Labor Department launches digital snapshot project
If a picture is worth a thousand words, how do you calculate what a snapshot of a 1.6 million-page Web site is worth?
The Labor Department doesn’t claim to know the answer, but it has resolved that its Web site is valuable enough to create a comprehensive snapshot for posterity.
The effort is aimed at providing greater depth and accessibility to the department’s Web pages than a similar, but much broader, project being led by the Library of Congress, said Patrick Pizzella, chief information officer and assistant secretary for administration and management at the Labor Department.
The Library of Congress and several partners embarked on a project that began in September to preserve a snapshot of U.S. government Web sites as the Bush administration was drawing to a close. The group, which includes the California Digital Library, the University of North Texas Libraries, and the Government Printing Office, commissioned Internet Archive to create a permanent copy of federal Web sites.
But the massive effort to crawl, capture and index between 300 million and 400 million government Web pages, each with associated images and files, will take months to complete. As Pizzella and Labor’s Webmaster Paul Craven discovered, there are limitations on what will be captured, the timing of the capture and the extent to which users can search the pages.
In the LOC project, agencies have no control over when their sites get crawled; DOL’s snapshot gives the department control over dates and areas on the site. DOL’s snapshot also will allow users to search for key words and text, in addition to URLs and domains.
DOL’s Web sites, like those at most federal agencies, have expanded significantly in recent years. The sites currently host roughly 1.6 million pages of information, Craven said, and drew 117 million visitors and 521.6 million page views in fiscal 2008. The volume of traffic has been growing about 9 percent annually over the past three years, he said.
“When we found out [the National Archives and Records Administration
] wasn’t going to be archiving [government Web sites] this year,” Craven said, a team was tasked to find ways to capture a wide range of programs that had evolved on DOL’s sites.
“Our first approach was to see if we could do this in-house,” he said. “We looked at several open-source applications. What became apparent was they didn’t include all the capabilities … and didn’t provide the kind of support we needed.” And many would place unacceptable strains on the department’s Web systems, Craven said.
Craven and project manager Mark Randazzo eventually learned about the Library of Congress’s work with San Francisco-based Internet Archive and an on-demand service it offers, called Archive-It.
Internet Archive is a non-profit organization that has gained notoriety for its efforts to collect and preserve historical snapshots of the World Wide Web. Founded in 1996, it has archived more than 115 billion URLs and 3 petabytes of data, said Kristine Hanna, director of Web Archiving Services at Internet Archive. The Archive-It service evolved from those efforts.
The service agreement, announced by DOL late last week, lets the department archive up to 5 million Web pages and related documents over the course of a year, at a subscription rate of less than $10,000 a year, Craven said.
Internet Archive is also working separately with the Energy Department, the National Institutes of Health and the National Library of Medicine, Hanna said.
Behind the scenes, Internet Archive uses Heritrix, a Web crawler that can be controlled to capture digital content behind firewalls, and Wayback Machine, an access tool that reconnects and renders the content, Hanna said.
Archive-It also uses NutchWAX, open-source search software that can search the text of collected material and display the results, similar to traditional search engines. Hanna said that Archive-It’s search capability will allow it to collect more material in the DOL snapshot than will generally be the case for the Library of Congress’s broader government snapshot.
The DOL snapshot will be hosted and maintained by Internet Archive, but will appear to the public on DOL’s Web site, and as part of its Wirtz Labor Library collection at www.library.dol.gov. All pages will have a disclaimer at the top indicating the page is an archived copy. Copies of the documents will also be available to help meet NARA records management requirements, Craven said.
The snapshot is expected to be completed and initially available by the end of the month. More information on the Web Snapshot Initiative at DOL is available at www.dol.gov/Archive.