As libraries become more involved in managing digital content, long-term digital preservation has become a crucial issue. While it is challenging enough to develop digital collections that fit within systems providing convenient access to library users, it's even more difficult to implement processes that guarantee long-term preservation of these materials. Libraries play an important role in ensuring that future generations have access to the artifacts of our cultural heritage. Books, manuscripts, and other physical media have been preserved within libraries for centuries, but in recent years, a tremendous volume of information has been transferred into digital formats. Much of the new work produced today is created digitally. The ever-increasing volume of digital content managed by libraries is creating a need for trusted digital repositories that provide confidence for long term preservation.
The challenge of long term preservation is closely tied to the type of media we use to store information and the rapid shifts in the technology behind it. Information on physical media like paper has proven to be quite durable when properly managed. In favorable environmental conditions, paper materials will endure for many centuries. CDs and DVD's, on the other hand, may last a couple of decades at best. Tapes and disk systems become obsolete after a just a few years. While the actual digital content may not deteriorate, the equipment needed to access the content on any type of media may not be available to future users.
The quest to preserve digital content into the distant future requires an active process that can overcome the transient nature of technology. We must assume that any given digital object will need to be migrated through many different media types and hardware platforms over time. We must also be prepared for the obsolescence of the basic formats in which digital content may reside. Library users several decades from now will not have the same versions of word processors and document viewers that exist today. Even formats that we consider standard today like TIFF, JPEG, PNG, and MPEG could eventually become extinct. Access to the content might depend on migrating formats, or providing the means for future systems to emulate those from the past.
In a major advancement in the digital preservation arena, the National Library of New Zealand, Ex Libris, and Sun Microsystems have developed a commercial product that addresses much needed requirements for long-term preservation of digital collections.
The National Library of New Zealand has been actively pursuing a strategy for the long-term preservation of its digital collections and has created the National Digital Heritage Archive, or NDHA. The content of the NDHA repository includes items submitted to the library for legal deposit, Web harvesting, content from the library's digitizing programs and other items in digital form donated to the library.
Beginning in 2004, the National Library of New Zealand began a process to develop detailed requirements and specifications to address its digital preservation needs. This process was completed in 2006 and led to the search for a vendor to create a product that addressed these requirements. The Library entered into the partnership with the expectation that a commercial product would be developed. If it developed a digital preservation system independently, it would have the ongoing responsibility for continued development and enhancement. By working with a commercial company, the library expects ongoing development and support to be provided by the company as it takes the product forward for itself and other customers.
The National Library of New Zealand signed an agreement with Endeavor Information Systems in August 2006 to develop a system that would address their needs in long-term digital preservation. Ex Libris acquired Endeavor in December of 2006. At the time of the acquisition, Ex Libris and the National Library of New Zealand began discussions on how to complete the project. Options included extending the Ex Libris DigiTool digital asset management system to meet the requirements or to develop a new product. In June 2007, the parties agreed to a partnership that also includes Sun Microsystems, who will develop a new product to address the library's requirements for digital preservation of its National Digital Heritage Archive.
The development of the project involved a peer review group of international experts from institutions including British Library, Cornell University Library, the Getty Research Institute, Helsinki University Library, Royal Library of Denmark (Koninklijke Bibliotheek), National Library of China, Singapore National Library, the University of Glasgow, and Yale University.
The Open Archival Information System, OAIS, stands as the primary international standard in digital preservation. OAIS provides a reference model that can be followed to create a trusted digital archive. Developed by the Consultative Committee for Space Data Systems at the request of International Standards Organization (ISO), OAIS has become well established as the ideal approach for long-term digital preservation. OAIS projspecifies the workflows involved in submitting items into an archive, storing items within the archive, and extracting items for the purpose of access. It describes the layers of metadata required, the validation that must be performed at each step. An OAIS compliant archive ensures all the transformations or emulations needed for each package to be accessed in the future. The model provides guidance on the myriad of issues involved in operating a trustworthy archive, like the number of copies needed for each package and how they should be distributed.
The agreement between Ex Libris and the National Library of New Zealand established as a requirement that the system produced be compliant with the OAIS reference model. The partnership also involved Sun Microsystems. The Digital Preservation System relies on very sophisticated storage technologies, and Sun Microsystems has developed storage systems capable of meeting the requirements of an OAIS compliant preservation system, with very large-scale capacities and with extremely good performance while maintaining data integrity. These systems include both disk and tape components. The StorageTek 990 from Sun, which can scale up to 330 terabytes and the Sun StorageTek Storage Archive Manager software provide some of the underlying infrastructure for the NDHA.
The software developed by Ex Libris provides the interfaces needed for submitting content into the archive, managing the internal operation of the archive, and providing access to the content in the archive. The Ex Libris Digital Preservation System maintains extensive metadata relating to the content in the archive which can be harvested for the purpose of search and display. The design of the system allows tools such as the company's Primo product to be use to provide access to the digital objects held in the repository.
The National Library of New Zealand put DPS into production on October of 2008. The general release of the product will be available to other libraries beginning in December 2008. A second phase of development is ongoing with completion expected in 2010.
Another interesting component of the NDHA project is the creation of the Web Curator to assist with the archiving of content from Web sites for the purpose of preservation. One can use the tool either to selectively harvest targeted content, or to comprehensively harvest all Web content within top-level domains. This component was developed jointly with the British Library and made available to the general public without cost as an open source project. The programming for the tool was contracted to a commercial company.
The Digital Preservation System targets national, research, and academic libraries involved with large amounts of digital content. Many of these libraries struggle to provide adequate safeguards for their collections in the short and medium term, much less have the infrastructure needed to guarantee their preservation for posterity. The Digital Preservation from Ex Libris provides an off-the-shelf system to meet this need, following the rigors of the OAIS reference model.