digital longterm archiving
Our desire for infinitely durable carriers and never changing formats for our archive holdings will not be fulfilled now or in the foreseeable future. The only option for preserving our audiovisual holdings is therefore the concept of "eternal migration". This means that the recordings must be migrated an infinite number of times, from carrier to carrier, from format to format, in order to preserve them.
This can only work if these migrations get carried out absolutely loss-free. Even with the slightest loss during these processes, the recordings lose more and more quality with the number of migrations. Images, sounds and data disappear. Our collections ultimately get lost.
The task of digital long-term archiving is to create the necessary conditions to ensure that this does not happen.
In the 20 years that I have been working at the Austrian Mediathek, together with Peter Bubestinger I have developed a system for managing video digitisation workflows (DVA-Profession, see below) , which has been successfully in use there since 2010. As a further innovation, we designed and implemented the hardware and software-optimised storage system for archiving purposes (DLP, see below).
DVA-Profession
In 2009, I had the opportunity to finally start digitising video as part of a project in which the Austrian Mediathek was to put 1000 video recordings on the web and digitise them as a prerequisite. I began to research the technical possibilities. At that time, there was only one format on everyone's lips in the international circle of AV archives for archiving video besides the uncompressed option: MXF (container) with Motion JPEG 2000 (codec)
The only system I knew of at the time that managed a digitisation workflow (metadata, checksum, analysis, packaging for the archive, etc.) and wrote such a format was "SAMMA Solo".
I had therefore included the purchase of such a device in the calculation for the project.
After the project had actually been approved, more detailed tests with original files from the Samma-Solo of the Austrian Phonogrammarchiv made me realise that I could only play back or convert these files on the original device.
This is a "no go" for any idea of long-term archiving.
In the search for alternatives, the open source codec FFV1 proved to be a sensible solution.
Since this codec was not served by any digitisation system, Peter Bubestinger and I developed our own system called "DVA-Profession".
Marion Jaks originally joined the team as a tester and from then on enriched the concept with down-to-earth suggestions for easier-to-understand operating functions.
DLP-Storage
The question of storage costs plays an existential role for the budget of an archive, especially when archiving digital video assets.
These costs are divided into hardware costs for physical storage and costs for the software that manages the data. Many solutions offered on the market are only available as a package of both areas.
At the same time, data management is often sold with volume-based licences.
Providers also primarily serve the market sector where large budgets are available (banks, insurance companies, broadcasters). Here, high availability is an important criterion. This cost-driving characteristic is less important for average audiovisual archives. A temporary outage is easier to cope with there. Any measure that ensures that the data is available again completely and unchanged after a restart is more important.
In order to reduce the costs to the features that are important for the archive, Peter Bubestinger and I developed the configuration for a cost-effective yet stable hardware solution for the Austrian Mediathek in collaboration with the companies Linforge, Anyfer and 45Drives. For the administration, we developed a concept that took into account all known requirements for digital long-term archiving - including the option of replacing it with a future system.
The costs have been greatly reduced as a result. Previously, it was not realistic to even think about a petabyte for budgetary reasons.