Digital Audiovisual Preservation Strategies

1. Digital Audiovisual Preservation Strategies October 21, 2011 Jimi Jones Adjunct Professor, Graduate School of Library and Information Science University of Illinois

2. Yeah, I work at the Library of Congress But today I’m not speaking for the Library of Congress. (I have to say that.)

3. Framing the Problem

4. Why Preserve AV Media? • AV materials are important parts of the cultural record • They can provide intimate, immediate records of important events • It is often difficult to predict what will have “value.”

5. Consider Home Movie Day…

6. Digitized vs. Born Digital Digitized = analog source information converted to digital information Born Digital = information was initially captured as digital and stays that way

7. Indiana University Media Preservation Survey • Nearly all of their half-million-plus audiovisual recordings are “actively degrading, some catastrophically.” • Nearly all are on obsolete formats • A high number of these recordings have high research value • “We have only a 15 to 20 year window of opportunity to digitally preserve” these recordings (emphasis mine) • At current 1:1 pace could take over a century – at least – to digitize their holdings

8. Challenges of Audiovisual Preservation • Formal AV preservation/digitization/cataloging knowledge is rare • Obsolescence • Formats are dependent on the market • Media collections are often siloed • Time is against us!

9. How Do We Mitigate Challenges? • Preserve originals (Walko’s presentation) • Create high-quality digital assets • Make informed decisions about sustainability • Migrate objects as necessary to maintain access • Describe objects in a structured way • COLLABORATE (and leverage existing expertise)

10. Preserve Originals

11. Why Do We Want to Preserve the Originals? • Often the best way to ensure image/sound quality • Originals may have value as artifacts in and of themselves. • As digitization technology advances, we may want to return to our originals and redigitize using newer technologies for better image and/or sound reproduction. • Preservation surveys can help us make target decisions about what to digitize and in what order

13. The Problem

14. How AvSAP Can Help

15. Create High Quality Digital Assets

16. A Few Words About Digitization and Digital Preservation • Digitization doesn’t mean you no longer need your originals. • All AV materials will have to be digitized at some point. • Digital preservation is an ongoing, active process. Not an end in itself. • I prefer the term “stewardship” to “preservation”

17. What is compression? Compression makes files smaller and easier to use In video, compression discards color information, resolution and, sometimes sound quality Once a file is compressed, that information cannot be recovered Compressed files are not considered archival quality

18. What is compression? Once a file is compressed, that information cannot be recovered (except J2K) Compressed files are not considered archival quality Lossless compression – file size is diminished without loss of information Lossy compression (most common) – file size is diminished by discarding information

19. Video Compression

20. Codecs Codec: Coder/Decoder –creates and/or plays back digital video and/or audio Also can stand for “Compressor/Decompressor” Unlike the audio world, video has no archival standard codec yet (JPEG200 is promising)

21. Codecs Creates the “bitstream” or “essence” of the file http://en.wikipedia.org/wiki/Category:Video_codecs http://en.wikipedia.org/wiki/Category:Audio_codecs

22. Some Common Video Codecs WMV (Windows Media Viewer) – proprietary DivX – proprietary Sorenson (used in Quicktime) – proprietary Xvid – open source Real Video - proprietary

23. Video Wrappers Contain the video essence information (the bitstream) Wrap the bitstream up with information that describes it and explains how to decode it Sometimes we say “file format” in place of “wrapper” – not entirely accurate but common

24. Common Video File Wrappers Quicktime MOV – proprietary; common on Apple platforms; well supported AVI (Audio Video Interleaved) – proprietary; common on Windows platforms; well supported Matroska – open source; free; support is growing MXF (Material Exchange Format) – defined by SMPTE, used in the broadcast industry and at LOC; support is growing

25. MPEG • MPEG-2 – standard devised by Motion Picture Experts Group • DVD encoding • Compressed • Used in broadcast • MPEG-4 – used online as downloadable video • very compressed • Used in iPod/iPhone/iPad video

26. JPEG 2000 Lossless video compression – discards no information (original data can be reconstructed from compressed data) Large file size becomes smaller when encoded as J2K Adoption is spotty but growing Has space for metadata Averages about 2.3:1 compression, depending on how “busy” the image content is

27. Digital Preservation Parallels Analog AV Preservation • Must preserve playback software alongside media files • Many institutions reformat their digital files • Masters and access copies • MXF-wrapped J2K – master • MOV H.264 15 Mbps – access copy

28. What the *&%$ Do We Do With Born Digital? • Keep a copy of the file in the format in which it was received (Internet Archive does this) • Transcode (change the file type) as necessary • Transcoding must be a very informed decision – can cause unnecessary “bloat” to files by inserting empty data (bigger files don’t necessarily mean higher quality)

29. What the *&%$ Do We Do With Born Digital? • File formats (wrappers) are rapidly evolving • How do we decide which formats are potentially long-lived and which are at risk?

30. Sustainability

31. What Kinds of Wrappers/Formats Do We Want? • Open standards • Widely supported • Doesn’t have patent or other kinds of restrictions to use • Ideally, something that is easily transcoded • Has minimal external dependencies – on particular software and/or hardware

32. Library of Congress Formats Sustainability Site

33. First Type of Evaluation Factor – Sustainability Factors

34. Second Type of Evaluation Factor – Quality and Functionality Factors

35. Digital Media Preservation • Must be part of an ongoing preservation workflow • There is no end to digital preservation (or any preservation) • Checksums can look at the data in a file and see if there is something missing or corrupted

36. Migrate Assets

37. Migration • Move file information from older version of a file format to a new one • Typically involves moving a large number of old files to a smaller number of contemporary formats • Usually about a 5-10 year migration schedule

38. Describe Assets

39. Access and Preservation • There is no preservation without access • and vice versa

40. Thoughts About Metadata Information ABOUT Information Organizes and describes objects Metadata can (and should) conform to schema, which conform to standards and can be format-specific MAKES OBJECTS FINDABLE

41. A World With Minimal Metadata

42. A World With Rich Metadata

43. Collective Access

44. Collaboration

45. Think Collaboration • Try to cut costs by working with other institutions • Consider working and thinking as “consortia” • Indiana University’s work: http://research.iub.edu/communications/media_preservation/iub_media_preservation_survey_FINALwww.pdf

46. Thoughts About Digital Materials and Preservation Consider your digital storage solution RAID drives back up information and prevent total data loss Many larger collections are using tape-based long-term storage Must preserve playback equipment to digitize media in-house

47. Thoughts About Digital Materials and Preservation Physical storage media for digital files must still be preserved (disks, hardware, tapes) Consider the LTO tape – a commonly-used storage medium that is based in magnetic tape

48. And…. The Future?

49. The Future of Digital Video Stewardship? preserving our analog originals long enough to digitize them the “fire hose” of born-digital materials isn’t slowing – it’s increasing do we compete with YouTube, Vimeo, etc or collaborate with them? Assets can exist anywhere

50. The Future of Digital Video Stewardship? Moving from “media” environments to “IT” environments Librarians, library students, people who can find and describe objects become more important Linking data becomes critical What can we automate or do en masse?