Preserving Digital Public Television ptvdigitalarchive/

1. Preserving DigitalPublic Televisionhttp://www.ptvdigitalarchive.org/ Howard Besser NYU Moving Image Archiving & Preservation Program http://besser.tsoa.nyu.edu/howard http://www.nyu.edu/tisch/preservation/ 1

2. NYU/Public Television Project • $6 million project -- 50% from LC/NDIIPP • Marry asset management to preservation • Preserve a broad set of elements (including ancillary material) • Life-cycle mgmt (add metadata as soon as a clip comes in) • Establish a community of stakeholders, working together for preservation (stations, university, librarians, journalists, historians, producers, scholars, …) • Build an OAIS Server • Explore appropriate file formats, wrappers, METS extensions • Develop sustainable business model 2

3. Project Partners • Thirteen/WNET & WGBH – Content and production expertise • The two largest television stations in the PBS system • Together produce largest percentage of national programs • Both have preservation Archives • Public Broadcasting Service – More content and network design • Distributes most of the national programming • Determines and keeps ‘broadcast’ versions • New York University – Facilitation and Resources • Leadership in designing digital libraries • Experience in process for setting standards • Has new Masters Program in Moving Image Archives 3

4. We were already collaborating to solve shared technical problems • The public television partners have been working together for a long time on common issues -- such as digital asset management, and a metadata dictionary. • Digitally-produced programs are at great risk of being lost --- because of the rapid changes in technology, lack of funds, and no preservation mandate. • Expanding our efforts to encompass preservation was a natural extension of our progress in standardizing a complex digital production and broadcast environment. 4

5. Public Television was analog for a long time 5

6. PBS Remote Tape Storage Facility 6

7. Computer-printed index of1955-69 WNET Holdings at LC 7

8. Public Television has becomeDigital for Production & Distribution • though some things still need to be streamlined 8

9. Programs are shot, assembled and edited digitally. Completed programs are packaged as digital files, often with many different elements. Typical AVID Editing Suite The same huge video files are digitized over and over again for different uses. A lot of video is not used for broadcast, but goes to the internet, to DVDs and to other media.

10. Local broadcasting is nearly all via digital systems Thirteen Master Control manages one high definition, two analog and three digital over-the-air broadcast channels. 10

11. Broadcast playback is from a digital server- no more tapes or tape machines ( All Thirteen broadcast programs are played from this server. It has to talk to the broadcast automation system, the satellite system, and other in-house and external networks. It is the size of a window air conditioner and holds 1700 hours of material. 11

12. Prior Work:PBCore • 2+ year collaborative effort • 48 elements based on Dublin Core • Intellectual content of a media asset or resource -- 13 elements • IP-creation, creators, and usage limitations of a media asset or resource -- 7 elements • Instantiation (in either digital and/or analog) -- 28 elements • http://www.utah.edu/cpbmetadata/ 12

13. Challenges of Digitalbrought the partners together • Formats & Versions issues (including variant forms of content, HD/SD quality, …) • Agreements on technical standards for handing off files from one organization to another • Repository design for long-term sustainability 13

14. Project activities include • Completing an inventory of at-risk materials to better quantify our holdings and prepare for selection • Reviewing best practices and most up-to-date developments in the field of video archiving • Conducting facilitated discussions on key topics to guide setting standards and policies • Establishing an Advisory Committee to assist with selection criteria • Ingesting sample materials and testing the repository • Presenting regular reports to public broadcasting and moving image archive community for ongoing feedback 14

15. Project Focus Areas • Appraisal and Selection – developing criteria and standards for what to preserve and by whom • File Formats and Packages – determining the best formats for our various uses, plus testing the suitability of file “packaging” for long term preservation • Metadata and Related Topics – specifying technical, descriptive and rights information • Repository Design – technical architecture, administrative policies and potential business models • Sharing Our Findings – Keeping the public broadcasting community involved and informed all along the way 15

16. Collaboration has been very productive • Now have standards that allow files to flow digitally from producer to PBS to stations • Studied various metadata schemes to zero in on what’s needed for preservation • Inventoried and located programs in non-regular locations • “Wrapper Roundtable” & aftermath • PBS & LC beginning to collaborate • The “American Archive” 16

17. Our Talks today • Repository Design (including Dspace and SRB)--James Bullen • Pushing metadata-gathering upstream, gathering it during the program production process--Kara Van Malssen 17

18. Preserving DigitalPublic Televisionhttp://www.ptvdigitalarchive.org/ 18

19. Pushing Metadata Upstream into the Content Production Process: Preliminary Studies of Public Television Howard Besser and Kara Van Malssen New York University 19

20. The Problem TRADITIONALLY… • Very little metadata required for preservation accompanies an object to a repository. • Archives, libraries and other repositories must create (or re-create) most of the necessary metadata. • This requires many manual hours, and significant resources - both time and money. IN THE DIGITAL WORLD… • Repositories will be unable to continue in this manner, as more metadata than ever is required. 20

21. “Once the (television) program is finished, it is passed on to the archive or library for safe keeping. Librarians will catalog and classify the content, possibly using a proxy copy, and enter the resulting informative metadata in their database so they can retrieve it in the future. However, rarely if ever is the metadata from the rest of the process passed onto them, except, perhaps, for the title, tape number, and basic technical information about recording formats. It has to be re-created, with all the associated risk of errors and lack of accuracy--not to mention the work and time involved.” - Cox, Tadic, and Mulder, Descriptive Metadata for Television (2006) 21

22. “The necessary or additional metadata cannot be effectively and satisfactorily produced either as an afterthought post-production process on the publisher’s side or as a pre-ingest conversion activity at the archive’s end. Approaching e-archiving in this fashion leads to distribution delays and a more complex production and distribution scenario, with all the accompanying potential to introduce production delays and errors.” - Yale University, YEA: The Yale University Archive, One Year of Progress, 2002 22

23. Solutions…? • Preservation becoming a shared responsibility between content creators, distributors, curators, and preservationists. • Partnerships are needed to come to unified solutions. • Preservationists seek reliable metadata back upstream in the production workflow... 23

24. Site Visits This report is based on workflow studies at public television stations, between June and August, 2006 by NDIIPP Research Assistants Caroline Rubens, Paula Felix-Dider, and Kara Van Malssen. Workflow report completed in September 2006. Additional insight was gained through metadata studies conducted by Mary Ide and Leah Weisse, Archivists at WGBH. 24

25. Site Visits • WGBH, Boston - June 19-20, 2006 • Interviewees included Archive and Media Library staff, members of Frontline, NOVA, and American Experience production units, and legal dept staff 25

26. Site Visits • WNET/Thirteen, New York - July 18 and August 2, 2006 • Interviewees included the Archivist, members of Broadway: The American Musical production unit, Broadcast Operations, and Broadcast Technology staff. • WNET/Thirteen, Washington DC - August 15, 2006 • Religion and Ethics production unit staff interviewed. 26

27. Public Television WorkflowBasically similar to workflows in other fields… 27

28. Recent Workflow • Hybrid digital/analog process. Production and post-production = file-based. Distribution and archiving = tape-based. • Programs and production materials are turned over to the archive after the production unit no longer has a need for them, or a place to store them. • Metadata gathering starts at this point, or sometime even further in the future. 28

29. Changes… • Public television (PTV) moving toward file-based workflow. • Distribution from PBS will soon be by the Next Generation Interconnction System (NGIS) = real and non-real time distribution of program files. • Largest stations (WNET and WGBH) implementing digital asset management (DAM) systems. • PTV working with the broadcast industry on the development of file wrappers for post-production, distribution, and preservation. 29

30. For digital preservation, this shouldn’t be the only place for metadata in the preservation workflow! This is far too late in the cycle! + 30

31. METADATA METADATA It also needs to be here! 31

32. As the workflow becomes file-based, the need for robust and accurate metadata will become critical. File relationships, video codecs, bit rates, and rights information must be explicit, accurate, and immediately accessible. This will require a much deeper level of metadata than is currently captured in tape-based archives. Obtaining the necessary metadata at the end of production and broadcast life cycle is not feasible. Metadata will need to be systematically gathered during the production lifecycle and submitted with the programs to the preservation repository. 32

33. Potential Points of Metadata Capture 33

34. Potential Points for Metadata Capture • Much of the necessary metadata for preservation is already generated by the production unit, but discarded after their internal use. This needs to be captured throughout the workflow. • “Those in the production unit are the creators and have first hand knowledge of who, what, where, when, and why the content was created.” -- Mary Ide and Leah Weisse, WGBH Archivists. 34

35. Creating Partnerships • It is in everyone’s interest to see that the content is saved. • Changes to the production workflow should be minimal and not disruptive. • Implementation of DAM is the perfect time to begin pushing for changes in metadata delivery. • This should provide immediate benefits to the producer’s work, as information and assets will be easier to find and manage. 35

36. Creating Partnerships WGBH AS A MODEL: • At production startup, the station’s archive provides templates and check lists for eventual program and metadata delivery. • Production units must deliver completed templates and all related materials before final payment. • Production team completes File Maker Pro databases (using templates supplied by archive) for original footage, stock footage, stills, and materials used in the final edit. 36

37. Creating Partnerships WGBH MODEL… • Metadata collected not enough for digital preservation, but important lessons have been learned: • Gathering metadata from one system and trying to use it for maintenance on another introduces QC and conversion problems (e.g. Users and archives have different version of File Maker Pro). • Extensive QC required because metadata coming from production units is often incomplete, inconsistent, or incorrect. Content creators need to be trained as well as convinced of the importance of metadata. 37

38. Creating Partnerships CONVINCING CONTENT CREATORS… • Create a clear financial incentive. DAMs save money (on stock footage and shooting) when editors and producers can search and retrieve station-owned assets right from their desk. • Mandate from funding agencies, such as Corporation for Public Broadcasting (CPB) that preservation planning must be part of the initial proposal • This is already happening in the sciences. NSF and NIH require long-term data sharing and preservation plans in research proposals. 38

39. Further Studies • Test of effectiveness for suggested methods of metadata capture. • Detailed study of metadata lifecycle throughout production and distribution workflow. • Look into questions of accuracy: • How can we be sure the metadata is reliable? • What methods are there to check the accuracy of metadata capture? 39

40. Conclusion • In order to develop systems to capture more metadata during the production process, producers and curators will need to work closely together. • Curators need to be sensitive to creator’s needs, and mindful of politics. • Producers must be trained in creating metadata according to standards, and will need to realize the necessity of producing metadata that can be utilized by others. • Many fields are working on this problem: social science data archives, e-journals, etc. • Ways of achieving the goals of upstream metadata capture should be shared between archivists and curators in all fields. 40

41. Preserving DigitalPublic Televisionhttp://www.ptvdigitalarchive.org/ 41