Lecture 13: Computational Media Theory

1. Lecture 13: Computational Media Theory IS246Multimedia Information (FILM 240, Section 4) Prof. Marc Davis UC Berkeley SIMS Monday and Wednesday 2:00 pm – 3:30 pm Spring 2003 http://www.sims.berkeley.edu/academics/courses/is246/s03/

2. Today’s Agenda • Review of Last Time • Theory of Computation • Computational Media Theory • Manovich on New Media • Bloch on Representing Film • Dorai and Venkatesh on Computational Media Aesthetics • Discussion Questions • Action Items for Next Time

3. Computation in Intellectual History • Computation as instrumentality • PCs, PDAs, embedded processors, etc. • Computation as ideas • Modeling process • Languages for modeling process • Primitives, combination, abstraction • Parameterization • Black boxing functionality • Optimization

4. Algorithms and Programming • Algorithm • A step-by-step description of a procedure to achieve a desired result • Programming • Primitives • Means of combination • Means of abstraction

5. Computation for Designing Artifacts • Four computational ideas/techniques from Carlo Sequin • Procedural generation • Parameterization • Optimization • Evolutionary power

6. Today’s Agenda • Review of Last Time • Theory of Computation • Computational Media Theory • Manovich on New Media • Bloch on Representing Film • Dorai and Venkatesh on Computational Media Aesthetics • Discussion Questions • Action Items for Next Time

7. What New Media Is Not Defined By • Digitized analog media • Media displayed on a computer • Random access media • Necessarily having less information than analog media • Necessarily being able to be copied without generation loss • Being “interactive”

8. Manovich on New Media • Numerical representation • Can be described formally (mathematically) • Can be manipulated algorithmically (programmability) • Modularity • Constructed out of substitutable components • Automation • Automation of media creation, manipulation, and access • Low level (bits) and high level (semes) automation • Variability • Media objects can have potentially infinite versions • Media database, separation of data and interface, customization/personalization, branching-type interactivity, hypermedia (links), periodic updates, scalability (e.g., resolution) • Transcoding • … Media => Data => Media …

9. Today’s Agenda • Review of Last Time • Theory of Computation • Computational Media Theory • Manovich on New Media • Bloch on Representing Film • Dorai and Venkatesh on Computational Media Aesthetics • Discussion Questions • Action Items for Next Time

10. Bloch “From Concepts to Film Sequences” • Describing shots • ACTION performed by the actors • SETTING • ACTORS • Direction of the LOOKs of the main actors • POSITIONs of the actors in the frame • Direction and speed of the apparent MOTION of the main actors or objects in motion • Constructing sequences • CHOICE • CONSTRUCTION • When APPRECIATION bad, CORRECTION • PROJECTION

11. Today’s Agenda • Review of Last Time • Theory of Computation • Computational Media Theory • Manovich on New Media • Bloch on Representing Film • Dorai and Venkatesh on Computational Media Aesthetics • Discussion Questions • Action Items for Next Time

12. The Semantic Gap • “However, the semantic gap between the rich meaning that users want when they query and browse media and the shallowness of the content descriptions that we can actually compute is weakening today’s automatic content-annotation systems.”

13. Computational Media Aesthetics • “ […] the algorithmic study of a variety of image and aural elements in media (based on their use in film grammar). It is also the computational analysis of the principles that have emerged underlying their manipulation in the creative art of clarifying, intensifying, and interpreting an event for an audience.” • “Our research systematically uses film grammar to inspire and underpin an automated process of analyzing, characterizing, and structuring professionally produced videos.”

14. CMA Challenges • Can we dynamically detect successful aesthetic principles with accuracy and consistency using computational analysis? • Can we build new postproduction tools based on this analysis for rapid, cost-efficient, and effective moviemaking and consistent evaluation? • How can we use these successful audio–visual strategies for improved training and education in mass communication? • How do we raise the quality of media annotation and improve the usability of content-based video search and retrieval systems?

15. Today’s Agenda • Review of Last Time • Theory of Computation • Computational Media Theory • Manovich on New Media • Bloch on Representing Film • Dorai and Venkatesh on Computational Media Aesthetics • Discussion Questions • Action Items for Next Time

16. Discussion Questions • On Manovich on New Media (Lisa Wang) • Manovich wrote, "Before, we would read a sentence of a story or a line of a poem and think of other lines, images, memories. Now interactive media asks us to click on a highlighted sentence. In short, we are asked to follow pre-programmed, objectively existing associations. Put differently... we are asked to mistake the structure of somebody else's mind for our own." (p. 314 of the reader) • When viewing a piece of media, each person accesses their individual repertoire of associations and connotations. Despite this subjectivity, there are many associations that are shared within a culture. Filmmakers draw upon these shared associations to tell their stories. Could a web or database of common associations and connotations be built? How could it be used in conjunction with semantic-based editing programs?

17. Discussion Questions • On Manovich on New Media (Phoebe Liu) • Through out the article, the author talks about one general tendency of new media that it favors individuality over conformity and the switch from mass standardization to individual customization. This means that the user is given many options to modify the performance of a program or a media object. In terms of designing multimedia information systems, say if we want to design a automatic editing program for some home videos, what are the important factors we will be looking for in order to achieve individual customization? 

18. Discussion Questions • On Bloch on “From Concepts to Film Sequences” (David Warthen) • In describing “the program,” Gilles refers to the program’s operators (e.g., CHOICE, CONSTRUCTION, etc.) “recognizing” and “examining” aspects of the film content in order to generate sequences. • Assuming that these operators are working with human-generated markups of the film content, what would this markup need to look like? In particular, considering our discussion of algorithms in the last class, what level of detail would be necessary to describe the shots so that the software could reasonably generate the sequence? • Assuming that the software itself is analyzing the content performing the “recognition” operations, what capabilities might this software require? • Gilles’ examples at the end relate to “live” film sequences, such as documentaries and news broadcasts. However, the state-of-the-art of computer generated film or video content has advanced greatly in recent years (e.g., Monsters Inc., Final Fantasy – The Spirits Within). How might the ability to generate content using such techniques extend Bloch’s ideas and system? • Bloch is pursuing greater automation in the generation of film. What is the possible downside if such systems go into wide deployment? (That is, what new problems might such systems create?)

19. Discussion Questions • On Dorai and Venkatesh on Computational Media Aesthetics (Lily Chen) • How can we deal with the • Ambiguous factors which are rooted in aesthetics • Factors which may come from personal bias when we do computational media analysis?

20. Today’s Agenda • Review of Last Time • Theory of Computation • Computational Media Theory • Manovich on New Media • Bloch on Representing Film • Dorai and Venkatesh on Computational Media Aesthetics • Discussion Questions • Action Items for Next Time

21. Readings for Next Time • Wednesday 03/12 Guest Lecturer: Jonathan Foote from FX Pal on “Automated Media Analysis for Audio” • Required • Jonathan Foote. An Overview of Audio Information Retrieval. Multimedia Systems, vol. 7, 1999; pp. 1-18. • Kenichi Minami, Akihito Akutsu, Hiroshi Hamada, and Yoshinobu Tomomura. Video Handling with Music and Speech Detection. IEEE MultiMedia, vol. 5, 1998; pp. 17-25. • Optional (On IS246 Course Web Site) • Christian Spevak and Emmanuel Favreau. Sound Spotter - A Prototype System for Content-Based Audio Retrieval. In: Proceedings of the 5th International Conference on Digital Audio Effects (DAFx-02). Hamburg, 2002. • George Tzanetakis, Perry Cook. Automatic Musical Genre Classification of Audio Signals. In Proc. International Symposium for Audio Information Retrieval (ISMIR 2001) Bloomington, USA, October 2001.