Computational Media

Prof. Marc Davis & Prof. Peter Lyman UC Berkeley SIMS Tuesday and Thursday 2:00 pm – 3:30 pm Spring 2005 http://www.sims.berkeley.edu/academics/courses/is146/s05/ Computational Media IS146: Foundations of New Media

Lecture Overview • Assignment Check In • Assignment 3: Documenting Artifact Usage • Review of Last Time • Computation: Programmability • Today • Computational Media • Preview of Next Time • New Media On The Go and In The Home

Programming Concepts • Basic programming constructs • Parameters • Loops • Procedural abstraction • Subroutines • Conditionals

Making a “C” • to c • params [height] • make halfheight :height/2 • left 90 • forward :height • right 90 • forward :halfheight • right 180 • forward :halfheight • left 90 • forward :height • left 90 • forward :halfheight • end

Making an “A” • to a • params [height] • make halfheight :height/2 • left 90 • forward :height • right 90 • forward :halfheight • right 90 • forward :halfheight • right 90 • forward :halfheight • right 180 • forward :halfheight • right 90 • forward :halfheight • left 90 • end

Making an “M” • to m • params [height] • make diagonal (:height/2)*7/5 • left 90 • forward :height • right 135 • forward :diagonal • left 90 • forward :diagonal • right 135 • forward :height • left 90 • end

Making an “R” • to r • params [height] • make halfheight :height/2 • make diagonal :halfheight*7/5 • left 90 • forward :height • right 90 • forward :halfheight • right 90 • forward :halfheight • right 90 • forward :halfheight • left 135 • forward :diagonal • left 45 • end

Making “MARC” • to marc • params [height kerning] • m :height • space :kerning • a :height • space :kerning • r :height • space :kerning • c :height • end

Making a Circle of “MARC” • to marccircle • params [letterheight letterkerning] • make marcnamewidth ((:letterheight*5/2)+(3*:letterkerning)) • repeat 360/:letterheight • [marc :letterheight :letterkerning • hopback :marcnamewidth • right :letterheight • ] • end

Conditionally Making “MARC” Circles • to marccirclecond • params [letterheight letterkerning circletightness] • make marcnamewidth ((:letterheight*5/2)+(3*:letterkerning)) • ifelse (:circletightness=0) • [make rotation :letterheight] • [make rotation :letterkerning] • repeat 360/:rotation [marc :letterheight :letterkerning • hopback :marcnamewidth • right :rotation • ] • end

Devin Blong on Papert • Papert repeats the idea that when children are taught to program, they are more self directed and active. • “By contrast, when a child learns to program, the process of learning is transformed. It becomes more active and self directed. In particular, the knowledge is acquired for a recognizable personal purpose […].The new knowledge is a source of power and is experienced as such from the moment it begins to form in a child’s mind.” • This is juxtaposed with the idea that programming is a normal, rather than strange and foreign skill for a child to learn. Why then is this knowledge more powerful than other types of knowledge?

Devin Blong on Papert • What good is a language if it is not spoken? • How does the programming process of constantly debugging relate to communication metaphors? • What are some of the self-imposed barriers that keep technology from moving forward today (e.g., QWERTY) ? • How does lacking a “vocabulary” in a particular area affect your understanding and learning in that area?

Trevor Newhouse on Papert • Papert writes of turtle geometry as being mathetic in nature, or knowledgeable about learning. If the logo turtle can be characterized by such a term, is there a better one to describe a developing child’s mind. If logo, or computer programming is the logically definitive way to systematically learn, how then can we account for human instinctual preference and our version of “once removed” learning. Can a computer be programmed with these devices? Has it already?

Trevor Newhouse on Papert • Is syntonic learning deemed as enjoyable because, in the case of programming, it asks you to break apart what you already know, eliminate and clutter, and then build it back up cleanly? Could it be that there is some relaxing quality in this kind of, Cartesian spring cleaning?

Trevor Newhouse on Papert • The conversation Papert highlights between two kids programming a flower utilizes a great deal of “repeat” commands and storing. Psychologically, and anthropologically speaking, is this ability of a computer to instantly copy work that took a man hours or days to create ultimately a good thing? Are there enough sci-fi movies out there to give us pause?

Trevor Newhouse on Papert • Papert describes juggling using computation certainty, however in real life juggling there exist myriad environmental variables that can affect performance. Weather, hangovers, noise levels, slippery fingers… such factors act as unforeseen variables going into the total juggling experience. Are there such variables in computer programming?

What New Media Is Not Defined By • Digitized analog media • Media displayed on a computer • Random access media • Necessarily having less information than analog media • Necessarily being able to be copied without generation loss • Being “interactive”

Manovich on New Media • Numerical representation • Can be described formally (mathematically) • Can be manipulated algorithmically (programmability) • Modularity • Constructed out of substitutable components • Automation • Automation of media creation, manipulation, and access • Low level (bits) and high level (semes) automation • Variability • Media objects can have potentially infinite versions • Media database, separation of data and interface, customization/personalization, branching-type interactivity, hypermedia (links), periodic updates, scalability (e.g., resolution) • Transcoding • … Media => Data => Media …

AutoBuddy Example

Central Idea: Movies as Programs Content Representation Producer Parser Media Media Parser Media Content Representation • Movies change from being static data to programs • Shots are inputs to a program that computes new media based on content representation and functional dependency (US Patents 6,243,087 & 5,969,716)

Driver and Gunner play “Zone Raiders” a first-person driving shooter videogame The Gunner can only shoot in the direction the Driver drives The Driver cannot shoot The Driver and Gunner can talk to and hear each other over headphones, but cannot not see each other They both hear and see the same videogame output AutoBuddy films the Driver and Gunner with digital video cameras AutoBuddy computes a “buddy driving movie” from edited according to the patterns of conversation and game play between the Driver and Gunner AutoBuddy: A Computed Movie Buddy Driving Movie Camera Camera TV TV Gunner Driver Computer w/ video game AutoBuddy Software

How AutoBuddy Makes the Movie Synchronize & Crop Create Shots Dialog-based Cutting Add Credits • Driver, Gunner, and Video Game 3 Digital Movies (QuickTime) • Synchronize movies (AudioStreams) • Find beginning and end of game • Create 3 new movies: - Driver in car - Gunner in car - Both in car • Add parametric special effects • Cut between 3 new movies and game video based on who is talking • Cutting rules for continuity editing • Insert stills of Driver, Gunner

AutoBuddy Dialog-Based Cutting Driver Pause Gunner Gunner Both Gunner Pause time • AutoBuddy analyzes the Driver and Gunner audio to determine who is speaking at each point in movie • Produces a stream of speech events with durations and values (Driver, Gunner, both, or neither)

Dialog-Based Cutting Input Speech Events: Driver Pause Gunner Driver Gunner Output Video Cuts: • AutoBuddy uses a set of cutting rules that cut between shots based on patterns of speech events • Example: if Driver speaks and then Gunner speaks, show Driver and cut to Gunner slightly before Gunner starts to speak • Example: if there is a long pause between Driver and Gunner speaking, cut to the game video

AutoBuddy Composite Shots • Driver, Gunner, and Both shots are multi-layer composites • View out the car rear window is generated video games rear view mirror image • Flipped, scaled, smoothed, and placed • Back of car is a static image from a 3D model • Images of Driver/Gunner are generated by background subtraction • Front of car is a static image from a 3D model

AutoBuddy Special Effects • Car is shaken based on “gas pedal” • Gas pedal parsed from acceleration indicator in game video • Car and people are shaken 90 degrees out of phase • Gunfire art added to frames based on game audio • Audio Streams used to detect gunfire in game audio • Able to detect gunfire even when other audio effects present • Explosions • Game video analyzed to determine when explosions happen • Images are lightened and rumbled during explosions

Computation for Designing Artifacts • Four computational ideas/techniques from Carlo Sequin • Procedural generation • Parameterization • Optimization • Evolutionary power

Procedural Generation • Rather than creating artifacts directly, the user may design a generating program that will then generate the desired artifact • The empowering aspect of this approach is that the generating procedure will not just create the one artifact originally desired, but, with minor variations to the program, it can produce many different artifacts that may all fit a specified set of constraints or usage

Parameterization • For classes of frequently needed artifacts, the procedural generation mentioned above can be captured in a robust and more general program that contains a modest number of parameters that can be easily adjusted by non-programming users • A judicious selection and coupling of such parameters can enhance the likelihood that any arbitrary combinations of parameters still produce a meaningful output, although it may be far from desirable or optimal with respect to some specific application • However, the ease of modifying the parameter values and previewing the expected outcome, would allow even novice users to achieve better than average results obtained by un-aided users

Optimization • Given that the tedium of creating individual artifacts can be greatly reduced by procedural generation, users can explore a far larger space of possibilities than they could if they had to craft each artifact individually • This allows them to home in on a more optimal solution than they could by building a few prototypes • If the constraints and goal functions are well understood, then the generating program may contain its own evaluation loop that allows it to explore many options on its own and gradually converge towards a local optimum

Evolutionary Power • The ease of exploration afforded by the use of procedural generation permits an informed user to more clearly see and locate the boundaries of the paradigm captured in a generating program • By making these boundaries more visible, it also becomes more obvious to ask what lies beyond • Often such questions can be answered with a modest re-programming effort that enlarges the scope of the generator

Steven Lybeck on Benjamin

Mark Martell on Manovich • Why does Manovich carefully avoid using "digital media" and "new media" interchangeably? Are they, in fact, interchangeable? Is one a subset of the other? Can you think of examples of non-digital new media?

Mark Martell on Manovich • In discussing the principle of Variability, Manovich asserts that: "Every hypertext reader gets her own version of the complete text by selecting a particular path through it… New media objects assure users that their choices [thoughts and desires] are unique, rather than preprogrammed and shared with others." Yet this assertion seems to overlook the non-linear and unique consumption patterns of "old media" such as magazines and newspapers (who reads either linearly from front to back?), visual art work and photographs (does everyone begin and end at the same points on a photo?), even radio and television (consider how the individual often flips his attention back and forth between programs and other activities or thoughts). Furthermore, web logs may reveal just a few patterns of site consumption among visitors – and these may, in fact, be deliberate. Webmasters may want visitors to consume the site in a particular path order and may engineer the site to achieve this end. Does Manovich exaggerate in making a case for the principle that new media produces unique consumption patterns – and if so, why?

Mark Martell on Manovich • Manovich also goes on to refute several popular notions of unique properties of media. He critically examines notions that: • New media alone allows for random access. He refutes this point by claiming that Edison developed a random access media format (which was never mass produced). Does his refutation hold water in light of the overwhelming prevalence of non-random access storage (video, photos, film)? • Digitally-encoded media can be copied endlessly without degradation. Manovich notes, however, that the overwhelming majority of distribution of digital media is done in compressed, lossy format. Is his argument valid?

Readings for Next Time • Paul du Gay, Stuart Hall, Linda Janes, Hugh Mackay, and Keith Negus. Doing Cultural Studies: The Story of the Sony Walkman, London: Sage Publications Ltd, 1997, p. 7-41. • Discussion Questions • Will Avla • Hugh Mackay. Consumption and Everyday Life, London: Sage Publications Ltd, 1997, p. 259-309. • Discussion Questions • Alex

Readings for Next Time • Paul du Gay, Stuart Hall, Linda Janes, Hugh Mackay, and Keith Negus. Doing Cultural Studies: The Story of the Sony Walkman, London: Sage Publications Ltd, 1997, p. 7-41. • Discussion Questions • Willian Avila • Hugh Mackay. Consumption and Everyday Life, London: Sage Publications Ltd, 1997, p. 259-309. • Discussion Questions • Allen Lew

For Next Time Assignment 3: Observing Artifact Usage DUE

Computational Media