Wireless sensor networks for art & entertainment applications

Presentation Transcript

Center for Embedded Networked Sensing

April 18, 2003

Jeff Burke

UCLA School of Theater, Film and Television

jburke@hypermedia.ucla.edu

HyperMedia Studio Background

Founded in 1997 by film professor Fabian Wagmister.
Located in a re-assigned television studio and edit rooms in the UCLA Department of Film, Television and Digital Media.

Behind the Bars (1999)

Iliad Project Team (2002)

Goals

Invocation and Interference (2001)

Investigate the impact of emerging technologies on traditional production of theater, film and television.
Explore new work inspired and enabled by their unique capabilities and qualities.
Build tools that help creators, educators, and students explore
novel uses of instrumented environments.

Iliad Project Rehearsal(2002)

Arena

Installation artworks
Live performance

Educational spaces
Film production

Behind the Bars (1999)

Macbett (2001)

Outline

Background
Application examples

Live performance
Installation artworks
Film production

Research topics
Collaboration

Research
Curriculum

Ubicomp as a catalyst for creation

Ubiquitous computing technologies canmake action and presence in physical space relevant to computing, networks, and media.
Process
Presence
Context

Art as a laboratory for ubicomp

Controlled slices of the real world
Development / rehearsal process
need to author on-the-fly, in-the-field
Specific performance requirements different tradeoffs
Challenge of basic assumptions
different perspectives

Live performance

Responsive stage environments

Arizona State University’s Intelligent Stage (Lovell, et al.)
University of Georgia (Saltz)
MIT Media Lab (Sparacino, et al.)

Audience interaction / incorporation

Carnegie Mellon ETC
Blast Theory (UK)
Many other companies

Fahrenheit 451 (2000)

Macbett

Directed by Adam Shive, Interactive systems by Jeff Burke (2001)

Department of Theater subscription-series production.
Large-scale theatrical lighting and sound controlled based on actor movement, as sensed by an off-the-shelf wireless tracking system.
Team of one graduate student and four undergraduate programmers, as well as the entire design team, cast, and crew.

Macbett

Seven networked workstations and servers communicated via UDP for

Sensor management,
Feature calculation,
Lighting control,
Sound control,
“Authoring” of interactive relationships,
System monitoring.

Macbett

The Iliad Project

Architecture: Jeff Burke, Jared Stein / Research direction: Fabian Wagmister, Edit Villareal, Jose Luis Valenzuela

An ongoing research project that involves the simultaneous development of an original theatrical script, design and acting technique, and technology.
Emerging focus

Customization of the script and media based on the attending audience.

The Iliad Project

Core technologies

Audience database
Radio-Frequency Identification
Geographic Information System
Image capture and manipulation
Text processing
Live video streaming
Middleware – “Kolo”

Live performance

Sensor network research areas

Localization: robust, precise, and scalable
Actor (and audience) ‘state’
Middleware
Monitoring and realtime feedback
Authoring tools

lights

1

2

intensity

ML001

intensity

colorC

colorM

colorY

focus

pan

tilt

Middleware for experiementation

Lightweight, consistent access to inputs and outputs.
Very high-level abstractions can be counterproductive,unless they can be built by the author/creator.

lights.ML001.colorM

magenta level of moving light #1

tracking

actors

macbett

banco

position

x

y

z

First, make input and output devices on the network available in a consistent way.
Let authors define hierarchies and abstractions.

or

lights.ML001, sound.pb1.left

left

lights

sound

syncCue1

intensity

members

ML001

intensity

pb1

intensity

colorC

colorM

colorY

focus

pan

tilt

Synchronization of actions in the environmentwithout centralized show control.

‘group’

Managing connections

Support for constructing ‘real-time’ relationshipsbetween data sources and sinks.
Arbitration between competing relationships.

Kolo middleware

Java-based middleware API(device drivers in C/C++)
Scripting language

Specifying simple spatial relationships

specifying simple spatial relationships22

Installation artwork

Non-narrative participatory experiences
Without performance definitions of performer/audience
Extensions into media-rich educational spaces

Participatory simulations
Embodied and kinesthetic learning

hamletmachine (2000-2)

cheLA demonstration (2002)

Time & Time Again…

Fabian Wagmister and Lynn Hershmann (1999)

Distributed interactive installation exploring complex relationships between our increasingly interlinked bodies and machines.
Site-specific work in the Ruhr region of Germany.

Live video streamed into musuem.
Real-time video compositing.
Silhouette image choice controlled by sensors in the installation space.
Robotic, telematic doll streaming images to and being controlled by web users.
History database of video fragments recorded by web visitors.

Time & Time Again…

Fabian Wagmister and Lynn Hershmann (1999)

Behind the Bars

Fabian Wagmister (1999)

Confrontational interactive environment treating Latin America’s history of physical and intellectual oppression.
Premiere:
Central American Film and Video Festival in Nicaragua.

Media-rich educational spaces

Research areas

Adapt a simple wireless ADC platform?
Middleware
Data storage / mining
Space configuration
Authoring tools

Film production

Equipping the film set with a wireless sensor network to support all aspects of production through

Instrumentation
Observation
‘Augmented footage’
Decision support
Control

Developing collaboration with CENS faculty.

Sensor networks on the film set

Not just application… laboratory:
‘Articulated chaos’,
Complex coordination to generate small slices of controlled reality,
Tradition of precise documentation,
Repetitive action,
Rapid, large scale field deployments,
Rigorous test environment already available on our campus (over 100 student films per year).

Terminology

Script
Location
Scene
Shot
Take

Exterior, Day, Suburban Street

Closeup of Heather at the fruit stand

Three time frames

Editing in post-production

Filmmaking Application Drivers

Primary / initial
Asset management
Continuity management
Secondary / future
Real-time control
General post-production
Special effects

Platform

Wireless articulated imagers
Commercial x86 low-power architecture
Active tags
Low-cost, low-power tags providing RF and/or acoustic localization
Passive RFID tags
Primarily for asset management

Continuity Management

Ensuring repeatable action by the crew and talent.
‘Script supervision’ is facilitated by several ‘continuity assistants’.
Fine to coarse grained documentation.
Squirrel twitches her nose in take 1a, but not in take 1d.
Male bystander correctly on the woman’s right in both shots.

Continuity / Asset Management

A superset of asset management:
Must track on-camera and off-camera items that create the film camera’s field of view for a shot:
Presence and qualities of props, scenery, and actors.
Equipment positions and settings.

Needs and constraints

Localization with multiple sensing modalities
Synchronization
Privileged viewpoint
Event repetition
The Script – a priori scene knowledge

Synchronization
Event repetition

Eventually, must localize O(100) objects for continuity on a large set.
Need for coarse-grained localization for O(1000) objects for asset tracking only is apparent.
Would like to know more than just location: orientation, gaze direction in closeups, etc.

Synchronization
Event repetition

Data recorded must be ‘frame-accurate’, synchronized with the SMTPE time code of the shoot.
24 frames/sec for film.
29.97 frames/sec for NTSC video.
Reasonable but fine-grained synchronization of wireless sensor network to this external clock.

Synchronization
Event repetition

All that matters is what the film camera sees – it is the ‘privileged viewpoint’ of the system.
Camera and film stock parameters can be used to estimate what is seen and what isn’t.
With this information, manage:
focus of attention,
power consumption,
processing cycles,
bandwidth,
sensor articulation.

Synchronization
Event repetition

Need to track and take advantage of the repetitive action built into the process:
multiple shots of one scene,
multiple takes of one shot,
off-camera rehearsal.
Leverage filmmakers’ experience to develop heuristics for what action is ‘crucial’ and what is not.

Synchronization
Event repetition

Unlike many ubicomp scenarios, the film shoot has a carefully defined blueprint for the action that will occur: the script.
Could consider ‘script reading’ research, or…
Use the consistent format of film scripts to recognize who and what are (or will be, or was) present in a scene or shot.

Decision support example: the 180o rule

Decision support example: the 180o line

decision support example the 180 o rule46

A simple position-based metric for perceived discontinuity of scene layouts from multiple perspectives.
Well-suited for real-time visualization.
Important use of ‘coarser’ grained continuity management.

Application Driver for What?

Localization and identification…
using several sensing platforms, high resolution for O(100) objects, low resolution for O(1000).
Asset management and tracking…
with multiple action-dependency relationships, and across multiple time frames.
Embedding sensor network generated metadata into media…
to create augmented footage.
Scene observation techniques…
for recognizing discontinuities across multiple perpsectives.
Field decision support…
considering an arbitrary ‘privileged viewpoint’.