Wireless sensor networks for art entertainment applications
Download
1 / 51

Wireless sensor networks for art & entertainment applications - PowerPoint PPT Presentation


  • 456 Views
  • Updated On :

Wireless sensor networks for art & entertainment applications. Center for Embedded Networked Sensing April 18, 2003. Jeff Burke UCLA School of Theater, Film and Television jburke@hypermedia.ucla.edu. HyperMedia Studio Background. Founded in 1997 by film professor Fabian Wagmister.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Wireless sensor networks for art & entertainment applications' - Jims


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Wireless sensor networks for art entertainment applications l.jpg

Wireless sensor networks for art & entertainment applications

Center for Embedded Networked Sensing

April 18, 2003

Jeff Burke

UCLA School of Theater, Film and Television

jburke@hypermedia.ucla.edu


Hypermedia studio background l.jpg
HyperMedia Studio Background

  • Founded in 1997 by film professor Fabian Wagmister.

  • Located in a re-assigned television studio and edit rooms in the UCLA Department of Film, Television and Digital Media.

Behind the Bars (1999)

Iliad Project Team (2002)


Goals l.jpg
Goals

Invocation and Interference (2001)

  • Investigate the impact of emerging technologies on traditional production of theater, film and television.

  • Explore new work inspired and enabled by their unique capabilities and qualities.

  • Build tools that help creators, educators, and students explore

  • novel uses of instrumented environments.

Iliad Project Rehearsal(2002)


Arena l.jpg
Arena

  • Installation artworks

  • Live performance

  • Educational spaces

  • Film production

Behind the Bars (1999)

Macbett (2001)


Outline l.jpg
Outline

  • Background

  • Application examples

    • Live performance

    • Installation artworks

    • Film production

  • Research topics

  • Collaboration

    • Research

    • Curriculum


Ubicomp as a catalyst for creation l.jpg
Ubicomp as a catalyst for creation

  • Ubiquitous computing technologies canmake action and presence in physical space relevant to computing, networks, and media.

  • Process

  • Presence

  • Context


Art as a laboratory for ubicomp l.jpg
Art as a laboratory for ubicomp

  • Controlled slices of the real world

  • Development / rehearsal process

  • need to author on-the-fly, in-the-field

  • Specific performance requirements different tradeoffs

  • Challenge of basic assumptions

  • different perspectives


Live performance l.jpg
Live performance

  • Responsive stage environments

    • Arizona State University’s Intelligent Stage (Lovell, et al.)

    • University of Georgia (Saltz)

    • MIT Media Lab (Sparacino, et al.)

  • Audience interaction / incorporation

    • Carnegie Mellon ETC

    • Blast Theory (UK)

    • Many other companies

Fahrenheit 451 (2000)


Macbett l.jpg
Macbett

Directed by Adam Shive, Interactive systems by Jeff Burke (2001)

  • Department of Theater subscription-series production.

  • Large-scale theatrical lighting and sound controlled based on actor movement, as sensed by an off-the-shelf wireless tracking system.

  • Team of one graduate student and four undergraduate programmers, as well as the entire design team, cast, and crew.


Macbett10 l.jpg
Macbett

  • Seven networked workstations and servers communicated via UDP for

    • Sensor management,

    • Feature calculation,

    • Lighting control,

    • Sound control,

    • “Authoring” of interactive relationships,

    • System monitoring.



The iliad project l.jpg
The Iliad Project

Architecture: Jeff Burke, Jared Stein / Research direction: Fabian Wagmister, Edit Villareal, Jose Luis Valenzuela

  • An ongoing research project that involves the simultaneous development of an original theatrical script, design and acting technique, and technology.

  • Emerging focus

    Customization of the script and media based on the attending audience.


The iliad project13 l.jpg
The Iliad Project

  • Core technologies

    • Audience database

    • Radio-Frequency Identification

    • Geographic Information System

    • Image capture and manipulation

    • Text processing

    • Live video streaming

    • Middleware – “Kolo”


Live performance14 l.jpg
Live performance

  • Sensor network research areas

    • Localization: robust, precise, and scalable

    • Actor (and audience) ‘state’

    • Middleware

    • Monitoring and realtime feedback

    • Authoring tools


Middleware for experiementation l.jpg

lights

1

2

intensity

intensity

ML001

intensity

colorC

colorM

colorY

focus

pan

tilt

Middleware for experiementation

  • Lightweight, consistent access to inputs and outputs.

  • Very high-level abstractions can be counterproductive,unless they can be built by the author/creator.

lights.ML001.colorM

magenta level of moving light #1


Middleware for experiementation16 l.jpg

tracking

actors

macbett

macbett

banco

position

x

x

x

y

y

y

z

z

z

Middleware for experiementation

  • First, make input and output devices on the network available in a consistent way.

  • Let authors define hierarchies and abstractions.

  • or


Middleware for experiementation17 l.jpg

lights.ML001, sound.pb1.left

left

lights

sound

syncCue1

intensity

members

ML001

intensity

pb1

intensity

colorC

colorM

colorY

focus

pan

tilt

Middleware for experiementation

  • Synchronization of actions in the environmentwithout centralized show control.

  • ‘group’


Middleware for experiementation18 l.jpg
Middleware for experiementation

  • Managing connections

    • Support for constructing ‘real-time’ relationshipsbetween data sources and sinks.

    • Arbitration between competing relationships.

  • Kolo middleware

    • Java-based middleware API(device drivers in C/C++)

    • Scripting language






Installation artwork l.jpg
Installation artwork

  • Non-narrative participatory experiences

  • Without performance definitions of performer/audience

  • Extensions into media-rich educational spaces

    • Participatory simulations

    • Embodied and kinesthetic learning

hamletmachine (2000-2)

cheLA demonstration (2002)


Time time again l.jpg
Time & Time Again…

Fabian Wagmister and Lynn Hershmann (1999)

  • Distributed interactive installation exploring complex relationships between our increasingly interlinked bodies and machines.

  • Site-specific work in the Ruhr region of Germany.

    • Live video streamed into musuem.

    • Real-time video compositing.

    • Silhouette image choice controlled by sensors in the installation space.

    • Robotic, telematic doll streaming images to and being controlled by web users.

    • History database of video fragments recorded by web visitors.


Time time again25 l.jpg
Time & Time Again…

Fabian Wagmister and Lynn Hershmann (1999)


Behind the bars l.jpg
Behind the Bars

Fabian Wagmister (1999)

  • Confrontational interactive environment treating Latin America’s history of physical and intellectual oppression.

  • Premiere:

  • Central American Film and Video Festival in Nicaragua.


Media rich educational spaces l.jpg
Media-rich educational spaces

  • Research areas

    • Adapt a simple wireless ADC platform?

    • Middleware

    • Data storage / mining

    • Space configuration

    • Authoring tools


Film production l.jpg
Film production

  • Equipping the film set with a wireless sensor network to support all aspects of production through

    • Instrumentation

    • Observation

    • ‘Augmented footage’

    • Decision support

    • Control

  • Developing collaboration with CENS faculty.


Sensor networks on the film set l.jpg
Sensor networks on the film set

  • Not just application… laboratory:

  • ‘Articulated chaos’,

  • Complex coordination to generate small slices of controlled reality,

  • Tradition of precise documentation,

  • Repetitive action,

  • Rapid, large scale field deployments,

  • Rigorous test environment already available on our campus (over 100 student films per year).


Terminology l.jpg
Terminology

  • Script

  • Location

  • Scene

  • Shot

  • Take

Exterior, Day, Suburban Street

Closeup of Heather at the fruit stand




Filmmaking application drivers l.jpg
Filmmaking Application Drivers

  • Primary / initial

  • Asset management

  • Continuity management

  • Secondary / future

  • Real-time control

  • General post-production

  • Special effects


Platform l.jpg
Platform

  • Wireless articulated imagers

  • Commercial x86 low-power architecture

  • Active tags

  • Low-cost, low-power tags providing RF and/or acoustic localization

  • Passive RFID tags

  • Primarily for asset management


Continuity management l.jpg
Continuity Management

  • Ensuring repeatable action by the crew and talent.

  • ‘Script supervision’ is facilitated by several ‘continuity assistants’.

  • Fine to coarse grained documentation.

  • Squirrel twitches her nose in take 1a, but not in take 1d.

  • Male bystander correctly on the woman’s right in both shots.


Continuity asset management l.jpg
Continuity / Asset Management

  • A superset of asset management:

  • Must track on-camera and off-camera items that create the film camera’s field of view for a shot:

  • Presence and qualities of props, scenery, and actors.

  • Equipment positions and settings.


Needs and constraints l.jpg
Needs and constraints

  • Localization with multiple sensing modalities

  • Synchronization

  • Privileged viewpoint

  • Event repetition

  • The Script – a priori scene knowledge


Needs and constraints38 l.jpg
Needs and constraints

  • Localization with multiple sensing modalities

  • Synchronization

  • Privileged viewpoint

  • Event repetition

  • The Script – a priori scene knowledge

  • Eventually, must localize O(100) objects for continuity on a large set.

  • Need for coarse-grained localization for O(1000) objects for asset tracking only is apparent.

  • Would like to know more than just location: orientation, gaze direction in closeups, etc.


Needs and constraints39 l.jpg
Needs and constraints

  • Localization with multiple sensing modalities

  • Synchronization

  • Privileged viewpoint

  • Event repetition

  • The Script – a priori scene knowledge

  • Data recorded must be ‘frame-accurate’, synchronized with the SMTPE time code of the shoot.

  • 24 frames/sec for film.

  • 29.97 frames/sec for NTSC video.

  • Reasonable but fine-grained synchronization of wireless sensor network to this external clock.


Needs and constraints40 l.jpg
Needs and constraints

  • Localization with multiple sensing modalities

  • Synchronization

  • Privileged viewpoint

  • Event repetition

  • The Script – a priori scene knowledge

  • All that matters is what the film camera sees – it is the ‘privileged viewpoint’ of the system.

  • Camera and film stock parameters can be used to estimate what is seen and what isn’t.

  • With this information, manage:

  • focus of attention,

  • power consumption,

  • processing cycles,

  • bandwidth,

  • sensor articulation.


Needs and constraints41 l.jpg
Needs and constraints

  • Localization with multiple sensing modalities

  • Synchronization

  • Privileged viewpoint

  • Event repetition

  • The Script – a priori scene knowledge

  • Need to track and take advantage of the repetitive action built into the process:

  • multiple shots of one scene,

  • multiple takes of one shot,

  • off-camera rehearsal.

  • Leverage filmmakers’ experience to develop heuristics for what action is ‘crucial’ and what is not.


Needs and constraints42 l.jpg
Needs and constraints

  • Localization with multiple sensing modalities

  • Synchronization

  • Privileged viewpoint

  • Event repetition

  • The Script – a priori scene knowledge

  • Unlike many ubicomp scenarios, the film shoot has a carefully defined blueprint for the action that will occur: the script.

  • Could consider ‘script reading’ research, or…

  • Use the consistent format of film scripts to recognize who and what are (or will be, or was) present in a scene or shot.





Decision support example the 180 o rule46 l.jpg
Decision support example: the 180o rule

  • A simple position-based metric for perceived discontinuity of scene layouts from multiple perspectives.

  • Well-suited for real-time visualization.

  • Important use of ‘coarser’ grained continuity management.


Application driver for what l.jpg
Application Driver for What?

  • Localization and identification…

  • using several sensing platforms, high resolution for O(100) objects, low resolution for O(1000).

  • Asset management and tracking…

  • with multiple action-dependency relationships, and across multiple time frames.

  • Embedding sensor network generated metadata into media…

  • to create augmented footage.

  • Scene observation techniques…

  • for recognizing discontinuities across multiple perpsectives.

  • Field decision support…

  • considering an arbitrary ‘privileged viewpoint’.


Art as a laboratory for ubicomp48 l.jpg
Art as a laboratory for ubicomp

  • Controlled slices of the real world

  • Development / rehearsal process

  • need to author on-the-fly, in-the-field

  • Specific performance requirements different tradeoffs

  • Challenge of basic assumptions

  • different perspectives


Research agenda l.jpg
Research Agenda

  • Short term goals

    • Localization platform

    • Middleware for ‘real-time’ relationships

    • Writing many input / output (and storage) drivers

  • Longer term interests

    • Fusing vision with other sensing modalities

    • Authoring tools


Collaboration l.jpg
Collaboration

  • Research

    • Project-based focus

      • Commissions

      • Externally funded research

    • Students and faculty from EE/CS, Theater, Film/TV, Design

  • Curriculum

    • Existing courses

      • FTVD 144/244, Theater 144C

    • Planned courses

      • Concurrent TFT / Electrical Engineering


Wireless sensor networks for art entertainment applications51 l.jpg

Wireless sensor networks for art & entertainment applications

Center for Embedded Networked Sensing

April 18, 2003

Jeff Burke

UCLA School of Theater, Film and Television

jburke@hypermedia.ucla.edu

http://hypermedia.ucla.edu/