wireless sensor networks for art entertainment applications l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Wireless sensor networks for art & entertainment applications PowerPoint Presentation
Download Presentation
Wireless sensor networks for art & entertainment applications

Loading in 2 Seconds...

play fullscreen
1 / 51

Wireless sensor networks for art & entertainment applications - PowerPoint PPT Presentation


  • 467 Views
  • Uploaded on

Wireless sensor networks for art & entertainment applications. Center for Embedded Networked Sensing April 18, 2003. Jeff Burke UCLA School of Theater, Film and Television jburke@hypermedia.ucla.edu. HyperMedia Studio Background. Founded in 1997 by film professor Fabian Wagmister.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Wireless sensor networks for art & entertainment applications' - Jims


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
wireless sensor networks for art entertainment applications

Wireless sensor networks for art & entertainment applications

Center for Embedded Networked Sensing

April 18, 2003

Jeff Burke

UCLA School of Theater, Film and Television

jburke@hypermedia.ucla.edu

hypermedia studio background
HyperMedia Studio Background
  • Founded in 1997 by film professor Fabian Wagmister.
  • Located in a re-assigned television studio and edit rooms in the UCLA Department of Film, Television and Digital Media.

Behind the Bars (1999)

Iliad Project Team (2002)

goals
Goals

Invocation and Interference (2001)

  • Investigate the impact of emerging technologies on traditional production of theater, film and television.
  • Explore new work inspired and enabled by their unique capabilities and qualities.
  • Build tools that help creators, educators, and students explore
  • novel uses of instrumented environments.

Iliad Project Rehearsal(2002)

arena
Arena
  • Installation artworks
  • Live performance
  • Educational spaces
  • Film production

Behind the Bars (1999)

Macbett (2001)

outline
Outline
  • Background
  • Application examples
    • Live performance
    • Installation artworks
    • Film production
  • Research topics
  • Collaboration
    • Research
    • Curriculum
ubicomp as a catalyst for creation
Ubicomp as a catalyst for creation
  • Ubiquitous computing technologies canmake action and presence in physical space relevant to computing, networks, and media.
  • Process
  • Presence
  • Context
art as a laboratory for ubicomp
Art as a laboratory for ubicomp
  • Controlled slices of the real world
  • Development / rehearsal process
  • need to author on-the-fly, in-the-field
  • Specific performance requirements different tradeoffs
  • Challenge of basic assumptions
  • different perspectives
live performance
Live performance
  • Responsive stage environments
    • Arizona State University’s Intelligent Stage (Lovell, et al.)
    • University of Georgia (Saltz)
    • MIT Media Lab (Sparacino, et al.)
  • Audience interaction / incorporation
    • Carnegie Mellon ETC
    • Blast Theory (UK)
    • Many other companies

Fahrenheit 451 (2000)

macbett
Macbett

Directed by Adam Shive, Interactive systems by Jeff Burke (2001)

  • Department of Theater subscription-series production.
  • Large-scale theatrical lighting and sound controlled based on actor movement, as sensed by an off-the-shelf wireless tracking system.
  • Team of one graduate student and four undergraduate programmers, as well as the entire design team, cast, and crew.
macbett10
Macbett
  • Seven networked workstations and servers communicated via UDP for
    • Sensor management,
    • Feature calculation,
    • Lighting control,
    • Sound control,
    • “Authoring” of interactive relationships,
    • System monitoring.
the iliad project
The Iliad Project

Architecture: Jeff Burke, Jared Stein / Research direction: Fabian Wagmister, Edit Villareal, Jose Luis Valenzuela

  • An ongoing research project that involves the simultaneous development of an original theatrical script, design and acting technique, and technology.
  • Emerging focus

Customization of the script and media based on the attending audience.

the iliad project13
The Iliad Project
  • Core technologies
    • Audience database
    • Radio-Frequency Identification
    • Geographic Information System
    • Image capture and manipulation
    • Text processing
    • Live video streaming
    • Middleware – “Kolo”
live performance14
Live performance
  • Sensor network research areas
    • Localization: robust, precise, and scalable
    • Actor (and audience) ‘state’
    • Middleware
    • Monitoring and realtime feedback
    • Authoring tools
middleware for experiementation

lights

1

2

intensity

intensity

ML001

intensity

colorC

colorM

colorY

focus

pan

tilt

Middleware for experiementation
  • Lightweight, consistent access to inputs and outputs.
  • Very high-level abstractions can be counterproductive,unless they can be built by the author/creator.

lights.ML001.colorM

magenta level of moving light #1

middleware for experiementation16

tracking

actors

macbett

macbett

banco

position

x

x

x

y

y

y

z

z

z

Middleware for experiementation
  • First, make input and output devices on the network available in a consistent way.
  • Let authors define hierarchies and abstractions.
  • or
middleware for experiementation17

lights.ML001, sound.pb1.left

left

lights

sound

syncCue1

intensity

members

ML001

intensity

pb1

intensity

colorC

colorM

colorY

focus

pan

tilt

Middleware for experiementation
  • Synchronization of actions in the environmentwithout centralized show control.
  • ‘group’
middleware for experiementation18
Middleware for experiementation
  • Managing connections
    • Support for constructing ‘real-time’ relationshipsbetween data sources and sinks.
    • Arbitration between competing relationships.
  • Kolo middleware
    • Java-based middleware API(device drivers in C/C++)
    • Scripting language
installation artwork
Installation artwork
  • Non-narrative participatory experiences
  • Without performance definitions of performer/audience
  • Extensions into media-rich educational spaces
    • Participatory simulations
    • Embodied and kinesthetic learning

hamletmachine (2000-2)

cheLA demonstration (2002)

time time again
Time & Time Again…

Fabian Wagmister and Lynn Hershmann (1999)

  • Distributed interactive installation exploring complex relationships between our increasingly interlinked bodies and machines.
  • Site-specific work in the Ruhr region of Germany.
    • Live video streamed into musuem.
    • Real-time video compositing.
    • Silhouette image choice controlled by sensors in the installation space.
    • Robotic, telematic doll streaming images to and being controlled by web users.
    • History database of video fragments recorded by web visitors.
time time again25
Time & Time Again…

Fabian Wagmister and Lynn Hershmann (1999)

behind the bars
Behind the Bars

Fabian Wagmister (1999)

  • Confrontational interactive environment treating Latin America’s history of physical and intellectual oppression.
  • Premiere:
  • Central American Film and Video Festival in Nicaragua.
media rich educational spaces
Media-rich educational spaces
  • Research areas
    • Adapt a simple wireless ADC platform?
    • Middleware
    • Data storage / mining
    • Space configuration
    • Authoring tools
film production
Film production
  • Equipping the film set with a wireless sensor network to support all aspects of production through
    • Instrumentation
    • Observation
    • ‘Augmented footage’
    • Decision support
    • Control
  • Developing collaboration with CENS faculty.
sensor networks on the film set
Sensor networks on the film set
  • Not just application… laboratory:
  • ‘Articulated chaos’,
  • Complex coordination to generate small slices of controlled reality,
  • Tradition of precise documentation,
  • Repetitive action,
  • Rapid, large scale field deployments,
  • Rigorous test environment already available on our campus (over 100 student films per year).
terminology
Terminology
  • Script
  • Location
  • Scene
  • Shot
  • Take

Exterior, Day, Suburban Street

Closeup of Heather at the fruit stand

filmmaking application drivers
Filmmaking Application Drivers
  • Primary / initial
  • Asset management
  • Continuity management
  • Secondary / future
  • Real-time control
  • General post-production
  • Special effects
platform
Platform
  • Wireless articulated imagers
  • Commercial x86 low-power architecture
  • Active tags
  • Low-cost, low-power tags providing RF and/or acoustic localization
  • Passive RFID tags
  • Primarily for asset management
continuity management
Continuity Management
  • Ensuring repeatable action by the crew and talent.
  • ‘Script supervision’ is facilitated by several ‘continuity assistants’.
  • Fine to coarse grained documentation.
  • Squirrel twitches her nose in take 1a, but not in take 1d.
  • Male bystander correctly on the woman’s right in both shots.
continuity asset management
Continuity / Asset Management
  • A superset of asset management:
  • Must track on-camera and off-camera items that create the film camera’s field of view for a shot:
  • Presence and qualities of props, scenery, and actors.
  • Equipment positions and settings.
needs and constraints
Needs and constraints
  • Localization with multiple sensing modalities
  • Synchronization
  • Privileged viewpoint
  • Event repetition
  • The Script – a priori scene knowledge
needs and constraints38
Needs and constraints
  • Localization with multiple sensing modalities
  • Synchronization
  • Privileged viewpoint
  • Event repetition
  • The Script – a priori scene knowledge
  • Eventually, must localize O(100) objects for continuity on a large set.
  • Need for coarse-grained localization for O(1000) objects for asset tracking only is apparent.
  • Would like to know more than just location: orientation, gaze direction in closeups, etc.
needs and constraints39
Needs and constraints
  • Localization with multiple sensing modalities
  • Synchronization
  • Privileged viewpoint
  • Event repetition
  • The Script – a priori scene knowledge
  • Data recorded must be ‘frame-accurate’, synchronized with the SMTPE time code of the shoot.
  • 24 frames/sec for film.
  • 29.97 frames/sec for NTSC video.
  • Reasonable but fine-grained synchronization of wireless sensor network to this external clock.
needs and constraints40
Needs and constraints
  • Localization with multiple sensing modalities
  • Synchronization
  • Privileged viewpoint
  • Event repetition
  • The Script – a priori scene knowledge
  • All that matters is what the film camera sees – it is the ‘privileged viewpoint’ of the system.
  • Camera and film stock parameters can be used to estimate what is seen and what isn’t.
  • With this information, manage:
  • focus of attention,
  • power consumption,
  • processing cycles,
  • bandwidth,
  • sensor articulation.
needs and constraints41
Needs and constraints
  • Localization with multiple sensing modalities
  • Synchronization
  • Privileged viewpoint
  • Event repetition
  • The Script – a priori scene knowledge
  • Need to track and take advantage of the repetitive action built into the process:
  • multiple shots of one scene,
  • multiple takes of one shot,
  • off-camera rehearsal.
  • Leverage filmmakers’ experience to develop heuristics for what action is ‘crucial’ and what is not.
needs and constraints42
Needs and constraints
  • Localization with multiple sensing modalities
  • Synchronization
  • Privileged viewpoint
  • Event repetition
  • The Script – a priori scene knowledge
  • Unlike many ubicomp scenarios, the film shoot has a carefully defined blueprint for the action that will occur: the script.
  • Could consider ‘script reading’ research, or…
  • Use the consistent format of film scripts to recognize who and what are (or will be, or was) present in a scene or shot.
decision support example the 180 o rule46
Decision support example: the 180o rule
  • A simple position-based metric for perceived discontinuity of scene layouts from multiple perspectives.
  • Well-suited for real-time visualization.
  • Important use of ‘coarser’ grained continuity management.
application driver for what
Application Driver for What?
  • Localization and identification…
  • using several sensing platforms, high resolution for O(100) objects, low resolution for O(1000).
  • Asset management and tracking…
  • with multiple action-dependency relationships, and across multiple time frames.
  • Embedding sensor network generated metadata into media…
  • to create augmented footage.
  • Scene observation techniques…
  • for recognizing discontinuities across multiple perpsectives.
  • Field decision support…
  • considering an arbitrary ‘privileged viewpoint’.
art as a laboratory for ubicomp48
Art as a laboratory for ubicomp
  • Controlled slices of the real world
  • Development / rehearsal process
  • need to author on-the-fly, in-the-field
  • Specific performance requirements different tradeoffs
  • Challenge of basic assumptions
  • different perspectives
research agenda
Research Agenda
  • Short term goals
    • Localization platform
    • Middleware for ‘real-time’ relationships
    • Writing many input / output (and storage) drivers
  • Longer term interests
    • Fusing vision with other sensing modalities
    • Authoring tools
collaboration
Collaboration
  • Research
    • Project-based focus
      • Commissions
      • Externally funded research
    • Students and faculty from EE/CS, Theater, Film/TV, Design
  • Curriculum
    • Existing courses
      • FTVD 144/244, Theater 144C
    • Planned courses
      • Concurrent TFT / Electrical Engineering
wireless sensor networks for art entertainment applications51

Wireless sensor networks for art & entertainment applications

Center for Embedded Networked Sensing

April 18, 2003

Jeff Burke

UCLA School of Theater, Film and Television

jburke@hypermedia.ucla.edu

http://hypermedia.ucla.edu/