A Wearable Wireless Sensor Platform for Interactive Dance Performances

1. A Wearable Wireless Sensor Platform for Interactive Dance Performances Chulsung Park and Pai H. Chou Center for Embedded Computer Systems Yicun Sun Department of Arts-Dance University of California, Irvine PerCom 2006 Presented by Jeffrey NTUEE/nslab

2. Outline • Abstract • Introduction • Related Work • Platform Design • Application Example • Conclusions and Future Work NTUEE/nslab

3. Outline • Abstract • Introduction • Related Work • Platform Design • Application Example • Conclusions and Future Work NTUEE/nslab

4. Abstract • Reports on recent development of a wearable wireless sensor platform • for interactive dance performances • At a fraction of a cubic-centimeter in volume • This platform is truly wearable and scalable in forming wireless networks • Integrated with a wide variety of different sensing devices • It is a real-time monitoring system for activities and physical conditions of the human body • Effectiveness of this platform is demonstrated with an interactive dance performance NTUEE/nslab

5. Outline • Abstract • Introduction • Related Work • Platform Design • Application Example • Conclusions and Future Work NTUEE/nslab

6. Introduction • Interactive dance environment • Live dancer’s movements are tracked and used to steer the synthesis of musical, graphical, and other various special effects in real-time • Available platforms today are not truly wearable, scalable, or able to support high-level interactivity NTUEE/nslab

7. Major Contributions (1/4) • Truly Wearable sensor platform • Eco • Ultra-compact and low power wireless sensor node • 648 mm3 without a battery • 720 mm3 with a battery • Smallest wireless sensor node in operation • Other wearable sensor platforms are at least 3 to 4 times larger NTUEE/nslab

8. Major Contributions (2/4) • Make Eco nodes form a scalable wireless network • Adopt ideas proposed for heterogeneous network architecture • Each dancer and sensor device can be uniquely identified • While multiple dancers wearing multiple sensing devices perform together NTUEE/nslab

9. Major Contributions (3/4) • Novelty in multi-modal sensing • Can collect data from multiple different types of sensing devices simultaneously • Motion tracking on dancers • Reading their physiological signs such as heartbeat • Opens up brand new possibilities for choreographers • As new creative tools for enhancing their performance NTUEE/nslab

10. Major Contributions (4/4) • Provides a seamless interface to Max/MSP and JITTER software packages using a wireless interface board • A choreographer can replace their current installation of interactive environment with proposed platform • Without any extra work on the software side NTUEE/nslab

11. Outline • Abstract • Introduction • Related Work • Platform Design • Application Example • Conclusions and Future Work NTUEE/nslab

12. Related Work • Interactive dance performance • Use the data from tracking the motion of a dancer to steer the generation of musical or graphical effects in real time • Motion tracking technologies • Computer vision based • Embedded or wearable sensors based NTUEE/nslab

13. Wearable Wireless Sensing Systems • Expressive Footware • A set of piezoelectric acceleration sensors are embedded into a pair of dancing shoes • 19.2Kbps radio NTUEE/nslab

14. Sensor Stack • Second generation Footware • 3-axial acceleration sensing • 3-axial angular velocity measurement • 115.2Kbps NTUEE/nslab

15. Wireless Inertial Measurement System (WIMS) • Flexible PCB • 101010 mm3 • Prone to breaking • Not suitable for mounting inertial sensors • Does not include a microcontroller, RF interface, and battery NTUEE/nslab

16. Max/MSP and JITTER • Max/MSP • A graphical environment for music, audio, and multimedia • Max: for MIDI, I/O control, user interface, and timing objects • MSP: a set of audio processing objects • JITTER • A set of matrix data processing objects optimized for video and 3-D graphics NTUEE/nslab

17. Max/MSP and JITTER NTUEE/nslab

18. Outline • Abstract • Introduction • Related Work • Platform Design • Application Example • Conclusions and Future Work NTUEE/nslab

19. Platform Design • Proposed wearable wireless sensor platform consists of three parts • wearable wireless sensor nodes • wireless data aggregators • wireless interface boards NTUEE/nslab

20. Platform Design NTUEE/nslab

21. Wearable Wireless Sensor Nodes • Activities and physical conditions of a dancer are first sensed and digitized by a set of wireless sensor nodes that the dancer wears • Then, each sensor node wirelessly transmits its data to the data aggregator worn on the dancer’s waist. NTUEE/nslab

22. Wireless Data Aggregators • Data aggregator collects and packetizes these data and send them to the wireless interface board NTUEE/nslab

23. Wireless Interface Boards • Converts the received data into digital/analog output signals • feeds them to the MIDI I/O terminal, which generates MIDI signals • Taking these MIDI signals as inputs • Max, MSP and JITTER process them and synthesize musical and visual effects as programmed by a choreographer • Effects are sent to the front projector, speaker, and lamps and displayed NTUEE/nslab

24. Scalability Issue • Uses two different networks • A body network • An 802.11b Wi-Fi network • Similar to Intel’s heterogeneous network architecture • Enables proposed platform to simultaneously monitor the activities and physical conditions of multiple dancers • Without degradation as the number of dancers increases NTUEE/nslab

25. Is A Single Network Scalable? • NO! • Every sensor node on each sensor would directly transmit its sampled data to the interface board • A body network for each dancer • A set of wearable wireless sensor nodes • One data aggregator • Use 2.4GHz ISM band radio • Use TDMA-based MAC protocol • Maximum data rate: 250Kbps • Transmission power level: 0dBm NTUEE/nslab

26. 802.11b Wi-Fi Network • Formed by the data aggregators on dancers and theater equipment • Overlaid on the body networks • Each data aggregator is linked to the access point of the interface board using 802.11b CF wireless card • Theoretically, up to 256 data aggregators can be connected to the access point simultaneously • In practice, 10-16 is more like the proper number of data aggregators connected to one access point • To guarantee the required bandwidth NTUEE/nslab

27. Wearable Wireless Sensor Node • Built based on the design of Eco • Three variants of Eco • Wireless transmitter unit (WT) • Acceleration, temperature, and light sensing unit (ATLS) • Image and gyro sensing unit (IGS) NTUEE/nslab

28. Wireless transmitter unit (WT) • Includes only a microcontroller and radio interface • With digital input/output and analog input interfaces • To connect to some big sensors • joint angle sensor • heartbeat sensor • infrared sensor • Turn on/off lanterns NTUEE/nslab

29. Wireless transmitter unit (WT) • nRF24E1 • A 2.4GHz transceiver with an embedded 8051-compatible microcontroller (DW8051) • DW8051 has a 512Byte ROM, a 4KByte RAM, one SPI (3-wire) interface, and a 9-channel 12-bit AD converter • Transceiver uses a GFSK modulation scheme in the 2.4GHz ISM band • 125 different frequency channels that are 1MHz apart • A chip antenna • 32K EEPROM • Maximum RF output power: 0dBm • Maximum data rate: 1Mbps • Maximum power consumption: 28mA at 3V NTUEE/nslab

30. Acceleration, temperature, and light sensing unit (ATLS) • Consists of • one 3-axial accelerometer (H34C) • A 3-axial accelerometer from Hitachi Metals • Acceleration measurement range is ±3g • It measures only 3.4 × 3.7 × 0.92 mm3 and consumes 0.36mA at 3V • one temperature sensor (embedded on H34C) • -20C to 65C • one light sensor (CdS photoresistor) • as well as what the WT unit has NTUEE/nslab

31. Acceleration, temperature, and light sensing unit (ATLS) NTUEE/nslab

32. Image and gyro sensing unit (IGS) • Built based on the Eco-Stick • a variant of the Eco • microcontroller and radio interface • has either an image sensor (VS6650) or a gyroscope (ADRS150) • VS6650 • 1.0-megapixel SMIA Camera Module from STMicroelectronics • measures 9.5 × 9.5 × 7.6 mm3 • consumes 30mA at 3V • interfaces with the nRF24E1 chip via the SPI port NTUEE/nslab

33. Image and gyro sensing unit (IGS) • ADXRS150 • a gyroscope from Analog Devices • measurement range is −150 to +150 degrees • current consumption of 13mA at 5V • measures 7 × 7 × 3.2 mm3 • Equipped with TWO 40mAh Li-Polymer batteries NTUEE/nslab

34. Wireless Data Aggregator • Consists of • An MSP430 16-bit microcontroller • An nRF2401A 2.4GHz transceiver • A WCF12 CompactFlash 802.11b card • 802.11b wireless card consumes a maximum of 250mA at 3.3V • Use a 700mAh Li-Polymer battery • To guarantee a minimum lifetime of one hour NTUEE/nslab

35. Wireless Data Aggregator NTUEE/nslab

36. Interface Board • Consists of • An MC9S12NE64 16-bit microcontroller with a built-in fast Ethernet control • One RJ-45 connector • Two serial ports • Digital/analog signal I/O interfaces • Provides a seamless interface between proposed platform and Max/MSP/JITTER software • Receives TCP/IP packets from data aggregators • Outputs digital/analog output signals fed to MIDI I/O board • When Max/MSP/JITTER output signals • It takes these signals and makes a TCP/IP packet that contains a proper destination address and control message • Sends out packets to the data aggregators NTUEE/nslab

37. Interface Board NTUEE/nslab

38. Outline • Abstract • Introduction • Related Work • Platform Design • Application Example • Conclusions and Future Work NTUEE/nslab

39. Application Example • Dreams in the Forbidden City • Devised by Yicun Sun • A live dance performance in an interactive environment • Describes the dreams of five concubines of the emperor in the Forbidden City • Their dreams are to please the emperor NTUEE/nslab

40. Dreams in the Forbidden City NTUEE/nslab

41. Dreams in the Forbidden City • According to the five dancers’ movement, the expression of the emperor varies • Sometimes the emperor punishes his concubines by thunder and lightning • Other times he expresses cheers by sweet music and bright light • All kinds of sound and visual effects are generated by a computer without any manual control NTUEE/nslab

42. Conventional Interactive Environment NTUEE/nslab

43. Improvements by Our Platform • Eliminate wiring between the stage equipment and the Max I/O terminal NTUEE/nslab

44. Improvements by Our Platform NTUEE/nslab

45. Outline • Abstract • Introduction • Related Work • Platform Design • Application Example • Conclusions and Future Work NTUEE/nslab

46. Conclusions • Propose a wearable wireless sensor platform for an interactive dance performance • Consists of • wearable wireless sensor nodes • data aggregators • wireless interface boards • Distinguishing features • Truly wearable • Highly scalable • Multi-modal sensing • Seamless interface NTUEE/nslab

47. Future Work • Choreographing interactive dance performances that take full advantage of proposed platform • At least tens of dancers will be performing together • Images transmitted from the dancers’ ISG units will be used to synthesize graphical effects on the stage • Dancers’ heartbeats and body temperatures will be monitored and converted into different colors and beats to reflect the dancers’ conditions on the stage equipment NTUEE/nslab

48. My Comments • Strength • Very good hardware technology • Interesting and practical applications • Cooperation with other departments in NTU? • Weakness • Truly scalable? • Interference between wireless sensor nodes of different dancers when they get close? NTUEE/nslab

49. Questions? NTUEE/nslab

50. Thank you very much for your attention! NTUEE/nslab