Large Group Musical Interaction Using Disposable Wireless Motion Sensors

1. Large Group Musical Interaction Using Disposable Wireless Motion Sensors Responsive Environments Group Mark Feldmeier, Mateusz Malinowski, Joseph A. Paradiso MIT Media Lab

2. Many input channels or sensors on body Expensive… (hundred to thousands of $) Multiplayer scaling? Wearable Interfaces for Electronic Dance Expressive Footwear Yamaha Miburi 16 sensor channels per foot Circa 20 input channels per body?

3. Wireless multisensor system has shrunk & increased BW Instrument upper and lower body of small dance ensemble Way too expensive for large groups! Scaling to several dancers…

4. Controllers/buttons in theater chairs Hardwired, nonmobile, potentially costly Bulk measurements via cameras (VNS), microphones (applause meters) Hard to measure/count/gauge deliberate gestures Background rejection difficult Techniques for large group interaction

5. E.g., the Pirates situated reality game Too expensive for very large groups Slow response Lots of wireless IP and protocol Each Participant has a PDA

6. Cinematrix (Loren and Rachel Carpenter) Everybody has bi-colored reflective paddle Binary vote Need line-of sight, not necessarily appropriate for dance environment Each participant has a passive optical “tag”

7. Galvactivator (Picard, Schrier) Wearable skin resistance monitor LED glows with decreasing resistance Video camera sees & integrates response Each participant has an active tag Biofeedback for large groups • The Sophisticated Soiree • Wearable, wireless heart rate sensor • 64 participants, each with 1 bit • Interactive content tries to synch heartbeats • Across several hours • Ars Electronica 2001 Autonomic systems – no direct physical control

8. New Scientist, 14-November-01 GA to optimally select tracks to play to keep people dancing (Dave Cliff – HP Bristol) Wearable mentioned: Heart rate monitor, perspiration sensor, accelerometer, Bluetooth link Expensive?! Proposed project? The HPDJ?

9. Wireless sensor system that responds to deliberate motion Non line-of-sight, appropriate for dance club environment Compact, wearable-or-holdable Very inexpensive – e.g., giveaway with ticket The Goal

10. Very simple motion sensor Cantilevered PVDF piezo strip with proof mass Activates CMOS dual monostable when jerked Sends brief (50 ms) pulse of 300 MHz RF 100 ms dead timer prevents multipulsing Can zone to within ~10 meters via amplitude Ultra low power – battery lasts up to shelf life Extremely cheap – e.g., under $1.00 in large quantity The Disposable Wireless Sensors

11. No ID sent to conserve bandwidth, $ Signals are extremely narrow e.g., 50 ms out of 100 ms max. F 0.05% overlap Low probability of collision across 10 meter range Current receivers are very simple Integrated pulse counts are updated every 16.38 ms Current platform supports 2 receivers Outputs counts in A only, A&B, B only for zoning MIDI output (mapping now done in MAX) Sensor Response 7 people clapping in unison

12. Pulse density Activity across various time windows Frequency of averaged density (from DFT) Tempo Significant density peaks Events at which several hits occur at similar times Extracted Features Early mapping in Lab

13. Interactive Raves at MIT

14. Mode of music is set by the mean activity level: <25% level_1 ambient 25 - 50% level_2 minimal techno 50 - 75% level_3 house 75 - 100% level_4 hardtrance >100% level_5 hardcore/chaos Activity compensated by number of people and tempo Activity is the #hits/#people/quarter-note duration BPM of music set to BPM of crowd (from DFT) + 2 Rolling averages over 80ms, 1/2s, 1s, 2s, 10s were used to modulate things like pitch of arpeggiated lines, depth of LFO, depth of effects, filter resonance, filter cutoff During the ambient level, when few hits were arriving, each hit is given a sharp, bright sound. As more hits enter, this dulls into a wall of sound that undulates w. energy. Over 25% cause a hit at the same time, a larger sound is created. After a succession of five of these beats, a base beat is slowly faded in, hence the dancers can start to build the initial rhythm Some rules used in interactive mappings

15. Interactive Raves at MIT Trial run with about 50 People at MIT (Sydney/Pacific) Sept. 7, 2002

16. Better run at Talbot, Sept. 12, 2002 • Fixed software errors that plagued Sydney-Pacific run of days before • 20-30 participants

17. 20-second samples Lower activity, lack of rhythm evident in ambient sections Talbot Event Data Sample Raw pulse count 10-sec LPF count FFT of above Ambient Section Energetic Section

18. Data across entire Talbot event Activity (# hits per 10 seconds) +2 BPM activated Perceived beats per minute

19. Responses Talbot Sydney-Pacific No Yes Seen to be more causal when working properly

20. Classes of 13-15 kids, 12-15 years old George Lewis (UCSD), Chris Csikszentmihalyi (MIT ML) collaboration, 8/02 Each sensor tuned to 1 of 3 separate frequencies 3 effects possible Zoned into 4 locations Used with children at CAMP in Kyoto

21. Sensors worked as planned Uniformity in sensitivity? Axis dependence? Ruggedness Amplitude location approximate Depend on crowd density Future devices are UWB? Transmitter trivial, receiver complex Perhaps circa 1 meter location? Battery lasts forever Transmitter drift? Conlusions New Old

22. Interactive Entertainment for Large Groups is never easy Audience found it more causal when working Plenty of room left… Better algorithms, statistics, features? e.g., cross correlation instead of DFT Different flavors of shakers - e.g., left/right hand, hands/feet, etc. Mappings that better encourage human schooling reflex Mapping Issues