Chuck! Parts 3 and 4: PLOrk and machine learning

1. Chuck!Parts 3 and 4: PLOrk and machine learning Rebecca Fiebrink Princeton University

2. Part 3: The Princeton Laptop Orchestra • 4 years old, first of its kind • Now laptop orchestras in Stanford U., Tokyo, London, Oslo, and many others • 15+ students • Learn ChucK, some Max/MSP, and digital music composition in two classes per week • Rehearse once per week • Pieces composed by students, faculty, and guest artists • Very hard compositional issues!

3. Dan Trueman: Silicon/Carbon • http://www.youtube.com/watch?v=YyD96blI4EU • http://www.youtube.com/watch?v=o71j1xGvvfA&feature=related

4. Equipment

5. Techniques • Network to synchronize computers • Sensors & game controllers • Incorporate voice & acoustic instruments • Often improvisational within a predetermined structure • Unique conducting gestures

10. Part 4 Machine learning for live performance

11. Machine learning for live performance • Problem: there is a semantic gap between the raw data that computers use and the musical, cultural, aesthetic meanings that humans perceive and assign.

13. One solution: A lot of code • What algorithm would you design to tell a computer whether a picture contains a human face?

14. The problem • If your algorithm doesn’t work, how can you fix it? • You can’t easily reuse it to do a similar task (e.g., recognizing monkey faces that are not human) • There’s no “theory” for how to write a good algorithm • It’s a lot of work!

15. Another solution: Machine learning (Classification) • Classification is a data-driven approach for applying labels to data. Once a classifier has been trained on a training set that includes the true labels, it will predict labels for new data it hasn’t seen before.

17. Train the classifier on a labeled dataset Data Set: A feature vector and class for every data point Classifier

18. Run the trained classifier on new data Classifier NO!

19. Candidates for machine learning • Which gesture did the performer just make with the iCube? • Which instruments are playing right now? • Who is singing? What language are they singing? • Is this chord major or minor? • Is this dancer moving quickly or slowly? • Is this music happy or sad? • Is anyone standing near the camera?

20. An example algorithm: kNN • The features of an example are treated as its coordinates in n-dimensional space • To classify an new example, the algorithm looks for its k (maybe 10) nearest neighbors in that space, and chooses the most popular class.

21. kNN space: Basketball or Sumo? Feature 2: Height Feature 1: Weight

22. kNN space: Basketball or Sumo? ? Feature 2: Height Feature 1: Weight

23. kNN space: Basketball or Sumo? K=3 ? Feature 2: Height Feature 1: Weight

24. kNN space: Basketball or Sumo? S Feature 2: Height Feature 1: Weight

25. Another technique: Neural networks • One of the first machine learning methods • Model functions as complex connections of interconnected nodes (neurons) • Inspired by the human brain • An input causes a cascade of neuron firings, resulting in an output value (e.g., a class label, or a real number) • Results in highly non-linear functions from input to output • Interesting for performance!

26. Combining Techniques Processing: Extract webcam features Java: Train a neural network to map features to sounds OSC OSC ChucK: Pass features to Java, receive results back and use them to make sound Example: Samson recognizer

27. Review • Machine learning is often easier than the alternatives • Use standard algorithms (exiting libraries) to do hard work • Present examples of inputs (features) with their outputs (desired labels) • Requires you to choose which features to use (different for audio, video, sensor) • Appropriate for camera, audio, sensors, and many other types of data • Live, interactive performance is a very interesting application area • E.g., can re-train classifiers as you get new data • See my handout: Classification_handout.pdf