1 / 103

Rebecca Fiebrink Princeton University / University of Washington

Real-time Human Interaction with Supervised Learning Algorithms for Music Composition and Performance. Rebecca Fiebrink Princeton University / University of Washington. function [ x flag hist dt ] = pagerank(A,optionsu ) [ m n ] = size(A ); if ( m ~= n )

summer
Download Presentation

Rebecca Fiebrink Princeton University / University of Washington

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Real-time Human Interaction with Supervised Learning Algorithms for Music Composition and Performance Rebecca FiebrinkPrinceton University / University of Washington

  2. function [x flag histdt] = pagerank(A,optionsu) [mn] = size(A); if (m ~= n) error('pagerank:invalidParameter', 'the matrix A must be square'); end; options = struct('tol', 1e-7, 'maxiter', 500, 'v', ones(n,1)./n, … 'c', 0.85, 'verbose', 0, 'alg', 'arnoldi', … 'linsys_solver', @(f,v,tol,its) bicgstab(f,v,tol,its), … 'arnoldi_k', 8, 'approx_bp', 1e-3, 'approx_boundary', inf,… 'approx_subiter', 5); if (nargin > 1) options = merge_structs(optionsu, options); end; if (size(options.v) ~= size(A,1)) error('pagerank:invalidParameter', … 'the vector v must have the same size as A'); end; if (~issparse(A)) A = sparse(A); end; % normalize the matrix P = normout(A); switch (options.alg) case 'dense’ [x flag histdt] = pagerank_dense(P, options); case 'linsys’ [x flag histdt] = pagerank_linsys(P, options) case 'gs’ [x flag histdt] = pagerank_gs(P, options); case 'power’ [x flag histdt] = pagerank_power(P, options); case 'arnoldi’ [x flag histdt] = pagerank_arnoldi(P, options); case 'approx’ [x flag histdt] = pagerank_approx(P, options); case 'eval’ [x flag histdt] = pagerank_eval(P, options); otherwise error('pagerank:invalidParameter', ... 'invalid computation mode specified.'); end; function [x flag histdt] = pagerank(A,optionsu)

  3. useful algorithms usable interfaces and appropriate interactions

  4. ? Machine learning algorithms

  5. Outline • Overviews of interactive computer music and machine learning • The Wekinator software • Live demo • User studies • Findings and Discussion • Conclusions

  6. interactive computer music

  7. Interactive computer music sensed action interpretation response (music, visuals, etc.) human with microphone, sensors, control interface, etc. audio synthesis or processing, visuals, etc. computer

  8. Example 1: Gesture recognition sensed action identification Bass drum: “Gesture 1” response computer

  9. Example 1: Gesture recognition sensed action identification Bass drum: “Gesture 2” Hi-hat response computer

  10. Model of sensed action to meaning sensed action model meaning response computer

  11. Example 2: Continuous gesture-to-sound mappings computer

  12. Example 2: Continuous gesture-to-sound mappings sensed action interpretation mapping sound generation human + control interface computer

  13. A composed system sensed action mapping/model/interpretation mapping/model/interpretation response

  14. supervised learning

  15. Supervised learning inputs training data algorithm model Training outputs

  16. Supervised learning inputs “Gesture 1” “Gesture 2” “Gesture 3” training data algorithm model Training outputs “Gesture 1” Running

  17. Supervised learning is useful • Models capture complex relationships from the data. (feasible) • Models can generalize to new inputs. (accurate) • Supervised learning circumvents the need to explicitly define mapping functions or models. (efficient) Has been demonstrated to be useful in musical applications, but no usable, general-purpose tools exist for composers to apply algorithms in their work.

  18. Weka: A model tool • General-purpose • GUI-based • Cited 11,705 times!

  19. Criteria for a supervised learning tool for composers General-purpose GUI-based Runs in real time Supports appropriate end-user interactions with the supervised learning process

  20. Appropriate interactions inputs “Gesture 1” “Gesture 2” “Gesture 3” training data algorithm model Training outputs “Gesture 1” Running

  21. Appropriate interactions inputs “Gesture 1” “Gesture 2” training data algorithm model Training creating training data outputs “Gesture 1” Running

  22. Appropriate interactions inputs “Gesture 1” “Gesture 2” training data model algorithm “Gesture 1” Training creating training data… evaluating the trained model outputs Running

  23. Appropriate interactions inputs “Gesture 1” “Gesture 2” “Gesture 3” training data algorithm model Training creating training data evaluating the trained model… modifying training data (and repeating) outputs “Gesture 1” Running

  24. Interactive machine learning (IML) • Training set editing for computer vision systems: Fails and Olsen 2003 • Application to other domains e.g. Shilman et al. 2006; Fogarty et al. 2008; Amershi et al. 2009; Baker et al. 2009 • Other types of interactionse.g., Talbot et al. 2009; Kapoor et al. 2010

  25. Research questions for end-user IML • Which interactions are possible and useful? • What are the practical benefits and challenges of incorporating end-user interaction in applied machine learning? • How can IML be useful in real-time and creative contexts?

  26. Outline • Overviews of interactive computer music and machine learning • The Wekinator software • Live demo • User studies • Findings and Discussion • Conclusions

  27. The Wekinator General-purpose GUI-based Runs in real time Supports appropriate end-user interactions with the supervised learning process • Built on Weka API • Downloadable: http://code.google.com/p/wekinator/

  28. Running models in real-time Feature extractor(s) .01, .59, .03, ... .01, .59, .03, ... .01, .59, .03, ... .01, .59, .03, ... time model(s) 5, .01, 22.7, … 5, .01, 22.7, … 5, .01, 22.7, … 5, .01, 22.7, … time Parameterizable process

  29. Interactive data creation and model evaluation “Gesture 1” model “Gesture 2” “Gesture 1” “Gesture 3” training data

  30. Real-time, iterative design

  31. Under the hood joystick_x joystick_y webcam_1 … Feature1 Feature2 Feature3 FeatureN Model1 Model2 ModelM … … Parameter1 Parameter2 ParameterM volume pitch 3.3098 Class24

  32. Under the hood Learning algorithms: Classification: AdaBoost.M1 J48 Decision Tree Support vector machine K-nearest neighbor Regression: MultilayerPerceptron … Feature1 Feature2 Feature3 FeatureN Model1 Model2 ModelM … … Parameter1 Parameter2 ParameterM 3.3098 Class24

  33. Tailored but not limited to music The Wekinator • Built-in feature extractors for music & gesture • ChucK API for feature extractors and synthesis classes Open Sound Control (UDP) Other modules for sound synthesis, animation, …? Other feature extraction modules

  34. Outline • Overviews of interactive computer music and machine learning • The Wekinator software • Live demo • User studies • Findings and Discussion • Conclusions

  35. Outline • Overviews of interactive computer music and machine learning • The Wekinator software • Live demo • User studies • Findings and Discussion • Conclusions

  36. Study 1: Participatory design process with composers • Process: • 7 composers • 10 weeks, 3 hours / week • Discussion and brainstorming ateach meeting • Final questionnaire • Outcomes: • Focus on instrument-building • 2 publicly-performed compositions • Much-improved software and lots of feedback

  37. Study 2: Teaching interactive systems building in an undergraduate course • Princeton Laptop Orchestra (PLOrk) • Midterm assignment • Students built 1 continuous + 1 discrete system • Logging + short answer questions • Outcomes: • Successful project completion • Used in midterm and final performances • Logs from 21 students

  38. Study 3: Bow gesture recognition • Case study with a composer/cellist • Task: Classify 8 types of standard bow gestures e.g., up/down bow (2 classes), articulation (7 classes)

  39. Study 3: Bow gesture recognition • Case study with a composer/cellist • Task: Classify 8 types of standard bow gestures e.g., up/down bow (2 classes), articulation (7 classes) • Method: • Tasks defined and directed by cellist • Logging, observations, final questionnaire • Cellist assigned each iteration’s classifier a quality rating (1 to 10) • Successful classifiers created for all 8 tasks (rated “9” or “10”)

  40. Study 4: Composer case studies • Clapping Music Machine Variations (CMMV) by Dan Trueman, faculty

  41. Study 4: Composer case studies • CMMV by Dan Trueman, faculty • The Gentle Senses / MARtLET by Michelle Nagai, graduate student

  42. Study 4: Composer case studies • CMMV by Dan Trueman, faculty • The Gentle Senses / MARtLET by Michelle Nagai, graduate student • G by Raymond Weitekamp, undergraduate

  43. Outline • Overviews of interactive computer music and machine learning • The Wekinator software • Live demo • User studies • Findings and Discussion • Conclusions

  44. Discussion of Findings Users took advantage of interaction in their work with the Wekinator. Users employed a variety of model evaluation criteria, and subjective evaluation did not always correlate with cross-validation accuracy. Feedback from the Wekinator influenced users’ actions and goals. The Wekinator was a useful and usable tool. Interactive supervised learning can be a tool for supporting creativity and embodiment.

  45. An iterative approach to model-building

More Related