Machine Discoveries: A few Simple, Robust Local Expression Principles

1. Machine Discoveries: A few Simple, Robust Local Expression Principles Written by Gerhard Widmer presented by Siao Jer, ISE 575b, Spring 2006

2. Presentation Overview • General Overview • Introduction • Training Data • Target Classes • Experimental Results • Quantitative Validation • Conclusion • Future Research

3. Gerhard Widmer • Head of the Department of Computational Perception at Johannes Kepler University Linz, Austria • Head of Machine Learning, Data Mining, and Intelligent Music Processing Group at the Austrian Research Institute for Artificial Intelligence • Numerous publication, awards, projects

4. General Overview • Discovering rules of expressive music performance • Inductive machine learning • Experiments with large data sets • Simple and general principles • Robust with surprisingly high level of accuracy

5. Introduction • What do performers do to make music “come alive?” • Studies done through a few classical approaches • Proposal of inductive machine learning • No preconceptions and expectations • Huge data sets allowed for more validity

6. Introduction • Previous work • Success in ability of machine learning (Widmer 2000) • Extremely complex • Attempt to find a complete model • Current goals • Testing new learning algorithm based on partial models • Learn rules of timing, dynamics, articulation • Testing degrees of fit over various styles and performers

7. Training Data • 13 complete Mozart piano sonatas • Performed by Roland Batik • On computer monitored grand piano • MIDI format • Includes hammer speed, impact times, pedal movements measured & xform’ed • Written score coded into computer format • Timing, dynamics, & articulation computed • 106,000 total notes • Melody restriction limits us to 41,000 notes

8. Target Classes • Objective: find note-level rules • Limit predictions to categorical decisions • Timing Dimension: note N is considered lengthened • If the note is lengthened relative to the instantaneous tempo over the previous note • If lengthened relative to local tempo over the last 20 notes • Analogous to this is a note shortened

9. Target Classes • Dynamics: louder if • Louder than previous note • And louder than average level of piece • Analogous to this is softer • Articulation • Staccato if played duration ratio (PDR) is less than 0.8 • Legato if greater than 1.0 • Portato otherwise, but study only concerned with staccato and legato • Pedaling not taken into account for articulation • Notes do not necessarily have to fall into one of these classes

10. Learning Partial Rule-based Models • No expectation to cover and describe all instances • Describe parts and define in meaningful terms • PLCG algorithm developed with these ideas in mind • Goal to come up with rules that covered lots of cases with good accuracy

11. Learning Partial Rule-based Models • General Steps • Separation into subsets • Learning partial rules within subsets • Merge all rules • Clustering of rules • One generalization per cluster • Optimize trade-offs (coverage vs. accuracy) • Result: 383 specialized rules narrowed to 17 general rules

12. Experimental ResultsTiming: Lengthening Notes • "Lengthen the middle note in a “cummulative” 3-note rhythm situation (ie, given 2 notes of equal duration followed by a longer note, lengthen the note that precedes the final, longer one).” • Most important one as it has highest prediction value • “Lengthen a note if it is followed by substantially longer note (ie the ratio between its duration and the duration of the next note is < 1:3)” • “Lengthen a note if it preceds an upward melodic leap of more than a perfect forth, if it is in a metrically weak position, and if it is preceded by (at most) stepwise motion” • 2 cases above have atleast 70% prediction rate

13. Experimental ResultsTiming: Lengthening Notes • “Lengthen a note if it preceds an upward melodic leap of more than a perfect forth, if it is in a metrically weak position, and if it is preceded by (at most) stepwise motion” • More of a “tendancy” than a rule • Interesting Note: • previously observed • But not over such a large data set

14. Experimental ResultsTiming: Shortening Notes • Difficult, but understandable • No strong rules, but a few tendencies • “Shorten a note in a sequence PN-N-NN if it is longer than its predecessor and longer than its successor.” • “Shorten a note in fast pieces in 3/8 time if the duratio ratio between previous note and current note is larger than 2:1, the current note is at most a sixteenth, and is again followed by a longer note.” • Example of a specialized rule • Correlation with Gabrielsson 1987

15. Experimental ResultsDynamics: Stressing Notes • Clear rules emerge, low coverage • Interesting note: relating stress to • melodic contour • Upward melodic movement • Observation by previous research as well • “Stress a note by playing it louder if it is preceded by an upward melodic leap larger than a perfect fourth.”

16. Experimental ResultsDynamics: Stressing Notes • “Stress a note by playing it louder if it forms the apex of an up-down melodic contour and is preceded by an (upward) leap larger than a minor third.” • “Stress a note by playing it louder if it at least twice as long as its predecessor, is reached by upward motion, and is in a quite strong metrical position.”

17. Experimental ResultsDynamics: Attenuating Notes • Difficult to predict • “Attenuate a note by playing it softer if it is less than 1/5 the duration of its predecessor.” • “Attenuate a note by playing it softer if it is preceded by a downward leap larger than a major third, is metrically weak, and is preceded by a note at least 1/3 of a beat long.” • “Attenuate a note by playing it softer if it is preceded by a downward leap larger than a perfect fifth and is metrically weak.” • Observation: linking metrically weak notes reached by downward leaps

18. Experimental ResultsArticulation Staccato • Most easily predictable, 4 strong rules • “Play a note staccato if the note is marked with a staccato dot in the score.” • “Play a note staccato if it is followed by a note of the same pitch (ie the interval between the note and its successor is a unison).” • Observations: • Combine for +90% accuracy & 6,000 cases • Previously observed in KTH Rules (Friberg 1995) • Physical reasons and explanations

19. Experimental ResultsArticulation Staccato • “Insert a micropause after a note if it precedes an upward leap larger than a perfect fourth and is metrically weak.” • “Insert a micropuase after a note of it is reached by downward motion and is followed by a note more than twice as long (ie the ratio between its duration and duration of the next note is < 0.4).” • Observations: • Correlation to lengthening rules • Supported by “Cumulative Rhythm” (Nramour 1977) • Articulation Staccato  30% of expression observed

20. Experimental ResultsArticulation Legato • Most difficult to predict • A LOT fewer instances vs. staccato • No markings on score • Low prediction rate (53.7%) • “Play a note legato if it is not marked staccato in the score, if it forms the apex of an up-down melodic contour, if it is quite short (<1/3 of a beat), and is metrically quite strong.” • Observations: • Melodic peak  legato?

21. Quantitative Validation:Generality I • Different Performer (Philippe Entremont) • Same pieces • No significant degradation in coverage and accuracy • Exception of “softer” • Higher coverage in • “lengthen” • “louder” • “staccato”

22. Quantitative Validation:Generality II • Testing on Different Styles & Artists • 2 Chopin pieces • 22 skilled pianist from Univ. of Music in Vienna • Surprising Results • “softer” and “legato”  unpredictable • “louder”  high % of positive examples, but high level of false predictions too • “lengthen,” “shorten,” & “staccato”  extremely good • Need more diversity of pieces

23. Conclusion • Small Step • Basic & simple rules • Robust model of local expression principles • Observations from other researchers • Autonomous discovery • Large data sets • Possible foundation

24. Further Research • Further evaluation of rules • different performers • Different types of music • Extension to other dimensions • (e.g. Harmony) • Going beyond note level • (e.g. phrase structure) • Comprehensive multi-level model