Melodic Features and Retrieval

1. Melodic Features and Retrieval ISMIR Graduate School, Barcelona 2004 Musicology 3-4 Frans Wiering, ICS, Utrecht University

2. Outline • yesterday’s assignment • demo: MIR outside academia (7:20; 44:10) • one-dimensional melody retrieval • Gestalt view of melody • advanced melody retrieval • assignment

3. one-dimensional melody retrieval • common assumption is (was?) pitch-only retrieval is sufficient • e.g. CCGGAAGGFFEEDDEC • mechanisms for fuzzy matching • variants • interval (distance between 2 pitches) • pitch-contour • same/up/down (Parson’s Code) • RURURDRDRDRDRUD • examples: • www.musipedia.com (Rainer Typke) • www.themefinder.org (CCARH)

4. Results from Musipedia • query is ranked 3 • other hits are very unlikely • unfortunately no notation/sound available • Haydn: evident false positive • why?

5. Themefinder • Several 1-dimensional search options, e.g. • pitch • interval • contour • rhythm • wildcards • each theme stored as a number of strings • matching by regular expressions • ca. 40.000 themes • Barlow and Morgenstern (1948) • ESAC encodings • Lincoln, 16th Century Motet (DARMS project)

6. results from Themefinder Query: +m2 +M2 P1 -M2 -m2 -M2 • Example from Byrd & Crawford (2001) • other hits • not as far-fetched as musipedia’s • different rhythm • different meter • still not very similar • is this what people have in mind?

7. Nice one we’ve just discovered • www.tuneteller.com • Pitch-only search of MIDI on the internet • many more MIR systems in Rainer Typke’s survey. URL is in your mailbox

8. Why pitch-only retrieval is unsatisfactory • information contribution of other 3 parameters (estimate for Western music; Byrd & Crawford 2001) • pitch: 50% • rhythm: 40% • timbre + dynamics: 10% • melodic confounds (Selfridge-Field 1998): • rests • repeated notes • grace notes, ornamentation • Mozart example

9. Why pitch-only retrieval is unsatisfactory • information contribution of other 3 parameters (estimate for Western music; Byrd & Crawford 2001) • pitch: 50% • rhythm: 40% • timbre + dynamics: 10% • melodic confounds (Selfridge-Field 1998): • rests • repeated notes • grace notes, ornamentation • Mozart example

10. Gestalt and melody • melody: coherent succession of pitches • from New Harvard Dictionary of Music • coherence important for similarity: creates musical meaning • bottom-up (pitches and durations) • top-down: segmenting, Gestalt • Gestalt theory of perception • late 19th/early 20th century, Germany, later US • perception of wholes rather than parts • explanations: Gestalt principles of grouping • application in visual and musical domain

11. Low-level Gestalt principles • Snyder mentions: • proximity • rhythmic • intervallic • similarity • duration • articulation • continuity • melodic • these produce closure of wholes • Example: Beethoven 5th symphony: beginning 1st movement • also illustrates high-level principles from Snyder (2001)

12. Low-level Gestalt principles • Snyder mentions: • proximity • rhythmic • intervallic • similarity • duration • articulation • continuity • melodic • these produce closure of wholes • Example: Beethoven • also illustrates high-level principles from Snyder (2001)

13. High-level Gestalt principles • parallellism • very strong in Mozart, Ah vous, second half of melody • intensification • important organisational principle in variations and improvisations • Mozart’s last variation from Snyder (2001)

14. Application in analysis and retrieval • Gestalt reduces memory overload: we can ignore the details • Analytical: Schering (1911) • 14th century Italian songs • basic melodic shape • might be nice for retrieval • Problem with Gestalt principles: • many different formulations • overlap; no rules for conflict • intuitive, cannot be successfully formalized from New Grove, Music analysis

15. The cognitive interpretation: chunking • what creates a boundary • interval leap • long duration • tonality (stable chords) • etc • Example of quantification: Melucci & Orio (2004) • using local boundary detection (Cambouropoulos 1997) • apply weight to intervals and durations • boundary after maximum • chunks forther processed for indexing

16. Organising chunks • STM problem: max. 5-7 different elements • very short span • solution: hierarchical grouping • melody schemas • contours of melody • cf. Schering ex. • examples: axial, arch, gap-fill • Mozart begins with gap-fill • next level: form • A-B-A from Snyder (2001)

17. Ah, vous dirai-je maman melody level analysis synthesis phrase level A A B chunk level subchunk level mental model of a song • analysis: from ear to LTM • (sub) chunks created by similarity and continuity • a lot of parallellism • boundaries by leaps and harmony • chunks may have a harmonic aspect too (I, V, V->I) • synthesis: from LTM to focus of attention • recollection • using general characteristics of phrases and chunks • performance • notes are reconstitued through some musical grammar

18. Problems of melody retrieval • People remember high-level concepts, not notes • often confused with poor performance abilities • theme-intensive music (fugues) stimulate formation of such concepts • melodic variability and change • transposition • augmentation/diminution • ornamentation • variation • compositional processes: inversion, retrograde • other factors • polyphony • harmony

19. Set-based approaches to melody retrieval in polyphony • General idea: • compare note sets: find supersets, calculate distance • usually take rhythm and pitch into account • hopefully more tolerant agains some of the problems of melodic variety • Clausen, Engelbrecht, Meyer, Schmidt (2000): • PROMS • matches onset times; wildcards • elegant indexing • Lemström, Mäkinen, Ukkonen, Turkia (several articles, 2003-4) • C-Brahms • algorithms for matching line segments • P1: onsets • P2: partial match onset times • P3: common shared time • attention to time complexity • Typke, Veltkamp, Wiering (2003-2004) • Orpheus system

20. Earth Mover’s Distance • The Earth Mover’s Distance (EMD) measures similarity by calculating a minimum flow that would match two set of weighted points. One set emits weight, the other one receives weight • Y. Rubner (1998); S. Cohen (1999)

21. Application to music • represent notes as weighted point sets in 2-dimensional space (pitch, time) • weight represents duration • other possibilities contour/metric position etc • other possible application: pitch event + acoustic feature(s)? here, the ‘earth’ is only moved along the temporal axis

22. Another example • interesting properties • tolerant against melodic confounds • suitable for polyphony • continuous • partial matching • disadvantage • triangle inequality doesn’t hold • less suitable for indexing: after alignment, the ‘earth’ is moved both along the temporal axis and along the pitch axis

23. Test on RISM A/II

24. Matching polyphony with the EMD • EMD’s partial matching property is essential • MIDI example used as query for RISM database • gross errors in playing are ironed out

25. Proportional Transportation Distance (PTD) • Giannopoulos & Veltkamp (2002) • EMD, weigths of sets normalised to 1 • suitable for indexing • triangle inequality holds • no partial matching

26. Test on RISM A/II • only hits with approximately same length • need 4 queries to find all known items

27. False positive (EMD) • problems arise when length and/or number of notes differs considerably

28. Segmenting • overlapping segments of 6-9 consecutive notes • not musical units • search results are combined • better Recall-Precision averages

29. Example of new search http://teuge.labs.cs.uu.nl/Rntt.cgi/mir/mir.cgi

30. Concluding remarks about melodic retrieval • lots of creativity go into melody; difficult to give rules • not a ‘basic musical structure’ (Temperley 2001) • essential to use multiple features • pitch, rhythm • harmony • segmentation • finding perceptually relevant chunks is not easy • finding complete melodies may be harder • arbitrary segments may also work • indexing strategies for melody • melodic change over time • several projects have tentative results for polyphony • gut feeling: false positives are big issue • notion of salience (Byrd and Crawford)