The need for interdisciplinary input in setting goals for Music Information Retrieval

1. The need for interdisciplinary input in setting goals for Music Information Retrieval Don Knox School of Engineering and Computing Glasgow Caledonian University

2. Music Information Retrieval • Emerging discipline, themes/goals include: • Audio content and score/symbolic music analysis, music retrieval by query, music similarity, rights management, music classification, recommendation systems, music visualisation, human interaction • QBH, QBE, genre and style classification, semantic classification

3. Music Information Retrieval • Methods • Content analysis – acoustic and music structure features, metadata, score/symbolic analysis • Statistical methods, signal analysis, machine learning, pattern matching etc.

4. Symposium themes and MIR • Innovation • MIR systems and approaches are innovative • Tools allow e.g. Musicological evaluation of genre, structure, novelty, similarity • Creativity • Analysis and classification of music semantics – attempts to evaluate creative intent (e.g. emotion) • Labour • Music recommendation now playing a central role in economic exploitation of musical work

5. Informing MIR goals • ‘Remarkable’ difference in the past between MIR assumptions and user needs regarding music query (Lee et al 2007) • Example: QBH early focus of MIR • Ignored user requirements, seeking behaviour – casual users don’t want it • Inappropriate format for musicological query

6. Informing MIR goals • Who are we innovating for? • Intended audience, how will systems be used? • Relevance of results – subjective views on what the results mean to different users • User interaction and query – useful? • Experiential issue – Downie (2003) • ‘The single most important challenge facing MIR’ • ‘To ignore the experiential aspect of MIR is to diminish the very core of MIR endeavour’ • Investigation of existing MIR practice should identify distinct user communities and investigate what they need from MIR systems (Futrelle and Downie 2002)

7. Users – who are they? • Casual, professional, researcher • Researchers need MIR tools • Industry – e.g. find music to match a specific need – advertising, film, QBE • Users need relevant applications to suit their needs and music seeking behaviour • User type affects how MIR systems are used • Music Psychology can tell us what type of user someone is (Uitdenbogerd and Yap 03)

8. Users/Experiential issue • Different disciplines play a key role around this issue • Agreed from the beginning that this is key to the future of MIR • Still very little emphasis on subjectivity, users, experience when setting goals for MIR systems

9. Example: genre classification • Motivation: • Industry use, user interaction with music, researching musical similarity Video clip. Marsayas genre meter: http://www.youtube.com/watch?v=NDLhrc_WR5Q

10. Why pursue this? • Do we understand how users define and use genre? • Does this lead to meaningful results for anyone? • Techniques usually based on expert annotation and then classification via feature extraction • MIR studies have found only 70% human agreement on genre (Tzanetakis and Cook 02) • Bottom-up approaches are leveling out as regards success rates

11. Genre classification • View of experts and ‘social’ view recognised as being important • Social, economic, cultural aspects affect genre identity (Sordo et al 2008) • McEnnis and Cunningham (2007) – social context is important • View of artists • Role of genre influence on artist– identification with communities (Toynbee 2000) • Very few MIR studies including artists participation • Recent call to evaluate these issues alongside the statistical/analysis methods applied (Craft et al 2007)

12. Musical similarity and recommendation • Similarity: Implicit in many MIR goals – classification, music query, recommendation, rights protection and tracking • Musicology influence: • Need MIR tools for query of music databases • Concerned with issues such as allusions to musical themes, musical influence, evolution of tune families • Can advise as to the nature of musical similarity • Danger of engineers and computer scientists defining ‘the nature of music’ – possible source of interdisciplinary tension (Futrelle and Downie 2002)

13. Recommendation systems • Huge industry interest (Genius, Amazon, Last.fm) • Usually based on artist, genre, purchasing behaviour, and ‘similarity’ (metadata) • Better similarity scores obtained by studying user preference (Slaney and White 2007) • i.e. It’s not just about musical similarity – it’s about like and dislike • Music psychology an invaluable source of information in this regard

14. Recommendation systems • Few MIR studies on user - cultural, social, contextual reasons for preference are ignored, despite being of ‘real practical use’ to MIR (Cunningham et al 2005) • Current trend in MIR in this area is study of social tagging and user music queries • Helpful for e.g. automatic playlist generation – gap between recommendation results and observed user behaviour (Cunningham et al 2006)

15. Recommendation systems • Usually based on study of tags, queries. • How good is this data? • One sociological study on motivation in social tagging (McEnnis and Cunningham 2007) • Input from music psychology recognised as being useful: • Emotional and social connection as part of music query (Downie and Cunningham 2002) • Friends, family channel for music seeking rather than expert opinion (Laplante and Downie 2006) • Despite this – there is little music psychology influence as regards music seeking behaviour for e.g. Activity, mood, well-being etc.

16. The semantic gap • Bottom up approach (structure, feature extraction) alone is not sufficient • But this is still a main focus for MIR efforts • Top-down, user centred studies make results and goals more relevant and accurate • Semantic/affective query, interaction identified as being ‘a promising new area’ for MIR • Content based MIR can benefit from study of user subjectivity and background (Casey et al 2008)

17. The semantic gap • The need for multidisciplinary input • How do we even begin to define these semantic descriptions of music? • Human factors essential to this task • Effect of gender, age familiarity, musical training etc. on semantic description (Hargreaves and North 99) • Need for guidance re music structure, composition, performance and semantic description – musical communication

18. Emotion classification • A recent focus of some MIR systems – ‘affective’ interaction with music collections • Driven by perception that listeners seek music according to mood (Juslin and Sloboda 2001) • Perhaps the most significant interdisciplinary overlap in MIR – feature extraction, musicology, user behaviour, composer intent, performer actions, cultural and social factors, subjectivity

19. Emotion classification • Subjective and complex issue of emotional effect of music • MIR takes an essentially uninformed approach to this important issue • Issue is either fudged, or examined by social tagging/collaborative filtering • Debate still raging in music psychology about whether this effect exists • Music psychology input regarding user experience is crucial if this is to be a valid goal

20. Emotion classification • Limited scope of MIR approaches thus far: • Limited study of features • Typically treated as a similarity/classification problem based on feature extraction • Expert annotation, subsequent comparison and classification

21. Emotion classification • Continued adoption of bottom-up approaches • Only beginning to look at music structure and emotion • Although e.g. tension/expectation well understood in musicology/psychology (Meyer 56)

22. Emotion classification • Limited participation by salient parties • No existing MIR studies concerning composer intent regarding emotion of the piece • Musical communication issue - composition and performance convey meaning • No analysis of performance cues – despite this being studied in music psychology (Juslin 2000)

23. Current MIR trends • Vast majority of ISMIR papers still about ‘how’ • Still very few studies from user perspective – but growing • Recent social tagging studies, some analysis of music queries and FOAF profiles (Lee et al 2007, Landone et al 2007, Celma et al 2005) • Few calls for interdisciplinary input since Downie and others around 2002/2003 • E.g. Crawford (2005) – MIR and the future of musicology, Craft et al (2007)– issues surrounding subjectivity of genre

24. Current MIR trends • Smiraglia (2006) study of all ISMIR publications • Vast majority of papers still produced by computer scientists • Teams tend to come from same institution, and same discipline • No psychology input at all (up to 2006) • Do we cite across disciplines, but continue to work within our own? • E.g. McEnnis and Cunningham (2007) a review of sociology literature by computer scientists

25. Recent MIR developments • Encouraging signs: • Recent series of workshops on ‘semantics of audio’ • Increasing MIR contribution in cross-disciplinary forums – e.g. ICMPC • ‘Limited but increasing’ number of user studies (Lee et al 2007) • User communities and organisation of artists into genres (Baccigalupo et al 2008) • Jacobson et al (2008), Facebook links between artists – alignment by genre • User motivation behind social tagging (McEnnis and Cunningham 07)

26. Conclusions • MIR efforts and goals must be informed by the needs of the user • Musicological/music theory contribution • Tools for researchers – e.g. query format etc. • Music structure • The nature of musical similarity • Structural effects upon listener, musical communication • Musicians, composers, performers • Participation in studies concerning musical communication

27. Conclusions • Music psychology contribution • Semantic description of query terms • User needs and music seeking behaviour • Cultural and social issues behind e.g. genre, music preference • Music and emotion – experience, subjectivity • Meaningful understanding of similarity and recommendation – social, contextual

28. References • Baccigalupo, C., Donaldson, J. and E. Plaza. 2008. Uncovering Affinity of Artists to Multiple Genres From Social Behaviour Data. Proceedings of the 9th ISMIR, 2008. • Casey, M.A., Veltkamp, R., Goto, M., Leman, M., Rhodes, C. and M. Slaney. 2008. Content-Based Music Information Retrieval: Current Directions and Future Challenges. Proceedings of the IEEE, Vol. 96, No. 4, April 2008. • Celma, A., Ramírez, M. and P. Herrera-Boyer. 2005. Foafing the Music: A Music Recommendation System based on RSS Feeds and User Preferences. Proceedings of the 6th ISMIR, 2005. • Craft, A., Wiggins, G. A., and T. Crawford. 2007. How many beans make five? the consensus problem in music-genre classification and a new evaluation method for single-genre categorisation systems. Proceedings of the 8th ISMIR, 2007. • Crawford, T. 2005. Music Information Retrieval and the future of Musicology. Online Chopin Variorum Edition technical report. Available from http://www.ocve.org.uk [Accessed June 2009] • Cunningham, S.J., Downie, J.S. and D. Bainbridge. 2005. "The Pain ; the Pain": Modelling Music Information Behavior and the Songs We Hate. Proceedings of the 6th ISMIR, 2005. • Cunningham, S.J., D. Bainbridge and A. Falconer. 2006. More of an Art than a Science: Supporting the Creation of Playlists and Mixes. Proceedings of the 7th ISMIR, 2006. • Downie, J.S. and S.J. Cunningham. 2002. Toward a Theory of Music Information Retrieval Queries: System Design Implications. Proceedings of the 3rd ISMIR, 2002. pp. 299-300.

29. Downie, J.S. 2003. Music Information Retrieval. Annual Review of Information Science and Technology. 37 pp. 295-340. 2003. Futrelle, J. and J.S. Downie. 2002. Interdisciplinary Communities and Research Issues in Music Information Retrieval. Proceedings of the 3rd ISMIR, 2002. Hargreaves, D.J. and A. North. 1999. The functions of music in everyday life: Redefining the social in music psychology. Psychol. Music, vol. 27, no. 1, pp. 71–83. 1999. Jacobson, K., Fields, B. and M. Sandler. 2008. Using Audio Analysis and Network Structure to Identify Communities in On-Line Social Networks of Artists. Proceedings of the 9th ISMIR, 2008. Juslin, P.N. 200. Cue Utilization in Communication of Emotion in Music Performance: Relating Performance to Perception. Journal of Experimental Psychology: Human Perception and Performance 2000, Vol. 26, No. 6, 1797-1813. Juslin, P.N. and J. A. Sloboda, Music and Emotion: Theory and Research. Oxford University Press, 2001. Landone, C., Harrop, J. and J.D. Reiss. 2007. Enabling Access to Sound Archives Through Integration, Enrichment and Retrieval: The EASAIER Project. Proceedings of the 8th ISMIR, 2007. Laplante, A. and J.S. Downie. 2006. Everyday Life Music Information-Seeking Behaviour of Young Adults. Proceedings of the 7th ISMIR, 2006. Lee, J.H., Downie, J.S. and M. Cameron Jones. 2007. Preliminary analyses of information features provided by users for identifying music. Proceedings of the 8th ISMIR, 2007. McEnnis, D. and S.J. Cunningham. 2007. Sociology and Music Recommendation Systems. Proceedings of the 8th ISMIR, 2007.

30. Meyer, L.B. Emotion and meaning in music. The University of Chicago Press, 1956. Selfridge-Field, E. 2006. Social Cognition and Melodic Persistence: Where Metadata and Content Diverge. Proceedings of the 7th ISMIR, 2006. Slaney, M. and W. White. 2007. Similarity based on rating data. Proceedings of the 8th ISMIR, 2007. Smiraglia, R.P. 2006. Music Information Retrieval: An Example of Bates' Substrate? In Information Science Revisited: Approaches to Innovation: Proceedings of the Canadian Association for Information Science annual conference. June 1st‐3rd, 2006. Sordo, M., Celma, O., Blech, M. and E. Guaus. 2008. The Quest for Musical Genres: Do the Experts and the Wisdom of Crowds Agree? Proceedings of the 9th ISMIR, 2008. pp. 255-260. Tzanetakis, G. and P. Cook. 2002. Musical genre classification of audio signals. IEEE Transactions on Speech and Audio Processing, 10(5):293–302, 2002. Uitdenbogerd, A.L. and Yap, Y.W. 2003. Was Parsons right? An experiment in usability of music representations for melody-based music retrieval. Proceedings of the 4th ISMIR, 2003, pp.75-79.