Enhancing Interactive Musical Performances: Solutions & Advantages

1. Approaches to Overcoming Problems in Interactive Musical Performance Systems Cory McKay Music Technology Area Faculty of Music McGill University Cory McKay

2. Topics • Introduction to interactive performance systems • Computer that performs with human performer(s) • Problems that have limited the use of such systems • General approaches that address these problems • Proposed long-term solution: • Standardized control parameter extractor • Reliance only on parameters extracted from performers’ audio signal Cory McKay

3. Interactive performance systems • Software / hardware that reacts musically to input from performer(s) • Can perform with: • A human soloist • A human accompanist • An ensemble of human performers • Other performance systems Cory McKay

4. Possible roles • Use score-following to dynamically timewarp pre-set scores • Initiate pre-recorded events triggered by performer actions • Basic mappings involving pre-set heuristics • e.g. harmonizations, dynamic sound processing, etc. • Use of advanced expert systems or non-deterministic artificial intelligence to model sophisticated musical knowledge • e.g. “jamming systems” Cory McKay

5. Advantages • Can perform actions that are too technically difficult for humans • Take advantage of extended sonic capabilities of computers • e.g. precise control of sound synthesis • Capitalize on essential expressivity and live dynamism of human performer(s) • Useful in an educational context Cory McKay

6. Problems with interactive systems • Lack of reliability during performance • Pieces can be too difficult or unintuitive for performers • Software can be unstable or too strenuous on hardware • Limited longevity • Reliance on software and hardware with limited life span • Poor interface with little or no documentation • Few performances without composer’s direct involvement • Hardware expensive and difficult to acquire and transport • Systems difficult to set up • Proper documentation and instructions rarely provided • Lack of effort to make pieces attractive to performers and audiences Cory McKay

7. Solutions relating to performers • Performers should not be expected to perform tasks that are alien to their training • Limitations of performer bandwidth must be considered • A clear explanation must be made of specifically how the system will respond to their actions • Intuitive feedback must be provided • Auditory, visual, haptic • Performers must be given sufficient time to learn how to use the system • Includes significant rehearsal time with functional system Cory McKay

8. Solutions relating to system robustness • Software should be well tested • Creative ambition should be tempered by technical realism • Simple equipment should be used whenever possible • The smaller the amount of electronics the better • Systems must be error tolerant • Not fooled by minor mistakes and should not fail entirely if major mistakes occur • Software interface should be simple • Basic overrides should be available Cory McKay

9. Solutions relating to dissemination and longevity • Only commonly available, cheap and easy-to-transport equipment should be used • Well-known software that is likely to have a long life-span should be used whenever possible • Applies to operating system and implementation language • Any need for proprietary or unique hardware or software should be avoided unless absolutely necessary • Systems should require only minimal supervision during performance • What supervision is needed should involve a simple, intuitive and well-documented interface Cory McKay

10. Advantages of standardization • Standardized performance parameter extractor would be beneficial: • Composers would not need to implement their own systems • Professional design would cause system to be more robust • Concert venues could be pre-equipped with system • Technicians at venues would be accustomed to the system • Addresses problems with reliability and dissemination of compositions as well as robustness • Mapping system should not be standardized • Would compromise creative freedom • Could use a client-sever architecture to facilitate porting of parameter extraction system to mapping system Cory McKay

11. Advantages of exclusive use of parameters derived from audio • Performers do not need to spend time learning modified instruments or new performance gestures • Increased familiarity makes performers less likely to make mistakes • Miniaturization and wireless technology can make microphones essentially invisible to performers • Rich sound of original instruments is available as well as information previously provided only by MIDI instruments • Only hardware needed is microphones and computer • Unlikely to become obsolete anytime soon • Reliable technology • Cheap, portable and easy to set up • Could mic audio output of hyper-instruments Cory McKay

12. Parameters to extract I • Want a general-purpose system • A large number of parameters provide a diverse palette • Too many parameters can be problematic • Compromises simplicity • Increases latency • Implement only the most clearly useful parameters • Should be sufficient for most pieces • Additional parameters can be added as plug-ins Cory McKay

13. Parameters to extract II • Primary output is MIDI transcription: • Note Ons and Note Offs • Time, pitch, velocity • Stream of onsets detected • Frequencies take longer to detect than onsets • Note Ons therefore have a greater latency than onset detections • Stream of detected frequency values • Needed for microtonal pieces and recognition of techniques such as glissandi • Overall signal amplitude • Gives an indication of overall dynamics Cory McKay

14. Parameters to extract III • Additional features related to timbre • This potentially very rich source of information is unavailable from MIDI instruments • e.g. relative amounts of energy in different spectral regions • More sophisticated features can be extracted from these basic parameters • Chord classification, beat tracking, note density, etc. Cory McKay

15. Inaudible cues • Performers often communicate with each other inaudibly • e.g. visual gestures • Could also be useful to communicate with an interactive system inaudibly • e.g. manual state switching if a cue is missed • Should avoid sophisticated, expensive or unusual equipment or anything requiring specialized training • Cameras and a standardized image processing system could provide a good long-term solution • Basic footswitches Cory McKay

16. Difficulties with real-time transcription • Monophonic transcription is known to work well • Can still have problems with latency • Accurate polyphonic transcription still an unsolved problem • Latency an even greater problem here • Can use localized microphones in ensembles to help with polyphonic transcription • Turns hard polyphonic transcription problem into multiple easier monophonic transcription problems • Does not solve problem of polyphonic instruments • Piano, guitar, harp, etc. • Advances in transcription technology will make real-time transcription more feasible as time goes by Cory McKay

17. Conclusions • There are currently significant problems relating to longevity, dissemination and reliability • A variety of fundamental guidelines have been presented to address these issues • Promising long-term solution to these problems: • A standardized real-time control parameter acquisition system relying only on auditory input Cory McKay