1 / 53

Recent Research in Musical Timbre Perception

Recent Research in Musical Timbre Perception. James W. Beauchamp University of Illinois at Urbana-Champaign Andrew B. Horner Hong University of Science and Technology Michael D. Hall James Madison University, Harrisonburg, VA. Starting Point.

bebe
Download Presentation

Recent Research in Musical Timbre Perception

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Recent Research in Musical Timbre Perception James W. Beauchamp University of Illinois at Urbana-Champaign Andrew B. Horner Hong University of Science and Technology Michael D. Hall James Madison University, Harrisonburg, VA

  2. Starting Point • Timbre experiments are based on musical instrument sounds.

  3. Starting Point • Timbre experiments are based on musical instrument sounds. • Perform short-time spectral analysis.

  4. Starting Point • Timbre experiments are based on musical instrument sounds. • Perform short-time spectral analysis. • Identify parameters of ST spectrum:

  5. Starting Point • Timbre experiments are based on musical instrument sounds. • Perform short-time spectral analysis. • Identify parameters of ST spectrum: • Partial (harmonic) amplitudes - Time variation - Spectral envelope (centroid, irregularity, etc.)

  6. Starting Point • Timbre experiments are based on musical instrument sounds. • Perform short-time spectral analysis. • Identify parameters of ST spectrum: • Partial (harmonic) amplitudes - Time variation - Spectral envelope (centroid, irregularity, etc.) • Partial (harmonic) frequencies - Time variation - Inharmonicity

  7. Methods for Studying Timbre Stimuli Preparation In Freq. Domain • Simplification • Perturbation • Normalization

  8. Methods for Studying Timbre • Listener Experiments • Discrimination (pairs) • Timbral Distance Estimation • Classification • Identification Stimuli Preparation In Freq. Domain • Simplification • Perturbation • Normalization

  9. Methods for Studying Timbre • Listener Experiments • Discrimination (pairs) • Timbral Distance Estimation • Classification • Identification Stimuli Preparation In Freq. Domain • Simplification • Perturbation • Normalization • Data Processing/Presentation • Discrimination (sensitivity) scores/plots • Multidimensional Scaling • Correspondence (R2) Measurements

  10. Studies Reviewed • 1999 Discrimination Study • 2006 Discrimination Study • 2006 Multidimensional Scaling (MDS) Study • 2009 Discrimination/Classification Study

  11. 1999 Discrimination Study(McAdams, Beauchamp, Meneguzzi, JASA) • Seven reference sounds • clarinet, flute, oboe, trumpet, violin, harpsichord, marimba

  12. 1999 Discrimination Study(McAdams, Beauchamp, Meneguzzi, JASA) • Seven reference sounds • clarinet, flute, oboe, trumpet, violin, harpsichord, marimba • Equalize F0, loudness, and duration.

  13. 1999 Discrimination Study(McAdams, Beauchamp, Meneguzzi, JASA) • Seven reference sounds • clarinet, flute, oboe, trumpet, violin, harpsichord, marimba • Equalize F0, loudness, and duration. • Test sounds: Apply six spectrotemporal simplifications.

  14. 1999 Discrimination Study(McAdams, Beauchamp, Meneguzzi, JASA) • Seven reference sounds • clarinet, flute, oboe, trumpet, violin, harpsichord, marimba • Equalize F0, loudness, and duration.. • Test sounds: Apply six spectrotemporal simplifications. • Subjects discriminate between original and simplified sounds.

  15. 1999 Discrimination StudyResults Discrim Score • Spectral envelope smoothing 96% • Spectral flux elimination 91% • Amplitude envelopes smoothing 66% • Frequency envelopes smoothing 70% • Freq. envs. harmonic locking 69% • Frequency variations elimination 71%

  16. 2006 Discrimination StudyHorner, Beauchamp, and So JAES • Eight sustained musical instrument tones • bassoon, clarinet, flute, horn, oboe, alto sax, trumpet, violin

  17. 2006 Discrimination StudyHorner, Beauchamp, and So JAES • Eight sustained musical instrument tones • bassoon, clarinet, flute, horn, oboe, alto sax, trumpet, violin • Modified by fixed random transfer function

  18. 2006 Discrimination StudyHorner, Beauchamp, and So JAES • Eight sustained musical instrument tones • bassoon, clarinet, flute, horn, oboe, alto sax, trumpet, violin • Modified by fixed random transfer function • F0, loudness, duration, centroid preserved

  19. 2006 Discrimination StudyHorner, Beauchamp, and So JAES • Eight sustained musical instrument tones • bassoon, clarinet, flute, horn, oboe, alto sax, trumpet, violin • Modified by fixed random transfer function • F0, loudness, duration, centroid preserved Typical spectral envelopes:

  20. 2006 Discrimination StudyHorner, Beauchamp, and So JAES • Objective: To discover which metrics based on the time-varying harmonic amplitudes give the best correspondence to discrimination between original and modified tones.

  21. 2006 Discrimination StudyHorner, Beauchamp, and So JAES • Objective: To discover which metrics based on the time-varying harmonic amplitudes give the best correspondence to the discrimination data. • Best results: obtained by relative-amplitude (harmonic) spectral error:

  22. 2006 Discrimination StudyHorner, Beauchamp, and So JAES Discrimination vs. error level (): R2=0.81

  23. 2006 Discrimination Study Horner, Beauchamp, and So JAES Discrimination vs. rel-amp spec error: R2=0.90 for a=1.0

  24. 2006 MDS Study Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) • Ten sustained musical instrument tones • bassoon, cello, clarinet, flute, horn, oboe, recorder, alto sax, trumpet, violin

  25. 2006 MDS Study Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) • Ten sustained musical instrument tones • bassoon, cello, clarinet, flute, horn, oboe, recorder, alto sax, trumpet, violin • F0, loudness, duration, attack & decay times, and average spectral centroid are equalized.

  26. 2006 MDS Study Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) • Ten sustained musical instrument tones • bassoon, cello, clarinet, flute, horn, oboe, recorder, alto sax, trumpet, violin • F0, loudness, duration, attack & decay times, and average spectral centroid are equalized. • Two types of tones: static (flux removed) and dynamic (flux retained).

  27. 2006 MDS Study Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) • Ten sustained musical instrument tones • bassoon, cello, clarinet, flute, horn, oboe, recorder, alto sax, trumpet, violin • F0, loudness, duration, attack & decay times, and average spectral centroid are equalized. • Two types of tones: static (flux removed) and dynamic (flux retained). • Subjects estimate timbral dissimilarity between instruments.

  28. 2006 MDS Study Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) • Ten sustained musical instrument tones • bassoon, cello, clarinet, flute, horn, oboe, recorder, alto sax, trumpet, violin • F0, loudness, duration, attack & decay times, and average spectral centroid are equalized. • Two types of tones: static (flux removed) and dynamic (flux retained). • Subjects estimate timbral dissimilarity between instruments. • Data processed by two multi-dimensional scaling (MDS) programs (SPSS & Matlab).

  29. 2006 MDS StudyBeauchamp, Horner, Koehn, and Bay (ASA Honolulu) Acoustical Correlates to Test: • Even/Odd: Ratio of even and odd harmonic rms amplitudes.

  30. 2006 MDS StudyBeauchamp, Horner, Koehn, and Bay (ASA Honolulu) Acoustical Correlates to Test: • Even/Odd: Ratio of even and odd harmonic rms amplitudes • Spectral IRregularity: Degree of jaggedness of a spectrum.

  31. 2006 MDS StudyBeauchamp, Horner, Koehn, and Bay (ASA Honolulu) Acoustical Correlates to Test: • Even/Odd: Ratio of even and odd harmonic rms amplitudes • Spectral IRregularity: Degree of jaggedness of a spectrum. • Spectral Centroid Variation: Standard deviation of the spectral centroid normalized by average value. For Dynamic Tones Only:

  32. 2006 MDS StudyBeauchamp, Horner, Koehn, and Bay (ASA Honolulu) Acoustical Correlates to Test: • Even/Odd: Ratio of even and odd harmonic rms amplitudes • Spectral IRregularity: Degree of jaggedness of a spectrum. • Spectral Centroid Variation: Standard deviation of the spectral centroid normalized by average value. • Spectral INcoherence: Degree of spectral change relative to the average spectrum (same as flux). For Dynamic Tones Only:

  33. 2006 MDS StudyBeauchamp, Horner, Koehn, and Bay (ASA Honolulu) 2D MDS Results: Correlations: E/O: R=0.78 SIR: R=0.69 Static Tone Case SPSS algorithm Stress=0.12

  34. 2006 MDS StudyBeauchamp, Horner, Koehn, and Bay (ASA Honolulu) 2D MDS Results: Correlations: E/O: R=0.79 SIR: R=0.75 Static Tone Case Matlab algorithm Stress=0.12

  35. 2006 MDS StudyBeauchamp, Horner, Koehn, and Bay (ASA Honolulu) 2D MDS Results: Correlations: E/O: R=0.71 SCV: R=0.68 SIN: R=0.56 SIR: R=0.39 Dynamic Tone Case SPSS algorithm Stress=0.17

  36. 2006 MDS StudyBeauchamp, Horner, Koehn, and Bay (ASA Honolulu) 2D MDS Results: Correlations: E/O: R=0.69 SCV: R=0.68 SIN: R=0.53 SIR: R=0.40 Dynamic Tone Case Matlab algorithm Stress=0.15

  37. 2009 StudyHall and Beauchamp (Canadian Acoustics) • Goals/Purpose • Exp. 1. Relative importance of spectral vs. temporal cues: Compare listener discrimination and classification performance for interpolations between two (impoverished) instruments with respect to spectral envelope and amplitude-vs.-time envelope.

  38. 2009 StudyHall and Beauchamp (Canadian Acoustics) • Goals/Purpose • Exp. 1. Relative importance of spectral vs. temporal cues: Compare listener discrimination and classification performance for interpolations between two (impoverished) instruments with respect to spectral envelope and amplitude-vs.-time envelope. • Exp. 2. Relative importance of spectral envelope (formant) structure vs. spectral centroid: Compare discrimination/classification performance for interpolated tones vs. tones obtained by filtration which matches the centroids of the interpolated tones.

  39. 2009 StudyHall and Beauchamp (Canadian Acoustics) • Experiment 1 Method • Reference stimuli: Impoverished (static) violin and trombone sounds. (Each sound has a fixed spectrum and a single amplitude-vs.-time envelope.)

  40. 2009 StudyHall and Beauchamp (Canadian Acoustics) • Experiment 1 Method • Reference stimuli: Impoverished (static) violin and trombone sounds. (Each sound has a fixed spectrum and a single amplitude-vs.-time envelope.) • Test stimuli: A 44 array of sounds is created based on interpolations of the spectral envelope and the amplitude-vs-time envelope between the violin and trombone timbres. Temporal Vn I01 I02 I03 I10 I11 I12 I13 I20 I21 I22 I23 I30 I31 I32 Tr Spectral

  41. 2009 StudyHall and Beauchamp (Canadian Acoustics) • Experiment 1 Method • Reference stimuli: Impoverished (static) violin and trombone sounds. (Each sound has a fixed spectrum and a single amplitude-vs.-time envelope.) • Test stimuli: A 44 array of sounds is created based on interpolations of the spectral envelope and the amplitude-vs-time envelope between the violin and trombone timbres. • Interpolation steps: Test tones differ from reference (original) tones by 1, 2, or 3 steps along either the spectral envelope or amplitude envelope dimension or both (3 steps gives the opposite timbre.).

  42. 2009 StudyHall and Beauchamp (Canadian Acoustics) • Experiment 1 Method • Reference stimuli: Impoverished (static) violin and trombone sounds. (Each sound has a fixed spectrum and a single amplitude-vs.-time envelope.) • Test stimuli: A 44 array of sounds is created based on interpolations of the spectral envelope and the amplitude-vs-time envelope between the violin and trombone timbres. • Interpolation steps: Test tones differ from reference (original) tones by 1, 2, or 3 steps along either the spectral envelope or amplitude envelope dimension or both (3 steps gives the opposite timbre.). • Subjects’ tasks: • 1) to discriminate tone pairs. • 2) to classify tones as ‘violin’, ‘trombone’, or ‘other’.

  43. 2009 StudyHall and Beauchamp (Canadian Acoustics) Experiment 1 Results Discrimination: reference stimuli: Note: Low sensitivity to temporal changes. High sensitivity to spectral changes.

  44. 2009 StudyHall and Beauchamp (Canadian Acoustics) Experiment 1 Results Classification: Note: Low sensitivity to temporal changes. High sensitivity to spectral changes.

  45. 2009 StudyHall and Beauchamp (Canadian Acoustics) • Experiment 2 Method • Reference stimulus: Original impoverished violin.

  46. 2009 StudyHall and Beauchamp (Canadian Acoustics) • Experiment 2 Method • Reference stimulus: Original impoverished violin. • Test stimuli: • 1) 3 violin tones interpolated with respect to spectral envelope (steps 1-3) (original violin amp env kept).

  47. 2009 StudyHall and Beauchamp (Canadian Acoustics) • Experiment 2 Method • Reference stimulus: Original impoverished violin. • Test stimuli: • 1) 3 violin tones interpolated with respect to spectral envelope (steps 1-3) (original violin amp env kept). • 2) 3 violin tones low-pass filtered in steps to match the spectral centroids of the 3 interpolated tones of 1).

  48. 2009 StudyHall and Beauchamp (Canadian Acoustics) • Experiment 2 Method • Reference stimulus: Original impoverished violin. • Test stimuli: • 1) 3 violin tones interpolated with respect to spectral envelope (steps 1-3) (original violin amp env kept). • 2) 3 violin tones low-pass filtered in steps to match the spectral centroids of the 3 interpolated tones of 1). • Subjects’ tasks: Discrimination and classification as in Exp. 1. (Which has the greater effect? Interpolation or filtration?)

  49. 2009 StudyHall and Beauchamp (Canadian Acoustics) Experiment 2 Results Discrimination: Classification:

  50. Conclusion Summary • 1999 discrimination study: • Spectral envelope detail and spectral flux are important for dynamic musical sounds, and they are more important than temporal detail.

More Related