computer music researches in ntuee
Download
Skip this Video
Download Presentation
Computer Music Researches in NTUEE

Loading in 2 Seconds...

play fullscreen
1 / 56

Computer Music Researches in NTUEE - PowerPoint PPT Presentation


  • 723 Views
  • Uploaded on

Computer Music Researches in NTUEE. Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication Engineering National Taiwan University. Research Areas of Shyh-Kang Jeng. Computer Music (JCMG) Agent Systems and Multimedia (JAMG)

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Computer Music Researches in NTUEE' - Faraday


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
computer music researches in ntuee

Computer Music Researchesin NTUEE

Shyh-Kang Jeng

Department of Electrical Engineering/

Graduate Institute of Communication Engineering

National Taiwan University

research areas of shyh kang jeng
Research Areas of Shyh-Kang Jeng
  • Computer Music (JCMG)
  • Agent Systems and Multimedia (JAMG)
  • Electromagnetic Theory and Applications (JEMG)
outline
Outline
  • Sound Output and Separation
  • Sound Synthesis
  • Music Information Retrieval
  • Music Perception
advantages of interactive 3 d sound system
Advantages of Interactive 3-D Sound System
  • Sound Localization
  • Spatial Impression
  • Fast Computer & DSP
  • It‘s very cheap!!!
what is hrtf
What is HRTF ?
  • HRTF(Head Related Transfer Functions) 在自由聲場下,聲音傳遞至人耳耳道的轉移函數,為達成聽覺虛擬實境最重要的關鍵.
  • 3D Sound=HRTF*(Mono Sound Source)
  • Getting HRTF -- By Measurement or Simulation
slide8
仰角

0

°

清大的

HRTF

麻省理工

HRTF

方位角

0

°和

可以辨認

:一人

可以辨認

:四人

180

°時的比較

不能辨認

:四人

不能辨認

:一人

方位角

0

°時

可以辨認

:四人

可以辨認

:四人

不能辨認

:一人

不能辨認

:一人

方位角

90

°時

均可辨認

均可辨認

方位角

180

°時

可以辨認

:四人

可以辨認

:四人

不能辨認

:一人

不能辨認

:一人

方位角

270

°時

均可辨認

均可辨認

HRTF的聽覺比較
virtual concert hall
10

unit : m

8

6

z - a x i s

4

2

0

10

8

6

4

2

0

-2

-4

10

8

6

-6

4

2

0

-8

-2

-4

-6

-10

-8

y - a x i s

-10

x - a x i s

Virtual Concert Hall
viess architecture
VIESS Architecture

irc2

simulation

Room

environment

HRTFL filter

Sound source

(*.wav file)

RTF filter

HRTFR filter

viess demonstration
Weak

Strong

VIESS Demonstration

Room

environment

irc2

simulation

Sound

Source

(*.wav file)

RTF filter

Original

slide15
歌曲人聲消除
  • 原音 軟體消音 自製消音
outline1
Outline
  • Sound Output and Separation
  • Sound Synthesis
  • Music Information Retrieval
  • Music Perception
physical modeling synthesis
Physical Modeling Synthesis
  • Synthesize sound of a musical instrument through computation based on its sound-generation model
  • The artificial instrument can have the same parameters as the real instrument
  • Expressivity of control is unlimited
outline2
Outline
  • Sound Output and Separation
  • Sound Synthesis
  • Music Information Retrieval
  • Music Perception
music retrieval service
Music Retrieval Service

Key-Melody

Notes

Key-MelodySearching Unit

MIDI Files

The Best Match

Desired Music

MIDI Sequencer

example of melody contour ds 1 st measure of moon river
Example of Melody Contour DS(1st Measure of “Moon River”)

67 4 –1 –1 –1 –1 1

3

4

1 4 5 7 8 9 10

edit distance computation
Edit Distance Computation
  • p={0 0 -4 2 0 0 -3} q={0 0 0 2 2 -2}
music query system prototype
Music Query System Prototype

Put the note into the piano bar

Choose what kinds of notes

Message Board

outline3
Outline
  • Sound Output and Separation
  • Sound Synthesis
  • Music Information Retrieval
  • Music Perception
integrated recognition model
Rhythm Recognition

Pitch Recognition

Timbre Recognition

Chord Recognition

Time-frequency analysis

Pattern recognition

Integrated Recognition Model

STFT

Constant-Q

Wavelet

slope of the attack
Slope of the Attack

Violin

Flute

AcousticPiano

Trumpet

String Ensembles

AcousticGuitar

results of timbre recognition unit in monophonic environments
Results of Timbre Recognition Unit in Monophonic Environments

Total Recognition rate = 80.56%

results of timbre recognition unit in polyphonic environments
Results of Timbre Recognition Unit in Polyphonic Environments

Total Recognition rate = 49.07%

models of chord classification
Cochlea

Cerebral

Cortex

Perception

Neural

Network

Wavelet

Transform

Recognized

Chords

Models of Chord Classification
transcription example
Transcription Example

Original

Transcription Result

the octave problem
2.5

2

Magnitude

1.5

1

0.5

0

0

200

400

600

800

1000

1200

1400

Frequency(Hz)

5

4.5

4

3.5

3

Magnitude

2.5

2

1.5

1

0.5

0

0

200

400

600

800

1000

1200

1400

Frequency (Hz)

The Octave Problem
pitch detection system
Digital audio signal

Constant-Q transform

Spectral peaks

64

Spectral peaks for every 64 samples

Grouping

Series G1

Series i

Series C8

sustained or silent

Onset/offset detection

off

on

Series i

Octave detector for series i

Pitch(es)

clear

Pitch memory for series i

Pitch Detection System
octave detection example
Octave Detection Example

Original

Test Results

beat tracking system
Beat Tracking System
  • Off-line method
  • Input signal sampling rate : 8 kHz
  • Platform : MATLAB
musical sound recognition model using som
Musical-Sound Recognition Model using SOM

Front-end Processing

The best-matching neuron is fired

experiment i musical notes in different pitches
Experiment I─ Musical-notes in different pitches
  • Input Pattern:
    • Timbre: Piano
    • Pitch: C3 ~ C5, include those black-notes
      • C,C#: (Red)
      • D,D#: (Orange)
      • E: (Yellow)
      • F,F#: (Deep Green)
      • G G#: (Light Green)
      • A A#: (Blue)
      • B: (Purple)
experiment result i learning process of som
Experiment Result (I) ─ Learning Process of SOM
  • Let’s see the learning process of the map while it listens to piano’s single-notes in different pitches over and over.
  • The map is in disorder at first.
experiment result i learning process of som cont
Experiment Result (I) ─ Learning Process of SOM (cont.)
  • As the iteration number goes up, the map is stable and well-organized.
experiment ii musical notes in different timbres
Experiment II─ Musical-notes in different timbres
  • Input Pattern:
    • Pitch: G3 A4 C4
    • Timbre: Piano (Red), Violin (Orange), Oboe (Yellow), Clarinet (Deep Green), Trumpet (Light Green) , Trombone (Purple), French Horn (Blue)
experiment result ii organization of notes in different timbres
Experiment Result (II) ─ Organization of notes in different timbres
  • For notes in different timbres and pitches, memories are mainly organized by pitches.
    • Piano (Red)
    • Violin (Orange)
    • Oboe (Yellow)
    • Clarinet (Deep Green)
    • Trumpet (Light Green)
    • Trombone (Purple)
    • French Horn (Blue)
ad