Development of a CALL system for Australian learners of Japanese Pronunciation

1. Development ofa CALLsystem forAustralian learners of JapanesePronunciation Dean Luo*，Nobuaki Minematsu**，Chiharu Tsurutani*** Yutaka Yamauchi**** and Keikichi Hirose* *Grad. School of Info. Sci. and Tech., Univ. of Tokyo **Grad. School of Frontier Sciences, Univ. of Tokyo *** School of Languages and Linguistics, Griffith University ****School of Business and Commerce, Tokyo International University

2. Outline • Introduction • Background • Previous work • Objectives of the project • Description of the project and the system • Development process of the system • Evaluations of the program from 3 perspectives • System performance evaluation • Phrase evaluation correlation between the system and the teacher • Students’ responses to program evaluation • Discussions and future work

3. Introduction • Background • The number of learners of Japanese language has increased dramatically, especially in Australia • There are not enough Japanese teachers • CALL (Computer Aided Language Learning) • Previous works • Automatic pronunciation scoring for language instruction (H.Franco, 1997) • Development of Japanese speech database read by non-native speakers for constructing CALL system (Minematsu et al, 2004) • Objectives of the project • Develop a CALL sysetem that focus on individual student • CALL that simulates the evaluation strategy of individual teacher

4. The project • Bring together language teachers with speech engineers • Create a database of individual teacher’s evaluation • System tunings focused on individual students • develop a program for self assessment • Analyze Japanese pronunciation by Australian students • Simulate the teacher’s evaluation of pronunciation, error detection, and feedback with the diagnostic information

5. The outline of the CALL system Analysis Server Web Server Log Server

6. User mode of the program

7. Administration mode

8. Categorization of pronunciation error patterns • Error patterns based on the experience and knowledge of the teacher • 15 segmental phoneme & prosodic error patterns • Additional error patterns by analyzing students’ speech data. • Speech data • 27 beginners and 19 intermediate Australian learners of Japanese • Articulation of 30 Japanese sentences that contain the 15 error patterns predicted by the teacher. • Transcription of the speech data • Transcribers • 15 Japanese university students majoring in linguistics who passed a short test to check the accuracy • when more than two people agreed, the transcription was judged correct • 4 more error patterns added to the list

9. Error pattern list 15 error patterns predicted by the teacher and 4 additional error patterns list proceeding from the most frequent error to the least frequent one

10. Most common patterns • Most common error patters: • 1.insertion of a consonante.g. hito (ひと） → hiqto（ひっと） • 2. shortening of a vowele.g. oba:san（おばーさん） → obasan（おばさん） • 3. omission of a consonante.g. iqte（いって） → ite（いて） • 4. lengthening of a vowele.g. hito（ひと）→ hi:to（ヒート) • More than 70% of the pronuncition errors were caused by phoneme duration HMM-based acoustic modeling is not capable of handling the length of phoneme Adequately⇔ Post-processing is necessary

11. Post-processing === phoneme alignment begin === id: from to n_score applied HMMs (logical[physical] or {pseudo}) ------------------------------------------------------------ 0: 0 89 -20.832420 silB 1: 90 103 -23.219648 y 2: 104 117 -21.528477 u: 3: 118 130 -24.951847 b 4: 131 141 -22.119362 i 5: 142 158 -24.029957 N 6: 159 183 -27.473242 ky 7: 184 201 -24.261204 o 8: 202 207 -20.726562 q 9: 208 210 -26.312134 k 10: 211 264 -24.831999 u • Post-processing HMM results • Short vowel: if longer than 120[msec], then substitute the corresponding long vowel for it. • Long vowel: if shorter than 10 [msecs], then substitute the corresponding short vowel for it. • Sokuon (double consonant): if shorter than 9 [msecs], then delete the phoneme. • Consonant (especially mora obstruents): if longer than 12 [msecs], then insert a sokuon.

12. Network Grammar 12 13 11 14 15 a: a: i R oi Start End a o k i: r o i k s a 7 10 6 1 4 5 8 9 2 3 q きいろいかさを きろいかさを きろいかっさを きRoいかさを きRoiかさを きろいかーさーを きいろいかさを

13. One example of the network grammar used in the program • Okyakusanni(1,2,5,6)chotto(3,18,5,6) matte(3,18)moratte(3,7,9,18)kudasai(13,16,17) • Learner-dependent grammar

14. Acoustic model • Acoustic Model • Japanese • ATR 4,000 Japanese Adults, gender-independent • Australian English • ANDOSL (Australian National Database of Spoken Language ) • 72 speaker aged 18~45, gender-independent • Left-to-right HMM • Monophone • Triphone • State clustering among Japanese HMMs and Australian English HMMs • Exp. a-R+i

15. System evaluations • System evaluations from 3 perspectives • System performance evaluation • Phrase evaluation correlation between the system and the teacher • Students’ responses to program evaluation

16. 3 sets of network grammars for system performance evaluation • 4 sets of network grammars • G1: apply the 19 error patterns without restriction. • G2: apply the 19 error patterns, considering the position where within the words the phoneme errors occur. • G3: restrict application of the network grammar to certain sounds, taking into consideration the tendency of English speakers’ error patterns. • Average paths per segment of each grammars • G1: 182 paths • G2: 66 paths • G3: 7 paths • Coverage of each grammars • G1: 92.3% • G2: 90.5% • G3: 89.6%

17. System performance evaluation System performance (without post-processing) System performance (after post-processing)

18. Phrase evaluation correlation between the system and the teacher • Segmental score prediction by CART • Data: The teacher’s assessment scores of 586 phrasesfrom 11 speakers • Factors • The number of pronunciation errors • Acoustic scores • Average and variance of vowel durations • Word confidence scores • Correlation between the system and the teacher • Average correlation: 0.75 (Text and speaker open)

19. Students’ responses to program • The evaluation sheet asked the following questions: • 1) Was the program easy to use? 1-1. Recording / 1-2. Evaluation • 2) Was the program fun to use? • 3) Did the program clearly tell weak points in your pronunciation? • 4) Did the program explain well what needs to be done to improve your pronunciation? • 5)Do you think it is useful? • The answers indicated in scores 1(good)-5(poor)

20. Discussions • New method for CALL system • Simulate the individual teacher’s strategies • Detect errors depending on the individual students’ levels • Collaboration between language teachers and engineers • More adaptable to the individual student • Flexibility and extensibility of the system enable individual teachers to administrate and improve the system performance.

21. Future work • Phoneme discrimination method needs to be improved • The best detection rate is about 70% • Detection rate increased dramatically after post processing based on Mora duration • Increase the reliability of the evaluation by the system • Clarify the factors that affect the teacher’s evaluation • Compare the teacher’s evaluation with the system