1 / 21

NCOLTCL 2011 Brigham Young University Shinsuke Tsuchiya

E licited I mitation and Automated S peech Recognition as a Placement Test: Analysis Of JSL Learners’ Language Development Process And Differences Between Instructed And Self-Instructed Learners . NCOLTCL 2011 Brigham Young University Shinsuke Tsuchiya. Overview. Introduction

clara
Download Presentation

NCOLTCL 2011 Brigham Young University Shinsuke Tsuchiya

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Elicited Imitation and Automated Speech Recognition as a Placement Test:Analysis Of JSL Learners’ Language Development Process And Differences Between Instructed And Self-Instructed Learners NCOLTCL 2011 Brigham Young University Shinsuke Tsuchiya

  2. Overview • Introduction • Purpose • Background • Research Questions • Test Design • Results • Analysis • Limitations • Implications • Future Research

  3. Introduction Assessment and Testing using technology can be: • Quick • Reliable • Objective Brigham Young University • Problem: Placement test for incoming learners is all multiple-choice questions (near-native learners, instructed and uninstructed learners) “Can we use Elicited Imitation and Automated Speech Recognition as a speaking portion of the placement test?”

  4. Background • Elicited Imitation • Automated Speech Recognition 1. Ohayoogozaimasu ‘Good morning.’ 2. _______________ “the more you know of a foreign language, the better you can imitate the sentences of the language.” (p. 247) Bley-Vroman& Chaudron (1994) A viable oral testing method (Erlam, 2006; Graham et al., 2008)

  5. BackgroundEnglish EI BYU PSST Research Group http://psst.byu.edu (Pedagogical Software and Speech Technology) • 1659 tests were administered 1279 ESL subjects (Lonsdale & Weitze, 2009) • “overall comparisons between EI scores of ESL learners and their scores on other measures of oral language proficiency are promising” (Hansen, Graham, Brewer, Brewer, & Tieocharoen) • “EI scores can predict Oral Proficiency Interview (OPI) scores within two levels of a scale of ten. (Graham, 2006; Lonsdale, Graham, & Madsen, 2005).

  6. BackgroundJapanese EI test using ASR Over 800 JSL learners have taken the EI. • the correlations between human grading and ASR grading was r=0.84 (p < 0.001) • the correlation between Japanese EI score and OPI scores was r=0.77 (p < 0.3) (Matsushita & LeGare, 2010) Sound recording ソノナカニナニハイッテンノ ‘sononakaninanihaitten no’ Student’s utterance ソノナカナニハイッテタショ ‘sononakaninanihaittetasyo’ Evaluation: C CCCD C CCCCCS SI Correct = 76.92% 10 ( 10) substitutions = 15.38% 2 ( 2) Deletions = 7.69% 1 ( 1) Insertions = 7.69% 1 ( 1) Errors = 30.77% 4 ( 4)

  7. Research Questions Can elicited imitation and automated speech recognition grading system be used as a placement test? • RQ 1. Can they differentiate instructed beginning, intermediate, and advanced learners of Japanese? • If so, what EI items? • RQ 2. Can they differentiate instructed and uninstructed learners? • If so, what EI items?

  8. Test Design • A total number of participants: 241 BYU JSL learners • EI was taken in the computer lab in Fall 2010 • 3-seconds pause before repeating back (Erlam, 2006) • 60 items (30 from textbooks, 30 from CSJ corpus) (15 min.) • Gender-neutral contents • no technical terms • Sentence length: 10-30 morae • Randomized order • Prompts (both male and female) • ASR graded mean scores for each level as well as individual items were compared (t-test) • Participants were divided into different groupsfor each research question

  9. Test items • Beginning (20) Ex. 明日行きませんか。 ‘asitaikimasenka’ ‘won’t you go tomorrow?’ • Intermediate (20) Ex. 昨日は学生さんが手伝ってくれたんですよ。 ‘kinoowagakusee san gatetudattekureta n desuyo.’ ‘As for yesterday, a student helped (me) as a favor.’ • Advanced (20) Ex. 日本がどんな国なのか知りたがっている人がたくさんいます。 ‘nihongadonnnakunina no kasiritagatteiruhitogatakusanimasu. ‘There are many people who are eager to know what kind of country Japan is.’

  10. Research Question 1 Can EI and ASR differentiate instructed beginning, intermediate, and advanced learners of Japanese? Instructed learners: • *1styear: beginning: 79 • 2nd year: intermediate: 34 • **3rdyear +: advanced: 21 *False beginners have been excluded **Native speakers and uninstructed learners who have been excluded.

  11. Research Question 2 Can EI and ASR differentiate instructed and uninstructed learners? • *Advanced Instructed learners: 38 • **1st semester Uninstructed learners in 3rd year level: 40 *second half of 2nd year students were added to equalize number of participants. **uninstructed learners who had more than 2 semesters of instruction have been excluded.

  12. Results (RQ 1) Beginning, intermediate, advanced learners’ mean scores were significantly different: Mean scores: • Beginning: 34.7 • Intermediate: 45.3 • Advanced: 58.8 p-values • Beginning vs. Intermediate: p=0.000 • Intermediate vs. Advanced: p=0.001 • Beginning vs. Advanced: p=0.000

  13. Results (RQ 1)Significant itemsAccuracy rate All progressed: • Beginning-intermediate 40 items • Intermediate-advanced 25 items • Beginning-advanced 58 items

  14. Results (RQ 2) Advanced instructed learners and 1st semester uninstructed learners in 3rd year level were also significantly different. Mean scores: instructed learners: 56.6 uninstructed learners: 63.4 p=0.034

  15. Results (RQ 2)Significant itemsAccuracy Rate 17 significant items Uninstructed learners > instructed learners (16 items) Instructed learners > uninstructed learners (1 item)

  16. Analysis (Summary) 1. EI and ASR used in this research can be used to differentiate: • beginning, intermediate, and advanced level learners. • Instructed and uninstructed learners. 2. JSL learners’ accuracy rate improved in each item in general: • beginning level items (beginning to intermediate) • advanced level items (intermediate to advanced) 3. Uninstructed learners outperformed instructed learners.

  17. Limitations Test • Technological problem (background noise, mis-recording, data loss etc.) • Face validity • L2 interlanguage errors that are not recognized by the present ASR system • Discouragement Subjects • Lack of advanced level instructed learners

  18. Implications EI and ASR could be used as a: • Placement test (listening/speaking portion) only if it is accompanied by other tests. • Computer Adaptive Test (efficient time & cost with reliable & consistent scores). • Program Review (inter/intraprogram assessment and evaluation) • Second Language Acquisition Research

  19. Future Research Test improvement: • Prevent technological problems • Increase face validity (include comprehension tasks) • Improve ASR (dealing with L2 interlanguage) • Increase Motivation • More advanced level classroom students Second Language Acquisition Research: • Item analysis • Grammatical elements (i.e. case particles) • Sentence complexity and length • Vocabulary acquisition

  20. Acknowledgement • Hitokazu Matsushita (Japanese EI/ASR creator) • Ray Graham, Ray Clifford, Dan Dewey (BYU Linguistics department) • Robert Russell, Paul Warnick and other BYU Japanese professors • BYU Japanese students • BYU JFSB Computer lab assistants

  21. Thank you! Questions? Shinsuke Tsuchiya s.tsuchiya777@gmail.com

More Related