Evaluating a TBLT Spanish immersion program

1. Evaluating a TBLT Spanish immersion program Greg Burwell – BPA – greg.burwell@dhs.gov Marta Gonzalez-Lloret –UHM – marta@hawaii.edu Katie Nielson – UMD (CASL) – knielson@casl.umd.edu COLLOQUIUM: Evaluating task-based language programs

2. Overview

3. TBLT at the Border Patrol Academy (BPA) • BPA Training Environment • Spanish at the BPA (before and after) • Students, instructors, and supervisors sought practical training • Developed in 2006-2007 • Needs Analysis • Genuine audio/video • Extensive input from actual Border Patrol agents

4. TBLT at the Border Patrol Academy (BPA) • First implemented in December of 2007 • First cohort of 15 students graduated in March 2008 • As of August 2009, 2,813 students have completed the program • To date, the program has started 180 classes with over 3,000 students • 100% pass rate

5. TBLT at the BPA- Curriculum • 8-week program • following Long’s (1985, 2005a) stages for designing and implementing a TBLT program 7 target tasks Several target task types Multiple pedagogic tasks

6. TBLT at the BPA - Curriculum Target Task Types:

7. TBLT at the BPA - Curriculum COMPLEXITY + Lab Activities + La Chamba interactive video game

8. TBLT at BPA- Assessment

9. TBLT at BPA - Assessment

10. TBLT at BPA- Assessment • Multiple versions of end-of-module assessments • Questions considered in each iteration: • Are these elements required to complete the task? • Which are critical? • How do we assess language? • Global accuracy, fluency, and pronunciation ratings • Inclusion of “using appropriate Spanish” with each required task • Removal of any mention of language

11. TBLT at BPA - Assessment

12. TBLT at BPA - Assessment

13. TBLT at BPA - Assessment

14. TBLT at BPA - Assessment • Instructor training • Video evaluation • Role-play evaluation • Reliability • Rubric modification • Perceived grade inflation • Language versus task accomplishment • Non-obligatory steps removed • Average grade changed from 94.10% to 95.48%

15. Pilot Study – Oral narration task Participants Grammar-based group (GB) (n=20) TBLT group (TBLT) (n=20) Oral narrative based on picture story Hypotheses: TBLT will perform better on measures of fluency, lexical complexity and structural complexity GB will perform better on measures of grammatical accuracy

16. Pilot study- cont • Compared on: • Fluency [syllables/second] (Gilabert, 2004) • Lexical complexity • [lexical words/words] (Robinson, 1995; Gilabert, 2004) • [1st time use of lexical word/words] • Ratio of lexical/functional words (Gilabert, 2004) • Structural complexity [words/utterance] (Ortega, 1999) • Grammatical accuracy • TLU of noun-modifier agreement (Ortega, 1999) • TLU of noun-verb agreement

17. Pilot study- Results - Fluency syllables/second

18. Pilot study- Results – Lexical complexity Lexical words/words 1st use lexical w/words Ratio of lexical/ functional words

19. Pilot study- Results- Structural complexity #of words/ utterance

20. Pilot study- Results –Grammatical Accuracy TLU of noun-modifier agreement TLU of noun-verb agreement

21. Pilot study - Discussion

22. Ordinate study Participants First 256 TBLT students Performance on computerized oral proficiency interview Hypotheses: TBLT will improve students’ overall proficiency TBLT will affect overall proficiency regardless of language abilities at the start of the course

23. Ordinate Study - continued • Ordinate (Versant) Computerized Oral Proficiency Exam • Used at BPA as a placement test • Scored between 20 – 80 • Overall • Sentence Mastery • Vocabulary • Fluency • Pronunciation • Students took the test before and after the 8-week TBLT program

24. Ordinate Study - Mean Scores

25. Ordinate Study – Change Scores

26. Ordinate Study - Correlation

27. Ordinate Study - Discussion

28. Students’ evaluations • Electronic (anonymous) surveys • 2 Groups: Pre-experienced Current students of the TBLT Spanish program n=41 Experienced Current BA Agents trained with TBLT Spanish program n=14

29. Students’ evaluation (pre-experienced)

30. Students’ evaluation In general students liked the best… • The work done with the role-players. • The direct application of the class to their job. “Role playing is very helpful. I feel that it helped me learn more spanish (sic) and become more comfortable speaking it.” “I enjoyed the hands on experience the most. Being able to read and write is important when learning a new language, but being able to incorporate it into your daily duties is beneficial.” “I have enjoyed that we get to have a lot of hands on in this program. Also helps a lot for us to be able to apply all that we have learned in basic and build apon (sic) basic.”

31. Students’ evaluations • Students suggestions: • More conversational Spanish (not for the job) • More grammar in class • More role-play work with Spanish speakers

32. Agents’ evaluation (experienced)

33. Agents’ evaluations • What they found most useful from the Spanish course was: • Commands and expressions • Knowing the procedures and forms in Spanish • What they would have liked to learn: • More conversation (outside scenarios) • More vocabulary • Grammar and verb conjugation

34. TBLT at BPA

35. Difficulties with evaluating TBLT at the BPA Difficulty of training instructors properly on methods and principles of TBLT Instructors reluctant to adopt new approach Lack of trained personnel to collect and analyze data Lack of support for follow-up research

36. Thank you Gracias Mahalo Questions?