How Much Does It Help to Score Language on Speaking Activities ?

1. How Much Does It Help to Score Language on Speaking Activities? Ben Taylor The Virginia Tech Language and Culture Institute

2. Background • 8-week, 10 hr/week Grammar, Listening and Speaking (GLS) class in an Intensive English Program • Advancement is not pass-fail for speaking: it is largely the accumulation of in-class speaking assignment grades (70% cutoff) – i.e. the grades matter. • In-class speaking assessed by a standard rubric

3. Overview • Current LCI speaking rubric • My questions • Literature --- • Methodology • Results --- • Discussion and implications • Future research • Questions

4. Standard LCI formative speaking rubric 80% of grade weighted to linguistic elements of speaking

5. (Quick discussion) • How is speaking assessed and graded in your classrooms? • How does linguistic feedback (vocabulary, fluency, grammar, pronunciation, etc.) factor into the grade/assessment?

6. My questions (Pedagogical angst) • In class (not placement), should we penalize students for being at a stage of interlanguage? • Do problems such as limited vocabulary, halted fluency, and imperfect grammar represent a lack of effort? Should they be graded just the same way we grade a lack of preparation for an assignment? • Is there a confusion or conflation between academic and linguistic assessment in our classes? • Would focusing the grade more on content and less on form allow more communicative freedom/risk taking and thus better language development? • For the communicative classroom, does providing mostly linguistic feedback convey an incorrect priority?

7. My questions (Pedagogical angst) • In class (not placement), should we penalize students for being at a stage of interlanguage? • Do problems such as limited vocabulary, halted fluency, and imperfect grammar represent a lack of effort? Should they be graded just the same way we grade a lack of preparation for an assignment? • Is there a confusion or conflation between academic and linguistic assessment in our program? • Would focusing grade on content instead of form allow more communicative freedom/risk taking and thus better language development? • For the communicative classroom, does providing mostly linguistic feedback convey an incorrect priority? Student: “How am I supposed to ‘work on my accent’?”

8. Literature on correction and motivation • Numerous studies have been conducted discussing language anxiety (MacIntyre & Gardner, 1991). • Fear of being corrected, or even penalized, by a professor for incorrect grammar or the inability to understand a certain language concept could keep them from trying to speak in the language at all (Capone, 2013). • According to Krashen’s (1982) affective filter hypothesis, anxiety, including that derived from the premature or excessive correction of student errors, interferes with the process of acquiring a second language. • ZoltánDörnyeidiscusses student motivation in many books and articles.

9. Literature on correction and motivation • There is much research on how motivation affects class grades, but very little on how class gradesaffect motivation. • I began a pilot study in my classes in hopes to open the door for institutionally based research.

10. Research question • Does associating a grade with feedback on linguistic factors of speaking performance in the L2 classroom help to improve speaking proficiency?

11. Methodology

12. Participants • 25 students in two high intermediate Grammar Listening and Speaking courses (GLS 450) at the Virginia Tech Language and Culture Institute • Saudi Arabia(12) • China (11) • Panama (1) • Kuwait (1) • All ages 17-40, majority 18-23 • Signed IRB consent forms • Randomly assigned to test/control groups • Blocked so that ½ students in both classes (8-10a/1-3p) were test and ½ were control • Taught with identical speaking assignments but given feedback by different rubrics

13. Standard Rubric

14. Test Rubric

15. Procedure Pre/post recordings of speaking were used to determine proficiency increase • Pre-test: March 25 2013 • Post-test: May 13 2013 • Both times, participants used a computer-based recorder with a headphone microphone • Recorded four speaking tasks of increasing complexities • Task description visible for 45 seconds/recorder invisible • Task disappears and recorder appears • No time limit

16. Tasks • The tasks contrasted the following three dimensions, following De Jong et al (2012): • complexity(complex versus simple topic) • formality(informal versus formal setting) • discourse type (descriptive versus argumentative) • Tasks varied slightly between pre-post to reduce direct recall but task dimensions remained constant

17. Tasks (pre) • (simple, informal, descriptive): You are speaking on the phone to a friend. Describe your apartment in Blacksburg to him/her. • (simple, formal, descriptive): A few days ago, you saw a car accident. Because you are a witness to the accident, you have been called to a courtroom to describe it to the judge. Tell the judge what happened. • (complex, informal, argumentative): Your friend told you she is planning to take a vacation to Africa. However, you think it is a bad idea. Talk to your friend and try to convince them NOT to go on the vacation. • (complex, formal, argumentative): You are the owner of a large supermarket in your town. You are trying to purchase some public land to build a parking lot. Your town is having a public meeting to decide if they should allow your company to purchase the land, and you allowed to give a speech at the meeting. Explain your reasons why the town should allow the supermarket to buy the land.

18. Tasks (post) • (simple, informal, descriptive): You are speaking on the phone to a friend. Describe the house that your family lives in, in your home country. • (simple, formal, descriptive): A few days ago, you saw a fight between two people. Because you are a witness to the fight, you have been called to a courtroom to describe it to the judge. Tell the judge what happened. • (complex, informal, argumentative): Your friend was planning to take a vacation to Africa. However, now they are saying that maybe they don't want to go. Talk to your friend and try to convince them to go on the vacation. • (complex, formal, argumentative): The owner of a large supermarket in your town is trying to purchase some public land to build a parking lot. Your town is having a public meeting to decide if they should allow the company to purchase the land. You are a citizen of the town and you allowed to give a speech at the meeting. Explain your reasons why the town should NOT allow the supermarket to buy the land.

19. Rating speech samples • Pre-post recording pairs were blindly rated by seven volunteer LCI teachers on scales of 1-10 for vocabulary, grammar, fluency, and pronunciation (the 4 categories on the rubric). • An equal number of test and control samples were given to each rater to eliminate the effect of teacher bias.

20. Rating speech samples • Distribution of samples

21. Rating speech samples • Rater rubric

22. Results

23. Data • Lost significant amounts of data • Class transfers • Students missed pre-test • Students missed post-test • Students recorded different tasks successfully in pre- and post-tests • Students had technical difficulty / submission errors • Unequal number of control/test samples for a given rater • 2516 participants

24. Results • 60% of the possible data is not present, so results must be taken with caution.(Pilot Study) • With the data collected, we could not statistically interpret the overalleffectof the rubric treatment on the proficiency improvement (independent of task type). • Surprisingly we were able to interpret differences compared by task type.

25. According to our analysis, it seems as if there is a downward trend in the test students in terms of their improvement from pretest to posttest, as they move from task to task, compared with the control group.

26. “simple, descriptive”

27. “complex, argumentative”

28. Formality was an element of complexity that added difficulty to both groups.

29. Both groups improved more with the simpler tasks than they did with the more difficult tasks.

30. Control students (normal rubric) reflected more stability of improvement depending on type of task.

31. Test students (test rubric) improved more than control students on the simple tasks, but their improvement decreased noticeably as task difficulty increased.

32. Results • We can tentatively observe that students who received the test rubric (no linguistic feedback associated with grade) performed better on simpler tasks than their partners who received graded language feedback, whereas they performed worse on more difficult tasks.

33. Discussion and Implications

34. Discussion and implications • The study seems to indicate that associating linguistic feedback with class grade suppresses fluency (slower improvement on easy tasks) but elevates accuracy (faster improvement on difficult tasks). In other words it affects the type of improvement more than overall improvement. • Limitations of the study • Possibility of teacher bias (rating was blind but not teaching) • Rubrics’ different formatting probably affected the types of comments and level of linguistic feedback I made (independent of whether it was graded). • Small sample sizes and amount of data

35. Future research • Repeated, expanded study to confirm • Larger sample sizes, >100 students (so the main effects can be measured independent of task type) • Identical format of feedback • 4 levels of grade weight to linguistic sections (20, 40, 60, 80) • Test results for a variety of different proficiency levels • Get large enough results to judge the overall effect of rubric independent of task type • What other aspects of Intensive English program curriculum/assessment affect motivation and language proficiency development? • Inter-program collaboration with a larger-scale follow-up study is welcome!

36. The big questions What kinds of proficiency improvement (fluency/accuracy) align with our class goals? And how does that affect our assessment and feedback? How can our assessment and feedback maximize student motivation?

37. Questions? Feedback?

38. Thank you! Ben Taylor Virginia Tech Language and Culture Institute bft@vt.edu