Rosalie Hirch April 26, 2013

1. Developing Theory-Based Diagnostic Tests of English Grammar: Application of ProcessabilityTheory Rosalie Hirch April 26, 2013

2. Order of the Presentation • Introduction • Literature Review: Processability Theory (PT) & Diagnostic Language Tests • Hierarchies • Errors • Task Types • Method • Participants • Instruments • Analyses • Results • Discussion, Limitations, & Conclusions

3. Introduction:Background & Motivation • Bridging the gap between testing and the classroom • Previous Research in Diagnostic Language Assessment • Empirical-based • Theory-based • Processability Theory • Already used for tests (RapidProfile) • Is it sufficient for diagnostic tests?

4. Introduction:Major Goals & Aims of the Study • To evaluate the reliability of a diagnostic grammar test for middle school students • To explore theoretical approaches to diagnostic language assessment • To investigate the application of Processability Theory for diagnostic grammar tests

5. Literature Review Processability Theory & Diagnostic Language Tests

6. Hierarchies ProcessabilityTheory • Based on Lexical Functional Grammar • Levels are implicational • Levels come from grammar tree • Problem: the PT hierarchy is very limited

7. Hierarchies ProcessabilityTheory S’-Procedure S-Procedure S-Procedure (Phrase) (Phrase) Phrasal Procedure (Phrase) N V D N SC N V PrP N Category Procedure was John Susan while tennis. decorated a cake playing Word/ Lemma

8. Hierarchies Diagnostic Tests • Other educational diagnostic tests also use hierarchies • Used for analyzing problems • Some are implicational • Tend to be very broad (covering as much as possible) • Suggestion that grammar, in particular, must cover a lot

9. Errors ProcessabilityTheory • Learners tend to make 2 types of errors • These account for interlanguages Is she at home? (Target Sentence) She Ø at home? (Deletion) She is at home? (Overuse)

10. Errors Diagnostic Tests • The primary focus of diagnostic tests • Can potentially show 2 elements in learner performance • Where the problem lies (error—observable outcome) • What thinking led to the error (weakness—underlying problem) • Requires careful planning • Before: Item Design • After: Rubric Design

11. Types of Tasks ProcessabilityTheory • Emphasis on implicit knowledge (automaticity) • Based on Levelt’s Speaking Model • Tasks tend to be productive (speaking, writing) • Analysis is done afterwards

12. Types of Tasks Diagnostic Tests • It is possible to use productive tasks, but not optimal • Difficult to control contexts • More likely to be discrete and, as a result “inauthentic” • Tasks from Norris (2005) and Chapelleet al. (2010) • Some qualities of multiple choice • Attempt to imitate productive

13. Research Questions • Can we achieve an acceptable level of reliability for the grammatical diagnostic test used for this study? • Do the items for the grammatical diagnostic test work well at an item level in terms of item discrimination and difficulty? Were there unexpected patterns? • What is the relationship between the subtest, full test, and self-assessment? • Were mastery and non-mastery patterns consistent with predictions based on the Processability Theory hierarchy?

14. Method Participants Instrument Analyses

15. Participants—Subjects • 219 middle school students • Outside Seoul • No overseas education

16. Participants—Raters • 2 rounds of rating • Round 1: Grammar • 6 Raters • All experienced in teaching; 4 in preparing tests • Scored the grammar tests and writing tests for the specific grammar points • Rated once (absolute answers) • Round 2: Holistic • 5 Raters • All experienced in scoring writing tests • Rated twice (3 times where raters differed by 2 or more)

17. Instruments • Grammar Test (see handout) • Writing test: picture task • Comparison purposes • PT grammar and additional levels

18. Analyses • Descriptive Statistics • Central tendency & dispersion measures • T-unit analysis • Test and subsection reliability (Alpha) • Item difficulty and discrimination • Correlation with the writing test • Fit to PT hierarchy

19. Results

20. Descriptive StatisticsGrammar Test & Writing Test

21. Reliability Statistics Grammar Test and Subsections & Writing Test

22. Index Item Numbers Item Difficulty and Discrimination Grammar Test

23. Correlation with the Writing Test Grammar Test and Subsections

24. Fit to Implicational Hierarchies • Coefficient of Scalability: • PT Only=94.1% • PT + Proposed Levels=89.3%

25. Discussion, Limitations, & Conclusion

26. Discussion • Overall reliability was quite good • Determiner and non-count section did not work • Exposed a problem with determiners generally • Task-types have good potential for diagnostic information • Grammar correlated fairly well with writing scores • Follows from complexity and accuracy • May also explain determiners & non-count nouns • Fit to PT of proposed levels suggests tasks are plausible

27. Limitations • Results are generalizable only to Koreans • Methods may be universal • Should have had a larger writing sample • Also, more feedback from students and teachers • More high-level students

28. Conclusions • Most of the grammar tasks can work well, but require more planning & research • Particular attention on error types • It may be possible to expand the PT hierarchy • Needed in order to be useful for diagnostic purposes