Assessment as a washback tool: is it beneficial or harmful?

1. Assessment as a washback tool: is it beneficial or harmful? Nick Saville Director, Research and Validation University of Cambridge ESOL Examinations October 2008

2. Outline • Introduction • Washback and impact: some definitions • Impact and the law of unintended consequences • Researching impact: the Cambridge ESOL approach • Conclusion

3. 1. Introduction • Examinations • access to opportunity • fairness • Exert a powerful influence on educational processes • possibly negative? • concern for social consequences, ethicality, accountability

4. Questions to be addressed: • How can educators and assessment providers foster beneficial effects from their tests and examinations? • How can harmful consequences be avoided or their impact mitigated?

5. “Impact by design” • Integral part of a framework for developing and validating examination systems • A concept akin to social impact assessment

6. 2. Washback and impact: definitions • Washback (or backwash) has been broadly defined in the assessment literature as the effect of testing on teaching and learning • One aspect of the broader phenomenon known as impact – as we shall see later

7. Washback • Alderson and Wall, 1993: • 15 washback hypotheses • Based on whoor whatmight be affected: • Teaching • Learning • Content • Rate of learning • Sequence of teaching/learning • Degree/depth of curriculum coverage • Attitudes of teachers/learners • Etc.

8. Washback • A continuum - stretching from harmful at one end, through neutral to beneficial at the other end Negative Neutral Positive - +

9. Washback • Negative? • Restriction of content – narrowing of curriculum • Too much time practising for the test • Positive? • Transparent objectives and outcomes • Increased motivation of learners • Increased accountability of teachers (?) • BUT – cause and effect explanations are rarely adequate …..

10. Washback Models • In the language testing literature: • Bailey (1996) • Watanabe (2004) • Cheng (2004, 2005) • Green (2007)

11. Bailey’s Model (1996) - based on Hughes,1993 • 3 Ps: • Participants • students • teachers • Processes • Products • learning • teaching • materials • curricula

12. Watanabe’s Model (2004) • Five dimensions: • Specificity • Intensity • Length • Intentionality • Value • Factors influencing theprocess of washbackare related to: • the test itself • status • stakeholders

13. Green’s Model See: Studies in Language Testing, 25, 2007: IELTS Washback in Context Washback will be most intense – have the most powerful effects on teaching and learning behaviours – where participants see the test as challenging and the results as important (high stakes) SEE BLUE ARROW

14. Impact Impact concerns the effects and consequences a test can have beyond the classroom and immediate learning context: • On individual career or life chances • In educational systems and in society

15. Impact • Impact deals with wider influences and includes the “macro contexts” - tests and examinations in society • Washback is an aspect of impact related to the “micro contexts” of the classroom and the school

16. Impact • Investigating impact is integral to validation • an essential component in establishing the usefulness of an assessment system - fitness for specific purposes and contexts of use • Consistent with Messick views of validity (1989, 1996) • Consequential aspects of validity • fairness and ethics

17. Impact • Educational systems – complex phenomena • Stakeholder constituencies • many different stakeholders in educational processes • complex network of relationships

18. Stakeholders in the testing constituency Learners Teachers Test writers/examiners Receiving institutions School owners Future employers Government agencies Professional bodies Test centre administrators Materials writers Publishers etc Learners Parents/carers Teachers Receiving institutions Employers School owners Examiners Government agencies Professional bodies Academic researchers Test writers/ Examiners etc Test constructs Test format Test conditions Test assessment criteria Test scores Testing System Inputs to test design Contexts of test use - consequences

19. Stakeholders in the testing constituency Learners Teachers Test writers/examiners Receiving institutions School owners Future employers Government agencies Professional bodies Test centre administrators Materials writers Publishers etc Learners Parents/carers Teachers Receiving institutions Employers School owners Examiners Government agencies Professional bodies Academic researchers Test writers/ Examiners etc Test constructs Test format Test conditions Test assessment criteria Test scores Testing System Inputs to test design Contexts of test use - consequences

20. Stakeholders in the testing constituency Learners Teachers Test writers/examiners Receiving institutions School owners Future employers Government agencies Professional bodies Test centre administrators Materials writers Publishers etc Learners Parents/carers Teachers Receiving institutions Employers School owners Examiners Government agencies Professional bodies Academic researchers Test writers/ Examiners etc Test constructs Test format Test conditions Test assessment criteria Test scores Testing System Inputs to test design Contexts of test use - consequences

21. Impact • Multiple voices • views, beliefs and attitudes • Important for examination providers • to collect feedback from the stakeholder • to take stakeholder perspectives into account • Dynamic relationship between micro and macro contexts

22. 3. The “law” of unintended consequences • “Any purposeful action will produce some unintended consequences” or side-effects • “Goodhart’s Law” (or “Campbell’s Law” in the USA) • a variant of the “law” of unintended consequences

23. “Goodhart’s Law” • “All performance indicators lose their meaning when adopted as policy targets” • Examples: • England - school achievement targets - school league tables • USA – No Child Left Behind (NCLB) • The clearer you are about what you want, the more likely you are to get it – but the less likely it is to mean what you wanted it to! (Dylan Wiliam, 2008)

24. Perverse incentives? • Assessment policy can create a tension between • educational objectives at the micro level(teaching and learning in schools) and • a requirement for accountability at the macro level

25. What role can examination providers play? How can examination providers ensure that : • examination systems are “fit for purpose”? • research is carried out to “find out what is going on” in contexts of use? • mitigating action is are carried out if/when negative effects and consequences occur?

26. 4. Researching impact • Cambridge ESOL • an international examinations board • Maxims of Test Impact • An attempt to integrate an action-oriented approach to investigating impact into working practices • Milanovic and Saville, 1996

27. Maxims for achieving/monitoring impact • Maxim 1 PLAN • Use a rational and explicit approach to test development • Maxim 2 SUPPORT • Support stakeholders in the testing process • Maxim 3 COMMUNICATE • Provide comprehensive, useful and transparent information • Maxim 4 MONITOR and EVALUATE • Collect all relevant data and analyse as required. Milanovic and Saville, 1996 - Considering the impact of the Cambridge examinations

28. Maxim 1 Plan Use a rational and explicit approach to test development • a process model – cyclical and iterative • creates the necessary conditions for appropriate tests to be developed and for on-going validation to take place • begins with the purpose - including anticipating • how the test should (or might) be used • how relevant and useful it is likely to be - social consequences and value implications • potential (unplanned) side-effects

29. Maxim 2 Support Support stakeholders in the testing process • Involvement of stakeholders - during test design and development • consultation on specifications/syllabus design • participation in pilot tests • Professional support programmes • for institutions and individual teachers/students etc. • Training of suitable personnel to work on all aspects of the examination cycle • item writers, examiners, etc.

30. Stakeholders in the testing constituency Learners Teachers Test writers/examiners Receiving institutions School owners Future employers Government agencies Professional bodies Test centre administrators Materials writers Publishers etc Learners Parents/carers Teachers Receiving institutions Employers School owners Examiners Government agencies Professional bodies Academic researchers Test writers/ Examiners etc Test constructs Test format Test conditions Test assessment criteria Test scores Testing System Inputs to test design Contexts of test use - consequences

31. Maxim 3 Communicate Provide comprehensive, useful and transparent information • Explain issues related to assessment adapted for the different stakeholders • the nature of the language constructs being assessed • the meaning of language test results • etc. • A major challenge for all test providers!

32. Maxim 4 Monitor and Evaluate Collect all relevant data and analyse as required • For example, feedback from the candidates and their teachers • from local contexts where the test is used (i.e. at the micro level) • A long-term endeavour as it involves: • the development of suitable instruments for the collection of adequate data • appropriate research methodologies e.g. mixed method designs, case studies etc. • Evaluate the test’s usefulness routinely • determine the need for changes and periodic revisions

33. Outcomes of impact studies • The World-wide survey of the impact of IELTS • The Italian Progetto Lingue 2000 (PL2000 project) • See Hawkey, (2006) • Studies in Language Testing, 24 The theory and practice of impact studies: Messages from studies of the IELTS test and Progetto Lingue 2000

34. Progetto Lingue 2000 • The policy was • “.... to introduce innovation into the teaching and learning of other languages by putting greater emphasis on the development of communicative competence in all grades of the school system” • It included: • the adoption of the Council of Europe’s Common European Frameworkof Reference (CEFR) as the basis for learning objectives and standards • certification of proficiency – the testing • (by a certificating body recognised internationally)

35. Students Parents Teachers Teacher-trainers Curriculum developers Testers Publishers Receiving institutions Employers Students Parents Teachers Teacher-trainers Curriculum developers Testers Publishers Receiving institutions Employers Italy - PL2000 Impact Project Materials Learning goals, curriculum, syllabus Methodology Teacher Support Testing

36. 5. Conclusion • Impact research now reflects the growing importance of evidence-based approaches to educational planning and evaluation • By adopting an “impact by design” approach - through careful test development and validation strategies, and by using the findings from impact research to guide future actions - more effective assessment policies and practices can be developed to meet the needs of contemporary education

37. Thank you