Download Policy: Content on the Website is provided to you AS IS for your information and personal use only and may not be sold or licensed nor shared on other sites. SlideServe reserves the right to change this policy at anytime. While downloading, If for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.
1. Impact, Washback and Consequences of Large-scale Testing Liying Cheng (Ph.D)
2. Overview Define the research terms washback, impact and consequences
Discuss this phenomenon in relation to test validity and social consequences
Argue for conducting further empirical evidence beyond Alderson & Wall, 1993 and Cheng et al 2004
Illustrate a series of empirical studies using different methodologies
Focusing on the influence of testing on students
3. Impact, washback, and consequences There is a set of relationships, intended and unintended, positive and negative, between teaching, learning and testing (Alderson & Wall, 1993).
measurement-driven instruction (e.g. Popham, 1987),
test-curriculum alignment (Shepard, 1990), and
consequences (Cizek, 2001) (see Cheng & Curtis, 2004 for a detailed review)
4. Impact, washback, and consequences Test Impact - the effects of tests on macro-levels of education and society, and
washback - the effects of language tests on micro-levels of language teaching and learning, i.e. inside the classroom (Bachman & Palmer, 1996; McNamara, 2000; Wall, 1997).
A view of test influence falling between the narrow one of washback and the all-encompassing one of impact (Hamp-Lyons, 1997).
5. Validity (theoretical models) Washback - only one form of testing consequences that need to be weighted in evaluating validity (Messick, 1996, p.243) promoting the examination of the two threats to test validity,
construct under-representation and
construct-irrelevant variance, to decide the possible consequences that a test can have on teaching and learning.
Bachman (2005) proposes a framework with a set of principles and procedures for
linking test scores and score-based inferences to test use and the consequences of test use
6. Social consequences to the society (philosophical models) Critical language testing - political uses and abuses of language tests (Shohamy, 2001)
Fairness framework (Kunnan, 2004) - drew on research in ethics to link validity and consequences - tests as instruments of social policy and control.
An encompassing ethics framework to examine the consequences of testing on language learning at the classroom as well as the educational, social and political levels Hamp-Lyons (1997).
7. Model of Washback
8. Washback studies Two areas of washback studies have recently been conducted:
those relating to traditional or existing tests which are thought to stifle innovative teaching, and
those relating to cases where a test has been specifically changed in order to encourage innovation in the classroom.
Survey methods interviews and questionnaire
Cheng, L., & Watanabe, Y., with Curtis, A. (Eds.) (2004).
Washback in language testing: Research contexts
and methods. Mahwah, New Jersey:
Lawrence Erlbaum Associates, Inc.
9. Major work on washback
10. Alderson & Wall (1993) 15 hypotheses A test will influence teaching*.
A test will influence learning?.
A test will influence what teachers teach*.
A test will influence how teachers teach*.
A test will influence what learners learn*.
A test will influence how learners learn?.
A test will influence the rate and sequence of teaching?.
A test will influence the rate and sequence of learning?.
11. Alderson & Wall (1993) 15 hypotheses A test will influence the degree and depth of teaching?.
A test will influence the degree and depth of learning?.
A test will influence attitudes to the content, method, etc., of teaching and learning*?.
Tests that have important consequences will have washback*.
Tests that do not have important consequences will have no washback*?.
Tests will have washback on all learners and teachers?.
Tests will have washback effects for some learners and some teachers, but not for others?.
12. Why studying the impact on students? of the many millions of people who will sit down to take (English) tests , virtually none will have participated in the tests design, in writing test items, in critiquing the test methods, in setting cut scores or in writing or commenting on the performance descriptions that tie to their all-important score.
Of all stakeholders in testing events, test takers surely have the highest stake of all (Hamp-Lyons, 2000, p. 581).
13. The Impact Study of Ontario Secondary School Literacy Test
14. Impact of test/task types, skills and strategies on Students
15. Impact of test formats on students
16. Impact of L2 test takers characteristics and test performance
17. Focus group (Cognitive lab) The research questions:
How do L1 and L2 test takers accounts of the OSSLT differ?
Is it the same for both L1 and L2 groups or does the construct change in important ways in relation to language background?
Do these differences pose a threat to the inferences drawn from the test results?
What other differences are evident in these test takers accounts of the OSSLT?
Fox, J. & Cheng, L. (in press). Did we take the same test? Differing accounts of the Ontario Secondary School Literacy Test by first and second language test takers. Assessment in Education: Principles, Policy & Practice, 14(1).
19. Students attitude and their CET performance (Zhao & Cheng, 2006) What are the attitudes of students toward CET-4?
What relationships exist between students attitudes and their performance on the CET-4?
Does sex difference exist in attitudes and their relation to test performance?
What attitudes differentiate high achievers (who score above 80 percentile) from low achievers (below 20 percentile)? What is the relationship between the two?
20. Four Attitudinal Factors
Factor Item Mean SD
1 Test-taking Anxiety/ 2,3,4,8,10,19,20,23,24 3.87 .72
Lack of Concentration 26,27,29,32,34,35,36,38
2 Test-taking Motivation 5, 7, 9, 22, 37 2.79 .58
3 Belief in CET-4 6,13,17,21,25,39 2.43 .58
4 Test Ease 12,14,30 2.35 .75
21. Multiple Regression
22. Multiple Regression Contd Multiple Regression: females attitudes toward CET-4 and their test performance (N=145)
Model Factor t p R2
1 Test-taking Anxiety/Lack of Concentration -.28 -3.44 .001 070
2 Test-taking Anxiety/Lack of Concentration -.26 -.327 .001 .104
Test-taking Motivation -.20 2.55 .012
Multiple Regression: males attitudes toward CET-4 and their test performance (N=63)
Model Factor t p R2
1 Belief in CET-4 .46 4.02 <.001 .197
23. Multiple Regression Contd Multiple Regression: High achievers attitudes toward CET-4 and their test performance (N=42)
Factor t p R2
2 Test-taking Motivation .58 4.18 <.001 .338
Multiple Regression: Low achievers attitudes toward CET-4 and their test performance (N=42)
Factor t p R2
2 Test-taking Motivation .32 2.11 .041 .322
24. Strategy use (Song & Cheng, 2006)
25. Students strategy use and their CET performance
26. How to establish the relationship between testing and its impact? Work backward from the test items (test design)
Explore test-takers characteristics over testing
learners academic background
L1 (native language), Culture, Ethnicity
Learning styles and personality (Field in/dependence)
Linking the effects to student test-performance using higher level analysis
27. Qualities of language tests Bachman and Palmers test usefulness framework (1996)
Reliability + Construct Validity + Authenticity + Interactiveness + Impact + Practicality
Kunnans test fairness framework (2004)
Validity + Absence of Bias + Access + Administration + social consequences
28. Future directions Washback/impact researchers need to fully analyze the test under study and understand its test use.
the extensive research on validity and validation has tended to ignore test use, on the one hand, while discussions of test use and consequences have tended to ignore validity, on the other. It is, then, essential for us to establish the link between test validity and test consequences (Bachman, 2005, p.7).
Therefore, it is imperative that washback/impact researchers work together with other language testing researchers as well as educational policy makers and test agencies to address the issue of validity, in particular, fairness and ethics of our tests.
29. Reflections It is clear that testing is never a neutral process and always has consequences (Stobart, 2003, p. 140). Tests are a differentiating ritual for students: for every one who advances there will be some who stay behind (Wall, 2000, p. 500). This is particular true to the large scale language tests.
Assessment (testing) is central to the teaching and learning process.
30. references Alderson, J.C. and Wall, D. (1993). Does washback exist? Applied Linguistics 14, 115-129.
Bachman, L. F. (2005). Building and supporting a case for test use. Language Assessment Quarterly, 2(1), 1-34.
Bachman, L. F. and Palmer, A.S. (1996). Language Testing in Practice, Oxford University Press, Oxford, England.
Bailey, K. M. (1996). Working for washback: A review of the washback concept in language testing, Language Testing 13, 257-279.
Cheng, L. (2005). Changing language teaching through language testing: A washback study. Studies in Language Testing: Volume 21, Cambridge University Press, Cambridge.
Cheng, L. and Curtis, A. (2004). Washback or backwash: A review of the impact of testing on teaching and learning, in L. Cheng and Y. Watanabe with A. Curtis. (eds.), Washback in Language Testing: Research Contexts and Methods, Lawrence Erlbaum Associates, Mahwah, New Jersey.
Cheng, L., & Watanabe, Y., with Curtis, A. (Eds.) (2004). Washback in language testing: Research contexts and methods. Mahwah, New Jersey: Lawrence Erlbaum Associates, Inc.
31. references Cheng, L., Klinger, D., & Zheng, Y. (2007). The challenges of the Ontario Secondary School Literacy Test for second language students. Language Testing, 24(2).
Cizek, G. J. (2001). More unintended consequences of high-stakes testing. Educational Measurement: Issues and Practrice, 23(3),1-17.
Hamp-Lyons, L. (1997). Washback, impact and validity: ethical concerns. Language Testing, 14(3), 295-303.
Hawkey, R. (2006). Impact Theory and Practice: Studies of the IELTS test and Progetto Lingue 2000. Cambridge University Press, Cambridge.
Klinger, D., Cheng, L., & Zheng, Y. (under review). Factors influencing ESL/ELD students performance on the Ontario Secondary School Literacy Test. Educational Assessment.
Kunnan, A. J. (2004). Test fairness. In M. Milanovic, C. Weir, & S. Bolton (Eds.). Europe language testing in a global context: Selected papers from the ALTE conference in Barcelona. Cambridge: Cambridge University Press.
Messick, S. (1996). Validity and washback in language testing, Language Testing 13, 243-256.
Popham, W. J. (1987). The merits of measurement-driven instruction. Phi Delta Kappa, 68, 679-682.
32. references Qi, L. (2005). Stakeholders conflicting aims undermine the washback function of a high-stakes Test, Language Testing 22, 142-173.
Shepard, L. A. (1990). Inflated test score gains: Is the problem old norms or teaching the test? Educational Measurement: Issues and Practice 9, 15-22. Shohamy, E. (2001). The Power of Tests: A Critical Perspective on the Uses of Language Tests, Longman, Essex, England.
Song, X., & Cheng, L. (2006). Language learner strategy use and test performance of Chinese learners of English. Language Assessment Quarterly: An International Journal, 3(3), 241-266.
Wall, D. (1997). Impact and washback in language testing. In Clapham, C. & Corson, D. (Eds.). Encyclopedia of Language and Education (p. 291-302).
Zhao, J. & Cheng, L. (2006, May). Exploring the relationship between students attitudes toward testing and their test performance. Paper presented at the Canadian Society for Study of Education, Toronto, Ontario.
Zheng, Y., Cheng, L. & Klinger, D. (under review). Do test formats in reading comprehension affect ESL/ELD and non-ESL/ELD students test performance differently? TESL Canada.