Testing of English- and French-Speaking Students in Canada / Évaluation des élèves anglophones et des élèves francophone

1. Testing of English- and French-Speaking Students in Canada /Évaluation des élèves anglophones et des élèves francophones au Canada Kadriye Ercikan, UBC, Marielle Simon, uOttawa, Maria Elena Oliveri, UBC & Normand Dufour, MÉLS Canadian Society for the Study of Education Canadian Education Researchers Association May-June 2010, Concordia University, Montreal

2. Objective Describe the context, challenges, actual procedures, other solutions and implications for achieving comparability of English and French student test results on large-scale assessments (LSAs) in Canada

3. Context • Outside of Quebec, Francophone students form between 1 to 34% of the population • In Quebec, Anglophone students make up 19% of the population • Anglophone and Francophone minority groups learn and are tested in their respective language. • Francophone students are provided education in French in most jurisdictions • Only New Brunswick officially adopted the two national languages

4. Context French Immersion programs are made widely available in all the provinces Comparative comments often made: “As was the case in PISA 200o and PISA 2003, students enrolled in the French-language school systems in Nova Scotia, New-Brunswick, Ontario and Manitoba performed significantly lower in reading than did students in the English-language systems in the same provinces” (Government of Canada, 2008 web site). Test scores are used for obtaining diploma, scholarship decisions, university entrance selection, etc.

5. Context 7 out of 10 jurisdictions surveyed in 2007 said comparability was intended 1 jurisdiction’s tests are developed and administered independently Holistic scoring by teachers in 1 jurisdiction needs validation committees to ensure comparability The two language versions of a test may be developed based on different target curricula, content specifications or teaching resources

6. Procedures

7. Challenges Ideal conditions for scores to be comparable: Equivalence of tests construct definitions items Equivalence of scores through parallel tests Comparability of scores through linking procedures simultaneous calibration anchor item linking

8. Challenges • Concern with comparability of scores is relatively recent; • There is great variation in provincial practices; • There is preference for development and validation committees over statistical procedures; • The small minority language sample sizes limit: • the types of psychometric procedures to create accurate score scales for the minority students; • linking score scales; • demonstration of equivalence.

9. Potential Solutions Adaptation solutions: Use parallel, successive (Rogers et al, 2003), concurrent (Solano-Flores et al, 2002) or simultaneous (Tanzer, 2005) test development strategies Expert reviews Content (Bolwes et al, 2008) Linguistic (for equivalence of meaning, cognitive requirements, difficulty of vocabulary and expressions and cue given to answer items)

10. Potential Solutions Psychometric evidence based on Classical test theory Unidimensionality analyses (Oliveri et al, in press) Identification of DIF and its sources Student cognitive processes using think aloud processes (TAPs) (Ercikan et al, 2010)

11. Final Note • We reviewed and summarized research, practices and challenges and recommended methodologies for examining comparability of test items and tests given in multiple languages. • We want to highlight the importance of optimizing comparability in the development phase of assessments. • We also want to encourage further research to investigate the comparability of cognitive processes of students from different language groups as integral components of validity investigations.

12. References • Bowles, M. S. & Stanfield, C. W. (2008). A Practical Guide to Standards-Based Assessment in the Native Language. Retrieved on March 28, 2008 from .ncela.gwu.edu/spotlight/LEP/2008/bowles_stansfield.pdf • Ercikan, K., Arim, R.,G., Law, D. M., Lacroix, S., Gagnon, F., & Domene, J. F. (2010). Application of think-aloud protocols in examining sources of differential item functioning. Educational Measurement: Issues and Practice, 29, 24-35. • Rogers, W. T., Gierl, M .J., Tardif, C., Lin, J., & Rinaldi, C. M. (2003). Differential validy and utility of successive and simultaneous approaches to the development of equivalent achievement tests in French and English. Alberta Journal of Educational Research, 49(3), 290-304. • Solano-Flores, G,, Trumbull, E., & Nelson-Barber, S. (2002). Concurrent Development of Dual Language Assessments: An Alternative to Translating Tests for Linguistic Minorities International Journal of Testing, 2, 107 – 129. • Tanzer, N.K. (2005). Developing tests for use in multiple languages and cultures: A plea for simultaneous development. In: R.K. Hambleton, P.F. Merenda & C.D. Spielberger (Eds.), Adapting educational and psychological tests for cross-cultural assessment. Mahwah, NJ: Erlbaum.