For, With, By: Researchers and Teachers CREOLE, City of Bristol College & CfBT Education Trust

1. For, With, By: Researchers and TeachersCREOLE, City of Bristol College & CfBT Education Trust Contextually Situated Research as a Driver Informing Policy, Practice & Theory Professor Pauline Rea-Dickins

2. Overview of Presentation • Introduce the research: SPINE • Stakeholder involvement & partnerships • Research drivers, including the stakes & consequences: real world problems • Some empirical findings from triangulated studies • Some conclusions

3. What is SPINE? Student Performance in National Examinations: the dynamics of language in school achievement (SPINE) www.bristol.ac.uk/spine (ESRC/DfID RES-167-25-0263) Bristol team: Pauline Rea-Dickins, Guoxing Yu, Oksana Afitska, Rosamund Sutherland, Federica Olivero, Sibel Erduran, Neil Ingram, Harvey Goldstein Zanzibar team: Zuleikha Khamis, Abdulla Mohammed, Amour Khamis, Mohammed Abeid, Shumbana Said, Haji Mwevura

5. Research Planning, Advisory Group Meetings • Partnership • Planning for the research • Ministry of Education & Vocational Training SMT Members – Advisory Group • National Launch • c. twice yearly updates of key findings • Discussion of research communication strategies/dissemination • Channels for interaction, n.b. Znz Research Team

6. Why is MoEVT Interested? Supportive? • Previous work by Bristol • Institutional links and respect (e.g. doctoral training) • Levels of achievement at end of Basic Education alarmingly low • Interest in SPINE: Evidence of policy changes • English as MoI to be brought in earlier at primary (std4) but commitment to raising children bilingually • SUZA: universities do research

7. International Research Initiatives & Policy Decisions • Examinations in English: high stakes ‘real world’ problem • Accommodations for ELL in US (e.g. Work of Abedi; Bailey & Butler; Rivera et al) • Teaching & learning of Science and Maths through English: reversal of policy in Malaysia (see also Lan, 2010) • Sri Lanka: at O and A level – students are using 2 languages to study & not compulsory to sit the exam in EL2: they can choose (Punch, 2009)

8. Policy Issues: The Stakes & the Consequences in SSA • Sub-Sahara Africa: what is the impact of an unfamiliar language on learning progression & education outcomes? • Research focus in sub-Saharan Africa more on classroom interactions & subject learning with less attention for the formal examining of this subject knowledge • Examining: Formal examinations at end of 2nd year of secondary in English • > 50% of school aged children leave school at the end of Basic Education as unsuccessful learners: what does the evidence say?

9. Theory-focused Issues: Applied Linguistics • L2 academic language development across school subjects • Working bilingually in the classroom • Monolingual (L1) examining • Negotiating double constructs in examinations • NS-orientation in examining; prescriptive vs supportive assessment modalities

10. SPINE: theoretical issues - assessment How valid: • is the dominant view of assessment as quality measurement model vs. a supportive & context sensitive model (e.g. Pollitt & Ahmed 2009) + dynamic assessment research (Poehner 2008; Poehner & Rea-Dickins 2010) • is the role of a dominant world language in LTA processes • are current constructs, e.g. that assume NS norms, and monolingual performance as the normal context for language use

11. SPINE: EXEMPLAR FINDINGS (STUDY 5.1) What are the four things that man could resemble whales? 45 students took this item: • 35.6% = no answer • 26.7% = wrong answer • 28.9% = partially correct answer • 8.8% = correct answer

12. Int: D1 – No response to Q3 explains (STUDY 5.1) D1: “because I did not understand by this … this … resemble” (lines 115-117) Int: “If I tell you that resemble means ‘to look like’ … can you do the question now? D1: “Yes” Int: OK so what’s the answer? D1: “Man … is warm blooded … and whales also … whales have lungs and man also have lungs …” (122-133) 12

13. Original Items: Summary of Findings (STUDY 5.1) • Very low mean scores across subjects • Student achievement lowest in Maths • Significant differences in achievement across the subjects • Strong correlation between English (exam + vocabulary test) & performance in other subjects • Factors affecting performance from learner interviews: • Not understanding the task, specific words & phrases or the meaning of tables and diagrams • Partial knowledge of topic area & question type • Low levels of students’ language proficiency

14. Modified Items - some findings (STUDY 5.1) • Increase in student response rates • Increase in accuracy of responses • Correspondence between ability to provide a correct translation of task & ability to solve/provide partially correct answer • Learners who had difficulty translating task generally performed poorly • Evidence of linguistic factor + other factors 14

15. COURSEWORK ASSESSMENT : implementation issues (STUDY 4) Significant variability among teachers: • the types of specific activities that constitute teacher assessment • the number of activities from which they take marks for each month • decisions on how to handle student absenteeism • how to arrive at the CWA mark: some picked the one that students did better on, some picked randomly, and some averaged of all marks

16. Schools A-F – limited exposure to English Tests of significance: comparison of students performance across the 3 different versions and per subject (STUDY 5.3)

18. Qualitative Analysis of Student Responses(STUDY 5.3) Top performing students Qn 5: Write about an animal you have studied • Name of the animal: • What does the animal look like? • Where does the animal live? • Describe how the animal eats?

19. Qualitative Analysis of Student Responses(STUDY 5.3) Top performing students Reaching potential: analysis of higher performing students (see handout)

20. Findings from National Form II Data: summary of the multilevel modelling analysis(STUDY 2) • It is very clear that ENGLISH language proficiency is a significant and substantial predictor of the students’ performance in MATH, BIO & CHEM. • But: the school-level variances explained in the cons models as well as in the models including ENGLISH as the single explanatory variable demonstrated that a substantial proportion of the variance is attributable to school factors

21. CONSEQUENCES: High stakes classroom assessment - the realities & impact 21

22. Enhancing Learning & Social Justice

23. Research Evidence as Driver for Change? For Awareness Raising? (1) For policy: • Potential through partnership: SUZA, MoEVT • Triangulated evidence, progressively focused studies • All learners affected, not only lower achieving learners For professional practice • Stakeholder engagement with findings e.g. through workshops For community • Through media, e.g. Kiswahili press

24. Research Evidence as Driver for Change? For Awareness Raising? (2) For Language Testing and Assessment Community/International Examining Boards • What evidence is there of the impact of research findings such these on actual testing and assessment policies and practices? (e.g. Shohamy) • Time to overthrow the dominant & unquestioned role that EL1 has in many examining contexts: time for a coup! • Time to reconceptualise constructs: NS orientations • Investigate supportive & context sensitive approaches to assessment

25. Research Evidence as Driver for Change? For Awareness Raising? (3) • Where is the most impact likely? • For policy? • For professional practice? • For community? • For LTA/Applied Linguistics community • For International Examining boards? • Where do partnership strengths lie?