Using the modified Angoff method to set fluency benchmarks in Rwanda

1. PHOTO BY NSHIMIYIMANA ALEXIS / SOMA UMENYE Using the modified Angoff method to set fluency benchmarks in Rwanda Norma Evans Evans and Associates Sharon Haba Soma Umenye Rwanda SOMA UMENYE ACTIVITY

2. Why set new standards and benchmarks? Previous P1 – P3 Kinyarwanda Benchmarks • Processes used, challenges encountered P1-P3 ORF benchmarks established using modified Angoff • Processes used, challenges encountered Achievements and next steps Overview

3. 2013 P3 ORF benchmarks - median method

4. 2014 ORF and RC Benchmarks, P1, P2 and P3 Sores in red = minimally meeting grade level expectations at each grade level

5. Difficulty replicating median method Median method only applicable to EGRA Documented challenges with median method (Jukes et al., 2018) • Insufficient % of pupils reading with comprehension to produce consistent results Other challenges USAID SOMA UMENYE

6. Establishing cut scores “There are no exact or correct cut scores for a test, but only more or less defensible ones. Defensibility is based largely on the method used to set the limit scores.’ Ferrara, Perie & Johnson, 2008

7. Most defensible method for establishing cut scores? • Most widely used Sireci & Biskin, 1992 & researched Cisek, 2012 • Only method that can be used for both EGRA and LARS-type assessments International best practices Angoff method Bookman method Nedelsky method Edel method Have panels of experts (curriculum specialists, master teachers) look at the questions asked or the tasks assigned and come to consensus as to whether pupils in each category (not meeting expectations, partially meeting, meeting, exceeding) would be able to answer them correctly. Zieky & Perie, 2006; Livingston & Zieky, 1984; Zieky & Perie, 2008 USAID SOMA UMENYE

8. The modified Angoff method Yes – No method

9. Evaluation specialists – 1 day Curriculum specialists – 2 days Alignment with curriculum For every evaluation instrument Evaluation & curriculum specialists (master teachers) 3 days % pupils in each category (short, medium, long term) Decision makers USAID SOMA UMENYE

10. Part A: Development of generic assessment framework (policy-level descriptors) USAID SOMA UMENYE

11. Part B: Adaptation of generic framework to reading in early primary USAID SOMA UMENYE

12. Development of grade-specific performance descriptors USAID SOMA UMENYE

13. Alignment of “meets expectations” for P2, P3 with SDG 4.1.1a USAID SOMA UMENYE

14. Using Modified Angoff to set ORF and RC cut scores for different performance categories Discussion leader P3 P1 P2 30 P1 to P3 master teachers from 5 regions (6 per region) 2 – 3 Kinyarwanda specialists per grade level USAID SOMA UMENYE

15. Teacher selection process Profile of teachers • Perceived by local or district education authorities as Master early primary teachers • Minimum 5 years of experience teaching Kinyarwanda in Early primary • Currently teaching Kinyarwanda in P1, P2 or P3 • Active participation in early grade reading trainings • Former or current early grade reading trainer or facilitator • Public school teachers USAID SOMA UMENYE

16. Example, 2 of 5 provinces Two Grade 1, 2 and 3 teachers per province USAID SOMA UMENYE

17. Process – Preparation for Modified Angoff Step 1: Validating performance descriptors for grade level Step 2: Learning how to collect oral reading fluency data Step 3: Collecting ORF and RC data in neighboring school (1/2 day) USAID SOMA UMENYE

18. Detour USAID SOMA UMENYE

19. Step 4: Understanding how the Angoff method works Borderline pupil – Pupil at the lowest point of a performance category • Teachers, curriculum specialists read through the performance description for fluency for “meets expectations ”. • They identified a pupil who would be a “borderline pupil” for this category USAID SOMA UMENYE

20. Step 5: For the category “meets expectation”, read down the sentences in the text and circle the last word their borderline pupil would attempt to read in 1 minute USAID SOMA UMENYE

21. The Angoff method – 2 rounds of scoring Round 1 Round 2

22. Step 7: Re-scoring and sharing Round 2 assessments. P3 Standard deviation 2.2 Step 8 : Calculating mean score for Round 1 - 41 Number of cwpm Standard deviation 5.9 Step 6: Sharing and discussing individual assessments (Round 1) USAID SOMA UMENYE

23. USAID SOMA UMENYE

25. Draft ORF Standards (CWPM) , Modified Angoff Scores in red are the mean, round 2 scores for each performance category USAID SOMA UMENYE

26. Regional Comparison, ORF benchmarks, other Bantu languages USAID SOMA UMENYE

27. Step 9 – Proposed adjusted scores (+/- 2 SE Mean) USAID SOMA UMENYE

28. USAID SOMA UMENYE

29. What we achieved… • Rwanda’s expectations now aligned with regional expectations – and with international expectations (SDG 4.1.1a) • Because the decision-making process was rooted in local expert knowledge, level of confidence that benchmarks are accurate • Ministry ownership of new benchmarks and standards • Change agents (master teachers) in each district who own these benchmarks, can explain and defend them, as well as teach others how to monitor progress towards them. • Learning assessment specialists from all subject areas able to replicate the modified Angoff method to set benchmarks and standards for other subject areas (systemic capacity) USAID SOMA UMENYE

30. Next steps Repeat the process, to generate grade or term-specific cut scores and benchmarks for other foundational skills Implement an accountability framework (3 pilot districts) where teachers measure, each term, pupils’ progress with respect to foundational skills and implement remediation activities for pupils in two lowest categories Develop an electronic dashboard so that school-based performance with respect to benchmarks is entered, aggregated at local, district and national level and used to direct resources to low-performing schools/areas. USAID SOMA UMENYE

31. Sharon Haba Director, MEL Soma Umenye/Chemonics shaba@soma-umenye.org Norma Evans Techncial Director Evans and Associates n.evans.associates@gmail.com

32. Recommendations for those of you working with countries thinking about setting stands Choose your teachers carefully. Involve local district staff, trainers (those who know teachers in the area) in the selection Trust your master teachers. They know the range of children’s reading performance at given grade levels and can describe it. Teach them to collect fluency data, and have them collect fluency data on a wide range of pupils before setting standards, so that they are able to attach quantitative descriptors to their qualitative judgments. Program this for the end of the school year. Have your assessments specialists from all disciplines involved in the process, so they can lead their own benchmarking sessions for their respective disciplines. USAID SOMA UMENYE

33. Comparison, P2 Benchmarks “Meets expectations’ - Median (2014, 2016) and Angoff method (2019) 2019 ANGOFF method 38cwpm 35 cwpmv 25 cwpm 35 cwpm 45 cwpm Median method, score 25th and 75th percentile 2016 25 cwpm 38 cwpm 2014 50 35 25 30 40 45 USAID SOMA UMENYE

34. USAID SOMA UMENYE

35. Modified Angoff method – 3 part process 2 days 3 days USAID SOMA UMENYE