Thabiso Nyabanyaba Wits Education Policy Unit

1. The fallacy of measuring mathematics ‘competencies’ in a context of a ‘high stakes’ external examination Thabiso Nyabanyaba Wits Education Policy Unit

2. Starting point and basic concerns • Speaking from sociology • Concern with epistemological access • Contribution to standards and • Why certain children fail • Basing reform on a deficiency rationale • In an information age • Basic reading and arithmetic are no longer sufficient • Decision making requires access, interpretation, analysis and use of information (Bond, 1994) • Employers are no longer satisfied with school graduates’ skills (Reville, 2002) • Higher institutions argue that school graduate emerge with poor knowledge • Basing reforms on noble intentions • epistemological and motivational’ concern • Representation calls • Problem solving or modeling movements

3. Basic concerns and background • Result • Public debates about low/high standards • Evidence of a wider variety of assessment practices • Critique that such reform is political-economic rather than research based (Cooper & Dunne, 2000) • Driven by a concern for epistemological access • On the surface there is poor performance • Studied examination changes • Draws on Basotho students performance • Deeper underlying differences/positioning • Interviewed a sample of Basotho teachers on positioning to changes • Interrogated (based on scenario) a sample of Basotho students on positioning to changes • Undertaken in a post-colonial context • No feedback • Examination for selection • Legacy of international examination boards

4. Language of analysis • Why certain categories of pupils fail? • Bernstein allows for an analysis of how pedagogic texts are • Produced: Textbooks in the UK (Dowling, 1998) • Acquired: classroom practices (Morais and Neves, 1998) • Assessed valued knowledge (Cooper and Dunne, 2000) • Relevance to/for whom (Dowling) • Relevance to what end (Cooper and Dunne) • Esoteric domain is most strongly classified • Realistic domain illustrate weak classification • Recognition rules allows for recognition of specialty of context • Realisation rules allows for production of ‘legitimate’ text

5. Valuing of integration

6. Psychological assumptions • Educational measurement based on assumptions • Bloom’s taxonomy for higher-order thinking • Intrinsic individual cognitive differences • Increased objectivity achievable • Differences contained in behavioral competencies • Comprehension • Application • Analysis • Synthesis • Evaluation

7. Sociological framework • Analysis framed by assumption that educational measurement • Reveal individual differences • Predicated by social differences • Schools confer advantages • Limitation of school effectiveness based on student outcomes • Can perpetuate social inequalities • Bernstein’s framework key • Recognition rules • Allows for recognition of • The demands • Speciality of context • Realisation rules • Production of • Legitimate text • Valued knowledge

8. Language of description • Exercise • Problem • Differentiating • Function of item/question • Relationship with the learner • Esoteric • Algorithmic • Challenging • Realistic • Mundane • Novel

9. Basotho students • Recognition • …and this way it confused me when it says they what ... the total shaded area remains unaltered, and yet the tape moved to the other side. • As we’re reading this latter part, we have forgotten what was required at the beginning. • Figuring out • Mathematical demands • Context of tape • Diagramatical relationship • Realisation rules • The context of absences • Once assisted could fulfill the maths demands

10. Basotho teachers • Power relations between agents in the field of recontextualisation of official discourses (Lamnias & Tsatsaroni, 1997): • Official state agents • Curricular construction • Textbooks producers • Academic community • Examination agents • Produces contradictions • Teachers criticised as teacher-centred and textbook bound with the main focus being to drill students for the examination (Kokome, 1990; Moru, 1994; Polaki, 1996) • In some classes, when one was observing, because we’re involved in the reviewing and rewriting of the textbooks in line with the changes in assessment ... you can actually sit in the class knowing what the next question is going to be because the teacher is practically following everything in the textbook (Education officer, 1998) • Standards are based on a number of factors, including • Examination material and level of difficulty • ‘Prototype’ comparison of performance • Statistical standardisation of the performance • Professional judgment of senior examiners

11. Contradictions • I … have you ever been compelled to write remarks about the COSC paper? • BT No! • I Why not? • BT Umm.. one, it’s not local. Two, I thought I should start from home. That was why I wrote for this J. C. There was no reaction. In fact, they didn’t write back at least saying we have received your something, and therefore thank you, or we shall look into that, or some … at least, a word of encouragement, yah, never came. And I felt it was not worthwhile if they don’t recognise your efforts. They don’t appreciate what you’re doing, yah. So I didn’t write anything about COSC. Umm, COSC was quite OK sometime. Like I told you sometime ago, the questions that they’re developing of late, are such that they are a bit verbose, and too long. And umm … require the students to read and find out, yah, unlike the previous years, OK, every year, you see that there are some changes, you know. And it is geared towards…, especially in this geometrical aspect of questions, I have not… I didn’t write. I don’t know, I think, now that you’ve drawn my attention to that, if I see that, probably next time I think I … you know, I thought that … because I was actually told that, even if I wrote, nothing will come out of it …

12. Context of absences • Absences • Poor competencies in relation to demands: • No, we didn’t even try it because when you see that something is very difficult, you didn’t know how to start. • Inability to recognise speciality of context: • I What did you see as being difficult? • BS1 Just seeing the tape! • [A lot of laughter all round at the comment above and a few picked up their question papers to demonstrate the horror at seeing the context of a tape] • I That’s very interesting!What confuses you about the tape? • S2 You look at it, You say tape! This question is no good.

13. Basotho students’ habitus • Students who have received strong preparation: • Attend private tutorials • Purchase past examination papers • AS1 I never even bother with such questions because I think there are enough questions in Section B … at least, there must be some graphs questions which will appear. And I have tried to make sure that with graphs, I attempt everything. It’s because here in school we haven’t done this topic; even there [in the tutorials], we haven’t done this topic, those which refer to … Normally when I open a question paper, any question that is based on this topic I never … I just look at it and then proceed to graphs and perhaps vectors. • I OK? [Referring to the other student] • AS2 Yes, I never do this one. • I You don’t do it at all? • AS2 Even when I revise, … even on my own, I never do it at all. That’s because even when I attempt it, I fail. • Translates into strong cultural capital • Awareness of context of high stakes examination and • choices available

14. Discussions • External examination pervades noble intentions (where they are drivers) • Students’ habitus more evident than • School effectiveness ideals • Individual competencies measurements ideals • Leads to underestimation of competencies • Teachers’ peripheral position • Results in their being removed from reform • International agencies produce contradictions in practice in the context of • Weak recontextualisation agencies