The impact of SBA: reflections on its implementation

1. The impact of SBA: reflections on its implementation Stephen Andrews University of Hong Kong Monday 29th June, 2009

2. This paper presents findings from a larger project that is supported by the Research Grants Council, Hong Kong (HKU7483/06H). The RGC’s support is gratefully acknowledged.

3. What do we mean by impact? • Bachman and Palmer (1996:29-30): tests have impact on society and educational systems (the macro-level) and upon the individuals within those systems affected by the test use (the micro-level). • Some researchers (e.g., Wall, 1997) distinguish between impact - the effect on the macro-levels of education and society - and washback (or backwash) - the effect on the micro-levels of teaching and learning. • In our study, the term impact is used in the broad sense suggested by Bachman and Palmer.

4. The aim of our impact study • Our focus is innovation effectiveness, specifically where an innovation is ‘owned’, and where it gets disseminated or blocked • Our aim is to enhance our understanding of the complex processes of innovation, with specific reference to the teaching-testing tension, by analyzing the processes of implementation from policy level through to the level of the individual school community, and by examining the perspectives of key stakeholders and the influence of mediators.

5. The specific foci of our impact study • The HK education authorities’ rationale and strategies regarding the introduction of SBA English Language • Other stakeholders’ perspectives on and interpretations of the innovation and the authorities’ implementation strategies • The processes by which SBA English Language is being implemented in Hong Kong secondary schools • The impact of the SBA innovation on teachers, students, and parents • The factors seeming to facilitate or inhibit the successful implementation of the SBA ESL innovation

6. The data of our impact study • Documents and interviews with key administrators, other stakeholders and ‘mediators’ • Classroom observation, interview, and questionnaire data from case studies conducted over a two-year period in three schools

7. The perspective of our impact study • Change implementation NOT a rational, problem-solving, technical endeavour • ‘Implementation (of change) … is taken to be a matter of struggle and conflict with material, vested and self-interests at stake. Policy is not just something that is done to people – it is essentially contested both in its formation and its implementation’ (Ball & Bowe, 1991:24) • i.e. a ‘micropolitical’ perspective

8. What the presentation is NOT going to do • The aim is NOT to present a full report of the project • NOR to present an evaluation of the success or otherwise of SBA, i.e. whether the impact has generally been ‘positive’ or ‘negative’ • Attempting to do either would be premature

9. What the presentation IS intending to do • To reflect on aspects of three interrelated issues relating to change implementation, with specific reference to SBA, drawing on data from the SBA impact project: • ‘Micropolitics’ and the implementation of SBA • ‘Ownership’ of innovations and SBA • The impact of ‘mediators’

10. The basic argument • Design/implementation of innovation is NOT a simple technical issue • ‘Micropolitics’ affects how an innovation is experienced by different stakeholders • Stakeholders take ownership or resist ownership in varying ways, affecting how they and others experience the innovation • ‘Mediation’ at intervening levels between policy and classroom affects how students, the ultimate stakeholders, experience the innovation

11. The ‘micropolitics’ of language education Alderson (2009:223): ‘Much of the literature is about rational solutions to language issues, informed by relevant theory. Authors argue about different approaches and theories to explain or cast light on issues, but on the whole the impression is given that if only we followed X method, used Y techniques, accepted Z approach, or applied a particular theory, we would make progress’

12. The ‘micropolitics’ of language education Alderson cont.(2009:223): ‘In contrast, myexperience convinced me that what seems to make the biggest difference to successes and failures in language education is not the theory or the policy, but the people who implement the policies, the teachers and trainers who have to interpret and operationalise the theory, and the organisations and institutions within which those individuals work’

13. The ‘micropolitics’ of language education Alderson (2009:3): ‘… Experience shows that in most institutions, development is a complex matter where individual and institutional motives interact and are interwoven … The literature, when it deals with development matters at all, gives the impression that language education is basically a technical matter … But behind that facade is a complex interplay of personalities, of institutional agendas and of intrigue’

14. ‘Ownership’ and SBA • Who owns an innovation like SBA at the policy/design level? BUT ALSO • Who owns SBA as a ‘product’? • The HKEAA? • The consultants/designers of the product? • Consumers of that product? • Those who mediate consumption of that product?

15. Who ‘owns’ SBA at the policy level? Education policy development, operations and implementation in HK (after Kennedy, 2005)

16. The complexity of ‘ownership’ Ownership may be shared between different agencies Ownership of design/implementation features may be contested Each individual with direct involvement in the design and implementation of the innovation has opportunity of some level of ‘mental ownership’ (Breiting, 2008) Others with peripheral involvement may appropriate the innovation to serve their own ends Students as passive ‘recipients’ or active ‘owners’?

17. The ownership of SBA The EMB’s ownership of the timeline The HKEAA’s ownership of responsibility for timely, reliable and valid assessment that satisfies public expectations The Secretary for Education’s ownership of responsibility for anything that goes wrong The HKEAA consultants’ ownership of the conceptualisation of the format (task types, delivery mechanisms, QA oversight)

18. Contested ownership at the policy level Contested ownership of the timeline EMB originally pressed for 2006 Strong resistance by HKEAA + other members of the ‘One Committee’ Eventual agreement on 2007 S4 teachers began implementation in Sept 2005 Early 2006, the Secretary for Education bowed to political pressure re teachers’ workloads  switch to a phased-in implementation of SBA

19. Contested ownership at the school level Contested ownership of the timeline Example: School 1 • Four teachers at S4 level • Teacher A • NET with experience with SBA in own country • involved in HK SBA development from the beginning • enthusiastic leader of SBA in School 1 from start in 2005 • when EMB introduced the phased-in implementation, Teacher A was on sick-leave for a short while • Teacher A’s S4 colleagues took advantage of her sick-leave to switch to a delayed implementation

20. Collaboration, conflict … School 1: The story moves on two years ….. Teacher A: 2007-2009 Involved again in SBA. Still very enthusiastic, fully integrating SBA within the curriculum Teacher B: collaborates closely with Teacher A to develop SBA- related ideas and materials BUT ‘… it’s all very well for, you know, SBA literature … for them to say, teachers should collaborate. But it doesn’t work if it doesn’t work, and you can’t force people to collaborate’ (Teacher A)

21. … and peaceful co-existence School 1: The story continued….. • Teachers C and D: ‘the other camp’ • refuse all invitations to collaborate • accept materials from Teacher A, but appear to make no use of them • share nothing of their own SBA-related classroom practices • Collaboration between the ‘two camps’ • occurs only when it cannot be avoided, i.e. in relation to SBA assessment events involving all the SBA teachers

22. The localised nature of conflict School 1: The bigger picture ….. The productive, positive collaboration among the other SBA cohort G: …for our form, okay we have a very good… H:Friendly partners. G:For the other…form, the story is not the same, it’s totally different, yeah. H:Maybe they rather prefer working independently. It’s just kind of working style. And we would rather depend on each other, and share the workload. G:It saves time, you know [laughing]. H:We are inefficient, therefore, we have to share…and they are too efficient, therefore they can handle the workload on their own… I’m not sure.

23. The role of ‘mediators’ • ‘Mediation’ and ‘ownership’ of an innovation • The potential for mediation of the SBA ‘message’ between design/dissemination and the way SBA is experienced by students • E.g. mediation via • Teacher educators • Textbooks • Tutorial schools • Teachers

24. The importance of textbook ‘mediation’ The potential importance of textbooks: • Cheng (2005:251-252): ‘The washback effect of the new 1996 HKCEE may be, to a large extent, determined by textbook publishers’ interpretation of the examination. Publishers want to sell their books. Teachers want something to rely on in their teaching’ • Cheng (ibid:122): ‘Whether the publishers are able to reflect fully the theory or belief underpinning the change is itself a complicated matter’

25. The ‘mediating’ role of textbooks • At best, textbooks reinforce a positive message • At worst, they misinterpret and/or misinform • Priorities are always commercial (i.e. what is going to sell? NOT what is educationally desirable?) • Exam-oriented textbooks tend to sell themselves as maximising student and teacher success rates • This may lead them to prioritise ‘test-wiseness’, practice of formats, stock phrases, memorisation, rather than learning

26. Textbook mediation and SBA? • For the written HKCEE papers, textbooks appear to play a significant role in shaping teachers’ and students’ knowledge of/preparation for the exam (Cheng, 2005) • Do SBA-related textbooks play a similarly influential role? If not, why not? • Timing? Textbooks were not ready early enough, so teachers had to cope, relying on own resources? • Teacher empowerment? Teachers feel comfortable/ confident with the idea of designing own materials?

27. Tutorial schools and ‘mediation’ Mr R • Involved in tutorial schools since mid-1990s • The business opportunities presented by the HKCEE changes in 1996: ‘Students were panic, so they rushed to tutorial centres and so I analysed the syllabus for them. I tell them what they had to do in order to score high marks step by step’

28. Tutorial school ‘mediation’ of public exams Mr R’s mediation of the HKCEE (post-1996) Oral: ‘For example, for the discussion part, I give them useful expressions. They like those because they want something to hold on to when they go to an examination. So I give them lists … like these … there are 80 ‘must-use’ expressions. So we give them stuff like that. They are our secret weapons. They give some gimmicks’

29. The ‘mediation’ style of tutorial schools Exam preparation, NOT teaching English Mr R: ‘Our slogan is ‘Help you excel in exams. Help you succeed in exams’. We actually tell them we are not teaching English, OK. After taking our courses, you may not be … better at English, but you will get a better score in the exam. So we try to tell them learning English is one thing, but being able to pass or get a distinction in the exam is another, so that they have to come, ha ha, to get good grades’

30. Tutorial school mediation of SBA Mr R: SBA as another business opportunity: • ‘So when the news came that the exam syllabus will change again in 2007, as tutorial schools we are really happy … because we see students are going to panic again’ • ‘I think we are giving them short-cuts and we are doing things that are gimmicks to make money … because actually we make money out of students’ worries, because … it’s SBA but it’s also an examination, so they are afraid of it … so that’s pretty much my role … I give them confidence and some extra help’

31. Tutorial schools and mediation of SBA? Do tutorial schools play as influential a role in mediating SBA as Mr R implies? Has it been the business opportunity he hoped for? If not, why not? • Does the ‘reduced pressure’ of SBA remove the student panic on which the tutorial school business feeds? • Do students feel confident/prepared for SBA anyway? Or do they simply not take SBA as seriously as the ‘exam’? • Does SBA not lend itself to the ‘exam tips’ approach? • Or are students’ own teachers already providing that (including ‘useful phrases’, strategies for avoiding reading the book etc)?

32. The ‘mediating’ role of teachers • The major mediating force impacting on each student’s experience of SBA is likely to be his/her school English teacher • That mediation may be significantly affected by chance factors, e.g. • The random allocation of students/teachers to classes • The life events that affect teacher continuity • The experiences in School 2, where both case-study classes had a change of teacher after S4

33. The experiences in School 2 School 2 • A co-educational CMI school • Mostly Band 2/Band 3 intake • Students mainly from lower/lower-middle income families • The two case study classes were the weakest in the year • After S4, both teachers left the school  each class experienced SBA mediated by two teachers

34. Class A – Teacher W W:‘I always threaten them that this is a public exam and their marks will go to HKCEE. So, everyday I bombard them with all these words and it seems that they have exam pressure and are willing to move a little bit more’ W:‘We just encourage them to read the book and try to give them guidelines, help them to understand the story more, and then in the lessons, we just teach them how to discuss, how to present, so these are actually oral skills’ S1:‘Maybe she was more strict with us, so we were motivated by her’ S2:‘And she would give us a lot of comments and ideas for us to think about. But we didn’t get the same kind of input from the teacher this year’

35. Class A – Teacher X X: ‘I think they were ok. Not too bad. I mean some of them are more shy. I think that because we have a bit more relaxed environment, so they are more willing to participate a bit more. I joke with them. I am a bit friendlier with them. We often laugh and talk about things, so I think that makes things a bit easier for them’ S1: ‘X likes to joke a lot. He’s too relaxed…Some people may feel he’s only fooling around’ S2: ‘He would try very hard to teach us if we asked him individually. For instance, if you asked him about the word “brave”, he would teach you a lot of synonyms and so on. But he wouldn’t do that during normal lessons …He’d joke with us because he likes to learn Cantonese from us’

36. Class B – Teacher Y Y: ‘Frankly speaking I don’t really want to teach English but I just teach English because of the pay’ Y: ‘What I have done was I actually taught them how to write their notes for Spider Man… and I actually show them like…how I copy and paste, how I change things and how to do research and then…to make a speech … I told them about Wikipedia and the website called Sparknotes … and a lot of them copy and paste from that website and change it to their own words or adding their own opinion to it’

37. Class B – Teacher Y’s students I:‘What did you gain most from Y’s class? What kind of skills did you learn from Y that you think are most beneficial to you in preparation for SBA?’ S1: ‘Not much actually. He basically asked us to write a script and then he corrected it’ I:‘Did he teach you how to prepare the script?’ S2: ‘He took us to the MMLC, gave us a web address and then we got on the Net together and wrote down something useful from the website’ I:‘So, he basically didn’t spend much time to help you with the SBA preparation’ S2:‘Well, he showed us the movie and also showed us some sample student clips’

38. Class B – Teacher Z Z: ‘I’m actually very pleased about their performance because most of them are, well, try to speak fluently, actually, they practice a lot before their assessment and I saw some of the students try not to, well, read from the notes, they try to react to the classmates, well, naturally, so well, I really see their effort so I am quite impressed’ S1:‘I think I can learn a lot of discussion skills in Miss Z’s class’ S2: ‘I can learn how to conduct a group discussion’

39. Variations in teachers’ SBA mediation • Teachers have different attitudes, styles and practices in relation to SBA • They have different degrees of willingness/readiness to take ownership • They all have to engage with SBA in that it’s part of the curriculum they must deal with • But how far do they ‘buy into’ it with genuine commitment in ways the designers hoped/intended? • All these variations in teacher mediation affect the students’ experience of the SBA innovation

40. Conclusions 1 • Aim was to introduce the ‘micropolitical’ perspective on innovation effectiveness adopted as the basis for our study of the impact of SBA • ‘Politics can be seen as methods, tactics, intrigue, manoeuvring, within institutions which are themselves not political, but commercial, financial and educational’ (Alderson, 2009:2-3) • ‘ …. Politics with a small p includes not only institutional politics, but also personal politics: the motivation of the actors themselves and their agendas’(Alderson, 2009:3)

41. Conclusions 2 Evidence from the study highlights the need to consider: • The influence of the ‘micropolitics’ at every level/stage of innovation implementation • The role and influence of ‘mediators’ • The ways in which stakeholders at every level (including mediators) may take ownership of an innovation and shape their own experience of it (and the experiences of others) to suit their particular agenda.

42. To end on a positive note……. ‘When I first started with the SBA, I thought it was a real headache, you know, and but this year it’s completely different. I enjoy this year and the feeling’s a bit different, I enjoy teaching SBA a little bit … and if you really know the skills to teach the SBA I think it’s kind of fun because it’s very different from teaching grammar and the comprehension’ ‘When you go over the SBA, the story with them, the characters and the climax, it seems that the students, but I cannot say all the students, most of the students enjoy the SBA lessons, provided that they have read the book …’ ‘It’s very different from the first year that I start with the SBA and, I am fine okay this year, I feel more confident, more comfortable teaching the SBA’ [Teacher G, School 1]

43. References Alderson, J.C. (ed.)(2009). The Politics of Language Education: Individuals and Institutions. Bristol: Multilingual Matters. Bachman, L. & Palmer, A. (1996). Language testing in practice. Oxford: Oxford University Press. Ball, S.J. & Bowe, R. (1991) Micropolitics of radical change: Budgets, management and control in British schools. In J.Blase (Ed.) The Politics of Life in Schools. London: Sage. Breiting, S. (2008). Mental ownership and participation for innovation in environmental education and education for sustainable development. In A. Reid. et al (Eds.) Participation and Learning, New York: Springer. Kennedy, K. (2005). Changing Schools for Changing Times. Hong Kong: Chinese University of Hong Kong Press. Wall, D. (2000). The impact of high-stakes testing on teaching and learning : can this be predicted or controlled? System, 28, 499-509.