The Findings of the Longitudinal Study of the First Two Years of SBA English and a Progress Report on its Continuation

1. The Findings of the Longitudinal Study of the First Two Years of SBA English and a Progress Report on its Continuation SBA Consultancy Team, Faculty of Education, The University of Hong Kong

2. The study • Mixed methods research design: Questionnaires, individual and focus group interviews, documentary analysis (video records of sample assessments, teacher notes and score sheets related to design and evaluation of assessment tasks), group forums, web-based assessment scoring) • Three rounds of main data collection: Feb-Mar 2005, Sept 2006, Feb-Mar 2007, plus follow up round, Sept 2007.

3. The schools • The study contracted to study 10 schools, but began with 14. Attrition occurred between Rounds 1 and 2 due to school workload (2) and introduction of SBA options (3). By Rounds 2 and 3 there were 9 participating schools

4. Quantitative data Written questionnaires from teachers (English) and students (Chinese) Teachers – Round 1 Round 2 Round 3 61 38 40 Students -- Round 1 Round 2 Round 3 2202 1584 1430 NB The student group was stable over course of study, but the teacher group changed in size and in composition.

5. Qualitative data • Written comments from teachers and students • Interviews with administrators and a small number of parents • Focus groups of teachers • Focus groups of students

6. Teachers’ views about the factors which influenced the school’s choice of options

7. Overview: Teacher data

8. Overview: Student data

9. Teachers’ views of their own preparation for SBA

10. Teachers’ views of the implementation of SBA

11. Teachers’ views of the effects of SBA on themselves

12. Teachers’ views of the effects of SBA on themselves

13. Teachers’ views of the potential problems of SBA

14. Summary of teachers’ views as shown on questionnaires • Teachers continue to feel pressure and experience increased workload, though this has decreased • Teachers are in general more positive towards SBA • Some teachers may have reached a plateau and begun to seek deeper understanding and new skills; many are still learning • Teachers need more system-level (non HKEAA) support to link assessment with learning

15. Workload: Teacher comments • Round 1: Many comments were negative and simplistic, eg., “too much workload”,“extra workload“; longer comments were highly critical • Round 2: Comments suggest more in-depth understanding of what SBA entails; indicators of positive change are seen, eg., “We're still just adding SBA on top of the curriculum instead of incorporating it into the curriculum.”

16. Workload: Teacher comments • Round 3: Fewer comments on workload and more are positive; greater sense that SBA may be worthwhile, eg., “Students can be motivated to read and watch movies in English more successfully. I find life much easier that way.” • But some teachers have not yet worked out how to moderate their workload through staged responses, groupwork management in the classroom, and the use of self and peer assessment

17. Fairness: Teacher comments • Round 1: Only four teachers referred to concerns about fairness; of these, one was a concern for parents’ opinions, and one was positive (that SBA would be fairer). • Round 2: There were only two references to fairness, both of which were positive. • Round 3: There were 3 comments, 2 were neutral, plus “Students think it is sometimes unfair to students of Band 3 because most Band 1 students find SBA very easy to do.”

18. Students’ views of their own preparation for SBA

19. Students’ views of the effects of SBA on themselves (1)

20. Students’ views of the effects of SBA on themselves (2)

21. Students’ views of potential problems with SBA

22. Summary of student views as shown on questionnaires • Students’ comments overall started neutral and stayed neutral • Main concern was with their workload • Secondary concern was with time to prepare and time to prepare in other parts of the English curriculum • A further concern was with whether SBA could be fair, i.e., whether their teacher or the system could be fair

23. Fairness: Student comments • Round 1: 42 comments, 27 about possible cheating • Round 2: 34 comments, eg.’ “The SBA may put the teachers in a dilemma as they want to be as fair as possible, yet, on the other hand, they tend to give students unreasonably higher grade.” • Round 3: 9 student comments relating to fairness (10 of these related to teachers as assessors, 3 related to unfairness of moderation, and the remainder were general claims that SBA was unfair without specific reasons given).

24. Teachers’ views about the problems of SBA over 4 rounds

25. Teachers’ views of student results Teachers’ views of the effect of SBA on higher proficiency students have remained almost stable (4.19 compared to 4.15 in the 3rd round), but very encouragingly they have become noticeably less concerned that SBA will disadvantage lower proficiency students (3.31 in the Round 3, down to 3.09 in the final round, having begun at 4.12 in the Round 1).

26. Effects on learning, teaching and assessment: Teachers’ comments • R1: Few comments about the effects of SBA on student learning; concerns mainly about procedures especially intrusive nature of (video) recording of assessment • R2: More comments relating to student learning, and more of these were positive; increasingly positive attitudes to (video) recording and to assessment changes • R3: More comments about SBA having a positive effect in increasing students’ exposure to and confidence in using English; increasing understanding of the potential of video records to be used for self and peer assessments, and teaching/program evaluation NB: Small-scale comparison of inter-rater reliability among teachers at beginning and end of study shows marked increase in reliability across all assessment domains, equivalent or better than international norms; results encouraging but cannot be generalized.

27. Patterns in SBA episodes 2006-7 INDIVIDUAL PRESENTATIONS • Formalised with large audiences and often more than one T present • Note cards overtly being used • Communicative strategies appear to be given low priority • Strict time limits in almost all schools • Formalised “invitation” for questions from audience—one question + reply suffices

28. Patterns in SBA episodes 2006-7 GROUP INTERACTIONS • Buzzers in several schools • Multiple “judges” in some cases • Prevalent overt use of a single note card; some evidence of detailed preparation, ie., too much prior notice of task • Formalised ‘turn-taking’ where turns seem fixed in advance; even transitions that necessitate prior knowledge BUT • Flashes of real communication, laughter, insider idea-sharing • Much more awareness of the formal characteristics of the genre of book report, narrative recount, simple turn-taking strategic language

29. Areas for future research and action (1) • Reading: Research, PD and pre-service • Classroom management for learning and assessing in groups: PD and pre-service; research on class size; flexible timetabling and facititative planning for teamwork • Task design for performance assessments: Action-research and PD • Grouping: Research, then PD into implications of differing grouping configurations

30. Areas for future research and action (2) • Feedback and interactive assessment: PD on feedback and interactive assessment that is “dialogic and sensitive to learners’ needs” ; reinforcement in all documents that “the teacher-assessor may, if they wish, interact individually with a student at any time to ask specific questions to clarify and/or extend the student’s ideas, to prompt and/or scaffold the student’s oral interaction and to probe the range and depth of their oral language skills”. 2008 HKCEE English Language Handbook for the School-Based Assessment Component (HKEAA 2006)

31. Areas for future research and action (3) • Research into the prevalence and effects of the tutorial/coaching schools • Reach out to these schools and educate staff in SBA • Investigate student reference to ‘cheating’ websites easily available in Hong Kong • Liaise with textbook publishers to ensure material accurately reflects SBA processes

32. Priorities • Priority 1: More specific PD and support material on the teaching of speaking The over-representation of a few specific group interaction activities and tasks, and the formality of the individual presentations, may be partly due to the predictable dearth of good SBA-appropriate practice in teaching practice in the schools at the time the oral SBA was being introduced. However, teachers continue to say that they do not know how to teach speaking. This study and several SBA-focused research studies and the Quality Education Fund project on school-based assessment of speaking in Forms 1-3 show that a strong programme of professional development on the teaching of speaking is badly needed. This lies outside the role of HKEAA and calls for intervention and high-level professional support from EDB/CDC.

33. Priorities • Priority 2: More varied PD on assessment Although it is to be expected that considerable new development work will be done in preparation for the implementation of SBA in the HKDSE, continuing PD opportunities for teachers are needed every year. Teachers have asked for ‘refresher’ courses or more advanced, hands-on courses. Short courses specifically for school SBA coordinators have been requested. • Priority 3: Updated support material At this stage the original exemplars and anchor samples do not (should not) represent the desired performance types or a sufficient range of tasks geared to the needs of specific student types. New samples are urgently needed. New samples are urgently needed to enhance validity and reliability

34. The continuation study • The continuation study aims to examine the continue to monitor and evaluate the impact of the introduction of the SBA In English Language on teacher workloads, and on students’ learning and attitudes to the use of English in a range of representative schools in Hong Kong from Dec 1, 2007 to Nov 30, 2009. • This study is needed as almost half the schools in Hong Kong chose Option 3 for the implementation of SBA in their school in 2007. It is important to monitor and evaluate their implementation of SBA and to see if there are systematic differences in the attitudes, practices and beliefs of early and late adopters of SBA. • The study is being undertaken by the same team of researchers from the Faculty of Education, The University of Hong Kong, in collaboration with teachers in the selected schools.

35. The schools • The continuation study, using a very similar methodological approach to the original study, contracted to study 18 schools • 11 schools have been recruited so far; particular difficulties with recruiting Option 3 schools

36. The data collection • Questionnaires/focus group interviews: S4/S5 teachers Mar, 08, Sept 08, Mar 09 & Sept 09 • Questionnaires/focus group interviews: S4/S5 students Mar, 08, Sept 08 & Mar 09 • Sharing sessions/mini-conferences June, 08, June 09