Quality Assurance in the School Based Assessment Component of a Public Examination

1. Quality Assurance in the School Based Assessment Component of a Public Examination by Stafford A. Griffith (Professor) Institute of Education University of the West Indies, Mona Campus

2. Introduction and Overview SBA in public examinations is an issue of continuing concern. Despite this, the secondary and post-secondary examination systems of many countries are expanding the use of SBA. Queensland in Australia, for example, abolished external examinations more than three decades ago and replaced them with a system of SBA (Maxwell 2004, Third ACEAB Conference). The UK Working Group on 14-19 Reform, suggested that for three of the four levels of certification proposed, “teacher-led assessment should be the predominant mode of assessment” while for the fourth “assessment should remain a balance between external examinations and in-course assessment” (Working Group on 14-19 Reform, 2004).

3. Nevertheless, the quality control issues that are often associated with the use of SBA cannot be ignored. A number of boards, including the Caribbean Examinations Council (CXC), have put in place certain quality assurance measures to address these issues. This paper examines the benefits of SBA in a public examination, with specific reference to CXC, and the quality assurance measures taken to respond to some of the critical concerns of stakeholders. It gives special attention to limitations that the teacher variable imposes on quality assurance and suggests ways of addressing these.

4. Benefits of SBA SBA is central to CXC examinations, both CSEC and CAPE. The SBA component does not mirror the external assessment measures. It seeks to explore the unique opportunity provided by this component to assess competencies that cannot be adequately or easily assessed using MC or essay tests. While SBA in CXC examinations may take a variety of forms, most tasks are in the form of projects. Projects require students to produce something and often include preparation of reports or the making of a presentation.

5. The advantages of SBA are, perhaps, well crystallised in its capacity to facilitate and promote authentic assessment. Authentic assessment focuses on the application of knowledge to real-life situations. It requires students to apply what they know and to demonstrate the competencies needed for success in the real world. The SBA component for many CXC subjects requires authentic assessment tasks. CSEC History syllabus provides an option that requires students to undertake a History research project where they behave like a historian • identify a suitable area of research, • collect historical data from a number of sources, and • prepare a report using writing conventions of historians.

6. SBA also facilitates the individualizing of a part of the curriculum. The teacher assesses the achievement of each student on tasks that are specially selected or defined to respond to the interest and the environment of the student. The extent to which the opportunity is provided, varies among subjects and public examination boards. CXC, in most of its subjects, provides for SBA to contribute 20 percent of the overall marks for the examination, thus providing the opportunity to individualise approximately 20 percent of the curriculum. Exceptions to this proportion are made for subjects that are more practical or performance oriented, where the percentage for SBA may account for 30 to 60 percent, depending on the nature of the subject.

7. The Issue of Fairness The limitations in SBA are generally related to issues of fairness that may affect the validity and reliability of the scores awarded. Fairness involves a lack of bias, equitable treatment in the testing process, and equal scores for students who have equal standing on the tested constructs (AERA, APA, NCME, 1999). Fairness is a central aspect of public examinations and any inherent unfairness in any aspect would be incongruous with the concept. While public examinations have gained acceptance, generally, as valid, reliable and fair measures of student achievement, there is still much concern about the fairness or unfairness of the SBA component.

8. Yip and Cheung (2005) in a HKEAA study found that teachers were concerned about fairness. They were of the view that SBA “could not provide a valid and reliable assessment of student work because of the great variation among different schools in the conditions under which assessment is made” The variations cited by teachers included: • the degree of guidance or assistance given to students, • the types of practical work being assessed and • the marking standards of teachers. CXC, like several other public examination boards, has instituted a number of quality assurance measures to: • reduce these potential sources of invalidity and unreliability and • counter possible unfairness. However, much more needs to be done to treat with these issues.

9. Quality Assurance Measures Quality assurance in SBA may be conceptualized as all the quality control mechanisms put in place by an examining board that provide confidence that the grades awarded for the SBA are in keeping with the required standards. Quality assurance measures are important to treat with issues of fairness that can affect the reliability and validity of the marks or grades awarded to students for the SBA component of a public examination. Like many other public examination boards, CXC uses moderation as the primary tool of quality control. Moderation may be described as “a process of aligning the standards of different examinations or different components of an examination, conducted in different places at different times and marked by different examiners” (Broomes, 1997).

10. The moderation procedures used by CXC involves procedures prior to, during, and after the assessment of students. Prior to initiating the SBA tasks, teachers and students are provided with guidance about what is expected. This guidance is contained in the subject syllabus and represents an initial effort to achieve a common understanding of what is required. Further guidance is provided through mark schemes or scoring rubrics for the award of marks for SBA tasks. The CXC syllabus guidance and scoring rubrics are developed by Panels of subject specialists.

11. The use of pre-defined mark schemes or scoring rubrics, provides a more objective approach, leading to increased scorer-reliability (Moskal, 2002). In addition to guidance in the subject syllabus and the scoring rubrics, CXC undertakes SBA workshops that seek to standardize or calibrate teachers who use the rubrics. Workshops are conducted, periodically, using the expertise of both contracted subject matter specialists and CXC staff.

12. Moderation measures are also instituted during SBA activities. Teachers are expected to provide guidance to students in keeping with guidelines provided in syllabuses and at workshops. Further guidance is provided through a SBA Manual for Principals. However, the teacher is permitted considerable independence of action in • working with students to undertake the required tasks and • assigning scores for the completed tasks. The degree of independence has caused concerns about • the validity of the assignments undertaken by some students and • the reliability of the scores awarded by some teachers. This is an area that requires greater attention for improved quality control.

13. CXC may benefit from the more detailed guidance given by boards such as the IEB for moderation of SBA by teachers both within and between schools. The moderation guidance addresses • measures for standardization, • fairness and equity, • validity of tasks for the subject, • accuracy and • standards of marking. The IEB provides extensive guidance on how various issues should be addressed, including the frequency, timing, venue and dates for meetings within schools and between schools. The adoption of similar measure by CXC will contribute, significantly, to the improvement of procedures for quality assurance.

14. The Council should also consider the recommendations made by the UK Working Group on 14-19 Reform for: “robust internal quality assurance systems including a potentially pivotal role for Chartered Assessors in institutions, responsible for co-ordinating assessment across the institution, and ensuring that standards are applied consistently and codes of fair practice are adhered to”. Both the recommendation of the UK Working Group and the practice of the IEB emphasize cross validation among teachers and schools during the conduct of SBA as an important measure for quality assurance. The adoption of such procedures will assist CXC with a more comprehensive response to the quality assurance challenges in SBA.

15. Apart from adopting quality control measures, through moderation, prior to, and during SBA, CXC undertakes a review of the scores or grades, and where necessary, makes adjustments. Broomes (1997) gives two important reasons for this review. • First, despite the fact that teachers may be scrupulously honest in making assessments of the work of their students, “the ‘halo’ phenomenon may cause a teacherto assess a studenttoo generously”. • Secondly, “as in any profession, there are a few teachers whose standards are not of the highest [quality]” . The review of scores or grades awarded by teachers assists in detecting any unfairness or malpractice that may have taken place.

16. CXC undertakes the review of teachers’ scores by using either of two procedures. • For some subjects, particularly those with a practical component, the moderator visits the school, participates in the assessment of some or all students and makes adjustments, where necessary, to the scores awarded by teachers. • For other subjects, assignments of five students, selected as requested by CXC, are remarked by CXC moderators. The scores of the moderators are compared with those of the class teacher and a relationship between the two sets of scores statistically established and used to generate moderated scores for all students in the class. This form of moderation, or variants of it, is widely used by several other public examination boards.

17. Quality Control and the Teacher Much of the concerns about fairness in SBA relate to the role of the teacher in the assessment process. The support for teachers’ involvement in the assessment of their students is widespread, particularly because of the value in improving instruction and, ultimately, student learning. However, differential degrees of guidance or assistance given to students in SBA is a source of continuing uneasiness among teachers, parents and other stakeholders. Much of the uneasiness is caused by misunderstandings about the dual role of the teacher as instructor and assessor.

18. The misinformed teacher or stakeholder may incorrectly regard teacher input into SBA as unfair. But teacher input in guiding students on a continuous basis is necessary and legitimate. This input is part of formative assessment that should be distinguished from summative assessment, which is also involved in SBA. In summative assessment, the aim is to report how far the student has reached in the learning journey.

19. This distinction between formative assessment and summative assessment is often unclear, leading to: • different practices among different teachers and schools and • different perceptions among different stakeholders about what is fair and acceptable. Differential inputs from teachers that may unfairly reward or penalize students are often the result of differential understanding of their role. At one end of the continuum are teachers who may treat the assignment purely as an external examination while at the other are teachers who end up doing much of the assignment for the student.

20. Moderation of the marks awarded by teachers is not an appropriate quality assurance measure for this type of improper practice among teachers. Moderation at that stage is concerned with the work submitted/completed. Once the marks awarded by the teacher for the assignment are in keeping with the CXC standards • no adjustment would be required and • the Council is not likely, without cause, to investigate the classroom practice that led to the quality of an assignment in the sample/assignments moderated. Under the circumstances, the Council depends on the teacher to follow fair and equitable practice.

21. Measures such as those that have been instituted by the IEB and recommended by the UK Working Group on 14-19 Reform, for quality control during the process of SBA, are helpful in alleviating the differential practices within and between schools. However, more effective training of teachers is required to ensure that they understand and can perform the dual role expected of them. The training of teachers to understand and perform the dual role as instructor and assessor in SBA should not be left to the workshops conducted by the examination board. It should form an integral part of the pre-service training provided by teacher training institutions.

22. Although stakeholder perception about the legitimacy of teachers’ guidance and assistance to students in SBA is not, in itself, a quality assurance measure, it is an important consideration for: • treating with any incorrect perception about the role of the teacher and • obtaining support for the quality control measures in relation to teacher involvement. As a complement to teacher training, therefore, more effective public education about the dual role of the teacher is necessary. In particular, such public education should seek to develop, among all stakeholders, a proper understanding of the importance of formative assessment in SBA.

23. Summary and Conclusions SBA is likely to remain a part of public examinations in the foreseeable future. It has significant educational benefits in developing and assessing important educational skills and in individualizing a part of the subject curriculum. However, concerns of fairness need to be addressed. Examination boards such as CXC have instituted a number of quality assurance measures to treat with this issue. In treating with quality assurance issues in SBA, the variable of teacher input requires the greatest attention.

24. The quality control measures undertaken by CXC may be augmented by adopting measures such as those • practiced by the IEB and • recommended by the UK Working Group on 14-19 Reform • that emphasize cross validation among teachers and schools during the conduct of SBA. However, teachers need to be trained more extensively to understand and to practice, more effectively, their dual role as instructor and assessor in SBA. Such training should not be left merely to workshops but should be an integral part of the teacher preparation undertaken by teacher training institutions.

25. THANK YOU