Umalusi Moderation & Release of the 2010 Standardisation Decisions National Consultative Forum Meeting 07 May 2011

1. Umalusi Moderation & Release of the 2010 Standardisation DecisionsNational Consultative Forum Meeting07 May 2011

2. WHAT CAN WE LEARN FROM THE NSC RESULTS? Context, principles, approaches and processes

3. Reasons for Disclosure • Unprecedented interest in the role of Umalusi • Request: Access to Information • Uninformed comments • Credibility of the NSC • Credibility of Umalusi

4. Legal Mandate • Chapter 2A of GENFETQA Act 58 of 2001 (as amended in 2008)

5. Section17 of the GENFETQA Act • (1) The Council must assure the quality of assessment at exit points • (2)(a) The Council must develop policy for the accreditation of assessment bodies other than departments of education and must submit it to the Minister for approval • (b) The Minister must make regulations in which the policy for accreditation is set out • (c) The Council must accredit an assessment body in accordance with the regulations contemplated in paragraph (b) • (3) The Council must perform the external moderation of assessment of all assessment bodies and education institutions • (4) The Council may adjust raw marks during the standardisation process.

6. Section17 of the GENFETQA Act • (5) The Council must, with the concurrence of the Director-General and • after consultation with the relevant assessment body or education • institution, approve the publication of the results of learners if the • Council is satisfied that the assessment body or education institution • has— • (i) conducted the assessment free from any irregularity that may • jeopardise the integrity of the assessment or its outcomes; • (ii) complied with the requirements prescribed by the Council for • conducting assessments; • (iii) applied the standards prescribed by the Council which a learner is • required to comply with in order to obtain a certificate; and • (iv) complied with every other condition determined by the Council. • (6) The Council must issue certificates to learners who have achieved qualifications or part qualifications

7. Framework for QA of Learner Achievement • Based on established and existing practices in assessment for certification • Prescribed components of External assessment (examinations) and Site based/ internal / continuous assessment • Use of systems, processes, and procedures to evaluate, inspect, monitor and report on examination systems, processes and procedures of public and private assessment bodies.

8. Framework for Quality Assurance of Assessment • Evaluation and /or accreditation of assessment bodies • Periodic inspection of assessment systems • Ongoing monitoring of assessment systems • Quality assurance of external examinations through: • Moderation of examination question papers • Monitoring and moderation of SBA • Monitoring the “State of readiness • Monitoring the conduct of examinations • Moderation of marking • Standardization of assessment outcomes

9. Approval of Results • Compliance with minimum requirements • Shift from Moderation to Verification (Moderation for verification) • Emphasis on Assessment Body reports • The monitoring of the conduct of examinations and the moderation of marking as well as the monitoring of the implementation of internal assessment and the moderation of internal assessment is undertaken by Umalusi to verify the veracity of the assessment body reports in respect of these process. • Broadened definition of irregularities (including assessment system processes)

10. Why Umalusi standardises results, and how • Provision of GENFETQA – Council may adjust raw marks. • International practice – large scale assessment systems • Standardisation – process used to mitigate the effect of factors other than learners knowledge and aptitude on the learners performance. • Sources of variability – difficulty in question paper, undetected errors, learner interpretation of questions

11. Objectives for Standardisation • To ensure that a cohort of learners is not advantaged or disadvantaged by extraneous factors other than their knowledge of the subject, abilities and their aptitude. • To achieve comparability and consistency from one year to the next.

12. Why Umalusi standardises results, and how • Assumptions – for large populations the distribution of aptitude and intelligence does not change appreciably • Process of standardisation • Moderation of question papers • Review of learner performance against historical performance of candidates in each subject. • Historical average (norm) constructed using past 3 to 5 years data. • Pairs analyses provides further comparisons of raw means • Statistical moderation of Internal assessment

13. Why Umalusi standardises results, and how • Qualitative input meetings • Reports (moderator, Chief marker and Internal Moderator) • Umalusi research (maintaining standards &post exam analysis) • Responsibility of Assessment Standards Committee • Committee of Council • Responsible for setting and maintaining standards • Observers (SAQA, HESA, Teacher Unions)

14. Process for Standardisation • Continuation of JMB and SAFCERT model • Assessment Standards Committee • Qualitative Reports • Pre-standardisation and Standardisation meetings. • Standardisation booklets (data) – subject raw mark distributions (external written component only) of entire cohort. • Subjects are standardised individually, in a linear and non- iterative manner

15. General principles applied in the standardisation of examination marks • 1. In general no adjustment should exceed 10% or the historical average. • 2. Adjustments in excess of 10% could be considered at the upper end to increase the number of distinctions in a subject. • 3. In the case of the individual candidate, the adjustment effected should not exceed 50%of the raw mark obtained by the candidate. • 4. If the distribution of the raw marks is below the historical average, the marks may be adjusted upwards subject to the limitations.

16. General principles applied in the standardisation of examination marks (cont.) • 5. If the distribution of the raw marks is above the historical average, the marks may be adjusted downwards subject to the limitations. • 6. The computer adjusted mark is calculated based on these principles. • 7. For those subjects with a practical component of 50%, raw marks could be accepted. • 8. Umalusi however retains the right to amend these principles where and when deemed to be necessary based on sound educational principles.

17. Main instruments of standardisation • 1.Historical average (HA) of raw scores over 3-5 years • 2. Distribution of raw and adjusted marks in 10% intervals • 3. Graph plot showing cumulative percentage of candidates at each mark • 4. Raw mark distribution including the number and cumulative percentage of candidates at each raw mark

18. Main instruments of standardisation • 5. Pairs analyses • 6. Qualitative inputs: examiners’, moderators’ and examining bodies’ reports. • 7. Qualitative and quantitative inputs: Umalusi’s ‘research group’.

19. Guiding Principles of Standardisation • 1. Apply both positive and negative adjustments • 2. No adjustments of more than 10% in either direction • 3. Positive adjustments should generally not push the results above the HA (norm) • 4. Marks that are well beyond the HA/norm should be adjusted downwards, but not necessarily as far as the norm

20. Guiding Principles of Standardisation • 5. Raw marks which are close to the norm should not be adjusted • 6. Take account of reports by examiners, moderators and research group • 7. Any adjustment – no change in rank order • 8. Small subjects generally not adjusted - raw

21. Specific issues / debates • Does Umalusi meet with government and look at the pass rates on raw exam marks? • Does Umalusi adjust marks to achieve a particular pass rate? • Is Umalusi subjected to political pressure to achieve a particular pass rate?

22. WHAT CAN WE LEARN FROM THE NSC RESULTS? STANDARDISATION DECISIONS ERCO NSC 2010

28. WHAT CAN WE LEARN FROM THE NSC RESULTS? STANDARDISATION DECISIONS IEB NSC 2010

36. WHAT CAN WE LEARN FROM THE NSC RESULTS? STANDARDISATION DECISIONS DBE NSC 2010