19 january 2010 dr mafu s rakometsi
Download
1 / 66

Portfolio Committee on Basic Education Report on the Quality Assurance of the NSC - PowerPoint PPT Presentation


  • 177 Views
  • Uploaded on

19 January 2010 Dr Mafu S Rakometsi . Portfolio Committee on Basic Education Report on the Quality Assurance of the NSC. WHAT CAN WE LEARN FROM THE NSC RESULTS? Introduction to context, principles, approaches and processes Dr Mafu Rakometsi - CEO of Umalusi. Regulatory Framework

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Portfolio Committee on Basic Education Report on the Quality Assurance of the NSC' - taji


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
19 january 2010 dr mafu s rakometsi l.jpg

19 January 2010

Dr Mafu S Rakometsi

Portfolio Committee on Basic EducationReport on the Quality Assurance of the NSC


Slide2 l.jpg

WHAT CAN WE LEARN FROM THE NSC RESULTS?

Introduction to context, principles, approaches and processes

Dr Mafu Rakometsi - CEO of Umalusi


Slide3 l.jpg

  • Regulatory Framework

  • Quality Assurance of Assessment

  • NQF Act Section 27 (h)

  • The QC must develop and implement policy and criteria for assessment for the qualifications on its sub-framework.


Slide4 l.jpg

  • Section17 of the GENFETQA Act

  • (5) The Council must, with the concurrence of the Director-General and

  • after consultation with the relevant assessment body or education

  • institution, approve the publication of the results of learners if the

  • Council is satisfied that the assessment body or education institution

  • has—

  • (i) conducted the assessment free from any irregularity that may

  • jeopardise the integrity of the assessment or its outcomes;

  • (ii) complied with the requirements prescribed by the Council for

  • conducting assessments;

  • (iii) applied the standards prescribed by the Council which a learner is

  • required to comply with in order to obtain a certificate; and

  • (iv) complied with every other condition determined by the Council.


Slide5 l.jpg

  • Framework for QA of Learner Achievement

  • Based on established and existing practices in assessment for certification

  • Prescribed components of External assessment (examinations) and Site based/ internal / continuous assessment

  • Use of systems, processes, and procedures to evaluate, inspect, monitor and report on examination systems, processes and procedures of public and private assessment bodies.


Slide6 l.jpg

  • Framework for Quality Assurance of Assessment

  • Evaluation and /or accreditation of assessment bodies

  • Periodic inspection of assessment systems

  • Ongoing monitoring of assessment systems

  • Quality assurance of external examinations through:

  • Moderation of examination question papers

  • Monitoring and moderation of SBA

  • Monitoring the conduct of examinations

  • Moderation of marking

  • Standardization of assessment outcomes


Slide7 l.jpg

  • Approval of Results

    • Compliance with minimum requirements

  • Shift from Moderation to Verification (Moderation for verification)

  • Emphasis on Assessment Body reports

  • The monitoring of the conduct of examinations and the moderation of marking as well as the monitoring of the implementation of internal assessment and the moderation of internal assessment is undertaken by Umalusi to verify the veracity of the assessment body reports in respect of these process.

  • Broadened definition of irregularities (including assessment system processes)


What can we learn from the nsc results standardisation dr sizwe mabizela chair of umalusi council l.jpg

WHAT CAN WE LEARN FROM THE NSC RESULTS?

Standardisation

Dr Sizwe Mabizela - Chair of Umalusi Council


Slide9 l.jpg

  • Why Umalusi standardises results, and how

  • Provision of GENFETQA – Council may adjust raw marks.

  • International practice – large scale assessment systems

  • Standardisation – process used to mitigate the effect of factors other than learners knowledge and aptitude on the learners performance.

  • Sources of variability – difficulty in question paper, undetected errors, learner interpretation of questions


Slide10 l.jpg

  • Why Umalusi standardises results, and how

  • Assumptions – for large populations the distribution of aptitude and intelligence does not change appreciably

  • Process of standardisation

    • Moderation of question papers

    • Review of learner performance against historical performance of candidates in each subject.

    • Statistical moderation of Internal assessment


Slide11 l.jpg

  • Why Umalusi standardises results, and how

  • Qualitative input

  • Reports (moderator, Chief marker and Internal Moderator)

  • Umalusi research (post exam analysis)

  • Responsibility of Assessment Standards Committee

  • Committee of Council

  • Responsible for setting and maintaining standards

  • Observers (SAQA, HESA, Teacher Unions)


Slide12 l.jpg

  • General principles applied in the standardisation of examination marks

  • 1. In general no adjustment should exceed 10% or the historical average.

  • 2. Adjustments in excess of 10% could be considered at the upper end to increase the number of distinctions in a subject.

  • 3. In the case of the individual candidate, the adjustment effected should not exceed 50%of the raw mark obtained by the candidate.

  • 4. If the distribution of the raw marks is below the historical average, the marks may be adjusted upwards subject to the limitations.


Slide13 l.jpg

  • General principles applied in the standardisation of examination marks (cont.)

  • 5. If the distribution of the raw marks is above the historical average, the marks may be adjusted downwards subject to the limitations.

  • 6. The computer adjusted mark is calculated based on these principles.

  • 7. For those subjects with a practical component of 50%, raw marks could be accepted.

  • 8. Umalusi however retains the right to amend these principles where and when deemed to be necessary based on sound educational principles.


Slide14 l.jpg

  • Reasons why Umalusi does not disclose standardisation decisions

  • Process is complex and context driven

  • Sensitive information

  • Could lead to varied erroneous interpretations

  • Prejudicial to learners

  • International practice

  • Credibility of the process of arriving at the decision rather than the decision itself.


Slide15 l.jpg

  • Standardisation decisions Decisions for 2010

  • Way forward

  • Public interest vs. international practice

  • Composite report to be tabled with Portfolio Committee

  • Composite report to be made public


Slide16 l.jpg

  • The Class of 2010 decisions

    • Third cohort of candidates to write the new National Senior Certificate (NSC)

    • Completely new qualification based on the National Curriculum Statement

    • Some 642 001 enrolled including (full time and part-time candidates)

    • Approximately 67,8% of all those who enrolled as full time and who wrote have met the requirements of a pass


Some criteria for comparison l.jpg
Some criteria for comparison decisions

  • The number of candidates passing

  • The quality of the results

  • The standardisation of the exam

  • The predictive quality of the exam for performance at HE level


The number of candidates passing l.jpg
The number of candidates passing decisions

  • Massification is a natural outcome of an education system that is non-racist, non-sexist and democratic

  • It is the clear that there has been a steady increase over the last decade in the number of learners who enrolled for and wrote the SC and then NSC – notable exception 2010

  • The underlying philosophy of the NCS had been designed to ensure that most people achieve the minimum requirement for a pass

  • The NSC determines a pass or fail by 40% in at least three subjects and 30% in three others


The number of candidates passing trends in sc nsc enrolment and passes l.jpg
The number of candidates passing decisions :Trends in SC/NSC enrolment and passes

Senior Certificate examination results, all schools, 1994-2008

Sources: DoE, 2004, 2005a, 2005b, 2005c, 2007, 2008, 2009


Sc nsc examination results and he l.jpg
SC/ NSC examination results and HE decisions

SC/ NSC examination results, all schools, 1994-2010

Sources: DoE, 2004, 2005a, 2005b, 2005c, 2007, 2008, 2009


The quality of the results l.jpg
The quality of the results decisions

  • Concerns about quality has become a thorny issue and the subject of much debate albeit uninformed at most times.

  • SA public uses the matric results as the main indicators of quality of the education system and so there is an understandable concern about what these results of a relatively new qualification is saying about the current state of education

  • It is important however that we interrogate some of our assumptions that inform our beliefs about standards and quality


Assumptions that affect our perceptions l.jpg
Assumptions that affect our perceptions decisions

  • The first assumption is that “more means less” or “more means worse”

    Stellenbosch University Website:

    “In the light of phasing out of matriculation exemption and of the distinction between subjects on the higher and standard grade, a significant number of learners will probably qualify for basic admission to university on the basis of their subject combination…..The implication is that many more learners may be able to apply for admission to Stellenbosch University. ……. With this background, it becomes clear why it is necessary for the University to have its own measure to interpret the meaning and value of the new NSC and also to generate additional information in order to make finer distinctions.”

    • Higher pass rate is as a result of:

      • Lowering standards

      • Upward adjustment of marks


Trustworthiness the standardisation of the exam l.jpg
Trustworthiness: decisionsThe standardisation of the exam

  • Another assumption is that we now use vastly different methods of standardising the results

  • Responsibility for matriculation results has changed hands:

    • JMB (1918 to 1992)

    • SAFCERT (1992 – 2001)

    • Umalusi (2002 – to date)

  • The one constant has been the standardisation process used for the examinations.

  • This is crucial to:

    • Obtain equivalence of the standard of the SCE across years, subjects and examination authorities

  • It is also necessary in order to deliver a relatively constant product to the HE sector and to the workplace

  • Trust in the statutory institutions established with particular mandates.


Umalusi s research on comparisons of nsc with nated 550 l.jpg
Umalusi’s research on comparisons of NSC with NATED 550 decisions

  • Another assumption is that the NCS is of a lower standard than the previous curriculum.

  • Research completed by Umalusi into the standard of the NCS curriculum confirms that in most cases the NCS presents a greater cognitive challenge

  • The NCS also represents modern, updated and more demanding versions of previous subjects

    (Ref: Maintaining Standards Reports 2008, 2009, 2010)


Sc nsc examination results l.jpg
SC /NSC examination results decisions

Senior Certificate examination results, all schools, 1994-2010

Sources: DoE, 2004, 2005a, 2005b, 2005c, 2007, 2008, 2009


Senior certificate examination results l.jpg
Senior Certificate examination results decisions

Senior Certificate examination results, all schools, 1994-2008

Sources: DoE, 2004, 2005a, 2005b, 2005c, 2007, 2008, 2009


Slide27 l.jpg

Quality Assurance of the 2010 National Senior Certificate Examination

Vijayen Naidoo – Sen. Manager : Quality Assurance of Assessments


Scope of the quality assurance processes l.jpg
Scope of the quality assurance processes Examination

Moderation of Question papers

Number of NSC 2010 question papers moderated


Slide29 l.jpg

PURPOSE Examination

  • to ensure that the question papers are of the required standard

  • To ensure that the question papers are relatively:

    - fair

    - reliable

    - representative of an adequate sample of the curriculum

    - representative of relevant conceptual domains

    - representative of relevant levels of cognitive challenge

  • External moderators

Moderation of question papers


Moderation of the question papers l.jpg
Moderation of the question papers Examination

Approach

  • Question papers set by panel of examiners – DBE

  • Internally moderated by DBE

  • Externally moderated by Umalusi

  • Subsequent moderations and approval.


Moderation of the question papers31 l.jpg
Moderation of the question papers Examination

Examples:

  • Approved at 3rd moderation (CAT; History; IT; Life Sciences P1; Math Lit P2; Physical Sc P1 etc.)

  • Approved at 2nd moderation (Business Studies; Economics; Life Sciences P2; Maths; Physical Sc P2 etc.)

  • Isizulu FAL P3 & Sesotho HL P1 rejected at 1st moderation.


Moderation of the question papers32 l.jpg
Moderation of the question papers Examination

Findings

  • Technical Criteria

  • Internal moderation – quality of internal moderator reports.

  • Content coverage

  • Cognitive Demand

  • Marking Guidelines

  • Language and Bias

  • Predictability


Moderation of the question papers33 l.jpg
Moderation of the question papers Examination

Maths Lit P1- Content coverage


Moderation of the question papers34 l.jpg
Moderation of the question papers Examination

Civil Technology- Cognitive Weighting


Slide35 l.jpg

Moderation of internal /continuous assessment Examination

  • Internal assessment refers to any assessment conducted by the provider, the outcomes of which count towards the achievement of the qualification

  • panels of moderators / subject specialists

  • Moderation at all levels

  • Directives for internal assessment

Scope of the quality assurance processes…


Slide36 l.jpg

Purpose Examination

  • Ascertain the appropriateness and standard of the assessment tasks being developed within colleges

  • Ascertain the degree to which assessment bodies/provinces are attempting to ensure standardisation across

  • Determine the extent and quality of internal moderation and educator development

  • Determine the reliability and validity of the assessment outcomes

Moderation of internal assessment (cont.)


Slide37 l.jpg

  • Moderation of SBA Examination

  • Previous model - moderation of learner and educator portfolios at the end of the year.

  • Implementation of new model in 2010 –focus on tasks rather than on portfolios

    • Ongoing – 3 moderations across 9 PDE’s

    • Feed back to educators

    • Improvement in SBA implementation



Slide39 l.jpg

  • Moderation of SBA Examination

  • Findings

  • Adherence to policy guidelines

  • Quality of Assessment tasks

    • Content Coverage

    • Cognitive demand and difficulty levels

    • Marking (tools, application and mark allocation)

  • Internal Moderation

    • Moderation instruments

    • Moderation at School level & district level

    • Feedback and Support


  • Slide40 l.jpg


    Slide41 l.jpg

    • Moderation of Life Orientation Examination

      • Findings

      • Good practice – common tasks

      • Concerns

      • Quality of tasks

      • moderation

      • Use of rubrics

      • Physical Education Tasks

      • Recommendations

      • At least one common task


    Scope of the quality assurance processes42 l.jpg
    Scope of the quality assurance processes Examination

    Focus of the monitoring visits

    • Policy development and implementation

    • Systems, processes and procedures

    • Plan for assessment and moderation

    • Availability and training of staff

    • Planning for the development of learner portfolios and educator portfolios


    Slide43 l.jpg


    Scope of the quality assurance processes44 l.jpg
    Scope of the quality assurance processes… Examination

    Monitoring of the NSC examination: writing

    Findings

    • General Management of the examinations

    • Candidate registration

    • Irregularities

    • Mainly technical

    • Packaging


    Scope of the quality assurance processes45 l.jpg
    Scope of the quality assurance processes… Examination

    Monitoring of the NSC examination: marking

    Findings

    • Delay in commencement of marking

    • Delay in capturing of marks

    • Concern with the appointment of markers

    • Large number of marking centers – security risk


    Slide46 l.jpg

    PURPOSE Examination

    • Moderation of marking determines the standard and quality of marking and ensures that marking is conducted in accordance with agreed practices

    • Umalusi engages the following during the moderation of marking

      1. Pre-marking/memorandum discussion: centralised memo discussions recommended - this will ensure consistency across marking centres

      2. Off-site moderation of marking

    Verification of marking


    Scope of the quality assurance processes47 l.jpg
    Scope of the quality assurance processes… Examination

    Verification of marking

    • Memoranda discussion meetings

      • Approved and signed off finalized memoranda/ marking guidelines for all the NSC subjects

      • Concern with non attendance and pre- memo discussion preparations


    Scope of the quality assurance processes48 l.jpg
    Scope of the quality assurance processes… Examination

    Centralized verification of marking


    Scope of the quality assurance processes49 l.jpg
    Scope of the quality assurance processes… Examination

    Centralized verification of marking

    Findings

    • Adherence to marking memo

    • Provision of alternate answers

    • Consistency and accuracy

    • Internal moderation


    Scope of the quality assurance processes50 l.jpg
    Scope of the quality assurance processes… Examination

    Centralized verification of marking

    Concerns

    • Change to marking guidelines

    • Competence of markers

    • Sample of scripts verified by Umalusi


    Scope of the quality assurance processes51 l.jpg
    Scope of the quality assurance processes… Examination

    Standardisation

    • 58 subjects standardised

    • Raw marks accepted: 39 subjects

    • Upward adjustments: 9 subjects

    • Downward adjustments: 10 subjects


    Scope of the quality assurance processes52 l.jpg
    Scope of the quality assurance processes… Examination

    Standardisation

    Concerns

    • African Languages

    • Consumer Studies



    Areas of good practice l.jpg
    Areas of good practice Examination

    • Generally papers catered for both the weaker & high performing candidates,

    • In many cases the Nov 2010 & March 2011 were presented at the same time for moderation, this ensured comparability of standards

    • External moderators and Umalusi staff signed the final papers off before dispatch to provinces for printing,

    • Great improvement observed in the monitoring of exams by provinces,

    • Security measures were intensified in the majority of centers across provinces,


    Areas of concern l.jpg
    Areas of concern Examination

    Registration of Candidates

    • Filtering of candidates

      Moderation of Question Papers

    • Late submission of question papers for moderation.


    Areas of concern56 l.jpg
    Areas of concern Examination

    Moderation of SBA

    • Majority of provinces do not conduct actual moderation of SBA tasks & learner evidence. Current moderation is a mere audit for compliance,

    • Majority of the SBA tasks reflect over reliance on past examination papers, educators lack creativity in developing suitable tasks,

    • SBA tasks developed by educators focus mainly on assessing the lower cognitive skills,

    • Educators battle with the development & use of rubrics used for scoring learners,


    Areas of concern57 l.jpg
    Areas of concern… Examination

    • Implementation & Assessment of Life Orientation (LO) requires serious attention. LO is 100% internally assessed and standards vary from one province to the other, even within provinces.

      Monitoring of exams:

    • Incidents of improper registration of candidates resulted in duplications and some candidates not registered or registered for the incorrect subjects,

    • The appointment of suitable Chief Invigilators is encouraged. This will address problems relating to the flouting of even basic examination regulations.


    Areas of concern58 l.jpg
    Areas of concern… Examination

    • Criteria for the appointment of markers should be strictly adhered to so as not to disadvantage candidates.

    • All stakeholders need to be consulted when changes are made relating to the appointment of markers.

      Moderation of marking:

    • Memoranda discussion meetings for small subjects were generally a disservice to the external moderators due to the non-attendance of the relevant provincial delegates. Added to this was the absence of input from the provinces.


    Areas of concern59 l.jpg
    Areas of concern… Examination

    • Additions were made to the final memoranda without the consent of the DBE and Umalusi.

    • Some markers do not have the ability and experience to handle higher-order cognitive level questions that required insight and logical reasoning.

    • There were instances of inaccurate totaling of marks and incorrect transferring of marks to the cover page during marking.


    Areas of concern60 l.jpg
    Areas of concern… Examination

    • Potential administrative and management problems in Limpopo, KwaZulu Natal and Free State

    • Large number of marking centers – security risk


    Recommendations l.jpg
    Recommendations Examination

    Moderation of SBA:

    • Introduction of a common (standardised task)

    • Moderation of the standard of SBA must be implemented at the school and cluster levels. Checks for compliance can still be part of the moderation process to ensure adherence to policy.

    • All the assessment bodies should ensure that all educators are familiar with the policy requirements to ensure the meeting of national standards.

      .


    Recommendations62 l.jpg
    Recommendations … Examination

    Verification of marking:

    • Criteria for the appointment of markers should be strictly adhered to so as not to disadvantage candidates.

    • All stakeholders need to be consulted when changes are made relating to the appointment of markers.

      Standardization of marks:

    • Common framework for the assessment of Languages.

    • Teaching and learning of Mathematics and Science needs to be strengthened.


    Examination irregularities l.jpg
    Examination Irregularities Examination

    • The majority of irregularities were of a technical nature and these were reported to Umalusi according to the established channels.

    • Some irregularities were as a result of registration-related problems, e.g. candidates nor appearing on mark sheets, some registered for incorrect subjects.

    • Umalusi represented on NEIC

      Major irregularities:

    • No major irregularities reported


    Conclusion l.jpg
    Conclusion Examination

    • The findings of the quality assurance processes are a clear indication of a maturing system that has, on the one hand, made positive strides towards improvement in certain areas of assessment and examination, but, on the other hand, still has a few challenges that need to be addressed.

    • The quality assurance of each of these processes presented above was conducted based on Umalusi criteria. Umalusi uses criteria that are subjected to constant review and refinement, to ensure that they are in line with current trends in assessment and examinations.


    Conclusion65 l.jpg
    Conclusion… Examination

    • In general Umalusi is pleased with the manner in which the 2010 NSC examination was administered.

    • Umalusi acknowledges that a number of technical irregularities were reported, but these were addressed in a fitting manner.

    • Umalusi takes this opportunity to express appreciation to the national & provincial departments of education for their concerted effort in ensuring a credible examination.

    • Umalusi expresses appreciation also to all the relevant stakeholders for the necessary support given in line with Umalusi quality assurance initiatives.


    Slide66 l.jpg

    Thank you! Examination


    ad