Controlling marking q uality in e-Marker ®

1. Controlling marking quality in e-Marker®

2. Marking quality control • Marking quality can be delivered by: • e-Standardisation • Seeding and qualification • Percentage double marking • S-Process

3. Seeding or percentage double marking? • If the marking guidance is detailed and objective and you would expect complete agreement among markers – use seeding • If the marking guidance is subjective – or gives guidance for the approach to marking, rather than what is a right or wrong answer – use percentage double marking • You can use seeding and percentage double marking on the same component • S-Process can be added to both quality methods for further quality control if desired

4. Seeding and qualification

5. Seeds • Seeds provide an in-built quality check • The administrators define and maintain the levels and frequency of the quality checks • Senior markers pre-mark a selection of responses - seeds • Quality control applies at individual question level • Quality is checked at the start of each day (qualification) and while the markers mark (seeding) • Markers can see the seeds they have failed so that they can improve their marking

6. Seeding Quality check: must not fail both seeds Quality check: must not fail both seeds Qualify at start of day Mark batch of questions Mark batch of questions Mark seeds Stop Marking Question Pass Fail Start marking after discussions with Senior or mark other questions Quality check: pass 9 out of 10

7. Seed window Quality check: 7 out of 10 within tolerance

8. Seed window Quality check: 7 out of 10 within tolerance

9. Seeding is set by senior markers • Seeding needs to be done before the markers start marking • Ensure there are enough seeds – the minimum seed bank size should be reached • If seeds are retired, they will need replacing • On a seeded question, the senior markers’ mark will be awarded to the candidate

10. Seeding for real

11. What makes a good seed? • Legible • Unambiguous and complete • Reflects the standards agreed in standardisation • No tricks! • Does not have to be the right answer • Don’t set “not attempted” responses as seeds

12. Percentage double marking

13. Percentage double marking • Percentage double marking involves: • Comparing two marking opinions in real time at question level; • Where differences in marking exceed a set tolerance, automated quality checks are used to invoke adjudication by a senior marker or peer

14. 1 2 Percentage double marking Marker 1 receives a portion to mark. It is selected by e-Marker® as a quality control item (benchmark item) and is forwarded to the next available marker Marker 2 marks the same clip, not knowing it has been marked before. If Markers 1 and 2 agree, no further action is taken and the mark is awarded to the candidate. In this example however, it is marked outside tolerance of Marker 1, so the marking of both markers is forwarded for adjudication by the Senior Marker The Senior Marker adjudicates. In this case he/she agrees with Marker 2, so Marker 1 receives a penalty

15. Percentage double marking • There are controls in e-Marker® administration to: • Ensure that markers mark at a similar pace; • Don’t collude in their marking; • Identify when discrepancies are forwarded to peer or senior; and • Identify when adjudication goes against a marker

16. Pioneer cap The pioneer cap defines how many benchmarks the fastest marker can be ahead of the next fastest. The prevents one marker marking too much without any quality control Too high? Too low? • First marker could mark their entire quota without being checked against other markers • Be particularly careful in small marker pools • Could slow down marking progress with markers being halted too often • Could make the frequency of the quality checks overt to markers

17. Partnering gap The partnering gap defines how frequently a marker’s benchmarks can be marked by each other marker Too high? Too low? • Marking is likely to be slowed down as new markers are needed • Could leave difficult, outstanding parts to mark at the end of marking • Could have markers colluding, or markers with similar opinions remarking each other’s work

18. Suspect cap The suspect cap defines how many adjudications can be outstanding for a marker. This prevents markers from getting too far ahead without the senior checking their work Too high? Too low? • Markers could potentially carry on marking when their marking is eventually shown to be substandard • Markers could be stopped from marking too frequently and, potentially, unnecessarily

19. Penalty cap The penalty cap defines how many adjudications can be delivered against a marker before the stopped marker management process is invoked Too high? Too low? • A marker below the standards required will be allowed to continue marking • Markers stopped too often, and potentially, unnecessarily • Additional work for senior markers in stopped marker management

20. Marker quality review • Marker review process is used during percentage double marking • Can be used if there are concerns about a particular marker or about marking standards at a particular time • Senior markers can request that a greater proportion of marking is checked • Can be done retrospectively, for example, work done in the past can be checked

21. Marker review process Marker 1 Marker 3 Marker 2

22. Stopped markers

23. Stopped markers • Senior markers can view quality failures of markers within stopped marker management. They can: • Permanently stop markers • Restart markers • For seeding and S-Process, can identify if quality check is not of a good standard

24. Setting the parameters

25. Seeding parameters • The function and application of seeding is governed by parameters set in the e-Marker® Administration system • These parameters define: • The number of seeds in qualification • The frequency at which seeds are presented in marking • The acceptable number of seeds that can be failed before a marker is stopped • The minimum seed bank level required

26. Percentage double marking parameters • The function and application of percentage double marking is governed by parameters set in the e-Marker® Administration system • These parameters define: • The frequency with which marking is benchmarked • The pioneer cap • The partnering gap • The suspect cap • The penalty cap

27. Parameter modelling

28. Controlling Marking Quality in e-Marker®