Optionality: forms, intended benefits and potential issues. Sandra Johnson, Assessment Europe SQA Research seminar, October 2013. What is “optionality”?.
Optionality:forms, intended benefits and potential issues
Sandra Johnson, Assessment Europe
SQA Research seminar, October 2013
Any feature in examinations that allows different candidates for the same qualification to achieve that qualification through different assessment routes
Appears in the context of the question rather than the content
Allows candidates to choose between items of mandatory content
Reflects optional content in courses
Embedded within questions
Answer either Question B1 or Question B2 and Question B3
Compare the importance of the Executive in making policy, with reference to two political systems you have studied.
In your answer you should compare three aspects of policy making.
Compare the importance of the Judiciary, with reference to two political systems you have studied.
In your answer you should identify three aspects of the Judiciary.
8.Answer either A OR B.
A Describe how animals survive adverse conditions.
B Describe recombinant DNA technology.
Labelled diagrams may be used where appropriate
Total marks — 40
SECTION 1 — Scottish Text — 20 marks
Read an extract from a Scottish text you have previously studied and attempt the questions.
Choose ONE text from either
Part A — Drama Pages 2–7
Part B — Prose Pages 8–17
Part C — Poetry Pages 18–25
Attempt All the questions for your chosen text.
SECTION 2 — Critical Essay — 20 marks
Write ONE critical essay on a previously studied text from the following genres — Drama, Prose, Poetry, Film and Television Drama, or language.
Your answer must be on a different genre from that chosen in Section 1.
To allow flexibility in curriculum coverage
To maximise learning motivation and test motivation for candidates
To provide opportunities for candidates to build on their strengths and interests
Should optional questions be used in examinations?
Stalnaker, J.W., School and Society, 1935
Optional questions in tests and examinations
Devadson, M.D., Teacher Education, 1963
Test performance and the use of optional questions
Ducette, J. & Wolk, S., Journal of Experimental Education, 1972
Question choice in examinations: an experiment in geography and science
Taylor, E.G. & Nuttall, D.I., Educational Research, 1974
O level examined: the effect of question choice.
Willmott, A.S. & Hall, C.G. Schools Council, 1975.
Validity (comparability of demand)
Wisdom of candidates’ choices*
Grading (all candidates in the undifferentiated test score distribution)
Fairness to candidates
*Willmott, A.S. & Hall, C.G. (1975) O level examined: the effect of question choice. Schools Council.
Candidates do not always make the wisest question choices
Mark variation in testing situations principally arises from genuine between-candidate differences in the construct being assessed (e.g. history knowledge, mathematical ability, investigation skills), but also from between-question differences (in apparent difficulty), between marker differences (in standards), and “interaction effects” (e.g. markers marking more or less severely than others on particular questions, candidates performing better than others on some but not all questions – “jagged profiles”).
Interaction effects contribute to measurement error in candidate ranking applications, including examination-based grading. Where candidates can choose to respond to different questions then grading validity issues arise.
Section A – 20 3-mark short-answer questions
Section B – 6 15-mark structured questions
Johnson, S., Johnson, R., Miller, L. & Boyle, A. (2013) Reliability of Vocational Assessment: An evaluation of level 3 electro-technical qualifications.Coventry: Ofqual.
Section B differentiated more among the candidates than did Section A while there was more between–question variation in Section A than in Section B. In both sections more than half of the variation in candidate-question-marker scores could be attributed to candidate-question interaction, i.e. inconsistent performances by individual candidates across the questions (examiners put this down to poor candidate preparation).
Reliability for the whole paper: 0.71 (95% CI around total test mark ± 20.3)
one of several alternative Unit 1 papers
presented three extended response questions
candidates were required to answer two questions
so in practice there were three different
pathways through the paper
each question was worth 60 marks for a 120-mark paper total
Candidates who opted for pathway q2q3 were more reliably assessed than those who opted for pathway q1q2 or pathway q1q3
Johnson, S. & Johnson, R, (2012) Component reliability in GCSE and GCE Coventry: Ofqual.
a 2-section paper
two three-part open-ended questions in each section allowed candidates to choose between items of mandatory content
candidates were required to answer one question from each section
so in practice there were four different pathways through the paper
each question was worth 35 marks for a 70-mark paper total
Pathways q1q4 and q2q4 were very much more popular than pathways q1q3 and q2q3.
The reliability statistics look similar, but the analysis could not take into account
all the measurement error contributions, since marking was clip-based (i.e. part-questions were electronically randomly allocated to markers for marking).
Baird, J-A., Hayes, M., Johnson, R. Johnson, S. & Lamprianou, I. (2013) Marker effects and examination reliability. Coventry: Ofqual.
NB. The lesser popularity of optional pathways involving q1 or q3, combined with the systematic random allocation of seeded clips to markers from across the whole candidate entry, meant that there were too few multiply marked clips available for the reliability of marking of q1 and q3 to be explored
Should optionality be encouraged or eliminated?
If continued in future paper design – what forms would be most appropriate?
How can comparability in level of knowledge/skill demand be assured across optional choices?
How should marker reliability studies be modified to accommodate differential option popularity?
How soon can comprehensive reliability studies be introduced to look at optional pathways?
Without question pretesting, what might be done to give all candidates fair grading outcomes in the context of optionality?