Assessment: Challenge for learners & markers. Professor Colin Mason. "Assessment is the most powerful tool teachers possess in moulding student effort and learning." Graham Gibbs and Trevor Habeshaw, 1988
Professor Colin Mason
Graham Gibbs and Trevor Habeshaw, 1988
"If we wish to discover the truth about an educational system, we must look into its assessment procedures .. the spirit and style of student assessment defines the de facto curriculum."
Derek Rowntree, 1987Why assess?
So that lecturers can demonstrate how expert they are in their subject.
Because they are a very efficient means of transmitting information to large numbers of students.
To provide a common space where students can be enthused by a lecturer exploring key concepts of their subject.
Students expect them as part of university (rather than school) education and prefer listening to having to contribute themselves.
Lectures are easier to prepare than more interactive resource-based techniques such as case study discussions or problem-based tutorials.
Fits aims and learning outcomes
Saves time (and effort - eventually)
For understanding and expertise
Design and create VLE course
‘Posts’ to discussion boards
Workbooks, diaries, logs
Authentic Work-based observations by professionals
Higher Cognitive skills, Transferable skills
See also: Integrative Assessment Guide no 3
‘Complex’ outcomes including higher order academic abilities (analysis, critical reasoning) and ‘soft skills’ (teamwork, leadership) are rarely and inconsistently defined
Advanced ‘skilful practice’ is acquired slowly (years?)
Precision & reliability of such assessment is often only attained at the expense of validity
Shift to FORMATIVE assessment
Provides diagnostic information for students and staff (even if only ‘scores’)
Provides opportunities for Feedback (corrections) and Feedforward (suggestions for improvement ‘for next time’)
Ensures wider coverage of all learning outcomes
Speeding up feedback accelerates subsequent learning
objective assessments with instant feedback;
Objective assessment (eg MCQ, matching etc), especially utilising online approaches, form a significant element of an overall assessment strategy; EVS also.
Problem-based learning with ‘ideal’ or exemplar solutions made available (often online)
Synoptic summative assessment
Multiple Choice Questions (MCQ)
More sophisticated Objective Tests
Computer-based (online) administration of assessments
SummativeBlended assessment between units: Synoptic approach?
What attracts me about using MCQ assessment?
What are my concerns about using MCQ assessment?
What is an MCQ?
What could I use MCQs for?
What are the features of MCQs?
Does adopting MCQ assessment mean \'dumbing down\'?
Are MCQ scores unrealistically high?
Can students guess their way to success in an MCQ?
Does using MCQ assessment encourage rote or surface learning?
Can MCQs test oral and written skills?
What should I think about before I design an MCQ test?
How can I write effective MCQs?
How can I provide effective feedback for my MCQs?
What can I learn from the student responses to the MCQs?
A Multiple Choice Question (MCQ) is one that students are asked to select one ‘answer’ from a given list of options in response to a question ‘stem’
“Which of the following is the currency unit used in Albania?”
a right or a best answer
but scope for crediting more than one right answer or \'near miss\' answers, depending on the marking scheme
Testing is efficient
greater part of syllabus assessed by compulsory questions.
Answers to Questions are easy to score or mark
Especially online or computer-assisted eg DSO, WebCT, Blackboard, QuestionMark.
Scoring uses the entire marking range (of 100% scale)
raises issues where the MCQs are used in summative assessment.
Results are quantifiable
possible to analyse the level of student achievement in each question and thus identify areas of student difficulty or problematic questions
Feedback can be targeted more effectively
Setting questions is time consuming and challenging
Initial investment is high, but questions can be re-cycled in databanks
Recycling of questions raises ‘quality’ issues for summative assessments
Fewer students submit assignments, reducing staff marking loads.
Students can choose whether or not to complete an assessment assignment.
It allows students to choose when, where and how they complete an assessment assignment.
It provides non-threatening, ‘low stakes’, opportunities for students to learn from their mistakes and feedback provided on their assessed work.
Academic staff need to worry less about the accuracy of their marking because it doesn’t count.
(If correct, R1=5 marks; R2=3 marks ;R3=2 marks; R4=1; R5=0)
MCQ tests are increasingly used in assessment in higher education coursesBECAUSEthey can easily be marked by computer systems and are thus more objective
The assertion and the reason are both correct, and the reason is appropriate or valid.
The assertion and the reason are both correct, but the reason is inappropriate or invalid.
The assertion is correct but the reason is incorrect.
The assertion is incorrect but the reason is correct
Both the assertion and the reason are incorrect
‘An MCQ with 5 options presents a 20% chance of \'guessing\' the correct answer’
adopting mathematical strategies to \'normalise\' the mark (see Bush, 1999)?
raising the pass mark for the MCQ element of assessment?
making MCQs one component (low weighting) of the assessment strategy?
concentrating on using MCQs in formative situations where the \'result\' is less important than the process?
(Design:1 mark –correct; 0 mark –incorrect)
“Monkey Mark” 50/100 = 50% (Pass?)
(Design: 1 mark correct; -1 mark –incorrect)
Excellent student – 100%
“Monkey Mark” – 0%
3 option MCQ
Unfair: 1 mark –correct; -1 mark –each incorrect distractor
1 mark correct; -0.5 mark each incorrect
But, easier ….
2 mark correct; -1 mark each incorrect
Score = Deserved score + undeserved score (for guessing)
100 MCQ with 5 options; correct option – 4 marks
25 ‘guesses’; 1 in 5 chance (20%)
undeserved score = 20% of 25 x 4 marks
undeserved score = 5 x 4 = 20 marks
Deduction for undeserved score = No. Qs (25-5) x I
Deduction for undeserved score = 20 x 1 = 20
So, adjust Score according to
Score = Deserved + undeserved – deduction undeserved
Score = Deserved +20 – 20
Score = Deserved
Situation: 100 MCQs (5 options)
Student ‘knows’ 50; guesses 25; but ‘educated guesses’ 25 (8 correctly; 17 incorrectly)
Deserved 50 x 4 = 200
Undeserved 5 (20% of 25) x4 – 20xI = 0
Partial 8 x4 -17 x1 = 15
Total 200 + 0 + 15 =215 (out of 400)
If no negative marking ie 0 for incorrect answer)
Total = 200 + (5x4 – 20x0) + (8x4 – 17x0)
Total + 200 + 20 + 32 = 252
Order of preference (or ranking) of options
Choose option and assign ‘confidence’ level (1, 2, 3)
If correct marks = 1, 2 or 3
If incorrect marks = 0, -2, -6
X → (x-20) * 5/4 (x = unadjusted mark)
Liberal (more attempts) MCQ
Eg 5 option MCQ
1 attempt – score 4/4 = 100% on Question
2 attempts – score (4-1)/4 = 75 % on Question
3 attempts – score (4-2)/4 = 50% on Question etc
Deakin Studies Online (DSO)
e.g. Blackboard, Moodle
Assessment Management Systems
e.g. QuestionMark, TRIADS
e.g. Respondus, StudyMate
Techniques for Formative Assessment
Fill in the blanks
What are the “Christian” names for the Islamic prophets Ibrahim, Daviyd, Musa and Isa?
& Alternative Answers
Same as Medium Technique
Randomisation of a question’s possible answers
Randomisation of questions
Randomised from bank
e.g. any 30 questions from 100
Pre-defined and randomised
e.g. first 10 defined then 30 random questions
Same as Medium Technique
Approaches to assessment and feedback that foster independent learning
Online assessment Strategies
UK resources to assist MCQ designers
Wimba Create (CourseGenie)