Assessment: Challenge for learners & markers

1. Assessment: Challenge for learners & markers Professor Colin Mason

2. "Assessment is the most powerful tool teachers possess in moulding student effort and learning." Graham Gibbs and Trevor Habeshaw, 1988 "If we wish to discover the truth about an educational system, we must look into its assessment procedures .. the spirit and style of student assessment defines the de facto curriculum." Derek Rowntree, 1987 Why assess?

3. Pick the best answerWhy are lectures still used so frequently in higher education? So that lecturers can demonstrate how expert they are in their subject. Because they are a very efficient means of transmitting information to large numbers of students. To provide a common space where students can be enthused by a lecturer exploring key concepts of their subject. Students expect them as part of university (rather than school) education and prefer listening to having to contribute themselves. Lectures are easier to prepare than more interactive resource-based techniques such as case study discussions or problem-based tutorials.

4. Tensions in Assessment(George Brown, 1996) • EFFECTIVENESS Fits aims and learning outcomes • EFFICIENCY Saves time (and effort - eventually) • ENABLEMENT For understanding and expertise

5. Some examples of innovation in methods of assessment: Exams Coursework Unseen exams Presentations Annotated Bibliography Seen exams Reports Design and create VLE course ‘Posts’ to discussion boards Essays PBL scenarios MCQs OSCEs Workbooks, diaries, logs Reflective Report Practical tests Authentic Work-based observations by professionals Higher Cognitive skills, Transferable skills Knowledge, Understanding See also: Integrative Assessment Guide no 3

6. Complex outcomes of learning ‘Complex’ outcomes including higher order academic abilities (analysis, critical reasoning) and ‘soft skills’ (teamwork, leadership) are rarely and inconsistently defined Advanced ‘skilful practice’ is acquired slowly (years?) Precision & reliability of such assessment is often only attained at the expense of validity Shift to FORMATIVE assessment

7. DIAGNOSTIC DIAGNOSTIC FORMATIVE FORMATIVE SUMMATIVE SUMMATIVE Forms (Modes) of Assessment ‘Reality’ ‘Theory’

8. So, formative assessment …. Helps learning Provides diagnostic information for students and staff (even if only ‘scores’) Provides opportunities for Feedback (corrections) and Feedforward (suggestions for improvement ‘for next time’) Ensures wider coverage of all learning outcomes

9. Speedier Feedback Speeding up feedback accelerates subsequent learning Solutions: objective assessments with instant feedback; exemplar answers Objective assessment (eg MCQ, matching etc), especially utilising online approaches, form a significant element of an overall assessment strategy; EVS also. Problem-based learning with ‘ideal’ or exemplar solutions made available (often online)

10. Assessment Efficiency Synoptic summative assessment Multiple Choice Questions (MCQ) More sophisticated Objective Tests Computer-based (online) administration of assessments

11. Unit 1 Unit 4 Unit 3 Unit 2 Unit 6 Unit 5 Formative Formative Summative Summative Blended assessment between units: Synoptic approach? LATERAL LONGITUDINAL TRIMESTER 2 TRIMESTER 1

12. Assessment Efficiency – MCQs?http://www.ukcle.ac.uk/resources/trns/mcqs/index.html What attracts me about using MCQ assessment? What are my concerns about using MCQ assessment?

13. Assessment Efficiency – MCQs?http://www.ukcle.ac.uk/resources/trns/mcqs/index.html What is an MCQ? What could I use MCQs for? What are the features of MCQs? Does adopting MCQ assessment mean 'dumbing down'? Are MCQ scores unrealistically high? Can students guess their way to success in an MCQ?

14. Assessment Efficiency – MCQs?http://www.ukcle.ac.uk/resources/trns/mcqs/index.html Does using MCQ assessment encourage rote or surface learning? Can MCQs test oral and written skills? What should I think about before I design an MCQ test? How can I write effective MCQs? How can I provide effective feedback for my MCQs? What can I learn from the student responses to the MCQs?

15. What is an MCQ – Design issueshttp://www.le.ac.uk/castle/resources/ A Multiple Choice Question (MCQ) is one that students are asked to select one ‘answer’ from a given list of options in response to a question ‘stem’ “Which of the following is the currency unit used in Albania?” Dinar Rouble Kwacha Lek Rupee Options ‘Answer’ Stem Distractors

16. What are the features of MCQs? Objective marking a right or a best answer but scope for crediting more than one right answer or 'near miss' answers, depending on the marking scheme Testing is efficient greater part of syllabus assessed by compulsory questions. Answers to Questions are easy to score or mark Especially online or computer-assisted eg DSO, WebCT, Blackboard, QuestionMark. Scoring uses the entire marking range (of 100% scale) raises issues where the MCQs are used in summative assessment.

17. What are the features of MCQs? Results are quantifiable possible to analyse the level of student achievement in each question and thus identify areas of student difficulty or problematic questions Feedback can be targeted more effectively Setting questions is time consuming and challenging Initial investment is high, but questions can be re-cycled in databanks Recycling of questions raises ‘quality’ issues for summative assessments

18. Rank (R) the options (1 - 5)There is an increasing interest in the use of formative assessment in higher education because … Fewer students submit assignments, reducing staff marking loads. Students can choose whether or not to complete an assessment assignment. It allows students to choose when, where and how they complete an assessment assignment. It provides non-threatening, ‘low stakes’, opportunities for students to learn from their mistakes and feedback provided on their assessed work. Academic staff need to worry less about the accuracy of their marking because it doesn’t count. (If correct, R1=5 marks; R2=3 marks ;R3=2 marks; R4=1; R5=0)

19. Evaluate the sentence explaining why MCQ tests are used in higher education these days MCQ tests are increasingly used in assessment in higher education coursesBECAUSEthey can easily be marked by computer systems and are thus more objective The assertion and the reason are both correct, and the reason is appropriate or valid. The assertion and the reason are both correct, but the reason is inappropriate or invalid. The assertion is correct but the reason is incorrect. The assertion is incorrect but the reason is correct Both the assertion and the reason are incorrect

20. High Marks and Guessing – What to do about it? ‘An MCQ with 5 options presents a 20% chance of 'guessing' the correct answer’ ‘negative scoring’? adopting mathematical strategies to 'normalise' the mark (see Bush, 1999)? raising the pass mark for the MCQ element of assessment? making MCQs one component (low weighting) of the assessment strategy? concentrating on using MCQs in formative situations where the 'result' is less important than the process?

21. True/False -a special case MCQ 100 T/F (Design:1 mark –correct; 0 mark –incorrect) “Monkey Mark” 50/100 = 50% (Pass?) 100 T/F (Design: 1 mark correct; -1 mark –incorrect) Excellent student – 100% “Monkey Mark” – 0%

22. MCQ – Negative mark Designs 3 option MCQ Unfair: 1 mark –correct; -1 mark –each incorrect distractor So ……. 1 mark correct; -0.5 mark each incorrect But, easier …. 2 mark correct; -1 mark each incorrect

23. I =C/(n-1)MCQ scheme for negative marking

24. Negative scoring for the whole test Score = Deserved score + undeserved score (for guessing) 100 MCQ with 5 options; correct option – 4 marks 25 ‘guesses’; 1 in 5 chance (20%) undeserved score = 20% of 25 x 4 marks undeserved score = 5 x 4 = 20 marks Deduction for undeserved score = No. Qs (25-5) x I Deduction for undeserved score = 20 x 1 = 20 So, adjust Score according to Score = Deserved + undeserved – deduction undeserved Score = Deserved +20 – 20 Score = Deserved

25. What about ‘educated guessing’? Situation: 100 MCQs (5 options) Student ‘knows’ 50; guesses 25; but ‘educated guesses’ 25 (8 correctly; 17 incorrectly) Scoring: Deserved 50 x 4 = 200 Undeserved 5 (20% of 25) x4 – 20xI = 0 Partial 8 x4 -17 x1 = 15 Total 200 + 0 + 15 =215 (out of 400) If no negative marking ie 0 for incorrect answer) Total = 200 + (5x4 – 20x0) + (8x4 – 17x0) Total + 200 + 20 + 32 = 252

26. Other Adjustments Order of preference (or ranking) of options Confidence Choose option and assign ‘confidence’ level (1, 2, 3) If correct marks = 1, 2 or 3 If incorrect marks = 0, -2, -6 Normalisation X → (x-20) * 5/4 (x = unadjusted mark) Liberal (more attempts) MCQ Eg 5 option MCQ 1 attempt – score 4/4 = 100% on Question 2 attempts – score (4-1)/4 = 75 % on Question 3 attempts – score (4-2)/4 = 50% on Question etc

27. Formative online assessment On-line approaches Deakin Studies Online (DSO) e.g. Blackboard, Moodle Assessment Management Systems e.g. QuestionMark, TRIADS Tools e.g. Respondus, StudyMate Techniques for Formative Assessment Simple Medium Advanced Fun

28. Self Assessment Exercises DSO Self Assessment Tool MCQs within content using (simple) javascript On-line Assessment: Simple

29. On-line Assessment: Medium DSO tests/quizzes Multiple Choice True/False Multiple Answer Matching Pair Fill in the blanks Short Answer Paragraph/Essay

30. Example What are the “Christian” names for the Islamic prophets Ibrahim, Daviyd, Musa and Isa?

31. Multiple Choice

32. Multiple Answer

33. Fill in the blanks Answered Correctly & Alternative Answers

34. Matching Pair

35. On-line Assessment: Advanced Same as Medium Technique PLUS Randomisation of a question’s possible answers Randomisation of questions All randomised Randomised from bank e.g. any 30 questions from 100 Pre-defined and randomised e.g. first 10 defined then 30 random questions

36. On-line Assessment: Advanced Same as Medium Technique PLUS • Timed assessments (“beat the clock”) • Timed released • Incentive driven e.g. • If score >80% release other learning materials • Extra support • If score <40% release more learning materials • + extra test

37. StudyMate On-line Assessment: Fun Flash based - can be used within DSO or on standard web pages e.g. • Crosswords • Pick a Letter • Challenge • And more…

38. Useful URLs Approaches to assessment and feedback that foster independent learning http://www.deakin.edu.au/itl/pd/tl-modules/assessment/index.php Online assessment Strategies http://www.deakin.edu.au/itl/dso/strategies-teaching/assessment.php UK resources to assist MCQ designers http://www.le.ac.uk/castle/index.html Studymate http://www.deakin.edu.au/itl/dso/tools/studymate.php Wimba Create (CourseGenie) http://www.wimba.com/products/wimba_create/ Votapedia http://www.votapedia.com/index.php?title=Main_Page

39. Contact us Institute of Teaching and Learning http://www.deakin.edu.au/itl/ ITL Support Service http://www.deakin.edu.au/itl/contact/contact-us.php And, Phone: 5227 8127 (ext. 78127) Email: itl-support@deakin.edu.au