1 / 39

DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University

CONSTRUCTED RESPONSE ITEMS: MEQ & SAQ. DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003. Hong Kong International Consortium. CONSTRUCTED RESPONSE ITEMS. ASSUMPTIONS • Higher cognitive processes

melodie-foreman
Download Presentation

DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CONSTRUCTED RESPONSE ITEMS: MEQ & SAQ DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer WorkshopOctober 28, 2003 Hong Kong International Consortium

  2. CONSTRUCTED RESPONSE ITEMS ASSUMPTIONS • Higher cognitive processes • Increased reliability through wider sampling ARE THE ASSUMPTIONS JUSTIFIED?

  3. MODIFIED ESSAY QUESTIONS: MEQs 1970s – 1980s RCGP (Hodgkin & Knox 1975) Newcastle Australia (Feletti 1980, Feletti and Smith 1986) Jefferson USA (Rabinowitz 1987, Rabinowitz and Hojat 1989)

  4. Learning objectives, concepts, deep structures Clinical scenario 7 – 10 questions Sequential (information question information question ...) Integrated (content, media – Rabinowitz 1985) MODIFIED ESSAY QUESTIONS: MEQs

  5. HIGHER COGNITIVE ORDER(Bloom 1956, Buckwalter et al 1981) 1. Knowledge Level 1 Recognition Recall 2. Comprehension Level 2 Comprehension Interpretation 3. Application 4. Analysis Level 3 Application5. Synthesis 6. Evaluation (Irwin and Bamber 1982) MODIFIED ESSAY QUESTIONS: MEQs

  6. LEVEL 1 RECOGNITION AND RECALL Know • What is …..? • List ….. • Identify ….. MODIFIED ESSAY QUESTIONS: MEQs

  7. MODIFIED ESSAY QUESTIONS: MEQs LEVEL 2 COMPREHENSION, INTERPRETATION Understand • Give reasons …. • Explain …. • Interpret ….

  8. LEVEL 3 APPLICATION Apply, Utilise, Judge • How does X assist in explaining Y? • What are the components of X?How are they related?How do they affect Y? • How does XYZ affect treatment and management? • Assess, judge X in terms of …. MODIFIED ESSAY QUESTIONS: MEQs

  9. MODIFIED ESSAY QUESTIONS – VALIDITY •CONTENT(Blueprints, objectives, deep structures) • CONSTRUCT(Level 1, 2, 3 questions)

  10. MODIFIED ESSAY QUESTIONS – VALIDITY CONSTRUCTNorman et al 1987 Effect of educational level MCQ and MEQ not different skills UEQ may be a different skill Small effect of contextFeletti 1980More Level 1 and 2 in first year undergraduate Irwin and Bamber 1986More Level 3 in final year undergraduate

  11. PREDICTIVE Irwin and Bamber 1986 Correlation with clinical examination 0.41Rabinowitz 1987 MCQ better predictor NBME I & IIRabinowitz and Hojat 1989 Higher correlation for MEQ than MCQ with overall clerkship ratings MODIFIED ESSAY QUESTIONS – VALIDITY

  12. Inter-rater reliability (model answers) Norman et al 1987Inter-rater reliability satisfactory Internal consistency (cumulative errors) Feletti 1980Alpha(60) 0.57 – 0.91 Stratford and Pierce-Fenn 1985Alpha(60) 0.39 MODIFIED ESSAY QUESTIONS – RELIABILITY

  13. Administration Marking Higher order questions MODIFIED ESSAY QUESTIONS – FEASIBILITY

  14. Model PBL process Determine objectives Construct scenario MINI-CASES: THE FLINDERS MEQ

  15. John Taylor (age 55) has been brought to the E.D. by his wife. John woke up this morning (one hour ago) with nausea and vertigo. He managed to walk to the bathroom but had to hold onto the furniture and walls to steady himself. He vomited when he reached the bathroom. 1. Give 3 hypotheses to account for John’s condition. 2. Explain the mechanisms which give rise to John’s condition. 3. What further information would you like to distinguish between hypotheses. Explain how this information will distinguish between hypotheses. MINICASES: THE FLINDERS MEQ

  16. • Provide further information • Cumulative error cues • Continue scenario MINI-CASES: THE FLINDERS MEQ

  17. John has had no significant medical problems in the last few years. He spent the previous evening at home watching TV and did not drink any alcohol or take any other substance. ….. The left pupil is smaller than the right ….. ….. John’s uvula deviates to the right ….. ….. Movements of the left hand are clumsy. MINICASES: THE FLINDERS MEQ

  18. • Further questions • Continue unfolding scenario • Page breaks, cumulative cues • Conclude with SAQs (Level 2/3) MINI-CASES: THE FLINDERS MEQ

  19. Face validity Construct validity Model answers Higher order questions MINI-CASES: THE FLINDERS MEQ

  20. Reform of RCGP MEQ– Swanson – Mullholand & McAleer Increase number (10 – 12 / 2 hours)– Reflect reality Increase independence Single marked – schedules SAQ (Lockie et al 1990) SHORT ANSWER QUESTIONS - SAQ

  21. response contextual information + Q criteria Flinders SAQ strategies Evaluation:“The ability to make a value-judgement in relation to relevant criteria” • Opportunity to target highest cognitive skills • application - analysis - synthesis - evaluation Bloom 1956

  22. response contextual information + Q criteria Targeting “evaluation” response • Provide multiple, detailed, relevant information • Patient • Problem • Clinical care setting • Your role

  23. Example scenario information • Ron Harkin is a 32 year-old father of two and self-employed gardener. He falls heavily. He is unable to stand and is in considerable pain. He is taken to hospital where you are the Intern in the Emergency Department. You note swelling and that his foot appears to be at an unusual angle to his leg. • A 7 year-old Aboriginal girl, Alice Mundi, presents with a mild fever and painful knee to a small rural hospital. Her mother has brought her 50km to the hospital. You are the GP called to see her.

  24. response contextual information + Q criteria Targeting “evaluation” response • 2. Pose question requiring value-judgement • focus the response • assume criteria are understood • relate to what you would do • 3. Extend information / probe further (optional)

  25. Example questions targeting “evaluation” • Rationalising the diagnosis • What further information from the history do you require? • On what aspects of the physical examination will you • focus? • What investigations would you undertake to establish the • diagnosis? • What are your differential diagnoses? Which do you • consider most likely?

  26. Example questions targeting “evaluation” • Formulating an appropriate management plan • What would be your immediate management goals? • Which treatments would you advise? • For what potential complications would you monitor? • What longer-term management strategy would you advise? • What advice would you give in relation to this new • development?

  27. Mr Dawes, a 58 yo man, presents to the Emergency Dept with a 24h history of vomiting and colicky abdominal pain. He has had no bowel movement for 24h. He had an appendicectomy at age 30 to remove a gangrenous appendix. On examination you find he has a distended abdomen with no focal tenderness. There is no palpable hernia. He has symptoms and signs consistent with dehydration of about 10%. Fluid resuscitation is commenced. A plain abdominal Xray shows distended loops of small bowel with air fluid levels and normal calibre large bowel. Q1. What is your most likely diagnosis for this patient? (2 marks) Q2. What possible causes of this would you consider? (3 marks) After 48h of conservative management, Mr Dawes’ condition is such that surgical intervention is warranted. Q3. What clinical indications would suggest to you that Mr Dawes needs surgery to resolve his problem at this time? (5 marks)

  28. Fairly reward sound responses Model answer (checklist) Writers / markers who understand: - curriculum content - learner stage Ongoing item improvement Flinders SAQ strategies

  29. Are the claims justified? Do they measure anything different?(Maquire, Skakun, Triska 1997) What role do they have in a balanced assessment program? CONSTRUCTED RESPONSE ITEMS

  30. REQUIREMENTS Write to a blueprint Sample widely Increase level 2 and 3 questions Model answers CONSTRUCTED RESPONSE ITEMS

  31. THE END

  32. You see Rebecca and Robert who are residents at a local training centre for 27 young mentally handicapped patients. They are accompanied by their care helper. Rebecca and Robert have scabies. Discuss your management • communication• clinical management• health in the community• sensitivity to the mentally handicapped• use of extended team• ethical issues Deep/Surface Structures (Lockie et al 1990) SHORT ANSWER QUESTIONS - SAQ

  33. VALIDITY ConstructWright et al (1983) – Independent cognitive abilities PredictiveWass et al (2001) – correlate with clinical tests RELIABILITY Low to moderateDes Marchais and Vu (1996) Wakeford and Roberts (1984) GoodWright et al (1983) – marking schedules SHORT ANSWER QUESTIONS - SAQ

  34. FEASIBILITY • Marking • Higher order questions SHORT ANSWER QUESTIONS - SAQ

  35. • One clinical scenario – multiple questionsPAQ (Des Marchais and Vu 1996) – no sequence SAQ (Flinders) sequence • Level 1, 2 and 3 questions on one scenario • ReliabilityPAQ – low to moderate • Feasibility HYBRID APPROACHES

  36. Face validity Construct validity Model answers Higher order questions THE FLINDERS HYBRID MODEL

More Related