CONSTRUCTED RESPONSE ITEMS:
Download
1 / 39

DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University - PowerPoint PPT Presentation


  • 103 Views
  • Uploaded on

CONSTRUCTED RESPONSE ITEMS: MEQ & SAQ. DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003. Hong Kong International Consortium. CONSTRUCTED RESPONSE ITEMS. ASSUMPTIONS • Higher cognitive processes

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University' - melodie-foreman


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

CONSTRUCTED RESPONSE ITEMS:

MEQ & SAQ

DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE

School of Medicine, Flinders University

Train the Trainer WorkshopOctober 28, 2003

Hong Kong International Consortium


Constructed response items
CONSTRUCTED RESPONSE ITEMS

ASSUMPTIONS

• Higher cognitive processes

• Increased reliability through wider sampling

ARE THE ASSUMPTIONS JUSTIFIED?


Modified essay questions meqs
MODIFIED ESSAY QUESTIONS: MEQs

1970s – 1980s

RCGP (Hodgkin & Knox 1975)

Newcastle Australia (Feletti 1980, Feletti and Smith 1986)

Jefferson USA (Rabinowitz 1987, Rabinowitz and Hojat 1989)


Learning objectives, concepts, deep structures

Clinical scenario

7 – 10 questions

Sequential (information question information question ...)

Integrated (content, media – Rabinowitz 1985)

MODIFIED ESSAY QUESTIONS: MEQs


Higher cognitive order bloom 1956 buckwalter et al 1981
HIGHER COGNITIVE ORDER(Bloom 1956, Buckwalter et al 1981)

1. Knowledge Level 1 Recognition Recall

2. Comprehension Level 2 Comprehension Interpretation

3. Application

4. Analysis Level 3 Application5. Synthesis

6. Evaluation

(Irwin and Bamber 1982)

MODIFIED ESSAY QUESTIONS: MEQs


LEVEL 1 RECOGNITION AND RECALL

Know

• What is …..?

• List …..

• Identify …..

MODIFIED ESSAY QUESTIONS: MEQs


MODIFIED ESSAY QUESTIONS: MEQs

LEVEL 2 COMPREHENSION, INTERPRETATION

Understand

• Give reasons ….

• Explain ….

• Interpret ….


LEVEL 3 APPLICATION

Apply, Utilise, Judge

• How does X assist in explaining Y?

• What are the components of X?How are they related?How do they affect Y?

• How does XYZ affect treatment and management?

• Assess, judge X in terms of ….

MODIFIED ESSAY QUESTIONS: MEQs


Modified essay questions validity
MODIFIED ESSAY QUESTIONS – VALIDITY

•CONTENT(Blueprints, objectives, deep structures)

• CONSTRUCT(Level 1, 2, 3 questions)


Modified essay questions validity1
MODIFIED ESSAY QUESTIONS – VALIDITY

CONSTRUCTNorman et al 1987 Effect of educational level MCQ and MEQ not different skills UEQ may be a different skill Small effect of contextFeletti 1980More Level 1 and 2 in first year undergraduate Irwin and Bamber 1986More Level 3 in final year undergraduate


PREDICTIVE Irwin and Bamber 1986 Correlation with clinical examination 0.41Rabinowitz 1987 MCQ better predictor NBME I & IIRabinowitz and Hojat 1989 Higher correlation for MEQ than MCQ with overall clerkship ratings

MODIFIED ESSAY QUESTIONS – VALIDITY


Inter-rater reliability (model answers)

Norman et al 1987Inter-rater reliability satisfactory

Internal consistency (cumulative errors)

Feletti 1980Alpha(60) 0.57 – 0.91

Stratford and Pierce-Fenn 1985Alpha(60) 0.39

MODIFIED ESSAY QUESTIONS – RELIABILITY


Administration

Marking

Higher order questions

MODIFIED ESSAY QUESTIONS – FEASIBILITY


Model PBL process

Determine objectives

Construct scenario

MINI-CASES: THE FLINDERS MEQ


John Taylor (age 55) has been brought to the E.D. by his wife. John woke up this morning (one hour ago) with nausea and vertigo. He managed to walk to the bathroom but had to hold onto the furniture and walls to steady himself. He vomited when he reached the bathroom.

1. Give 3 hypotheses to account for John’s condition.

2. Explain the mechanisms which give rise to John’s condition.

3. What further information would you like to distinguish between hypotheses. Explain how this information will distinguish between hypotheses.

MINICASES: THE FLINDERS MEQ


• Provide further information wife. John woke up this morning (one hour ago) with nausea and vertigo. He managed to walk to the bathroom but had to hold onto the furniture and walls to steady himself. He vomited when he reached the bathroom.

• Cumulative error cues

• Continue scenario

MINI-CASES: THE FLINDERS MEQ


John has had no significant medical problems in the last few years. He spent the previous evening at home watching TV and did not drink any alcohol or take any other substance.

….. The left pupil is smaller than the right …..

….. John’s uvula deviates to the right …..

….. Movements of the left hand are clumsy.

MINICASES: THE FLINDERS MEQ


• Further questions years. He spent the previous evening at home watching TV and did not drink any alcohol or take any other substance.

• Continue unfolding scenario

• Page breaks, cumulative cues

• Conclude with SAQs (Level 2/3)

MINI-CASES: THE FLINDERS MEQ


Face validity years. He spent the previous evening at home watching TV and did not drink any alcohol or take any other substance.

Construct validity

Model answers

Higher order questions

MINI-CASES: THE FLINDERS MEQ


Reform of RCGP MEQ years. He spent the previous evening at home watching TV and did not drink any alcohol or take any other substance.– Swanson – Mullholand & McAleer

Increase number (10 – 12 / 2 hours)– Reflect reality

Increase independence

Single marked – schedules

SAQ

(Lockie et al 1990)

SHORT ANSWER QUESTIONS - SAQ


Evaluation the ability to make a value judgement in relation to relevant criteria

response years. He spent the previous evening at home watching TV and did not drink any alcohol or take any other substance.

contextual information + Q

criteria

Flinders SAQ strategies

Evaluation:“The ability to make a value-judgement in relation to relevant criteria”

  • Opportunity to target highest cognitive skills

    • application - analysis - synthesis - evaluation

Bloom 1956


response years. He spent the previous evening at home watching TV and did not drink any alcohol or take any other substance.

contextual information + Q

criteria

Targeting “evaluation” response

  • Provide multiple, detailed, relevant information

    • Patient

    • Problem

    • Clinical care setting

    • Your role


Example scenario information years. He spent the previous evening at home watching TV and did not drink any alcohol or take any other substance.

  • Ron Harkin is a 32 year-old father of two and self-employed gardener. He falls heavily. He is unable to stand and is in considerable pain. He is taken to hospital where you are the Intern in the Emergency Department. You note swelling and that his foot appears to be at an unusual angle to his leg.

  • A 7 year-old Aboriginal girl, Alice Mundi, presents with a mild fever and painful knee to a small rural hospital. Her mother has brought her 50km to the hospital. You are the GP called to see her.


response years. He spent the previous evening at home watching TV and did not drink any alcohol or take any other substance.

contextual information + Q

criteria

Targeting “evaluation” response

  • 2. Pose question requiring value-judgement

    • focus the response

    • assume criteria are understood

    • relate to what you would do

  • 3. Extend information / probe further (optional)


  • Example questions targeting “evaluation” years. He spent the previous evening at home watching TV and did not drink any alcohol or take any other substance.

    • Rationalising the diagnosis

    • What further information from the history do you require?

    • On what aspects of the physical examination will you

    • focus?

    • What investigations would you undertake to establish the

    • diagnosis?

    • What are your differential diagnoses? Which do you

    • consider most likely?


    Example questions targeting “evaluation” years. He spent the previous evening at home watching TV and did not drink any alcohol or take any other substance.

    • Formulating an appropriate management plan

    • What would be your immediate management goals?

    • Which treatments would you advise?

    • For what potential complications would you monitor?

    • What longer-term management strategy would you advise?

    • What advice would you give in relation to this new

    • development?


    Mr Dawes years. He spent the previous evening at home watching TV and did not drink any alcohol or take any other substance., a 58 yo man, presents to the Emergency Dept with a 24h history of vomiting and colicky abdominal pain. He has had no bowel movement for 24h. He had an appendicectomy at age 30 to remove a gangrenous appendix. On examination you find he has a distended abdomen with no focal tenderness. There is no palpable hernia. He has symptoms and signs consistent with dehydration of about 10%. Fluid resuscitation is commenced. A plain abdominal Xray shows distended loops of small bowel with air fluid levels and normal calibre large bowel.

    Q1. What is your most likely diagnosis for this patient?

    (2 marks)

    Q2. What possible causes of this would you consider?

    (3 marks)

    After 48h of conservative management, Mr Dawes’ condition is such that surgical intervention is warranted.

    Q3. What clinical indications would suggest to you that Mr Dawes needs surgery to resolve his problem at this time? (5 marks)


    Fairly reward sound responses years. He spent the previous evening at home watching TV and did not drink any alcohol or take any other substance.

    Model answer (checklist)

    Writers / markers who understand:

    - curriculum content

    - learner stage

    Ongoing item improvement

    Flinders SAQ strategies


    Are the claims justified? years. He spent the previous evening at home watching TV and did not drink any alcohol or take any other substance.

    Do they measure anything different?(Maquire, Skakun, Triska 1997)

    What role do they have in a balanced assessment program?

    CONSTRUCTED RESPONSE ITEMS


    REQUIREMENTS years. He spent the previous evening at home watching TV and did not drink any alcohol or take any other substance.

    Write to a blueprint

    Sample widely

    Increase level 2 and 3 questions

    Model answers

    CONSTRUCTED RESPONSE ITEMS


    THE END years. He spent the previous evening at home watching TV and did not drink any alcohol or take any other substance.


    You see Rebecca and Robert who are residents at a local training centre for 27 young mentally handicapped patients. They are accompanied by their care helper. Rebecca and Robert have scabies.

    Discuss your management

    • communication• clinical management• health in the community• sensitivity to the mentally handicapped• use of extended team• ethical issues

    Deep/Surface Structures

    (Lockie et al 1990)

    SHORT ANSWER QUESTIONS - SAQ


    VALIDITY training centre for 27 young mentally handicapped patients. They are accompanied by their care helper. Rebecca and Robert have scabies. ConstructWright et al (1983) – Independent cognitive abilities PredictiveWass et al (2001) – correlate with clinical tests

    RELIABILITY Low to moderateDes Marchais and Vu (1996) Wakeford and Roberts (1984) GoodWright et al (1983) – marking schedules

    SHORT ANSWER QUESTIONS - SAQ


    FEASIBILITY training centre for 27 young mentally handicapped patients. They are accompanied by their care helper. Rebecca and Robert have scabies.

    • Marking

    • Higher order questions

    SHORT ANSWER QUESTIONS - SAQ


    • One clinical scenario – multiple questions training centre for 27 young mentally handicapped patients. They are accompanied by their care helper. Rebecca and Robert have scabies.PAQ (Des Marchais and Vu 1996) – no sequence SAQ (Flinders) sequence

    • Level 1, 2 and 3 questions on one scenario

    • ReliabilityPAQ – low to moderate

    • Feasibility

    HYBRID APPROACHES


    Face validity training centre for 27 young mentally handicapped patients. They are accompanied by their care helper. Rebecca and Robert have scabies.

    Construct validity

    Model answers

    Higher order questions

    THE FLINDERS HYBRID MODEL


    ad