Absolute standards in progress testing: A growth curve analysis approach

1. Absolute standards in progress testing: A growth curve analysis approach Paul Lambe, Tom Gale, Steven Burr, Colin Ferguson & David Bristow paul.lambe@pms.ac.uk

2. The plan Progress Tests Norm-reference, Absolute reference (criterion) standard setting Study rationale Absolute and Growth Rate (AGR) standard setting Results Conclusions

3. Progress Test (PT) of Applied Medical Knowledge 5 year BMBS programme 20 PTs, 4 /year, set at level appropriate for a graduate, same tests sat by all students irrespective of study stage 125 multi choice questions (5 options & don’t know option, correct=1, incorrect = -0.25, don’t know = 0). Study year 5, PTs Absolute (criterion) standard Study years 1- 4, PTs Norm-referenced standard

4. Absolute & Norm-referenced standard setting Absolute (criterion referenced) based on judgements about the performance of individual examinees (Study Year 5) Norm-referenced (relative) standard based on judgements about groups (cohorts) of test takers (Study Years 1=4) Norm-referenced PT cut-score = approx. 1 standard deviation below cohort mean score approx. 15% fail

5. Study rationale Progress testing advantages:- multiple measures of continuous learning/ cumulative performance-based progression decisions/ early intervention for struggling students, curriculum & teaching quality feedback Norm-referenced progress testing disadvantages:- fixed fraction bound to fail, inter-cohort variation in ability, variance in test difficulty, student perceptions of unfairness & arbitrariness ‘A system … which combines norm-referenced standard setting with the undeniable attraction of pre-fixed cut-off scores representing the minimally required standards of mastery’ (Cohan-Schotanus J. Van der Vleuten C. A standard setting method with the best performing students as point of reference. Medical Teacher 2010;32:154-160)

6. Absolute standard setting Norm-referenced cut-score = cohort mean – 1 standard deviation, approx. 15% failure rate Imposition of absolute standard increased variance in failure rate 2% to 47% (Muijtjenset al Relative or Absolute standards in Assessing Medical Knowledge in Progress Tests. Advances in Health Sciences Education 1998;3:81-87) Absolute & Growth Rate (AGR) method: Muijtjens method + criterion of test-to-test rate of growth (Tan et al A distribution-free approach for comparing growth of knowledge. Journal of Educational Measurement 1994;Vol3, No1:51-65)

7. Progress tests in academic years 2009/10, 2010/11 & 2011/12

8. Results Muijtjens Absolute standard setting method:- mean failure rate = 15.09%, sd= 6.7, min = 2.4%, max = 35.7% Absolute & Growth Rate standard setting method:- mean failure rate = 8.16%, sd=4.8, min = 2.4%, max = 22.9% Over 3 academic years AGR reduced awards of Borderline/Unsatisfactory grade by 45%

9. Limitations Attrition rate low, but accept results may be biased by scores of students who dropped out and by those who participated in remediation. No account taken of the effect on student examination behaviour that an actual introduction of an absolute standard might have had. No ‘gold standard’ in standard setting & absolute fairness to every candidate impossible to attain

10. Conclusion AGR method:- produces failure rates comparable to norm referencing alone provides students with a stable & unambiguous level of competency to aim for at each stage of study across the medical degree course

11. Questions ?????

12. Results

13. Study year 4 in academic years 2009/10, 2010/11 and 2011/12, PUPSMD norm-referenced cut-scores & failure rates, compared with Muijtjens & AGR method