Criteria for Good Assessment John Norcini, Ph.D.
Overview • Some background • What is a ‘test’ or assessment? • How does assessment affect learners? • How are assessments changing? • Who cares about assessment? • Characteristics of good assessment • How do they relate to purpose? • How do they relate to the stakeholders?
What is a test or assessment? • Two definitions of ‘test’ • ‘To discover the worth of something by trial’ • Summative, evaluation • ‘To improve the quality of something by trial’ • Formative, assessment • For today, a ‘test’ or ‘assessment’ is any measurement process • Formative or summative • These are central to • Understanding the importance of assessment • Understanding the application of assessment • Identifying criteria for good assessment
How does assessment affect learners? Assessment is a powerful tool in education Motivates students to learn Directs learning by signaling what it important Unintended effects when there is a mismatch between curriculum and assessment Ensures that learning has occurred Creates learning
How are assessments changing? Past “That part of eternity with some small fraction of which we have a slight and regrettable acquaintance.” A Bierce Future “That period of time in which our affairs prosper, our friends are true, and our happiness is assured.” A Bierce Authenticity Active person/patient involvement More structured workplace training Life-long learning and quality improvement • Little realism • Little person/patient involvement • Lack of structure during workplace training • Episodic and infrequent continuing education
Who cares about assessment? • Variety of stakeholders • Students • Key to learning and demonstration of competence • Teachers and individual/collective educational institutions • Accountability and self-assessment • Person/patients and healthcare systems • Accountability and efficiency • Regulatory bodies • Accountability and maintenance of competence
Characteristics of good assessment • No single set of criteria apply equally to all assessment situations • Different purposes may lead to different values for the criteria • Different stakeholders and different mixes of stakeholders may value the criteria differently • Ottawa conference produced a set of criteria and how they are modified by purpose and stakeholders
Characteristics • Validity or coherence • Is there a body of evidence that ‘hangs together’ and supports the use of assessment for a particular purpose • Is a property of the inferences drawn from a test, not the test itself • Is a matter of degree • Requires the ongoing collection of data “In seeking absolute truth we aim at the unattainable, and must be content with finding broken portions.” William Osler, 1889
Characteristics • Reliability or reproducibility • Will scores be the same if examinees are reassessed? • History of reliability • Test-retest reliability • Alternate form reliability • Split-half reliability • Reliability index “Arithmetic…is a bad master, but it is a good servant.” Anonymous Lancet, 1921
Characteristics • Equivalence • Different versions of an assessment yield equivalent scores or decisions • A challenge for assessment in the workplace • Catalytic effect • The assessment provides results and feedback in a fashion that enhances learning • A requirement for formative assessment
Characteristics • Educational effect • Theassessment motivates those who take it to prepare in a fashion that has educationalbenefit • How do students prepare for the test? • Acceptability • Stakeholders find the assessment process and results to be credible • Feasibility • The test is practical, realistic, and sensible, given the circumstances and context
Summary • Characteristics of good assessment • Validity, reliability, equivalence, catalytic effect, educational effect, acceptability, feasibility • Influenced by the purpose • Formative and summative • Influenced by the stakeholders • Learners, teachers-institutions, patients-healthcare system, regulators