Program Evaluation Session 1. Karen V. Mann, PhD Professor Division of Medical Education Dalhousie University Harvard Macy Program for Health Professions Educators January 20, 2009. Objectives. To review elements of program evaluation
Karen V. Mann, PhD
Division of Medical Education
Harvard Macy Program for
Health Professions Educators
January 20, 2009
Reflection and analysis
Wilkes & Bligh, 1999.
The curriculum in action
The curriculum on paper
(planned or formal)
The curriculum: students’ experience
Fowell et al, 1999
Kirkpatrick’s hierarchy of levels of evaluation. Complexity of behavioural change increases as evaluation of intervention ascends the hierarchy.
Evaluationof results(transfer or impact on society)
Evaluation of behaviour(transfer of learning to workplace)
Evaluation of learning(knowledge or skills acquired)
Evaluation of reaction(satisfaction or happiness)
Kirkpatrick, 1967. In Hutchison, 1999.
4b. Benefits to
4a. Change in organizational
3. Behavioural change
2a. Acquisition of knowledge, skills
2b. Modification of attitudes, perceptions
1. Evaluation of reaction
A Critical review of Evaluations of Health Professions Education (2002)
(Freeth, Barr et al.)
Focus on improvement
Focus on demonstrating results
1. Is it learning? - individual and/or group?
2. Is it behaviour change? – individual, group, and/or organization?
3. Is it processes of teaching/ learning/ change?
- at individual, group, organization levels?
With a partner, discuss your own project regarding:
Methods of evaluation/ data collection approaches (using attached table)
Timing of measures
What are the factors that will influence the evaluation you conduct?
Purpose of the evaluation
Priorities for information
Limitations of information
Potential enabling factors
Questions should inform program planning and activities
Programs are not static
Evaluation is important to program accountability
Evaluation should assist understanding how a program produces the effects seen
Bennett J (2003). Evaluation methods in research. Continuum Research Methods Series. London UK: Continuum.
Bond SL, Boyd SE, Montgomery DL (1997). Taking stock: a practical guide to evaluating your own programs. Chapel Hill, NC: Horizon Research, Inc.
Creswell J (2002). Research design. Qualitative, quantitative and mixed methods approaches. Thousand Oaks CA: Sage.
Des Marchais JE, Bordage G (1998). Sustaining curricular change at Sherbrooke through external, formative program evaluations. Acad Med 73:494-503.
Fowell SL, Southgate LJ, Bligh JG (1999). Evaluating assessment: the missing link? Med Educ 33: 276-281.
Freeth D, Hammick M, Kippel I, Reeves S and Barr H. (2002). A critical review of evaluations of interprofessional education. The interprofessional education joint evaluation team. London: Learning and Teaching Support Network for Health Sciences and Practice.
Henry Gary T (2002). Choosing criteria to judge program success. Evaluation 8(2): 182-204.
Kern DE, Thomas PA, Howard DM, Bass EB (1998). Curriculum development in medical education. a six step approach. Step 6: evaluation and feedback. Baltimore MD: Johns Hopkins University Press, pp. 70-98.
Kirkpatrick DL (1994). Evaluating training programs. The four levels. San Francisco CA: Berrett-Koehler