Evaluating Program Quality: Criteria and Methodologies

1. A Close Examination of Policy-Relevant Education Evaluations: Criteria for Judging Quality Matthew Linick & Diane Fuselier-Thompson

2. Implicit vs. Explicit Criteria for Judgments of Program Quality • Explicit criteria for judging program quality: • Can be clearly discerned in the text. • “A successful program will display the following characteristics…” • Implicit criteria for judging program quality: • Can be inferred by research questions. • “We will measure various aspects of the program…”

3. Methodologies of Research Reports • Types of Studies: • Impact and Outcome reports (14) • Implementation reports (12) • Methodologies: • Mixed Method: Interview and Survey (17) • Qualitative: Interview and/or focus group (6), Observation (5) • Quantitative: RCT (4), Quasi-experiment (6), Comparative Statistical Analysis (4)

4. Explicit Statements of Criteria for judging program quality • Explicit criteria were primarily included in implementation and outcome evaluations • When reports included explicit criteria, program quality was judged along methodological standards • Statistical significance in quantitative studies • Logic Model often used as rubric in implementation studies • Most evaluation reports refrain from making actual judgments of program quality • Authors tend to be uncritical of the evaluated program • Evaluations tend to report findings in lieu of making judgments

5. Examples of Explicit Criteria used during Program Evaluations • Implementation evaluation: ‘Ending Violence in Schools: A Study of Morton North High School’ • Logic model used as rubric • Evaluators constructed logic model based on relevant research and used this model to evaluate the implementation of violence prevention approaches used by the school • Impact evaluation: ‘Start Reading: Impact Study’ • Statistical significance used as explicit criteria for judging program • Statistically detectable differences between treatment and control schools in using a regression discontinuity • student reading achievement • classroom reading instructional practices • student time engaged with print

6. Explicit Statements of Criteria for judging program quality • Few explicit statements of criteria: • 9 of 31 reports have explicit statements • Explicit criteria are stated more often when the program is deemed to be successful • 6 of 9 reports with explicit statements were found to be successful

7. Implicit Statements of Criteria for judging program quality • Frequently Provided as a basis for judging program quality: • Statistical significance was often set as a goal of a research model attempting to estimate the positive impact of a program. • Research questions were used to establish the goal of the study, but the questions often did not contain criteria for making judgments. • Program goals were often referenced as the desired outcomes of the stakeholders or clients, but evaluators usually avoided such statements.

8. Example of Implicit Criteria used during Program Evaluations • Outcomes evaluation: ‘Extended School Day Program’ • Evaluators framed evaluation questions as research questions • What are the outcomes for students, teachers, and schools in this program? • What were the effects on test scores, attendance, teacher attitudes, etc.?

9. Implicit Statements of Criteria for judging program quality • Many of the reports imply that stakeholder expectations are a guiding principle for program ‘quality’. • Implicit Criteria for Program Quality in each of the 31 reviewed reports. • Not easily discernable (found in discussion of results in 20/31 reports) • Implied criteria tied to stakeholder expectations (25/31 reports)

10. Implicit Statements of Criteria in Reports • Examining the Program/Quality Criteria/Methodology. • Implicit criteria reflects stakeholders’ desired outcomes. • Desired outcomes influence methodological choices. • Methodological choices influence the criteria used to judge program quality.

11. How do Statements of Criteria relate to Methodology Used?

12. Questions, Comments, or Praise? • Contact Information: • Matt Linick • mlinic1@illinois.edu • Diane R. Fuselier-Thompson • diat@illinois.edu