A Close Examination of Policy-Relevant Education Evaluations: Criteria for Judging Quality
This presentation is the property of its rightful owner.
Sponsored Links
1 / 12

A Close Examination of Policy-Relevant Education Evaluations: Criteria for Judging Quality PowerPoint PPT Presentation


  • 44 Views
  • Uploaded on
  • Presentation posted in: General

A Close Examination of Policy-Relevant Education Evaluations: Criteria for Judging Quality Matthew Linick & Diane Fuselier -Thompson. Implicit vs. Explicit Criteria for Judgments of Program Quality. Explicit criteria for judging program quality: Can be clearly discerned in the text.

Download Presentation

A Close Examination of Policy-Relevant Education Evaluations: Criteria for Judging Quality

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


A close examination of policy relevant education evaluations criteria for judging quality

A Close Examination of Policy-Relevant Education Evaluations: Criteria for Judging Quality

Matthew Linick & Diane Fuselier-Thompson


A close examination of policy relevant education evaluations criteria for judging quality

Implicit vs. Explicit Criteria for Judgments of Program Quality

  • Explicit criteria for judging program quality:

    • Can be clearly discerned in the text.

    • “A successful program will display the following characteristics…”

  • Implicit criteria for judging program quality:

    • Can be inferred by research questions.

    • “We will measure various aspects of the program…”


A close examination of policy relevant education evaluations criteria for judging quality

Methodologies of Research Reports

  • Types of Studies:

    • Impact and Outcome reports (14)

    • Implementation reports (12)

  • Methodologies:

    • Mixed Method: Interview and Survey (17)

    • Qualitative: Interview and/or focus group (6), Observation (5)

    • Quantitative: RCT (4), Quasi-experiment (6), Comparative Statistical Analysis (4)


A close examination of policy relevant education evaluations criteria for judging quality

Explicit Statements of Criteria for judging program quality

  • Explicit criteria were primarily included in implementation and outcome evaluations

  • When reports included explicit criteria, program quality was judged along methodological standards

    • Statistical significance in quantitative studies

    • Logic Model often used as rubric in implementation studies

  • Most evaluation reports refrain from making actual judgments of program quality

    • Authors tend to be uncritical of the evaluated program

    • Evaluations tend to report findings in lieu of making judgments


A close examination of policy relevant education evaluations criteria for judging quality

Examples of Explicit Criteria used during Program Evaluations

  • Implementation evaluation: ‘Ending Violence in Schools: A Study of Morton North High School’

    • Logic model used as rubric

    • Evaluators constructed logic model based on relevant research and used this model to evaluate the implementation of violence prevention approaches used by the school

  • Impact evaluation: ‘Start Reading: Impact Study’

    • Statistical significance used as explicit criteria for judging program

    • Statistically detectable differences between treatment and control schools in using a regression discontinuity

      • student reading achievement

      • classroom reading instructional practices

      • student time engaged with print


A close examination of policy relevant education evaluations criteria for judging quality

Explicit Statements of Criteria for judging program quality

  • Few explicit statements of criteria:

    • 9 of 31 reports have explicit statements

  • Explicit criteria are stated more often when the program is deemed to be successful

    • 6 of 9 reports with explicit statements were found to be successful


A close examination of policy relevant education evaluations criteria for judging quality

Implicit Statements of Criteria for judging program quality

  • Frequently Provided as a basis for judging program quality:

    • Statistical significance was often set as a goal of a research model attempting to estimate the positive impact of a program.

    • Research questions were used to establish the goal of the study, but the questions often did not contain criteria for making judgments.

    • Program goals were often referenced as the desired outcomes of the stakeholders or clients, but evaluators usually avoided such statements.


A close examination of policy relevant education evaluations criteria for judging quality

Example of Implicit Criteria used during Program Evaluations

  • Outcomes evaluation: ‘Extended School Day Program’

    • Evaluators framed evaluation questions as research questions

      • What are the outcomes for students, teachers, and schools in this program?

      • What were the effects on test scores, attendance, teacher attitudes, etc.?


A close examination of policy relevant education evaluations criteria for judging quality

Implicit Statements of Criteria for judging program quality

  • Many of the reports imply that stakeholder expectations are a guiding principle for program ‘quality’.

  • Implicit Criteria for Program Quality in each of the 31 reviewed reports.

    • Not easily discernable (found in discussion of results in 20/31 reports)

    • Implied criteria tied to stakeholder expectations (25/31 reports)


A close examination of policy relevant education evaluations criteria for judging quality

Implicit Statements of Criteria in Reports

  • Examining the Program/Quality Criteria/Methodology.

  • Implicit criteria reflects stakeholders’ desired outcomes.

  • Desired outcomes influence methodological choices.

  • Methodological choices influence the criteria used to judge program quality.


A close examination of policy relevant education evaluations criteria for judging quality

  • How do Statements of Criteria relate to Methodology Used?


A close examination of policy relevant education evaluations criteria for judging quality

Questions, Comments, or Praise?

  • Contact Information:

  • Matt Linick

  • [email protected]

  • Diane R. Fuselier-Thompson

  • [email protected]


  • Login