slide1
Download
Skip this Video
Download Presentation
A Close Examination of Policy-Relevant Education Evaluations: Criteria for Judging Quality

Loading in 2 Seconds...

play fullscreen
1 / 12

A Close Examination of Policy-Relevant Education Evaluations: Criteria for Judging Quality - PowerPoint PPT Presentation


  • 68 Views
  • Uploaded on

A Close Examination of Policy-Relevant Education Evaluations: Criteria for Judging Quality Matthew Linick & Diane Fuselier -Thompson. Implicit vs. Explicit Criteria for Judgments of Program Quality. Explicit criteria for judging program quality: Can be clearly discerned in the text.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' A Close Examination of Policy-Relevant Education Evaluations: Criteria for Judging Quality' - susan


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

A Close Examination of Policy-Relevant Education Evaluations: Criteria for Judging Quality

Matthew Linick & Diane Fuselier-Thompson

slide2

Implicit vs. Explicit Criteria for Judgments of Program Quality

  • Explicit criteria for judging program quality:
    • Can be clearly discerned in the text.
    • “A successful program will display the following characteristics…”
  • Implicit criteria for judging program quality:
    • Can be inferred by research questions.
    • “We will measure various aspects of the program…”
slide3

Methodologies of Research Reports

  • Types of Studies:
    • Impact and Outcome reports (14)
    • Implementation reports (12)
  • Methodologies:
    • Mixed Method: Interview and Survey (17)
    • Qualitative: Interview and/or focus group (6), Observation (5)
    • Quantitative: RCT (4), Quasi-experiment (6), Comparative Statistical Analysis (4)
slide4

Explicit Statements of Criteria for judging program quality

  • Explicit criteria were primarily included in implementation and outcome evaluations
  • When reports included explicit criteria, program quality was judged along methodological standards
    • Statistical significance in quantitative studies
    • Logic Model often used as rubric in implementation studies
  • Most evaluation reports refrain from making actual judgments of program quality
    • Authors tend to be uncritical of the evaluated program
    • Evaluations tend to report findings in lieu of making judgments
slide5

Examples of Explicit Criteria used during Program Evaluations

  • Implementation evaluation: ‘Ending Violence in Schools: A Study of Morton North High School’
    • Logic model used as rubric
    • Evaluators constructed logic model based on relevant research and used this model to evaluate the implementation of violence prevention approaches used by the school
  • Impact evaluation: ‘Start Reading: Impact Study’
    • Statistical significance used as explicit criteria for judging program
    • Statistically detectable differences between treatment and control schools in using a regression discontinuity
      • student reading achievement
      • classroom reading instructional practices
      • student time engaged with print
slide6

Explicit Statements of Criteria for judging program quality

  • Few explicit statements of criteria:
    • 9 of 31 reports have explicit statements
  • Explicit criteria are stated more often when the program is deemed to be successful
    • 6 of 9 reports with explicit statements were found to be successful
slide7

Implicit Statements of Criteria for judging program quality

  • Frequently Provided as a basis for judging program quality:
    • Statistical significance was often set as a goal of a research model attempting to estimate the positive impact of a program.
    • Research questions were used to establish the goal of the study, but the questions often did not contain criteria for making judgments.
    • Program goals were often referenced as the desired outcomes of the stakeholders or clients, but evaluators usually avoided such statements.
slide8

Example of Implicit Criteria used during Program Evaluations

  • Outcomes evaluation: ‘Extended School Day Program’
    • Evaluators framed evaluation questions as research questions
      • What are the outcomes for students, teachers, and schools in this program?
      • What were the effects on test scores, attendance, teacher attitudes, etc.?
slide9

Implicit Statements of Criteria for judging program quality

  • Many of the reports imply that stakeholder expectations are a guiding principle for program ‘quality’.
  • Implicit Criteria for Program Quality in each of the 31 reviewed reports.
    • Not easily discernable (found in discussion of results in 20/31 reports)
    • Implied criteria tied to stakeholder expectations (25/31 reports)
slide10

Implicit Statements of Criteria in Reports

  • Examining the Program/Quality Criteria/Methodology.
  • Implicit criteria reflects stakeholders’ desired outcomes.
  • Desired outcomes influence methodological choices.
  • Methodological choices influence the criteria used to judge program quality.
slide12

Questions, Comments, or Praise?

ad