Designing the evaluation
Download
1 / 21

Designing the Evaluation - PowerPoint PPT Presentation


  • 113 Views
  • Uploaded on

Designing the Evaluation. Sonal R. Doshi MS, MPH. Program Evaluator CDC/OSTLTS/DPHPI TB Education, Training, and Evaluation Lights, Camera, Action: Setting the Stage for TB Elimination 18 September 2012. National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Designing the Evaluation' - luella


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Designing the evaluation
Designing the Evaluation

Sonal R. Doshi MS, MPH

Program Evaluator

CDC/OSTLTS/DPHPI

TB Education, Training, and Evaluation

Lights, Camera, Action: Setting the Stage for TB Elimination

18 September 2012

National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention

Division of Tuberculosis Elimination


Session outline
Session Outline

  • Introduction

    • Session objectives

  • Background

    • What is evaluation?

    • Research vs. evaluation

    • CDC Evaluation Framework

  • What is evaluation design?

    • Considerations for determining evaluation design

    • Type of evaluation designs

  • Exercise

  • Wrap-up



Session objectives
Session Objectives

  • Identify key elements for determining the evaluation design.

  • Describe the difference between process and outcome evaluation and other evaluation designs



What is evaluation
What is Evaluation?

  • “the process of determining the merit, worth, or value of something, or the product of that process.” Michael Scriven

  • “Evaluation is a field that applies systematic inquiry to help improve programs, products, and personnel, as well as the human actions associated with them.” *

* AEA guiding principles - http://www.eval.org/aea09.eptf.policy.handouts.pdf


Research vs. Evaluation

SystematicMethods

Evaluation

Research

  • Knowledge intended for use

  • Program- or funder-derived questions

  • Judgmental quality

  • Action setting

  • Role conflicts

  • Often not published

  • Multiple allegiances

  • Production of generalizable knowledge

  • Researcher-derived questions

  • Paradigm stance

  • More controlled setting

  • Clearer role

  • Published

  • Clearer allegiance

7


“Research seeks to prove, evaluation seeks to improve…”

M.Q. Patton


Cdc framework for program evaluation
CDC Framework for Program Evaluation

MMWR, 1999. Framework for Program Evaluation in Public Health.



Evaluation and evaluation design
Evaluation and Evaluation Design

  • To improve program…

  • Art of evaluation design*

    • An “…exercise of the dramatic imagination” – L.J. Cronbach (pg 249)

    • Involves creating a design that is appropriate for a specific situation and particluar action or policy-making context

    • There is no single best plan for an an evaluation, not even for an inquiry into a particular program, at a particular time, with a particular budget

    • Any given design is necessarily an interplay of resources, possibilities, creativity, and personal judgements by the people involved

* From Patton MQ. Utilization-focused evaluation: the new century text. 1997. Sage Publications, Inc.


Types of evaluation design
Types of Evaluation Design

  • Many different ways to categorize evaluation designs

  • Examples

    • Randomized or true experimental/quasi-experimental/non-experiments

    • Summative/formative

    • Process/outcomes

    • Economic – CEA, CBA, and CUA

    • Mixed methods


Considerations for evaluation design
Considerations for Evaluation Design

  • Determine expected uses by intended users

    • Judgement oriented

    • Facilitating improvements

    • Generating knowledge

  • Have a clear understanding of what, if any, decisions will be influenced by the evaluation

  • Reflect on the program evaluation standards

    • Utility

    • Feasibility

    • Propriety

    • Accuracy

* Citations, references, and credits – Myriad Pro, 11pt


Considerations for evaluation design cont
Considerations for Evaluation Design, cont.

  • Considerations from “Real World Evaluation”*

    • Understand clients information needs.

    • Understand the dynamics and timetable of the decision-making process.

    • Define the program theory on which the program is based in close collaboration with key stakeholders.

    • Identify Budget, time, and data constraints and prioritize their importance,.

    • Understand political context.

  • Answer these questions**

    • What is the purpose of the evaluation?

    • How will the information be used?

    • What will we know after the evaluation that we don’t know now?

    • What actions will we be able to take based on evaluation findings?

* Bamberger M, Rugh J, Mabry L. RealWorld evaluation: working under budget, time, data, and political constraints. 2006. Sage Publications, Inc.

** From Patton MQ. Utilization-focused evaluation: the new century text. 1997. Sage Publications, Inc.



Evaluation questions
Evaluation Questions

  • What are the barriers to successful tuberculosis contact investigations in Utah?

  • Did implementation of the TB cohort review process as a program evaluation tool improve treatment and contact elicitation outcomes?



Take aways
Take Aways

  • This is evaluation; not research

  • Be clear on the evaluation question you are trying to answer

  • Be clear on the intended use of the evaluation

  • Be realistic

  • Others??


Thank you
Thank-you!

For more information please contact Centers for Disease Control and Prevention

1600 Clifton Road NE, Atlanta, GA 30333

Telephone: 1-800-CDC-INFO (232-4636)/TTY: 1-888-232-6348

E-mail: cdcinfo@cdc.gov Web: http://www.cdc.gov

The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.

National Center for HIV/AIDS, Viral Hepatitis, STD , and TB Prevention

Division of Tuberculosis Elimination


Designing the evaluation1
Designing the Evaluation

Sonal R. Doshi MS, MPH

Program Evaluator

CDC/OSTLTS/DPHPI

sdoshi@cdc.gov

TB Education, Training, and Evaluation

Lights, Camera, Action: Setting the Stage for TB Elimination

18 September 2012

National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention

Division of Tuberculosis Elimination