evaluation 101 a workshop n.
Skip this Video
Loading SlideShow in 5 Seconds..
Evaluation 101: A Workshop PowerPoint Presentation
Download Presentation
Evaluation 101: A Workshop

Loading in 2 Seconds...

play fullscreen
1 / 20

Evaluation 101: A Workshop - PowerPoint PPT Presentation

  • Uploaded on

Evaluation 101: A Workshop. Overview, Concepts and Application. Today:. Review some definitions Talk about practical concepts for AETC evaluation Discuss a useful framework Apply these concepts and the framework to your programs Identify other evaluation resources.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

Evaluation 101: A Workshop

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
evaluation 101 a workshop

Evaluation 101: A Workshop

Overview, Concepts and Application

  • Review some definitions
  • Talk about practical concepts for AETC evaluation
  • Discuss a useful framework
  • Apply these concepts and the framework to your programs
  • Identify other evaluation resources
definition rossi freeman and lipsey
Definition: Rossi, Freeman and Lipsey
  • “Program evaluation is the use of social research procedures to systematically investigate the effectiveness of … programs.”
concepts rossi et al s 5 program domains
Concepts: Rossi et al’s 5 program domains
  • Program evaluation typically involves assessment of one or more of the five program domains:
      • The need for the program
      • Design of the program
      • Program implementation and service delivery
      • Program impact or outcomes
      • Program efficiency
concepts chlemisky s perspectives
Concepts: Chlemisky’s Perspectives
  • Accountability: Information collected for decision makers (emphasis: outcomes)
  • Developmental: Information collected to improve performance (emphasis: process measures)
  • Knowledge: Information collected in the interest of generating understanding and explanation (emphasis on processes and outcomes).
scriven and context
Scriven and context
  • Evaluation has two arms:
    • Data gathering
    • Contextualizing results
some definitions
Some definitions…
  • Process evaluation:
    • Addresses questions about how well the program is functioning
    • Is useful for diagnosing outcome
    • Is critical in quality improvement
some definitions1
Some definitions…
  • Key questions in process evaluation:
    • Who is served?
    • What activities or services are provided?
    • Where is the program held?
    • When and how long?
some definitions2
Some definitions…
  • Outcome evaluation:
    • Gauges the extent to which a program produces the intended improvements it addresses
    • Addresses effectiveness, goal attainment and unintended outcomes
    • Is critical in quality improvement
some definitions3
Some definitions…
  • Key questions in outcome evaluation:
    • To what degree did the desired change(s) occur?
  • Outcomes can be initial, intermediate or longer-term
  • Outcomes can be measured at the patient-, provider-, organization or system level.
some definitions4
Some definitions…
  • Impact is sometimes used synonymous to “outcome.”
  • Impact is perhaps better defined as a longer-term outcome. For clinical training programs, impacts may be improved patient outcomes.
some definitions5
Some definitions…
  • Indicators or measures are the observable and measurable data that are used to track a program’s progress in achieving its goals.
  • Monitoring (program or outcome monitoring, for example) refers to on-going measurement activity
some definitions6
Some definitions…
  • CQI/QM:
    • HAB’s definition of quality: The degree to which a health or social service meets or exceeds established professional standards and user expectations
some tools for planning
Some tools for planning…
  • United Way’s Outcomes Manual
  • HRSA’s Quality Improvement Handbook
  • W.K. Kellogg Foundation Evaluation Handbook
some methods
Some methods…
  • Identify some Quantitative Methods
  • Identify some Qualitative Methods
  • Which is best, qual or quant?
a framework for aetc evaluation kirkpatrick
A framework for AETC evaluation: Kirkpatrick
  • Measure Reaction
  • Measure Learning
  • Measure Behavior
  • Measure Results
  • Identify some ways to measure each in an AETC training setting
application to your program
Application to Your Program:
  • Identify Program Goals
  • For each goal:
    • Identify Process Objectives
    • Identify Outcome Objectives
  • For each objective:
    • Identify Indicators
    • Identify Data Source
    • Plan Data Collection
    • Plan Data Analysis
recommended reading
Recommended Reading
  • Definitely:
    • Kirkpatrick
    • Cruz the American Journal Evaluation (the Journal of the American Evaluators’ Association)
  • If you have time:
    • Chelimsky
    • Rossi
  • If you want a good read:
    • Hunt
  • Table 1.1 from Chapter 1 in Chelimsky: The Coming Transformations in Evaluations
  • Exhibit 2-P An Australian Teams’ Ten Step Approach to Program Evaluation (p. 75 in Rossi et al)
  • Chapters 1-3 in Kirkpatrick
  • American Evaluators’ Association Guiding Principles for Evaluators
  • Isenberg’s CQI Program Handout