Practical Program Evaluation Follow Up: Working a Case Example

Practical Program Evaluation Follow Up: Working a Case Example PowerPoint PPT Presentation

  • Uploaded on
  • Presentation posted in: General

2. Today

Download Presentation

Practical Program Evaluation Follow Up: Working a Case Example

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

1. Practical Program Evaluation—Follow Up: Working a Case Example By: Thomas J. Chapel, Office of Workforce and Career Development Betty Apt, Division of STD Prevention Centers for Disease Control and Prevention

2. 2 Today… Review main points from overview by applying to a case example Choosing a component to evaluate Identifying stakeholders that matter most Developing simple logic models Choosing a strong evaluation focus Discuss November workshop

3. 3 Why Bother? Because We Want Programs to Improve Continuous Quality Improvement (CQI) cycle. Planning—What actions will best reach our goals and objectives. Performance measurement— How are we doing? Evaluation—Why are we doing well or poorly?

4. 4 What can Program Evaluation do? Increase Program Knowledge Direct Program Improvement Maximize Resources Provide Accountability

5. 5

6. 6 Two Pieces of Work Choose a program component to evaluate Apply evaluation steps and standards

7. 7 Choosing a Program Component to Evaluate Evaluation can focus on any/all parts of a large program like the immunization program

8. 8 Some Potential Criteria.. Where am I spending the most? Where am I concerned the most? Where are my big opportunities/new areas? Where are my big successes? Other… Other…

9. 9 Some Potential Components to Evaluate Immunization information systems Adolescent immunization Perinatal hepatitis B prevention AFIX VFC

10. 10 Today’s Case Example: AFIX—Reasons for Selecting Component Substantial program effort Research shows it works… But implemented in various ways… Want to ensure “our” approach is working

11. 11 Next—Apply Evaluation Steps and Standards

12. 12 Evaluation Step—Stakeholders Identify and engage stakeholders Describe the program component Focus the evaluation design Gather credible evidence Justify conclusions Ensure use and share lessons learned

13. 13 Which Stakeholders Matter Most? Who is Affected by the program? Involved in program operations? Intended users of evaluation findings?

14. 14 Some Key Stakeholders

15. 15 Payoffs for Involving Stakeholders Strong input on purpose and design of the evaluation “Reality check” on utility and feasibility of evaluation plan and methods. Increase credibility of evaluation findings. Increase likelihood that evaluation results will be acted upon.

16. 16 Involve Stakeholders from the Start! Potential stakeholder roles at all steps: Input on evaluation design Input on data collection methods Assist with data collection/analysis Outreach to the target population for the evaluation Interpretation and use of results

17. 17 Case Example: AFIX Stakeholders Program staff Clinical staff (clinic managers, nurses, physicians)

18. 18 Evaluation Step—Description Identify and engage stakeholders Describe the program component Focus the evaluation design Gather credible evidence Justify conclusions Ensure use and share lessons learned

19. 19 Seeking Consensus On… The activities and outcomes depicted The “roadmap” Which outcomes = program “success” This will facilitate future discussions of: How much progress on outcomes = program “success” Choices of data collection/analysis methods

20. 20 Describing the Program: Complete Logic Model

21. 21

22. 22

23. 23

24. 24 Case Example: AFIX Logic Model

25. 25

26. 26 AFIX Full Logic Model

27. 27

28. 28 Evaluation Step—Focusing the Evaluation Identify and engage stakeholders Describe the program component Focus the evaluation design Tailor evaluation to program and stakeholder’s needs Determine resources and personnel available Develop and prioritize evaluation questions Gather credible evidence Justify conclusions Ensure use and share lessons learned

29. 29 Evaluation Standards Help Choose Focus “Utility”— Which stakeholders/others want the evaluation findings? How/for what purpose will they use the findings? “Feasibility”— How reasonable are these questions given: How long program has been in existence; program intensity; data collection logistics/constraints.

30. 30 Generating and Prioritizing Evaluation Questions Utility: Questions which: Are important to stakeholders; Are consistent with program goals and objectives Will provide information that can be acted upon Feasibility: Questions which: Can be answered with available resources and within available timeframe Are consistent with program maturity and intensity Will be supported by program decision-makers.

31. 31 Setting Focus: Process and Outcome Evaluation Process Evaluation Determines if program activities are implemented as intended. Tracks who, what, when, and where program information. Provides feedback loop for program improvement. Conducted throughout project life. Outcome Evaluation Measures program effects; changes in target population’s knowledge, attitudes, self-efficacy, skills, intentions, behaviors; or organizational changes (e.g., policy adoption). Linked to process evaluation. Provides feedback loop for program improvement

32. 32 Case Example: Setting AFIX Focus Utility Purpose/User: Program staff Use: Determine if: AFIX feedback is actually implemented by providers. If not, determine barriers and challenges to use of feedback And, determine how to improve activities and followup to surmount identified barriers and challenges Feasibility Stage/Intensity: Well-established and multi-faceted program. Should be making these impacts if working Resources: [Assumption] Evaluation questions of interest have some ready data sources that would be credible for those interested in findings

33. 33 AFIX Logic Model—Focus

34. 34 AFIX—Sample Evaluation Questions Proposed focus: both process and outcomes Outcome: Are AFIX recommendations implemented by practices within 6 months of visit? (short term outcome) Process Factors that may be related to implementation/non-implementation of AFIX recommendations: Are AFIX staff appropriately trained? Are practice contacts the right people? What is the quality of the feedback? Are practices satisfied with feedback received?

35. 35

36. 36 But… These steps more likely to be done well because we did a good job on earlier steps!!!

37. Practical Program Evaluation Coming! This Fall…

38. 38 Fall Training Session Reaffirm/reinforce today’s points Work through remaining steps: Define indicators Identify data collection sources and methods Consider data analysis and interpretation Maximizing use

39. 39 Pre-Work for November Consider criteria for choosing a program component Give some preliminary thought to: Potential program components Key stakeholders Purpose/use of evaluation

40. Practical Program Evaluation Life Post-Session

41. 41 Helpful Publications @

42. 42 Helpful Resources NEW! Intro to Program Evaluation for PH Programs—A Self-Study Guide: Logic Model Sites Innovation Network: Harvard Family Research Project: University of Wisconsin-Extension: CDC/DASH: CDC/STD: Texts Kellogg Foundation Logic Model Development Guide: W.K. Kellogg Foundation Evaluation Resources: Rogers et al. Program Theory in Evaluation. New Directions Series: Jossey-Bass, Fall 2000 Chen, H. Theory-Driven Evaluations. Sage. 1990

43. 43 Community Tool Box

44. 44 This document can be found on the CDC website at:

  • Login