1 / 34

Program Evaluation and the Logic Model

Program Evaluation and the Logic Model. Research Methods for Public Administrators Dr. Gail Johnson. What to Evaluate?. Projects : a single intervention in one location or a single project implemented in several locations.

fergusonv
Download Presentation

Program Evaluation and the Logic Model

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Program Evaluation and the Logic Model Research Methods for Public Administrators Dr. Gail Johnson Dr. G. Johnson, www.researchdemystified.org

  2. What to Evaluate? • Projects: a single intervention in one location or a single project implemented in several locations. • Programs: an intervention comprised of various activities or projects which are intended to contribute to a common goal. • Organizations: multiple intervention programs delivered by an organization. Dr. G. Johnson, www.researchdemystified.org

  3. When to Evaluate? • Before program starts: • To improve design • During the implementation: • To improve implementation • Identify barriers to be removed • Lessons learned about implementation Dr. G. Johnson, www.researchdemystified.org

  4. When to Evaluate • Mid-term evaluation • Relevance, effectiveness, efficiency • Lessons learned: management tool • Impact evaluation • Either at the end of the project or a few years after the program has been operating: assessing a mature program • Can also look at: effectiveness, efficiency, early signs of impact and sustainability • Lessons learned: future projects Dr. G. Johnson, www.researchdemystified.org

  5. Why Is Evaluation Useful? • Feedback • Accountability • Learning • Improvement • Results • Testing underlying assumptions or theory • Funding decisions Dr. G. Johnson, www.researchdemystified.org

  6. Fear of Evaluation If evaluation is so useful, why do some people fear evaluation? Dr. G. Johnson, www.researchdemystified.org

  7. Evaluation Questions Dr. G. Johnson, www.researchdemystified.org

  8. Types of Evaluations • Auditing: accounting for money • Is the money being spent according to plan? • Efficiency and effectiveness. • Monitoring: measuring implementation and results • Is the intervention producing the intended results? • Process: measuring operations and service delivery • Are there problems in service delivery? Dr. G. Johnson, www.researchdemystified.org

  9. Types of Program Evaluations • Feasibility evaluations • Before the program begins • Intended to improve program design • Evaluability assessments • Assesses potential usefulness of the evaluation • Used to test out different strategies for conducting an evaluation • What is doable given the situation? Dr. G. Johnson, www.researchdemystified.org

  10. Evaluability Assessment • Helps to define the actual objectives, implementation and management of a program. • The actual objectives may differ from those initially planned. • Determines the coherence of the program: are goals, activities, program infrastructure linked? Dr. G. Johnson, www.researchdemystified.org

  11. Evaluability Assessment Key steps in the process: • Interview key program staff to actual program mission, goals, objectives and activities. • Site visits to observe and get a sense of what is going on. • May include interviews with key stakeholders. • Observe program delivery. Dr. G. Johnson, www.researchdemystified.org

  12. Evaluability Assessment • Reach agreement as to: • Whether to conduct the evaluation. • Scope and objectives of the evaluation. • The decision could be to not conduct the evaluation. Dr. G. Johnson, www.researchdemystified.org

  13. Evaluability Assessment: Challenges • Key components of the program may not be well defined: • Lack of agreement on program objectives. • Lack clear, measurable indicators of performance and/or impact. • Target group may not be clearly defined. • The delivery system is poorly articulated. Dr. G. Johnson, www.researchdemystified.org

  14. Types of Program Evaluations • Formative evaluations • During implementation • Feedback about operations and processes • Used to make mid-course corrections Dr. G. Johnson, www.researchdemystified.org

  15. Definition: Performance Monitoring • Performance monitoring: • the continuous process of collecting and analyzing data to compare how well a project, program or policy is being implemented against expected results. • Traditional: • focus on inputs, activities and outputs. Dr. G. Johnson, www.researchdemystified.org

  16. Types of Evaluation:Monitoring • On-going review: • On-time • On-budget • On-target • Linked with on-going management • Measured against established baselines • Indicators of progress toward targets Dr. G. Johnson, www.researchdemystified.org

  17. Types of Program Evaluations • Summative Evaluations • At the end of the program or after the program has been running long enough to achieve its goals (with “mature” programs) • Identify lessons learned • Other issues: unintended outcomes, program sustainability, program efficiency, costs and benefits • Sometimes called impact evaluations and ex-post evaluations Dr. G. Johnson, www.researchdemystified.org

  18. Program Evaluation • Summative Evaluation Question: Do Public Programs Work? • Implied cause-effect relationship • Did the program cause a desired outcome? • Performance-based: Focus on outcomes, results, impacts, goal achievement. Dr. G. Johnson, www.researchdemystified.org

  19. Differences • Formative Evaluations • Project Monitoring • Early Years of Implementation • Key Question: • Are we doing things right? • Have we hired the right people with the right skills? • Have we marketed the program effectively? • Have we met our strategic objectives? • Have we spent our money according to our plan? Dr. G. Johnson, www.researchdemystified.org

  20. Differences • Summative Evaluations • Measuring Results or Impacts • A longer time before results or impacts are visible • Key Question: • Are we doing the right thing? • This gets back to the theory or underlying assumptions of the program: • We can do an excellent job at training people but if the problem is not about the larger structural economic issues, a training program, no matter how well implemented, may show little result. Dr. G. Johnson, www.researchdemystified.org

  21. Working with Models • Visualize a program in context • Systems approach, within an environment • Identify the relationships between various components • Identify cause and effect • Identify key assumptions Dr. G. Johnson, www.researchdemystified.org

  22. Models: Cause and Effect: Did the program cause something to happen? Education Employment Dr. G. Johnson, www.researchdemystified.org

  23. Hierarchy of Objectives Reduced Poverty Improved Quality of Life Increased Income Job Training Unemployed Dr. G. Johnson, www.researchdemystified.org

  24. Logic Models • The focus is on results or impacts rather than inputs and activities • We are not training people just for the sake of training people • We believe if we train the chronically unemployed, then there quality of life will be improved and poverty will decrease. • Our goal is to reduce poverty • Also called Program Outcome Model or Measuring for Results Dr. G. Johnson, www.researchdemystified.org

  25. Logic Model Impact Inputs Outcomes Outputs Activities Dr. G. Johnson, www.researchdemystified.org

  26. Elements of the Logic Model Inputs: what resources are used • University inputs: budget, number of faculty, number of staff, number of buildings, number of classrooms • Activities: what the program does • University activities: teaching, research, and service Dr. G. Johnson, www.researchdemystified.org

  27. Elements of the Logic Model • Outputs: the services or products produced • University outputs: number of students that graduate, number of articles and books published by faculty • Outcomes: what happened: immediate results • Graduates are sought after, get good jobs, active alumni who donate big bucks • Faculty well-known, obtain big grants, enhance rating of university Dr. G. Johnson, www.researchdemystified.org

  28. Elements of the Logic Model • Impacts: the “so what.” Larger, long term results, usually tied to program goals. • A more informed and engaged citizenry, preserves democratic institutions, future leaders. Faculty research contributes to knowledge. Dr. G. Johnson, www.researchdemystified.org

  29. Logic Model Logical Connections: • Inputs to do activities • Activities lead to outputs • Outputs lead to one or more outcomes • Outcomes lead to impacts Dr. G. Johnson, www.researchdemystified.org

  30. Logic Model: Training Program Inputs Activities Outputs Outcomes Impacts • Training Programs • Dress for success coaching • Interview coaching • Resume assistance • Products • Number of graduates per training session • % graduate rate • Benefits changes • Increased skills • % Obtain jobs • % Obtain high paying, quality jobs • Increased self-esteem • Goals • Increased income • Self-sufficiency • Family stability • Reduction in poverty • Resources • money • staff • Supplies • mentors Dr. G. Johnson, www.researchdemystified.org

  31. Takeaway Lesson • Program evaluation takes many forms but all follow the same research planning process. • Evaluation of programs have shifted from reporting inputs and activities to attempting to measure results: the difference the program actually made. Dr. G. Johnson, www.researchdemystified.org

  32. Takeaway Lesson • Measuring results is harder than it appears. • Program goals and objectives may be fuzzy. • Sites may vary in how they have implemented the program. • It takes money to collect and analyze data. • Results may not be observable for many years. • The operating environment may make it hard to see results. Dr. G. Johnson, www.researchdemystified.org

  33. Takeaway Lesson • Measuring results is harder than it appears. • But there is much that can be learned from engaging in this process and doing the best job possible. • Remember: do not quickly conclude a program does not work just because you cannot measure the result. The research tools available may not be up to the job. Dr. G. Johnson, www.researchdemystified.org

  34. Creative Commons • This powerpoint is meant to be used and shared with attribution • Please provide feedback • If you make changes, please share freely and send me a copy of changes: • Johnsong62@gmail.com • Visit www.creativecommons.org for more information Dr. G. Johnson, www.researchdemystified.org

More Related