USING A LOGIC MODEL FRAMEWORK FOR PROGRAM PLANNING AND EVALUATION

1. USING A LOGIC MODEL FRAMEWORK FOR PROGRAM PLANNING AND EVALUATION Judith Hager Belfiori, MA, MPH Brianna Gass, MPH Family Health Outcomes Project (FHOP) California Conference on Childhood Injury Control October 5, 2005

2. This Session • Provide rationale / benefits of using a logic model framework • Describe a logic model framework and its components • Present a logic model example • Break into groups to critique and modify a program logic model

3. Why Use a Logic Model? • To guide program development and evaluation • To assess the potential for demonstrating the intervention yields results • To assist in identifying problems in program design or implementation, so that they may be addressed • Often required

4. Use for Planning and Implementation • A tool for program development • Provides to program funders and implementers a common understanding of the program and its intended outcomes • Promotes team building • Assists focused program management and problem solving

5. Use for Evaluation: • Depicts how the program is designed to work and aids assessment of whether it is logical • Useful to assess whether the program is operating as designed • Useful to communicate how a program works • Assists with identification of important evaluation questions

6. Tool only as useful as you make it! • It is a framework that relies on “critically” thinking through logic and relationships to develop your program model—only as good as the user • It is a representation—program implementers job to assure it becomes reality • Depicts your theory of change – It does not replace a detailed implementation plan

7. A Logic Model Framework • Organizes information into a visual sequence • Depicts how a program works to achieve its intended outcomes, shows relationships • May be a flow chart, table, diagram, etc • Has common elements

8. A Logic Model Can Look Like:

9. When Is a Program Logic Model Useful? Needs Assessment Problem Analysis ProgramDevelopmentLogic Model Evaluation Logic Model

10. Scope of A Logic Model • Statewide • Community (e.g., a Coalition of agencies) • Individual program • Single intervention

11. Using A Logic Model Framework to Assist Program Planning • Articulate the problem • Identify objectives • Determine intervention points using research, best practices, experts, local experience • Identify intervention strategies and target groups • During this process, use the framework working backwards from objectives

12. Developing the Logic Model

13. Elements of the Logic Model • Inputs (resources) • Outputs (activities, units of service, persons served) • Outcomes (short, intermediate and long-term) • Some models also include other elements such as problem statement, assumptions, environment

14. A graphic representation that shows logical relationships between inputs, outputs and outcomes of a program Elements of the Model: Problem statement Inputs Outputs Outcomes Assumptions Environment Today, we will use the UWEX* Logic Model Framework * Ellen Taylor-Powell, University of Wisconsin - Extension

15. The Logic Model Framework Program P R O B L E M INPUTS OUTPUTS OUTCOMES ASSUMPTIONS 1. 2. 3. ENVIRONMENT (external factors that influence outcomes)

16. The Problem Statement Crucial! Ask the following: • What does the program seek to change? • Long Term? • Program Life?

17. Program Components INPUTS OUTPUTS OUTCOMES • Resources Short term Inter- mediate Activities Participation Long term What is done Who is reached What is invested With what results

18. INPUTS STAFF (special requirements) MONEY LOCATION VOLUNTEERS PARTNERS EQUIPMENT TECHNOLOGY

19. OUTPUTS What program does Who it reaches ACTIVITIES Treatment Classes Counseling Skill development Case management Curriculum design Trainings Conferences PARTICIPATION Participants Providers Policy makers

20. PROGRAM OUTCOMES What results for individuals, agencies, communities? SHORT Learning Awareness Knowledge Attitudes Skills Opinions Aspirations Motivation INTERMEDIATE Action Behavior Practice Decisions Policies Systems change LONG-TERM Conditions Mortality Morbidity Quality of Life Environmental

21. PROGRAM OUTCOMES What results for individuals, agencies, communities..… SHORT Learning / Program process Materials Services Awareness Knowledge Attitudes Skills Opinion Aspirations Motivation INTERMEDIATE Action Behavior Practice Decisions Policies Systems change LONG-TERM Conditions Mortality Morbidity Quality of Life Environmental

22. Working Backwards from Objectives Problem: Proper use of Infant Car Seats INPUTS OUTPUTS OUTCOMES Parents increase knowledge of infant car seat use Design parent ed. curriculum Staff Reduced rates of infant injuries due to auto accidents Parents use improved skills Targeted parents attend Money Provide 6 training sessions Parents increase skill in use of car seat Partners

23. Assumptions and Environmental Factors • Assumptions Proven theories (research literature/best practices Promising program results Experience Expected conditions • Environmental/External Factors What, besides program interventions, can account for program results?

24. Risks to Teen Drivers • 1 in 5 new drivers experiences a collision during first year • Highest risk is during 1st month of driving • 6 months on--risk drops to twice the risk of adults until 3-4 years later • Developmental issues: teens tend to overestimate their abilities and misperceive risky situations • Environmental influences: media, peers, parents • Inexperience

25. Graduated Drivers License (GDL) Place restrictions on driving to minimize risky situations while allowing new driver to gain critical experience Highest risk situations: • Teen/peer passengers • Night driving

26. Factors that Reduce Affect of GDL • Night restrictions start too late • Limits on passengers still allow for one • Exceptions made for work/activities • Lack of DMV involvement • Low compliance/enforcement • Some parents placed restrictions prior to implementation of GDL

27. Factors that increase GDL affect • Conditional advancement • Education about GDL and its rationale • Parent involvement • Type of restrictions: e.g., earlier restrictions on night driving Today's example: Increasing parent involvement to achieve higher impact of GDL

28. Let’s Practice

29. Using the Logic Model • Develop the logic model • Assess The Logic: Do the elements relate to each other and the whole? • Determine what will be evaluated

30. Logical Linkages: ExampleSeries of If Then RelationshipsProblem: Inadequate Parental Restrictions on Teen Drivers IFTHENIFTHENIFTHENIFTHEN Decrease hours/ circum-stances teen drivers are at risk of accident Program invests time & money Design Parent Education curriculum Parents increase knowledge/motivation Parents increase restrictions placed on teen drivers INPUTS OUTPUTS OUTCOMES

31. Logical Linkages: ExampleSeries of If Then RelationshipsProblem: Improper use of infant car seat IFTHENIFTHENIFTHENIFTHEN Program invests time & money Design infant car seat use curriculum Parents increase knowledge/skills Parents increase proper use of infant car seat Decrease rates of infant injuries/mortality INPUTS OUTPUTS OUTCOMES

32. Example Problem: Improper Use of Infant Car Seats INPUTS OUTPUTS OUTCOMES Parents increase knowledge of infant car seat use Design parent ed. curriculum Staff Reduced rates of infant injuries due to auto accidents Parents use improved skills Targeted parents attend Money Provide 6 training sessions Parents increase skill in use of car seat Partners

33. Assessing Program Logic:Basic Questions • How well conceptualized is the program? • Is it grounded in research, best or promising practices? • Does it address specific causes/risks of the problem? • Are the outcomes clearly related to the program’s interventions? • Is the program reasonable, feasible to implement? Is the program logic valid?

34. Using the Logic Model • Develop the logic model • Assess the logic • Determine What Will Be Evaluated

35. Evaluation Problem: Inputs Strategy Results Evidence

36. PLANNING INPUTS OUTPUTS OUTCOMES • Resources Short Inter- mediate Long term Activities Participation EVALUATION

37. Evaluating a Program Using a Logic Model Framework: • Start with inputs and work toward outcomes • Assess problem statement/program relationship • Review the program’s theories of change / logic model • Is the theory valid? Supported by research / local experience? • Is the progression from inputs to outcomes logical? • If valid, develop objectives and measures and proceed with evaluation design

38. Questions: Data: Were sessions delivered? # of sessions delivered # of target parents attending session? demographics of parents Increase in know/ motivation? pre-post session survey Behavior change? Contracts Follow-up interviews Decrease in rates? from program records vs. comp group Using the Logic Model to Develop the Evaluation INPUTS OUTPUTS OUTCOMES Parents increase knowledge of their role Parent/ teen contracts Design parent ed. curriculum Staff Reduced rates of collisions involving teen drivers Targeted parents attend Money Parents increase motivation to restrict teen driving Provide 6 training sessions Parents report enforcing restrictions Partners EVALUATION: What do you want to know? What data do you need?

39. Need a Data Collection Plan (examples)

40. Review of Benefits ofUsing a Logic Model Framework: • Helps clarify program rationale and expectations • Identifies gaps in logic and uncertain assumptions • Summarizes complex program to communicate with stakeholders/others • Helps identify appropriate evaluation questions • Builds understanding and consensus (shared approach) • Focus is on achieving and demonstrating outcomes

41. Benefit: Can help to integrate: PLANNING AND EVALUATION INPUTS OUTPUTS OUTCOMES • Investments Short Inter- mediate Long term Activities Participation

42. Problem: Program: INPUTS OUTPUTS OUTCOMES Resources Activities Participation Short Intermediate Long-Term PROGRAM ASSUMPTIONS ENVIRONMENTAL FACTORS Logic Model: WORKSHEETS

43. References: • Taylor-Powell, E., Jones, L., & Henert, E. (2002) Enhancing Program Performance with Logic Models. Retrieved September 1, 2005 from the University of Wisconsin-Extension website: http://www.uwex.edu/ces/pdande/ • Developing An Effective MCH Planning Process: A Guide for Local MCH Programs, 2nd edition, September 2003 FHOP Website and contact information http://www.ucsf.edu/fhop (415) 476-5283

44. Exercise Your Turn