Using Logic Models for Project Evaluation & Improvement

1. Using Logic Models for Project Evaluation & Improvement MeHAF Integration Initiative Grantee Web Seminar March & April, 2009 Presented by Mary Ann Scheirer Maryann@ScheirerConsulting.com

2. Why Are We Here? • What do you want to learn from this web seminar? • Remember, for 2009 grantees, your project’s Logic Model is due as part of your first 6-month report in August, 2009. • Be sure to review these materials just before you draft your own Logic Model!

3. Definition of “evaluation” – “The use of systematic data to aid users in their management of a targeted set of activities”

4. Planning Program Evaluation • Evaluation (collecting & using data) should be a continuing part of your project management. • When planning evaluation, one needs a clear description of what the project is – use a logic model!

5. Logic Models - Warm-Up Exercise • Think about a recipe for a favorite dish • How would you go about making that dish?

6. What is a Logic Model? • Picture/graphic of how a project or program works • A systematic, visual way to show the connections among parts of a project • Shows a ‘theory of change’ connecting what happens (activities) with outcomes

7. Format for a Logic Model Participants: Resources (Inputs) Activities (Outputs) Short-term Outcomes Long-term Outcomes Environmental Context

8. Why a Logic Model? A program is a set of activities designed to achieve change, which should be based on IF - THEN LOGIC. If my organization provides “X,” Then the target audience will “Y”

9. Why is a Logic Model Important?

10. How Can You Use a Logic Model? • For Project Planning • To understand & specify project elements • By involving stakeholders - agreement? • For Project Management • Monitoring & improving project implementation • Communicating & building consensus • For Project Evaluation • Shows assumptions in line of reasoning • Suggests measures needed for evaluation • For Communications • Writing proposals for further funding → sustainability

11. Simplified MeHAF Integration Grantee Logic Model Participants: NProviders and Npatients in XYZ health system • Activities • Create PC-BH team • Train providers • Screen/treat/ follow-up on patients • Short-term Outcomes • # providers in integ. team • # pts rec’ving integ. screening/ • treatment/f-up • # pts reporting integ./pt.-ctrd. care • Long-term Outcomes • Integrated care delivered consistently • X % of pts. w/ chronic diseases improve • Care is more pt.- centered • Resources • MeHAF funding • Evidence • Staff experience • Environmental Context • Fragmented service delivery Licensing, regulation, reimbursement • Workforce Shortages  Translation of Research to Practice

12. Resources are whatever the agency needs to deliver the program Usually are nouns Also called “inputs” Examples: Staff & Volunteers Time Money Space in agency Evidence Base Partners Materials & Technology Parts of a Logic Model - Resources

13. A summary of the actions or steps an organization takes to deliver a program. …what the projects does with its resources. …use verbs like provide counseling or serve meals. ... show what you do. Examples: Train providers Recruit clients - outreach Deliver MH/SA/PC treatment Develop products & resources Create collaboration with partners Implement new clinical protocols Adopt shared clinical records Parts of a Logic Model - Activities

14. Outcomes: How the target audience changes or benefits after participating in program’s activities. Short term: Immediate actions desired among participants Measures are usually expressed quantitatively Shows what “they” do Examples: Providers implement integration components # of new clients reached  # of Clients screened % Clients engaged in treatment % Clients receive Follow-up Partners implement agreed-on components Change in agency “agendas” Parts of Logic Model:Short-Term Outcomes

15. Weak: “Increased capacity of primary care to provide addiction treatment” “Patient assessment tools using SDOH evidence designed and utilized” “Increase in knowledge, attitudes and beliefs around integrated care for practice personnel, mental health providers and third-party payers.” Better: “80% of PC sites improve scores on SSA tool” “ # of SA patients treated in primary care” “ At least 150 patients assessed/year, using evidence-based tools” “90 % of providers practice integrated model consistently” LM: Short Term Outcomes: Examples

16. Reflect your long-term goals for the project. “What will they do then?” Ideally, measured quantitatively, behaviorally. May be derived from evidence base. Include LT outcomes for agency, e.g., sustainability Examples: Integration components used consistently by all providers Increased client access to care % clients with continued behavior change Higher retention rate in SA treatment New policies enacted Partners sustain integrated services Parts of Logic Model:Long-Term Outcomes

17. Weak: “Improved health status” “Evaluate sustainability of project financially – set up data base for estimating revenue capacity” “Integrated quality health care for clients, leading to longer lives” “Add to the existing evidence base” Better: “X % of patients with chronic diseases improve on relevant heath status measures” “Revenue sources secured for sustaining integrated services” “High quality integrated care for clients continued after project (as per SSA)” LM: Long Term Outcomes – Examples

18. Check the logic and assumptions underlying your project!

19. Feedback Loops • Information about the project is sent back as an input to the system • Can lead to project improvements • A key use for process evaluation, from your short-term outcome measures

20. Logic Model - MeHAF Integration Initiative Clients/Participants: Health providers, funders and consumers in Maine • Long-Term Outcomes • Maine’s health care system is more integrated • Increased % of Maine’s pop. receives integrated care • Sustainable integration created at practice & system levels • Support integration through: • reimbursement, • regulations & • licensing • Value of integrated care demonstrated • Non-grantee providers take up integrated approach • Unintended effects are avoided • Resources • Results of visioning & grassroots discussions • Evidence base re clinical practices • Staff/consultants’ knowledge of Maine • Collaborative relationships with stakeholders & other Maine initiatives • Maine’s data resources • State Health Plan • Inputs from Board & Community Advisory Group • Other MeHAF programs • Alignment with other efforts • Activities (Outputs) • Grants Program: • Plan and implement initiative with 10 – 20 new projects/year • Manage grant funding &resources • Technical Assistance and Support: • Assist projects: • - Site visits/calls • - Quarterly meetings • - Peer learning • - Facilitate evaluation • Sustainability and Dissemination: • Communicate results & learning • Fund research • Engage policy groups to promote system change • Align with state and national efforts Environments • Fragmented service delivery  Social norms, individual beliefs  Workforce shortages  Regulations, licensing, reimbursement  Translation of research to practice • Short-Term Outcomes • For MeHAF: • High quality pool of applicants developed • Models/culture of pt.-centeredness and integration created • Trends, issues & opportunities...leveraged • Systems change .... • MeHAF viewed as partner • For Grantees: • Increased .... level of pt.-centeredness & integration • Changed delivery systems • Improved pt. outcomes • Priority pop.’s served • Pt’s/families become advisors/advocates • Data systems support integration

21. Conclusions • The process of developing a Logic Model may require time & multiple stakeholders. • Prior literature and stakeholders should be consulted. • Logic Models are powerful, “living” tools for planning, management, evaluation, and communication, at any stage of project implementation!

22. Logic Model –Integrated Primary Care – Sacopee Valley Health Center Clients/Participants: Patients of and community members referred to SVHC for services. Environment: Fragmented delivery system, Rural area, Poverty, Workforce Shortages, Cost of Gas, Reimbursement issues

23. Resources • Funding from MeHAF • Information on other practices gathered through research.Staff: IPP Coordinator, Medical Mobile Unit, MMHP’s Quality Improvement Committee. • External partners and key staff in steering committee. • Information from local mental health providers on capacity to offer MH/SA services to MSFWs. • Information on MH/SA needs from MSFW communities. • Evaluation Consultant • Activities (Outputs) • Research: • Review of literature on comparable programs. • Review of existing models. • Surveying: • MSFW community survey. • Mental health / Substance abuse provider survey. • Guidance: • Feedback from members of steering committee. • Advice on designing evaluation plan from external evaluator (consultant). • Short-Term Outcomes • List of best practices and models for mental health/substance abuse services integrated to primary care programs. • Compiled and tabulated data on: • 182 MSFW’s barriers and needs for MH/SA services. • 23 Local mental health providers’ capacity to meet the needs of MSFWs. • A coherent evaluation plan for the project. • Long-Term Outcomes • Strategic mix of resources needed to meet the behavioral health needs of Maine’s MSFWs identified. • Working model of care that adequately and appropriately integrates behavioral health services into MMHP’s primary care program has been developed or adapted. (ready to pilot by January, 2009) Environments Lack of MH/SA services for MSFWs. Staffing to obtain needed information. Lack of information on existing models, capacity of services, and real needs and barriers of target population. Translation of research into a pilot program. Logic Model - Integrated Behavioral Health for Farmworkers (Plan phase-year 1) Clients/Participants: Maine Migrant Health Program (MMHP), and Migrant and Seasonal Farmworkers in Maine (MSFWs)

24. Your Turn! • Questions? • Work as a project team to develop your draft LM. • Logic Models will be due with interim reports, in August • Contact me, if questions arise.

25. Overview of Evaluation Plans for MeHAF Integration Initiative • Multiple methods used • Combined with grantee level evaluation • Focused on data collection and feedback over time, across grantees • Includes assessing both provider system-level change and care received by clients • Includes both “process” and “outcome” evaluation – linked over time

26. Major Components of MeHAF Evaluation Plan • Grantee Level Data Sources: • Do clinical services grantees make progress in implementing the components of integrated, patient-centered care? • Measured by the Site Self Assessment tool • How many & what types of clients receive integrated care, across all projects? • Data from Client Data Elements form • Do clients receiving integrated care show improvement in outcomes measured by each site? • Data submitted on Client Data Elements form

27. Major Components of MeHAF Evaluation Plan (Continued) • Grantee Level Data Sources • How are grantees implementing integrated care? • Qualitative data from site visits, annual reports; conducted by external evaluator - JSI • Are patients/families and providers engaged in, satisfied with their experience with integrated care? • Method(s) to be developed (TBD) • Can the costs of integrated care at grantee sites be documented? • Method(s) to be developed (TBD)

28. Major Components of MeHAF Evaluation Plan (Continued) • From State-Level Data Sources (Methods to be developed) • To what extent has “integrated care” spread among Maine’s population as a whole? • What are the total costs of providing integrated care & cost changes over time? • Do providers and/or consumers better understand and utilize integrated care? • Does the initiative prompt changes in reimbursement, regulatory and/or licensing policies?

29. Site Self Assessment Tool (SSA) • Many of you have completed the tool • Any comments after using the tool? • Note that SSA can also serve as a measuring tool for site level progress in implementing integrated care - it should be included in your Logic Models

30. Client Data Elements Form (CDE) • Developed with input from Round 1 grantees • 4 sections: I - Numbers of clients II – Demographics of clients III - Follow-Up IV – Client outcome measures • Discuss additional comments or concerns • Client data to be reported in CDE should be included as outcomes in your grantee-level logic models • An evaluation data management contractor will be available, with TA if needed