Performance measurement: Finding our way from outputs to outcomes

1. Performance measurement: Finding our way from outputs to outcomes

2. Finding our way from outputs to outcomes • How do we know we are having an impact on children, youth and their families? • How can we come together to ensure that the services we offer are appropriate and effective?

3. Panel members • Dr. Melanie Barwick: The role of CAFAS in measuring organizational and system outcomes for children and youth’s mental health • Roger Rolfe: The CYTS experience with CAFAS • Samantha Yamada: Building research and evaluation capacity at PRI

4. Overview: Performance measurement • WHAT is it? • WHY is it necessary? • WHO benefits? • How is it done?

6. What is performance measurement? “The regular collection of information for monitoring how a policy, program or initiative is doing at any point in time. It can be used to report on the level of attainment of planned results and on performance trends over time.” - Treasury Board Secretariat of Canada

7. Why is it necessary? • “Serves as a descriptive tool” on how a project, policy or program is doing • Serves as an early warning if the direction of a program, policy or project is not going as planned

8. Who is the audience? • End users: clients, families caregivers • Service providers, educators, program staff • Organization or network • Health system • Public at large

9. PM: provides regular snapshots of how a program or policy is doing; focuses on what the outcome is Client satisfaction surveys is often a key indicator of PM Evaluation: can provide insight into how and why an outcome is occurring Client satisfaction is part of the process evaluation and can influence outcomes How is it different from evaluation?

10. OUTCOMES ACTIVITIES SITUATION INPUTS OUTPUTS Quantity of work, products or participants Resources of a program Change in target audience Performance measurement and evaluation

11. How is it done? • Program logic models • Balanced score cards • Strategy maps

12. Balanced scorecard approach on health promotion Source: ICES, 2004

13. Improve client focus of addiction services Improve access to appropriate addiction treatment Further develop & increase equitable resources and capacity Improve healthy behaviours, health promotion and disease prevention Reducing Risk through influencing the broader determinants of health Ensure the continuum of interventions includes prevention, health promotion, early intervention, harm reduction and treatment services. Ensure evidence informed practices are developed, implemented and maintained across province Increase productive use & appropriate allocation of resources across system Increase linkages, transition & integration within addiction services Improve health outcomes at the individual and population level Ensure quality assurance within the addiction system Increase availability and retention of the qualified human resources Improve linkages and transitions between addiction, mental health, health, education, social and justice systems Increase sustainability and equity of the addiction & health systems Addiction System Strategy Map Ontario Federation of Community Mental Health & Addictions Programs

14. Child welfare performance measurement Source: OACAS QA Framework, 2004

15. Types of performance measures • Outcome measures • Intermediate outcome measures • Process measures • Output measures • Input measures

17. Strategic priorities by the Select Committee

18. Challenges and Opportunities • Selecting measures • Valid and reliable • Relevant, feasible, sensitive to changes • Developmentally and culturally appropriate • Information management capacity within agencies and in the government • Collaboration, buy-in and cultures that foster learning

20. Questions? Dr. Evangeline Danseco Head of Evaluation and Research 613.737.7600 Ext. 3319 edanseco@cheo.on.ca

21. The role of CAFAS in measuring organizational and system outcomes for children and youth mental health Melanie Barwick, PhD, C.Psych. Associate Scientist, Hospital for Sick Children Lead Implementer, CAFAS in Ontario (c) Barwick

22. Overview 1 • Evidence Base for Outcome Measurement 2 • CAFAS Measure 3 • CAFAS Implementation 4 • Ontario Outcomes (c) Barwick

23. 1 • Evidence Base for Outcome Measurement When clinicians are given feedback about how clients were responding to treatment (as expected, normally functioning, failure to respond), they have an opportunity to improve outcomes and reduce deterioration in the patient. Lambert, Whipple, Smart, et al., (2001) found that they could identify potential treatment failure based on initial level of disturbance and early negative response to treatment. Providing feedback to therapists enhanced outcomes and reduced deterioration. Those identified as potential treatment failures stayed in therapy longer and had better outcomes when feedback was provided to their clinician. Patient outcomes can be improved if therapists are alerted to treatment response . This is called “outcome management”. (c) Barwick

24. Utility of Outcome Measurement Using outcomes to MODIFY treatment if necessary Assessing outcomes DURING treatment to track client progress Assessing the outcome of treatment or service (c) Barwick

25. Benefits of Outcome Measurement (c) Barwick

26. Elements of a Successful System of Care in CYMH? • business practices • human resources • practice change • implementation • CAFAS • BCFPI • EBP • Best practices (c) Barwick

27. 2 • Overview of the CAFAS Measure • Provincially mandated use of a the Child and Adolescent Functional Assessment Scale (Hodges, 2002) to measure level of functioning outcomes among 6-17 year old children and youth receiving mental health services in Ontario • Begun in 2000 with training of over 3000 practitioners over 3 years; now reaching 6,000 practitioners! • 120 CYMH organizations selected by MCYS to participate in the initiative • CMHCs also participate in use of a systematic intake screening interview called the Brief Child and Family Phone Interview; oversight and training for BCFPI is provided by Children’s Mental Health Ontario (c) Barwick

28. AGE GUIDELINES CAFAS - Child and Adolescent Functional Assessment Scale children ages 6-17 PECFAS - Pre-school and Early Childhood Functional Assessment Scale children ages 4-7

29. CAFAS Subscales (c) Barwick

30. Levels of Functional Impairment (c) Barwick

32. CAFAS Caregiver Subscales (c) Barwick

33. Caregiver Resources, Material Needs… Caregiver difficulties in providing for the child’s material needs - housing, food, clothes - and there is a negative impact on level of functioning Child’s needs for food, clothing, housing, medical attention are not being met, causing severe risk Insufficient material needs leads to frequent negative impact on the child An occasional negative impact due to this depravation

34. Caregiver Resources, Social Support Caregiver difficulties in providing a home setting that is free of known risk factors (abuse, parental alcoholism) or in providing for the child’s emotional & social needs Caregiver is hostile, rejecting, or does not want child to return to the home Family members are insensitive, angry and/or resentful to the youth Family not able to provide warmth, security, & sensitivity

35. Score Interpretation

36. Milestones in Implementation and Uptake (c) Barwick

37. 3 • CAFAS Implementation (c) Barwick

38. Implementation Supports (c) Barwick

39. Annual Reports (c) Barwick

40. 4 • Ontario Outcomes (c) Barwick

41. Total cases submitted by export deadline: N = 52,423Next: Restrict to the exporting time-frame Find cases outside of the admission date interval required by last export (01/04/2008 - 31/03/2009) Exclude N = 151 Cases outside the range or with erroneous admission dates N = 52,423 Next: Restrict to the reporting time-frame Retain Find cases closed prior to 01/04/2008 or cases with a T1 evaluation after 31/03/2009 Exclude N = 24,144 N = 28,128 Next: Restrict to cases within the 6-18 yrs old age range Retain Find cases outside the age range Exclude N = 516 N = 27,612 Next: Restrict to cases without a pre-treatment evaluation (T1) Retain Find cases without a T1 Exclude N = 638 Cases without a T1 but with a T14 (Exit) CAFAS evaluation: N = 219 Cases without T1 or a T14 (Exit) CAFAS evaluation: N = 419 Retain Total analyzable cases: N = 26,974 Cases with just T14 CAFAS evaluationN = 170 Cases with T14 and at least one other CAFAS evaluation: N = 49 (c) Barwick

42. Number and Regional Distribution of Mandated Agencies Submitting Data (c) Barwick

43. Gender Distribution of Children and Youth Receiving CMH Services and CAFAS Rating (c) Barwick

44. Age Distribution (c) Barwick

45. Children with Complex Needs (c) Barwick

46. Complex Needs (2) (c) Barwick

47. Severity at Entry to Treatment for Ontario and Regions(2005: N=9,065; N= 2006: N=18,255; 2007: N=23,566; 2008: N=26,974) (c) Barwick

48. Severity at Entry to Treatment (2) (c) Barwick

49. Severe Impairment on CAFAS Subscales at Entry to Treatment – years 2005 to 2008 (c) Barwick

50. Average CAFAS Subscale Score at Entry to Treatment (T1) by Sex (N for Boys varies between 15,371 and 15,394 and N for Girls varies between 11,437 and 11,455 due to missing subscale scores) (c) Barwick