521: The Quality Service Review Process

1. 521: The Quality Service Review Process

2. Day 1 - Agenda • Training Overview • Overview of the QSR Process • Overview of the Indicators • Roll-Up Sheet • The QSR Interviews • Child/Youth and Family Status Indicators

3. Day 2 – Agenda • Practice Performance Indicators • Feedback Session with the Caseworker/Supervisor • Case-Specific Team Debriefing • The QSR Written Case Review Summary • Closing and Evaluation

4. Learning Objectives • Identify the components of the QSR Process; • Recognize the QSR Indicators; • Apply the interview techniques relative to the protocol; • Assign reliable status and performance indicator scores; • Demonstrate a Caseworker/Supervisor Feedback Session; • Identify the steps to prepare for the Case-Specific Debriefing; and • Recognize the components of the QSR Written Case Review Summary.

5. QSR’s Connection to CQI Positive Change QSR PA CQI

6. DAPIMTM and “Learning by Doing” Define Work products – baseline surveys, alignment notes, initial feelings, ground rules, defined areas for improvements, desired future state Assess Work products – findings: strengths and gaps, root causes and general remedies Monitor Work products – monitoring quick wins and other changes, evaluations and measures of progress and impact Performance & Capacity Implement Work products – team activities, action plans, charters for working teams, communication efforts Plan Work products- quick wins, mid- and long-term improvement plans, communication and capacity plans

7. Purposes of the QSR Process • To appraise the current status of the focus child/youth and family and the performance of key practice areas; • To assure that the actions of a team of child welfare professionals are guided by clear and comprehensive policies and are rooted in Pennsylvania’s Practice Model/Standards; • To improve the quality and usefulness of information needed to evaluate the safety, permanence and well-being of children and youth;

8. Purposes of the QSR Process (continued) • To assure the quality and effectiveness of services for children, youth and families by regularly reviewing programs and practices; • To support the efforts of child welfare agencies to maintain an environment that encourages learning and program improvement; and • To stimulate actions and next steps, to build on strengths and support efforts to improve outcomes.

9. What is the QSR? The QSR is designed to conduct an in depth review of front line practice in specific locations and points in time. It is used for: • Appraising the current status of a child/youth in key life areas; • Appraising the current status of the parent/caregiver; and • Appraising the performance of key practices for the same child/youth/family.

10. How is the QSR Different From Other Quality Assurance Methods? Traditional Review Were visits timely? Is the child’s IEP in the file? Was there a referral to mental health? Quality Service Review Are children currently safe? Is the child progressing well academically? Is the child functioning well emotionally?

11. How is the QSR Different From Other Quality Assurance Methods? (continued) Traditional Review Is there a current plan in the file? Did the parents sign the plan? Is there a written assessment? Quality Service Review Is the plan likely to lead to permanence? Do the parents feel that the plan is their own? Does the assessment accurately identify underlying needs?

12. Skills and Abilities Discuss with your partner the following questions: • Which three challenges or needs are most important for you to overcome in order to become a successful QSR Reviewer? • On a scale of 1-10, with one being a minimal challenge or need and 10 being a major challenge or need, how would you rate your three identified challenges or needs? • What will it take for you to decrease your rating 1-2 points?

13. The QSR Process • Review Team Pairings • Sample Selection • Focus Groups and Stakeholder Interviews • Case Reviews • Feedback Session with the Caseworker/Supervisor • Case-Specific Debriefing • QSR Written Case Review Summary • Exit Conference (optional for reviewers) • Final Report • Next Steps Meeting • County Improvement Plan

14. What Happens Prior to the Onsite Review? • Identification and pairing of internal and external reviewers; • Random sample of a county’s cases are selected and stratified by a service area; • Family permission obtained to participate in the review and consents/releases obtained; and • Key interviews in each case are identified and appointments are scheduled.

15. Focus Groups and Stakeholder Interviews Purpose: • Obtain feedback specific to systemic issues rather than case-specific information; and • Provide a forum for open dialogue about challenges and solutions within the system.

16. Focus Groups and Stakeholder Interviews Invited to… Focus Groups: • Caseworkers, Supervisors, Managers, Service Providers, and Resource Parents Stakeholder Interviews: • Judges, Attorneys, Police, Probation Officers, County Commissioners, etc.

17. Exit Conference – What Happens? • A few reviewers will be asked to present the highlights from their case review. • The Local and State Site Leads deliver the aggregate findings from all the cases reviewed along with trends discovered from the focus groups and stakeholder interviews. • Key strengths and areas needing improvement are identified.

18. Final Report – What Happens? • Final Report which includes analysis of demographic information, quantitative analysis of indicator scores, and information from the County Specific Data Packet is shared. • The county will use the data and analysis to develop a County Improvement Plan.

19. Next Steps Meeting – What Happens? • Approximately 6-8 weeks after the onsite review; • Led by the Local and State Site Leads; • Lasts approximately two hours; • Discuss the recommendations for next steps in working toward an action plan for improving frontline practice and system performance; and • Begins the next step in the CQI process.

20. What is Your Role? Each pairing of reviewers will: • Conduct case reviews: • Case file review - overview of the case record; • Interviews - meet with key players involved in the case; • Roll-Up Sheet - demographic information and scoring; • Provide feedback to the caseworker (and supervisor) - immediate feedback including suggestions and potential next steps for the case; • Present information gathered from case review and feedback on the family story at the Case-Specific Team Debriefing; • Complete a QSR Written Case Review Summary; and • Reviewers may be invited to attend the Exit Conference.

21. Child/Youth and Family Status Indicators • Safety: Exposure to Threats of Harm; • Safety: Risk to Self/Others; • Stability; • Living Arrangement; • Permanency; • Physical Health; • Emotional Well-Being; • Early Learning and Development; • Academic Status; • Pathway to Independence; and • Parent or Caregiver Functioning.

22. Practice Performance Indicators • Engagement Efforts; • Role and Voice; • Teaming; • Cultural Awareness and Responsiveness; • Assessment and Understanding; • Long-Term View; • Child/Youth and Family Planning Process; • Planning for Transitions and Life Adjustments; • Efforts to Timely Permanence; • Intervention Adequacy and Resource Availability; • Maintaining Family Relationships; and • Tracking and Adjusting.

23. What Time Period is Considered in Scoring? • Child/Youth and Family Status Indicators look at the past 30 days • Exceptions are: Stability and Academic Status • Practice Performance Status Indicators look at the past 90 days • Exceptions are: Planning for Transitions and Life Adjustments and Efforts to Timely Permanence • NOTE: Recent performance improvements have some influence on scores

24. How are Cases Scored? Each indicator is scored on a scale of 1 – 6 • 6 – Optimal • 5 – Substantial Acceptable Range • 4 – Fair • 3 – Marginal • 2 – Poor Unacceptable Range • 1 – Adverse Maintenance Zone: 6-5 Refinement Zone: 4-3 Improvement Zone: 2-1

25. Distinction Between 3 and 4 The Groundhog Rule If this case were frozen in time as it is today, would performance or status be considered acceptable or unacceptable?

26. The Central Scoring Rules Rules Include: • Concentrate on the Focus Measure and Core Concepts in each indicator. • Read each indicator’s instructions: • Some indicators score results, others score effort or intent. • Not Applicable rating criteria are identified as appropriate. • Score each sub-category within each indicator, as appropriate. • Stay within the time-based observation windows associated with each indicator. • Rate indicators based on events that have occurred or conditions that were present within the time-based observation window.

27. Roll-Up Sheet Purpose Purpose • Gather Data and Monitor Practice Model Implementation: • Demographic information • Case information • Indicator scores

28. Roll-Up Sheet Completion • Basic case information is filled out by the caseworker prior to onsite review. • Remaining information is gathered through case file review and interviews. • Reviewers will submit a validated Roll-Up Sheet electronically to Site Leads on a flash drive on the final day of onsite review. 

29. Roll-Up Sheet Components • General Review Information • Child/Youth’s Demographic Information • Child/Youth’s Current Placement • Child/Youth’s Educational Information • Case Information • Child/Youth and Family Status Domain • Practice Performance Status Domain

30. Planning QSR Interviews • Know the case history; • Plan relevant questions for each respondent; • Have accurate directions and be punctual; and • Organize the interview to help the respondent participate.

31. Prior to the Interview • Review the interview schedule and begin preparation by identifying potential questions; • Identify the lead interviewer and notetaker for each interview; • Remember that you are a reviewer and not a caseworker during the QSR process; and • Learn something about the interests of the child/youth to better engage him/her.

32. During the Interview • Take notes to document the interview; • Take every opportunity to recognize family strengths; • Listen more than talk; • Use the list of QSR questions to keep you on track; • When needed, ask questions on the Roll-Up Sheet;

33. During the Interview (continued) • Always ask about the child’s safety(if safety threats are identified, call site lead immediately); • Be patient with family stories; • Gather information, do not identify solutions to the case’s or family’s concerns; • Develop the relationship before inquiring about sensitive family issues; and • Avoid leading questions.

34. After the Interview • Do not share information gathered during one interview to the next interview; • Resist reaching conclusions prior to the completion of all interviews; • Contact your site lead or next interview if you are running a few minutes behind; • Note areas needing confirmation or follow-up; and • Contact your site lead if interviews lead to other key people in the child/youth’s life that you feel would be important to talk to regarding the case review.

35. The QSR Interview – Exploring Questions for Providers • For clinicians - What are your goals for this child/youth/family? • What are the child/youth/family’s goals? • What are the strengths of this child/youth/family? • What worries you most about this case? • Tell me what you are working on now. • Who do you see as members of this family’s team? • What’s the next big hurdle for this child/youth/family? • What will success look like? • On a scale of 1-10, how likely do you think success will be achieved? Ask follow-up questions. • Where do you see this case in six months?

36. Interviewing Practice – Sara’s Case • Both reviewers read the handout, Facts about the Fisher Family. • Reviewer playing the role of the mother reads the handout, Interview with Cynthia Fisher. • Reviewer playing the role of reviewer considers, What do you want to learn in the interview with Sara’s mother? • Conduct the interview.

37. Child/Youth and Family Indicators • With your partner, review Handouts #6-10. • Summarize the indicator, score the indicator and identify two good questions for each indicator you are assigned. Group 1: 5 and 1a Group 2: 2 and 1b Group 3: 4 and 5 Group 4: 3 and 6 Group 5: 7b and 4 Group 6: 6 and 9 • Be prepared to join other teams in a group discussion about the scoring experience.

38. Child/Youth and Family Indicators • With your partner, review Handouts #6-10. • Summarize the indicator, score the indicator and identify two good questions for each indicator you are assigned. Group 1: 1a and 9 Group 2: 2 and 1b Group 3: 4 and 5 Group 4: 3 and 6 Group 5: 7b and 4 Be prepared to join other teams in a group discussion about the scoring experience.

39. Practice Performance Indicators • With your partner, review Handouts #6-10. • Summarize the indicator, score the indicator and identify two good questions for each indicator you are assigned. Group 1: 1a and 5 Group 2: 9 and 11 Group 3: 2 and 10 Group 4: 3 and 7 Group 5: 4 and 8 Group 6: 1b and 6 • Be prepared to join other teams in a group discussion about the scoring experience.

40. Practice Performance Indicators • With your partner, review Handouts #6-10. • Summarize the indicator, score the indicator and identify two good questions for each indicator you are assigned. Group 1: 1a and 5 Group 2: 6, 9, and 11 Group 3: 2 and 10 Group 4: 1b, 3, and 7 Group 5: 4 and 8 • Be prepared to join other teams in a group discussion about the scoring experience.

41. Caseworker/Supervisor Feedback Session • Thank the caseworker and supervisor for their time. • Review the QSR purpose. • Outline the strengths of the case. • Remind caseworker of the high QSR standards. • Very briefly summarize family story – ensure facts are correct. • Share new information, if applicable. • Ask caseworker for ideas for next steps. • Discuss the strengths and challenges of the indicators. • Share reviewer ideas for next steps. • Thank the caseworker/supervisor and emphasize that they had a meaningful role in making a difference for children, their families and the system.

42. Feedback Session Practice • Please complete Handout #13, Notes for Providing Feedback to Caseworkers. • When completed, one partner will be the caseworker; the other partner, the reviewer. • Practice a feedback session providing the caseworker feedback on Sara’s case. • Compare your notes with your partner from Handout #13, did you identify similar: • Strengths/Accomplishments; • Practice Challenges/Limiting Conditions; • Opportunities for Advancing Practice/ Improving Local Conditions (Next Steps).

43. Case-Specific Team Debriefing • Inter-rater reliability check for consistency between case description and scores. • Identification of themes for feedback to the county and written report. • Preliminary communication about review findings to leadership.

44. Case-Specific Team Debriefing Outline Case Presentation Outline – 15 minutes • Case Summary (2 minutes) • Child/Youth and Family Status Indicators – Strengths and Challenges (4 minutes) • Practice Performance Indicators – Strengths and Challenges (4 minutes) • Next Steps (2 minutes) • Group Questions/Feedback (3 minutes)

45. The QSR Written Case Review Summary - Purpose • Explain the “story” behind the data; • Provide feedback to the caseworker and local leadership; and • Be utilized as an effective training tool within the county agency.

46. The QSR Written Case Review Summary- Additional Guidelines • Should not tell the case story/history; but rather include information you gathered to support the ratings of the indicators; • Written about the ratings and why the factors contributing to the indicators are either favorable or unfavorable; • Practical and helpful recommendations; and • Make a note of your due date: • Three day Review – Due prior to the conclusion of the 3rd day • Two day Review – Due approximately 7-10 days following the onsite QSR NOTE: There will be an editing process done by the Site Lead Team, so you may be asked to edit your Written Case Review Summary and/or provide more supportive information.

47. CQI Continues… Remember QSR is NOT a one-time event. It’s part of the bigger CQI process that leads to improved outcomes of Safety, Permanence and Well-Being for children, youth and families.