Assessing & Improving Youth Program Quality Nicole Yohalem, Forum for Youth Investment

1. Assessing & Improving Youth Program Quality Nicole Yohalem, Forum for Youth Investment Quality Youth Development Practice Webinar April 2, 2009

2. Why focus on quality? • Quality matters • Quality is measureable (we can “know it”) • Quality is malleable (we can “grow it”)

3. Quality Matters While improving youthoutcomes is the goal we are all working toward, we believe that to get there at scale, our focus needs to be on helping leaders improve the quality and reach of programs and services available in the community.

4. Quality Matters There is basic agreement that young people need structured, voluntary activities to fill their out-of-school hours. • The good news: There is now strong evidence that these programs can produce positive change in things we all care about – academic achievement, social and emotional development. • The bad news: Many programs do not.

5. Quality Matters Why do only some programs make a difference? • Controlling for participation, young people who participate in high quality programs achieve greater gains than those who do not. • In 2007 meta-analysis, Durlak and Weissberg grouped 73 programs into 2 clusters based on quality features. Programs with those features showed positive effects on almost every outcome – school performance, social behavior, attitudes and beliefs. Programs that did not have the SAFE features showed no effect on any outcome.

6. Quality Matters In programs that score well on the YPQA, young people are more likely to have: • Improved program attendance • Improved school-day reading • Reduced school-day suspension • Self-reported interest in program, sense of challenge and sense of growth Key Point: Programs can improve outcomes by improving the quality of their services.

7. Quality is Measurable • Assessing Afterschool Program Practices Tool (APT) National Institute on Out-of-School Time and the MA Department of Education • CORAL Observation Tool (CORAL) Public/Private Ventures • Out-of-School Time Observation Instrument (OST) Policy Studies Associates • Program Observation Tool (POT) National Afterschool Association • Program Quality Observation (PQO) Deborah Vandell and Kim Pierce • Promising Practices Rating Scale (PPRS) WI Center for Education Research and Policy Studies Associates, Inc. • Quality Assurance System (QAS) Foundations Inc. • Program Quality Self-Assessment Tool (QSA) New York State Afterschool Network • School-Age Care Environment Rating Scale (SACERS) Frank Porter Graham Child Development Center, UNC • Youth Program Quality Assessment (YPQA) High/Scope Educational Research Foundation Measuring Youth Program Quality A Guide to Quality Assessment Tools Updated January 2009

8. Quality is Measurable There is a lot of similarity across definitions. Common elements include: • Relationships • Environment • Engagement • Social Norms • Skill Building Opportunities • Routine/Structure

9. Quality is Measurable There are some differences among tools, however: • Emphasis on the point-of-service vs. program administration. • The extent to which measures actually teach • The extent to which measures are clear and unambiguous

10. Quality is Malleable Can these aspects of quality be improved? Once we know it, can we grow it? YES

11. Youth Program Quality Intervention `` ` l • Systemic quality improvement systems anchored by the YPQA are being developed in: • States: MI, ME, RI, KY, NM, AR, MN, IA, WA, NY • Cities/Counties: Austin, Chicago, Rochester, Detroit, Grand Rapids, Palm Beach County, Baltimore, Nashville, St. Louis, Louisville, Georgetown Divide/Sacramento, Columbus IN, Indianapolis IN, Tulsa OK Seattle Rochester Minnesota Grand Rapids Washington* M i n n e a p o l i s Maine Chicago New York e t r o i t Iowa Indianapolis Rhode Island Sacramento/ Georgetown Divide m b u s Columbus Baltimore St. Louis Kentucky Oklahoma Nashville Arkansas New Mexico Austin West Palm Beach County

12. YPQI Focus: POS quality in context POS Point-of-Service Youth PQA Form A Engagement Interaction Support Safety PLC Professional Learning Community Youth PQA Form B • Org policies/practices • Management values • Performance feedback • Continuity/staffing • Standards and metrics • Staff development SAE System Accountability Environment

13. Quality is Malleable Components of effective quality improvement systems: • Quality standards that address what should happen at the point of service • Ongoing assessment of how well services compare to the standards • Targeted plans for how to improve • Training and technical assistance (including on-site coaching) that fits those plans

14. Quality Improvement: Lessons Learned • Have well defined purposes for the system. • Focus on the point-of-service. • Anchor improvement efforts with data about POS. • Create incentives for continuous improvement. • Build in on-site, ongoing technical assistance/coaching. • Be intentional about pilot participation. • Build learning communities. • Recognize that management is a key lever.