Moving Toward Evidence-Based Practice: Innovative Ideas for the CSBG Network

1. Moving Toward Evidence-Based Practice: Innovative Ideas for the CSBG Network NASCSP September 2012 Barbara Mooney, Ed. D. Karen Walker, Ph.D. National ROMA Training and Certification Project

2. Performance? Evidence? • We know we have a groundwork in the CSBG program for Performance Measurement and Performance Management, but how do we build on this foundation to move forward? • We need strategies for moving the CSBG Network toward greater adoption of evidence-based practices.

3. Workshop Objectives • Definitions and examples of • Evidence-based, evidence-informed • Participants will understand the regional and national factors pushing toward new practice. • Participants will consider how the core activities in IM 49 support universal implementation of high quality practices • Participants will learn about examples of using evidence to evaluate and improve programs • Participants will brainstorm “next steps” for action to move us toward more evidence based practices

4. What Does it Mean to be “Evidence-based”?

5. What Do People Mean by Evidence-based? • Multiple terms are in use: • Evidence-based practice • Evidence-based program • Evidence-informed program …How do we make sense of these terms?

6. What do we mean by Evidence-based programs and practices? • Evidence-based programs: • Interventions that have been found to have positive effects or impacts on targeted outcomes (Nurse Family Partnership, Parent Management Training) • Evidence-based practices: • Intervention strategies or program practices that have been found to improve outcomes for children and youth (role-modeling, positive reinforcement, avoiding grouping children with problem behaviors together)

7. Evidence-informed Program Intervention that has been designed with research evidence in mind. Some of the program’s strategies may be evidence-based practices, they may be informed by the basic research, or both. Evidence-informed program may also refer to a program that uses performance information to monitor and guide its progress.

8. Time Out for a brief description of what we mean by Randomized-Controlled Impact and Quasi-Experimental Evaluations • Randomized-Controlled Impact Evaluation • Provides most certainty about program effectiveness, but must be well-designed and well-implemented • Study participants randomly assigned to control or intervention group (think lottery). • Compares the difference in outcomes between the intervention and control group.

9. Quasi-Experimental Study Like randomized-controlled impact study (RCT) a quasi-experimental study includes a comparison between people who received the intervention and those who did not. Unlike an RCT, the comparison is not based on random assignment. As a result, researchers are less certain that the comparison is a good one.

10. What Can You Do to Become Evidence-Based or Evidence-Informed? Adopt evidence-based programs and monitor their implementation to ensure that they are being replicated correctly Assess “home-grown” programs to determine whether or not they include evidence-based or evidence-informed practices Set the program on a path toward rigorous evaluation …These strategies are not mutually exclusive

11. Becoming an Evidence-Based Program* Collect and Use Data on Performance & Outcome Measures Targeting Implement Program Conduct Ongoing Performance Management Identify Population Conduct Needs Assessment Select Intervention, Develop Logic Model & Identify Indicators Conduct an Implementation Evaluation Conduct a Quasi-Experimental Outcomes Evaluation Figure adapted from: Performance Management and Evaluation: What’s the Difference by Karen E Walker and Kristin Anderson Moore. Child Trends Conduct a Randomized-Controlled Impact Evaluation

12. Federal, State and Local Efforts and Policies • Widespread move toward evidence-based programs and practice and performance management • Federal funding tied to evidence • State initiatives and funds for intermediaries • Local efforts include both public and non-profit efforts • Administration’s guidance from OMB to agencies about encouraging evidence-based efforts.

13. Definitions from OCS: Evidence-Based Practice – Approaches to prevention or treatments that are validated by some form of documented scientific evidence. These could be findings established through scientific research, such as controlled clinical studies or other comparable and rigorous methods. Evidence-Informed Practice – Approaches that use the best available research and practice knowledge to guide program design and implementation within context. This informed practice allows for innovation and incorporates the lessons learned from the existing research literature.

14. Use Data to Show Results and Improve Performance • “Success should be judged by results, and data is a powerful tool to determine results. We can’t ignore facts. We can’t ignore data.” • President Barack Obama • July 24, 2009 • “The test of a performance management system is whether it’s actually used….Federal managers and employees at all levels must use performance goals and measures to set priorities, monitor progress, and diagnose problems.” • Chief Performance Officer Jeffrey Zients • October 29, 2009 14

15. New Guidance:GPRA Modernization Act 2011 Shift emphasis from production of annual reports to collecting and using performance information more frequently throughout the year to set priorities and make decisions that improve results Establish common Federal Government performance indicators with Quarterly Targets and Quarterly Milestones

16. The plan to target CSBG resources “Building on the President's budget request to target CSBG resources to high-performing, innovative agencies, ACF plans to work with Congress and a wide variety of organizational stakeholders to establish a set of core Federal standards that States will be required to use to assess whether an eligible entity is meeting a high standard of service delivery. These standards of service delivery will include a focus on compliance with basic requirements of the CSBG Act, and will also go beyond basic compliance to focus on high-quality performance across the CSBG Network. Core performance standards will focus on organizational management issues, as well as strategic process for performance management.” (OCS T/TA overview, 2012)

17. The CSBG National T/TA Strategy is designed to: Measure and document performance in accordance with CSBG reporting requirements. Assure performance measurement and use of performance data at all levels within the CSBG program (local, State, and Federal). Mitigate risks associated with the use of CSBG grant funds. Promote exemplary practices and innovative programming that stimulates the creation and sharing of information and knowledge in the CSBG T/TA Network. Measure and document performance in accordance with CSBG reporting requirements. Work with States to help ensure CSBG-eligible entity boards know, understand, and meet their fiduciary responsibilities to CSBG (i.e., participation in the development, planning, implementation, and evaluation of CSBG-funded activities and services). Ensure the management and delivery of CSBG-funded services creates and strengthens sustainable economic resources in communities

18. ROMA Next Generation and Organizational Standards Two targeted T/TA Centers of Excellence embody OCS focus for coming years

19. Organizational Standards The CSBG T/TA Organizational Standards Center of Excellence (COE) will coordinate the development and dissemination of a core set of standards with input from the State CSBG Lead Agencies as appropriate, regional and national partners, including the CSBG T/TA Regional Performance and Innovation Consortia (RPIC). The central mission of this effort will be to ensure that all CSBG-eligible entities have the capacity to provide high-quality services to low-income individuals and communities, and are able to meet high-quality organizational performance standards. The movement toward continued accountability, risk mitigation, and performance plays a key role in developing organizational standards toward improved performance management and the capacity to thrive within the CSBG Network.

20. ROMA Next Generation Progress has been made in recent years in the quality and breadth of State and local performance reporting systems through ROMA and the NPI framework… (which) provides a foundation for additional targeting of resources based on outcomes and effectiveness of services. However, practices of data collection and reporting within the ROMA system vary from State to State; while some States‘ reporting systems reflect the full ROMA cycle, other States only report NPI data. Such inconsistencies within the ROMA system minimize the capacity of ROMA to be an integrated and comprehensive effort that lends to accurate and reliable performance management and accountability within the CSBG model.

21. Key goals of the ROMA Next Generation COE will be: To review and strengthen the current performance measures and performance indicators with input from experienced CSBG organizations, as well as performance management experts from outside the CSBG Network; To work with State CSBG agencies to communicate consistent standards for performance management throughout the ROMA cycle, including community needs assessment, program planning, and performance measurement; To assure a coordinated, national strategy for nationwide implementation of an updated performance management system; To assure that a performance management system can be used, as necessary, for comparison of the effectiveness of local entity use of CSBG resources.

22. We are keeping the basics: Six National Goals ROMA Cycle (core activities for eligible entities) Core activities for state offices Checklist for Monitoring (IM102) Risk Mitigation (IM 112) Key elements of the Logic Model

23. Six National Goals 1. Low-income people become more self-sufficient. 2. The conditions in which low-income people live are improved. 3. Low-income people own a stake in their community. 4. Partnerships among supporters and providers of services to low-income people are achieved. 5. Agencies increase their capacity to achieve results. 6. Low-income people, especially vulnerable populations, achieve their potential by strengthening family and other supportive systems.

25. IM 49 Core Activities for State Offices • The state has developed, in coordination with eligible entities and the State CAA association, a State-wide vision statement that speaks to the goals and purposes of community action within the State and that supports the 6 national ROMA goals. • The state is encouraged to participate in, and contribute to, broader State anti-poverty/community development initiatives with outcome measures and goals compatible with ROMA; • The agency has trained all its eligible entities (staff and boards) in outcome-based management, and that 80% of the entities use ROMA concepts to guide needs assessment, agency mission review, activity planning, resource allocations, service delivery, measuring and reporting results;

26. Core Activities for State Offices (cont) • 80%of the plans and program reports received from eligible entities in the State describe plans to achieve projected outcomes, and evaluate results based on measurable improvements of condition(s) among clients and/or communities served; • The agency submits complete, accurate, and timely annual reports to OCS on the measured performance of the State and the eligible entities in the State as required by Section 678E of Public Law 105-285, the Community Services Block Grant Reauthorization Act of 1998.

27. ROMA Logic Model

28. Community Federal, State and Local Policies Population Culture (norms and behaviors) Funding Organization Mission & Goals Leadership Resources Policies Program Monitoring Staff assessments Program Activities Staff roles and skills Training Quality and Fidelity Participation Outcomes

29. ExamplePerformance Management In Action • After-school tutoring program assisting at-risk youth – attempting to improve their math and reading performance. • Collect the following data: • Demographics and risk/protective factors • Program attendance • Amount of service (hours and topics of tutoring) • Report card grades (first / last grading period)

30. Example (cont.)Performance Management in Action • Finding #1: Halfway through program, program attendance is low among youth from Neighborhood A. • Action: Reach out to teachers in schools in Neighborhood A • Action: Talk with youth in Neighborhood A • Finding #2: Math grades increased, but reading grades did not change. • Action: Investigate. • Finding: Tutors unfamiliar with reading curriculum • Action: Train tutors in reading curriculum

31. Consider Resources for Performance Management • Executive leadership • To set the tone, to implement policies • Staff expertise • To identify measures, to collect data, and to analyze data • Staff time • To collect, enter, analyze, share, and discuss data • Technology • Forms (paper or electronic), spreadsheet software, management information systems/commercial software

32. Create Policies for Performance Management • Performance management policy outlining: • Staff roles and responsibilities related to performance management • Description of data collection and reporting methods • How often regular meetings should take place to review and discuss findings • Performance appraisal process which evaluates staff on how well they carry out performance management roles and responsibilities

33. Example Questions in Performance Management Are we reaching our intended population with the intended services? Are participants getting the amount of program time we want them to get? Are participants engaged in the process? Are participants’ outcomes changed? Do our staff carry out activities as intended? Are we providing the training our staff need? If the answers are no, organizations need to determine why and what needs to change

34. Video Clip from “Saving Philanthropy” http://www.savingphilanthropy.org/preview-footage/ • An example about how sharing performance management data on outcomes can increase staff buy-in, inspire youth, and create a learning culture

35. Implementation will take resources • Time and attention to develop a plan of action • Staff time and support (travel, meeting costs, provision of materials, etc.) • At the state level, for local EEs • Follow up to evaluate actions and assess new needs/resources

36. T/TA from National ROMA Trainers • The demand for training and technical assistance exceeded the available resources in the ROMA Trainer network to address the needs. • Two specific areas were identified: • The certified trainers do not have any resources to bring local CAAs together as they begin to implement the full range of ROMA activities. Implementation support is critical to working out solutions to local challenges and establishing a consistent use of terms, concepts and processes. • NCRTs do not have the time or the expertise to provide individual on-site technical assistance to local CAAs as they apply the principles to agency systems. • We are working to increase the number of available NCRTs in partnership with the RPICs and State Offices

37. State CSBG Discretionary Dollars • NASCSP Resources on discretionary funds: • Annual Reports and appendix tables • Issue Brief: State Discretionary Spending

38. Distribution of FY 2011 CSBG Discretionary Funds by Purpose

39. CSBG National Resource Center & Clearinghouse Resources online Evidence-based discussions Forums Lists of consultants

40. Child Trends’ Resources Using Data in Multi-Agency Collaborations to Ensure Accountability and Improve Performance. http://www.childtrends.org/Files//Child_Trends-2012_02_23_FR_UsingData.pdf Performance Management and Evaluation: What’s the Difference? http://www.childtrends.org/Files//Child_Trends-2011_01_19_RB_PerformMgmt.pdf at www.childtrends.org Leap of Reason: Managing to Outcomes in an Era of Scarcity, available at leapofreason.org

41. What’s next? How can you identify and focus resources so our network can move toward evidence based practices? What specific steps do you need to take to begin to identify what works and how to collect the evidence? What do you need to consider?

42. Steps toward improving practice

43. IMPROVING REPLICATION AND “SCALE-UP” The history of replication and scale-up efforts indicates the difficulty associated with these efforts. As President Bill Clinton frequently declared, “You can find virtually every problem in our country solved by somebody somewhere in an astonishingly effective fashion….The challenge for us is to figure out how to replicate that.” The day before his first inauguration, Clinton told the nation’s governors that his number one disappointment as governor had been that it was so hard to “take something that works to the next level.” He said he “could never figure out a way to make the exception the rule, and that is our enduring problem in America in public life.”

44. For more discussion: Karen Walker Senior Research Scientist and Senior Program Area Co-Director Child Trends 4301 Connecticut Ave NW Suite 350 Washington, DC 20008 • 202-572-6005 kwalker@childtrends.org Barbara Mooney Project Director National ROMA Training and Certification Project 243 E. High St. Waynesburg, PA 724-852-2272 barbaramooney@ windstream.net