SOUTH DAKOTA’S SFP SIG EVALUATION

1. SOUTH DAKOTA’S SFP SIG EVALUATION Doug Piper, Senior Scientist Population Health Institute School of Medicine and Public Health University of Wisconsin – Madison SPF SIG Advisory Council Meeting Chamberlain, South Dakota September 20, 2010 Population Health Institute - University of Wisconsin - Madison

2. The SPF SIG Project was designed by the folks at CSAP to change the way substance abuse prevention is done in the states. • Emphasizing data driven planning • Evidence-based programs • Evaluation • Complex • Resource intensive • Largely driven by CSAP requirements Intention of the SPF SIG Project Population Health Institute - University of Wisconsin - Madison

3. Expectations for State and Local SPF SIG Grantees • Fidelity implementing the SPF process at the state, tribal and coalitions level will increase prevention capacity and a build a stronger prevention infrastructure at all levels. • “Infrastructure” is the underlying foundation or basic framework which is necessary for an organization or system to function. • The components of strong prevention infrastructure might include: • A unifying theory of prevention; • Effective leadership, • Readiness to collaborate organizations and agencies; Population Health Institute - University of Wisconsin - Madison

4. Expectations for State and Local SPF SIG Grantees: Components of a strong Infrastructure (cont.) • A logic model; • A set of consistent definitions of such terms as evidence-based prevention; • A blend of individual and environmental strategies; • Monitoring and evaluation; • Workforce development (e.g. training or accreditation), • Structural organization, • Resources available, • Cultural competence, • Marketing and sustainability. Population Health Institute - University of Wisconsin - Madison

5. Expectations for State and Local SPF SIG Grantees • Increased prevention capacity & stronger prevention infrastructure at the local level will lead to…. • Community-level prevention system changes independent of the direct allocation of SPF SIG funds. • Delivery of evidence-based, culturally appropriate direct service and environmental prevention programs. • Further refinement and efficiency of planning and implementation. Population Health Institute - University of Wisconsin - Madison

6. Expectations for State and Local SPF SIG Grantees • The delivery of evidence-based programs implemented with fidelity will lead to improvements in community level and individual outcomes. • Integration of cultural competency principals and practices at the state and local levels will promote sustained prevention capacity and infrastructure development. Population Health Institute - University of Wisconsin - Madison

7. South Dakota’s AOD Prevention System • Funding source • SD governmental unit • Programs • State-wide advisory body • MOUs with DADA to run programs Population Health Institute - University of Wisconsin - Madison

8. South Dakota SPF SIG Goals for the Local Coalitions • The South Dakota SPF SIG Advisory Committee, based on extensive epidemiological data, have chosen the following goals: • Reduce the incidence and consequences of underage drinking in South Dakota among youth ages 12 to 20 years. • Reduce the incidence and consequences of binge drinking among 18 to 25 year olds in South Dakota. Population Health Institute - University of Wisconsin - Madison

9. South Dakota’s 30 day use for alcohol and binge drinking for 18 to 25 year olds compared to National data NSDUH data Population Health Institute - University of Wisconsin - Madison

10. Primary Evaluation Objectives • Describe the process through which the SPF project is implemented in South Dakota and at the community and tribal levels. • Measure prevention capacity and infrastructure development in South Dakota and the community and tribal levels. • Determine to what extent the SPF has produced changes in targeted communities and individual outcomes. Population Health Institute - University of Wisconsin - Madison

11. Evaluation Types • Monitoring is checking records and doing site visits to ensure that the intervention included in the contractual work plan is being implemented with reasonable fidelity. This is DADA’s role and not that of the evaluation team. • Process evaluation documents the program implemented by tracking the input or activities. It defines the independent variable. • 3. Outcome evaluation measures the program’s effects that you expect to achieve after the program is implemented Program actually implemented Inputs Outcomes – targeted behavior Population Health Institute - University of Wisconsin - Madison

12. Environmental Strategies • Attempt to change the surroundings in which people choose to use, or not use, substances. • Aim at changing environmental conditions that contribute to substance abuse. • Accessibility of substances, • Law enforcement efforts to reduce illegal activity, penalties for illegal activity, as well as • Social norms regarding substance use. • Often involve changing laws, policies, and regulations (LPRs) to make it more difficult for people to abuse substances. • Intended to change individual-level behavior by creating environments that dissuade substance abuse. Population Health Institute - University of Wisconsin - Madison

13. Logic Models and Evaluation* ENVIRONMENTAL EXAMPLE 1

14. Logic Models and Evaluation* NVIRONMENTAL EXAMPLE 2

15. This approach to prevention endeavors to change individuals within the community or society. Their surrounding environment is not a target for change to any great degree. They may aim to improve an individual’s refusal skills, increase their connection to the community or promote positive family interaction. Examples are: • School based programs • Indicated programs which focus on high-risk individuals Individual-focused Prevention Population Health Institute - University of Wisconsin - Madison

16. Logic Models and Evaluation INDIVIDUAL-BASED PROGRAM EXAMPLE

17. Challenges for Evaluating Community Outcomes of Environmental Strategies • Dearth of data corresponding to the Coalition’s service area and the project’s target populations. • Sensitivity of the data to our interventions, e.g. alcohol-related car crash deaths. • Immediacy of the intervention’s affects on the target population. • Dosage of the intervention penetrating to the target population. • May require additional data collection for 20 to 25 year olds for social norms, etc. Population Health Institute - University of Wisconsin - Madison

18. Need a comparison group to show more than just change. • CSAP requires all participants in individual-focused programs to be surveyed: • Must include NOMs questions • Data collection methodology must be up to standards • Periodicity of the surveys must be approximately the same. • Local Coalitions may be able to secure permission from local school districts to use their data. Only relevant to the underaged drinking target population of 12 to 18 year olds. Challenges for Outcome Evaluation of Individual-based Prevention Population Health Institute - University of Wisconsin - Madison

19. Evaluation Tasks for the Coalitions (1 of 3) Population Health Institute - University of Wisconsin - Madison

20. Evaluation Tasks for the Coalitions (2 of 3) Population Health Institute - University of Wisconsin - Madison

21. Evaluation Tasks for the Coalitions (3 of 3) Population Health Institute - University of Wisconsin - Madison

22. Suggestions for recruiting an on-site evaluator for the South Dakota SPF SIG Project to be a member of the evaluation team, housed in Pierre and an employee of the University of Wisconsin – Population Health Institute. Population Health Institute - University of Wisconsin - Madison

23. Doug Piper • 5901 Research Park Blvd. • Madison, WI 53719 • 608.263.7629 • DPiper2@wisc.edu Population Health Institute - University of Wisconsin - Madison