Improving Prevention Effectiveness
This presentation is the property of its rightful owner.
Sponsored Links
1 / 49

William B. Hansen, Ph.D. Tanglewood Research, Inc. Greensboro, NC PowerPoint PPT Presentation


  • 65 Views
  • Uploaded on
  • Presentation posted in: General

Improving Prevention Effectiveness The Maryland Alcohol and Drug Abuse Administration Annual Management Conference October 5, 2006. William B. Hansen, Ph.D. Tanglewood Research, Inc. Greensboro, NC. Profile population needs, resources, and readiness to address needs and gaps.

Download Presentation

William B. Hansen, Ph.D. Tanglewood Research, Inc. Greensboro, NC

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


William b hansen ph d tanglewood research inc greensboro nc

Improving Prevention EffectivenessThe Maryland Alcohol and Drug Abuse Administration Annual Management ConferenceOctober 5, 2006

William B. Hansen, Ph.D.

Tanglewood Research, Inc.

Greensboro, NC


Samhsa s strategic prevention framework

Profile population needs, resources, and readiness to address needs and gaps

Monitor, evaluate, sustain, and improve or replace those that fail

Mobilize and/or build capacity to address needs

Implement evidence-based prevention programs and activities

Develop a Comprehensive Strategic Plan

SAMHSA’s Strategic Prevention Framework

Assessment

Evaluation

Capacity

Planning

Implementation


Samhsa s strategic prevention framework1

SAMHSA’s Strategic Prevention Framework

Assess

Evaluate

Select a Strategy

Implement

Develop Capacity


What do programs want to do

What Do Programs Want To Do?

Assess

Evaluate

Select a Strategy

Implement

Develop Capacity


What do programs want to do when they are required to evaluate

What Do Programs Want To DoWhen They Are Required to Evaluate?

Assess

Evaluate

Select a Strategy

  • The quality of delivery.

2. The effects achieved.

Implement

Develop Capacity


What do we mean quality of delivery

What Do We Mean: Quality Of Delivery?

  • Dosage

    • How much?

    • How often?

  • Adherence

    • Was the program delivered as intended?

    • Was new content added?

    • Was important content deleted or modified?

  • Relevance

    • Was the program engaging to participants?

    • Did the program meet participants’ needs?


Dosage 1

Dosage 1

  • Meta-analysis of 25 SAMHSA model programs.

  • Programs that were delivered more frequently, generally had larger effects.


Dosage 2

Dosage 2

  • Same 25 SAMHSA model programs

  • Programs that had more opportunities for contact were generally more effective.


All stars dosage tracking

All Stars Dosage Tracking


All stars dosage tracking1

All Stars Dosage Tracking


All stars dosage tracking2

All Stars Dosage Tracking


Adherence

Adherence

  • An evaluation of Life Skills Training

  • Observers rated the percent of objectives met and lesson points covered

  • High-fidelity classes (>60% adherence) did best.


Adherence1

Adherence

  • An evaluation of Life Skills Training

  • Observers rated the percent of objectives met and lesson points covered

  • High-fidelity classes (>60% adherence) did best.


Adherence2

Adherence

  • An evaluation of Life Skills Training

  • Observers rated the percent of objectives met and lesson points covered

  • High-fidelity classes (>60% adherence) did best.


Local adherence

Local Adherence

  • Drug Strategies assessed adherence of Life Skills Training implemented in Baltimore.

  • Teachers re-taught lessons they had previously delivered.

  • Observers rated adherence.

  • Teachers implemented 65% of objectives (Range = 45-100%).

  • Teachers implemented 58% of main points (Range = 38-93%).


Local adaptation

Local Adaptation

  • All teachers made adaptations

  • 3.5 definable adaptations, on average, per observed session (range 1 to 7)

  • Overall, 63% of adaptations were judged to be negative


Helpful adaptations

Helpful Adaptations

  • The addition of reading material, videos, and testimonials

  • Changes in methods to make them more interactive

  • Inclusion of examples for cultural relevance or interest


Important correlates of adherence

Important Correlates of Adherence

  • Teacher’s Understanding of LST

    (r=.784; p<.01)

  • Quality of Process

    (r=.663; p=.03)

  • Level of Experience

    (r=.756; p<.01)


All stars approach to assessing adherence

All Stars Approach to Assessing Adherence


Relevance

Relevance

  • As part of the SAMHSA program meta-analysis, we coded relevance.

  • Three aspects of relevance were significantly correlated with outcomes:


All stars approach to assessing student engagement

All Stars Approach toAssessing Student Engagement


Quality of implementation summary

Quality of Implementation Summary

Data are needed to assess:

  • Whether a sufficient dose has been delivered

  • How closely delivery adhered to design

  • The relevance of implementation for participants

    Gathering and reporting data will improve quality of implementation.


Outcome evaluation

Outcome Evaluation

  • Everybody is afraid of outcome evaluation.

  • Why?

    • No one likes to fail.

    • It is perceived to be mysterious, complex, and expensive.

    • Outcomes are not controllable.

    • It shows you are normal.


What are prevention goals

What Are Prevention Goals?

The goal of prevention is not behavior change but either:

  • Non-behavior maintenance

  • Delay in onset

  • Reduce the intensity of use


How do programs work

How Do Programs Work?

  • All Programs are based on a logic model.

  • The program changes a mediator:

    • Characteristics of the participant (skill, motivation)

    • Characteristics of the social environment

    • Characteristics of the physical environment

  • Characteristics targeted for change affect behavior.


All stars logic model example

All Stars Logic Model Example

All Stars Core targets:

  • Lifestyle incongruence (idealism)

  • Normative beliefs

  • Commitment

  • Bonding to school

  • Positive parental attentiveness


Mediators in prevention

Motivation

Attitudes

Bonding

Beliefs consequences

Commitment

Normative beliefs

Lifestyle incongruence

Personal Competencies

Academic skills

Decision-making skills

Emotional self-regulation

Goal setting skills

Self-esteem

Social Competencies

Resistance skills

Media literacy

Communication skills

Social problem solving skills

Social skills

Environment

Availability, access, enforcement

Alternatives

Classroom management

Family management

Monitoring and supervision

Positive peer affiliations

Support and involvement

Mediators in Prevention


The easy part of outcome evaluation

The Easy Part ofOutcome Evaluation

  • Collecting survey data is easy.

  • After over 30 years development, there are:

    • Many measures for assessing alcohol, tobacco, drug use, and consequences of use.

    • Many measures for measuring mediators (risk and protective factors targeted for change).


Sample outcome results all stars in a community setting

Sample Outcome ResultsAll Stars in a Community Setting


Sample outcome results all stars in a community setting1

Sample Outcome ResultsAll Stars in a Community Setting


Sample outcome results all stars in a community setting2

Sample Outcome ResultsAll Stars in a Community Setting


What do when programs succeed

What Do When Programs Succeed?

Assess

Evaluate

Select a Strategy

Implement

Develop Capacity


Results looking at mediators

Results Looking at Mediators


Results of the all stars community trial

Results of the All Stars Community Trial


Sample local evaluation results from a community program in mn

Sample Local Evaluation Resultsfrom a Community Program in MN


What do you do when you have not yet succeeded

What Do You Do When YouHave Not Yet Succeeded?

Assess

Evaluate

Select a Strategy

Implement

Develop Capacity


Working backwards

Working Backwards

Evaluate

Implement

Develop Capacity


How to develop capacity

How To Develop Capacity?

  • Two predictors of quality implementation:

    • Experience

    • Training


Experience counts

Experience Counts!

  • Teachers with more experience were:

    • Most adherent when the taught LST

      (r = .630)

    • More likely to meet objectives

      (r = .590)

    • More likely to cover major points

      (r = .756)

    • More likely to make positive adaptations

      (r = .577)


Developing skill

Developing Skill


Training counts too

Training Counts, Too!

  • Teachers’ understanding of LST was strongly correlated with adherence (r = .784).


Improving understanding

Improving Understanding

  • Program-specific training

  • Coaching and feedback

  • Independent study


Program specific training

Program-Specific Training

  • Introductory training

    • Basics of theory and methods

  • Technical assistance

    • Help with specific issues and adaptations

  • Booster training

    • When important questions are asked

  • Certification of Mastery

    • A process of demonstration and certification


Does training matter

Does Training Matter?

  • Video Training Project

  • Two conditions

    • 3-hour course without video

    • 3-hour course with video

  • Topic: Norm Setting

  • Knowledge pretest-posttest survey

http://www.PreventionABCs.com


All stars certification of mastery

All Stars Certification of Mastery

  • Basic All Stars facilitator training

  • Implement one cycle of All Stars

  • Videotape implementation with feedback about mechanisms of delivery

  • Videotape implementation with feedback about interactivity

  • Implement Strategies for Success

  • Implement Parent Intervention

  • Complete online course (Prevention ABCs)

  • Videotape to demonstrate expert delivery

  • Improved student outcomes


Just in time support

Just-In-Time Support

  • New project

  • Life Skills Training

  • Emailed helpful hints just before you teach

  • Links to a streaming video demonstration

  • Recruiting test schools

    [email protected]

    1-888-692-8412


Conclusion

Conclusion

  • I was asked to answer two questions:

    • How can prevention programs use data to improve program effectiveness?

    • How can programs become data-driven?


Improving effectiveness

Improving Effectiveness

  • How can prevention programs use data to improve program effectiveness?

    • Quality of implementation data

    • Behavioral outcome data

    • Targeted mediating variable data

  • All improve the potential of a program to be implemented with greater rigor


Improving effectiveness1

Improving Effectiveness

  • How can programs become data-driven?

    • Collect data

    • Look at the data you have collected

    • Start with modest expectations

    • Find meaning in the data

    • Find alternatives in the data


  • Login