Evidence-Informed Program Improvement:
Download
1 / 42

Evidence-Informed Program Improvement: Using principles of effectiveness to enhance the quality and impact of youth and - PowerPoint PPT Presentation


  • 266 Views
  • Uploaded on

Evidence-Informed Program Improvement: Using principles of effectiveness to enhance the quality and impact of youth and family programs. STEPHEN SMALL Professor of Human Development & Family Studies, University of Wisconsin-Madison and

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Evidence-Informed Program Improvement: Using principles of effectiveness to enhance the quality and impact of youth and' - adamdaniel


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Evidence-Informed Program Improvement:Using principles of effectiveness to enhance the quality and impact of youth and family programs

STEPHEN SMALL

Professor of Human Development & Family Studies,

University of Wisconsin-Madison

and

Extension Human Development & Family Relations Specialist

University of Wisconsin-Extension

Iowa State Webinar

November 17, 2009


Agenda
Agenda

  • What are evidence-based programs, practices, and organizations?

  • What are some common principles of effective evidence-based youth and family programs?

  • How can Evidence-Based Program Improvement (EIPI) be used to improve existing programs?


Evidence based programs and practices
Evidence-based Programs and Practices

Years of research have demonstrated that specific approaches, practices and strategies can reduce problem behaviors and enhance positive developmental outcomes

The most effective programs and practices are termed “evidence-based” – these are the “gold standard”


Why the growing interest in evidence based programs and practices
Why the growing interest in evidence-based programs and practices?

  • Accountability: Increasingly required by funders within political climate

  • Efficiency: Reduces development costs associated with creating a new program

  • Availability of good science: Body of scientific evidence has reached a critical mass

  • Cost-effectiveness: Increased likelihood that limited resources will be used wisely

  • Marketability: Documented evidence can help sell and justify program

  • Ethics: Responsibility to use what is known to be effective


The evidence based programs and practices continuum
The Evidence-Based Programs and Practices Continuum practices?

Evidence-based Programs

Evidence-based Kernels and core elements

Evidence-based Practices

Evidence-based Principles

Evidence-based Organizational

Systems


Evidence based programs
Evidence-Based Programs practices?

A new class of programs that:

  • Are based on a sound scientific evidence and theory

  • Have been carefully implemented, evaluated and replicated using rigorous scientific methods

  • Have evaluation findings that have been subjected to critical review

  • Often “endorsed” as evidence-based by a federal agencies or respected research organizations (e.g., Campbell Collaboration, SAMHSA, CDC)


Terminology
Terminology practices?

Evidence-based Research-based

Evidence-based = Research-based + Rigorously tested


Examples of evidence based programs
Examples of Evidence-based practices?Programs

Family/Parent Education Programs

  • Nurse Family Partnership: 15-year follow-up showed ↓arrests (54%) among adolescents; $3.59 return on the dollar

  • Strengthening Families Program for Parents and Youth 10-14:↓ aggression, alcohol, tobacco, marijuana, meth; $7.82 return


Examples of evidence based programs1
Examples of Evidence-based practices?Programs

School-based programs

  • Life Skills Training: 6-year follow-up showed ↓alcohol, tobacco, marijuana, narcotics, hallucinogens; $25.61 return on the dollar

  • Olweus Bullying Prevention Program:↓in bullying and victimization, vandalism, fighting, truancy; ↑ in school attachment


  • Examples of evidence based programs2
    Examples of Evidence-based practices?Programs

    Mentoring

    Big Brothers Big Sisters:↓ in violence, alcohol and drug use; ↑ in school performance; $1.01 return


    The effect of the strengthening families 10 14 program on teen aggressive and hostile behaviors
    The effect of the Strengthening Families 10-14 Program on teen aggressive and hostile behaviors

    Aggression and Hostility Index

    SOURCE: Spoth, R., Redmond, C., & Shin, C. (2000) Reducing adolescents' aggressive and hostile behaviors: Randomized trial effects of a brief family intervention four years past baseline. Archives of Pediatrics and Adolescent Medicine 154, 1248-1257


    What are some limitations of evidence based programs
    What are some limitations of teen aggressive and hostile behaviorsevidence-based programs?

    • Not always easy to find

    • Can be costly to implement in both time & $

    • May not address particular issues or audiences

    • Tend to be problem focused

    • May downplay local knowledge and community ownership

    • Overlooks value of promising grassroots programs

    • Often difficult to transport to new settings

    • Other?


    The evidence based programs and practices continuum1
    The Evidence-Based Programs and Practices Continuum teen aggressive and hostile behaviors

    Evidence-based Programs

    Evidence-based Kernels

    and core elements

    Evidence-based Practices

    Evidence-based Principles

    Evidence-based Organizational

    Systems


    Evidence based kernels
    Evidence-Based Kernels teen aggressive and hostile behaviors

    • Kernels = units of behavioral influence and practice that have been shown through experimental evaluation to produce reliable effects on behavior

    • Kernels =Active ingredients of effective programs


    The evidence based programs and practices continuum2
    The Evidence-Based Programs and Practices Continuum teen aggressive and hostile behaviors

    Evidence-based Programs

    Evidence-based Kernels

    Evidence-based Practices

    Evidence-based Principles

    Evidence-based Organizational

    Systems


    Definitions of evidence based practices
    Definitions of teen aggressive and hostile behaviorsEvidence-based practices

    • “The integration of the best available research with clinical expertise in the context of patient characteristics, culture and preferences.” (American Psychological Association)

    • “The integration of best-researched evidence and clinical expertise with patient values.” (Institute of Medicine of the National Academies)


    Evidence based practice
    Evidence-Based Practice teen aggressive and hostile behaviors

    Practitioner

    Expertise & Experience

    Evidence-Based

    Practices

    Research Evidence


    The evidence based programs and practices continuum3
    The Evidence-Based Programs and Practices Continuum teen aggressive and hostile behaviors

    Evidence-based Programs

    Evidence-based Kernels

    Evidence-based Practices

    Evidence-based Principles

    Evidence-based Organizational

    Systems


    EVIDENCE-INFORMED PROGRAM IMPROVEMENT: teen aggressive and hostile behaviors

    Using principles of effectiveness to enhance the quality

    and impact of family-based prevention programs

    February 2009, Family Relations, 58, 1-13.

    http://whatworks.uwex.edu


    Principles of evidence based youth and family programs
    Principles of evidence-based youth and family programs teen aggressive and hostile behaviors

    • Design & Content

    • Relevance

    • Delivery/Implementation

    • Assessment & Quality Assurance


    Program design and content
    Program design and content teen aggressive and hostile behaviors

    Effective programs…

    • Target relevant assets and/or risk and protective factors

    • Are theory-driven

    • Are of sufficient dosage and intensity

    • Are comprehensive

    • Use active learning approaches & appropriate delivery formats


    Theory driven
    Theory-driven teen aggressive and hostile behaviors

    • Based on strong, scientific theory

    • Target relevant risk, protective factors and assets to accomplish objectives

    • Logical, well-developed program theory

    A well thought out recipeis essential


    Theory driven example
    Theory driven - Example teen aggressive and hostile behaviors

    Nurse-Family Partnership (early childhood/parent ed)- grounded in theories of human ecology (Bronfenbrenner, 1979, 1992), self-efficacy (Bandura, 1977, 1982), and human attachment (Bowlby, 1969)


    Theory driven1
    Theory-driven teen aggressive and hostile behaviors

    • Based on strong, scientific theory

    • Target relevant risk, protective factors and assets to accomplish objectives

    • Logical, well-developed program theory

    A well thought out recipeis essential


    Have sufficient dosage and intensity
    Have sufficient dosage and intensity teen aggressive and hostile behaviors

    Enduring change takes effort


    Thinking about sufficient dosage and intensity
    Thinking about sufficient dosage and intensity… teen aggressive and hostile behaviors

    For a mentoring program to be effective, the relationship on average has to exist for at least…

    2 weeks

    1 month

    6 months

    9 months

    1 year


    Comprehensive
    Comprehensive teen aggressive and hostile behaviors

    Simple solutions rarely work



    Program relevance
    Program relevance format

    Effective programs…

    • Are developmentally appropriate

    • Are socio-culturally relevant

    • Are appropriately timed

    • Are sensitive to local context

    • Are responsive to individuals

    One size does not fit all


    Program delivery
    Program delivery format

    Effective programs…

    Foster good relationships

    Are delivered by well-trained and committed staff

    Address environmental barriers to program delivery and behavior change

    -Provide a safe setting for the intervention

    Are embedded in a supportive organization infrastructure


    Well trained and committed staff examples
    Well trained and formatcommitted staff - Examples

    Big Brothers, Big Sisters (mentoring program) – Orientation, volunteer screening, youth assessments, matches, supervision

    Promoting Alternative Thinking Strategies (PATHS, classroom program)- Teachers receive training in a two- to three-day workshop and in bi-weekly meetings with the curriculum consultant.


    Program assessment and quality assurance
    Program assessment and quality assurance format

    Effective programs…

    • Are well-documented

    • Focus on evaluation and refinement

    • Provide opportunities for ongoing feedback


    Make sure the program is well documented
    Make sure the program is formatwell-documented

    It’s important to understand the program in order to implement it well

    Maintaining program fidelity requires that people know what the program is about

    Program replication and evaluation is difficult if the program isn’t well documented


    Focus on evaluation refinement
    Focus on formatevaluation & refinement

    See evaluation as your friend

    Evaluation takes many forms

    Evaluation isn’t always easy

    Consider EIPI as a first step


    Evidence informed program improvement
    Evidence Informed Program Improvement format

    http://whatworks.uwex.edu/attachment/whatworks_manual.pdf


    Summary principles of effective youth and family prevention programs
    SUMMARY formatPrinciples of Effective Youth and Family Prevention Programs

    Program design and content

    Program relevance

    Program delivery

    Program assessment and quality assurance


    The evidence based programs and practices continuum4
    The Evidence-Based Programs and Practices Continuum format

    Evidence-based Programs

    Evidence-based Kernels

    Evidence-based Practices

    Evidence-based Principles

    Evidence-based Organizational

    Systems


    Evidence based organization
    Evidence-based organization format

    • Promotes an organizational culture that values science and evidence-based approaches

    • Creates an infrastructure that supports effective practices and programs:

      • Uses existing evidence-based programs when available and appropriate

      • Employs a strong theoretical/science approach to new program design

      • Provides regular opportunities to reflect on programs and effective practices

      • Uses scientific tools to assess needs/assets, understand local context and adapt programs

      • Provides ongoing technical support for selecting, implementing and evaluating effective programs and practices

      • Builds organizational and community capacity to identify relevant goals and implement appropriate and effective approaches to address these goals



    ad