evaluating outcomes of prevention programs
Download
Skip this Video
Download Presentation
Evaluating Outcomes of Prevention Programs

Loading in 2 Seconds...

play fullscreen
1 / 62

Evaluating Outcomes of Prevention Programs - PowerPoint PPT Presentation


  • 103 Views
  • Uploaded on

Evaluating Outcomes of Prevention Programs. FRIENDS National Resource Center for Community Based Child Abuse Prevention A Service of the Children’s Bureau www.friendsnrc.org. Overview of the Day. Review of Logic Models: a template for planning, evaluating, and reporting

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Evaluating Outcomes of Prevention Programs' - stian


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
evaluating outcomes of prevention programs

Evaluating Outcomes of Prevention Programs

FRIENDS National Resource Center for Community Based Child Abuse Prevention

A Service of the Children’s Bureau

www.friendsnrc.org

overview of the day
Overview of the Day
  • Review of Logic Models: a template for planning, evaluating, and reporting
  • Using data to develop and maintain a CQI environment
  • Administering the Protective Factors Survey (PFS)
  • Identifying other methods for measuring outcomes
  • Managing and reporting on outcomes
slide3

Tell me, why evaluate?

If you measure it, it’s more likely to get done.

If you don’t measure results, how do you know you’ve succeeded?

And, how do you tell others what you’ve done? How do you learn what needs improvement?

We also need to show funders, staff, & parents that their time, money & efforts are well spent.

Plus, with limited funding opportunities . . . .

Good evaluation could lead to better chances of funding?

Well. . . yes

slide4

Evidence-based and evidence-informed programs require evaluation

I’m convinced. . . How do I get started?

It all begins with a logic model . . .

slide5

What is that fancy one-paged chart?

Is it a program map? A tool for planning evaluation? A report template? Is it a requirement for grant applications?

It’s all of that and more! Friends, it’s a Logic Model!

slide6

A Program Map? !

  • Yes!
  • A logic Model outlines:
  • Where you want to go.
  • How you plan to get there
  • What will you see that tells you you’ve arrived at your destination
      • If we do this . . . Then this will happen . . .
slide7

A tool for planning evaluations?

Yes! A logic model can specify the tools that will be used to measure to what degree you achieved your desired outcomes

slide8

A report template?

Yes!

When you’re ready to report on your services, pull out your logic model and use it as a template to organize your report.

slide9

A requirement for grant applications?

Yes!

Most grant applications are asking for a logic model as part of the proposal.

Don’t leave home without one.

logic model components
Logic Model Components
  • Vision
  • Population
  • Services
  • Assumptions
  • Resources
  • Outcomes
  • Indicators
  • Measurement Tools

Remember: These components sometimes go by different names.

vision goal impact long term outcome
Vision (Goal, impact, long-term outcome)

A broad statement of well-being; the long-term impact on social, economic or environmental conditions.

Examples:

* Children &Youth are Nurtured, Safe & Engaged

* Families are Strong & Connected

* Communities are Caring and Responsive

* From Pathways to the Prevention of Child Abuse & Neglect, Pathways Mapping Initiative , Schorr, & Marchand, 2007

http://www.dss.cahwnet.gov/CDSSWEB/entres/pdf/Pathway.pdf

population consumers participants audience
Population (consumers, participants, audience)
  • Who are your participants?
  • What are their characteristics and demographics?
  • What are their needs?
  • How do you know?
    • Needs assessments
    • Focus groups
services outputs activities
Services (outputs, Activities)
  • Based on the needs assessment of your target population-
    • What services are you providing/will you provide?

(You will revisit this section as you move further in your logic model)

    • What will they look like?
    • How frequently will they occur?
assumptions theory of change research base
Assumptions (theory of change, research base)
  • What assumptions are you making that suggest your services will be successful?
  • Services should be based on what is known to be effective.
  • Your research into assumptions should result in understanding the degree to which you are providing evidence-based services.
assumptions continued
Assumptions, continued

If uncertain of the empirical foundations of your approach, now is the time to stop and investigate what works. This link will help get you started:

http://www.childwelfare.gov/preventing/programs/whatworks/index.cfm

The FRIENDS website has additional guidelines on Evidence-Based Practices. Please check the site regularly http://www.friendsnrc.org

slide16

Services

  • Twelve 1 hour parenting education sessions using the ”Safe Homes” curriculum.
  • Access to concrete materials (such as fencing, stair gates, car seats, etc.)

Assumptions

Two evaluations of the Safe Homes curriculum (Fisher, 2000, Hale & Carter, 2003), each conducted over a 3-year period, found that children of parents who had participated in the program were 50% less likely to experience preventable injuries than children in comparison groups.

resources inputs
Resources (inputs)

Do you have the infrastructure needed?

  • Based on the services you identify,
    • What resources do you need?
    • If you don’t have them, can you get them?

The Implementation Plan worksheet on the FRIENDS web-site can help. http://www.friendsnrc.org/download/dtappendices.pdf

page 45

outcomes goals objectives
Outcomes (Goals, objectives)

Movement towards or attainment of a change in attitude, belief, behavior.

Ask yourself: “If you conduct your program’s activities, and succeed with participants, what do they believe, know, have or do as a result?

short term outcomes engagement knowledge
Short-Term Outcomes(engagement, knowledge)
  • Engagement outcomes: First steps: gaining trust, interest and involvement
  • Learning outcomes: Changes in knowledge, attitudes, beliefs
intermediate outcomes practice
Intermediate Outcomes(practice)

Intermediate

  • Moving from learning to action
  • Practicing and refining new skills
  • Adopting or changing behaviors
long term outcomes integration
Long-Term Outcomes(integration)

Long-term

  • When participants have integrated new behaviors into their lives over time.
  • Consistently practicing new skills and behaviors
writing outcome statements
Writing Outcome Statements

Who (example: Participants)

Will Do (example: will describe)

What (example: a child-safe home)

The FRIENDS Evaluation toolkit has examples of outcomes a click away! http://www.friendsnrc.org/outcome/toolkit/outin.htm

slide24

Formatting Outcomes

  • Short-term: Participants demonstrate knowledge of how to appropriately supervise their children.
  • Intermediate: Participants provide appropriate supervision of their children.
  • Long-term: Participants provide appropriate supervision according to the changing needs of the children as they grow and develop.
slide25

Outcomes need indicators

Desired Change

Evidence of Change

indicators performance indicators
Indicators (performance indicators)
  • Indicators are specific signs needed to track progress
  • Indicators answer the question: What would I see or hear that would tell me that the outcome was being achieved?
  • Can spell out a level of achievement to attain in #’s or %’s
participants appropriately manage child behavior
Participants appropriately manage child behavior.

Participants encourage positive behaviors

Participants set realistic, age-appropriate household rules.

Participants consistently enforce their household rules.

Participants enforce rules without coercion or physical punishment

participants encourage positive behaviors
Participants encourage positive behaviors

Participants state expectations clearly

Participants look for their children’s “good” behavior

Participants use words of encouragement to their children

Participants model appropriate behavior

setting benchmarks
Setting Benchmarks
  • You may need to identify performance targets (benchmarks) for each indicator).
  • If possible, set benchmarks based on earlier evaluations.
  • If you need to set benchmarks before you provide service, consult with peers to set realistic benchmarks and adjust them based on your evaluation findings.
slide30

Parents demonstrate knowledge of how to provide a safe home environment for their children.

80% of the participants name 6 features of a child-safe house.

80% of the participants identify at least 6 common household hazards.

80% of participants describe how to eliminate or reduce at least 6 household hazards.

measures
Measures

How will you know if you’ve achieved targeted outcomes?

Outcomes

Indicators

measurement tools test whether the indicator was achieved
Measurement Tools “test” whether the indicator was achieved.

Outcome

Student is proficient in 4th grade math.

One Indicator:

Student knows multiplication tables up to 12

One Measure:

Student scores 90% in the Tedious Tables in the Nuts about Numbers text book

participants foster their infant s social emotional development
Participants foster their infant’s social/emotional development

Parents correctly identify their infants verbal and non-verbal cues

Subscale 3 on the NCAST PCI Feeding and Teaching Scales.

selecting measurement tools
Selecting Measurement Tools

Look for tools that are

  • linked to outcomes
  • culturally appropriate
  • valid and reliable
  • practical (think time & money)

Compendium of annotated tools: http://www.friendsnrc.org/outcome/toolkit/annot.htm

simple counts checklists
Simple Counts & Checklists
  • Children receiving a dental check-up
  • Parents who attended a parent/teacher conference
  • Mothers who received pre-natal care
  • Number of fathers who maintained regular contact with their children
example counts and checklists

Is your home safe for children? Bathroom:

All medications are kept out of reach and sight of my children.

Chemicals I use for cleaning sinks, toilets and floors are inaccessible to children.

Example Counts and Checklists
public agency records
Public Agency Records
  • Police Reports
  • CPS Records
  • Court Records
  • School Records

A potentially good data source but there’s a lot to consider before using them.

Informed consent, reliability of data, accessibility . . .

slide38

You may choose to construct your own tools This should be undertaken very cautiously. Check out the FRIENDS Evaluation Toolkit for information. friendsnrc.org

qualitative data
Qualitative Data

Captures information that evaluation tools can’t always capture. Can be gathered through

Surveys, interviews, focus groups, key informants, case notes, observations

For more information:

FRIENDS Guide to Qualitative Evaluation: http://www.friendsnrc.org/download/outcomeresources/qualitativedata.pdf

slide40

“The horror of that moment,” the King went on, “I shall never, NEVER forget!”

“You will, though,” the Queen said, “if you don’t make a memorandum of it.”

Lewis Carroll,

Through the Looking Glass

continuous quality improvement is
Continuous Quality Improvement is.

a process to ensure programs are systematically and intentionally increasing positive outcomes for the families they serve. It is about getting better and better. 

Instead of “if it ain’t broke, don’t fix it,” CQI emphasizes that even if programs aren’t broken, they can always be improved.”

FRIENDS National Resource Center for Community Based Child Abuse Prevention

A Service of the Children’s Bureau

plan do study act cycle
Plan, Do, Study, Act Cycle

PLAN services that are

  • responsive to consumer needs & desires
  • evidence based or evidence informed
  • mapped in a logic model

Develop a manual

Plan your evaluation methods

FRIENDS National Resource Center for Community Based Child Abuse Prevention

A Service of the Children’s Bureau

slide43
Do

Provide services

  • Hire, train and supervise staff
  • Document activities
  • Monitor fidelity
  • Collect data
      • formally
      • informally

FRIENDS National Resource Center for Community Based Child Abuse Prevention

A Service of the Children’s Bureau

study
Study

Study your data

  • formally, in the course of staff supervision, full staff meetings, board meetings, and
  • informally, through daily discussions with staff and participants; self-assessment of job performance, observation of day-to-day participant progress and satisfaction.

FRIENDS National Resource Center for Community Based Child Abuse Prevention

A Service of the Children’s Bureau

slide45
Act

Act by adjusting practices

  • formally, at the agency level by adopting new practices, programs, policies and procedures based on findings
  • informally, by making personal adjustments to improve job performance

Plan, Do, Study, Act may begin as a linear process but soon all steps are happening all the time.

what drives cqi
What Drives CQI?
  • System Wide
  • Consumer Involvement
  • Logic Model
  • Manual
  • Staff
  • Evaluation
  • Data Driven Decision Making

Safe Environment

cqi self assessment tool
CQI Self Assessment Tool
  • What CQI activities are occurring in your agency?
  • How you would like to strengthen the atmosphere of CQI in your agency?
the protective factors survey pfs
The Protective Factors Survey (PFS)

It is difficult to show that a program prevented negative things from happening

Increasing protective factors minimizes the risk for maltreatment

The Protective Factors Survey is one way to measure the effectiveness of prevention programs.

protective factors
Protective Factors
  • There is no single cause of child maltreatment.
  • a number of risk factors or attributes commonly associated with maltreatment.
  • a number of protective factors to reduce the occurrence of child maltreatment.
risk factors
Risk Factors

Parent Factors – mental health, substance abuse, attitudes and knowledge, age

Family Factors – economics, stress, domestic violence, household size,

Child Factors – Age, disabilities, temperament

Environmental Factors – poverty, unemployment, lack of social support, and community violence

understanding the risk factors
Understanding the Risk Factors

Even though certain factors are present where maltreatment occurs, this does not mean that the presence of these factors will always result in child abuse and neglect.

Most people living in poverty do not harm their children.

protective factors1
Protective Factors

Factors that can protect families and promote resilience.

A Protective Factors framework focuses on prevention strategies based on building strengths with families rather than exclusively on risks and deficits.

the relationship between the pfs other measures
All four subscales of the PFS were:

significantly negatively correlated with child abuse potential and stress

significantly positively related to optimism and the adaptive coping strategy of positive reframing

All subscales except Concrete Support were significantly negatively related to depression and positively related to positive affect

The relationship between the PFS & other measures
predictive validity of the pfs
Predictive validity of the PFS
  • Time One PFS subscales were negatively related to Time Two stress and depression
  • Time One PFS subscales were positively related to all six subscales of the Rand Health Survey at Time Two
survey administration

Staff preparation

  • Setting
    • Individual or group
    • Informed Consent
  • Instructions
    • Scripts
    • Paraphrasing

Survey Administration

data management
Data Management

Enter data

Store raw data in secure location

Destroy surveys within prescribed time frames

FRIENDS PSF Database

resources
Resources
  • Survey materials
    • Training manual, PFS technical information handout, survey, reports all available at www.friendsnrc.org
    • FRIENDS database
how can i find other evaluation tools
How can I find other evaluation tools?

The FRIENDS Evaluation Toolkit and Logic Model Builder includes a compendium of evaluation tools.

Access it at:

http://www.friendsnrc.org/outcome/toolkit/annot.htm

reporting on outcomes
Reporting on Outcomes

The report should include:

  • Executive Summary
  • Body of the Report
    • Use your logic model as a map for developing this major section of the report
  • Conclusion
the body of the report
The Body of the Report

Population Targeted/Population Served

Services Planned/Services Provided

(include info on Implementation & Fidelity)

Assumptions/Underlying theory

Outcomes Projected

Evaluation Methodology

Outcome Achievement

Analysis of Results

Future Planning

contact information
Contact Information

Casandra Firman

Training and Technical Assistance CoordinatorFRIENDS National Resource Center for Community Based Child Abuse Prevention(360) 769-7167 [email protected]

FRIENDS National Resource Center for Community Based Child Abuse Prevention

A Service of the Children’s Bureau

ad