Evaluating outcomes of prevention programs
1 / 62

Evaluating Outcomes of Prevention Programs - PowerPoint PPT Presentation

  • Updated On :

Evaluating Outcomes of Prevention Programs. FRIENDS National Resource Center for Community Based Child Abuse Prevention A Service of the Children’s Bureau www.friendsnrc.org. Overview of the Day. Review of Logic Models: a template for planning, evaluating, and reporting

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Evaluating Outcomes of Prevention Programs' - stian

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Evaluating outcomes of prevention programs

Evaluating Outcomes of Prevention Programs

FRIENDS National Resource Center for Community Based Child Abuse Prevention

A Service of the Children’s Bureau


Overview of the day
Overview of the Day

  • Review of Logic Models: a template for planning, evaluating, and reporting

  • Using data to develop and maintain a CQI environment

  • Administering the Protective Factors Survey (PFS)

  • Identifying other methods for measuring outcomes

  • Managing and reporting on outcomes

Evaluating outcomes of prevention programs

Tell me, why evaluate?

If you measure it, it’s more likely to get done.

If you don’t measure results, how do you know you’ve succeeded?

And, how do you tell others what you’ve done? How do you learn what needs improvement?

We also need to show funders, staff, & parents that their time, money & efforts are well spent.

Plus, with limited funding opportunities . . . .

Good evaluation could lead to better chances of funding?

Well. . . yes

Evaluating outcomes of prevention programs

Evidence-based and evidence-informed programs require evaluation

I’m convinced. . . How do I get started?

It all begins with a logic model . . .

Evaluating outcomes of prevention programs

What is that fancy one-paged chart? evaluation

Is it a program map? A tool for planning evaluation? A report template? Is it a requirement for grant applications?

It’s all of that and more! Friends, it’s a Logic Model!

Evaluating outcomes of prevention programs

A Program Map? ! evaluation

  • Yes!

  • A logic Model outlines:

  • Where you want to go.

  • How you plan to get there

  • What will you see that tells you you’ve arrived at your destination

    • If we do this . . . Then this will happen . . .

Evaluating outcomes of prevention programs

A tool for planning evaluations? evaluation

Yes! A logic model can specify the tools that will be used to measure to what degree you achieved your desired outcomes

Evaluating outcomes of prevention programs

A report template? evaluation


When you’re ready to report on your services, pull out your logic model and use it as a template to organize your report.

Evaluating outcomes of prevention programs

A requirement for grant applications? evaluation


Most grant applications are asking for a logic model as part of the proposal.

Don’t leave home without one.

Logic model components
Logic Model Components evaluation

  • Vision

  • Population

  • Services

  • Assumptions

  • Resources

  • Outcomes

  • Indicators

  • Measurement Tools

Remember: These components sometimes go by different names.

Vision goal impact long term outcome
Vision evaluation(Goal, impact, long-term outcome)

A broad statement of well-being; the long-term impact on social, economic or environmental conditions.


* Children &Youth are Nurtured, Safe & Engaged

* Families are Strong & Connected

* Communities are Caring and Responsive

* From Pathways to the Prevention of Child Abuse & Neglect, Pathways Mapping Initiative , Schorr, & Marchand, 2007


Population consumers participants audience
Population evaluation(consumers, participants, audience)

  • Who are your participants?

  • What are their characteristics and demographics?

  • What are their needs?

  • How do you know?

    • Needs assessments

    • Focus groups

Services outputs activities
Services (outputs, Activities) evaluation

  • Based on the needs assessment of your target population-

    • What services are you providing/will you provide?

      (You will revisit this section as you move further in your logic model)

    • What will they look like?

    • How frequently will they occur?

Assumptions theory of change research base
Assumptions evaluation(theory of change, research base)

  • What assumptions are you making that suggest your services will be successful?

  • Services should be based on what is known to be effective.

  • Your research into assumptions should result in understanding the degree to which you are providing evidence-based services.

Assumptions continued
Assumptions, continued evaluation

If uncertain of the empirical foundations of your approach, now is the time to stop and investigate what works. This link will help get you started:


The FRIENDS website has additional guidelines on Evidence-Based Practices. Please check the site regularly http://www.friendsnrc.org

Evaluating outcomes of prevention programs

  • Services evaluation

  • Twelve 1 hour parenting education sessions using the ”Safe Homes” curriculum.

  • Access to concrete materials (such as fencing, stair gates, car seats, etc.)


Two evaluations of the Safe Homes curriculum (Fisher, 2000, Hale & Carter, 2003), each conducted over a 3-year period, found that children of parents who had participated in the program were 50% less likely to experience preventable injuries than children in comparison groups.

Resources inputs
Resources (inputs) evaluation

Do you have the infrastructure needed?

  • Based on the services you identify,

    • What resources do you need?

    • If you don’t have them, can you get them?

The Implementation Plan worksheet on the FRIENDS web-site can help. http://www.friendsnrc.org/download/dtappendices.pdf

page 45

Outcomes goals objectives
Outcomes evaluation(Goals, objectives)

Movement towards or attainment of a change in attitude, belief, behavior.

Ask yourself: “If you conduct your program’s activities, and succeed with participants, what do they believe, know, have or do as a result?

Short term outcomes engagement knowledge
Short-Term Outcomes evaluation(engagement, knowledge)

  • Engagement outcomes: First steps: gaining trust, interest and involvement

  • Learning outcomes: Changes in knowledge, attitudes, beliefs

Intermediate outcomes practice
Intermediate Outcomes evaluation(practice)


  • Moving from learning to action

  • Practicing and refining new skills

  • Adopting or changing behaviors

Long term outcomes integration
Long-Term Outcomes evaluation(integration)


  • When participants have integrated new behaviors into their lives over time.

  • Consistently practicing new skills and behaviors

Writing outcome statements
Writing Outcome Statements evaluation

Who (example: Participants)

Will Do (example: will describe)

What (example: a child-safe home)

The FRIENDS Evaluation toolkit has examples of outcomes a click away! http://www.friendsnrc.org/outcome/toolkit/outin.htm

Evaluating outcomes of prevention programs

Formatting Outcomes evaluation

  • Short-term: Participants demonstrate knowledge of how to appropriately supervise their children.

  • Intermediate: Participants provide appropriate supervision of their children.

  • Long-term: Participants provide appropriate supervision according to the changing needs of the children as they grow and develop.

Evaluating outcomes of prevention programs

Outcomes need indicators evaluation

Desired Change

Evidence of Change

Indicators performance indicators
Indicators evaluation(performance indicators)

  • Indicators are specific signs needed to track progress

  • Indicators answer the question: What would I see or hear that would tell me that the outcome was being achieved?

  • Can spell out a level of achievement to attain in #’s or %’s

Participants appropriately manage child behavior
Participants appropriately manage child behavior. evaluation

Participants encourage positive behaviors

Participants set realistic, age-appropriate household rules.

Participants consistently enforce their household rules.

Participants enforce rules without coercion or physical punishment

Participants encourage positive behaviors
Participants encourage positive behaviors evaluation

Participants state expectations clearly

Participants look for their children’s “good” behavior

Participants use words of encouragement to their children

Participants model appropriate behavior

Setting benchmarks
Setting Benchmarks evaluation

  • You may need to identify performance targets (benchmarks) for each indicator).

  • If possible, set benchmarks based on earlier evaluations.

  • If you need to set benchmarks before you provide service, consult with peers to set realistic benchmarks and adjust them based on your evaluation findings.

Evaluating outcomes of prevention programs

Parents demonstrate knowledge of how to provide a safe home environment for their children.

80% of the participants name 6 features of a child-safe house.

80% of the participants identify at least 6 common household hazards.

80% of participants describe how to eliminate or reduce at least 6 household hazards.

Measures environment for their children.

How will you know if you’ve achieved targeted outcomes?



Measurement tools test whether the indicator was achieved
Measurement Tools environment for their children. “test” whether the indicator was achieved.


Student is proficient in 4th grade math.

One Indicator:

Student knows multiplication tables up to 12

One Measure:

Student scores 90% in the Tedious Tables in the Nuts about Numbers text book

Participants foster their infant s social emotional development
Participants foster their infant’s social/emotional development

Parents correctly identify their infants verbal and non-verbal cues

Subscale 3 on the NCAST PCI Feeding and Teaching Scales.

Selecting measurement tools
Selecting Measurement Tools development

Look for tools that are

  • linked to outcomes

  • culturally appropriate

  • valid and reliable

  • practical (think time & money)

Compendium of annotated tools: http://www.friendsnrc.org/outcome/toolkit/annot.htm

Simple counts checklists
Simple Counts & Checklists development

  • Children receiving a dental check-up

  • Parents who attended a parent/teacher conference

  • Mothers who received pre-natal care

  • Number of fathers who maintained regular contact with their children

Example counts and checklists

Is your home safe for children? developmentBathroom:

All medications are kept out of reach and sight of my children.

Chemicals I use for cleaning sinks, toilets and floors are inaccessible to children.

Example Counts and Checklists

Public agency records
Public Agency Records development

  • Police Reports

  • CPS Records

  • Court Records

  • School Records

A potentially good data source but there’s a lot to consider before using them.

Informed consent, reliability of data, accessibility . . .

Evaluating outcomes of prevention programs

You may choose to construct your own tools developmentThis should be undertaken very cautiously. Check out the FRIENDS Evaluation Toolkit for information. friendsnrc.org

Qualitative data
Qualitative Data development

Captures information that evaluation tools can’t always capture. Can be gathered through

Surveys, interviews, focus groups, key informants, case notes, observations

For more information:

FRIENDS Guide to Qualitative Evaluation: http://www.friendsnrc.org/download/outcomeresources/qualitativedata.pdf

Evaluating outcomes of prevention programs

developmentThe horror of that moment,” the King went on, “I shall never, NEVER forget!”

“You will, though,” the Queen said, “if you don’t make a memorandum of it.”

Lewis Carroll,

Through the Looking Glass

Continuous quality improvement is
Continuous Quality Improvement is. development

a process to ensure programs are systematically and intentionally increasing positive outcomes for the families they serve. It is about getting better and better. 

Instead of “if it ain’t broke, don’t fix it,” CQI emphasizes that even if programs aren’t broken, they can always be improved.”

FRIENDS National Resource Center for Community Based Child Abuse Prevention

A Service of the Children’s Bureau

Plan do study act cycle
Plan, Do, Study, Act Cycle development

PLAN services that are

  • responsive to consumer needs & desires

  • evidence based or evidence informed

  • mapped in a logic model

    Develop a manual

    Plan your evaluation methods

FRIENDS National Resource Center for Community Based Child Abuse Prevention

A Service of the Children’s Bureau

Evaluating outcomes of prevention programs
Do development

Provide services

  • Hire, train and supervise staff

  • Document activities

  • Monitor fidelity

  • Collect data

    • formally

    • informally

FRIENDS National Resource Center for Community Based Child Abuse Prevention

A Service of the Children’s Bureau

Study development

Study your data

  • formally, in the course of staff supervision, full staff meetings, board meetings, and

  • informally, through daily discussions with staff and participants; self-assessment of job performance, observation of day-to-day participant progress and satisfaction.

FRIENDS National Resource Center for Community Based Child Abuse Prevention

A Service of the Children’s Bureau

Evaluating outcomes of prevention programs
Act development

Act by adjusting practices

  • formally, at the agency level by adopting new practices, programs, policies and procedures based on findings

  • informally, by making personal adjustments to improve job performance

    Plan, Do, Study, Act may begin as a linear process but soon all steps are happening all the time.

What drives cqi
What Drives CQI? development

  • System Wide

  • Consumer Involvement

  • Logic Model

  • Manual

  • Staff

  • Evaluation

  • Data Driven Decision Making

Safe Environment

Cqi self assessment tool
CQI Self Assessment Tool development

  • What CQI activities are occurring in your agency?

  • How you would like to strengthen the atmosphere of CQI in your agency?

The protective factors survey pfs
The developmentProtective Factors Survey (PFS)

It is difficult to show that a program prevented negative things from happening

Increasing protective factors minimizes the risk for maltreatment

The Protective Factors Survey is one way to measure the effectiveness of prevention programs.

Protective factors
Protective Factors development

  • There is no single cause of child maltreatment.

  • a number of risk factors or attributes commonly associated with maltreatment.

  • a number of protective factors to reduce the occurrence of child maltreatment.

Risk factors
Risk Factors development

Parent Factors – mental health, substance abuse, attitudes and knowledge, age

Family Factors – economics, stress, domestic violence, household size,

Child Factors – Age, disabilities, temperament

Environmental Factors – poverty, unemployment, lack of social support, and community violence

Understanding the risk factors
Understanding the Risk Factors development

Even though certain factors are present where maltreatment occurs, this does not mean that the presence of these factors will always result in child abuse and neglect.

Most people living in poverty do not harm their children.

Protective factors1
Protective Factors development

Factors that can protect families and promote resilience.

A Protective Factors framework focuses on prevention strategies based on building strengths with families rather than exclusively on risks and deficits.

The relationship between the pfs other measures

All four subscales of the PFS were: development

significantly negatively correlated with child abuse potential and stress

significantly positively related to optimism and the adaptive coping strategy of positive reframing

All subscales except Concrete Support were significantly negatively related to depression and positively related to positive affect

The relationship between the PFS & other measures

Predictive validity of the pfs
Predictive validity of the PFS development

  • Time One PFS subscales were negatively related to Time Two stress and depression

  • Time One PFS subscales were positively related to all six subscales of the Rand Health Survey at Time Two

Survey administration

  • Staff preparation development

  • Setting

    • Individual or group

    • Informed Consent

  • Instructions

    • Scripts

    • Paraphrasing

Survey Administration

Data management
Data Management development

Enter data

Store raw data in secure location

Destroy surveys within prescribed time frames


Resources development

  • Survey materials

    • Training manual, PFS technical information handout, survey, reports all available at www.friendsnrc.org

    • FRIENDS database

How can i find other evaluation tools
How can I find other evaluation tools? development

The FRIENDS Evaluation Toolkit and Logic Model Builder includes a compendium of evaluation tools.

Access it at:


Reporting on outcomes
Reporting on Outcomes development

The report should include:

  • Executive Summary

  • Body of the Report

    • Use your logic model as a map for developing this major section of the report

  • Conclusion

The body of the report
The Body of the Report development

Population Targeted/Population Served

Services Planned/Services Provided

(include info on Implementation & Fidelity)

Assumptions/Underlying theory

Outcomes Projected

Evaluation Methodology

Outcome Achievement

Analysis of Results

Future Planning

Contact information
Contact Information development

Casandra Firman

Training and Technical Assistance CoordinatorFRIENDS National Resource Center for Community Based Child Abuse Prevention(360) 769-7167 cfirman@wavecable.com

FRIENDS National Resource Center for Community Based Child Abuse Prevention

A Service of the Children’s Bureau