program evaluation l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Program Evaluation PowerPoint Presentation
Download Presentation
Program Evaluation

Loading in 2 Seconds...

play fullscreen
1 / 47

Program Evaluation - PowerPoint PPT Presentation


  • 126 Views
  • Uploaded on

Program Evaluation. Needs Assessment Survey of Local Coalitions. How many evaluators does it take to change a light bulb?. One to do a needs assessment One to do a feasibility study One to do a qualitative study to find out what bulb to change One to empower the bulb to change

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Program Evaluation' - base


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
program evaluation

Program Evaluation

Needs Assessment Survey of Local Coalitions

how many evaluators does it take to change a light bulb
How many evaluators does it take to change a light bulb?
  • One to do a needs assessment
  • One to do a feasibility study
  • One to do a qualitative study to find out what bulb to change
  • One to empower the bulb to change
  • One to tender a contract for further study
  • One to write performance indicators for success
  • One to do a cost benefit analysis of the best bulb to buy
  • One to do a meta-evaluation showing that all previous studies have left everyone in the dark
so how many evaluators does it take
SO.......how many evaluators does it take?

None, actually. Evaluators don't change bulbs, that's an implementation problem!

-Program Evaluation gets a lot of flack!

objectives
Objectives
  • Understand the purpose of program evaluation
  • Review local coalition survey results
most commonly asked pe question
Most commonly asked PE question
  • How do I know I’m making a difference?
  • Answer : Institute best practices…
  • With fidelity, innovation, and critical interpretation.
  • Unfortunately, outcome evaluation is expensive and rarely feasible with community intervention.
most commonly asked pe question7
Most commonly asked PE question
  • How do I know I’m making a difference?
  • Answer : Institute best practices…
  • With fidelity, innovation, and critical interpretation.
  • Unfortunately, outcome evaluation is expensive and rarely feasible with community intervention.
process evaluation helps implement best practices with fidelity
Process evaluation helps implement best practices withfidelity.
  • Is my programming “evidence-based”?
  • Nicotine patches are great. Stick one over each eye and you can't find your cigarettes.
  • Is the “best practice” appropriate for my population?
  • What does the tobacco science say about adapting best practices to my population?
  • Am I appropriately following the science?
process vs outcome evaluation
Process vs. Outcome Evaluation

Processevaluation: Determines whether best-practices were adopted properly.

Outcome evaluation: Determines whether program was successful.

process evaluation
Process Evaluation

Assesses the delivery or implementation of a program.

What happened and why? How is this different from what was planned?

To be evaluated:

  • Activities
  • Materials
  • Delivery
  • Numbers of staff
  • Audience
  • Logistics
process evaluation11
Process Evaluation

TATU Example:

How many youth participated in TATU trainings?

What were the youths’ perceptions of each training?

What were the implementation problems?

outcome evaluation
Outcome Evaluation

Measures what progress the program has made towards its goals.

To be evaluated:

  • Behaviors
  • Attitudes
  • Knowledge
  • Skills
outcome evaluation13
Outcome Evaluation

Example goals to evaluate for TATU:

Short term – youth who participate in TATU gain outreach skills.

Intermediate – TATU youth reach fellow students.

Long term – Youth who TATU participants reach actually use skills to abstain when tobacco is available.

uw tobacco surveillance evaluation program
UW Tobacco Surveillance & Evaluation Program

“It is now proved beyond doubt that smoking is one of the leading causes of statistics.”

http://www.medsch.wisc.edu/mep/

uw tobacco surveillance evaluation program15
UW Tobacco Surveillance & Evaluation Program
  • Indirectly provides outcome evaluation of Wisconsin’s coordinated tobacco control activities.
  • Fields surveys.
  • Analyzes data to monitor state trends.
  • Identifies progress and emerging challenges.
  • Future – more geographically specific information; better disparities data.

http://www.medsch.wisc.edu/mep/

conclusion uses of program evaluation
Conclusion : Uses of program evaluation
  • Integral part of every successful organization.
  • Clarifies and re-focuses program goals and strategies.
  • Verifies that current strategies are still appropriate.
  • Refreshes staff expertise in best practices and evidence-based programming.
  • Documentation for supporters, funders, and stakeholders.
survey results
Survey Results
  • Results reported here are for fully-funded coalitions ($20,000 or more).
  • 95% of fully-funded coalitions responded (40 of 42 coalitions that were contacted).
  • 76% of coalitions responded in total

(55 of 72).

  • 94% of coalitions funded at $10,000 or more responded (44 of 47).
number of full time equivalent employees ftes in wisconsin
Number of full-time equivalent employees (FTEs) in Wisconsin:
  • 33 FTEs statewide
  • 0.82 per coalition

(33 hours per week)

  • About 40¼ total staff.
slide24
Indication of urban-efficiency gains.
  • Building partnerships is vital.
slide25
Most coalitions are funded entirely through a TPCP contract.
  • This raises the potential for instability given a potential change in state government.
  • What steps are coalitions taking to diversify?
priority populations choose three
Priority Populations (Choose three)
  • Targeting youth has likely paid off.
  • After rising through 1999, youth cigarette use now drops every year.
priority populations choose three27
Priority Populations (Choose three)
  • More resources needed for 18-24 year olds.
priority populations choose three28
Priority Populations (Choose three)
  • Less than 50% target policymakers.
  • “Other” populations reported include health care providers.
coalition objectives check all that apply
Coalition Objectives(Check all that apply)

“Other” reported goals:

  • Reducing tobacco use among pregnant women
  • Increasing the tobacco tax.
major coalition activities check all that apply
Major Coalition Activities (Check all that apply)
  • Coalitions selected an average of 10 activities.
  • All selected more than one activity.
major coalition activities check all that apply31
Major Coalition Activities (Check all that apply)
  • Less than 100% recruit supporters and seek community partnerships.
  • Less than ¼ pursue alternative funds.
major coalition activities check all that apply32
Major Coalition Activities (Check all that apply)

“Other” activities:

  • working with state partners.
  • helping worksites create individual policies.
  • asthma workgroups.
  • the N-O-T program.
  • Oral health screenings in disparate populations.
current program evaluation activities check all that apply
Current Program Evaluation Activities(Check all that apply)
  • More than 50% document their basic program activities.
  • Herein lies a wealth of experience to share.
slide35

Current Program Evaluation Activities(Check all that apply)

“Other”:

  • focus groups
  • police reports
  • CDC’s SPF indicators
  • WI Wins data
  • First Breath statistics
  • parent surveys
  • oral health surveys
  • data from local agencies
in your own words
In your own words…
  • Coalitions’ answers to open-ended survey questions:
  • What questions would you want a program evaluation to answer?
  • Please describe any sources of information/data that might be available for use in program evaluation.
1 what questions would you want a program evaluation to answer
1. What questions would you want a program evaluation to answer?
  • Open-ended responses were grouped into:
  • process questions
  • outcome questions
  • technical needs
process questions
Process Questions:
  • Is the program operating as intended?
    • Are training materials and meeting times best?
  • Is the program structure optimal?
    • Are we reaching the intended population?
  • Are our activities “best practice”?
    • Are we using best practices for helping pregnant women quit?
  • Generally, what is the ranking of most- to least-valuable practices?
  • What is the need/demand for our program?
  • How important are non-cigarette tobacco products?
process questions39
Process Questions:
  • Is the program operating as intended?
    • Are training materials and meeting times best?
  • Is the program structure optimal?
    • Are we reaching the intended population?
  • Are our activities “best practice”?
    • Are we using best practices for helping pregnant women quit?
  • Generally, what is the ranking of most- to least-valuable practices?
  • What is the need/demand for our program?
  • How important are non-cigarette tobacco products?
outcome questions
Outcome Questions:
  • What difference has the program made?
    • Are we changing attitudes about second hand smoke?
    • Have our education materials impacted attitudes about smoke-free workplaces?
  • What behavior changes resulted?
    • Have pledge cards increased smoke-free homes?
  • How many/who benefits?
  • Progress on goals of multi-year action plan?
  • Do results justify the costs?
technical needs
Technical needs:
  • Where can we find detailed local statistics?
    • Where to find numbers of youth smokers, by school and by grade?
    • How can City of Milwaukee data be separated from Milwaukee county data?
  • How do we write survey questions that identify attitude and behavior changes?
    • Where can we find template surveys that can be changed to fit the needs of our community?
  • How can we use program evaluation data to promote tobacco control?
  • What are indicators of whether progress is being made?
slide42
2. Please describe any sources of information/data that might be available for use in program evaluation.
  • Open-ended responses were grouped into:
  • Qualitative
  • Quantitative
qualitative
Qualitative
  • Testimony from youth, school leaders, adults who work with youth, partners, law enforcement.
  • Documentation of earned media.
  • Minutes of Board of Health meetings.
  • Supporter data base.
  • Mid-year and year-end reports on consolidated contract tobacco objectives.
  • Community needs assessment (every 5 years).
  • Community Health Assessment data.
quantitative
Quantitative
  • Statewide surveys/data:
  • YRBS, BRFS, UW Burden of Tobacco, UW survey of smoke-free municipal and government-owned buildings, Worksite survey, WIC data on tobacco use prevalence
  • Local surveys:
  • Dane County Youth Assessment, PRIDE survey in county schools, South West Youth Survey (SWYS), evaluation of local smoke-free ordinance, county smoke free air/tobacco use survey (400 residents surveyed), community health assessment data, survey data collected by community health improvement coalitions, oral health screening results
  • Program surveys/data:
  • Friends Helping Friends (youth peer group) survey data, WI Wins compliance checks data, First Breath data, Quit-line and Fax to Quit statistics, pre- and post-test data for participants, member survey
next steps
Next Steps…
  • Potentially, there may be regional trainings in program evaluation methods in each of the DHFS service areas.
  • Show of hands:
    • No PE training or experience
    • Some PE training
    • Some PE experience
    • Some PE success
thanks
Local coalitions team:

Sue Marten, Chair

Dianna Forrester

Kathy Cahill

Themis Flores Ramos

Connie Olson

Heidi Foster

Deb Bruning

Paula Silha

Deb Gatzke

Karen Hagemann

Jody Moesch-Ebeling

T&TA

Lynn Habrik

Kathy Cahill

Deb Bruning

Mary Hilliker

Sue Marten

Randy Glysch

Cindy Musial

Sue Marten

Thanks!