What do programs want
This presentation is the property of its rightful owner.
Sponsored Links
1 / 23

What Do Programs Want? PowerPoint PPT Presentation


  • 77 Views
  • Uploaded on
  • Presentation posted in: General

What Do Programs Want?. Jeanette Treiber, PhD Project Director Center for Program Evaluation and Research UC Davis School of Medicine. Objective. Determine local agencies’ evaluation capacity building needs. Background.

Download Presentation

What Do Programs Want?

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


What do programs want

What Do Programs Want?

Jeanette Treiber, PhD

Project Director

Center for

Program Evaluation and Research

UC Davis School of Medicine


Objective

Objective

Determine local agencies’ evaluation capacity building needs


Background

Background

  • California Department of Public Health (CDPH)’s Tobacco Control Program is carried out by

    • 61 Public Health Departments

    • Approximately 40 Competetive Grantees


More background

CDPH Strategy

Norm Change

Community base

Community Assessments

3yr work plans

Evaluation

More Background


Evaluation components

Evaluation Components

Local Program Evaluation requirements

  • Comprehensive Evaluation Plan

  • Process and Outcome Evaluation

  • Internal or external evaluators

  • Progress Reports

  • Final Evaluation Reports


Tobacco control evaluation center tcec

Tobacco Control Evaluation Center (TCEC)

TCEC provides Evaluation Capacity Building services to California TC programs through:

  • TA

  • Training

  • Tools

  • FER scoring

    and feedback


Rationale for tcec services

Rationale for TCEC Services

“…pressures have been accompanied by increasing awareness (and often frustration) on the part of funders as they confront the limited ability of the organizations they fund to meet the challenge of evaluating programs in ways that are likely to yield either genuine outcome-oriented accountability or useful information for formative improvement programs”Stevenson et.al. 2002


Evaluation capacity building

Evaluation Capacity Building

  • Observed need and response in past decade

  • Various approaches to CB

  • Various definitions

  • Various approaches to

    measuring effect of CB

  • Still little research


Method

Method

  • A 21 item electronic survey (using the online service surveymonkey.com) to a list of 170 project directors and evaluators.

  • Analysis of access database for technical assistance from 2004 to 2009 (approx. 4.5 yrs)


Results survey

Results (survey)

Need for training or materialsHigh orModerate

Need

Using culturally competent evaluation methods ……………….79.4 %

Writing evaluation plans ……………………………………….59.9 %

Adapting/developing data collection instruments ……………...57.1 %

Interpreting evaluation results…………………………………..54.1 %

As a reference during evaluation activities…………...................51.5 %

Writing evaluation reports……………………………………….50 %

Collecting evaluation data……………………………………….42.9 %


Projected use of tcec survey

Projected Use of TCEC (survey)

Projected useResponse Percent

Finding, developing,

or adapting data collection

Instruments………………………………………..68.6

Writing an evaluation plan ……………………….57.1

As a reference during

evaluation activities ………………………………48.6

To build overall evaluation

capacity of the project…………………………….42.9


Preferred format survey

Preferred Format (survey)

Webinar 68 %

“How-to” evaluation

guides on website 57.1%

Individual consultation

by phone or e-mail 45.7%

Website strategy

exchange 42.9%


Requests for assistance ta log

Requests for Assistance (TA Log)

Total number of TA requests between 11/2/04 and 7/16/09

512

Average: 11 requests/month

Up to 27 requests/agency total


Main request types ta log

Main Request types (TA Log)


Type of data collection instrument

Type of Data Collection Instrument

189

44

39


Ta demand

TA Demand

(79%)

(71%)


Fer score and ta requests

FER Score and TA requests

Final Evaluation report scores for Local Lead Agencies were correlated with frequency of TA request.

No statistically

significant correlation

was found.


Summary conclusion

Summary/Conclusion

  • Majority of local health departments AND community organizations in CA TC require evaluation assistance

  • Greatest need is with DCIs, developing Eval plans, and help conducting evaluation with diverse populations.

  • Most DCI dev. need is with surveys

  • Webinars are the most desired mode of training delivery

  • Individual assistance requests vary greatly in frequency, scope, and content


Limitations

Limitations

  • Response rate to survey was only 20%

  • TA assistance was not consistently entered into log in the same manner

  • Requests for TA are somewhat dependent on marketing of TA

  • TCEC team has changed personnel throughout the 4.5 years


What do programs want1

What do Programs Want?

  • Flexible,

  • Varied

  • Evaluation

  • Assistance;

  • Convenient

  • Delivery


References

References

California Department of Public Health Services, Tobacco Control Program. http://www.cdph.ca.gov/programs/tobacco/Pages/default.aspx

Innovation Network – Transforming Evaluation for Social Change. http://www.innonet.org/

Preskill, Hallie. (2005). Building Evaluation Capacity. 72 Activities for Teaching and Training. Thousand Oaks (Sage).


More resources

More Resources

Stevenson, John, Paul Florin, Dana Scott Mills, and Marco Andrade. (2002). Building Evaluation Capacity in Human Service Organizations: A Case Study. Evaluation and Program Planning 25, 233-243.

Tobacco Control Evaluation Center (TCEC). Center for Program Evaluation and Research, UC Davis. http://programeval.ucdavis.edu

University of Wisconsin Extension. Program Development and Evaluation. http://www.uwex.edu/ces/pdande/evaluation/index.html


Contact

Contact

Jeanette Treiber, PhD

Project Director

Center for Program Evaluation and Research

Tobacco Control Evaluation Center

UC Davis School of Medicine

[email protected]

http://programeval.ucdavis.edu

1616 Da Vinci Court

Davis, CA 95618


  • Login