program evaluation l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Program Evaluation PowerPoint Presentation
Download Presentation
Program Evaluation

Loading in 2 Seconds...

play fullscreen
1 / 31

Program Evaluation - PowerPoint PPT Presentation


  • 160 Views
  • Uploaded on

Program Evaluation. It’s Not Just for OMB Anymore…. Questions. Think about your state What has your public investment been in workforce development in your state in the last five years? How many people participated in what services?

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Program Evaluation' - lew


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
program evaluation

Program Evaluation

It’s Not Just for OMB Anymore…

questions
Questions

Think about your state

  • What has your public investment been in workforce development in your state in the last five years?
  • How many people participated in what services?
  • What has been the net impact of participation in the services?
  • What has been the return on this investment?
it s not just a good idea it s the law
It’s Not Just A Good Idea,It’s the Law

[PL 105-220] WIA Chapter 6, Section 136(e)(1)

  • “Using funds made available under [WIA], the State, in coordination with local boards, shall conduct ongoing evaluation … of workforce investment activities …”
why evaluate
Why Evaluate?
  • “ … in order to promote, establish, implement, and utilize methods for continuously improving the activities … to achieve high-level performance …
  • Evaluation ensures that states are getting the highest possible return on the investment of public funds
  • If you don’t know what is working and what isn’t, your performance will be driven solely by happenstance and not by design
    • The best defense IS NOT accidental offense
what should evaluations of the workforce investment system include
What Should Evaluations of the Workforce Investment System Include?
  • Analysis of Customer Feedback
    • Not simply the collection of the data for WIA reporting purposes
  • Outcome Measures
  • Process Measures in the Workforce Investment System
fundamental evaluation questions
Fundamental Evaluation Questions
  • What is the program doing; what are the activities?
  • What are the customers’ reactions?
    • Participants and Employers
  • What are the results?
  • How are the results achieved?
  • What is the return on investment?
isn t monitoring enough
Isn’t monitoring enough?
  • Monitoring determines the level of compliance with administrative policies and procedures designed to interpret the legislative intent of the program.
  • Monitoring does not measure impact, process or outcomes.
okay how about reviews
Okay, how about reviews?
  • Reviewdetermines the extent to which program plans are consistent with legislative intent of the program
  • Compliance Reviews do not measure impact, process or outcomes
  • Reviews are primarily designed to ensure compliance with laws and guidance and are not an evaluation of the program
performance reporting on 17 measures must be sufficient
Performance Reporting on 17 Measures Must be Sufficient…
  • Performance Measurement determines the extent to which immediate program outputs and short-term outcomes meet performance expectations established and consistent with the legislative intent.
  • Just because a state reports high performance levels, this does not mean that they are running a program worth emulating. Only an evaluation can make that determination.
  • Success on performance measures under WIA could also reflect negotiation skills and not strong performance design
evaluation is the only way
Evaluation is the Only Way
  • Evaluation determines through longitudinal-type studies the extent to which a program is achieving its broader legislative intent in terms of producing the expected effects on the individuals or other entities for whom the program was created.
  • None of the WIA measures provide information on UI savings, the return on investment or business services.
types of evaluation
Types of Evaluation

There are seven (7) types of evaluations that satisfy

the legislative intent of the Workforce Investment Act:

  • Program Evaluation
  • Assessment of Need
  • Assessment of Program Theory and Design
  • Process/Implementation Evaluation
  • Outcome Evaluations
  • Impact Evaluations Defined by GAO
  • Cost-Benefit and Cost-Effectiveness Analysis
1 program evaluation
1. Program Evaluation
  • The systematic collection, analysis, and interpretation of information that answers questions about the efficiency and effectiveness of program implementation and outcomes, using the principles and methods of social science research.
2 assessment of need
2. Assessment of Need
  • Considers the assumptions that Employment and Training programs utilize in regards to problems they address for their target population. Problems would be determined through existing data sources, surveys, and forecasts.
3 assessment of program theory and design
3. Assessment of Program Theory and Design
  • Compares intervention specified in the program with the needs the program is expected to address, and determines whether the program plan can reasonably expect to improve the problem.
4 process implementation evaluation
4. Process/Implementation Evaluation
  • Assesses the extent to which a program is operating as it was intended. This type of evaluation can provide a linkage between the process and program outcomes (i.e. impact evaluations).
5 outcome evaluations
5. Outcome Evaluations
  • Focuses on intended and unintended outputs and outcomes to judge program effectiveness; may also employ a process evaluation to establish a complete picture of how the outcomes are produced.
6 impact evaluations defined by gao
6. Impact Evaluations:Defined by GAO
  • One that assesses the net effect of a program by comparing program outcomes with an estimate of what outcomes would have happened in the absence of the program.
  • The “gold” standard is a random assignment study to assign individuals (or …) to one of two groups: a control and an experimental group.
7 cost benefit and cost effectiveness analysis
7. Cost-Benefit and Cost-Effectiveness Analysis
  • Compares program outputs and outcomes (i.e. outcome evaluation) with the “costs” to produce them.
  • Both measure efficiency of a program using dollars expended; however, Cost Benefit Analysis aims at identifying all relevant costs and benefits of a program.
  • Cost Effectiveness Analysis compares programs with similar goals and compares their costs.
examples
Examples
  • Massachusetts
  • Washington
take away points
Take Away Points
  • Start with a question
  • Let question(s) drive the choice of evaluation design/study
  • Remember that often mixed methods will be needed
  • Avail yourself of resources out there
  • Start small to make your point (“proof of concept” studies)
  • Maximize secondary analyses of available administrative data

It can be done – It is not an unfunded mandate

what are you doing
What are you doing?
  • Audience discussion on evaluation activities
websites resources
Websites/Resources
  • Government Accountability Office (GAO) http://www.gao.gov/special.pubs/erm.html
  • Social Policy and Research Associates (SPRA) http://www.spra.com/publications.shtml
  • Mathematica http://www.mathematica-mpr.com
  • Environmental Protection Agency (EPA) http://www.epa.gov/evaluate/workshop/index.htm
  • Sage Associates http://www.sageways.com/home/et_types.cfm
  • Commonwealth Corporation – Center for Research and Evaluation http://www.commcorp.org/cre/index.html
presenter information
Timothy Theberge

ETA - Region 1

617-788-0139

theberge.timothy@dol.gov

Johan Uvin

Director of Adult Literacy

RI Dept. of Elementary and Secondary Education

401-222-8950

johan.uvin@ride.ri.gov

Presenter Information