Some perspectives on the importance of policy evaluation
1 / 19

Some perspectives on the importance of policy evaluation - PowerPoint PPT Presentation

  • Uploaded on

Some perspectives on the importance of policy evaluation. Joost Bollens HIVA- K.U.Leuven. Evaluation of ALMP’s. Why evaluate ALMP’s ? How to measure effectiveness ? Some practical issues Unanswered questions. ALMP’s. Active Labour Market Policies Training for the unemployed ;

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about ' Some perspectives on the importance of policy evaluation' - etta

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Some perspectives on the importance of policy evaluation

Some perspectives on the importance of policy evaluation

Joost BollensHIVA- K.U.Leuven

Joost Bollens

Evaluation of almp s
Evaluation of ALMP’s

  • WhyevaluateALMP’s?

  • Howtomeasureeffectiveness?

  • Some practical issues

  • Unansweredquestions

Joost Bollens

Almp s

  • Active LabourMarketPolicies

    • Training for the unemployed;

    • Private sector incentiveschemes (wage subsidies, start-upgrants,…);

    • Direct employment programmes;

    • Counseling, monitoring, job search assistance, sanctions;

Joost Bollens

Public expenditureonALMP’s

Joost Bollens

Why evaluate almp s

  • Active policies : beneficialeffects

  • Strong beliefs

  • Is thisreally the case?  Impact evaluation

    • Different programmes in one country

    • Allequallyeffective?

Joost Bollens

Why evaluate almp s1

  • Evidencebasedpolicy : givenevaluationresults, decideto:

    • Continue the programme

    • Expand the programme

    • Restructureorredesign the programme

    • Abolish the programme

  • In the end : a matter of accountability

Joost Bollens


  • Processevaluation

    • How is the programmeimplemented?, Management quality?, Proper design?, Selectionprocesses?,…

  • Impact evaluation : effectiveness

  • Efficiency : costeffectiveness

    • Twoequallyeffective programmes may have a quite different cost per participant

Joost Bollens

Impact evaluation
Impact evaluation

  • Effectiveness : a lot of possibleoutcomes

    • % of participantsthatfind a job, % thatleaveunemployment, % thatfind a stable job orstableemployment,…,

    • % thatfind a decent job, effectsonhealth, psychologicaleffects, effectsonwell-being

Joost Bollens

Gross versus net effectiveness
Gross versus net effectiveness

  • Observedoutcome : effect of programmeparticipation + effect of factors outside the programme

    • Therefore, if we observethat 6 monthsafter finishing the programme e.g. 60% of the participants do have a job, thiscannotentirelybeattributedtoprogrammeparticipation : even without participating in the programme, someunemployedwould have found a job within 6 months

Joost Bollens

Net effectiveness
Net effectiveness

  • In order tofind the proper impact of the programme (the “valueadded”, or the “net effectiveness”, or the “impact”) , we have to correct the observedgross effect :

    Net effect = [Gross effect] - [the % of participantsthatwould have found a job even without participating]

  • Sinceparticipantscannot at the same time benon-participants, the red quantitycannotbeobserved (“counterfactual”) and must beestimated

Joost Bollens

Estimating counterfactuals

  • Non-experimentalapproaches(includingquasi-experiments)

    • Several, more orless sophisticated approaches

    • Basically : compose a comparisongroup of personswho are comparabletoparticipants, BUT whodidnotparticipate

    • Potentialweakness : comparabilitynot complete, e.g. dueto (self-) selectioneffects. Example : motivation

Joost Bollens

Estimating counterfactuals1

  • Experimentalapproaches

    • Basically : take the group of personswho are willingtoparticipate in a programme, and randomlyassign half of themto a experimentalgroup, and half of themto a controlgroup

    • Experimentalgroup is allowedtoparticipate, controlgroupnot

    • Results of controlgroup serve as counterfactual

    • Advantage : betterguaranteeforcomparability, factors like e.g. motivationwillon average be the same in bothgroups

    • Strong resistance in a lot of countriestothisapproach : “unequaltreatment”. However, given the cost of ALMPs and the intrinsicuncertainty as totheireffects, thisshouldbereconsidered

Joost Bollens

Some practical issues
Some practical issues

  • Planning helps

    • Plan before the introduction of a newprogramme

    • However, avoid the evaluation of a brand newprogramme

    • Radicallychanging (orabolishing) a programmebefore the end of the evaluationmakes the resultssomewhat irrelevant

Joost Bollens

Some potential conflicts

  • Time is onourside?

    • Policy makers, evaluation sponsors, programme administrators want immediatelyevaluationresults↔evaluatorwillinsistthat a thoroughevaluationtakes time

    • Impact evaluationresultsnecessarilywillonlybeavailablesome time afterparticipation

    • The resulting “this is old stuff”-argument is not per sevalid

Joost Bollens

Some other potential conflicts

  • Different expectations: “usableinformation” (e.g. whatcanbeusedto fine tune the programme) ↔whereasevaluatorsoften are (somewhatmyopically?) in the first place interested in the validity of their impact estimates

  • Makeevaluation more usefulbyuncoveringrelationshipbetweeneffectiveness and design aspects

Joost Bollens

Some other potential conflicts1

  • Moreover, policy makers etc. onlyseemtobeinterested in impact estimateswhen these are positive, whilenegativeresultsoften are downplayedoroutrightneglected

  • (apparently?) contradictoryconclusions

  • Meta-analysiscan help

Joost Bollens

Remaining questions 1
Remainingquestions 1

Is net effectivenessrelatedto …

  • …specificgroups? What does (doesn’t) workforwhom and why (not)?

  • …combination of severalpolicies? Order?

  • …timing of intervention ?

  • …labourmarketinstitutions?

  • … intensityor “dose” orduration?

Joost Bollens

Remaining questions 2
Remainingquestions 2

Is net-effectiveness different between …

  • …public versus private provider?

  • …local versus nationalprogramme?

  • …favourable and unfavourable business cycleconditions?

  • …short run and long run ? (locking in?)

  • …sample in evaluationstudy, and futureparticipants? (externalvalidity)

Joost Bollens

Macro effects of almp s
Macro-effects of ALMP’s

  • Thusfar: effect onparticipants

  • Butalso :

    • Effect onnon-participants? Substitution, displacement, deadweight loss,….;

    • General equilibriumeffects

    • Effectsonemployment, unemployment, productivity, matchingeffectiveness, ….

  • Very important, yet a lot of uncertainty

Joost Bollens