Some perspectives on the importance of policy evaluation
1 / 19

Some perspectives on the importance of policy evaluation - PowerPoint PPT Presentation

  • Uploaded on

Some perspectives on the importance of policy evaluation. Joost Bollens HIVA- K.U.Leuven. Evaluation of ALMP’s. Why evaluate ALMP’s ? How to measure effectiveness ? Some practical issues Unanswered questions. ALMP’s. Active Labour Market Policies Training for the unemployed ;

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Some perspectives on the importance of policy evaluation' - etta

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Some perspectives on the importance of policy evaluation

Some perspectives on the importance of policy evaluation

Joost BollensHIVA- K.U.Leuven

Joost Bollens

Evaluation of almp s
Evaluation of ALMP’s

  • WhyevaluateALMP’s?

  • Howtomeasureeffectiveness?

  • Some practical issues

  • Unansweredquestions

Joost Bollens

Almp s

  • Active LabourMarketPolicies

    • Training for the unemployed;

    • Private sector incentiveschemes (wage subsidies, start-upgrants,…);

    • Direct employment programmes;

    • Counseling, monitoring, job search assistance, sanctions;

Joost Bollens

Some perspectives on the importance of policy evaluation

Public expenditureonALMP’s

Joost Bollens

Why evaluate almp s

  • Active policies : beneficialeffects

  • Strong beliefs

  • Is thisreally the case?  Impact evaluation

    • Different programmes in one country

    • Allequallyeffective?

Joost Bollens

Why evaluate almp s1

  • Evidencebasedpolicy : givenevaluationresults, decideto:

    • Continue the programme

    • Expand the programme

    • Restructureorredesign the programme

    • Abolish the programme

  • In the end : a matter of accountability

Joost Bollens


  • Processevaluation

    • How is the programmeimplemented?, Management quality?, Proper design?, Selectionprocesses?,…

  • Impact evaluation : effectiveness

  • Efficiency : costeffectiveness

    • Twoequallyeffective programmes may have a quite different cost per participant

Joost Bollens

Impact evaluation
Impact evaluation

  • Effectiveness : a lot of possibleoutcomes

    • % of participantsthatfind a job, % thatleaveunemployment, % thatfind a stable job orstableemployment,…,

    • % thatfind a decent job, effectsonhealth, psychologicaleffects, effectsonwell-being

Joost Bollens

Gross versus net effectiveness
Gross versus net effectiveness

  • Observedoutcome : effect of programmeparticipation + effect of factors outside the programme

    • Therefore, if we observethat 6 monthsafter finishing the programme e.g. 60% of the participants do have a job, thiscannotentirelybeattributedtoprogrammeparticipation : even without participating in the programme, someunemployedwould have found a job within 6 months

Joost Bollens

Net effectiveness
Net effectiveness

  • In order tofind the proper impact of the programme (the “valueadded”, or the “net effectiveness”, or the “impact”) , we have to correct the observedgross effect :

    Net effect = [Gross effect] - [the % of participantsthatwould have found a job even without participating]

  • Sinceparticipantscannot at the same time benon-participants, the red quantitycannotbeobserved (“counterfactual”) and must beestimated

Joost Bollens

Estimating counterfactuals

  • Non-experimentalapproaches(includingquasi-experiments)

    • Several, more orless sophisticated approaches

    • Basically : compose a comparisongroup of personswho are comparabletoparticipants, BUT whodidnotparticipate

    • Potentialweakness : comparabilitynot complete, e.g. dueto (self-) selectioneffects. Example : motivation

Joost Bollens

Estimating counterfactuals1

  • Experimentalapproaches

    • Basically : take the group of personswho are willingtoparticipate in a programme, and randomlyassign half of themto a experimentalgroup, and half of themto a controlgroup

    • Experimentalgroup is allowedtoparticipate, controlgroupnot

    • Results of controlgroup serve as counterfactual

    • Advantage : betterguaranteeforcomparability, factors like e.g. motivationwillon average be the same in bothgroups

    • Strong resistance in a lot of countriestothisapproach : “unequaltreatment”. However, given the cost of ALMPs and the intrinsicuncertainty as totheireffects, thisshouldbereconsidered

Joost Bollens

Some practical issues
Some practical issues

  • Planning helps

    • Plan before the introduction of a newprogramme

    • However, avoid the evaluation of a brand newprogramme

    • Radicallychanging (orabolishing) a programmebefore the end of the evaluationmakes the resultssomewhat irrelevant

Joost Bollens

Some potential conflicts

  • Time is onourside?

    • Policy makers, evaluation sponsors, programme administrators want immediatelyevaluationresults↔evaluatorwillinsistthat a thoroughevaluationtakes time

    • Impact evaluationresultsnecessarilywillonlybeavailablesome time afterparticipation

    • The resulting “this is old stuff”-argument is not per sevalid

Joost Bollens

Some other potential conflicts

  • Different expectations: “usableinformation” (e.g. whatcanbeusedto fine tune the programme) ↔whereasevaluatorsoften are (somewhatmyopically?) in the first place interested in the validity of their impact estimates

  • Makeevaluation more usefulbyuncoveringrelationshipbetweeneffectiveness and design aspects

Joost Bollens

Some other potential conflicts1

  • Moreover, policy makers etc. onlyseemtobeinterested in impact estimateswhen these are positive, whilenegativeresultsoften are downplayedoroutrightneglected

  • (apparently?) contradictoryconclusions

  • Meta-analysiscan help

Joost Bollens

Remaining questions 1
Remainingquestions 1

Is net effectivenessrelatedto …

  • …specificgroups? What does (doesn’t) workforwhom and why (not)?

  • …combination of severalpolicies? Order?

  • …timing of intervention ?

  • …labourmarketinstitutions?

  • … intensityor “dose” orduration?

Joost Bollens

Remaining questions 2
Remainingquestions 2

Is net-effectiveness different between …

  • …public versus private provider?

  • …local versus nationalprogramme?

  • …favourable and unfavourable business cycleconditions?

  • …short run and long run ? (locking in?)

  • …sample in evaluationstudy, and futureparticipants? (externalvalidity)

Joost Bollens

Macro effects of almp s
Macro-effects of ALMP’s

  • Thusfar: effect onparticipants

  • Butalso :

    • Effect onnon-participants? Substitution, displacement, deadweight loss,….;

    • General equilibriumeffects

    • Effectsonemployment, unemployment, productivity, matchingeffectiveness, ….

  • Very important, yet a lot of uncertainty

Joost Bollens