Measuring Impact – what can we learn from recent DWP evaluations?

1. Measuring Impact – what can we learn from recent DWP evaluations? Simon Marlow

2. Overview What support has been/is available to Jobcentre Plus claimants? Focus on ‘work experience’ and Future Jobs Fund Impacts Cost Benefit Analysis of Future Jobs Fund Future thoughts

3. Claimant journey for 18-24 year olds on JSA Post mid-2011 New claim Maximum of 9 months JCP offer Includes: 1) Core interventions – signing etc. 2) Flexible interventions eg. work experience, Mandatory Work Activity 3) Back to work support 4) Flexible Support Fund Work Programme – Mandatory for 2 years. Early access at 3 months for disadvantage groups and with long benefit history Post Work Programme - trial Length of JSA claim 2009 - mid-2011 New claim Flexible New Deal Phase 1 areas - 1 year Flexible New Deal – Mandatory for 1 year Early Access for some groups Young Person’s Guarantee (includes Future Jobs Fund) Flexible New Deal Phase 2 – 6 months New Deal for Young People – Mandatory Early Access for some groups

4. Work experience (WE) and Future Jobs Fund (FJF) WE – January 2011 - present Who it is for? Mainly 18-24yrs old (a few 25+) Guidance - 13wks + JSA claim duration JCP adviser discretion to refer Aim Help people gain practical experience they need to secure a job What is it? 2-8 week placement with employer Employers expected to allow time for job search where possible Conditionality Voluntary scheme once referred Continue signing; claiming benefit No sanctions for non-completion Cost - £325/per participant FJF – October 2009 – March 2011 Who was it for? Mainly 18-24 (a few 25+ in hotspots) Guidance - 26 weeks JSA duration Voluntary Aim Build skills/work experience for disadvantaged jobseekers to help them find unsubsidised employment What was it? 6m subsidised placement, 25 hrs/wk at least min wage Additional jobs, benefit local communities Organisations mainly from public sector and third sector Conditionality Voluntary, but from Apr 10 if refused, had to do ‘Community Task Fund’ (another strand of the Young Person’s Guarantee) Stop claiming benefit Cost - Employers were paid £6,500/per placement – about £650m total

5. WE and FJF reach similar levels of take up By end of Nov 11 WE starts reach 6k/month By Apr 10 FJF starts reach 6k/month

6. Methodology Want to understand programme impact on benefit receipt and employment Isolate impact by trying to establish a plausible counterfactual In absence of an RCT, Propensity Score Matching used: Regress on observable characteristics from DWP data to try and get probability of participation (controlling for selection) Match to non-participant group (JSA claimants from same period without a start on the programme) Pseudo starts given to non-participant group so that outcomes can be compared Main assumption: hope that observable characteristics can explain participation (ie needs to explain participant ‘motivation’)

7. Matching quality on observables is very good Before matching After matching

8. Work Experience – 1,300 individuals, Jan – March 2013 . Benefit Impacts – 6 ppts at 21 weeks/5 fewer days Benefit receipt rate Employment rate Employment Impact – 8ppts at 21 weeks/8 additional days

9. FJF impacts on Benefit/Welfare support – first 12,000 individuals from Oct 2009 – April 2010 Benefit impacts Over 6m - 127 fewer days 18-24m – -7 ppts/ 41 fewer days Benefit receipt rate ‘Welfare support’ (benefit or FJF job) impacts Over 6m - 33 additional days 18- 24m – average -7 ppts/ 41 fewer days ‘Welfare support’ (benefit or FJF job) receipt rate

10. FJF impacts on employment/unsubsidised employment (caveat – using HMRC P45 data) Employment impacts 0-6m – 99 additional days 18-24m – average 10 ppts/ 57 additional days Employment receipt rate Unsubsidised impacts 0- 6m – 45 fewer days 18-24m – average 10 ppts/ 57 additional days Unsubsidised employment (job other than FJF job) receipt rate

11. WE - by 21 weeks: Benefit Impact 6 ppts off benefit – overall 5 fewer days. Employment Impact 8 ppts in employment – overall 8 additional days FJF – over 2 years Benefit/Welfare Impact 0-6m – 127 fewer days off ben/33 more days on welfare support 18-24m – average 7 ppts off ben; 41 fewer days off ben Employment/Unsub employment impact 0-6m – 99 add days in employment/45 fewer days in unsub employment 18-24m – 10 ppts in employment; 57 additional days in employment Summary of Impacts • Caveats • Work experience first early cohort – only 1,300 starts; only 21 weeks • More likely than not to be overestimates for both • HMRC data poor quality so employment impacts are less robust Compare with New Deal for Young People – 64 fewer days off JSA over 4 years International job creation scheme evidence poor

12. Costs & benefits of the Future Jobs Fund If cohort is representative, then net cost to exchequer ~ £330m (against gross cost of programme - £720m)

13. Sensitivity to assumptions

14. Future • Developing our ways to measure impacts and cost benefit analysis of programmes • Extend work experience analysis • Day one support trial • Work Programme evaluation

15. References Early work experience impacts http://statistics.dwp.gov.uk/asd/asd1/adhoc_analysis/2012/early_impacts_of_work_experience.pdf Future Jobs Fund impacts, costs and benefits http://statistics.dwp.gov.uk/asd/asd1/adhoc_analysis/2012/impacts_costs_benefits_fjf.pdf Two main sources of guidance for CBA: The HM Treasury Green Book http://www.hm-treasury.gov.uk/d/green_book_complete.pdf The DWP Social-Cost Benefit Analysis Framework http://research.dwp.gov.uk/asd/asd5/WP86.pdf