Evaluating labour market policy experiments: ETU, ONE and ERA

1. Evaluating labour market policy experiments:ETU, ONE and ERA Alan Marsh Policy Studies Institute

2. Three evaluation programmes in ‘welfare-to-work’ policy • The Earnings Top-up experiment (1995-2000 – what happens when you extend Family Credit to people without children? • The ONE initiative (1999-2002) – the prototype of Jobcentre Plus • The Employment Retention and Advancement Scheme (2003-2010) – the ‘next step’ in welfare-to-work policy

3. ETU • Two versions A & B: Added typically £20-£30 a week to lowest paid worker’s wages • Ran for three years 1996-99. • Compared ‘matched’ areas (A, B and ‘Controls’), which had similarly difficult labour markets.

4. Location of ETU and Control Areas

5. Aims of the evaluation • What were the effects of ETU on the people who received it? • What was the take-up rate among eligible workers? • What were the effects on: • employment? • wages? And….Were there any unintended affects?

6. What was done • Surveys of workers, recipients, unemployed people and employers, pre- and post ETU • Qualitative studies of these, and staff discussion panels • Local labour market studies using secondary sources (JUVOS, ES vacancies, CofP, Benefit outflows, LFS, etc..) • ‘Parallel analysis’ of Family Credit records.

7. ETU evaluation surveys with potential customers, recipients and employers

8. The programme of qualitative studies

9. Secondary data analysis • ‘Before and after’ analysis of JUVOS data • Analysis of outflows from benefit • Analysis of Family Credit applications in ETU and ‘control’ areas.

10. What happened.. • ETU found its way to young, single unskilled workers. • Many were already in work but few had housing costs. • They had claimed OOW benefits in the past and were people used to unemployment and hardship • But they were a small minority of the eligible population (max 30%) • Few of these had heard of ETU

11. Underlying reasons for non-take-up • Geography: eligible workers too scattered • Social isolation: Eligible Non-claimants had quite different social networks, or none • Critical mass: Geography+isolation=low density • Skill transfer: recipients sensitised to claiming by unemployment and hardship • The lack of publicity.

12. The search for effects: employment No significant impact on…. • employment volumes, or flows, or duration • low paid recruitment • claimant volumes or flows • hours worked • no incentive response from longer-term unemployed • Employers’ policy or behaviour, except that the care home industry had got onto it…… Possible impact on job retention among lowest-paid workers entering from JSA, but low-skilled workers may have lost out to the lowest-skilled

13. The search for effects: wages • Did ETU lower wages? Probably not……. • Employers denied it….but employers who were aware of ETU did appear to offer lower entry wages • By 1999, the entry wages of medium-term unemployed people finding work were lower in ETU areas (small samples). • There were no knock-on effects among FC recipients.

14. Mean wage per week of Family Credit recipients

15. The lessons for Employment Tax Credit • It will be an efficient anti-poverty measure • It will meet the growing problem of equity with families • It will improve incentives, BUT • A clash with Housing Benefit will re-appear as a problem. • Low-take-up of ETU is not necessarily a problem, because: • eligible workers will achieve critical densities, especially alongside workers with children; • National publicity will work; • Employment gains will become visible; • Wage effects will be underwritten by the NMW. • Social gains from ‘unwaged work’

16. The ONE Pilots • Three ‘models: Basic, Call Centre and Private/Voluntary Sector models • Each in four areas • Compared with 12 matched ‘control areas’ • Target Groups: Lone parents applying for Income Support, JSA customers, and Sick and Disabled people

17. The Aims of ONE • To change the culture of the benefits system towards independence and work,; • To increase the level of sustainable employment • To put more benefit recipients in touch with the labour market • To improve the assessment and delivery of benefits to ensure clients receive an individual service that is efficient and tailored to their needs.

18. Evaluation scheme • An interview/observation study of staff practice and IT use in a selection of ONE offices by a team from the Tavistock Institute (Kelleher et al 2001). • A qualitative interview study of ONE clients and non-participants (Osgood et al 2001) • Longitudinal quantitative interview surveys of two successive cohorts of ONE clients compared with their equivalents in Control Areas (Green et al 2001) • Econometric analysis of unemployment flows (Kirby & Riley (2003) • Cost/benefit analysis (DWP)

19. Labour market entry: summary

20. Summary • ONE had no significant effect on the labour market behaviour of lone parent or JSA clients. • ONE may have increased the rate of entry of sick or disabled clients into shorter-hours work, especially in the Basic Model areas.

21. However…… • ONE moved from a voluntary to a compulsory phase. • Yet customers retained a positive view of the new service • Staff overcame fears of several kinds • ONE provide valuable guidelines for Jobcentre Plus, especially for training

22. DWP was commended by the Parliamentary Select Committee on Work and Pensions for ‘….the quality and quantity of research it has commissioned to monitor the effects of the ONE pilots...’ • [1] ‘ONE’ Pilots: lessons for Jobcentre Plus, First report of Session 2001-02, House of Commons, 426, pp 9 para 11.

23. The Employment Retention and Advancement Scheme (ERAS)

24. Employment Retention and Advancement Scheme (ERAS) • A new post-engagement service to lone parents and long term unemployed people • It is provided by Jobcentre Plus, since October 2003 • Clients given Advancement Support Advisers (ASAs) for 33 months, plus financial incentives • They follow an individually-tailored Advancement Action Plan, which • balances short-term requirements with longer term ambitions and goals • is sensitive to local labour market opportunities • lays down steps to achieve goals • is reviewed in face-to-face meetings • connects to other services to address special barriers

25. Policy aims • To promote a work-based welfare policy • To break the ‘low-pay-no-pay cycle’ • Lessen the ‘scarring effect’ of unemployment • Improve ‘job matching’ • Increase human capital • Provide longer term ‘treatment’ for barriers to work • Increase (even further) financial incentives to work and reduce marginal deductions

26. Incentives Retention and advancement bonus • Up to six payments of £400 for full time workers • PLUS: An Emergency Discretion Fund of up to £300 • Training bonus

27. Target groups • One working group: • Lone parents working part-time and receiving Working Tax Credits (WTC) • Two non-working groups (inflow): • Lone parents on the New Deal for Lone Parents (NDLP) • Unemployed people on New Deal for 25plus (ND25+)

28. Scheme Areas • Benefit Districts (70 sites - about 6% of working pop.) • South East Wales • Manchester • Derbyshire • Gateshead & South Tyneside • W. Scotland: Renfrewshire, Inverclyde, Argyll & Bute • North East London

29. The choice of a random control trial • A history of uncertain outcomes from non-experimental pilots of employment interventions (ETU, ONE) • A contrasting history of clear(er) outcomes from RCTs of similar interventions in USA (GAIN) • The compelling logical superiority of well-conducted RCTs • Strong support from the Cabinet Office and Treasury to carry out a well-conducted large-scale RCT to test their usefulness in evaluating major LM initiatives • To encourage evaluation skill transfer between US and UK, building capacity in UK

30. A simple design • Eligible clients are invited to consent to Random Assignment • Divided 50/50 into Programme and Control groups (12 month inflow n=26,000)

31. The Evaluation Programme • A Process Study – to unlock the black box • An Impact Study – to count the outcomes • A Cost Study – to find out what it really cost • A Cost-benefit Study – to find out whether the outcomes are worth the cost

32. The Process Study: aims • Provide a careful narrative of how ERA was put into practice • Explain how barriers to work, retention and advancement were addressed • Gather evidence of the joint and several effects of cash incentives and case management • Learn how clients respond to these opportunities • Understand differences in delivery • Look for resource bias • Understand the dialogue between ASAs and their clients and the judgements they make together

33. The Process Study‘Everything is data’ • TAs’ records, diaries, incident logs • Quantitative surveys of samples of P&C clients at 12, 24 and 60 months on from RA (ONS telephone surveys with face-to-face follow-ups) • Quantitative surveys of Staff • Qualitative surveys of staff, clients and employers at 6 months, 12 months

34. The Impact Study • Measures of net impact: Programme compared to Controls (surveys, LMS and IR data) • Retention: • Weeks in employment/claiming benefit post RA • Hours per week • Advancement: • Earnings/income • pay per hour • fringe benefits - holidays, sick leave, pensions • contracts of employment, better training • Hours and earnings growth in-work (selection) • Non-employment outcomes: • Family change, migration, housing tenure, education, child well-being, parental stress

35. The Cost Study • Identify the ERA program components • Locate sources of cost data for the components • Collect costs during baseline period and ERA, for ERA clients and controls • Examples: Find changes to staff costs; Determine costs of the financial incentives, training etc. • Compute net costs per client for each of the client categories (NDLP, ND25+, WTC).

36. The Cost Benefit Study • Benefits: ERA impacts on earnings, fringe benefits, improved child well-being, reduced parental stress • Costs: ERA impacts on work related costs eg.childcare and transport; Training expenses, job search counselling etc; Substitution/displacement effects; net operating costs; • Transfers: ERA impacts on tax payments, out of work benefits, tax credits, ERA financial incentives.

37. Some difficult challenges... • Carry out randomisation accurately • Gather huge amount of varied data to high standard • Not lose the control group • Solve ‘black box’ issues, such as the joint and separate effects of incentives and active case management • Understand that not all non-employment outcomes are ‘bad’ • Understand that an ‘unfavourable’ C/B assessment might still be ‘good’ • Successfully distinguish between methodological and programme outcomes. • Analyse net impacts for subgroups in different places • possible use of non-experimental techniques • define conditional impacts • variations of strength of impacts • effects of office regime