Using longitudinal administrative data to evaluate area-based initiatives

Using longitudinal administrative data to evaluate area-based initiatives George Smith & David McLennan

Background Brief history Example 1: Evaluating the Kent Supporting Independence Programme (SIP) Example 2: Evaluating the national Neighbourhood Nurseries Initiative (NNI) Example 3: Evaluating the New Deal for Communities (NDC) Future developments? (Not covering ethical, legal, contractual and access issues) Outline

Administrative data: data primarily collected for administrative purposes, converted for use as a research tool Such data traditionally released in an aggregated format but increasingly available in individual format Increasingly linkable laterally (across different datasets) and over time (but still a very long way short of what is possible in Denmark and Sweden). Key strengths: potentially virtual universal coverage of many of these datasets, and regular (sometimes continuous) updating; also may be virtually the only way to study some social phenomena Weaknesses: may only contain data that is needed for administrative purposes and there may be ad hoc changes in criteria or definitions (through policy or admin changes). Administrative data and survey data: very powerful combination? Background: Using Administrative Data for Social Research

Brief history – where are we coming from? • Phase 1: late 1980s: impact ofnational1988 welfare benefit changes assessed by using an individual claimant extract from an LA housing benefit (HB) system taken before the reforms to estimate the post 1988 reform ‘gains and losses’. • Phase 2: mid 1990s: local longitudinal studies (e.g. Lone Mothers Moving On and Off Benefit), using linked HB extracts to plot benefit dynamics. Such data increasingly well postcoded (and linkable through LA specific ‘HBref’). • Phase 3: 1999 onwards: access by SDRC to national data systems at individual level (e.g. DSS/DWP) for the new Index of Multiple Deprivation (ID2000), led on to longitudinal linkage via encrypted NINOs e.g. for the SEU (e.g. Changing Fortunes , 2001, Growing Together or Growing Apart? 2002, which drew on individually linked welfare benefit data over 5 years, to give both national and local trends). • Phase 4: 2002 onwards: using such linked datasets to assess local programmes. Key strengths of such data are that it is already collected in a consistent form nationally. Is typically more than 99% postcoded and can be linked over time. • IMD work gave access to a very wide range of individual level admin data (welfare benefits, employment, education, health, crime). Strong focus on local areas but within a setting where we needed consistent information across all areas (a prime requirement for the IMD). • Previous experience with attempts to evaluate local area based programmes suggests that unless programmes are highly focussed and targeted, any effects are likely to be very small.

Example 1: Evaluating the Kent ‘Supporting Independence Programme’ (SIP) • Study commissioned by HM Treasury 2002-5 (and follow up to 2006/7) to evaluate impact of Kent’s SIP programme as part of the local Public Service Agreement (LPSA) - Target 12 ‘to reduce dependency and increase employment .. across the whole of Kent with an initial focus on particular districts’. • SIP was essentially an umbrella programme of local initiatives more or less targeted at Kent’s ‘priority wards’ (initially in East Kent) and later expanded to 33 wards in all 12 districts. These had a claimant population of about 18,000 in 2001. • The SIP programme had already begun by the time the evaluation was commissioned; it was a series of loosely targeted initiatives, in some cases with new facilities e.g. increased childcare; in other cases with advice centres or enhanced coordination between local and national agencies. • Some of the key outcomes could be directly measured using national welfare benefits data. So an opportunity to test the use of admin data in a local evaluation. Was it possible?

The Kent LPSA Evaluation Stage 1: Baseline of ‘benefit spend’, based on data extract for April 2001, for all wards and districts in Kent. Numbers of cases and actual ‘spend’. Stage 2: Repeated cross-sectional extracts at yearly intervals of major national welfare benefits (IS, JSA. IB/ SDA), allowing trends over time for the target and other areas.

Stage 3: Monitoring individual change in claiming patterns over time, and tracking geographical mobility on benefit. Key to uncovering how far area change is affected by movement into or out of benefit by claimants.Also effects of geographical mobility on claiming patterns • Stage 4: Comparing the pattern of change by claimants in the Kent priority wards, with the pattern of change in the rest of the South East of England over the same period Modelling the pattern of exits for a cohort followed from 2001-2005 on IS, JSA, IB and SDA.

Some interim results • Limited number of explanatory variables in the admin data, or available from other sources at the local area level. • No clear cut overall effects, though possible improvement in 2003 as claimants in the most highly targeted wards had exit rates in line with other areas in the SE; also slightly higher than expected exit rates for some groups in these areas e.g. younger lone parents with one child under five. • Admin data comes up with numbers large enough to identify such groups, even at a local level. But problem of attributing this apparent change to the programme, without more evidence of a direct link. Issue may not be simply ‘does it work? But for whom, when and where?’ • Further two years of data analysed to explore these patterns further. Report in preparation.

Evaluating the Neighbourhood Nursery Initiative (NNI) • NNI – national programme to increase childcare in the most disadvantaged areas (approximately 45,000 places in about 1400 centres by 2005) • Evaluation ( by groups at Oxford, Nat/Cen and IFS). covered implementation, quality of provision, neighbourhood change, and the impact of NNI provision on entry to employment,. Study made extensive use of admin data, including longitudinal DWP benefit data, WPLS and DfES PLASC data. • NNI did not serve tightly specified areas. So necessary to identify potential target areas.

NNI Rich and NNI Poor Areas • Identified all census output areas (OAs) in the 20% most disadvantaged Super Output Areas (SOAs) using the IDAC measure (itself generated from admin data). • Using individual level DfES PLASC data, analysed the distance travelled to primary school by children aged 5-7 in each area to establish the accessibility criterion for an NNI nursery. • Used this information to identify disadvantaged areas with reasonable access to NNI provision and places (‘NNI rich’) and those without (‘NNI poor’). • Propensity score matching techniques used to identify two matched groups of families with preschool children drawn from the child benefit system resident in NNI rich and NNI poor areas. Samples invited to take part in a screening survey to assess their intentions to (re)enter the labour market and make use of childcare. • Subsequent impact study (Nat/Cen and IFS) drew on the ‘work ready’ subset but was able to compare results from four different sources; those actually using NNI provision; their own self assessment of impact; ‘work ready’ families in NNI rich and poor areas, and the overall population of families with young children in these two areas. This included drawing on WPLS to check on the subsequent entry to labour market outcomes. The advantage is that the study includes both the effects on users and at the community level. One key point that emerged that even in the most highly targeted areas (NNI rich) the take up of NNI childcare was around 10% at best. So any effect on ‘users’ is significantly reduced when looking ‘community wide’. Report to be published shortly.

Tackling worklessness in NDC areas

Worklessness Worklessness: “involuntary exclusion from the labour market and receipt of selected out-of-work benefits” • Unemployment • Job Seekers Allowance (JSA) • Work-limiting illness • Incapacity Benefit (IB) • Severe Disablement Allowance (SDA) Work and Pensions Longitudinal Study (DWP)

Tackling worklessness Has the NDC programme reduced worklessness in the 39 partnership areas? (1) Area level change: • Reduction in area-level worklessness rates? (2) Individual level change: • Increased likelihood of NDC residents moving out of worklessness and into employment?

Cross-Sectional Worklessness • 24 of the 39 partnerships saw a reduction in the absolute percentage point ‘gap’ with local authority worklessness rate between 1999 and 2005 • 31 out of 39 saw a reduction in absolute gap with local authority on unemployment rate • 4 out of 39 saw a reduction in absolute gap with local authority on work-limiting illness rate • So are 31 NDCs successfully tackling unemployment?

Unemployment example • Change in unemployment rate could be due to…

Longitudinal analyses • Work and Pensions Longitudinal Study enables individuals to be tracked over time: • into, out of and between benefits and employment • geographically when claiming benefits • Enables analysis of the individual level dynamics driving area level changes

NDC Worklessness ‘inflows’ and ‘outflows’

Programme impact? • Has the NDC programme increased the likelihood of people leaving workless? And entering employment? • Need to track people over time and control for individual, household and area level factors to ‘isolate’ any programme effect • Propensity score matching and Difference-in-Difference modelling using WPLS

Previous results • NDC Evaluation Phase 1 results (June 2005): • Unemployed NDC residents were 1.1 times more likely to exit workless benefits compared to control group • NDC residents unable to work due to limiting illness were 1.6 times more likely to exit workless benefits compared to control group But… • Used GMS-ONE • Unable to track people into employment • Control group matched across whole of England

Current developments • NDC Evaluation Phase 2 analysis: • Using WPLS • Tracking people into employment • Control group matched to people in similarly deprived local neighbourhoods • Results forthcoming: spring/summer 2007

Next Steps / Future Developments

NRF areas Non- NRF areas Next Steps: WPLS • Add evaluations: • E.g. Neighbourhood Renewal Fund

Possible outcome measures include: Likelihood of: • Exiting to employment • Exiting to sustained employment • Returning to workless benefits NRF areas Employment Zones NDC areas Future possibilities: WPLS • Compare (people within) programmes:

Future possibilities : WPLS • Find cumulative impact of people and place programmes New Deal 25+ Employment Zones NRF areas

No programme Future possibilities: WPLS • Find cumulative impact of combinations people and place programmes People based programme Place based programme Both programmes

Access to Higher Education (UCAS) Pupil Level Annual School Census & National Pupil Database (DfES) Post 16 Qualifications, such as ILR, SERAP, NVQD (DfES) Future possibilities:Multi-Source Data Linkage • Assessing the impact of change in one outcome measure on change in another - Linking data on the same person

Work and Pensions Longitudinal Study Health data: e.g. GP Prescription Data? Future possibilities:Multi-Source Data Linkage 6. Assessing the impact of change in one outcome measure on change in another - Linking data on the same person

Work and Pensions Longitudinal Study (DWP) Child Benefit records (HMRC) Pupil Level Annual School Census & National Pupil Database (DfES) Future possibilities:Multi-Source Data Linkage 7. Assessing the impact of change in one outcome measure on change in another - Linking data on different people

Conclusion • Huge increase in interest and usage of longitudinal linkage of administrative data • Invaluable for unpicking individual-level dynamics driving area-level changes • Invaluable for small area analyses – surveys rarely robust at neighbourhood level • Great deal of untapped potential – especially for targeting, monitoring and evaluating interventions to tackle social disadvantage

Using longitudinal administrative data to evaluate area-based initiatives