Presentation to Joint Meeting of DG REGIO Evaluation Network and ESF Evaluation Partnership

1. Making an Appropriate Choice of Methods to Evaluate Cohesion PolicyHarvey Armstrong, University of Sheffield Presentation to Joint Meeting of DG REGIO Evaluation Network and ESF Evaluation Partnership 8 July 2011, Gdansk

2. Key Fifth Cohesion Report themes (pp256-7) • Move on from preoccupation with financial ‘absorption’ • Fewer, simpler policy priorities (‘concentration’) • Focus on results and impacts (i.e. performance). • More rigorous evaluation (e.g. CIE, CBA for quant; case studies for qual; beneficiary surveys for both quant and qual) • Triangulation • From productivity to well-being (happiness) and sustainability For the presentation I will focus on 4, 5 and 6

3. Making the right choice: The example of beneficiary surveys • CED priority (P6 – Targeted Action for Key Deprived Areas) in 1994-99 Yorkshire & Humber Objective 2 programme • ERDF Measure 6.21: ‘support for community based economic projects’ – 111 (of the 178) CED projects • Main types of projects funded: - Micro-finance and credit - Social enterprises (‘community businesses’) - Labour market access, training and ILM - Education - Targeted environmental improvements - Transport access to jobs - Pure capacity building

4. What we did • Two-stage sampling method: - managers of all the projects contacted for lists of beneficiary enterprises and other organizations - individual enterprises and organizations randomly sampled from the lists (simple random sampling) • Useable lists from 33 of the 111 projects, but all the biggest projects chased down • Random sampling of 242 organizations and enterprises (110 First Sector enterprises serving mainly non-local markets; 44 First Sector serving mainly local market; 15 community enterprises; 64 mixed social/market organisations; 9 purely social organisations)

5. Did we do anything right?Direct counterfactual questions

6. Did we do anything right?Lots of nice new stuff on ‘why it works’

7. What did we do wrong (in Fifth Cohesion Report terms)? • Rigour • Should have used stratified sampling method • Only one bias check (sub-areas) • No proper confidence intervals • Triangulation • Could have used open questions and a third stage of SSIs as a qual-on-quant triangulation • Could have analysed secondary data sets for quant-on-quant triangulation • Well-being and sustainability • Beneficiaries = organisations. Wrong. • Should have done residents’ survey • Should have added some sustainability questions

8. Making the right choice: beneficiary surveys guidance

9. Case studies: what’s being done well? • Example: ERDF 2000-06 Work Package 4: Structural Change and Globalisation • Thematic: effect of, and policy response to globalisation • Holistic: quantitative and qualitative • Quantitative: Three main elements: - By core team statistical analysis - By case study teams - Pilot beneficiary surveys; secondary data analysis • Qualitative: - In-depth analysis of documents - Semi-structured interviews with regional experts, policy makers and beneficiaries (176 in total)

10. Case studies: what’s being done well? • Multiple full case studies (12) • Transparent and logical selection criteria: ‘representative’ approach rather than extreme/deviant, maximum variation, critical or ‘pardigmatic’ (Flyvbjerg, 2006) • In-depth knowledge of context, including historical context (flexible – back to early 1980s for many) • Narrative: ‘telling the story’ • Mini-case studies • Common template but with flexibility for regional teams • Central core team • Generalization: accumulation of evidence; power of the ‘deviant’ case

11. Case studies: what next? • Rigour • Fundamentally a qualitative method, but needs more quant • Beneficiary surveys • Proper CIE and secondary econometric data analysis? • More rigorous qualitative methods • Triangulation • Already happening, but • Regional team SSI/document analysis and core team quant • Beneficiary survey as quant-on-qual by regional team • Well-being and sustainability • Potentially excellent method

12. Making the right choice: case studies guidance

13. From productivity to well-being: A warning from history: British enterprise zones in the 1980s • An urgent need to prioritize amongst the many well-being and sustainability measures. • ‘Concentration’ will narrow the field for each programme, but…….. still and awful lot, especially for well-being • A ‘must’ for all future evaluations: distribution of income and wealth • A warning from history: Erickson and Syms, Regional Studies, 1986. Salford-Trafford Park EZ, Manchester. • Tax reductions (especially property tax – ‘Business Rates’ • ‘Place prosperity’ versus ‘people prosperity’: who wins, who loses. Distribution: combined property tax (rates) plus real rents: 36% of value of the property tax exemption went to industrial tenants and 64% to initial property owners, many ex-regional. We may not like the results we get.

14. Reserve slides

15. From productivity to well-being and sustainability: Challenges for other methods? • Heavily affected? - cost-benefit analysis - SWOT - Evaluability assessment, logic models, concept mapping - Delphi methods, focus groups, experts panels - Observational methods - Stakeholder consultation and participative techniques - Meta-analysis • Less heavily affected? - Input-output - Macro models CGE