120 likes | 155 Views
Theory-Based Evaluation:. Untapped potential for EU Regional Policy. Marielle Riché Evaluation Unit, DG Regional policy, European Commission Helsinki, 4th October 2012 4 October 2012. Content of paper. Evaluations in EU regional policy: state of play
E N D
Theory-Based Evaluation: Untapped potential for EU Regional Policy Marielle Riché Evaluation Unit, DG Regional policy, European Commission Helsinki, 4th October 20124 October 2012
Content of paper • Evaluations in EU regional policy: state of play • TBE are relevant for regional policy: throughout policy cycle and for policy making • TBE still has much untapped potential: too much focus on funds absorption and lack of clear theories of change • 2014-2020: new opening for TBE (stronger focus on results, new regulatory requirements) • Looking forward
Evaluations in EU regional policy • Programme and evaluation management: MS or regions (EC: ex post evaluations) • Programmes run for 9 years (2007-2013/15) • Cover research, ICT, urban generation, SMEs, environment, social infrastructures etc. • May focus on specific territories, group of people, sectors • May use "integrated approaches", may support "innovation systems" – complex programmes
Evaluations in EU regional policy • MS/Regions: some 266 evaluations since 2007 • Most on implementation issues and policy relevance, few on outcomes (27%) • Most based on traditional qualitative techniques and monitoring data • EC role: improve evaluation practice • Long-range aim: more rigorous evidence for policy making, building shared knowledge ("Regio collaboration")
TBE are relevant for regional policy • Programmes (should) rely on a theory of change • TBE are relevant throughout the programming cycle (ex ante, in itinere, ex post) • Analyse the assumptions underlying the programme • Assess the plausibility of programme achievements as planned • Assess contribution, analyse mechanisms leading to results, context, unintended effects
TBE are relevant for regional policy • Fields of intervention where counterfactual impact evaluations not possible • Complicated/complex programmes • Context analysis allows for policy adaptation and replication • Communication on programme outcomes (narratives) • Participative evaluations (partnership principle)
An untapped potential • Programmes more focused on funds absorption than on results • Evaluations carried out at the beginning of the period, focused on implementation • No explicit theories in programmes • Over-confidence in indicators to measure actual outcomes • "Myopia" of programmes (focus on first results) • Reluctance to commit to outcomes which are not "under control"
2014-2020: a new opening for more TBE • Political commitment for a "result-based" policy • New logical framework, taking into account context and other factors, where impact is the contribution to observed results • Stronger requirements for ex ante evaluations • Compulsory impact evaluations for each priority axis of programmes (covers different interlinked interventions) • Our preparatory work with Member States: guidance and pilots
How to implement these new requirements for 2014-2020? Our advice to Member States: • Use the evaluation ex ante to strengthen the programme theories of change • Be open to new approaches and allow for discussion on methods in Terms of Reference • Be realistic: take into account budget and time constraints, think of the primary use of evaluation (improve implementation or assess effectiveness? target audience?)
How to implement these new requirements for 2014-2020? Our advice to evaluators: • Help commissioners to refine evaluation questions • Propose and defend TBE designs mixing methods according to evaluation questions, timing, budget, planned use • Document data collection and analysis processes in evaluation reports
Conclusion • We should not miss the opportunity given by ex ante evaluations • Common goals: create a body of evidence on regional development • Need for critical mass of good quality TBE: mix of methods framed by a rigorous design • Need to develop strategies to minimise potential bias in data collection and analysis • Need to develop criteria and standards for theory-based designs
Reference documents On InforegioWebsite:http://ec.europa.eu/regional_policy/information/evaluations/index_en.cfm Guidance documents Impact evaluation center Evaluation library (programmes) EC evaluations RegionalPolicy