Theory Driven Evaluation: tracing links between assumptions and effects

1. Theory Driven Evaluation: tracing links between assumptions and effects Sixth European Conference on Evaluation of Cohesion Policy Warsaw, 30 November – 1 December 2009 Karol Olejniczak, k.olejniczak@uw.edu.pl EUROREG – University of Warsaw: www.euroreg.uw.edu.pl

2. Theory-Driven Evaluation in a nutshell • It relates evaluation research with the scholarly socio-economic theories • It treats programme as optimistic assumption about causal relations:If we DO…. than we GET… and than… • Programme is a set of theories… • Underlying Theories (knowledge, experience, influences) • Theory of Change (assumption about strategic change) • Theory of Implementation (the way every-day work is organized) • ...that works in a certain context & circumstances • TDE is and approach, it is method-neutral • Logic models • 5 stages procedure

3. 5 stages procedure

4. Case Study of Ex post Evaluation Neighbourhood Program INTERREG/TACIS CBC Poland-Belarus-Ukraine 2004-06 Theory Driven Evaluation in Practice

5. Programme • Border regions from Poland, Belarus, Ukraine • Two programmes – two pots of money: INTERREG 37,8 mln EUR vs TACIS 8 mln EUR • Beneficiaries – local communities, local services, NGO • 173 major projects + 300 micro-projects • Contracting Authority: • Polish Ministry of Regional Development, Territorial Cooperation Unit • Evaluator: • EGO – Evaluation for Government Organizations s.c. • Contract: • 5 monhts (July-Nov 09)

6. Stage 1: Understanding the context Questions: Answers: CBC depends on permeability of the border & partners complementarities Highly peripheral, underdeveloped area, border as barrier Minor financial impulse, could be visible only on a local scale • What theories tell us about cross-border cooperation? • What type of border it is? • What was the scale of intervention? • Methods: • Literature review, review of earlier empirical research • Analysis of general statistics • Review of socio-political situation • Interviews with the Directors of the Departments • Secondary data • Effectiveness indicators from Task-oriented budgets • Staff turnover • Other

7. Stage 2: Defining Theory of Change & tracing its Underlying Theories Questions: Answers: Dual objective: IF we act on 5 thematic fields THAN quality of life AND socio-economic integration will be improved in a border area Reasonable fields of intervention but no measures for border permeability Stakeholders choices, intuition, earlier experiences of Poland-Germany CBC Programme • What strategic objectives have been set? • What were the assumptions and inspirations for these decisions? • Methods: • Desk research (programme documents) • Interviews with key programme stakeholders involved in the programme design • Interviews with the Directors of the Departments • Secondary data • Effectiveness indicators from Task-oriented budgets • Staff turnover • Other

8. Stage 3: Reconstructing detailed Theory of Change and its indicators Questions: Answers: Effects both planned and side-effects have to be measured on 2 scales: quality of life & integration They have to be traced in 3 dimensions: thematic (projects topics), relational (partnerships) and territorial (local communities) Analysis & assessment criteria base on logic models, they differ in details but scale stays the same • How should we define the change in terms of indicators & assessment criteria? • How programme designers defined these changes in terms of programme & projects indicators? • Methods: • Logic models for each thematic group and partnerships • Assessment system – two scales: quality of life vs integration • Review of the programme indicators

9. Stage 4: Tracing real changes – outputs & effects Questions: Answers: High disproportion between 3 sides of the border Focus on improving quality of life Local and close-to-border effects The main integration effect was brought by soft-projects, it was institutional, limited integration of local societies Balanced effects (quality + integration) brought by tourism & border security projects • What was the funds spatial & thematic distribution? • What was the number & structure of partnerships? • What were the effects of thematic group of projects? • What were the effects of partnerships? • What were the impact on local communities? • Methods: • Local visits of all project sites (different observation tools applied, depending project’s topic) • Survey of project beneficiaries, survey of partnerships • Social survey of twin communities • Interview with local stakeholders, review of local press

10. Stage 5: Explaining the obtained effects Questions: Answers: Unbalanced money & procedures; higher experience of Polish teams High peripheriality & local needs; unclear demarcation line Border as a sealed barrier, small funds spread spatially Selection criteria, limited trust – focus on smaller projects, expert-type projects, micro-projects as top-down not bottom up initiatives Tourism easy to combine with next initiatives (multiplier effects), security required official cooperation agreements • Why disproportion? • Why focus on quality of life? • Why close-to-border and local effects? • Why institutional integration? • Why tourism & border security projects had best effects? • Methods: • Brainstorming with experts • Second review of qualitative & quantitative data again • Interviews with programme managers • Survey of unsuccessful and potential applicants • Interviews with the Directors of the Departments • Secondary data • Effectiveness indicators from Task-oriented budgets • Staff turnover • Other

11. Summing up the case study Main message: Close to border and local effects. In a given context every joint project was a success How TDE helped us? Dealing with complexity - packing & unpacking issues Focusing exploration on the right level Making the fair judgement – understanding contextual limitations Writing the concise report - clear narrative

12. TDE for cohesion programmes Advantages Challenges Using too rigid model can lead to tunnel vision and omitting side-effects There is a trade-off between level of details and clarity of the models Too much sophisticated theoretical considerations can alienate stakeholders and turn evaluation into scholar research • Articulates rationality of the programme • Provides clear conceptual foundation for the study • Focuses on effects and treats implementation issues only as one of the explaining factors • Relates to scholarly theories and give bigger picture • Allows to discuss causal relations

13. Bibliography Chen, H.T. (2004) Practical Program Evaluation: Assessing and Improving Planning, Implementation, and Effectiveness. Thousands Oaks: Sage Publications, Inc. Donaldson, S.I. (2007) Program Theory-Driven Evaluation Science: Strategies and Applications. New York: Lawrence Erlbaum. Knowlton, L.W. & Phillips, C.C. (2008) The Logic Model Guidebook: Better Strategies for Great Results. Los Angeles: Sage Publications, Inc. Leeuw, F.L. (2003) Reconstructing Program Theories: Methods Avaliable and Problems to be Solved. American Journal of Evaluation, 24(1), pp.5-20. Patton, M.Q. (2008) Utilization-focused evaluation. 4th edition. Los Angeles, London: Sage Publications. Pawson, R. (2009) "Introduction to Realist Evaluation and Realist Synthesis", lecture on The Academy of Evaluation, EUROREG – University of Warsaw, Warsaw, 7.02.2009 Weiss, C.H. (1997) How can theory-based evaluation make greater headway? Evaluation Review, 21(4), pp.501-524. Weiss, C.H. (2004) On Theory-Based Evaluation: Winning Friends and Influencing People. Evaluation Exchange, IX(4), pp.2-3.

14. Contactdetails Karol Olejniczak, PhD EUROREG – University of Warsaw www.euroreg.uw.edu.pl E-mail: k.olejniczak@uw.edu.pl Phone: +48 22 826 16 54 Mobile: +48 696 41 22 82