1 / 23

From compliance to performance and beyond: towards utility-focused evaluation

From compliance to performance and beyond: towards utility-focused evaluation. Jarosław Górniak Centre for Evaluation and Analysis of Public Policies Jagiellonian University in Krakow. Why does a demand to focus on outcomes appear on the agenda?.

aggie
Download Presentation

From compliance to performance and beyond: towards utility-focused evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. From compliance to performance and beyond: towards utility-focused evaluation Jarosław Górniak Centre for Evaluation and Analysis of Public Policies Jagiellonian University in Krakow

  2. Why does a demand to focus on outcomes appear on the agenda? • Re-focusing on outcome indicators is an indicator of problems with policy outcomes • Policy performance gap in terms of outcomes (aspirations versus achievements) is a manifestation of the dominance of effective money spending over effective social problems solving in the actual policy implementation • The dominance of effective money spending is an outcome of the dominance of a compliance audit over a performance evaluation in a multilevel policy management process Jarosław Górniak, Center for Evaluation and Analysis of Public Policies, Jagiellonian University in Krakow

  3. Why compliance with regulations dominates effectiveness of programmes • Lower complexity of rule-based management over performance management • Proneness to safety in administrative behaviour • Deficits in domestic and European development strategies • Lack of mechanisms transmitting performance evaluation to policy decisions, BUT: • Imposing a universal system of output and result indicators and using negative incentives to enforce declared targets • may deteriorate the performance-oriented policy framework • Means the “one best way” approach – policy Fordism – with its consequences Jarosław Górniak, Center for Evaluation and Analysis of Public Policies, Jagiellonian University in Krakow

  4. Programmes are focused on activities and rules – evaluations too CEAPP: Meta-evaluation of evaluation reports of project selection criteria in operational programmes co-financed by EU in Poland 2007-2013 Jarosław Górniak, Center for Evaluation and Analysis of Public Policies, Jagiellonian University in Krakow

  5. Outcome evaluation? Yes, but what is an outcome? • FabrizioBarca, Philip McCann et al.: “It is therefore clear that a systematic and rigorous use of well-defined outcome indicators is a very powerful managerial tool for increasing policy effectiveness” – TRUE, BUT: is the concept of outcome clear? • B & McC: outcomes=results (“outcomes/results”) or outcomes = result + impacts? • Is outcome: • “the specific dimension of the well-being (…) to be modified” OR • a change – neteffect – on this dimension? Jarosław Górniak, Center for Evaluation and Analysis of Public Policies, Jagiellonian University in Krakow

  6. Outcomes • Outcomes = results (immediate + delayed) • Observed in the target group • Directly causally dependent on the intervention • But they have their temporal dimension • Impact – indirectly dependent on the intervention and observed beyond the target group • Hard to measure • Attribution to the intervention is dependent on the insight in transmission mechanisms Jarosław Górniak, Center for Evaluation and Analysis of Public Policies, Jagiellonian University in Krakow

  7. Focus on outcomes in evaluation is necessary… • To mainstream programmes towards development goals • To build a knowledge base on causal effects in policy implementation – counterfactual analysis/net effects • To legitimize spending of European tax-payers’ money showing the value added • To facilitate the evidence-based learning process in policy design and implementation Jarosław Górniak, Center for Evaluation and Analysis of Public Policies, Jagiellonian University in Krakow

  8. … but not sufficient … • … if the planned programme objectives are the only outcomes considered; • … if actual outcomes are measured/observed only using indicators selected to define intended outcomes = targets; • … if important actual outcomes – positive or negative – are ignored if they are not intended or expected; • ... if the evaluation does not grasp a total utility of the programme and the mechanism producing all important consequences. Jarosław Górniak, Center for Evaluation and Analysis of Public Policies, Jagiellonian University in Krakow

  9. The Concept of „Utility” is known…Evalsed 2009 socialneeds public intervention IMPACT Regardless of the statedgoals, theory and logic of the program Jarosław Górniak, Center for Evaluation and Analysis of Public Policies, Jagiellonian University in Krakow

  10. …but the concept of utility is: • misunderstood in the vast majority of evaluations – in the Polish case, this was determined in the research conducted by CEAPP in 2010; • largely ignored; • not to be linked easily with setting objectives; • unfolding complexity and risks; • hard to measure and assess normatively. Jarosław Górniak, Center for Evaluation and Analysis of Public Policies, Jagiellonian University in Krakow

  11. Goal-FreeEvaluation • Michael Scriven • too much focus on originally stated goals leads to tunnel vision and bias • impact of the unplanned effects may be surprisingly high • stated objectives as a safety blanket for evaluator /Scriven M., Prose and Cons about Goal-Free evaluation,  Evaluation Comment, December 1972, Vol. 3, No. 4, pp. 1-4/ Jarosław Górniak, Center for Evaluation and Analysis of Public Policies, Jagiellonian University in Krakow

  12. EffectsMatrix GOAL-LIMITED Jarosław Górniak, Center for Evaluation and Analysis of Public Policies, Jagiellonian University in Krakow

  13. Corrected Definition of Utility „Balance of importantconsequences (bothbenefits and costs) of intervention in the light of currentsocio-economicproblems” /CEiAPP, 2010/ anti-utility utility Jarosław Górniak, Center for Evaluation and Analysis of Public Policies, Jagiellonian University in Krakow

  14. Methodology Jarosław Górniak, Center for Evaluation and Analysis of Public Policies, Jagiellonian University in Krakow

  15. Methodology Jarosław Górniak, Center for Evaluation and Analysis of Public Policies, Jagiellonian University in Krakow

  16. Methodology Jarosław Górniak, Center for Evaluation and Analysis of Public Policies, Jagiellonian University in Krakow

  17. Methodology Jarosław Górniak, Center for Evaluation and Analysis of Public Policies, Jagiellonian University in Krakow

  18. Methodology Jarosław Górniak, Center for Evaluation and Analysis of Public Policies, Jagiellonian University in Krakow

  19. Strengths of the approach • identifying actions undertaken in the process of implementation and defining latent objectives behind them; • indicating all the actors and policy impact groups; • defining the widest possible range of consequences of an assessed policy regardless of whether they were planned or not, positive or negative; • creating quantitative indicators of policy effects whenever possible; • measuring outcomes even if the objectives are poorly defined or missing in the programme documentation; • recommending a mix mode methodology joining qualitative and quantitative techniques typically used in social sciences Jarosław Górniak, Center for Evaluation and Analysis of Public Policies, Jagiellonian University in Krakow

  20. Evaluation and ex-post conditionality • There is a time lag between the programme outputs and – sometimes essential – outcomes • It is not possible to totally plan the total programme utility • The real question is: • will the frozen programme targets (conservatively set to avoid failure) simply amend the system of detailed, bureaucratic, but really financially executed rules or • will the cohesion and structural policy concentrate on innovative development in Europe Jarosław Górniak, Center for Evaluation and Analysis of Public Policies, Jagiellonian University in Krakow

  21. Open Method of Coordination revisited • Common set of indicators, common learning process based of collected evidences and MS declaration of policy targets were corner stones of the OMC – well known but recently neglected • The common indicators in OMC are general welfare/development indicators rather then programme result indicators • OMC may be converted into performance informed budgeting system rather than performance based budgeting of cohesion and structural policy • Attrition of OMC is not a defect of the method but the lack of commitment • Revival of OMC is possible in the area of innovative development – here objectives are real motivators Jarosław Górniak, Center for Evaluation and Analysis of Public Policies, Jagiellonian University in Krakow

  22. Towards European performance framework • OMC after an upgrade and merger with the evaluation system • delivering evidence • utility focused is inline with the first option for performance framework presented in the issue paper of the Polish Ministry of Regional Development for this conference: “The first model assumes that European rules regarding performance framework are general and only common set of principles such as the obligation of establishing performance network in a given MS ….” p. 5. Jarosław Górniak, Center for Evaluation and Analysis of Public Policies, Jagiellonian University in Krakow

  23. Thank You for Your attention! www.ceapp.uj.edu.pl Jarosław Górniak, Center for Evaluation and Analysis of Public Policies, Jagiellonian University in Krakow

More Related