The Policy makers' evaluation use — between rationalizing and legitimating Dr. Daniel Tsygankov

1. The Policy makers' evaluation use — between rationalizing and legitimating Dr. Daniel Tsygankov Head of the Center for Program and Policy Evaluation (Russia, Moscow) European Evaluation Society Conference - 2008 Lisboa, 2. October 2008

2. Introduction • In recent decades the number of evaluation research carried out is rapidly growing; now this trend is also referred to states of the former Soviet block and China. • A lot of countries in transit have already entered the influence orbit of countries and international organizations (e.g. EU) with formed evaluation institutes. • Which risks exist for Russia / CIS states? Evaluation is developing here "wildly", fragmentarily, with own (not yet clear) scenarios.

3. Cooperation’s models between policy maker and expert / scientist • “Technocracy” (expert adapts himself to policy maker demands); • “Meritocracy” (policy maker has „no idea“, and expert decides everything – from theme formulation up to the research design); • “Policy Learning“ (both sides find mutual understanding by multistage negotiations).

4. Knowledge utilization?(1) Simplifying the policy makers taking political and administrative decisions pursue two groups of objectives: • scientifically centralized objectives (knowledge may increase the rationality of political practice); • applied objectives: • the legitimating of decisions/ proposals; • political win.

5. Knowledge utilization?(2) In accordance with this fact… • the supporters of amplification of use/integration of evaluation into public sector speak of challenges of modern administration “efficiency”(eg. NPM, Performance Audit, etc.); • the critics refer to the “verification rituals” of modern auditing society (Michael Power) and forming of new audit culture (Marylin Strathern) which can affect social reality (e.g. by means of forming of performance indicators system) . • Here we put aside those evaluations and surveys which are objectively useless due to their methodological mistakes.

6. Evaluation percolation through fields of power • On the one hand, evaluation results’ use in political practice is still not a general rule. => The information on evaluation, its concepts and arguments is gradually percolating through society by means of force groups, expert influence and mass-media. • On the other hand, evaluation as “new” cross-border discipline – beside management and marketing research – compares favourably with policy science, sociology and philosophy. • The advocates of evaluation implementation themselves perceive both weak professionalism in some countries and “shapeless” evaluation (everything that names itself in such a way is accepted just so by those as well) with an anxiety, and neatly characterize them as pseudo-evaluations or quasi-evaluations.

7. Rationalizing vs. Legitimating • Therefore the development of evaluation as public practice and research strategy can be analytically placed on the axis between the poles: “Rationalizing” <–> “Legitimating” • For further analysis of chances of its institutionalization the new dimensions can be introduced: • country institution environment • waves of public sector reforms • development of boundary science disciplines • structure of the power field / the intellectual field, etc.

8. Attributes of the "non-rational" intentions The peculiarities of formulating of conditions and evaluation topics by policy makers can point out such a “removal” from rational pole: • the topic is contradictorily, confusingly and at length formulated; • policy maker (or his representative) avoids explaining research topics; • research reports are never published; • the companies which are absolutely unknown in the market participate in tenders; • the price of the contract obviously contains “a corrupt constituent”.

9. Conclusion & Acknowledgment • The understanding of consistent patterns of such development possesses not only pure scientific interest, but it is also important in principle for the countries in transit which did not enter the influence orbit of the formed evaluation culture. For such countries there is a risk the relegation of evaluation to the “Legitimating” pole is high. • Therefore there is a question whether the existing approaches (creation of national association and work-out of evaluation standards, development of master programs and programs of further education, publication of magazines and books in mother-tongue, etc.) will be enough for risks reduction? Or… • Whether it will be necessary to take non-standard steps which will contribute to more rational use of evaluation as cross-border discipline?

10. Contact information • Head of CPPE – Dr. Daniel Tsygankov • Address in Russia:125493, Moscow, Smolnaya street, 10. • Phone: +7 (499)504-4147 • E-mail: mail@iopp.ru • Web: www.cppe.info (English site) evaluationrussia.wordpress.com (English blog) www.iopp.ru (Russian site) • European office (Paris) – Maria Smirnova • E-mail:smirnova@iopp.ru