1 / 23

EVALUATION OF RESEARCH PERFORMANCE: A System’s Approach (Klaus Schuch, ZSI)

EVALUATION OF RESEARCH PERFORMANCE: A System’s Approach (Klaus Schuch, ZSI). SCIENCE EVALUATION AS A PREREQUISITE FOR PROMOTING EXCELLENCE IN RESEARCH NOVEMBER 6, 2012 CHISINAU, MOLDOVA. Contents. General aspects Different levels of Science and Technology Policy Evaluation

abiba
Download Presentation

EVALUATION OF RESEARCH PERFORMANCE: A System’s Approach (Klaus Schuch, ZSI)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. EVALUATION OF RESEARCH PERFORMANCE: A System’s Approach (Klaus Schuch, ZSI) SCIENCE EVALUATION AS A PREREQUISITE FOR PROMOTING EXCELLENCE IN RESEARCH NOVEMBER 6, 2012 CHISINAU, MOLDOVA

  2. Contents • General aspects • Different levels of Science and Technology Policy Evaluation • Project Evaluation • Programme/Instrument Evaluation • Institution Evaluation • Policy Evaluation • Situation in Southeast Europe • The EVAL-INNO project EVAL-INNO

  3. Public Interest 3 • Democratic values: transparency and fairness • S&T policy values: • S&T policy has to be in a position to show how and why R&D investments are worthwhile • a developed culture of evaluation is an integral part of a strategically oriented S&T policy • Evaluations help to reduce the risk of future interventions to combat market and system failure • Evaluations contribute external knowledge and opinions to policy-making and policy-delivery and also provide the necessary documentation of how policies are implemented • But evaluations do not replace policy action! 03/10/2014 EVAL-INNO

  4. Regulatory Framework – Example Austria 4 • Public Procurement Law • Opt for “economically most advantageous offer” and not always “best price” • Include also knowledgeable experts from outside the country • In Austria a soft law is in place that demands evaluations for each R&D programme • Guidelines to support economic-technical research and technology development (2007 ) • Recommendations of the Austrian Council for Research and Technology Development on the Evaluation and Monitoring of RTDI Programmes from 12 April2005 • Evaluation Standards in Research and Technology Policy issued by the Austrian Platform for Research and Technology Policy Evaluation (FTEVAL) 03/10/2014 EVAL-INNO

  5. 5 03/10/2014 EVAL-INNO

  6. Utilisation of Evaluations 6 • Create a demand for evaluation results by • Ensuring the support of top-level decision-makers • Raising realistic expectations • Integrating those involved in and affected from • Create an appropriate environment by • Carrying out evaluations systematically • Planning evaluations within the policy cycle • Ensuring access and collecting all relevant data – and only that! • Reserving sufficient funding for a sound evaluation • Ensuring the methodological quality of evaluations • Follow-up on the implementation of results • Designing appropriate communication strategies etc. EVALUATIONS ARE NOT CARRIED OUT FOR THEIR OWN SAKE! 03/10/2014 EVAL-INNO

  7. Functions of Evaluations • Legitimisation: • Information: • Learning: • Steering: • Mediation: justifying the use of public funds to the public how funds are being used and to what effect for all stakeholders to do things better for establishing policy objectives, planning of measures etc. balancing interest and dialogue EVAL-INNO

  8. Evaluation Pyramid In a developed NIS evaluation occurs on different levels! Policy evaluations System evaluations Institution evaluations Portfolio evaluations Instrument evaluations Programme evaluations Project evaluations EVAL-INNO

  9. Project Evaluation 9 • The basis of any evaluation system is a sound project evaluation. • A project is an individual activity with its own fixed time plan and its own budget. Projects produce concrete results (output, outcomes). • Ex-ante evaluations project proposals are of regular component of justifying funding decisions. They are either based on peer reviews or jury decisions (with external experts). • They assess the relevance of the proposal, its likelihood of reaching the objectives (of the project and contribution to the programme, if any), the quality of its content and methodology (research design), the quality of its management, its efficiency in times of expected input-output ratio, its economic viability (if economic R&D), it potential impact etc. • Larger projects should also be subject to interim or monitoring and terminal or ex-post evaluation. 03/10/2014 EVAL-INNO

  10. Project Evaluation in Austria Austrian Science Fund Applied Research (FFG) • Only competition based – all fields of science compete (no quota) • Only peer review based • Only international peers invited • Proposals mostly in English • Effect on the Austrian participation in European RTDI • Open and targeted (e.g. thematic calls) – often programme based • Economic content requires discretion • Mix of peer reviews and jury reviews involving external experts • Proposals often in German language EVAL-INNO

  11. Programme/Instrument Evaluation (1) 11 • A programme is a combination of measures where the underlying intentions refer to each other and which aim at achieving one or several specific defined objectives. A programme usually exists for a fixed time period and has its own allocated and often centrally administered budget. • An instrument describes a specific form of an intervention. Monetary instruments (funding, financing) are to be differentiated from non-monetary instruments (consultancy, awareness-raising measures, regulations, standards). Programmes can use one instrument or a combination of instruments (mix of instruments). 03/10/2014 EVAL-INNO

  12. Programme/Instrument Evaluation (2) 12 • Ex-ante concept evaluations review the mission, assumptions, fundamental hypotheses and framework conditions of any possible interventions. • Design evaluations deal with the organisational structure of an intervention (timing, budgeting, conflict resolution), control whether the measures used are suited to meet the objectives and if it was possible to reach them. They are typical elements of programme evaluations at all stages (ex-ante, interim, ex-post). The trend goes from a mere “before and after” comparison (black-box evaluation) to process evaluation (typical for formative interim evaluations). • Impact analyses (mostly ex-post) assess the extent to which the objectives of an intervention have been achieved and aim to identify all relevant directly or indirectly, intentional or unintentional caused effects in a summative way. They look at impacts, outcomes and outputs, scientific effects, social effects and economic effects. 03/10/2014 EVAL-INNO

  13. Institution Evaluation 13 • Institutions are mostly of permanent nature. Focus is on universities with their combination of research and teaching, research institutions, intermediaries, and funding institutions or agencies. • Institution evaluations are often peer based. They should help to avoid lock-in phenomena and blind spots. • When assessing the research performance the focus should be on internal, external and negotiated factors concerning • knowledge generation • knowledge utilisation • knowledge diffusion • The RECORD Benchmarking Manual provides a useful methodology for transition and catching-up countries. • Example: Benchmarking of Nanotechnology Research Institutes in EECA 03/10/2014 EVAL-INNO

  14. Policy Evaluation 14 • Policy constitutes a set of activities (funding instruments, procedures, regulations etc.) which may be very different in type but share a common motivation or objective. In contrast to projects and programmes, policy is generally not limited by time nor budget. • Policy evaluations are often carried out in form of international peer reviews (e.g. DAC peer reviews of OECD in the field of development assistance). In S&T policy initiated through CREST (now ERAC). • They critically examine the existing S&T policy mix by using a “critical friends” approach, provide policy recommendations and engage in a dialogue with policy makers and S&T experts from the host country to increase mutual understanding and learning. • A S&T Policy Mix Peer Review is rather a (mutual) learning than a rigid assessment exercise! 03/10/2014 EVAL-INNO

  15. S&T Policy Mix Peer Review Moldova 15 • Carried out 2012 within the INCO-NET EECA project by involving S&T policy experts from Austria, Belarus, Estonia, Germany and Greece. • Hosted by the Academy of Science of Moldova • Focus: • Public research and innovation system • Private sector RTDI • Funding of RTDI • Human Ressources • Internationalisation and FP7 association • Regional dimension 03/10/2014 EVAL-INNO

  16. Situation in SEE 16 03/10/2014 EVAL-INNO • Number of R&D programmes, applied instruments etc. grows, but … • Capacities to evaluate RTDI programmes, instruments, policies and organisations are underdeveloped • lack of knowledge on professional tendering procedures (including public procurement laws) to obtain the best evaluation results • Potential of evaluations for strategic intelligence building is not exploited (no policy cycle; no systematic approach) • Difficulties to access comparable RTDI evaluation information and good practices • Difficulties to access international databases with important output data (e.g. patents, publications) • Lack of completed good practice RTDI programmes, institutions and policy evaluations • Lack of systematic exchange with evaluators in EU and globally.

  17. EVAL-INNO Objectives 17 03/10/2014 EVAL-INNO Key objective is to strengthen regional as well as national evaluation capacities in order to improve the framework conditions for innovations (policies, programmes, projects, institutions). In particular, the project aims • to promote the role of RTDI evaluation for a reflexive learning innovation system • to develop the needed capacities for comprehensive RTDI evaluations • to provide procedural and methodological know-how and tool-kits both on the side of evaluators and on the side of awarding authorities.

  18. EVAL-INNO Core Activities 18 03/10/2014 EVAL-INNO mapping evaluation experts and RTDI programmes in SEE training evaluators and policy delivery systems strengthening the use of external evaluation through right- and meaningful public procurement establishing a virtual regional RTDI evaluation platform developing standards running RTDI programme evaluations and benchmarking exercises linking to international networks

  19. EVAL-INNO Training Invitation 19 03/10/2014 EVAL-INNO • 4 Training Courses • Separate training courses for evaluators and policy delivery systems including programme owners • Bulgaria: March 2013 • Hungary: April 2013 • Montenegro: June 2013 • Serbia: October 2013

  20. EVALUATORS POLICY DELIVERY AND PROGRAMME OWNERS Eval-INNO Training Invitation 20 03/10/2014 EVAL-INNO Hermeneutics and theory of evaluation Rules and ethics Real evaluation cases from different policy spheres Concrete evaluation methods (qualitative & quantitative) Exercises Hermeneutics and theory of evaluation Rules and ethics Real evaluation cases from different policy spheres Public Procurement ToR construction Development of Monitoring Systems

  21. Training Course for Programme Owners 21 03/10/2014 EVAL-INNO

  22. Training Course for Evaluators 22 03/10/2014 EVAL-INNO

  23. Contact Klaus Schuch ZSI – Centre for Social Innovation Austria http://www.eval-inno.eu EVAL-INNO

More Related