1 / 33

Evaluation of European Union Research Programmes Recent experiences and future challenges

This conference presentation discusses the evaluation system of EU research programmes, including the FP7 and Horizon 2020. It explores the characteristics, budget, evaluation principles, and networking of these programmes, as well as the long-term impact study. The presentation concludes with an overview of Horizon 2020 and its focus on excellent science.

bagosto
Download Presentation

Evaluation of European Union Research Programmes Recent experiences and future challenges

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Evaluation of European Union Research Programmes Recent experiences and future challenges Evaluation 2012 Conference Minneapolis, 25 October 2012 Dr Peter Fisch European Commission - DG Research and Innovation

  2. Roadmap • Introducing the EU Research Framework Programme (FP) • FP Evaluation System • Study on longer-term impact of the FP • Horizon 2020 – the new Research and Innovation Programme 2014 – 2020 • Challenges for the evaluation system

  3. Characteristics of EU Framework Programmes • Project funding only • True common put – no national quota • Competitive process with annual calls and peer review proposal evaluation • Main focus on collaborative projects with partners from several countries • Open to academia and industry alike • Open to the world, not restricted to Europe

  4. FP7 Structure Cooperation – Collaborative research Multi-partner projects, top-down agenda Ideas – Frontier Research Individual projects, bottom-up agenda Capacities – Research Capacity Sharing of infrastructures … People – Marie Curie Actions Mobility programmes for post-docs, Training courses, … + Other activities (EURATOM, JRC, …)

  5. FP7 Budget (in millions of €)

  6. FP7 in Key Figures • 79.000 Applications • 386.000 Participants in applications received • 16.000 Projects funded • 85.000 Participants in projects funded 38% Universities 28% Industry (17% SME) 26% Research Organisation • 25.3 Billion € EU funding • 169 Countries participating

  7. Global scientific collaborations

  8. FP7 Evaluation Basic Principles • Systemic Approach • Clear planning and timing; complete coverage • Evidence based • Quantitative and qualitative data • Policy oriented • Focus on recommendations for concrete improvements • Decentralised • Each activity “owns” its evaluation

  9. FP7 Evaluation Networking • Across all services managing FP7 • Exchange of experiences • Planning • Within all Commission services • Methodologies • Common administrative issues • Network with Member States • Interaction with experts and administrators • Forum for national experiences • Opportunity for “soft coordination”

  10. FP7 Evaluation Studies • Around 20 to 30 evaluation studies per year in FP7 • “Thematic Studies”, initiated by a specific activity to analyse a particular area, such as • Transport Research • Social Sciences & Humanities • “Horizontal Studies”, initiated centrally to analyse overarching issues, such as • Effects of simplification measures • Long term impact of the FP

  11. FP7 Evaluation Full Programme Evaluations • Political decision to carry out evaluations of the entire FP in regular intervals • Kind of “Meta-Evaluation”, mainly based on available material from Monitoring and Studies • Ex-Post Evaluation of FP6 (2008 – 2009) • Interim Evaluation of FP7 (2010) • Ex-Post Evaluation of FP7 (2015) • http://ec.europa.eu/research/evaluations

  12. Long-Term Impact Study • Study on longer term impact of FP • Analysing impact of FP4, FP5 and FP6 "from the distance" • Attempt to combine quantitative and qualitative methodologies • Quantitative approach not as successful as hoped for … • Carried out by Technopolis

  13. Long-Term Impact StudyCombination of methods Definition Critical reflections Implementation Literature review Co-word analysis Interviews Intervention Logic 6 Case studies Citation network Breakthrough analysis Bibliometrics Interviews Methods Impact mechanisms Methodological Framework Effects Typology Focus areas Findings on Impacts Conclusions

  14. Long-Term Impact StudyA Case-Based Approach to Impact 6 case studies: • Quantum Information Processing and Computing • Stratospheric Ozone Research • Solar Photovoltaics (PV) • Brain Research • Automotive Industry • Manufuture Technology Platform

  15. Long-Term Impact StudyKey Findings • Very difficult to access valid information • 10 years is a long period … • Long-term impact different from short- term impact • Community building vs. Publications • Long-term impact varies per theme / area • Industry integration, discipline building , … • Importance of referring back to programme objectives

  16. Long-term Impact StudyAttempt for an Intervention Logic

  17. Long-Term Impact StudyConclusions • Long-term perspective needed to complement usual short-term analysis • Current monitoring and data tracking systems not suitable for capturing long-term effects • Further efforts needed to develop methodologies further ("beyond case studies")

  18. Horizon 2020Basics • A single programme bringing together three separate programmes/initiatives (FP/CIP/EIT) • Coupling research to innovation – from research to retail, all forms of innovation • Focus on societal challenges facing EU society, e.g. health, clean energy and transport • Simplified access,for all companies, universities, institutes in all EU countries and beyond

  19. Horizon 2020Priority 1: Excellent science • World class science is the foundation of tomorrow’s technologies, jobs and wellbeing • Europe needs to develop, attract and retain research talent • Researchers need access to the best infrastructures

  20. Horizon 2020Priority 1: Excellent science

  21. Horizon 2020Priority 2: Industrial leadership • Strategic investments in key technologies (e.g. advanced manufacturing, micro-electronics) underpin innovation across existing and emerging sectors • Europe needs to attract more private investment in research and innovation • Europe needs more innovative SMEs to create growth and jobs

  22. Horizon 2020Priority 2: Industrial leadership

  23. Horizon 2020Priority 3: Societal challenges • Concerns of citizens and society/EU policy objectives (climate, environment, energy, transport etc.) cannot be achieved without innovation • Breakthrough solutions come from multi-disciplinary collaborations, including social sciences & humanities • Promising solutions need to be tested, demonstrated and scaled up

  24. Horizon 2020Priority 3: Societal challenges

  25. Horizon 2020 Novelties • Integration of the innovation dimension • Additional emphasis on Public-Private-Partnerships and Public-Public-Partnerships • New approaches to SME support • Massive increase in financial instruments (loan guarantees for Mid-Caps) • Even greater variety of governance modes (Knowledge and Innovation Communities for the European Institute for Technology and Innovation)

  26. Evaluation SystemLegal documents Explicit  intervention logic Objectives specified General objectives Specific objectives Operational objectives Key indicators Specified with benchmark figures

  27. Evaluation SystemHierarchy of objectives

  28. Evaluation SystemWork in progress Internal Working Group Community building Complementing objectives and indicators Gathering and handling of data Commission Staff Working Document Spring 2013 Key elements of Horizon 2020 evaluation strategy Timetable

  29. Evaluation ChallengesObjectives and Indicators Objectives and indicators in the legal documents provide the "back bone" for the evaluation system Need to complement these by more specific objectives and indicators Old problem of the "missing middle" Preparing valid indicators in view of Horizon 2020 Ex-Post Evaluation in … 2023          

  30. Evaluation ChallengesEconomic and societal impact High political emphasis on providing information on economic and societal impact Difficulties to trace back research and innovation effects   Most commonly used indicators "stop too early" Little attention paid on human resource indicators (Training, Career tracking)        

  31. Evaluation ChallengesLonger term footprint Recent study highlighted difference in focus between short term and long term analysis Need to prepare the ground for adequate capturing of longer term effects Difficulty to define adequate proxies for phenomena like Community building New disciplines …

  32. Evaluation ChallengesGovernance and coherence Horizon 2020 implemented through (currently) 134 activity lines Managed by several DGs, executive agencies, and numerous other actors (Joint Undertakings, …) Huge variety of actions Small scale to global challenge Top-down vs. Bottom-up Projects, Networks, Careers, Risk Capital, Infrastructures, Mobility, …

  33. Contact Dr. Peter Fisch Head of Unit “Ex-post Evaluation and Reporting” European Commission – DG RTD A.6 SDME 2/41 1049 Brussels / BELGIUM peter.fisch@ec.europa.eu http://ec.europa.eu/research/evaluations

More Related