AEA Conference Orlando, 12 November 2009 Dr. Peter Fisch European Commission

1. Evaluation of the current 7th European Research Framework Programme AEA ConferenceOrlando, 12 November 2009 Dr. Peter Fisch European Commission

2. Monitoring FP7 The Second Monitoring Report (2008) Evaluation of FP7 FP7 Interim Evaluation Open Issues for FP Evaluation Challenges in a wider context Roadmap

3. Move from “ad hoc” campaigns using external experts (FP6) towards a systematic internal monitoring (FP7) Annual reports based on a core set of indicators Information source for future FP7 evaluations Flexible system to develop as FP7 will become more “mature” (outputs) FP7 Monitoring System

4. Absolute figures (2007 + 2008): 25.000 proposals received, 160.000 applicants 5.500 proposals retained, 35.000 participants Funding from the EU 10 billion € Simplification: Improvements from FP6 to FP7 (e.g. on audit certificates) Positive feedback from research community Monitoring Report 2008Key Findings (1)

5. Participants: Universities and Research Organisations in the lead (30+%) 15.5% SME participants Gender: 21.6% female “scientific contact persons” 35.4% female “fellows” in Marie Curie actions International cooperation: 136 countries involved in FP7 Monitoring Report 2008Key Findings (2)

6. Proposal Evaluation: 96% of evaluators judge FP evaluation system “similar” or “better” than national systems Redress / Ethical review: 1170 redress requests, 17 re-evaluations 539 ethical reviews, no project stopped Monitoring Report 2008Key Findings (3)

7. Embedded in the overall Commission Evaluation system Evaluations to be carried out by the services responsible for an activity as part of the management responsibilities FP7 Evaluation system spelled out in detail in the FP7 decision and the FP7 Impact Assessment Multi-layer system consisting of thematic evaluations at programme level, studies to analyse general issues and evaluations at FP level FP Evaluation SystemBasics

8. Monitoring, Project Database Output indicators Self assessments Thematic evaluation studies and reports Horizontal evaluation studies National Impact Assessments Feedbacks, surveys … FP Evaluation SystemIngredients

9. FP7 Decision: Interim Evaluation “no later than” 2010 To cover FP as a whole Specific reviews in some areas (European Research Council, Euratom, …) Carried out by a group of external experts Meetings from February to September 2010 Final Report expected in October 2010 FP7 Interim EvaluationBasics

10. Key questions: General objectives achieved? How to improve impact of FP on the European Research Area and other policies ? FP7 role in positioning Europe on the global map? Efficiency of novel measures (European Research Council, Joint Technology Initiatives)? How to better address interdisciplinary “grand challenges”? Simplification measures effective? Progress on issues raised in FP6 evaluation? FP7 Interim EvaluationPossible Key Questions

11. Specify a clear intervention logic in the basic legal acts Develop a hierarchy of (measurable) objectives throughout the different levels (programmes, projects) FP evaluation: Difficult to achieve in a complex environment like the European Union Need to develop new types of indicators in order not to be blocked by just aiming at what you can measure … Open IssuesIntervention Logic

12. Clear evaluation strategy from the outset Complete coverage of all activities Right timing in view of revisions and development of new actions FP evaluation: Concise long term-time planning Need to improve on overall coherence of the evaluation activities carried out in different fields of the FP Open IssuesOverall Evaluation Strategy

13. Calls for strong coordination and “harmonisation” clash with the need to use a wide spectrum of different evaluation approaches FP evaluation: Diversity is one of the big assets of Europe Different traditions and schools across Member States Unique opportunity to bring together wide range of evaluation approaches Challenge to turn “constructive chaos” into operational structures Open IssuesDiversity and Coordination

14. Different views on what drives evaluation Need to control activities and actors Wish to understand what is going on Major implications on the organisation and focus of evaluations FP evaluation: Focus on evaluation as a management task to gain a better understanding of the activities undertaken Open IssuesControl or Understanding

15. Full impact of research activities can only be assessed after a long period of time, as full implementation can take several years FP evaluation: In order to be on time (politically), evaluations are carried out very early … Test-run in preparation for a study looking back at projects which were finished some 5 to 10 years ago Easier said than done … Open IssuesLonger Term Impact

16. A too much detailed level of analysis prevents many evaluation reports from having a real impact in the wider political and societal context FP evaluation: Meta-Evaluation through expert panels as a tool to “condense” otherwise too dispersed evaluation findings Further efforts are needed to focus on key messages for communication with the target audiences Open IssuesConcise Messages

17. There is no “ideal” evaluation All actors are trying out different approaches Need to organise more mutual learning FP evaluation: European RTD Evaluation Network, involving experts from 30+ countries Interest in more active exchange at global level Open IssuesOrganise Mutual Learning

18. Dr. Peter Fisch Head of Unit “Evaluation and monitoring of Programmes” European Commission – DG RTD A.3 SDME 2/41 1049 Bruxelles - Belgium peter.fisch@ec.europa.eu http://ec.europa.eu/research/evaluations Contact