Evaluating Research Dynamics Using Network Analysis (in the context of EU funded R&D)

1. Evaluating Research Dynamics Using Network Analysis (in the context of EU funded R&D) Frank Cunningham DG Information Society and Media European Commission and Caroline WagnerGeorge Washington University CES/AEA Conference, Toronto October 27 2005

2. Shift Towards Evidence Based Policy Making in EU • Commission has become more an ‘executive arm’ of the European Institutions • Manages a budget of €100 B /year • Rising (possibly) to €1000 B in period 2007-13 • Shift towards ‘evidence-based’ policy development reinforced by enlargement to 25 Member States

3. EU R&D Framework Programmes • Currently in the 6th Framework Programme • Total budget of €17B over 4 years (2003-6) • Information Society Technologies (IST) €3.8B • Annual EU investment in IST of €1B • EU investment about 3% of total IST-RTD in Europe.

4. Current Evaluation and Impact Assessment Methodologies • Commission has considerable experience in evaluation and monitoring of effectiveness of EU RTD investments over the last 15 years. • Focused mainly on ‘outputs’ rather than impacts. • Focused mainly on ‘project’ level and on the ‘private-return’ to RTD participants. • Now increased attention is given to results-based assessment and macro-economic and social impacts of EU RTD.

5. Gaining a better Understanding of Impact of EU RTD Funding • Work under way in DG INFSO since 2002 • Established that EU funding has created an intensely linked network of European research collaborations. • New tools of ‘data mining’ and ‘link analysis’ were applied to EU IST databases. • Confirmed that network of collaborations is ‘small world’ with strong clustering • High degree of connectivity dominated by a small number of ‘hubs’

6. New Evaluation Methodologies • Innovative firms do not innovate in isolation but in interaction with other ‘actors’ • In ‘Innovation Systems’ the links between innovation actors are crucial • Collaborative R&D Programmes are one important way of strengthening linkages • Partnerships through collaboration networks helps to achieve ‘critical mass’ of dispersed EU R&D resources.

7. Series of Evaluative Studies • DG INFSO is commissioning a series of evaluative studies to answer evaluative questions about the networking aspects and dynamics of: • IST RTD • Innovation & Deployment • Part of our strategy for network evaluation and impact assessment

8. Series of Evaluative Studies • 1. ERAnets Study (RAND Europe) • (subject of this presentation) • 2. Evaluation of Progress towards ERA for IST (CESPRI) • (subject of following presentation) • 3. Networks of Innovation in IS Development and Deployment in Europe • (tenders in evaluation process)

9. ERAnets Study (RAND Europe) • Main Objectives were to Assess: • The degree to which IST Researchers in Europe collaborate. • How integration of IST Research in ERA changed with introduction of new FP6 Instruments. • Carried out from Jan 2004 to March 2005

10. Collaboration Has Grown SignificantlyEurope - 1990

11. Collaboration Has Grown Significantly Europe - 2000

12. ERAnets Study Breaking New Ground in Evaluation • An Evaluativestudy supporting the Evaluation and Monitoring Unit of DG Information Society • Analysed the networks in IST Research and compared the current networks emerging in the Framework Programme with those emerging in other frameworks and communities • Providing insight into the structuring effects of FP6 on European IST RTD • Developed, and refined tools and datasets that can be used by the INFSO Evaluation and Monitoring Unit and others

13. FP6 Networks of Excellence Produce Highly Clustered Network NoEs, call 1 retained, projects as cliques, component 1, diameter 3, average path length 2.1 (colour is number of steps away from the centre, from 1-3)

14. ERAnets Study – Main Findings (1) • Main Findings show that IST RTD funding at the European level is most effective in: • Connecting universities and business • Connecting research in different themes and disciplines • Integrating new Member States • Integrating key patent-holders; and • Integrating small businesses

15. ERAnets Study – Main Findings (2) Main Findings also show that: • FP6 network participants are more tightly interconnected than previous FPs • FP provides a major integrating function …however • Large institutes and companies act as ‘gate-keepers’ • SMEs have been ‘crowded out’

16. Next steps: 2005-6 • Broadened RTD network analysis to the “ERA in the Global context”: CESPRI (in 2005) • Extending network analysis from RTD to include innovation and deployment : to be contracted in 2006. • Clarifying the “causality links” between IST-RTD and FP and “Lisbon” objectives : Technopolis (10/04-8/05) • Initiating the systematic use of key indicators for monitoring the output and impact of IST-RTD at the Strategic Objective-level; • Identification and assessment of the capabilities of “system dynamic” model and tools for macro-economic/social impacts of investments in IST-RTD and ICT deployment : to be contracted;

17. Further Information DG INFSO Evaluation Activities http://europa.eu.int/comm/dgs/information_society/evaluation/index_en.htm  IST RTD (FP 6 Thematic Priority) http://www.cordis.lu/ist/  Communication: i2010 - A European Information Society for growth and Employment http://europa.eu.int/information_society/eeurope/i2010/index_en.htm

18. Further Information Frank Cunningham DG INFSO Evaluation and Monitoring Unit European Commission Tel : + 32-2-296-8119 Office BU 31 0/52 Ave de Beaulieu 31 1160 Bruxelles Belgium Email: frank.cunningham@cec.eu.int