Evaluating Research Dynamics Using Network Analysis (in the context of EU funded R&D)

Presentation Transcript

Frank Cunningham DG Information Society and Media European Commission

and

Caroline WagnerGeorge Washington University

CES/AEA Conference, Toronto October 27 2005

Shift Towards Evidence Based Policy Making in EU

Commission has become more an ‘executive arm’ of the European Institutions
Manages a budget of €100 B /year
Rising (possibly) to €1000 B in period 2007-13
Shift towards ‘evidence-based’ policy development reinforced by enlargement to 25 Member States

EU R&D Framework Programmes

Currently in the 6th Framework Programme
Total budget of €17B over 4 years (2003-6)
Information Society Technologies (IST) €3.8B
Annual EU investment in IST of €1B
EU investment about 3% of total IST-RTD in Europe.

Current Evaluation and Impact Assessment Methodologies

Commission has considerable experience in evaluation and monitoring of effectiveness of EU RTD investments over the last 15 years.
Focused mainly on ‘outputs’ rather than impacts.
Focused mainly on ‘project’ level and on the ‘private-return’ to RTD participants.
Now increased attention is given to results-based assessment and macro-economic and social impacts of EU RTD.

Gaining a better Understanding of Impact of EU RTD Funding

Work under way in DG INFSO since 2002
Established that EU funding has created an intensely linked network of European research collaborations.
New tools of ‘data mining’ and ‘link analysis’ were applied to EU IST databases.
Confirmed that network of collaborations is ‘small world’ with strong clustering
High degree of connectivity dominated by a small number of ‘hubs’

New Evaluation Methodologies

Innovative firms do not innovate in isolation but in interaction with other ‘actors’
In ‘Innovation Systems’ the links between innovation actors are crucial
Collaborative R&D Programmes are one important way of strengthening linkages
Partnerships through collaboration networks helps to achieve ‘critical mass’ of dispersed EU R&D resources.

Series of Evaluative Studies

DG INFSO is commissioning a series of evaluative studies to answer evaluative questions about the networking aspects and dynamics of:

IST RTD
Innovation & Deployment

Part of our strategy for network evaluation and impact assessment

Series of Evaluative Studies

1. ERAnets Study (RAND Europe)

(subject of this presentation)

2. Evaluation of Progress towards ERA for IST (CESPRI)

(subject of following presentation)

3. Networks of Innovation in IS Development and Deployment in Europe

(tenders in evaluation process)

ERAnets Study (RAND Europe)

Main Objectives were to Assess:

The degree to which IST Researchers in Europe collaborate.
How integration of IST Research in ERA changed with introduction of new FP6 Instruments.

Carried out from Jan 2004 to March 2005

Collaboration Has Grown SignificantlyEurope - 1990

Collaboration Has Grown Significantly Europe - 2000

ERAnets Study Breaking New Ground in Evaluation

An Evaluativestudy supporting the Evaluation and Monitoring Unit of DG Information Society
Analysed the networks in IST Research and compared the current networks emerging in the Framework Programme with those emerging in other frameworks and communities
Providing insight into the structuring effects of FP6 on European IST RTD
Developed, and refined tools and datasets that can be used by the INFSO Evaluation and Monitoring Unit and others

FP6 Networks of Excellence Produce Highly Clustered Network

NoEs, call 1 retained, projects as cliques, component 1, diameter 3, average path length 2.1 (colour is number of steps away from the centre, from 1-3)

ERAnets Study – Main Findings (1)

Main Findings show that IST RTD funding at the European level is most effective in:

Connecting universities and business
Connecting research in different themes and disciplines
Integrating new Member States
Integrating key patent-holders; and
Integrating small businesses

ERAnets Study – Main Findings (2)

Main Findings also show that:

FP6 network participants are more tightly interconnected than previous FPs
FP provides a major integrating function

…however

Large institutes and companies act as ‘gate-keepers’
SMEs have been ‘crowded out’

Next steps: 2005-6

Broadened RTD network analysis to the “ERA in the Global context”: CESPRI (in 2005)
Extending network analysis from RTD to include innovation and deployment : to be contracted in 2006.
Clarifying the “causality links” between IST-RTD and FP and “Lisbon” objectives : Technopolis (10/04-8/05)
Initiating the systematic use of key indicators for monitoring the output and impact of IST-RTD at the Strategic Objective-level;
Identification and assessment of the capabilities of “system dynamic” model and tools for macro-economic/social impacts of investments in IST-RTD and ICT deployment : to be contracted;