Dr. Jim Ryan

1. Evaluation of Science Foundation Ireland: A Case Study of a Programme Designed to Enhance National Research Quality Dr. Jim Ryan Michael Fitzgibbon

2. CIRCA Group Europe • CIRCA is an international consulting company, established in 1991 in Dublin, Ireland. • Member of ETAN (European Technology Assessment Network) and the European S&T Observatory (ESTO). • 12 Consultants in different areas of S&T, all with significant experience of European S&T issues and systems. • The major Irish consultancy company specializing in S&T policy and management issues.

3. Forfás is the national board responsible for providing policy advice to Government on enterprise, trade, science, technology and innovation in Ireland. Legal responsibility for the promotion and development of these sectors is vested in Forfás by the State, through the Department of Trade, Enterprise and Employment. About Forfás www.forfas.ie Michael.Fitzgibbon@forfas.ie 3

4. Presentation Outline • Irish infrastructure – the context • Background to Science Foundation Ireland • Evaluation Process

5. Ireland in Summary • Population 3.98 million • Total at work 1.81 million (was 1.1 million in 1990) • GDP per capita€33,919 • A GROWING ECONOMY - 2003 • GDP €135 bn • Exports €113 bn • Imports €92 bn • Trade Surplus €21 bn - 15.5% of GDP • The average annual rate of increase in Irish GDP has been 9% over the 7 year period 1994 - 2000, the highest in the EU and the OECD

6. S&T Context: Some notable features • Ireland has historically had very low S&T investment • Universities & Institutes of Technology are almost the solepublic source of technology & RTD services: Only Public RTD institute is in Food/Agric. • 3. S&T was a low priority, and the agency structure was under-developed and staffed, until very recent times

7. Profile of Enterprise in Ireland Most of the products we manufacture are designed elsewhere Bulk of our exports are marketed /sold by organisations outside Ireland

8. Changing the Capability Profile of Enterprise in Ireland • Need to Build World Class Expertise • International Sales & Marketing • Operations • R&D/Technology • Focus on Sectors & Niches

9. “A world class research capability in selected niches of … enabling technologies is an essential foundation for future growth. These key technologies require new skills, and in particular the development of a cadre of world class S&T personnel.” Technology Foresight Ireland: Conducted 1998; published by Forfás 1999

10. Foreign Direct Investment is important to economy (~130,000 jobs; 14% of GDP) • Innovative capacity seen as important to the attraction and retention of FDI firms • Also vital to the growth of indigenous companies • Investment in RTD was seen as important: • To provide an incentive for Foreign firms • To develop indigenous high-technology industries ENTERPRISE IRELAND Ireland –S&T & the Economy

11. “.... There will be a major accelerated increase in Research,Technological Development andInnovation (RTDI) investment with the objectives: • to strengthen the capacity of Irish third level institutions and other research establishmentsto conduct research relevant to the needs of the Irish economy; • to strengthen the capacity of Irish firms to assimilate the results of R&D into their productsand processes, and; • to provide support for sectoral research in agriculture, food, marine and the environment....”

12. RTD and Innovation Funding in National Development Plan 2000 – 2006Total : €2.5 billion PRTLI Capital Facilities & Research Infrastructure in Universities €690m Other Sectoral & Industry Supports €1,160m Science Foundation Ireland. Quality and Scale of Basic R&D €650m

13. Science Foundation Ireland provide researchers Research Community view: SFI v PRTLI PRTLI provides infrastructure for research performance Bricks & Mortar Grey Matter

14. Conway Institute–Dublin Programme for Research in Third Level Institutions(PRTLI) Forced Universities to develop Strategic Plan for RTD Built 33 new Research Centres Provided an additional 90,000 m2 research space Funded 62 new or expanded research programmes Increased undergraduate Post-grad places by approx 1,500 Encouraged inter-disciplinary research and inter-institutional collaboration Encouraged greater interaction between research and teaching Materials & Surface Science Institute – Limerick Nanofabrication facility – Cork

15. ‘Helping Ireland Recruit and Retain Research Groups’ • SFI was established in 2000 and became operational in 2001 • ‘SFI is investing €646 million in academic researchers who are likely to generate new knowledge, leading edge technologies, and competitive enterprises in the fields underpinning two broad areas: • Biotechnology • Information & Communications Technology’ Vision Statement Through strategic investments in the people, ideas and partnerships essential to outstanding research in strategic areas, SFI will help build research of globally recognised excellence and nationally significant economic importance.

16. Grants are allocated solely on the criterion of excellence as determined by external, international peer review: relatively unusual in EU • Have established 163 research groups led by principal investigators of whom 34 have come to Ireland from laboratories abroad • SFI groups employ over 1150 research staff and 450 PhD students • Established 6 major R&D Centres in collaboration with industry

17. Six Centres for Science, Engineering and Technology (CSETs) have been established in core thematic fields and with considerable industry involvement. • The CSETs are in the following areas : • Adaptive Nanostructures and Nanodevices (CRANN in Trinity College Dublin) • Regenerative Medicine (REMEDI in NUI Galway) • Digital Enterprise Research (DERI in the NUI Galway) • Alimentary Pharmabiotics (APC in University College Cork) • Telecommunications Value Chain (Trinity College Dublin) • Human Proteomics (College of Surgeons, Dublin)

18. Requested by Department of Enterprise, Trade & Employment at end ‘04 Managed by Forfás. Department ‘Accept that SFI is a long-term investment ... …. but also know that the early stages are critical’ it is therefore ‘not too early to form an initial impression about the operational performance of SFI to date’ Review of the performance and impact of SFI to date.

19. 1. International review panel led the evaluation. Small size of Ireland makes it difficult to find local independent panel-members for such evaluations. External panels are therefore the norm. 2. Consulting groups were commissioned (by public tender) to provide support and information inputs to the panel. CIRCA Group conducted the Peer Review Assessment and the Bibliometric analysis ( with CWTS, Leiden, NL) Technopolis (UK) conducted a survey of industry views Review of the performance and impact of SFI to date.Evaluation Methodology:

20. Professor Sir Richard Brook (Chairman) Director of the Leverhulme Trust (UK) Prof. Karin Markides Deputy Director General of Vinnova, Swedish agency for Innovation (Sw) Dr. David Clark Chair of Computer Science & Telecomm Board, US National Academies (US) Dr. Wilhelm Krull Secretary General of the Volkswagen Foundation (D) Prof. David Finnegan Institute of Cell & Molecular Biology, Univ. of Edinburgh (UK) Mr. Pat Toole Retired Senior VP of Corporate Manufacturing & Technology of IBM (US) Members of International Panel

21. Appropriateness or Efficacy: Are the objectives still consistent with the Irish research system and with national research and innovation policies? Is the investment in SFI appropriate to meet its objectives? Are there sufficient and appropriate complementary measures to SFI which would allow the desired economic effects to materialise? Effectiveness: Is SFI effective in meeting its objectives? Are its programmes and activities likely to lead to the desired outcomes? What are the outputs and impacts of its activities? What impact is SFI having on the research system as a whole? Efficiency: Examine the operational efficiency of SFI in relation to a range of relevant issues (e.g. application and review procedures, monitoring). Terms of Reference

22. Conducted discussions with a representative cross-section of relevant groups: SFI-Funded Researchers (44) , post-doctoral staff (33) and students (34) funded under SFI awards; SFI management and administration (5) Management staff of institutions housing SFI researchers (13) Other key stakeholders: Public agencies, National Research Funding bodies and private sector (6) Discussions were conducted in 5 venues over 1 week Panel Methodology

23. Process Design Peer selection: qualifications, nationality*, experience, gender, etc Peer process: criteria used, information provided to peers; ranking and rating processes; Internal SFI Process: mechanisms and speed of decision-making; rejection rate etc Supports provided to successful and unsuccessful applicants Oversight and monitoring: conflict-of-interest rules; appeals process; internal audit/evaluations Process Operation Random Case files were examined to assess the practical operation of the review process. These were balanced by discipline, university, successful/unsuccessful applicants, and by type of award. Evaluation of the Peer Review systemCIRCA Group * No Irish reviewers are used

24. Bibliometric Analysis CIRCA Group and CWTS (Leiden) Two studies were commissioned to assess: • Quality of the SFI-funded researchers from non-Irish institutions prior to their coming to Ireland. This was to determine whether the SFI system had selected excellent researchers • The publication record of researchers who had been working in Irish institutions prior to funding. This was to assess whether the receipt of funding had affected their publication output and/or quality Note that appointments of researchers by SFI commenced in 2001

25. Percentage of SFI-funded researchers with a higher than average number of papers in the top 10% and Top 5% of papers in their respective fields * i.e. researchers who were working outside Ireland when receiving SFI award

26. Industry View of SFITechnopolis (UK) In-depth face-to-face or telephone interviews with 24 companies were conducted in January/ February 2005. The interviewees were from a mix of large foreign-owned multinationals, large indigenous firms and small high technology start-ups, all operating in business sectors with an interest in biotechnology or information and communications technology research. • Generally very positive about SFI approach and activities • Some concern about lack of incentives in HE sector re IP protection etc • Recognise that time is required for policy to bear fruit

27. Overall View of Panel • SFI is a ‘bold and well-designed path to transform Irish research’ • There are ‘persuasive indications of the suitability and effectiveness of its process’ • The defining aspect of SFI is its ‘demand for excellence in research quality’ • Continued success will need ‘ongoing commitment to sustenance of a research culture by government and universities’ • Report to be formally published by Forfás in November 2005