An international comparative analysis

1. Research support to the Fagerberg CommitteeInternational comparison of goal-oriented public governance in research policyErik Arnold25 October 2010

2. An international comparative analysis • The team has been busying itself with desk research to answer the study questions, for each of our seven countries • Canada, Denmark, Finland and The Netherlands • New Zealand, Sweden and the UK • The work has been organised around the seven topics listed in the study specification • Description of research systems and allocation of funding • Efficiency of the research performing sectors • Research education • Research funding mechanisms and criteria • Monitoring performance of research grants • Cataloguing research production • Examples of successful goal-oriented research policy

3. Progress to date • The team has produced draft reports for the seven countries • Each paper is organised using a common structure, written around the seven topic areas • The seven papers have been compiled in a compendium, which will form an annex to our final report • The team has begun to carry out horizontal analyses of each topic, in order to develop our synthesis and final report • The team is drawing on the facts and figures in the individual country papers and cross-referencing this empirical material with more conceptual and evaluative material where practicable • We have so far produced 4 of our seven cross-sectional papers • Research efficiency, research education, funding mechanisms, monitoring of grants • The three other papers are in progress • We have also been compiling selected international statistics, principally from the OECD MSTI to help structure the synthesis • We have taken three of these cross-sectional papers and turned these into a series of slides, to show preliminary results and illustrate our treatment of the questions

4. Progress (Vertical analysis) • Draft Reports and research focusing on the countries

5. Progress (Horizontal Analysis) • Draft reports and research focusing on the questions

6. Context – Research Policy Goals • High-level goals determined at government level • Research typically comes under responsibility of departments for education or industry/economics, or a focused ministry for science/ research/technology/ innovation • Key goal for all countries is research quality • Further goal is research impact i.e. level of innovation and resultant economic/social benefits –however this latter goal is stated more less explicitly in different countries • Indicators assigned to government/ ministerial goals and, in some cases, targets set e.g. in UK (for quality, BERD, innovation activity), Denmark (GERD – increase to 3% of GDP). Indicators monitored at regular intervals (e.g. annual in UK) • Hierarchy of goals filters down to funding agencies, with appropriate indicators set at each level that link back to key goal of research quality (and innovation where required)

7. Context – Research Quality and Impact • Research Quality: • Assessed via peer review and/or objectives metrics such as bibliometrics, external research • Research Impact: • At national level this is determined indirectly via, for example, innovation indicators • UK currently attempting to assess/measure research impact at HEI/ research group level via a system based on representative case studies • Assessment of quality and/or impact can be used to: • Determine effectiveness of policy • And/or to assign( future) research funds

8. Topic 2 – The efficiency of research administration • Information on the research performing sectors with regard to spending of funds for research versus spending of funds for research administration, and on the development of this over time.

9. 2.1 – Research efficiency • We began with an hypothesis that an increasingly goal-oriented approach to the governance of research policy might require proportionately more administrative effort • To determine where to invest, whether due to anticipated future challenges or measured, past performance • To confirm that those investments have been expended as planned and that they have produced the outputs and outcomes sought • Following a wide-ranging review, we concluded that we could not answer this question in any definitive sense • The only international statistics of any relevance were the OECD data on total R&D personnel by occupation and by research performing sector • These statistics are still somewhat limited however • They do not include staff within central administrations (e.g. the 5-person team working in a university research office helping bid EU RTD FP calls or addressing research performance exercises) • Nor can headcount statistics differentiate between research and research administration within an broad occupational grouping (e.g. a professor standing as a member of a peer-review panel) • We found no studies, in any of the 7 countries, which investigated this phenomenon, however we did find several studies that looked at narrow aspects of research costs • Academic and other community-wide consultations express concern about the additional burden of performance-based funding as compared with the more traditional, self-regulated system where peer review of grant applications and books / articles governs all • The UK government did commission a study of the costs (cash and in-kind) of the research assessment exercise and peer review, which found the former amounted to around 1% of the science budget annually while the latter amounted to 10%

10. 2.2 – Trends in research employment • The OECD statistics for 2005 suggest that research administrators account for • 10-20% of headcount in business • 10-35% of the headcount in the government / institute sector • 10-12% in the HE sector (data are patchier here) • The previous years’ data (previous 5 years) show that • Almost every sector in every country has decreased the % of support staff over time • Almost every sector in every country has increased the % of researchers over time • The %of technicians has been broadly stable • The data suggest that all organisations, particularly business, have been striving to improve efficiency of their research spend by reducing in-house support staff • We can draw no strong conclusions here however, due to • The data limitations (occupational headcounts, technical change or outsourcing of administrative functions) and • The vintage of these data, which pre-date the gathering global commitment to switch to a more goal-oriented approach to the governance of research policy

11. 2.3 – Distribution of R&D personnel by occupation and by sector (2005) Source: Technopolis Based on OECD statistics

12. 2.4 – Research efficiency (methodology, definitions) • Using OECD data on types of personnel in R&D, we can determine the percentage of support staff, researchers, and technicians in a given sector and over time (1981 onwards). • Definitions • Researchers- Staff engaged in the conception and/or creation of new knowledge /products. Personnel involved in the planning or management of scientific and technical aspects of R&D projects, and software developers. • Technicians- Staff engaged in technical tasks in support of R&D, normally under the direction and supervision of a researcher. • Support Staff- Includes administrative and managerial staff working on, or directly associated with, R&D activity. Does not include staff outside the R&D performing unit providing indirect support. For example central finance or personnel services and central support services e.g. information services and cleaning. • Combined with qualitative analysis, from desk research • National and institutional Annual Report data is of very limited use because administration staff / expenses are not solely for research

13. Topic 3 – Research education • Information on how recruitment to research (research education) is organized/financed, and by which criteria PhD scholarships are funded (if applicable).

14. 3.1 – Funding mechanisms for doctoral training • 5 of 7 countries (all but UK & Canada) allocate block grants to universities to allocate to departments and doctoral researchers – with varying degrees of direction from funding bodies • 2 countries (UK & Finland) also have a competitive element for doctoral programmes – open competition in Finland, in UK the allocation dependent on research council (competitive) grant income • In 5 of 7 cases, Government funding covers university costs in full through employment contracts or grants • In Canada and the UK, a substantial proportion of the costs is covered by fees (20-30%) • In the UK, 20-30% of total costs are covered by the universities through fee waivers, with some cross-subsidy from international students and from student contributions to the execution of research council-funded projects • In a proportion of cases, stipends to doctoral researchers • Some additional funding from industry for individual doctoral researchers

15. 3.2 – Criteria for doctoral training • Criteria: • Block grant to institutions – various combinations of capacity & quality measures (typically based on the same method used to allocate block research grants – e.g. NZ bases doctoral allocation on same quality measure (PBRF) as block research grants) • Competitive doctoral programmes • UK: based on competitive grant income from each research council • Finland: open competition – review and selection by funding agency • Doctoral applicant quality – undergraduate relevance and performance, academic/ employer references, interview – fairly standard approach across all countries

16. Topic 4 – The basis for the allocation of public funding • Information on which criteria and mechanisms form the basis for allocation of public funding for research in the different performing sectors and or institutions.

17. 4.1 – Research funding overview Objective: Quality and Impact of (publicly funded) Research Government R&D expenditure Funding source: Over-arching allocation method: Institutional core funding Project funding Impact e.g. potential impact of proposed research projects Capacity/ volume Quality e.g. quality of past research Quality e.g. quality of individual proposed research projects and/or Criteria: Assessment method: Objectiveindicators e.g. no. of research staff Objective indicators e.g. no. of outputs bibliometrics Qualitative indicators e.g. Peer review Qualitative indicators – Peer review Qualitative indicators – Peer review

18. 4.2 – Funding mechanisms • For universities • A majority of countries under review has implemented a performance-based funding mechanism, to improve the international standing of academic research nationally • In 5 of 7 cases, governments allocate a meaningful proportion of total national HE research income, based on adjudged past performance • Denmark, Finland, NZ, Sweden and the UK operate bimodal systems, providing institutional funds through a performance-based mechanism and project awards through competitions • Canada allocates the great majority of its funds for centres or projects, through competitions • The Netherlands uses direct funding of institutions by ministry of education and science, combined with competitions / calls • For research institutes • All countries provide core institutional funding through a negotiated procedure, which is non-competitive but often linked (softly) to performance targets • Most countries allow institutes to compete for funding within selected national competitions • For firms, in all 7 countries, public research funds are awarded through competitive measures, whether that is programmatic calls for proposals or project-specific ITTs

19. 4.3 – Performance-based reviews • In all cases, PBRs are national in scope • They cover all disciplines • They are comprehensive exercises, run periodically, 5 – 7 years • Research quality is the dominant criterion • In 3 of the 5 countries where PBRs operate, they are built around peer review • In Denmark, the exercise involves objective data only • In NZ, peer review of quality determines 60% of future income while more objective data on achievements (e.g. external research income and completed degrees) determine the balance • Evaluation processes • Quantification of achievements (e.g. esteem and prizes won, or appointments secured) • Bibliometric analyses, taking national and international publication and citation data from Thomson Reuters or Elsevier SCOPUS • Peer review of monographs, journal articles, non-text artefacts etc (e.g. top four articles for each submitted researcher) • Evaluation criteria • Research quality (e.g. adjudged excellence, citation rate normalised against average for field internationally) • Contribution to social and economic benefits (e.g. external research income) • Efficiency / productivity (e.g. degree completions) • Funding formulae • Usually include volume measures as well as performance measures • Usually provide a non-linear distribution, wherein the best research groups / institutions might be awarded a multiple of the funding awarded to those groups rated just one scale lower

20. 4.4 - Public-sector research competitions • We find a high-degree of commonality here • All countries provide project-based funding to universities through research competitions addressing most if not all disciplines, with the aim of maintaining strong disciplines • Targeted or strategic competitions where relevance (to the target) is given equal weight alongside quality and originality • Response mode often open calls, where quality / originality is the primary criterion • Countries are extending compliance tests around for example verification of approval by appropriate research ethics / governance committees or the existence of policies relating to environment to diversity • Success rates vary from 20-50%, typically • All countries operate targeted research programmes some of which are open to institutes or companies and which have strategic objectives (e.g. industrial competitiveness or grand challenges) • In these cases, the competitions are almost always closely targeted on issues of strategic importance to the country in question • The evaluation process will typically involve proposals being scored and judged by peer review panels, using a longer list of criteria than one might find in academic competitions • Relevance, coherence and economy are scored alongside originality and methodological quality • Peer review panels often involve research users as well as researchers • The application process is more likely to operate with multiple stages, perhaps beginning with expressions of interest, moving on to short proposals score by administrators and invited full proposals being scored and peer reviewed • Success rates are highly variable, but typically range between 5-30%

21. 4.5 – Effectiveness • Performance-based research assessment systems • We found only one official evaluation of a national system, which is the UK Roberts review of the RAE2001 (2003) • UK academic papers (e.g. Martin 2009) show improving share of all research output being rated as internationally outstanding and a gradual concentration of • However, this is a largely national peer-review of national research and OECD citation statistics don’t show a strong positive trend (Norway / Finland have registered much stronger improvement in previous 20 years, without an RAE) • NZ and the UK have commissioned independent bibliometric analyses, which suggest performance is being sustained • Other countries tend make do with a much more limited and aggregate set of citation data taken directly from the OECD MSTI • Competitions tend to be evaluated indirectly, as part of broader reviews • International disciplinary reviews, wherein international peer review panels will be invited to judge the quality of a country’s scientific endeavour against their view of the international state of the art (which can include consideration of the effectiveness of instruments). This is seen in Finland, Sweden and the UK • Programme evaluations, wherein an external, independent review will typically be required to look at both effectiveness and efficiency. The latter is likely to include a test – with ‘users’ and against international benchmarks – of the transparency, fairness and efficacy of the operational arrangements

22. 4.6 – Good practice • Good practice in goal-oriented allocation of research funds remains somewhat subjective (without substantive evaluations of the effectiveness of the specific mechanisms) • However, the scale, complexity and cultural shift of PBRs does mean that most have been introduced only after wide-ranging consultation • The arrangements are specified in great detail in most cases, in publicly-available, English language guidance material (DK, NZ, SE, UK) • The results of consultations are also available in some cases • New departures / refinements have also been extensively piloted and documented (e.g. UK piloting of the use of bibliometrics, wherein all STEM subjects at 11 HEIs were run through a citation analysis and the results profiled on four points [i.e. below world average for the field, at world average, twice world average and 4 times world average, with the latter being the benchmark for internationally outstanding work])

23. Increasingly common for a proportion of HERD to be driven by a PBR 35% of HERD in the UK to 15% in NZ, big enough to cause institutions to conform Spectrum in PBR approach ‘Objective’ (Denmark has implemented a metrics-based approach in past 12 months) A hybrid approach as in NZ where peer review determines 60% of the marks and metrics the rest A moderated approach whereby metrics are provided to peer review panels to inform judgements New criteria in evidence UK is proposing to allocate up to 25% of future institutional funding against adjudged performance on research impact (breadth and depth of non-academic inputs as demonstrated by case study) Simplification / efficiency projects UK and NZ pushing hard at streamlining their process (e.g. on-line process for applications and for peer-review material) DK chose a metrics-based approach in part to minimise the costs of the new procedure (although the approach is contingent on a national research cataloguing project to permit the national agency to award ‘bibliometric’ points to universities, e.g. 6 for a monograph, 5 for a thesis) Incentives to researchers to take part in peer review processes UK Research Council (EPSRC) Reviewers Incentive Scheme for academics Other countries limit for instance the number of reviews (e.g. two calls for participation in the Academy of Finland) 4.7 – Developments

24. Next steps for the study • Next steps • Conclude the cross-sectional analyses (last three topics) • Revisit draft country reports to fill key gaps where practicable and to improve consistency across the seven papers • Complete the synthesis report • Prepare a summary with conclusions (and recommendations?) • Submit draft report to the Committee by end November 2010 • Today … • Any questions …