Evaluation of R&D Programs: State of the Art

1. Evaluation of R&D Programs:State of the Art Presented at Washington Research Evaluation Network - WREN Workshop June 6, 2008 Gretchen Jordan Sandia National Laboratories, USA gbjorda@sandia.gov, 505-844-9075

2. Interest is high– and all over the map To name just a few – • WREN • OECD Workshop on Rethinking Evaluation in Science and Technology. Paris 30.10.2007 (TIP Working Group) • Atlanta S&T Policy Conferences • American Evaluation Association Annual Conference- RTD Evaluation Topical Interest Group (most international of the TIGs!) • European RTD Evaluation Network • European Court of Auditors review of Evaluation of European Union RTD Programs • Investments in Health Research: Defining the Best Metrics --Canadian Academy of Health Sciences • Economic returns of medical research --Swedish Research Council, Medicine • NSF Science of Science and Innovation Policy grants program • American Chemical Society Petroleum Research Fund Jordan WREN June 2008

3. Research Assessment in U.S. G. Jordan, AAAS 2005 G. Jordan 02/18/2005 3 Jordan WREN June 2008

4. A glimpse of state of the art of RTD Evaluation From presentations of that Topical Interest Group at the American Evaluation Association 2007 Conference • Evaluation Systems • Frameworks • Methods View 2007 RTD presentations at http://rtd.aea.googlepages.com/ 2002-2006 at http://www.wren-network.net/ Join us at AEA in Denver, Colorado, November 5-8, 2008 Jordan WREN June 2008

5. Evaluation Systems

6. Evaluation in the Policy Cycle Technology Assessment Wolfgang Polt 30-10-2007 Foresight Technology Roadmapping Jordan WREN June 2008

7. Thematic level evaluations Evaluation system components – FP7 FP 5 FP 6 FP 7 Annualmonitoring FP6 ex post 5 yearassessment FP7 Ex ante Impact Assessment FP8 Ex ante Impact Assessment FP7 mid-term review National Impact studies 2002 1998 2000 2004 2006 2008 Jordan WREN June 2008 EUROPEAN COMMISSION - Research DG – November 2007

8. Program Evaluation in National S&T Activities National S&T Planning S&T Level & Trend Analysis Technology Level Assessment National Standard S&T Classification S&T Indicators & Statistical Analysis S&T Foresight & Roadmap R&D Survey & Analysis National R&D Priority Setting National R&D Master Plan(5 years) National R&D Budget Allocation Ministry Action Plan (every year) Ministry R&D Programs National R&D Program Evaluation Performance Review (every year) Overall Coordination for National R&D Programs 2007 AEA Conference KISTEP session

9. Systemic Evaluation Frameworks

10. The intellectual battle has culminated in a ‘national innovation systems’ perspective … Source: Arnold and Kuhlmann, 2001 Jordan WREN June 2008

11. High risk capital – available where Capabilities – Level, mix, availability Modes of coordination – effective? Socio economic outcomes Technical progress Network connectedness Organizational profiles – do attributes match the profile? Portfolios - need more/ less radical, large scope? RTD arenas – are there sufficient funds A Theory-based Framework for Evaluating Diverse Portfolios of Scientific Work All these work together…Key indicators for innovation bottlenecks/ policy objectives Macro- Institutional Rules as they affect the sector Meso - Performance by sector and arena Commercialization research Quality research Basic research INNOVATION Manufacturing research Applied research Development research if not, check for bottlenecks Micro - funds allocation by arena and profile Jordan, Hage, Mote Vienna April 24, 2006 Jordan WREN June 2008

12. An Example of Roadmap Jordan WREN June 2008

13. The Logic of Indirect Programs to Diffuse Technologies or Practices: EERE programs typically undertake these activities Fund and Promote Adoption Build Infrastructure Develop Technical Information Assist Public Entities Assist Businesses Outreach and Partner Assist and Fund Purchases Provide Tools and Technical Assistance Reviewing and Reporting Analyze and Plan Partnering with or targeting these audiences Technical and other personnel in laboratories, government, firms, colleges, universities Federal, state, and local agencies and nongovernmental organizations Investors and financiers, manufacturers, distributors, retailers, architects, engineers, trades people End user organizations, firms and individuals To achieve the following intermediate outcomes Market and product knowledge Create, advance, and package market and technical knowledge to make energy efficiency more accessible and implementable Change the policies, structure and operation of public entities to smooth the advance of energy efficiency and clean energy supply Create and enhance products, create and align market channels, enhance marketing, and develop installation and support infrastructures New knowledge, alternative institutional arrangements and processes, new product and service ideas, new opportunities, Adopt, replicate, institutionalize, and enculturate energy efficient and clean energy supply practices and technologies That produce the following long-term outcomes or impacts Reduced energy use and emissions, increased clean energy supply, and enhanced productivity and global security U.S. Department of Energy • We need to describe and measure the expected response of: • Knowledge workers • Public Entities • Businesses and manufacturing • End-users Knowing these activities and their corresponding outputs John H. Reed To show how activities are connected to impacts Jordan WREN June 2008

14. Methods

15. Recent attempts to use peer review at a high level recognise systems complexity - and are being driven to use background studies, increasingly using innovation system and evaluation specialists Examples • Finnish NIS Review • EU 5-year assessments • OECD ‘Innovation System’ reviews • EU-CREST ‘Policy Mix’ reviews Evaluation customer Secretariat Panel Expert(s) Background work Erik Arnold Jordan WREN June 2008

16. In research phase, good collaboration among private companies. - using “Follow-Up Chart” Case Study ＜Device development by collaborating with upstream and downstream industrial technology＞ Potential for practical use Achieved world record performance High Further technical problems requiring solutions before they can be practically applied Network of personal contacts Increased opportunities for business Achieved three times as high as performance by new method. ->Appropriate evaluation based on the R&D results. Post Project Evaluation In the development phase, upstream and downstream industry work closely together. legend Excellent collaboration between industry and universities Plus element Minus element In the practical phase, it can be difficult to maintain effective collaboration among companies. Intermediate evaluation Accelerated R&D efforts by setting very challenging target Low NEDO project encouraged newcomers to collaborate with universities, allocate new budget for R&D, etc. planning 1st yr. 2nd yr. 3rd yr. 4th yr. 5th yr. Post project Evaluation present Jordan WREN June 2008 AEA 2007

17. Historical Tracing Method • Forward tracing from R&D to downstream outcomes • Backward tracing from a selected outcome to upstream R&D Innovation 4 R&D Innovation 1 Innovation 3 Innovation 2 ? ? ? Target Innovation R&D TIA Consulting, Inc. Jordan WREN June 2008

18. Second Generation Patent Tree for US 5348822, Issued to Ovonic Battery Company in 1994 TIA Consulting, Inc. Jordan WREN June 2008

19. Research Network Deployment Network Organisations Participating in Both Networks working as Gatekeepers Bridging Links Question 2: What are the Network Characteristics? Gatekeepers: Bridging Research and Deployment Networks • There are 277 gatekeeper organizations • 1/3 of the links in each of the two networks are bridging links Franco Malerba, Nicholas Vonortas, Caroline Wagner, Lorenzo Cassi, Nicoletta Corrocher Jordan WREN June 2008

20. Measuring the Interdisciplinarity of a Body of Research David Roessner, Alan Porter, Anne Heberger, Alex Cohen, and Marty Perrault Jordan WREN June 2008

21. Summary and Conclusions Vision and Agenda of NSF-sponsored 2001 Workshop remain: Vision: More valuable and valued R&D assessment contributes to broad community understanding of S&T innovation systems and how they lead to and are influenced by social, technical, economic, and political development. Proposed Agenda: • Better networks, more training • Better data and methods • Research on Innovation, Societal and Regional impacts, and Complexity of emerging research organizations Communicate Better! Jordan WREN June 2008