Osamu NAKAMURA, Ph.D. Director for Technology Evaluation

1. Report about R&D Evaluation Workshop in Japan:Evaluation of Science and Technology Policy System in Japan: Country Case Study Osamu NAKAMURA, Ph.D. Director for Technology Evaluation Technology Evaluation and Research Division, METI JAPAN http://www.mri.co.jp/PROJECT/2005/20051011_ird01.html

2. Evaluation of Science and Technology Policy System in Japan: Country Case Study Organizers: Japan Society for Science Policy and Research Management (JSSPRM) Ministry of Economy, Trade and Industry (METI) Ministry of Education, Culture, Sports, Science, and Technology (MEXT) National Institute of Advanced Industrial Science and Technology (AIST) National Institute of Science and Technology (NISTEP) … In cooperation with WREN! (Washington?/World? Research Evaluation Network) Date: June 2-3, 2005 Place: Mita Public Hall, Tokyo, JAPAN

3. Policy Evaluation General Discussion in Seoul (May 30 & 31, 2005) Country Case Study Exercise in Tokyo (June 2 & 3, 2005) Program Evaluation Project Evaluation Policy Evaluation Exercises in Seoul / Tokyo In cooperation with WREN!

4. Evaluation of Science and Technology Policy System in Japan: Country Case Study • Session 1: Science and Technology Policy System in Japan moderator: Prof. Kondoh (Yokohama Natl. Univ.) commentator: Dr. Holland (White House, OSTP) Prof. Georghiou (Manchester Univ.) • Session 2: Evaluation of Science and Technology Policy and Reflection of its Result in Japan moderator: Prof. Niwa (Natl. Graduate Ins. for Policy Studies) commentator: Dr. Reeve (EC) Dr. Shipp (NIST) • Session 3: Evaluation of Science and Technology Programs and Reflection of its Result in Japan moderator: Prof. Miyazaki (Tokyo Inst. of Technology) commentator: Dr. Oros (USDA) Mr. Teather (NRC of Canada) • Session 4: Evaluation of National Research Institutes moderator: Prof. Hayashi (Natl. Inst. for Academic Degrees and Univ. Evaluation) commentator: Dr. Jordan (SNL) Mr. Valdez (DOE)

5. Evaluation of Science and Technology Policy System in Japan: Country Case Study Kick-Off: Prof. Vonotas (GWU)

6. National Guideline on the Evaluation for Governmental Research and Development (III) 2005 Progress of R&D Evaluation in Japan Ex-ante Monitoring/ Ex-post Impact As. 2002 Government Policy Evaluations Act Policy 2001 National Guideline on the Evaluation for Governmental Research and Development (II) Program Project 1997 National Guideline on the Evaluation for Governmental Research and Development (I) 2001 Law on the General Rules of Incorporated Administrative Agencies Institute Ryo Hirasawa, 30/08/2005, rh@rhirasawa.info

7. Evaluation of Science and Technology Policy System in Japan: Country Case Study Session 1:Science and Technology Policy System in Japan Japanese Science and Technology Policy System and Evaluation Dr. Tomohiro IJICHI Associate Professor, Institute of Innovation Research, Hitotsubashi University Affiliated Fellow, National Institute of Science and Technology Policy, MEXT

8. Ministry of Education, Culture, Sports, Science, and Technology Ministry of Economy, Trade and Industry Ministry of Health, Labour, and Welfare Ministry of Agriculture, Forestry, and Fisheries

9. Evaluation of Science and Technology Policy System in Japan: Country Case Study Investigation 6% Development 23% Basic 44% 20 Billions US$ Applied 27% Public R&D Budget in Japan

10. Evaluation of Science and Technology Policy System in Japan: Country Case Study Comments from Commentators: Evaluation results should be reflected to the budget on the uniform basis across different types of researches. In active research fields, many researchers at different research units should compete to stimulate each other. It would produce the new discipline with high yield. So, the balance between the administrative efficiency and competition of researchers is very important. Japanese evaluation system has no definite framework of ex-ante evaluation, so few projects/programs have clear targets to be accomplished. An incentive system to R&D projects based on the results of evaluation should be carefully set in order to promote reasonable competition.

11. Evaluation of Science and Technology Policy System in Japan: Country Case Study Session 2: Evaluation of Science and Technology Policy and Reflection of its Result in Japan Current Situation and Challenges for Policy Level Evaluation in Japan Dr. Jun SUZUKI Professor, Technological strategy, S&T policy, Graduate School of Engineering Management, Shibaura Institute of Technology

12. Law on the Evaluation of Science and Technology Policy Conducted by the Government 　　　　　　　　　　　　　　　　　　　 （Government Policy Evaluation Act, April 2004） METI MEXT METI Basic Plan for Policy Evaluations Basic Plan for Policy Evaluations achievement ・3 evaluation procedures ex-ante, ex-post, and synthesized evaluation ・ ex-ante and ex-post evaluation ex-ante: 78 projects ex-post: 8 projects ・achievement evaluation: 42 policy measures achievement 　・ex-ante evaluation for 113 policy measures, an integration of 840 projects since FY2002 　・of which 75 policy measures (42 are R&D) subject to ex-post evaluation 1. Backgrounds and track records for evaluation of science and technology policy Case：solar power generation Case：HIMAC He stressed the importance to employ the Balanced Score card and Behavioral Additionality to assess the effects of project performance.

13. Evaluation of Science and Technology Policy System in Japan: Country Case Study Comments from Commentators: It is very important to evaluate not only intended effects in the mid-term and long-term but also secondary effects. Besides objectives and goals, management of the performance is also very important element of program management. Project budget must be managed in a consistent manner throughout the project period. It is unclear in Japan who is responsible for the evaluation results. The evaluation system in EU sets importance in the balance of: accountability transparency justification learning trust in evaluation

14. Evaluation of Science and Technology Policy System in Japan: Country Case Study

15. Evaluation of Science and Technology Policy System in Japan: Country Case Study Session 4: Evaluation of National Research Institutes Revised Evaluation System to Reflect the Future A model＠National Institute of Advanced Industrial Science and Technology (AIST) ・Independent administrative institution under METI ・To perform innovation-oriented basic researches Dr. Osamu NAKAMURA Senior Researcher, Evaluation Department, AIST

16. Sustainable Development by Full Research Society Outcomes AIST Public resources People’s Interest (applause) dream nightmare reality Universities Type-I basic research Type-II basic research Industrial development Industries AIST Full Research Outcomes AIST places its highest priority on the pursuit of complete research, "Full Research", engaging Type-I Basic Research to the development of “products” by conducting intensive Type-II Basic Research.

17. Evaluation from the view point of OUTCOMES Procedure of Evaluation external reviewers: Specialists or Intellectuals internal reviewers: Principal Reviewers Strategy of AIST Research Units to make sure the research strategy ＊Roadmap Evaluation ＊Output Evaluation ＊Management Evaluation ＊Roadmap for outcomes ＊Output indexes ＊Management to know the level of performances and achievements to encourage researchers to perform Full Research Evaluation: Rating with comments Comments Resource Allocation Evaluation Department Reorganization Nation President of AIST Evaluation Committee Nation Evaluation Committee

18. Notes: • Evaluation is for Encouraging with Love. • Evaluation is for Communication. • Evaluation is for Strategy. • Evaluation is for Creation. • Evaluation is to be fed back. • Evaluation reflectstheFuture. • Evaluation is like a Compass for Voyage. “Revised Evaluation System to Reflect the Future” (O. Nakamura, AIST)

19. Evaluation of Science and Technology Policy System in Japan: Country Case Study Session 4: Evaluation of National Research Institutes Evaluation System of RIKEN ・Independent administrative institution under MEXT ・To perform comprehensive researches for natural science Dr. Masayuki UCHIDA Evaluation Section, Policy Planning Division RIKEN

20. 2-1 Advisory Council System of RIKEN RIKEN Discovery Research Institute & Harima Institute Directors, DRI & HI AC RIKEN Advisory Council AC RIKEN Frontier Research System Director，FRS AC RIKEN Brain Science Institute Director，BSI AC RIKEN BioResource Center Director，BRC AC RIKEN Genomic Sciences Center Director，GSC AC RIKEN Plant Science Center Director，PSC Recommendation AC RIKEN SNP Research Center Director，SNP AC RIKEN Research Center for Allergy and Immunology Director，RCAI President The Board of Executive Directors AC RIKEN Center for Developmental Biology Director，CDB Review Report Advice Response

21. Evaluation of Science and Technology Policy System in Japan: Country Case Study • Comments from Commentators: • It is very good to perform evaluation of a research institute based on • its strategic roadmap, outputs, and research management. • If you want to improve the research environment, you should set a • target and monitor indicators across years. • It must be emphasized to motivate researchers by setting • appropriate criteria for evaluation. • Important elements to create a good business model of a research • institute: • Core competency • Initiative and products (deliverables) from it • Leadership based on a rational criteria • Risk management (monetary risks, market risks, scientific risks, non-technical risks, etc.)

22. Evaluation of Science and Technology Policy System in Japan: Country Case Study Day2 Session 1: Applicability of logic model to policy evaluation Dr.Gretchen Jordan (SNL) Mr.George Teather (NRC of Canada) Session 2: Application of US PART for Japanese public RTD evaluation Mr. Bill Valdez (DOE) Dr. Mike Holland (White House, OSTP) Session 3: Application of EU FW6 Evaluation system for Japanese public RTD Prof. Luke Georghiou (UM) Dr. Neville Reeve (EC) Session 4: Current program evaluation methods Dr. Stephanie Shipp (NIST) Session 5: Training program for policy evaluation/R&D evaluation Prof. Nicholas Vonortas (GEU) Dr. Cheryl Oros (USDA) Session 6: Newest evaluation tools Dr. Tom Fiddaman (Ventana Systems)

23. Evaluation of Science and Technology Policy System in Japan: Country Case Study Summary: We conducted extensive discussions on methodologies/ focuses on policy/program/project evaluations on Japanese science and technology policy system as a country case study with the guidance of WREN– for the first time ever. Next Steps for Japan (METI): We are trying to introduce logic models into planning and evaluation of METI projects and programs. We consider that the some criteria in PART (US) or FP (EU), such as those for management, must be incorporated into Japanese S&T policy evaluation system. We should get into the network of the world evaluation community in order to apply global top-level evaluation methodologies to Japanese S&T policy evaluation system.

24. Evaluation of Science and Technology Policy System in Japan: Country Case Study

25. Thank you ! Evaluation of Science and Technology Policy System in Japan: Country Case Study

26. Evaluation of Science and Technology Policy System in Japan: Country Case Study Appendix

27. MIC 2% MLIT 2% MAFF 3% others MHLW 3% 4% JDA 4% 36 Billions US$ METI 17% MEXT 65% Evaluation of Science and Technology Policy System in Japan: Country Case Study R&D Budget of Japanese Ministries (including expenses for labors and facilities, 2005)

28. 3. Case study 2: Heavy Iron Medical Accelerator in Chiba (HIMAC)(3) Heavy Iron Medical Accelerator in Chiba (HIMAC) ◆Background to development ・　Construction of Heavy Iron Medical Accelerator in Chiba (HIMAC) began in 1984 as the world’s first heavy particle accelerator for medical treatment, based on the Comprehensive 10-Year Strategy for Cancer Control. ・　Clinical experiments began in 1994. ・　The device has been used in 2,184 people up to February 2005. ・　HIMAC was provided to researchers both in Japan and overseas as equipment for utilization in biological experiments that would provide the basis for medical treatment, as well as in physical and engineering experiments. Over 500 outside researchers use the device each year. ・　Highly-advanced medical treatment began in November 2003, and 341 patients have undergone treatment. ・　R&D of downsizing the equipment was implemented to spread heavy particle cancer radiotherapy. ◆Special features： ・　Causes no pain, placing less burden on patients ・　Induces little or no side effects ・　Can treat lung and liver cancer within a short period (less than 1 week) ・　Can treat refractory cancers (e.g., that of the bone and soft tissues) that respond poorly to other treatment methods