Woo Chul Chai

1. Reformation of National R&D Program Evaluation System Woo ChulChai

2. I Overview of S&T in Korea I I I I I Contents National R&D Program Evaluation System Concluding Remarks 2

3. Overview of S&T in KOREA 3

4. Changes of S&T Environments in Korea 1960s 1970s 1980s 1990s 2000s Policy trend Technology- oriented policy Industry- oriented policy Enhancing technology innovation Building R&D infrastructure Promoting R&D GRIs established (e.g. KIST) MOST established in 1967 Daedeok Science Town built in 1974 Enforcing the national R&D programs Promoting the university -based researches (e.g. SRC, ERC) S&T policy direction Increasing efficiency of R&D investment (coordination of S&T- related policies) Planning Total Roadmap Heavy industry goods & electronic products Light & heavy industry goods Change in focusing industry Light industry goods Electronic & transport products Primary goods 4

5. Brief History of National R&D Programs Year New Growth Engine Program MOST National R&D Program MOCIE Industrial Technology Program MKE MIC Information & Communication Technology Program ’08 MOHW MEST Health & Medical Technology Program ME Environmental Technology Program MAF Agricultural Technology Program MOCT Construction & Transportation Technology Program Academic Research Promotion Program MOE 5

6. Science Competitiveness Technology competitiveness S&T Development in Korea (Quantitative Growth) 2th 0 5th 6th 6th 5 8th 12th 7th 12th 10 14th 14th 15th 16th World ranks 15 17th 19th 21th 20 25 27th 30 2001 2002 2003 2004 2005 2007 2008 2006 Source : IMD reports 6

7. National R&D Program Evaluation System 7

8. History of National R&D Program Evaluation Increasing Needs of Effectiveness of R&D Investment Performance Management& Evaluation Enhancement of Effectiveness, Efficiency & Accountability Performance Evaluation Transformational Stage Survey· Analysis· Evaluation ○ ’08~ ○ Law on Performance Evaluation and Performance Management of National R&D Program Revised(‘08.2) ○ Focus : Performance Evaluation & Management for Enhancing Effectiveness of R&D Investment Effectiveness of R&D Investment Growing Stage ○ ’06~’07 ○ Law on Performance Evaluation and Performance Management of National R&D Program Enacted(‘05.12) ○ Focus : Performance Achievement based on the Results of R&D Activities Entering Stage • ○ ’97~’05 • ○ S&T Framework Law • Enacted(’01.1) • ○ Focus : Appropriateness • of Input and Output 8

9. Architecture of Program Evaluation in NES NSTC Review of Plan Reports of Results Specific Evaluation Meta Evaluation MOSF • Basic Plan for Performance Evaluation (5 yr) • Action Plan for R&D Evaluation (annually) • To evaluate the appropriateness of Self-Evaluation • To review evaluation procedure and methods of Self-Evaluation • In-depth evaluation on major national R&D programs • Long-term/large-scale programs • Joint program among ministries • Programs which need to remove redundancy and require connectivity • Programs at national issue • Mainly examined by the Evaluation Committee Evaluation Results Evaluation Guideline Self Evaluation Ministries • Planning Self-Evaluation according to MOSF’s guideline • Implementing Self-Evaluation • Annually practiced by Ministries • Implementation based on self-made performance indicators and methods 9

10. Lifecycle of National R&D Program Evaluation Evaluation Range Feasibility Analysis Self- Eval Meta-Eval Specific (In-Depth) Eval Self- Eval Meta-Eval Self- Eval Meta-Eval Follow-up Eval +3year +3year + 1year Time Start year Closing year Implementation Ex-ante Ex-Post 10

11. Self/Meta-Evaluation of National R&D Programs (1) Designed after PART (Program Assessment Rating Tool)of the U.S. Federal Government Line Ministriesassess their own programs every 3 years The assessment is based on 25 checklist (questions)for all types of programs - Types of programs: R&D (Basic, Applied Research, Development), Infrastructure Investment, Procurement of Large-scale Facilities and Equipment, Human Resource Development MOSF reviews the assessment results and reflects them in annual draft budgets and the National Fiscal Management Plan 11

12. Self/Meta-Evaluation of National R&D Programs (2) • Answers to the questions take the form of “Yes (5)” or “No (0) • - In case of the questions regarding the achievement of • program goals, 2-scale answers (5, 0) is given. • A different score is assigned to each question and the result of • assessment is given by the sum of score • - Classified as “Effective (95-100),”“Moderately Effective (90-94),”Adequate (75-89),” and “Not Effective (0-74).” 12

13. Evaluation Items and Score (3) 13

14. Concluding Remarks 14

15. Limitations Little enthusiasm from line ministries Line ministries did not set upa clear framework of mission and strategy Performance indicators were not derived from ministerial missions in a systematic way Assessment of performance relies on subjective assessment by outside experts and in-house staff, not systematically utilizing indicators Cooperation and coordination among players are not sufficiently made 15

16. Future Works for Evaluation Developing Strategic Performance Management Frame - Reviewing Performance Indicators, Monitoring R&D Activities, etc. Enhancing Education & Consulting for R&D Program Officers - Organizing Education Program of Performance Management - Developing Logic Model Manual for Various R&D Program Types Linking Evaluation Result with Budget Allocation 16

17. Thank You!! Chai, Woo Chul wcchai@kistep.re.kr