In-depth Evaluation of R&D Program in Korea

1. In-depth Evaluation of R&D Program in Korea Seung Jun Yoo, Boo-jongGill, Woo ChulChai

2. Contents 1. Overview of Public R&D Program Evaluation 2. Objective 3. Procedure 4. Methodologies 5. Utilization of Evaluation Results 6. Challenges and Discussion

3. Agency Agency Agency Agency Agency Overview of Public R&D Program Evaluation 1 - Players Report Taking Charge of R&D Evaluation and Budget Allocation MOSF NSTC Evaluation Supporting Groups KISTEP (Evaluator) R&D programs of each ministry MIFAFF MOE MKE MW MEST …… *NSTC (National Science & Technology Council), MOSF(Ministry of Strategy and Finance)

4. Evaluation strategy & Data collection R&D Budget Survey/Analysis Evaluation Programs/Projects implemented In-depth Corrections* Feedback Overview of Public R&D Program Evaluation 2 - R&D program management process Self → Meta Evaluation process Recommendations to program ministries Correction process

5. Overview of Public R&D Program Evaluation 3 - Overview of Public Finance Program

7. Overview of Public R&D Program Evaluation 4 - In-depth evaluation & Self/Meta evaluation In-depth Evaluation • ~ 10 programs with evaluation issues • logic model, evaluation questions, in- • depth analysis, communications, • recommendations, coordination, etc. • - depends on evaluation questions R&D Program Evaluation Self → Meta Evaluation • 1/3 out of all programs • (70 programs out of 207 programs, ‘09) • self evaluation by each ministry • meta evaluation by MOSF/KISTEP • - depend on indicators with weight

8. Objective of the Evaluation • To increase the efficiency and effectiveness • : find out and diagnose the problems at all aspects • : improve the program by applying evaluation results

9. Procedure 1 - 4(5) steps 0. Selecting target program

10. Procedure 2 - 7-month schedule (depends on the cases) - (month 0) : selected by selection committee based on special issues, etc. ¶ In-depth evaluation procedure for selected program(s) - month 1 : form evaluation group, gather program(s) data, study target R&D program(s), find major evaluation questions - month 2 : develop logic model and methodologies - month 3/4 : perform in-depth analysis (relevance, efficiency, effectiveness, program design & delivery, etc)

11. Procedure 3 - month 5 : interview (researchers, program managers, etc.) - month 5 : report interim evaluation result (MOSF, ministries) - month 6 : report final evaluation result & recommendations Large program : ~ 10 months Specific needs for short-term : 2 ~ 3 months (specify the needs → perform evaluation)

12. Methodologies

13. Methodologies_IPA

14. Methodologies_Survival Analysis Following Study Modified from: DeVol, R. & Bedroussian, A.(2006), Mind to Market: A Global Analysis of University Biotechnology Transfer and Commercialization. Milken Institute

15. Methodologies_System Dynamics Source: Ahn, N. (1999), A system dynamics model of a large R&D program, MIT Press Yoo, S. et al. (2009), In-depth Evaluation of Health & Medical R&D Program in Korea, KISTEP

16. Methodologies_PortfolioAnalysis

17. Utilization of Evaluation Results

18. Self/Meta Challenges and Discussion Efficient R&D Management In-depth • Evaluation as R&D Management Tool Measure the achievement according to performance plan • Efficient budgeting • Improve goal achievement Diagnose problems and correct to improve the efficiency/effectiveness

19. Challenges and Discussion • UCI concept among stakeholders • : Understanding – Change - Improvement • is important for raising the accountability • (responsibility + acceptability) of evaluation • : Understanding = communication with the facts • : Change = 4 types of corrections • : Improvement = efficiency & effectiveness

20. Thank you! Seung Jun Yoo, PhD biojun@kistep.re.kr www.kistep.re.kr