Zheng Yonghe Deputy Director General, Bureau of Science Policy, NSFC Kunming, May 23, 2014

1. Evidence-based policy making practice based on the international evaluation of funding and management of NSFC Zheng Yonghe Deputy Director General, Bureau of Science Policy, NSFC Kunming, May 23, 2014

2. Outline • NSFC’s funding policy development: Dimension discussion • The basis of policy making: Evidence-based evaluation • Essential work of evaluation:Evidence collection • Case study: International evaluation of funding and management of NSFC and the policy making

3. NSFC’s funding policy development: Dimension discussion • Principals: Problem-driven; reform-driven

4. The basis of policy making: Evidence-based evaluation The national performance evaluation system has not established in China, currently. NSFC’s international evaluation on funding and management performance provides a typical success case for comprehensive evaluation in China. Evaluation as an research-based approach plays an important role in policy making. Evidence-based evaluation insure the legality and credibility of policy in policy argument process. Question-oriented evaluation and independent third-party evaluation are more feasible and convincing.

5. The basis of policy making: Evidence-based evaluation The pathways and period of suggestion into policy

6. Essential work of evaluation: Evidence collection • Designing evaluation plan based on evidence Evidence-based Evidence-based Evidence-based

7. Case Study: International evaluation of funding and management of NSFC and the policy making • Brief overview of international evaluation of funding and management of NSFC • The evaluation is commissioned by MOF and NSFC together. The State Council attached importance to this evaluation. • Dual Objectives of the Evaluation: • Accountability: Provide an independent assessment of the overall performance of NSFC’s funding and management during the past 25 years. • Lesson learning: Improve the NSFC’s funding and management performance; redefine NSFC’s strategic role within the NIS of China

8. Brief overview of international evaluation of funding and management of NSFC • Evaluation Approach and Principals • Modality: Domestic preparation followed by international evaluation • Approach: Evidence-based • Principals: Independent, Objective

9. Evaluation Framework Based on the dimension discussion

10. Brief overview of international evaluation of funding and management of NSFC • Functions of Actors in the evaluation Synthesis evidence report: Provided by independent evaluation organization • Typical cases: • Provided by NSFC • International evaluation report

11. Brief overview of international evaluation of funding and management of NSFC • Implementation of the Evaluation

12. Cases report Thematic report Literature metrology report Survey report Interview report Special report • 2. Evidences and the evaluation procedure Designing evaluation scheme design Typical cases Thematic research Collecting evidences Synthesis evidence report Domestic preparation Individual Evaluating by expert Focused evaluating by Expert Panel Writing report by Chair & rapporteur International evaluation 第12页（共25页） International evaluation report

13. Designing evaluation: • Factor analysis in designing framework based on evidence

14. Domestic preparation: collecting evidences from questionnaire surveys, interviews and other activities Six sets of questionnaire surveys: delivered 77799, returned 20221

15. Domestic preparation: collecting evidences from questionnaire survey, interview and other activities Interviews: Focus group meeting & face-to-face interview 53times, involving 294 persons

16. International evaluation: From ‘Synthesis evidence report’ to ‘International evaluation report’ IEC presents the evaluation results in ‘International evaluation report’ based on ‘Synthesis evidence report by NCSTE individually. NCSTE IEC

17. Case Study: International evaluation of funding and management of NSFC and the policy making • Evidence-based policy making: After Evaluation • Case 1: Raising ‘Excellent Young Scientists Fund’ • Case 2: One year ‘Time out’ for applicants who have been unsuccessful in two consecutive years.

18. Case 1: Raising “Excellent Young Scientists Fund” based on age distribution of DYS fund • Evidence Naturally, having fostered young scientists in its talent programs NSFC is now seeing them age. The effect is especially marked in the DYS found, where increasingly those just below the age limit win the grants.

19. Case 1: Raising “Excellent Young Scientists Fund” based on age distribution of DYS fund • Suggestion from ‘International evaluation report’: We recommend that a new program should be established for younger researchers, no longer based on biological age but rather on ‘scientific age’…… and, recommend that a substantial ‘China Chair’ program (in the style of the Canada Research Chairs) be established by NSFC for junior investigators with sustained support over years……

20. Case 1: Raising “Excellent Young Scientists Fund” based on age distribution of DYS fund Type: Directly into policy Evidences source: Data statistics, Evaluation report • New policy making of NSFC Raising “Excellent Young Scientists Fund” in 2012 The ‘Excellent Young Scientists Fund’ plans to support with funding of 1 million RMB per project.

21. Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years. • The challenge of NSFC in 2009 More management pressure on NSFC staff: The growth in applications has been much faster than the growth in staff in recent years. The annual number of applications per permanent member of staff has risen from 142 in 2001 to 505 in 2009. More pressure on Reviewers comparing with NSF or DFG. More pressure on Reviewers

22. Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years. • The challenge of NSFC in 2009 Rapid increasing proposals of GP: As the main instrument of NSFC, the General Program application has been increasing rapidly since 2002. ？ the future?

23. Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years. • Alternatives of Policy 1 2 3 How to make the decision? Evidence-based evaluation Policy making

24. Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years. • Step 1: Analysis the reason of rapid increasing • NSFC grants are increasingly attractive to actual and potential beneficiaries. • Universities and other organizations are using receipt of NSFC funding as an esteem indicator for promotions and in the assessment of large projects, further increasing application pressure. Evidence: Questionnaire survey of supporting institutions • 75% of respondents use NSFC grants as an important basis for various merit-based assessments of research staff. • 58% of respondents regard NSFC grants as a condition for job promotion or assignment. • 56% of respondents allocate additional research funds to NSFC projects. • 51% of respondents pay cash bonuses or other benefits to NSFC grantees. • Only 8% of respondents have no special policies for NSFC awards or awardees.

25. Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years. • Step 2: Evaluation based on the evidence of stakeholder surveys and data analysis • Stakeholder 1: Staff of NSFC • When interviewed, the staff from the Science departments complain that “The current workload is too heavy, and the only time we can have for ourselves is the time for sleep.” • Most of the staff who attended the group meeting also complained about the overloading. One of the participants pointed out that a program director had only eight minutes to review a proposal in the given timeframe.

26. Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years. • Stakeholder 2: Institutes and universities • Most institutes and universities oppose to limit the number of applications, because more conflicts will emerge between applicants and institutions, and more and more applicants will get the opportunity by “old-friend network”. • 43.5% of 841 institution respondents considered the current success rate of application for the GP was appropriate, while 46.0% considered it moderately low or low.

27. Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years. • Stakeholder 3: PIs of General Program The results from the questionnaire survey of PIs of the GP show that, 53.8% of 10288 respondents considered the grant size of GP was appropriate, while 43.5% considered it was less than the actual research costs. Only 0.6% of the respondents thought it was more than actual research costs. The rest 2.1% of respondents’ answers to the question is “I’m not sure”.

28. Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years. • Stakeholder 4: Reviewers 45.5% of reviewer respondents thought the current success rate of the GP was appropriate, while 49.8% thought that it was moderately low or low.

29. Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years. • Stakeholder 5: Rejected Applicants 44.1% of rejected applicant respondents agree that applicants who were consecutively denied for two years are not permitted to submit proposal in the third year, but can submit proposal in the fourth year.

30. Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years. • Evaluation suggestion: At least one year time our for applicant who has been unsuccessful in two consecutive years.

31. Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years. • Step 3: Further research for policy making after evaluation • The success rates of GP after 2-4 consecutive rejected years are lower than the average rate. • But the success rates of YSF and LDR after 2-4 rejected consecutive years are higher than the average rate. • The trend will be more obvious following the increasing of consecutive rejected years. General Program(GP) 2.85% Young Scientists fund(YSF) Fund for Less Developed Region(LDR) -3.35% -1.92%

32. Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years. • Step 3: Further research for policy making after evaluation • The proportion of applicants who have been unsuccessful in two or above consecutive years maintains 70%- 80% in recent years. • The prediction said that it would decrease about 30000 proposals if NSFC applies a ‘time-out’ policy to forbid the proposals of applicants who were rejected in both 2011 and 2012. (The potential applicants proportion was proposed 80%)

33. Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years. • Step 4: Policy making & performance of the new policy • Policy making: • Enforcing one year ‘time out’ for applications who have been unsuccessful in two consecutive years since 2014. NSFC had informed the message to all potential applicants in 2013. • Adjusting the grant size, and the duration of General Program from 3 years to 4 years.

34. Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years. • result of the new policy: The number of the whole proposals is obviously decreasing in 2013 and 2014 because of the ‘time-out’ policy.

35. Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years. • result of the new policy: In contrast to the growth trend, the number of proposal for the GP began decreasing dramatically in 2013.

36. Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years. • result of the new policy: The grant size and success rate of GP began increasing obviously since the implementing of the new policy. The grant size and success rate of General Program in 2009-2013

37. Thanks！