Soon Cheon Byeon Byung Yong Hwang KISTEP

1. AEA 2009 Orlando, Florida Effectsof GRIs Evaluation System on the Evaluation Result Acceptability and Performance : Focusing on Economic and Human Society Research Council in Korea Soon Cheon Byeon Byung Yong Hwang KISTEP (Korea Institute of Science and Technology Evaluation and Planning) Seoul, South Korea

2. Contents I. Introduction II. Backgrounds 2.1. Performance Evaluation of GRIs 2.2. Characteristics of GRIs and Acceptability III. Research Model and Hypotheses 3.1. Research Model 3.2. General Model 3.3. Variables and Hypotheses IV. Methodology 4.1. Targets for Questionnaire 4.2. Sampling 4.3. Analysis Method V. Results 5.1. Analysis on Validity and Reliability 5.2. Results of Hypothesis Verification VI. Conclusions and Suggestions

3. I. Introduction 1.1. Current Status of Research Councils and GRIs • On 2005, the Economic Society Research Council and the Human Society Research Council were integrated into the Economic and Human Society Research Council(23 GRIs) • On 2008, Basic/Industrial/Public Technology Research Councils were reorganized as KRCF(Korea Research Council of Fundamental S&T, 13 GRIs) and ISTK(Korea Research Council for Industrial S&T, 13 GRIs) • 23 GRIs in EHS Research Council are classified as follows according to their research areas

4. I. Introduction 1.2. Purpose of Research Council and GRIs • Governing Principles of the Research Council System • Competition in the research and development • securing of independency and autonomy of GRIs and their researches • avoiding of redundant and overlapped investment • Purpose of the 23 GRIs under Economic and Human Society Research Council • to satisfy government's demands on policies which address social issues arising from Korea's fast economic development and science and technology advancement during 1970s and 80s, diversified international competition, and social conflicts and confrontation.  • Purpose of the establishment of the EHS Research Council • to support national R&D policy • to contribute toward the development of knowledge-based industry • by support and promotion of research institutes in the area of Economic and Human Society • and by systematic management of GRIs

5. I. Introduction 1.3. Functions and Roles of EHS Research Council • Functions and Roles of the EHS Research Council

6. I. Introduction 1.4. Achievements and Problems of EHS Council • Major Achievements in … • Establishment of autonomous and accountability management of GRIs • Facilitation of Industry-Academy-Research Institute joint researches • Enhancement of key research capabilities of GRIs • Increased reasonable use of budget through avoiding redundant and overlapped projects • Still several Problems in … • Establishment of identity of Research Council (too small joint research fund) • Coordination of GRIs functions through evaluation (low implementation of evaluation result) • Acquisition and distribution of research resources for GRIs (limited activity in acquisition process) • Development strategies of GRIs (including low activity in the maintenance of high quality HR)

7. I. Introduction 1.5. Necessity of Research National concerns on R&D performance Improvement of the functions and roles of the Research Council & GRIs Evaluation system Focusing on improving Performance of GRIs ▪ Recent changes in the environment surrounding the Research Council and GRIs ▪ Reorganization of National Science and Technology Council (NSTC) by the new government and Selection of the policy to keep the existing Research Council system ▪ Criticism and Demand for improvements on the roles and management systems of the Research Council ▪ Improvement of GRIs performance under the Research Council ▪ Improvement of Pubic trust in government through reporting performances

8. I. Introduction 1.6. Orienting Questions • Questions on the Functions and Roles of the Research Council • Have the functions and roles of the Economic and Human Research Council been well established and institutionalized? • If any, what are the problems of the Research Council system? • What can be done to stabilize the system and enhance the performance of GRIs? • Survey result on the satisfaction on the EHS Research Council

9. I. Introduction 1.7. Orienting Questions • Questions on the GRIs Evaluation System of the Research Council • Dissatisfaction with GRIs evaluation system within the functions and roles of the Research Council • Has the evaluation system been fulfilled well to maximize research performances? • GRIs’ dissatisfaction than acceptance • The satisfactions point on GRIs evaluation: 49.4 points • What area should be improved to motivate GRIs and maximize performances?

10. I. Introduction 1.8. Objectives of Research • Measuring the relationship among GRIs evaluation system, Evaluation result acceptability and GRIs performances • Analysis on the status of GRIs evaluation system and the recognition & acceptance attitude on GRIs evaluation system by institutes with high performances • Suggestion of enhancement directions of GRIs evaluation system to improve the performances of GRIs • Approach to problems of the Research Council and GRIs through integrated view

11. II. Research Backgrounds 2.1. Performance Evaluation of GRIs • Performance Evaluation by Government • a process which defines its mission and desirable outcomes, establishes standard on performance, connects performance to budget, reports results, and makes managing level officials take the accountability for the results. (Andris Vinelli, 2000) • - Act on R&D Performance Evaluation and Management (Dec. 2005)   - National Evaluation System (NES) • Researches on the Performance Evaluation on GRIs in Korea • studies focusing on the system design for GRIs evaluation • studies suggesting the balanced scorecard (BSC) approach for GRIs performance evaluation • a possibility of adoption the intellectual capital model to performance evaluation • needs to encourage GRIs to raise their performance through stabilizing the functions and roles of Research Councils • Not enough study on the factors consisting of functions and roles of the Research Council which is the basis of such R&D activities • Insufficient efforts to analyze on how the functions and roles of the Research Council affects the performance of GRIs

12. II. Research Backgrounds 2.1. Performance Evaluation of GRIs • For non-profit organizations and public organizations, performance related to their missions are regarded more important than financial performance. (Brinckerhoff, Peter C., 1994) • Recently, when it comes to GRIs' performance, more interests are placed in the operation of an organization as well as tangible results. • Models identifying performance of an organization focusing on the effectiveness of the organization of a research institute • These Models relate innovation, creativeness and excellence of science technology to structure, culture, characteristics, activities and processes of a research organization. • These Models made large contributions in that they identified factors affecting performances of research institutes and suggested such performances can be analyzed through the analysis on those factors. (Jordan and Streit, 2000) • However, the Models failed to include comprehensive performance related to mission and goals of the research organization, relationship between the organization and outside entities, and learning aspect. (Jang Jae Lee and others, 2003)

13. II. Research Backgrounds 2.1. Performance Evaluation of GRIs • Models adopting BSC(balanced scorecard) to performance evaluation of the research institute • Min Hyeong Lee (2001, 2005) was the first to employ BSC approach into performance evaluation of GRIs. • Jang Jae Lee and others (2003) utilized general framework suggested by Kaplan & Norton (1996) to analyze performance based on four aspects including finance, customer, internal process and innovation and learning. • cases of May Institute, a non-profit research institute in Massachusetts, the U.S. Department of Transportation, Sandia National Lab, and museums • In this study, considering the characteristics and missions of GRIs, the GRIs’ performances are classified as follows based on the BSC, which includes comprehensive performances, relations and learning aspect.

14. II. Research Backgrounds 2.1. Performance Evaluation of GRIs • BSC(balanced scorecard)-based Performance Evaluation for GRIs

15. II. Research Backgrounds 2.2. Characteristics of GRIs and Acceptability • Characteristics of GRIs should be considered to determine the effect of the functions and roles of Council on GRIs performances. • Research area, budget size, established year • The evaluation result acceptability could be considered to determine the effect of evaluation system on the GRIs performances by focusing on variables related to the responding attitude to the evaluation result. • Will to reflect evaluation recommendations, will to reflect negative evaluation result

16. III. Model and Hypothesis 3.1. Research Model • Previous studies related to the operation of Research Council insist that the functions and roles of the Council have significant impact on the performance of GRIs. • But no discussion on which factors of the Council's functions and roles have affected what kind of GRIs' performance • This study will examine the factors which comprise functions and roles of the Research Council first. • The effect of acceptability of evaluation result on the GRIs performances will be investigated. • This study will explore desirable future directions in the functions and roles of the Research Council consistent with the objectives of GRIs, by investigating the specific effect of the functions and roles of the Research Council on the performances of GRIs.

17. III. Model and Hypothesis 3.2. General Model Characteristics of GRIs Research areas Functions and roles of Research Council Size of GRIs GRIs performance Established Year Research planning by areas Financial performance H2 Planning GRIs’ Development Direction H1 Customer performance Coordinating and adjusting GRIs’ functions Internal process performance Evaluation Result Acceptability H4 H3 Learning & growth performance Evaluation on GRIs H5 Promoting joint research

18. III. Model and Hypothesis 3.3.1. Variables and Hypothesis(1) • The objective of this study is to discuss the detail effects of the functions and roles of the Research Council on the specific performance of GRIs. • control factors are added to identify the effects of GRIs' characteristics including research areas, size and established year of GRIs. GRIs’ Characteristics • Research Areas • Size • Established Year control factors GRIs’ Performances • Financial Performance • Customer Performance • Internal Process Performance • Learning & Growth Performance Independent variables Dependent variables Functions and Roles of EHS Research Council • Research Planning • Planning GRIs’ Development Direction • Coordination and Adjusting GRIs’ Functions • Evaluation on GRIs • Promoting Joint Researches

19. III. Model and Hypothesis 3.3.1. Variables and Hypothesis(1) • Hypothesis 1. The functions and roles of Research Council will have significant impact on the performance of GRIs. • Hypothesis 2. The performance of GRIs will be significantly different according to the characteristics of GRIs and the functions and roles of the Research Council. • Hypothesis 2-1. The performance of GRIs will be significantly different according to the research area of GRIs and the functions and roles of Research Council. • Hypothesis 2-2. The performance of GRIs will be significantly different according to the size of GRIs and the functions and roles of Research Council. • Hypothesis 2-3. The performance of GRIs will be significantly different according to the established year of GRIs and the functions and roles of Research Council.

20. III. Model and Hypothesis 3.3.2. Variables and Hypothesis(2) • The other objective of this study is to discuss the detail effects of the evaluation system on the specific performance of GRIs. • Mediating variables are added to identify the effects of evaluation system on GRIs performances, such as ‘will to reflect recommendation’ and ‘will to reflect negative evaluation results’. Mediating variables Dependent variables Independent variables • GRIs performance GRIs evaluation system Evaluation Result Acceptability H3 H4 H5

21. III. Model and Hypothesis 3.3.2. Variables and Hypothesis(2) • Hypothesis 3. The quality of GRIs evaluation system will have significant effect on the acceptability of GRIs evaluation results. • Hypothesis 4. The acceptability of evaluationresults will have significant effect on the GRIs performances. • Hypothesis 5. The quality of GRIs evaluation system will have significant effect on the GRIs performances.

22. IV. Methodology 4.1. Methodology and Targets for Questionnaire • Empirical methodology • A research model was established by collection and analysis on the related laws and literatures, and then the hypotheses were verified by an empirical methodology which verifies hypotheses through statistical analysis based on a survey result.  • Process  • The survey was carried out on researchers in 23 GRIs under the Economic and Human Society Research Council. • The samples consist of 551 persons including vice president, administrative (planning) leader, research coordination (planning and budget) team leader, and president of labor unions. • Via e-mail • Analyzed by using SPSS for Windows 15.0

23. IV. Methodology 4.2. Sampling • To survey for the empirical methodology, a qualitative research was performed on a group including staff, union members, researcher, administrative employee who are directly related to the functions and roles of the Research Council and the performance of GRIs. • the sample group was smaller than that in general studies, but had higher level of understanding for all questions. • Although the sample was small, it can be considered as appropriate and reasonable given that it was the qualitative research.

24. IV. Methodology • Composition of samples

25. IV. Methodology • Composition of samples

26. IV. Methodology 4.3. Analysis method

27. V. Results 5.1. Analysis on validity and reliability • It is generally accepted that a measured value is considered as reliable when the Cronbach’s α value exceeds 0.6. • All the α values satisfy such requirement according to the following analyses. • enabling the analysis on all the factors in a concerned item with a single measure without undermining reliability.

28. V. Results 5.2. Results of Hypothesis Verification(1)

29. V. Results 5.3. Analysis Results of Structural Equation Model(2) * All path coefficient values have significance at the level of p<.001

30. V. Results 5.4. Cause-effect analysis among factors(2)

31. V. Results 5.5. Analysis Results • The GRIs Evaluation System has significant and positive influence on the acceptability of evaluation results and GRIs’ performances. Furthermore, the acceptability of evaluation results has relations with GRIs’ performances • Direct effect: 0.592, Indirect effect: 0.610 • All the factors consisting of the GRIs Evaluation System serves as important deciding factors • ‘Sufficient interactions’, ‘Fairness of evaluation criteria’, and ‘Systematic evaluation items and indicators’ receive more weights • ‘Confidence of participants’ and ‘Appropriate incentives’ have relatively less weights • This might be interpreted as the importance of developing the evaluation criteria • and indicators which would most reflect the characteristics of GRIs in the operation • of GRIs Evaluation System, through effective interactions

32. V. Results 5.5. Analysis Results • The factors for acceptability of evaluation results are also crucial determining factors • All the factors consisting of GRIs’ performances serve as important determining factors • ‘Internal process performances’ takes up more importance than ‘Customer performances • It means that the evaluation results greatly influence on the improvement • of GRIs’ management process, while they have less influence on the research • performance or customer satisfaction

33. V. Conclusions and Suggestions • First, this study conducts empirical analysis on the effects of functions and roles of the Economic and Human Society Research Council on the performance of GRIs. • As result of survey on 551 employees from 23 GRIs, functions and roles of the Economic and Human Society Research Council have significant impacts on the performance of GRIs. • In particular, 'research planning by each research area' has more impacts on the GRIs' performance, while functions and roles of the Council have the most impacts on GRIs' financial performance. • Furthermore, 'research area‘ and 'size‘ have control effects on customer performance, internal process performance and learning & growth performance, and internal process performance respectively. • In conclusion, the Research Council needs to play different functions and roles considering research area, size and work type of GRIs. 

34. V. Conclusions and Suggestions • Second, the GRIs evaluation system has significant influence on GRIs performances, and the acceptability of evaluation results have a relation with GRIs performance. • Overall weakness of the GRIs Evaluation System results in the low acceptability, and also brings limitation to the active use of evaluation results. • In conclusion, the Research Council needs to improve the evaluation system and develop methods to enhance the acceptability of the evaluation results. 

35. V. Conclusions and Suggestions • Based on the above results, the following development measures and improvements are suggested for the Research Council. • First, the Economic and Human Society Research Council should endeavor to establish and develop its identity. It should expand its functions to include 'planning on government R&Ds' through regular monitoring of current R&Ds, as well as planning and establishment of new government R&Ds. • Second, the Council's function of obtaining and distributing research resources for GRIs should be expanded. Considering that the Council's functions and roles have the most impacts on 'financial performance' of GRIs, the most desirable role for the Council should be 'assisting GRIs with obtaining research budget and excellent human resources. • Third, the Council should strengthen its role as function coordinator among GRIs. Therefore planning ability and expansion of expertness of the Research Council are required.   • The participation and discussion process of each actor is crucial to improve • the functions and roles of the Research Council and to establish development plans

36. V. Conclusions and Suggestions • Also, GRIs evaluation system should be focused on the motivation of GRIs and maximization its performances. • First, the objectives of evaluation (and results use) should be specific and reasonable so that they serve as specific guidelines for the GRIs evaluation participants. • Second, Evaluation indicators should be employed and the weights be placed by most reflecting on the characteristics of GRIs. Furthermore, the indicators should be agreed among the government, Research Council, and GRIs. • The achievement should be evaluated by absolute evaluation method based on their own goals. And the efficacy should be enhanced through the connection to internal performance management. • Third, the government and the Research Council as evaluators need to strengthen the collection and accumulation of relevant information and knowledge including current states and issues of GRIs. They also make further endeavors to form an evaluation committee in consideration of expertise, integrity, and accountability, and to provide the committee with prior knowledge about GRIs. Moreover, face to-face contacts among evaluation participants should be continuously expanded to consolidate interactions. The volume of incentive and its distribution should also be improved.   • Fourth, Enhancement of acceptability: The establishment of monitoring system

37. V. Conclusions and Suggestions • Research limitations • Limitation of generalization. • Measuring GRIs performance by subjective satisfaction of individual members of GRIs rather than based on quantitative data. • Future research • Suggest the indicators of GRIs performances reflecting the characteristics of public research institutes in EHS sciences. • Faithfully measure the concepts of the acceptability and effectiveness of the GRIs evaluation system.

38. Thank you for your attention For more questions : sbyeon@kistep.re.kr byhwang@kistep.re.kr