H. Lee*, Y.-J. Kim, H.-K. Lee
[R&D Budget Strategy Division
- Objective: Estimating Government R&D Program Efficiencies as a part of our assistants to the Government R&D Budget Compilation Process
- Programs are units, comprising the Government R&D budget
- Korean Government adopted Digital Budget & Accounting System, Top-down Budgeting, Performance Based Budgeting several years ago.
- Measuring Program Efficiency assists the budgeting process .
- - Efficiency is used in Program Performance Evaluation
- - Yr n-1 Performance of a Program affects its Budget Size in Year n+1 in the Top-down budget process.
- Our Challenges prior to Measuring Program Efficiency (PE)
- 1/3 of R&D Programs get In-depth Performance Evaluation per annum
- Strategy : Tracking project performances and aggregating these statistics by program
- Program budgeting for Yr n+1 with performance information from Yr n-1 while programs end, merge, regroup, and are reorganized
- Strategy: Human Labor to match projects of Yr n-1 to appropriate programs in Yr n+1
- Collect Project Statistics and Obtain Proper Input & Output Measures by Program, and Assign Appropriate Categories for each Program
- Strategy: Use Experts to Validate these statistics
- Computing Efficiencies and Finding Efficient Programs/Program Types
- Used Input Oriented BCC Model (Banker et al, 1984)
- Tobit Regression Analysis to find Factors(Program Types) of efficiency
bipolar: many low PE while many PE close to 1./Note we did not use bimodal
- Performance based Budgeting : Allocation of funds to achieve programmatic goals and objectives as well as some indication or measurement of work, efficiency, and/or effectiveness
- Links between performance information & budget allocation
- Measures: Inputs, Outputs, Effectiveness, Efficiency, Workloads
- Efficiency = Output/Input
Results from Tobit Regression Analysis is that Programs comprised of
bottom-up projects are more efficient than those of top-down projects
individual projects are more efficient than those of group projects
& Preplanning shows more efficient than Programs without preplanning.
- Statistical Challenges :
- Data Matching & Classification & Statistical Inferences
- Challenge 1: matching projects in Yr 2008-2010 to a program with budget request in Yr 2012
- Challenge 2: obtaining coherent statistics from this matching
- Challenge 3: classifying programs into various categories based on project information and program plans
- Strategy: Experts reduce errors in measuring inputs/outputs for more accurate efficiency measures
- Challenge 4: measuring efficiency with multiple inputs and outputs
- Strategy: Data Envelop Analysis (DEA) has been employed (Farrell, 1957; Banker et al, 1984)
- Challenge 5: incorporating statistical and systematical uncertainties
- - survey data have various uncertainties
- - inference on groups of efficiency measures based on different types of outputs, results in different error structures/distributions
- (basic research does not produce patents and royalties).
- On going workto tackle this statistical challenge
Use of Efficiency Measure and Tobit Regression Analysis in Budget Process
[We can suggest]
[…] officers in charge of R&D programs efficient program planning and budgeting strategies
[…] officers in NSTC and MoSF increase/reduce the budget size of relatively efficient/inefficient R&D programs in the budget allocation and compilation process (A Program with good performance gets budget increment whereas one with poor performance gets deduction)
[…] program planning schemes and adjust the program budget size accordingly as a part of policy analysis.
[…] the midterm R&D budget projection as a part of settling new budget systems
- Brief R&D Budget Process for FY 2012
- [Ministries submit] proposals on their priorities in R&D (Oct. 31, 2010)
- […] midterm R&D program plans with estimated budget (Jan. 31, 2012)
- Ministry gets R&D budget ceiling by the midterm plans (Apr. 30, 2012)
- […] detail budget requests by program (Jun. 30, 2012).
- Budget (Re)Allocation by NSTC (Jul. 31, 2012)
- NSTC finalizes the R&D budget deliberation (Sept. 15, 2012)
- MoSF submits the budget deliberation to the National Assembly (Oct. 2, 2012)
- We (KISTEP) make contributions to these yellow steps
1.1 tr. won ~ $1 mil
- Shortcomings of DEA from the statistical inference perspective :
- Statistical Uncertainties are not take into account.
- Curse of Dimensionality: the more input/output measures, the more DMUs have efficiency 1.
- Efficiency measures are results of a Nonparametric Method, valued between 0 and 1, difficult to link regression analysis methods, assuming some distributional properties (In quest of adopting nonparametric regression analysis schemes while using the efficiency measures as response).
- On Data Envelop Analysis(DEA) to measure Program Efficiency:
- Program Input: Expenditure in 2008-10
- - only coherent input measure across R&D programs
- Program Outputs by the Program Type
- - Basic Research(BR): No. of well received Papers (R2nIF ≥ 1.0) [=Papers]
- - Fundamental Research(FR): No. of Registered Patents [=Patents] & Papers
- - Applied and Development Research(ADR) : Patents, Tech Transfer Fees, Royalties, and Sales Increases
- - SME R&D aids(SME): Same as ADR outputs
- These Programs are Decision Making Units (DMUs) for DEA.
- KISTEP currently survey and analyze all Gov. funded R&D projects including their performances during 2011.
- Projects are funded through appropriate Programs during each fiscal yr.
- Scientific Papers and Patents are the examples of project outputs
- By aggregating these outputs and inputs, project performance statistics are analyzed at KISTEP
- These statistics are not fully used for program budgeting
- References: written in Korean are omitted
- Banker, Charnes & Cooper (1984) Management Science vol.30(9) pp. 1078 –
- Farrell (1957) with discussions. J. of Royal Statistical Society A vol. 120 pp.11-
- Shah & Shen (2007) “A Primer on Performance Budgeting,” World Bank
This work was supported by the Korean Ministry of Strategy and Finance(MoSF)