1 / 34

WORKSHOP IN MONTREAL

WORKSHOP IN MONTREAL. 2004. MEASURING THE IMPACTS FROM PUBLIC-SECTOR SCIENCE AND TECHNOLOGY: NEW METHODS. J U N E 2 0 0 4. Elie Geisler. Professor and Associate Dean for Research Stuart Graduate School of Business Illinois Institute of Technology Chicago, Illinois

quon-hubbard
Download Presentation

WORKSHOP IN MONTREAL

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. WORKSHOP IN MONTREAL 2004

  2. MEASURING THE IMPACTS FROM PUBLIC-SECTOR SCIENCE AND TECHNOLOGY: NEW METHODS J U N E 2 0 0 4 Elie Geisler Professor and Associate Dean for Research Stuart Graduate School of Business Illinois Institute of Technology Chicago, Illinois email: geisler@stuart.iit.edu June 17, 2004 Prepared for Presentation at the “Workshop on Measuring the Impacts of Science,” Montreal, Canada, June 16-18, 2004

  3. TERMINOLOGY W O R K S H O P • Public-Sector Science & Technology (S&T) • Impacts on Industry • Impacts on Society • Commercialization of S&T • Technology Transfer from Federal Laboratories IN M O N T R E A L

  4. METHODOLOGIES W O R K S H O P STATISTICAL, CO-VARIATION DESIGN • Comparing databases distinct in time and phenomenon (ontology): temporal & conceptual gaps • Ex post factor Explanation as Substitute for Strong Theory IN M O N T R E A L OPENING THE BOX • Process Approach • Linkages explored between stages • Monitoring transformations of constructs

  5. WHAT DO WE STUDY? W O R K S H O P • PROCESSES: How public sector laboratories conduct their activities? Issues of efficiency & effectiveness. • OUTCOMES IN M O N T R E A L • How successful are public-sector laboratories • Success: Defined as commercialization and technology transfer • Success: Defined as contributions to parent agency • IMPACTS on industry & society

  6. A MODEL OF OUTCOMES AND IMPACTS W O R K S H O P • What are the stages of transformation of public R&D into technology transferred and commercialized? • How can we measure such a process: What are the metrics? IN M O N T R E A L

  7. A GENERALIZED MODEL OF METRIC SELECTION Cultural variables (sector, industry, company, or Government agency) W O R K S H O P Available pool of metrics Type of activity to be measured Management policies, strategy, and biases Development of Selection Criteria & Selection Process IN M O N T R E A L Customers’ choices and preferences Prior experience with evaluator and metrics How managers perceive the link between S&T and the strategic objectives of the organization Perceptions of the value and effectiveness of metric Interests and preferences of other stakeholders Impacts on behavior of managers regarding S&T Impacts on the organization (effects on the strategy and culture) Feedback

  8. KEY ISSUES IN METHODOLOGYAND ANALYSIS OF SCIENCE INDICATORS W O R K S H O P IN M O N T R E A L

  9. ISSUES IN THE USE OF INDICATORS TO MEASURE COMPLEXPHENOMENA AND CONCEPTUAL CONSTRUCTS W O R K S H O P IN M O N T R E A L

  10. The Process-Outcomes Model of the Linkages Between the R&D Process and Social and Economic Systems 2 1 Immediate or direct outputs from R&D/ science (potential inputs to social subsystems), e.g., Publications Patents Ideas Theories Discoveries Methods Inputs to R&D (in particular R&D sector or the total national S&T system), e.g. People & skills Funding Guidance & gain Other resources and restraints A The R&D process, e.g. Energy Health care Industry Education Other factors in the specific situation (e.g., the R&D sector or the social subsystem) which affect the transition, transportation, adoption, usefulness, cost or economic benefit from transfers between adjacent and more distant stages. Such factors may include economic, cultural, organizational, technical, personal, or political ones. Some are particular to a given stage (e.g., the barriers and difficulties involved in designing economical and socially acceptable energy or safety devices and systems or the diffusion problems in curing a disease); others may apply to several stages in the overall process (e.g., capital shortages or regulations); and still others are pervasive across the whole process (e.g., organizational barriers to innovation, individual risk preferences, diffuse decision-making responsibility).

  11. 3 C Intermediate outputs of R&D factual inputs to social sub- systems, e.g., New and improved products or processes Methods of organizing, managing, or evaluating B Transformation, and diffusion processes in the economic & social sub- systems, e.g., Health care delivery Transportation services Energy system Consumer or industrial products & services Transformation and diffusion processes, e.g., Marketing Adoption Design Implementation Other factors in the specific situation (e.g., the R&D sector or the social subsystem) which affect the transition, transportation, adoption, usefulness, cost or economic benefit from transfers between adjacent and more distant stages. Such factors may include economic, cultural, organizational, technical, personal, or political ones. Some are particular to a given stage (e.g., the barriers and difficulties involved in designing economical and socially acceptable energy or safety devices and systems or the diffusion problems in curing a disease); others may apply to several stages in the overall process (e.g., capital shortages or regulations); and still others are pervasive across the whole process (e.g., organizational barriers to innovation, individual risk preferences, diffuse decision-making responsibility).

  12. 4 5 Pre-ultimate R&D (outputs of social/ economic subsystems), e.g., Mortality and morbidity Extinction of particular causes of death Improved safety of products and work environments Productivity rates in individual firms or sectors Ultimate R&D outputs (quality of life and health of the economy and society), e.g., International balance of trade Energy independence Gross national product Components of quality of life D Transformation and diffusion processes in the society and the economy Other factors in the specific situation (e.g., the R&D sector or the social subsystem) which affect the transition, transportation, adoption, usefulness, cost or economic benefit from transfers between adjacent and more distant stages. Such factors may include economic, cultural, organizational, technical, personal, or political ones. Some are particular to a given stage (e.g., the barriers and difficulties involved in designing economical and socially acceptable energy or safety devices and systems or the diffusion problems in curing a disease); others may apply to several stages in the overall process (e.g., capital shortages or regulations); and still others are pervasive across the whole process (e.g., organizational barriers to innovation, individual risk preferences, diffuse decision-making responsibility).

  13. METRICS W O R K S H O P • CORE indicators of outputs/outcomes from public-sector laboratories. • ORGANIZATION-SPECIFIC indicators of outputs/outcomes from public-sector laboratories IN M O N T R E A L

  14. ILLUSTRATIVE CORE INDICATORS AND MEASURES OF THE OUTPUTS FROM R&D • IMMEDIATE OUTPUTS • 1. Written Scientific and Technical Outputs • 1.1 Number of publications in refereed journals. • 1.2 Number of technical reports. • 1.3 Number of patents. • 1.4 Number of citations in refereed journals. • 1.5 Number of patent disclosures. • Other Outputs • 2.1 Number of licenses signed for own patents. • 2.2 Number of new products conceived • 2.3 Number of key improvements suggested. • 2.4 Number of new and improved test methods, models, standards, concepts, and databases. • 2.5 Number of new ideas transferred downstream. • 2.6 Number of problems solved for users/clients. • 3. Overall Reputation • 3.1 Number of complaints by clients/users. • 3.2 Judgment of quality of R&D by clients/users. • 3.3 Number of awards received. • 3.4 Milestones/objectives met. W O R K S H O P IN M O N T R E A L

  15. II. INTERMEDIATE OUTPUTS • 1. Scientific/Technical Impacts on Direct Users of R&D • 1.1 Number of improved or new products produced. • 1.2 Number of improved or new processes applied. • 1.3 Number of improved or new materials made. • 2. Economic/Financial Impacts on Direct Users of R&D • 2.1 Income from licensing patents and inventions. • 2.2 Cost reductions/savings from new and improved • products/processes • 2.3 Improvements in productivity of people and • materials/equipment. • 2.4 Costs associated with implementation and adoption of • new products/processes (e.g., training, regulatory adjustments). • 3. Responsiveness of R&D • 3.1 Judgment by direct clients/users (their satisfaction; • benefits from R&D). • 3.2. Judgment by other organizations (R&D and non-R&D). W O R K S H O P IN M O N T R E A L

  16. III. PRE-ULTIMATE OUTPUTS W O R K S H O P • 1. Economic Benefits and Costs 1.1 Improvements in productivity levels in people and equipment by sector and industry. 1.2 Savings, cost reductions, and income generated by improved health, productivity, safety, and mobility of the workforce at sectoral and national levels. 1.3 Costs to the economy and to society from the absorption of R&D into social/economic subsystems (such as transportation, energy, information, telecommunications, and health care). • 2. Improvements and Problems in Social Conditions 2.1 Improvements in overall health of population. 2.2 Improved life expectancy of population. 2.3 Improved satisfaction and levels of optimism of the population. 2.4 Problems in social conditions (associated with the pre- ultimate outputs and their impacts on society and the economy) such as increased alienation, decrease in feelings of security and job loyalty. 2.5 Changes (positive and negative) in the nature of work. IN M O N T R E A L

  17. IV. ULTIMATE OUTPUTS W O R K S H O P 1. Economic Benefits and Costs 1.1 Improved gross national product. 1.2 Improvements in the balance of trade and balance of payments. 1.3 Improved GDP/capita. 1.4 Costs of living in a highly technological economy. 2. Social Benefits and Costs 2.1 Improved level of overall satisfaction and happiness of population. 2.2 Expanded “middle” or “professional” class. 2.3 Costs of living in a technological society. IN M O N T R E A L Source for some of the indicators and measures is from Table 2 and Appendix A in Geisler, “An Integrated Cost-Performance Model of Research and Development Evaluation,” Omega, 23(3), 1995, 281-294.

  18. ILLUSTRATIVE ORGANIZATION-SPECIFIC INDICATORS AND MEASURES OF THE OUTPUTS FROM S&T W O R K S H O P • 1. Level of Technical Expertise • 1.1 Ratio of doctorate holders to scientific workforce. • 1.2 Relative experience of scientists and engineers (total years of technical work). • Attractiveness of the R&D Organization • 2.1 Number of candidates applying for scientific position, per position. 2.2 Age profile of scientists and engineers. • 2.3 Judgment of quality of the organization by peers. IN M O N T R E A L

  19. II. INTERMEDIATE OUTPUTS W O R K S H O P • Level of Investment in Exploitation of R&D • 1.1 Funds allocated to technology commercialization by the focal organization (e.g., company, government agency). • 1.2 Number of personnel from non-R&D units working with R&D units (per scientists & engineers). • Level of Importance of R&D Outcomes • 2.1 Role of new products/services in the organization’s success and survival (as perceived by senior managers of the organization). • 2.2 Perceived success of the transfer process and implementation/absorption of R&D outputs in the organization (perceived by senior managers of the organization). • Climate and Leadership • 3.1 Degree of support that senior management gives to R&D generation, adoption, and transfer. • 3.2 Overall “climate” in the organization: favorable or unfavorable to R&D (perceived by workers and managers). IN M O N T R E A L

  20. III. PRE-ULTIMATE OUTPUTS W O R K S H O P • 1. Investments in Adoption of R&D • 1.1 Funds allocated to the total adoption, adaptation, and utilization of R&D intermediate outputs by the focal organizations (as ratio of total budgets, and over time). • 1.2 Ratio of products, services, and processes impacted by R&D outputs that are adopted, implemented, and utilized by the organization. • 2. Structure of the Industry • 2.1 Structural variables such as size, centralization, and vertical integration. • 2.2 Traditional perception of S&T in the industry: How important is R&D to the progress and survival of the industry? • 3. Strategy and Life Cycle • 3.1 Role of R&D in strategic management of the industry and sector. • 3.2 Impact of R&D on the industry in its stage of life cycle. IN M O N T R E A L

  21. IV. ULTIMATE OUTPUTS W O R K S H O P • Role and Importance of R&D • 1.1 Role of R&D in the economy, as perceived by the population. • 1.2 Role of R&D in social progress, as perceived by the population. • 1.3 Importance of R&D as a political issue. • S&T Level of Population • 2.1 Levels of R&D literacy in population. • 2.2 Acceptance of the importance of investments in R&D by the population. IN M O N T R E A L Source for some of the indicators and measures if from Table 2 and Appendix A in Geisler, “An Integrated Cost-Performance Model of Research and Development Evaluation,” Omega, 23(3), 1995, 281-294.

  22. COMPUTATION OF LOADING OUTPUTS INDICATORS W O R K S H O P s = stage pf [process/outputs )1=4_ n = number pf ;leading output indicators i = each indicator w = weight of ith indicator IV = index value of indicator, so that IN M O N T R E A L wia = weight of the ath measure of indicator I dia = each component indicator n(i) = number of measures of ith indicator ia = value of ath measure of indicator i

  23. KEYOUTPUT INDICATOR W O R K S H O P = normalized weights for each LOI IN M O N T R E A L (This is the integration of all four stages, from immediate to ultimate outputs.)

  24. MEASURING IMPACTS BY STAGE W O R K S H O P • (alpha) = KOI for LOI1 • (immediate outputs) • These are outcomes from the public laboratory. • (beta) = KOI for LOI1-2 • (immediate + intermediate outputs) • These are outcomes from the public laboratory & inputs to parent agency, industry, and society. IN M O N T R E A L

  25. W O R K S H O P (gamma) = KOI for LOT1-3 (immediate, intermediate, and preultimate outputs) These are outcomes from public laboratories and impacts on parent agency, industry, and society. IN M O N T R E A L (omega) = KOI for LOT1-4 (immediate ultimate outputs) These are overall impacts on society’s goals and objectives.

  26. ADVANTAGES OF PROCESS-STAGES METHOD W O R K S H O P • - format allows for choice of “upstream” or ‘downstream emphasis in the measurement of outputs. • This format allows for comparison of outputs and impacts between “middle” and “downstream” for the same public laboratory. • This format allows the evaluator (including management at laboratory or parent agency, or constituents) to assign different weights to each stage and to the “upstream” or “downstream” portion of the process, thus tailoring the evaluation to strategic objectives. IN M O N T R E A L

  27. Illustrative Development of Indexes of Leading Output Indicators for Immediate and Intermediate Outputs for Two Public Laboratories (1) Immediate Outputs Laboratory A Core Indicators

  28. Laboratory A Organization-Specific Indicators From the formula in equation (3): IV1 = 16.75 (Benchmark = 17.00) IV2 = 8.40 (Benchmark = 5.00) IV3 = 8.00 (Benchmark = 17.50) IV4 = 0.30 (Benchmark = 0.90) Weights assigned to IV1: IV1 = 0.4 IV2 = 0.2 IV3 = 0.2 IV4 = 0.2 Applying equation (4): LOI(A)1 = 10.04 (Benchmark = 11.48)

  29. Laboratory B Core Indicators

  30. Laboratory B Organization-Specific Indicators From the formula in equation (3): IV1 = 23.75 (Benchmark = 23.75) IV2 = 18.00 (Benchmark = 18.00) IV3 = 27.50 (Benchmark = 27.50) IV4 = 2.40 (Benchmark = 2.40) Weights assigned to IV1: IV1 = 0.6 IV2 = 0.1 IV3 = 0.2 IV4 = 0.1 Applying equation (4): LOI(B)1 = 28.02 (Benchmark = 21.75)

  31. (2) Intermediate Outputs Laboratory A By applying equation (3): IV1 = 2.4 (Benchmark = 2.0) IV2 = 11.10 (Benchmark = NA) Weights assigned to IV1: IV1 = 0.3 IV2 = 0.7 LOI(A)2 = 8.42 (Benchmark = N/A)

  32. Laboratory B By applying equation (3): IV1 = 0 IV2 = 50 (weight assigned = 0.8) (weight assigned = 0.2) By applying equation (4): LOI(B)2 = NA

  33. (3) KOI for Laboratory A Weights assigned: LOI(A)1 (immediate outputs) = 0.3 LOI(A)2 (intermediate outputs) = 0.7 From equation (5): KOI(A) = (10.04)  0.3 + (8.42  0.7) = 8.90 (4) KOI for Laboratory B Weights assigned: LOI(B)1 (immediate outputs) = 0.8 LOI(B)2 (intermediate outputs) = 0.2 From equation (5): KOI(B) = (28.02)  0.8 + (10.0)  0.2 = 24.41

  34. WORKSHOP IN MONTREAL 2004

More Related