Quantitative Models for Guiding Complex S&T Investment Strategies

1. International Workshop on the Evaluation of Publicly Funded Research Quantitative Models for Guiding Complex S&T Investment Strategies Robert E. Roberts Pamela Ebert Flattau Bhavya Lal 26 September 2005 Berlin S T P I

2. Complex S&T Investments Strategies • The US government routinely engages in the assessment of its S&T programs as part of the federal budget process • However, recent emphasis on inter-agency research “priority areas” has revealed shortcomings in federal assessment practices • These shortcomings are chiefly related to current data collection practices. • Improvements in data collection strategies are needed even as appropriate methods are being developed to guide complex S&T investment decisions

3. Presentation Overview • “Relevance, Quality and Performance” • “Hot Topics” and the S&T Investment Process • Emerging Directions • Harmonization of administrative recordkeeping • Monitoring changes in the S&T infrastructure • Assessing Federal S&T investments on innovation and economic development

4. Complex S&T Investments Strategies “Relevance, Quality and Performance”

5. “The budget system of the United States Government (USG) provides the means for the President and Congress to decide how much money to spend, what to spend it on, and how to raise the money they have decided to spend.” “Analytic Perspectives, Budget of the US Government FY 2006” www.omb.gov The USG budget process has 3 key stages: Formulation of the President’s proposed budget, Congressional action on the budget, and Budget implementation by the agencies The US Federal S&T Budget

6. Departments and Agencies The Office of Science and Technology Policy (OSTP) with the Office of Management and Budget (OMB) The National Science and Technology Council (NSTC) The President’s Council of Advisors for Science and Technology (PCAST) Office of the President: OMB OSTP NSTC PCAST US Federal Departments and Agencies Key Actors in the Formulation of the President’s S&T Budget

7. Congress established OSTP in 1976 with a broad mandate to advise the President and others within the Executive Office of the President on the effects of science and technology on domestic and international affairs The 1976 Act also authorizes OSTP to lead an interagency effort: to develop and to implement sound science and technology policies and budgets, and to work with the private sector, state and local governments, the science and higher education communities, and other nations toward this end Dr. John H. Marburger, Director, confirmed in October 2001, serves as Science Advisor to President Bush. Dr. Marburger also co-chairs the President’s Committee of Advisors on Science and Technology (PCAST) and supports the President’s National Science and Technology Council (NSTC) The White House Office of Science and Technology Policy (OSTP)

8. Together with the Director of the Office of Management and Budget (OMB), the OSTP Director Issues a Memorandum each summer To “Heads of Executive Departments and Agencies” The memorandum Highlights the Administration’s R&D priorities Provides general guidance for setting priorities among R&D programs, and interagency R&D efforts, The memorandum reiterates the R&D Investment Criteria To improve investment decisions www.ostp.gov/html/budget07.html R&D Investment Criteria (2005) President’s Management Agenda (PMA) directs Agencies to use R&D investment criteria Relevance Quality Performance Agencies must undertake specific activities to demonstrate fulfillment of the R&D investment criteria Many activities are incorporated into the Program Assessment Rating Tool (PART)tailored for R&D programs Initiating the President’s Budget Process

9. “The Program Assessment Rating Tool (PART) is a systematic method of assessing the performance of program activities across the Federal government. The PART is a diagnostic tool; the main objective of the PART review is to improve program performance. The PART assessments help link performance to budget decisions and provide a basis for making recommendations to improve results.” The Program Assessment Rating Tool (PART) http://www.whitehouse.gov/omb/part/fy2006/2006_guidance.pdf

10. PART consists of 25-30 questions about performance and results, supported by evidence Section 1: Program Purpose & Design Are the purpose and design clear and sound? Section 2: Strategic Planning Does the program have valid long-term and annual measures? Section 3: Program Management Rate Agency management, including financial oversight and program improvement Section 4: Program Results/Accountability Rate program performance on measures and targets from section 2 and other evaluations. OMB provides a workbook of questions and a guidance document Agencies: Select and use the specific worksheets that apply There are 7 templates including one for R&D Submit a draft to their OMB examiner Enter into negotiations with OMB Key Features of PART

11. Uses of PART by OMB • PART is the technical input to the President’s Office of Management and Budget (OMB); • PART is used by OMB to identify programs that need more scrutiny than others • Examiners rank order programs based on score, and • Then focus on the low performers • PART scores do not determine future funding levels • Agency examiners give recommendations and options based on PART performance and other inputs • Final decisions made at the Branch Chief and Director levels with input from the White House • PART is kept opaque to prevent “gaming” of the system

12. Some Results from PART

13. The USG R&D spending may total $132 billion in FY 2006 USG R&D supports activities across a range of agencies The Department of Defense and the National Institutes of Health receive largest share of R&D support …The President’s Budget Request Source: AAAS, 2005

14. The US Congress After the President’s S&T Budget has been submitted to the US Congress • Congress engages in careful analysis of the proposed package of S&T funding on an agency-by-agency basis • There is extensive use of expert testimony during the course of Congressional deliberations • To further guide their deliberations, Congress passed the Government Performance and Results Act (GPRA) • Although in reality there is limited use of GPRA when it comes to Congressional S&T budgeting

15. Government Performance and Results Act (GPRA) • The Government Performance and Results Act (GPRA) • Instructs agencies to prepare strategic plans, performance plans, and performance reports that set goals and report on achieving them • GPRA differs from PART • PART is a tool of the Executive Branch of government; GPRA results in a series of reports to Congress • PART is a method for assessing program performance and how the program achieves goals; GPRA results in an Agency-based goal-setting procedure

16. …Congressional Budget Action

17. Budgeting within US Federal Agencies… • DOE • Decision-making driven by large “field reviews” and expert panels • Expert recommendations turned into budget priorities • NASA • Decentralized – though changing especially since the new administrator came onboard • Decision-making has a strong quantitative flavor • Sometimes decisions driven by historical and political factors • Optimization and sensitivity studies form the basis of recommendations • NSF • Bottom-up • Driven by program managers and division directors working in consultation with the Agency director

18. “Relevance, Quality, and Performance”– In Summary • The US government uses a combination of qualitative and quantitative information at various stages of developing its S&T investment strategies • There is greater evidence for the use of formal tools like PART in decisionmaking at the program level when formulating the President’s budget proposal to the US Congress • GPRA represents an important step by the US Congress to require more formal performance reporting at the Agency-level • But chiefly depends on expert testimony for making S&T investment decisions at all levels

19. Complex S&T Investments Strategies “Hot Topics” and The S&T Investment Process

20. Complex S&T Investments Strategies • The US Administration often identifies S&T “priority areas” apart from those announced by individual agencies • These priorities typically represent “hot topics” in science and technology • In the national interest • In which the Federal government may initiate funding • Often with the expectation that private sector funding will follow • Agencies may align their S&T activities in support of these announced “priority areas” • The National Nanotechnology Initiative represents a recent exampleof a priority area

21. Launched in FY 2001 Over 2500 active research projects in 2004 Research projects at over 500 universities, Government labs, and other research institutions in all 50 states. Built up an infrastructure of over 35 nanotechnology research centers, networks, and user facilities impact on 10,000 graduate students and teachers in 2004 alone. Funded by 11 Federal agencies Department of Agriculture (USDA) Department of Defense (DOD) Department of Energy (DOE) Department of Homeland Security (DHS) Department of Justice (DOJ) Environmental Protection Agency (EPA) National Aeronautics and Space Administration (NASA) National Institute of Standards and Technology (NIST, Department of Commerce) National Institute for Occupational Safety and Health (NIOSH) National Institutes of Health (NIH) National Science Foundation (NSF) PLUS 11 other non-funding agencies The National Nanotechnology Initiative Source: PCAST June 2005

22. This Example Alone Demonstrates the Need for… • Comparable administrative record-keeping across agencies • Information to understand the role of Federal S&T investments in developing the S&T infrastructure in “priority areas” • The development of realistic models to estimate and monitor the outcomes and impacts of public and private S&T investments in the economy

23. NSTC plays a critical role in the resolution of many of these measurement challenges The National Science and Technology Council (NSTC) was established by an Executive Order of the President of the United States on November 23, 1993. This Cabinet-level Council is the principal means for the President to coordinate the diverse parts of the Federal research and development enterprise The President chairs the NSTC Membership consists of: the Vice President of the United States, Assistant to the President for Science and Technology (OSTP) The President’s Council of Advisors on Science and Technology (PCAST) Cabinet Secretaries And Agency Heads with significant science and technology responsibilities, As well as other White House officials National Science and Technology Council (NSTC)

24. STPI Aligns With OSTP, NSTC, and PCAST to Provide Analytic Support OSTP STPI Reach-Back and Support Resources Assistant Directors Core Group Homeland Security Homeland & National Security Dr. Steve Kramer IDA Studies & Analyses FFRDC Divisions National Security Studies & Analyses FFRDC Division Directors Studies & Analyses FFRDC Division Directors Studies & Analyses FFRDC Division Directors Space, Aeronautics, & Physical Sciences Dr. Dan Garretson Space and Aeronautics Physical Sciences Social, Behavioral & Education Sciences Dr. Pamela Ebert Flattau Social, Behavioral & Education Sciences Abt Associates, Inc Life Sciences Life Sciences & The Environment Dr. Judy Hautala Environment DSSG & Other University Resources Telecommunications & Information Technology Dr. Bill Brykczynski Telecommunications and Information Technology Other Subcontractors and Consultants NSTC (Exec. Sec.) PCAST & NSTC Support Dr. Bob Roberts PCAST (Exec. Dir.)

25. Complex S&T Investments Strategies Emerging Directions

26. Many areas of research today require multi -institutional, -sectoral, and -disciplinary teams To facilitate team research, OSTP issued a a memorandum to all Federal research agencies in January 2005 Requiring them to allow formally more than one principal investigator (PI) on individual research awards Implementation planning launched formally in July 2005 Source: Federal Register July 18 2005 The “E-Government Act of 2002” promotes integrated reporting across agencies The harmonization of administrative practices can be expected to lead to the harmonization of administrative recordkeeping In 2004, NSTC explored the utility of administrative records analysis using text-mining and data visualization techniques to understand inter-agency and multidisciplinary S&T funding patterns Harmonization of Administrative Practices

27. Text-Mining and Data Visualization Text mining tools coupled with data visualization techniques can facilitate interagency and multidisciplinary S&T planning and evaluation

28. Data Visualization as an Evaluation Tool • Federal agencies have already begun to examine the role of text-mining and data visualization in understanding large and complex S&T investments • The US Department of Energy analyses of patterns of publications and patenting in Nanotechnology demonstrates the utility of such techniques

29. Use of patent and citation data to examine contributionof disciplines in a field Nanotechnology Data Visualization Source: Bill Valdez, DOE 2005

30. Data Visualization • Use of patent and citation data to examine key milestones and contributions of Federal entities Source: Bill Valdez, DOE 2005

31. Federal statistical budget ranges from High of $629 million (Census Bureau) Low of $32 million for NSF Science Resources Statistics Division (FY 2004) Complex array of surveys and analyses S&T workers Degree production Employment patterns S&T investments Federal Industrial Other S&T performers A recent study by the US National Research Council outlined important changes that will make the Federal survey system more responsive to changes in the S&T infrastructure, including: Taxonomy (i.e., field definitions) Emerging definitions of the complex innovation system In 2004, PCAST convened a workshop on the Federal-State S&T innovation infrastructure Pointing to further framework development and data collection issues at the Federal-regional interface Monitoring Changes in the S&T Infrastructure

32. States and regions are increasingly and systematically using science and technology (S&T) as part of their economic development portfolios. Regional S&T-based economic development initiatives are those that promote the establishment and growth of S&T-driven companies. Innovation is a complex process and is increasingly being conceptualized as occurring within an innovation system. The innovation systems approach is concerned with the complex set of relationships among various actors – companies, government and universities. Regional S&T-based economic development initiatives are those that promote the establishment and growth of science and technology-driven companies Networked Industrial District (NID) The typical Networked Industrial District (NID) is comprised of networks of small, locally owned firms, which enable the evolution of strong local cultural identity and shared industrial expertise. Firms in the same or related industries are able to rapidly adapt to changing markets and differentiated demand through collaboration and use of new technologies. Examples include Silicon Valley, Boston (Route 128), and northern Italy. Mapping the Innovation Infrastructure

33. “The combination of finite resources and a multitude of new research opportunities requires careful attention to funding priorities and wise choices by agency managers...Agencies must vigorously evaluate existing programs and, wherever possible, consider them for modification, redirection, reduction, or termination in keeping with national needs and priorities…” July 8 2005 “Memorandum for the Heads of Executive Departments and Agencies” “… We need econometric models…” in sufficient detail to guide complex decision-making Jack Marburger, May 20 Editorial Science magazine In July 2005, STPI convened a panel of experts to respond to this challenge. There is a Need for Large-Scale Models…

34. How Case Studies Could Advance Modeling • Studies of private-sector R&D show that: • The financial returns can be significant; probably as high or higher than other investments; • Social returns may exceed the direct financial returns to investors by a factor of about 2 to 1. • Studies of Federal R&D • Range from large-scale agency-wide analyses to project-level cost-benefit studies • Show small to medium financial returns to research, though may be subject to selection biases, and • Use disparate methods that make it difficult to aggregate across studies • A systematic program of case studies of Federal R&D programs, decisions and practices could provide • Better understanding of needed R&D data (metrics, relationships) • Important lessons for decision-making • Improved measures of the social returns to federal R&D

35. Implications for Evaluation Strategies • Evaluation is becoming increasingly forward-looking and adapting to the new “innovation ecosystem” • Emerging frameworks • Portfolio/system-level assessments • Advanced bibliometrics, paper-patent analysis • Increasingly moving beyond citation analysis to understand linkages between basic research and innovation (paper-patent analysis) • Increasingly looking internationally (triadic patents) • Social network methods • Assess collaboration, multi-disciplinarity by understanding who works with whom and how • Most useful when assessing initiatives where collaborations, inter-disciplinarity are critical to achieving outcomes • Often use bibliometric data as input, but can use any data source (administrative records, surveys) • Incorporate visualization and quantitative tools

36. Improvements in data collection strategies are needed even as appropriate methods are being developed to guide complex S&T investment decisions Activities already underway today : Important changes in Federal administrative practices and recordkeeping Use of text-mining and data visualization tools can lead to a better understanding of Federal patterns of support of multidisciplinary, interagency funding Evaluators have already demonstrated the use of data visualization to document complex patterns of S&T investment over time Furthermore, numerous sources have described what changes are needed in Federal statistical surveys to reflect the emergence of multidisciplinary, interagency investments in science and technology One remaining challenge occurs at the Federal-regional interface Case studies of the relationship between Federal investment in innovative S&T also has the potential for refining large-scale econometric models to meet the needs of planners and policymakers …so clearly laid out by the President’s Science Advisor. Final Thoughts