Building a Scientific Basis for Research Evaluation

1. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior Researcher Research Trends Seminar October 17, 2012

2. Outline • Science of science policy • A proposed conceptual framework • Empirical approaches: • NSF Engineering Dashboard • ASTRA – Australia • HELIOS – France • Final thoughts

3. Outline • Science of science policy • A proposed conceptual framework • Empirical approaches: • NSF Engineering Dashboard • ASTRA – Australia • HELIOS – France • Final thoughts

4. The emergence of a science of science policy • Jack Marburger’s challenge (2005) • Science of Science & Innovation Policy Program at the National Science Foundation (2007) • An emerging, highly interdisciplinary research field • Science of Science Policy Interagency Task Group publishes a “Federal Research Roadmap” (2008): • The data infrastructure is inadequate for decision-making • STAR METRICS (2010)

5. Why a science of science policy? • Evidence-based investments • Good metrics = good incentives • Science is networked and global • Build a bridge between researchers and policymakers • Researchers ask the right questions • The adjacent possible: leverage existing and new research and expertise • New tools to describe & measure communication

6. The timing is right:

7. A conceptual framework for a science of science policy

8. Getting the right framework matters • What you measure is what you get • Poor incentives • Falsification • Usefulness • Effectiveness

9. A proposed conceptual framework Adapted from Ian Foster, University of Chicago

10. A framework to drive person-centric data collection • WHO is doing the research • WHAT is the topic of their research • HOW are the researchers funded • WHERE do they work • With WHOM do they work • What are their PRODUCTS

11. Challenge – The data infrastructure didn’t exist However, some of the data do exist

12. Empirical Approaches Leveraging existing data to begin describing results of the scientific enterprise

13. An empirical approach • Enhance the utility of enterprise data • Identify authoritative “core” data elements • Develop an Application Programming Interface (API) • Data platform that provides programmatic access to public (or private) agency information • Develop a tool to demonstrate value of API

14. Topic modeling: Enhancing the value of existing data Automatically learned topics (e.g.): … t6. conflict violence war international military … t7. model method data estimation variables … t8. parameter method point local estimates … t9. optimization uncertainty optimal stochastic … t10. surface surfaces interfaces interface … t11. speech sound acoustic recognition human … t12. museum public exhibit center informal outreach t13. particles particle colloidal granular material … t14. ocean marine scientist oceanography … … NSF proposals • Topic Model: • Use words from • (all) text • Learn T topics t49 t18 t114 t305 Topic tags for each and every proposal David Newman - UC Irvine

15. Stepwise empirical approach • Enhance the utility of enterprise data • Identify authoritative “core” data elements • Develop an Application Programming Interface (API) • Data platform that provides flexible, programmatic access to public (or private) agency information • Develop a tool to demonstrate value of API

17. Stepwise empirical approach • Enhance the utility of enterprise data • Identify authoritative “core” data elements • Develop an Application Programming Interface (API) • Data platform that provides programmatic access to public (or private) agency information • Develop a tool to demonstrate value of API

18. Outline • Science of science policy • A proposed conceptual framework • Empirical approaches: • NSF Engineering Dashboard • ASTRA – Australia • HELIOS – France • Final thoughts

24. Outline • Science of science policy • A proposed conceptual framework • Empirical approaches: • NSF Engineering Dashboard • ASTRA – Australia • HELIOS – France • Final thoughts

25. Linking administrative and grant funding datain Australia

26. Outline • Science of science policy • A proposed conceptual framework • Empirical approaches: • NSF Engineering Dashboard • ASTRA – Australia • HELIOS – France • Final thoughts

27. Describing public-private partnerships in France People People

28. What does getting it right mean? • A community driven empirical data framework should be: • Timely • Generalizable and replicable • Low cost, high quality • The utility of “Big Data”: • Disambiguated data on individuals • Comparison groups • New text mining approaches to describe and measure communication • ??

29. Final thoughts

30. Policy makers can engage SciSIP communities: • Patent Network Dataverse; Fleming at Harvard and Berkeley • Medline-Patent Disambiguation; Torvik & Smalheiser at U Illinois) • COMETS (Connecting Outcome Measures in Entrepreneurship Technology and Science); Zucker & Darby at UCLA

31. The power of open research communities • Internet and data technology can transform effectiveness of science: • Informing policy • Communicating science to the public • Enabling scientific collaborations • Interoperability is key • Publishers are an important part of the community

32. THANK YOU! Rebecca F. Rosen, PhD E-Mail: rrosen@air.org 1000 Thomas Jefferson Street NWWashington, DC 20007 General Information: 202-403-5000TTY: 887-334-3499 Website: www.air.org