1 / 33

Building a Scientific Basis for Research Evaluation

This seminar discusses the science of science policy and proposes a conceptual framework for research evaluation. It explores empirical approaches used by organizations like NSF Engineering Dashboard, ASTRA in Australia, and HELIOS in France. The session also highlights the importance of linking administrative and grant funding data, and describes public-private partnerships in France.

mazzella
Download Presentation

Building a Scientific Basis for Research Evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior Researcher Research Trends Seminar October 17, 2012

  2. Outline • Science of science policy • A proposed conceptual framework • Empirical approaches: • NSF Engineering Dashboard • ASTRA – Australia • HELIOS – France • Final thoughts

  3. Outline • Science of science policy • A proposed conceptual framework • Empirical approaches: • NSF Engineering Dashboard • ASTRA – Australia • HELIOS – France • Final thoughts

  4. The emergence of a science of science policy • Jack Marburger’s challenge (2005) • Science of Science & Innovation Policy Program at the National Science Foundation (2007) • An emerging, highly interdisciplinary research field • Science of Science Policy Interagency Task Group publishes a “Federal Research Roadmap” (2008): • The data infrastructure is inadequate for decision-making • STAR METRICS (2010)

  5. Why a science of science policy? • Evidence-based investments • Good metrics = good incentives • Science is networked and global • Build a bridge between researchers and policymakers • Researchers ask the right questions • The adjacent possible: leverage existing and new research and expertise • New tools to describe & measure communication

  6. The timing is right:

  7. A conceptual framework for a science of science policy

  8. Getting the right framework matters • What you measure is what you get • Poor incentives • Falsification • Usefulness • Effectiveness

  9. A proposed conceptual framework Adapted from Ian Foster, University of Chicago

  10. A framework to drive person-centric data collection • WHO is doing the research • WHAT is the topic of their research • HOW are the researchers funded • WHERE do they work • With WHOM do they work • What are their PRODUCTS

  11. Challenge – The data infrastructure didn’t exist However, some of the data do exist

  12. Empirical Approaches Leveraging existing data to begin describing results of the scientific enterprise

  13. An empirical approach • Enhance the utility of enterprise data • Identify authoritative “core” data elements • Develop an Application Programming Interface (API) • Data platform that provides programmatic access to public (or private) agency information • Develop a tool to demonstrate value of API

  14. Topic modeling: Enhancing the value of existing data Automatically learned topics (e.g.): … t6. conflict violence war international military … t7. model method data estimation variables … t8. parameter method point local estimates … t9. optimization uncertainty optimal stochastic … t10. surface surfaces interfaces interface … t11. speech sound acoustic recognition human … t12. museum public exhibit center informal outreach t13. particles particle colloidal granular material … t14. ocean marine scientist oceanography … … NSF proposals • Topic Model: • Use words from • (all) text • Learn T topics t49 t18 t114 t305 Topic tags for each and every proposal David Newman - UC Irvine

  15. Stepwise empirical approach • Enhance the utility of enterprise data • Identify authoritative “core” data elements • Develop an Application Programming Interface (API) • Data platform that provides flexible, programmatic access to public (or private) agency information • Develop a tool to demonstrate value of API

  16. Stepwise empirical approach • Enhance the utility of enterprise data • Identify authoritative “core” data elements • Develop an Application Programming Interface (API) • Data platform that provides programmatic access to public (or private) agency information • Develop a tool to demonstrate value of API

  17. Outline • Science of science policy • A proposed conceptual framework • Empirical approaches: • NSF Engineering Dashboard • ASTRA – Australia • HELIOS – France • Final thoughts

  18. Outline • Science of science policy • A proposed conceptual framework • Empirical approaches: • NSF Engineering Dashboard • ASTRA – Australia • HELIOS – France • Final thoughts

  19. Linking administrative and grant funding datain Australia

  20. Outline • Science of science policy • A proposed conceptual framework • Empirical approaches: • NSF Engineering Dashboard • ASTRA – Australia • HELIOS – France • Final thoughts

  21. Describing public-private partnerships in France People People

  22. What does getting it right mean? • A community driven empirical data framework should be: • Timely • Generalizable and replicable • Low cost, high quality • The utility of “Big Data”: • Disambiguated data on individuals • Comparison groups • New text mining approaches to describe and measure communication • ??

  23. Final thoughts

  24. Policy makers can engage SciSIP communities: • Patent Network Dataverse; Fleming at Harvard and Berkeley • Medline-Patent Disambiguation; Torvik & Smalheiser at U Illinois) • COMETS (Connecting Outcome Measures in Entrepreneurship Technology and Science); Zucker & Darby at UCLA

  25. The power of open research communities • Internet and data technology can transform effectiveness of science: • Informing policy • Communicating science to the public • Enabling scientific collaborations • Interoperability is key • Publishers are an important part of the community

  26. THANK YOU! Rebecca F. Rosen, PhD E-Mail: rrosen@air.org 1000 Thomas Jefferson Street NWWashington, DC 20007 General Information: 202-403-5000TTY: 887-334-3499 Website: www.air.org

More Related