Barry Bozeman, Arizona State University Jan Youtie, Georgia Institute of Technology USA - PowerPoint PPT Presentation

Barry bozeman arizona state university jan youtie georgia institute of technology usa
1 / 17

  • Uploaded on
  • Presentation posted in: General

Credibility and Use of Scientific and Technical Information in Policy Making: An Analysis of the Information Bases of the National Research Council’s Committee Reports. Barry Bozeman, Arizona State University Jan Youtie, Georgia Institute of Technology USA. Motivation.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

Barry Bozeman, Arizona State University Jan Youtie, Georgia Institute of Technology USA

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Barry bozeman arizona state university jan youtie georgia institute of technology usa

Credibility and Use of Scientific and Technical Information in Policy Making: An Analysis ofthe Information Bases of the National Research Council’s Committee Reports

Barry Bozeman, Arizona State University

Jan Youtie, Georgia Institute of Technology




  • Contribution: little empirical understanding of use of STI

    • Much literature on use of formal information in decision-making

    • No literature on use of STI in science, technology and innovation (S&T) policy

  • Research questions for study

    • Does the perception of the limited use of formal scientific and technical information (STI) accord with empirical reality?

    • What types of information “compete” with STI for inclusion in science policy-making (where one might expect greatest receptivity)?

    • How does the choice of various types of information relate to the use and impacts of science policy reports and recommendations?

Definition of sti

Definition of STI

  • Open scientific and technical literature appearing in peer-reviewed academic journals or proceedings.

  • STI used in a narrow sense v. typical in the literature (McClure, 1988; Walker and Hurt, 1990)

National research council nrc

National Research Council (NRC)

  • Performs research work for the production of reports on science and technology issues within the National Academies

    • National Academy of Science (NAS)

    • National Academy of Engineering (NAE)

    • Institute of Medicine

  • National Academies serves as advisor about science and technology intensive policy issues to Congress

  • Little research on the NRC

    • Ellefson (2000): single case on non-forest federal land management

    • Policansky (1999): an ecologist and staff member at the NRC, argues that well-constructed committees with high level of trust given a precisely constructed policy question are most successful

    • Shapiro and Guston (2006): bureaucracies will shirk their duties, relying on the peer review process to for correction

    • Fein (2011): NRC plays an increasingly important role in regulatory peer review

    • Parascandola (2007): conflict of interest policy history in the NAS and NRC

    • Martin and Irvine (1989): lack of priority setting in NRC reports

National research council process

National Research Council Process

National Academies (2006) Our Study Process: Ensuring Independent, Objective Advice.

Group based credibility model

Group-based Credibility Model



  • All National Academies reports published 2005-2012

  • Exclusion of workshop, narrow/very particular studies: Transportation Bureau (NAE), Health and Safety (IOM), repeat Congressionally authorized standing studies

  • Focus on board appointed/empaneled single shot studies (mostly NAS)

  • Results=589 reports



  • Report variables: publication year, policy area, National Research Council department, Congressional authorization, # pages, # committee members, # reviewers by sector (business, government, academia, etc.) and geographic location (domestic, foreign), # cited references, # congressional briefings

  • Committee variables: chair versus member, terminal degree and concentration, industry affiliation by three-digit NAICS, sectoral affiliation (business, government, academia, etc.), title (e.g., president, secretary of agency, gov’t laboratory director)

  • Committee publications: # publications in the Web of Science and Scopus, # citations, primary Web of Science category, self-citation in reference list

  • Report outcomes: # Web of Science and Scopus papers, newspapers, industry trade press, newsletters, magazines, news transcripts, hearing transcripts, Congressional record daily issues, bills, House and Senate reports, Congressional Research Service reports.

  • Reference variables: STI (journal article, published proceeding) v. government report/document, book or book chapter, working paper, industry and nonprofit organization document, interview/testimony, website, expert opinion

Database linkages

Database Linkages

  • NRC annual reports

  • Web of Science

  • Scopus

  • LexisNexis (news, web)

  • Proquest, UNT Digital Library (hearings, legislation, bills, Congressional reports)

Preliminary findings reports

Preliminary Findings: Reports

  • 14% Congressionally authorized

  • Natural Resource and Defense areas most common and most likely Congressionally authorized (~ 30% of studies in these areas)

Average number of pages declining

Average Number of Pages Declining

  • Average number of pages=188

Policy areas vary in use of references in nrc reports 2005 2006

Policy Areas Vary in Use of References in NRC reports (2005,2006)

Preliminary findings outcomes

Preliminary Findings: Outcomes

  • NRC reports have greater impact than expected

    • 37% self-reported Congressional communications

      • Informal briefings (30%) more common than formal testimony (15%)

    • 20% reports have independent presence in Congressional documents

      • Not necessarily overlapping with self-reports (only 28% of reports with any Congressional impact are independent + self reports)

    • Taken together 45% of NRC reports have either (or both) of the above impacts

Preliminary findings reports and outcomes

Preliminary Findings: Reports and Outcomes

  • ~ over time (except for independent outcomes being much higher, and self-reported outcomes lower, in 2005)

  • Congressionally authorized studies are associated with self-reported briefings (29% versus 6% for nonauthorized) and independent outcomes (26% versus 12% for nonauthorized)

  • Longer NRC studies tend to have self-reported briefings and independent outcomes (briefings=216 mean # pages versus 171 for no briefings; outcomes=213 mean versus 181 for no outcomes)

Preliminary findings references and outcomes

Preliminary Findings: References and Outcomes

  • Inclusion of references makes little difference to outcomes

Plans and some issues

Plans and some issues

  • Planned analysis

    Impact f(STI, active researchers on committee, other committee information, report characteristics)

  • Limitations

    • Not every NRC report has references; reference formats differ

    • We are not analyzing report content

    • We would like to do some interviews



  • This work was supported by the US National Science Foundation, Science of Science and Innovation Policy, Award #1262251. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the sponsors.

  • For more information

  • Login