slide1 l.
Skip this Video
Loading SlideShow in 5 Seconds..
Bibliometric analyses of publications and grant productivity: SPIRES, a new Web-based tool. PowerPoint Presentation
Download Presentation
Bibliometric analyses of publications and grant productivity: SPIRES, a new Web-based tool.

Loading in 2 Seconds...

play fullscreen
1 / 44

Bibliometric analyses of publications and grant productivity: SPIRES, a new Web-based tool. - PowerPoint PPT Presentation

  • Uploaded on

Bibliometric analyses of publications and grant productivity: SPIRES, a new Web-based tool. . The National Institute of Environmental Health Sciences The National Institutes of Health. Today’s discussion points : Background - Program Analysis Branch Goals

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Bibliometric analyses of publications and grant productivity: SPIRES, a new Web-based tool.' - fruma

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Bibliometric analyses of publications and grant productivity: SPIRES, a new Web-based tool.

The National Institute of Environmental Health Sciences

The National Institutes of Health

  • Today’s discussion points:
      • Background
      • - Program Analysis Branch Goals
      • - Current methods for evaluating research including bibliometric analysis
      • Introduction to SPIRES (Scientific Publications Information REtrieval System)
      • Results of a Validation Experiment
      • Future Directions/
      • Live demo
pab analysis goals
PAB Analysis Goals
  • Information transfer– Communicate the high-quality research produced by the NIEHS portfolio.
  • Track current DERT supported research– Examine number of grants and annual support broken down into major fields of science.
  • Analyze all incoming grants– Develop a two-three sentence description of each funded grant, place into a web searchable data base. Track study sections and priority scores.
  • Evaluate scientific and public health impact of grants.Track publications for each grant award and center. Every RFA and PA needs to be tracked for publication productivity and scientific impact.

Evaluation Tools

  • peer review– accepted “gold standard of evaluation,” provides a review of quality of research
  • – usually applies to specific projects, difficult to use for the evaluation of programs or fields, time consuming, expensive and labor intensive
  • bibliometric analysis– examines publications, impact factors, literature citations, and patent counts; is based on the premise that a publication represents quality research as it passed peer review, and cited literature has merit
  • – quantitative can measure aggregate quality of fields or programs, but not always useful across fields
needs for tracking publications
Needs for tracking publications
  • To systematically track the productivity of the portfolio, specific tools are needed by the Program Analysis Branch (PAB) that are not currently available.
  • According to NLM, NIEHS-supported grantees published 1637 publications in 2000.
  • The idea was to link the grant database (IMPAC II) to the National Library of Medicine (NLM) database.
  • Web-based routines will allow NIEHS staff to display information and generate multiple outputs.
  • System should be robust so that new NLM downloads can be accepted and operator updates are feasible.
development of spires joining impac ii with nlm
Development of SPIRES: joining IMPAC II with NLM.
  • SPIRES :Scientific Publication Information REtrieval System
  • NLM keeps all its data in flat files.
  • Jerry Nehls downloaded ES/NIEHS data from NLM to a local server.
  • Paul Jordan developed computer algorithms to search publications for Grant numbers, (ES-----) and developed tables in Oracle to create a relational database linking IMPAC II with NLM data.
  • Bob Hoppin developed a Web-based interface.
  • Paul Jordan and Ying Gao currently upgrading functionality and usability.
status of spires s cientific p ublication i nformation re trieval s ystem
Status of SPIRES.Scientific Publication Information REtrieval System
  • Currently have 1995-2001 (August) data loaded with 9,615 publications
  • SPIRES is being used by the PAB:

– Finding and removing bugs

– Working on developing a better interface and reportingstructure

  • SPIRES is accessible to anyone in NIEHS
  • Working on the ambiguous Project code problem.

– ???ES007218 – rarely is the prefix captured: R01, T32, F32, P01

– ES-007218 – hyphen present or zero missing?


Status of SPIRES (contin.).Scientific Publication Information REtrieval System

  • Implement and expand system through theSPIRES Working Group: PAB, Computer Applications Specialists, DERT Staff, Sheila Newton, Larry Wright
  • Running test reports: looking at data quality through validation experiments
  • SPIRES as a tool for analysis– develop reports on Arsenic, Oxidative Stress, Cardiovascular disease
  • Quarterly updates to SPIRES
  • V 1.0 was beta-tested by several groups at NIH
  • V 2.0 being developed and tested



Searching for pub information

Searching for grant information

Crisp terms


SPIRES Validation – Jerry Phelps

  • Compare SPIRES data to information contained in the grant file (progress reports and competitive renewals)
  • Grants Years
  • – Pilot studies : MERIT n=16 1995-2001
  • R03 n= 16(44) 1995-2001
  • – Larger study: R01 n=20 1995-2001 P01 n=2 1995-2001
  • P30 n=3 2000
  • P42 n=3 2000































SPIRES Validation

Pilot studies

Larger study


“R” AWARDS: 475/593 “P” AWARDS: 167/393

80% 47%


SPIRES Validation

  • Based on these validation studies, Paul Jordan and Jerry Nehls have made improvements in the download process and decoding algorithms that will improve SPIRES’ ability to capture papers from PUBMED.
  • PAB is following up with an analysis of the entire P30 program to give the Center Directors data that they will use to increase the number of papers that cite Center support.
  • DERT and PAB will be contacting PI’s who are not citing their grants in their publications.

SPIRES as an Analysis Tool

1. Comparisons of productivity for MERIT nominees.

2. Productivity assessments for competitive renewal applications.

3. Identify high-impact papers from grantees who are visiting NIEHS or being visited by NIEHS leadership.

4. Program Evaluations – Productivity assessments for grant programs such as arsenic, oxidative stress, P01, Children's centers, and R03.

5. Customized reports in Excel spreadsheets allow for easy sorting according to impact factor, date, journal, PI, institution.


SPIRES – Future Directions

  • Develop better reporting features
  • Integrate with Centers, Superfund, Contracts, and DIR databases
  • Incorporate links to PubMed and ISI Citation reports
  • Demo to other IC’s, working towards building a NIH-wide system
  • Consider incorporating SPIRES into the Quick View Reports (QVR) section of the Electronic Council Book
  • Add additional tracking fields (through QVR): grant descriptors, disease, chemical, and organ systems
  • Consider making a site available on Web for grantees

Special thanks to:

Paul Jordan, Ying Gao, Bob Hoppin & Jerry Nehls (CTB)

Nancy Stegman (CTB)

Ben Van Houten, Jerry Phelps, Martha Barnes, Edith Lee, Judy Hanson (PAB)

Thor Fjellstedt, Bill Suk, & Anne Sassaman (DERT)

Larry Wright (DIR, Library)

Sheila Newton (OPPE)