1 / 21

How good is our research? New approaches to research indicators

How good is our research? New approaches to research indicators. Average is a metric; distribution is a picture. ‘Average impact’ is a good bibliometric index but not sufficient A tool for reporting but not for action Data are skewed, so average is not central

Download Presentation

How good is our research? New approaches to research indicators

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. How good is our research?New approaches toresearch indicators

  2. Average is a metric; distribution is a picture • ‘Average impact’ is a good bibliometric index but not sufficient • A tool for reporting but not for action • Data are skewed, so average is not central • Many papers are uncited and a few papers are very highly cited • New approach looks at where the spread of performance falls • Activity is located within distribution by more than a single metric • Thresholds help in describing peak of performance. • This improves descriptive power, information content and management value

  3. Traditional impact indicators

  4. Distribution of “research performance”

  5. Distribution of “research performance” A good ‘indicator’ should capture and reflect this in some meaningful way Do current metrics do this?

  6. Distribution of data values - income Minimum Maximum

  7. Distribution of data values - impact The variables for which we have metrics are skewed and therefore difficult to picture in a simple way

  8. Simplifying the data picture • Scale data relative to a benchmark, then categorise • Could do this for any data set • All journal articles • Uncited articles (take out the zeroes) • Cited articles • Cited less often than benchmark • Cited more often than benchmark • Cited more often but less than twice as often • Cited more than twice as often • Cited less than four times as often • Cited more than four times as often

  9. Categorising the impact data This grouping is the equivalent of a log 2 transformation. There is no place for zero values on a log scale.

  10. UK ten-year profile 680,000 papers MODE (cited) AVERAGE RBI = 1.24 MODE MEDIAN THRESHOLD OF EXCELLENCE?

  11. Implications • Is the UK as good as we thought? • YES - the average is unchanged • What lies beneath just became apparent • The effective peak is very concentrated • Other countries would probably look similar • New metrics are needed • Average impact not indicative of distribution • Need to add median, mode • Proportion of activity at thresholds of excellence • Above world average, More than 4 x world average, etc • Evaluate methodology • Does it work by year and by subject? • How can we apply it?

  12. Time profile

  13. Subject based curves

  14. Subject & site profiles – molecular biology

  15. HEIs – 10 year totals - 1

  16. HEIs – 10 year totals - 2

  17. HEIs – 10 year totals – 4.1 Smoothing the lines would reveal the shape of the profile

  18. HEIs – 10 year totals – 4.2 Absolute volume would add a further element for comparisons

  19. HEIs – 10 year totals – 4.3

  20. What next? • Profiles • Create a view of the distribution of performance • Provide more information useful to management • Require a change in metrics • Applications • Disaggregate the components of the research base • Track institutional profiles against benchmark • Evaluate the link between platform and peak • Track papers through time: e.g. leaders vs. climbers

  21. How good is our research?New approaches toresearch indicators

More Related