1 / 17

Accounting for Research Quality: Research Audits and the Journal Rankings Debate

Kent Business School Journal Rankings: Love ‘em or hate ‘em? Weds 30 2013. Accounting for Research Quality: Research Audits and the Journal Rankings Debate. Michael Rowlinson Professor of Organization Studies

shalom
Download Presentation

Accounting for Research Quality: Research Audits and the Journal Rankings Debate

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Kent Business School Journal Rankings: Love ‘em or hate ‘em? Weds 30 2013 Accounting for Research Quality: Research Audits and the Journal Rankings Debate Michael Rowlinson Professor of Organization Studies Co-editor The Association of Business Schools Academic Journal Quality Guide (version 4, 2010)

  2. Distribution of ratings for Business and Management in RAE2001 and RAE2008

  3. Criticism of editors of the ABS Guide • Biased (Hoepner & Unerman), self-appointed gerontocrats, who know they could never recognize academic excellence for themselves (Burrell) • Auto-asphyxiation fetishists who can find time to endlessly revise their list but not for “the critical task of reading” (Willmott) • If our academic leaders, such as Research Directors, “actually read lots of material” then they would be able to “recognize quality itself”, but the availability of lists distracts them from the necessary academic task of critical reading (Burrell)

  4. Feedback from the 2008 RAE panel “there was not a perfect correlation between the quality of a piece of work and its place of publication. Although much top-quality work was indeed published in what are generally regarded as leading journals, top quality work could also be found in journals occupying a lower position in conventional rankings. Similarly, some of the work considered that had been published in so-called leading journals was thought to be of less than top quality. The proportions in each of these categories also varied across sub-disciplines. There was also a considerable amount of work published in books or other formats, some of which was of world-leading quality. It would therefore be inappropriate in the future to use assessments of journal quality alone to assign quality ratings to individual items of work”

  5. Should the REF use The ABS List? (we woz robbed in the RAE 2008!)

  6. Four critiques of journal lists • Expert peer review is sufficiently reliable so there is no need to consult journal quality lists • Therefore the best way to assess outputs for submission is to read them • Variation in citations for articles in the same journal • Therefore the journal citation impact factor is not reliable as a guide to the quality of individual articles • Subject Field Bias in Journal Ratings • The ABS Guide is allegedly biased against accounting, operations research, and inter-disciplinary research • Journal rankings distort the REF as a research audit • if it wasn’t for journal rankings the REF would work well for resource allocation, which is it’s main function

  7. Was the 2008 RAE more inclusive than the ABS Guide? • Proposition 1. • Outputs from more journals received the highest grades, 3 and 4, from the Business and Management RAE2008 panel than would be the case if the ABS Guide is used to rate outputs; • Proposition 2. • More journal outputs received the highest grades, 3 and 4, from the Business and Management RAE2008 panel than would be the case if the ABS Guide is used to rate outputs; • Proposition 3. • Outputs from more research active staff received the highest grades, 3 and 4, from the Business and Management RAE2008 panel than would be the case if the ABS Guide is used to rate outputs.

  8. Count of ABS 2010 rated jnl outputs from individual staff RAE2008

  9. Alternative methods for assessing the quality of research outputs (RA2) • Individualized citation metrics • Informed by Web of Science, Scopus, or Publish or Perish • Journal ratings (e.g. The ABS Guide) • Informed by citation impact factors, peer review, or a hybrid combination • Peer review (e.g. Bus and Mgt REF panel) • Informed by expert judgment

  10. Conclusions • Is it time to consider a more appropriate, inclusive, and economical form of ranking for research in business and management? • A ranking system should recognize the importance of research in relation to teaching in business schools. • Journal ratings would have role to play in an improved ranking system.

  11. RAE2008 Cum % bus & mgt outputs in journals

More Related