Scholarly Communications: Changes to Peer Review

1. Scholarly Communications:Changes to Peer Review • Bradley Hemminger • School of Information and Library Science • University of North Carolina at Chapel Hill

2. Scholarly Communications Process Present to colleagues V2 Present at conference V3 Idea V1 Submit to journal V4 Revision to include additional new results V8 Revision to update analysis V7 Referees Revision for journal V5 Journal Final Revision V6

3. Scholarly Communications Process formulate discussion discussion, revision Present to colleagues V2 Present at conference V3 Idea V1 Submit to journal V4 comments comments comments Author revision Revision to include additional new results V8 Revision to correct analysis V7 Referees Revision for journal V5 Journal Final Revision V6 Copyproofing Criticisms, new thoughts, revision new results, revision Two peer reviews

4. Scholarly Communications Process: What’s Produced Journal Final Revision V6

5. Scholarly Communications Process:What I’d like to see saved! formulate discussion discussion, revision Present to colleagues V2 Present at conference V3 Idea V1 Submit to journal V4 comments comments comments Author revision Revision to include additional new results V8 Revision to correct analysis V7 Referees Revision for journal V5 Journal Final Revision V6 Copyproofing Criticisms, new thoughts, revision new results, revision Two peer reviews

6. Peer Review Output With Respect to XYZ… Accept reject revise Review (Peer) Qualitative Grade Article Qualitative Comments Comments to Author

7. Generalized Review Model Score (1-10) Quantitative Grades Review (open, peer, machine) Accept, Reject, Revise, With respect to XYZ Qualitative Grade Article Qualitative Comments Comments to Author

8. Overview of Peer Review Quantitative Grade Score (1-10) Filter Published Article Review Peer, Open, Machine Accept, reject, revise with respect to XYZ standards Article submitted Send elsewhere Qualitative Grade Comments to Author Reject Qualitative Comments

9. General Review Model Parallels • In general, you have sample (material) which is judged/scored quantitatively and qualitatively by an identified observer. • Current Peer Review • Automated Scoring Systems (lab tests) • Moderated email lists (announce) • Moderated Eprints servers (arXiv)

10. Peer Review Options Comment Quantitative Score (1-10) Score (1-10) Score (1-10) Score (1-10) Score (1-10) #citations #hits #number of related discussions Qualitative Rel Yes/No Group Rel Yes/No Group Absolute Absolute Rel Yes/No Group Rel Yes/No Group Y Y Y Y Y ? • Human Judgement • Expert peer review (status quo) • Certified expert peer review • Open Peer Review BMJ, BioMed • Open comment review pyscprints • Computer Judgement • Computer peer review • Human Usage • Citation-based (CiteSeer) • Usage counts (CiteSeer) Example • Quantity of discussion • Coarse Categorization • Two Tier (grey/gold) • Moderator (current arXiv) • No review (old arXiv)

11. Judgment based on some combination of reviews/comments formulate discussion discussion, revision Present to colleagues V2 Present at conference V3 Idea V1 Submit to journal V4 comments comments comments Author revision Revision to include additional new results V8 Revision to correct analysis V7 Referees Revision for journal V5 Journal Final Revision V6 Copyproofing Criticisms, new thoughts, revision new results, revision Two peer reviews

12. What areas of improvement? • Review Process Change • Search, Retrieval Process Change • Service Provider Process Change

13. Review Process Changes • Include open reviews and comments to get additional feedback. • Support normal scholarly discourse, allowing give and take with author responding. • Add quantitative scores to allow better filtering based on quality during retrieval • Add machine (automated) reviews

14. Search and Retrieval Changes • Universal Archive: all material freely available. • Universal Searching: standardized metadata (Dublin Core) for general searching. • Automated agents to bring material of interest to your attention. • Use additional review scores (public reviews, machine) to help filter search. • Example: article scores > 7.0, refereed, citation count above X, type=research article, search terms = schizophrenia, geneX)

15. Provider Service Change • What is worth paying for? • Quality review (Faculty of 1000) • Proofing, citation linking, professional presentation (CiteSeer, Cite-base) • Archival (JStor) • Who hosts material: • Society (arXiv) • Commerical Publishers (Elesiever,BioMedCentral) • University Library (MIT Dspace)

16. New Frameworks for Peer Review • As an enabling technology: frameworks like NeoRef supports all of the above models in any combination at the same time, while eliminating many of the costs. • Requirements: Based on OAI and Dublin Core, and expectation of logical universal archive, an universal unique object IDs (URL, DOIs) and person IDs.

17. Example Model (NeoRef) • All material and metadata are author contributed to a public OAI archive (author retains ownership). • All materials universally available via search engines that harvest metadata from OAI archives. • OAI archives have automated or manual moderator to filter out “junk”. • Everything--articles, reviews, comments, indexings, etc., are stored as digital content on archive using the same mechanism. Reviews contain quantitative score, qualitative grade, qualitative comments. Logically (although not physically), a two tier (Grey & Gold) system for materials • High quality keep forever material reviewed by known entity • Grey material (everything else)

18. NeoRef for Movies, Dates, DocSouth • The same process used by NeoRef to support Scholarly Communication could be used for almost any purpose. All that is required is storage of Digital Content Items, and linking of reviews, comments, etc to them. • Movies: Grey is everyone’s reviews; Gold is Siskel and Ebert reviews • DocSouth: self cataloged and indexed items are Grey; librarian/archivist cataloged and indexed items are Gold.

19. Can we save the Gold and Grey? formulate discussion discussion, revision Present to colleagues V2 Present at conference V3 Idea V1 Submit to journal V4 comments comments comments Author revision Revision to include additional new results V8 Revision to correct analysis V7 Referees Revision for journal V5 Journal Final Revision V6 Copyproofing Criticisms, new thoughts, revision new results, revision Two peer reviews

20. NeoRef Storage Model Auto-indexing Revision to include additional results and analyses V8 Author Indexing Journal Final Revision V6 Comments on V3 Journal Submission V4 Comments on V6 Conference paper (v3) Material expressing content Local powerpoint Presentation v2 Two peer reviews Machine Review Digital Archive Filter (Moderate) Author Grey Literature Automated Recognized Expert Top Tier (Keep Forever) Open (anyone)

21. What do users want?

22. The Association of Learned and Professional Society Publishers (ALPSP) SurveyAuthors and Electronic Publishing • Scholarly research communication has seen far-reaching developments in recent years. • Most journals are now available online as well as in print, and numerous electronic-only journals have been launched; • the Internet opens up new ways for journals to operate. • Authors have also become conscious of alternative ways to communicate their findings, and much has been written about what they ought to think.

23. ALPSP felt that it would be timely to discover what they actually thought and what they actually did. This survey aimed to discover the views of academics, both as authors and as readers. Some 14,000 scholars were contacted across all disciplines and all parts of the world, and nearly 9% responded; their detailed comments make thought-provoking reading. Alma Swan and Sheridan Brown. Authors and Electronic Publishing: The ALPSP Research Study on Authors' and Readers’ Views of Electronic Research Communication. (West Sussex, UK: The Association of Learned and Professional Society Publishers, 2002). http://www.alpsp.org/pub5.htm

24. Importance of the Peer Review Process http://www.alpsp.org/pub5.ppt

25. Importance of journal features

26. Importance of the peer review process

27. Factor Responses as authors Responses as readers Peer review 81 80 Gathering articles together to enable browsing of content 64 49 Selection of relevant and quality-controlled content 71 54 Content editing and improvement of articles 60 39 Language or copy editing 50 34 Checking of citations/adding links 46 28 Marketing (maximising visibility of journal) 44 20 Importance of publishers’ roles

28. Dissemination method Very important plus important categories Ranking Traditional print + electronic journal 91 1 Discipline-based electronic reprint archive 78 2 Traditional print journal 77 3 Traditional electronic-only journal 66 4 Institution-based electronic reprint archive 60 5 New forms of electronic-only journal 49 6 Discipline-based electronic preprint archive 44 7 Institution-based electronic preprint archive 33 8 Importance of future dissemination channels

29. http://www.update-software.com/Cochrane/MR000016.pdf

30. Cochrane Methodology Review • Despite its widespread use and costs, little hard evidence exists that peer review improves the quality of published biomedicalresearch. • There had never even been any consensus on its aims and that it would be more appropriate to refer to itas ‘competitivereview’. Caroline White, “Little Evidence for Effectiveness of Scientific Peer Review,” BMJ 326 (February 1, 2003): 241 http://bmj.com/cgi/reprint/326/7383/241/a.pdf

31. Cochrane Methodology Review • On the basis of the current evidence, ‘the practice of peer review is based on faith in its effects, rather than on facts,'state the authors, who call for large, government funded researchprogrammes to test the effectiveness of the [classic peer review] system and investigatepossiblealternatives. Caroline White, “Little Evidence for Effectiveness of Scientific Peer Review,” BMJ 326 (February 1, 2003): 241 http://bmj.com/cgi/reprint/326/7383/241/a.pdf

32. Cochrane Methodology Review • The use of peer-review is usually assumed to raise the quality of the end-product (i.e. the journal or scientific meeting) and to provide a mechanism for rational, fair and objective decision-making. However, these assumptions have rarely been tested. Tom O. Jefferson, Phil Alderson, Frank Davidoff, and Elizabeth Wager, Editorial Peer-review for Improving the Quality of Reports of Biomedical Studies. (Middle Way, Oxford: Update Software Ltd, 2003). http://www.update-software.com/Cochrane/MR000016.pdf

33. Cochrane Methodology Review • The available research has not clearly identified or assessed the impact of peer-review on the more important outcomes (importance, usefulness, relevance, and quality of published reports) • … [G]iven the widespread use of peer-review and its importance, it is surprising that so little is known of its effects Tom O. Jefferson, Phil Alderson,Frank Davidoff, and Elizabeth Wager, Editorial Peer-review for Improving the Quality of Reports of Biomedical Studies. (Middle Way, Oxford: Update Software Ltd, 2003). http://www.update-software.com/Cochrane/MR000016.pdf

34. FURTHERMORE … • 16%said that the referees would no longer be anonymous • 27%said that traditional peer review would be supplemented by post-publication commentary • 45%expected to see some changes in the peer-review system within the next five years Fytton Rowland, “The Peer-Review Process,” Learned Publishing 15 no. 4 (October 2002): 247-258. Report version: http://www.jisc.ac.uk/uploaded_documents/rowland.pdf