Peer review in the google age
1 / 21

Peer Review in the Google Age - PowerPoint PPT Presentation

  • Uploaded on

Peer Review in the Google Age. Is technology changing the way science is done and evaluated? Peggy Dominy & Jay Bhatt. When did peer review start?.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about ' Peer Review in the Google Age' - colorado-alexander

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Peer review in the google age

Peer Review in the Google Age

Is technology changing the way science is done and evaluated?

Peggy Dominy & Jay Bhatt

When did peer review start
When did peer review start?

Some would say that “Peer Review” goes back as far as the 17th century, when it was known as “The Inquisition of the Holy Roman and Catholic Church”. Scholars’ works were examined for any hints of “heresy”.



Peer review in modern times
Peer review in “modern times”

Peer review (known as refereeing in some academic fields) is used in:

  • Publication process

  • Awarding of funding for research

  • Patents

  • Standards

    Each of these involve slightly different practices, but ultimately colleagues are evaluating each other.

Process of peer review
Process of peer review

Once a paper has been submitted for consideration of publication, the editor will select 1-2 or 3 scholars from a pool of volunteers to read and evaluate the paper.

Typically it is a double blind process: the reviewers do not know who the author is and the author does not know who the reviewers are. That way only the merits of the paper are evaluated.

Process of peer review cont
Process of peer review (cont.)

The reviewers (within a reasonable time period) respond with their comments which are then forwarded to the author for response to or compliance with reviewer’s suggestions. In the days before the “Internet”, this added weeks (months?) to the publishing process.

Today, moving text back and forth electronically has dramatically accelerated the process, though the imposition on an overburden volunteer researcher has not changed much.

Why do peer review
Why do peer review?

  • Filter

    • More papers submitted than could be “printed”

    • Eliminate “bad” science, pseudo-science, harmful science...

  • Aura of “quality” (only the best gets in)

  • Collegial stamp of approval

  • Professional obligation to the principles of one’s discipline

So what s the problem
So, what’s the problem?

  • Famous papers that were published and did NOT get peer reviewed:

    • Watson & Crick’s 1951 paper on the structure of DNA in Nature

    • Abdus Salam’s paper “Weak and electromagnetic interactions” (1968). Led to Nobel Prize

    • Alan Sokal’s “Transgressing the Boundaries...” in 1996 turned out to be a hoax. Now known as the Sokal Affair.

  • Famous papers that were published and passed peer review that later proved to be fraudulent:

    • Jan Hendrik Schon (Bell Labs) submitted and passed peer review 15 papers published in Science and Nature (1998-2001) found to be fraudulent.

    • Igor and Grichka Bogdanov 1999 & 2002 published papers in theoretical physics believed by many to be jargon-rich nonsense.

  • Famous papers that got rejected that later turned out to be seminal works:

    • Krebs & Johnson’s 1937 paper on the role of citric acid on metabolism was rejected by Nature as being of “insufficient importance”, was eventually published in the Dutch journal Enzymologia. This discovery, now known as the Krebs Cycle, was recognized with a Nobel prize in 1953.

    • Black & Scholes 1973 paper on “the pricing of options and corporate liabilities”, rejected many times, was eventually published at the intersession of Merton Miller to get it accepted by the Journal of Political Economy. This work led to the Nobel Prize.

Who s worried about peer review
Who’s worried about peer review?

Using Google Scholar search engine, a search using “peer review” and limiting to subject areas (as defined by Google folks):

7,220 Biology, Life Sciences, and Environmental Science

12,000 Business, Administration, Finance, and Economics

12,500 Chemistry and Materials Science

157,000 Engineering, Computer Science, and Mathematics

15,300 Medicine, Pharmacology, and Veterinary Science

35,900 Physics, Astronomy, and Planetary Science

35,100 Social Sciences, Arts, and Humanities

Two recent articles
Two Recent Articles...

  • “Is Peer Review Broken?"

    by Alison McCookThe Scientist, vol 20 (2), Feb 2006, pg 26.

    Submissions are up, reviewers are overtaxed, and authors are lodging complaint after complaint about the process at top-tier journals. What's wrong with peer review?

  • "Journal lays bare remarks from peer reviewers"

    by Emma MarrisNature, vol. 439, 9 February 2006, page 642

    Cloak of anonymity shed by new publication. Editors of a journal launched this week are out to revolutionize peer review. By publishing signed reviews alongside papers, they hope to make the process more transparent and improve the quality of the articles.

So what s changed
So, what’s changed?

  • Papers can be “published” on the Web without the constraints (peer reviewers and editors) of traditional “journals”.

  • Papers can be “published” on the Web with the constraints (peer reviewers and editors) of traditional “journals”.

    Is this a good thing or a bad thing?

Without constraints
Without “constraints”…

  • Wild and Open—no gatekeepers—no censors

  • “Where’s the beef?”—will the good stuff percolate to the top?

  • Discernment—how do you know if it’s good?

  • Who’s the authority?

With constraints
With “constraints”…

  • American Physical Society

  • BioMed Central

  • Blackwell

  • BMJ

  • Cambridge University Press

  • IEEE

  • MIT Press

  • ACM

  • AIP

  • IOP

  • American Mathematical Society

  • Public Library of Science

  • Royal Society of Chemistry

  • Kluwer

  • PNAS

  • And many more

  • The good stuff is vetted

  • Scholarship is monitored and maintained

  • Exposure beyond the internet?

  • Publishers “sharing” content with the internet (a short list):

Peer review
Peer Review?

  • It’s not perfect—grist for a lot of mills

  • The Web has made it less of a obstacle to access

  • Different disciplines have different perspectives—different issues

  • Pedagogical yardstick for students

Peer review on the internet
Peer review on the Internet

  • Using emailemail based online peer reviewSee Peer Review of Scholarly Publications in Health,

    Online Manuscript Peer Review and Tracking Systems and Physics of Plasmas Online Manuscript Submission and Peer Review

  • Discussion approachbetter interaction among authors, reviewers and the editorial bodyJIME – Open Peer Review Process

  • WikisImmense potential to conduct peer review

  • Blogspost publication commentsSee Article Note: On Blogging as Tool, but Really About Using RSS

Access to scientific literature
Access to Scientific Literature

  • Author home pages linking their papers Google finds them.

  • Institutional RepositoriesProvide access to faculty authored research papers. See Publisher Policies that shows listing of which publishers allow either publisher or post print version on IRs

  • Indexed by Google; increases visibility of scholarly material

Access to scientific literature1
Access to Scientific Literature

  • SHERPA: Securing a Hybrid Environment for Research Preservation and Access. It is developing open-access institutional repositories in a number of research universities to disseminate research findings worldwide

  • Preserving EPrints:Scaling the Preservation Mountain

  • DSpace at Drexel

  • University of Pennsylvania Institutional Repository

  • Institutional Repositories are increasing and hence open access to scholarly literature increasing

Global benefits
Global benefits

  • Worldwide increase in access to scientific literature

  • Increased opportunities for collaboration among experts worldwide

  • Increased speed to disseminate scientific literature with electronic communities

  • More informal peer reviews

  • Quality needs to be maintained

  • See Scholarly Electronic Publishing Weblog

A very very brief bibliography
A (very, very) Brief Bibliography

  • Chubin, D. E., & Hackett, E. J. (1990). Peerless science: Peer review and U. S. science policy. Albany: State University of New York Press.

  • Davenas, E., Beauvais, F., Amara, J., Oberbaum, M., Robinzon, B., & Miadonnai, A. et al. (1988). Human basophil degranulation triggered by very dilute. Nature, 333(6176), 816-818. (published with editorial reservation on validity)

  • Emch, A. (1937). Rejected papers of three famous mathematicians. National Mathematics Magazine, 11(4), 186-189.

  • Garfield, E. (1993). Essays on refereeing and peer review. Retrieved 2/22, 2006 from

  • Godlee, F., Gale, C. R., & Martyn, C. N. (1998). Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: A randomized controlled trial. JAMA, 280(3), 237.

  • Harnad, S. (2000). The invisible hand of peer review. Exploit Interactive(5), February 15, 2006

  • Harnad, S. (1996). Implementing peer review on the net: Scientific quality control in scholarly electronic journals. In R. Peek, & G. Newby (Eds.), Scholarly publication: The electronic frontier (pp. 103). Cambridge MA: MIT Press.

A very very brief bibliography1
A (very, very) Brief Bibliography

  • Harnad, S., & Hemus, M. (1997). ALL-OR-NONE: NO STABLE HYBRID OR HALF-WAY SOLUTIONS FOR LAUNCHING THE LEARNED PERIODICAL LITERATURE INTO THE POSTGUTENBERG GALAXY. In I. Butterworth (Ed.), The impact of electronic publishing on the academic community (pp. 18). London: Portland Press.

  • Horrobin, D. F. (1990). The philosophical basis of peer review and the suppression of innovation. JAMA, 263(10), 1438-1441.

  • Judson, H. F. (1994). Structural transformations of the sciences and the end of peer review. Second international congress on biomedical peer review and scientific publication, Chicago, JAMA 272, 92-94.

  • Justice, A. C., Cho, M. K., Winker, M. A., Berlin, J. A., Rennie, D., & and the PEER Investigators. (1998). Does masking author identity improve peer review quality?: A randomized controlled trial. JAMA, 280(3), 240-242.

  • Kassirer, J. P., & Campion, E. W. (1994). Peer review: Crude and understudied, but indispensable. Second international congress on biomedical peer review and scientific publication, Chicago, JAMA, 272 96-97.

A very very brief bibliography2
A (very, very) Brief Bibliography

  • Mahoney, M. J. E. -. (1977). Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research, 1(2), 161-175.

  • McNutt, R. A., Evans, A. T., Fletcher, R. H., & Fletcher, S. W. (1990). The effects of blinding on the quality of peer review. A randomized trial. JAMA, 263(10), 1371-1376.

  • Moller, A. P., & Jennions, M. D. (2001). Testing and adjusting for publication bias. Trends in Ecology & Evolution, 16(10), 580.

  • Scaria, V. (2003). Peer review of scholarly communication in health: Perspectives in the internet age. Internet Health, Journal of Research, Application, Communications & Ethics, 2(6)

  • Scaria, V. (2003). Scholarly communication in biomedical sciences, open access and the developing world. Internet Health, Journal of Research, Application, Communication & Ethics, 1(1)

  • Wenneras, C., & Wold, A. (1997). Nepotism and sexism in peer-review. Nature, 387(6631), 341-343.

  • Wikipedia. (2006). Peer review. Retrieved February 20, 2006, 2006 from

A very very brief bibliography3
A (very, very) Brief Bibliography

  • Wilkinson, S. L. (1998). Electronic publishing takes journals into a new realm; publications slip off restrictions of print world and carve out a unique identity Chemical & Engineering News, 76

  • JAMA Peer Review Theme Issues

    Containing abstracts and articles from the Fourth, Third, and Second Peer Review Congresses.

    June 5, 2002 Issue

    July 15, 1998 Issue

    July 13, 1994 Issue

  • International Congress on Peer Review and Biomedical Publication

    There have been 5 since 1991