1 / 25

Some Thoughts On Peer Review In The Global Internet Context

Some Thoughts On Peer Review In The Global Internet Context. Narayanan Komerath Daniel Guggenheim School of Aerospace Engineering Georgia Institute of Technology Atlanta. http://farm4.static.flickr.com/3112/3157622458_b19d039158.jpg. IMETI 2010, Orlando, FL June 2010.

lok
Download Presentation

Some Thoughts On Peer Review In The Global Internet Context

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Some Thoughts On Peer Review In The Global Internet Context Narayanan Komerath Daniel Guggenheim School of Aerospace Engineering Georgia Institute of Technology Atlanta http://farm4.static.flickr.com/3112/3157622458_b19d039158.jpg IMETI 2010, Orlando, FL June 2010

  2. “What is a scholarly or peer reviewed journal?” http://knowledgecenter.unr.edu/help/using/peer.aspx “Scholarly and professional journals feature articles written by researchers and practitioners in a particular subject area.” “The authors often have particular specialties. Peer groups of researchers, scholars and professionals within a specific discipline are the audience for scholarly literature.” “Well-accepted indicator of quality scholarship.” “Process by which an author's peers read a paper submitted for publication.” “A number of recognized researchers in the field will evaluate a manuscript and recommend its publication, revision, or rejection.” “Articles accepted for publication through a peer review process implicitly meet the discipline's expected standards of expertise.” “Articles selected by an editor or board. Standards of scholarship in such journals are often equal or comparable to those of peer-reviewed publications, although this is not always the case.”

  3. From West Virginia University Library “What is a Peer-Reviewed Journal? ..Sends articles submitted by authors to a group of experts in the field for review before publication. …The review process helps to ensure that research published in the journal is of high quality and contributes new information to the field. Characteristics of Peer-Reviewed Journals: Purpose is to share the results of original research with the rest of the scholarly world. Usually have a plain cover with lengthy and in-depth articles that contain charts, graphs, and tables. ..Minimal use of photographs and color. Any advertising is geared towards the subject discipline. Articles are written by researchers, scientists or scholars in the field. Authors cite their sources in footnotes or bibliographies, which are often extensive. Uses the technical terminology appropriate to the discipline and assumes that the reader will have a similar scholarly background.”

  4. http://undsci.berkeley.edu/images/us101/peerreview.gif

  5. http://www.whitehead.mit.edu/news/paradigm/spring_2008/img/peer_review.jpghttp://www.whitehead.mit.edu/news/paradigm/spring_2008/img/peer_review.jpg

  6. http://1.bp.blogspot.com

  7. Reviewer’s S.O.P. According to n.k. • Read the paper back and forth at least 3 times. Relate conclusions to what is shown. Put away for a couple of days and mull on the paper. • 2. Consider originality. Search for related prior work if necessary. • Follow author’s train of thought and check for reasonableness. • 4. Read carefully and note questions • 5. Compose review and pose questions • 6. Develop summary recommendations and ratings • 7. Get the review back to the editor in a timely manner. • If requested to review again, do so quickly. • A good peer review should engage the brains of the authors and the editors. • Takes time to do it right, but your professional standards are at stake. • How would you like someone like yourself as reviewer when you are the author?

  8. My Background 32 years, full-time researcher/ academic tenure track/ tenured professor in a major technological institution - Combustion diagnostics / propulsion experiments - Turbulent combustion/ fluid dynamics - DSP / Optical and other diagnostics development - Experimental aerodynamics / wind tunnel operations / vortex flows / combat aircraft / helicopters - Guiding 15+ engineering PhDs. - Institutional peer review panels. - Proposal and paper reviewer for national and international organizations ~ 250 publications, ~90 of them peer-reviewed. ~ 14 years internet / interdisciplinary / digital library / resource building & maintenance ~ 14 years in formally multidisciplinary endeavors ~ 12 years in Strategic / International / Security affairs ~ 8 years in social sciences interactions in community vs. academics

  9. Some Questions From the Cross-disciplinary Perspective

  10. “Killing” (or sneaking in) a Paper is Easy if the Editor is Asleep (or Worse) The dual role of quality control and motivation encounters conflicts of interest, especially when there are underlying competitive aspects and financial pressures. • Delay • Dismissive rejection, or glowing acceptance • Unusually harsh or mild comments • Demand for “additional work” • Delay following rebuttal • “Re-review” to think up new • excuses for rejection http://2.bp.blogspot.com

  11. So why have Peer Review? Thoughtful review and comments from interested and knowledgeable persons is more relevant than ever to motivating original thought, disciplined research and progress. • Help set in context of prior work • Help author improve thinking, calculation, presentation and writing: • Ask questions that may not have occurred to author • Catch errors • Provide opportunity for author to rebut/revise/explain • Increase utility to other readers through an open discussion of • important issues. What I have deleted is “Accept only High-Quality Papers”

  12. Observations on rejecting papers • Some process for rejection is essential, if only to encourage other authors. • High rejection rate is a low-quality metric of high quality. • Professional organization’s responsibility to not publish nonsense is outweighed by the risk of rejecting innovative work. • There are better ways ….. 

  13. Not possible in traditional model, to eliminate conflicts of interest. • Reviewers are anonymous, • and do not have to defend their (in)actions • Editors can delay publication • Willing reviewers are hard to find • Competent reviewers are harder to find • – and are usually competitors

  14. Evolving technology and habits of researchers, and the attendant fragmentation of time that many experts face, have driven traditional peer reviewed paper journals to the verge of irrelevance. • Adversarial / arrogant attitude (IP is author’s!) • Poor value addition by sloppy reviewing • Cost (“paper charges”) • Delay in review and publication • Poor visibility: Libraries cannot afford to subscribe to all, or most, journals • Misses the Google Searcher Audience • Citations will be more on papers that • are accessible! • Present generation does not follow linear or hierarchial methods of knowledge organization • “If u can’t b found on Google screens 1-3, u don’t exist” www.columbia.edu/cu/21stC/issue-1.1/peer.htm

  15. Casual adoption of technology in review systems poses its own problems, especially when used in reviewing innovative work • “Review Panels typically support the status quo” (priceless quote from an experienced authority from this conference!) • Adding technical convenience does not improve the amount or quality of time and thought devoted to review. http://francisthemulenews.files.wordpress.com/2009/04/dibujo20090425_peer_review_not_always_useful_for_editors.jpg

  16. Academic promotion/tenure processes are relying more on “prestige” and “percentile rankings in a field” metrics along with number counts of publications Fragmentation & proliferation of journals inhibits cross-disciplinary innovation. I reject this route as being destructive of original thought and cross-disciplinary knowledge dissemination. http://imvu.regnarts.com/news/wp-content/uploads/2008/06/peer-review-tears.png

  17. “Professor Komerath is one of the Top 3 Experts in the World in measuring pressure-velocity correlations in turbulent premixed flames using cross-spectral analysis of response-compensated Pitot microphone probes with ceramic tips and a coherence metric” (Great, but who else would want to join that club?) Interim solution adopted by specialist communities to the lack of competent “expert” reviewers in their narrow fields: Creation of ever‐tighter “peer” circles, leading to fragmentation of knowledge and the inability of any library to maintain a comprehensive journal base. Thus the most “prestigious” publications may be so exclusive that few can even find them.

  18. Scholarly publication and academic freedom vs. responsible community behavior Egregious examples of abuses in “religion” academia. Refusal of the “Academy” to publish well-informed rebuttals and exposures from lay practitioners destroys academy’s credibility. “Journals feature articles written by researchers and practitioners in a particular subject area…Peer groups of researchers, scholars and professionals within a specific discipline are the audience for scholarly literature.” Lesson: Scholarly communities that do not welcome and facilitate well-reasoned disagreement from “lay persons”, are not worthy of respect

  19. Technological capabilities and evolving trends in capturing knowledge from prior work • Internet Search Engines • “Google Books” and “Google Scholar” • Delivery to personal communication devices (Iphone, Ipad) • Global access • ModeratedDiscussion forum • “Twitter”

  20. Open, two-stage peer-review From Nature (2006) | doi 10.1038/nature04988 Journal of Atmospheric Chemistry and Physics Stage 1: Rapid pre-screening (access review) and immediate publication as “discussion paper” on website. Public interactive discussion for 8 weeks, with comments of designated reviewers & others are published alongside the discussion paper. Reviewers sign or remain anonymous, but comments by other scientists must be signed. Stage 2: traditional manuscript revision and peer review. If accepted, final papers are published in the main journal. Every discussion paper and interactive comment remains permanently archived and individually citable.

  21. Peer review model for cross-disciplinary innovation

  22. Preliminary empirical observations • Social-media forum to motivate experts to participate: • Examples: • Strategic affairs discussion forum • Space Solar Power forum • Discussions on major news items (Gulf of Mexico oil leak) • Most engineering journals are not likely to generate many comments from readers. • Getting authors to submit articles, requires recognition of quality control processes and archival reliability of “journal” • Strong moderation is required • Author ego is a problem with many authors unused to social • media • Notes: • J. Atmospheric Science reports excellent impact factor. • “Deters submission of low-quality papers: low rejection rate”

  23. Conclusions • Purpose of peer review should be carefully re-emphasized • Abuses cannot be avoided with traditional paper journal review system • Swift & unobstructed publication essential. • Open but moderated, interactive discussion provides good quality • Archived reviews/rebuttal is a good experiment • Motivating participation requires thought and resources http://www.the-funneled-web.com/images_2008/PeerReview-ACS.gif

  24. ACKNOWLEDGEMENTS The work reported in this paper was made possible by resources being developed for the “EXTROVERT” cross-disciplinary learning project under NASA Grant NNX09AF67G S01. Mr. Anthony Springer is the Technical Monitor. http://docinthemachine.com/wordpress/wp-content/uploads/2007/02/gspz0625.jpg

More Related