nikolaus kriegeskorte mrc cognition and brain sciences unit n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Nikolaus Kriegeskorte MRC Cognition and Brain Sciences Unit PowerPoint Presentation
Download Presentation
Nikolaus Kriegeskorte MRC Cognition and Brain Sciences Unit

Loading in 2 Seconds...

play fullscreen
1 / 30

Nikolaus Kriegeskorte MRC Cognition and Brain Sciences Unit - PowerPoint PPT Presentation


  • 195 Views
  • Uploaded on

Nikolaus Kriegeskorte MRC Cognition and Brain Sciences Unit. Overview. What’s right and wrong with the current system? What features define the future system of scientific publishing? How can we transition to the future system? . What’s right and wrong with the current system?.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Nikolaus Kriegeskorte MRC Cognition and Brain Sciences Unit' - luthando


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
nikolaus kriegeskorte mrc cognition and brain sciences unit
Nikolaus Kriegeskorte

MRC Cognition and Brain Sciences Unit

overview
Overview
  • What’s right and wrong with the current system?
  • What features define the future system of scientific publishing?
  • How can we transition to the future system?
positive functions of the current system
Positive functionsof the current system
  • Journal prestige as an evaluative signal that helps select papers to read
  • Beautiful professionallayout for papers
positive functions of the current system1
Positive functionsof the current system
  • Journal prestige as an evaluative signal that helps select papers to read
  • Beautiful professionallayout for papers
problems with the current system
Problems with the current system

open access

(OA)

  • Not generally open access
  • Long publication delays
  • Excessive costs
  • Journal prestige is the only immediate evaluative signal for choosing papers
  • Intransparent and unsatisfactory paper evaluation process

open post-publication

peer review

(OPR)

what s wrong with journal prestige as an evaluative signal
What’s wrong with journal prestige as an evaluative signal?
  • reflects journals, not individual papers
  • too weakly correlated with paper quality
  • 3-4 reviews provide too noisy an evaluative signal to justify the de-facto influence of high-impact publications on
    • the attention of the scientific community
    • public policy
    • science funding
    • individual scientists’ careers
  • no continuous quality ratings on multiple scales
some current developments that point in the right direction
Some current developmentsthat point in the right direction
  • arXivopen-access paper repository
  • PLoS, PLoS ONEopen-access, traditional journals, inviting postpublication commentary
  • Faculty of 1000commercial source for alternative paper evaluations from selected experts
  • ResearchBlogging.orgcollects blog responses to peer-reviewed papers
  • Frontiers journal familycombineopen access and democratic postpublication selection for greater visibility
two features
Two features

access to the literature

open access (OA)

open post-publicationpeer review (OPR)

evaluation of the literature

current

re

vie

ws

Current

10 years later

citations

1 year later

secret

peer

review

research,

writing

reception,

citation

unpublished

paper

published

paper

journal prestige

future
Future

review

review

review

review

review

review

review

review

review

open peer review

and reception

(merged process)

instantly

published

paper

research,

writing

citing paper

citing paper

citing paper

time

0 months

3 months

1-10 years

open post publication peer review1
Open post-publication peer review
  • open to post: anyone can instantly post a review
  • open to access: anyone can instantly access any review
  • each review is permanently linked to the paper
  • reviews are digitally authenticatedat different levels
    • signed reviews(author authenticated and publicly identified)
    • unsigned reviews by authenticated group members (e.g. member of professional group such as SfN)
    • unauthenticated
open post publication peer review2
Open post-publication peer review
  • In order for reviewing to be open, it has to be post-publication.
  • Review and reception are an integrated ongoing process after publication.
  • Reviews do not decide about or delay publication.
open post publication peer review3
Open post-publication peer review
  • Peer review is not perfect, but it is the best evaluation mechanism we have.
  • The most serious drawbacks of peer review derive from the fact that it is currently a secret process.
the nature of a review
Current

secret communication to authors and editors

decides about publication

reviewer’s motivation

selfless: scientific objectivity

selfish: science politics

a weak argument can kill a paper

Future

open letter to the community

evaluates published work

reviewer’s motivation

selfless: scientific objectivity

selfish: looking smart and objective in public

an argument is as powerful as it is compelling

The nature of a review

Questions?

slide18

peer-to-peer editing

  • authors ask a senior scientist to edit the paper
  • editor chooses 3 reviewers and asks them to openly review the paper
  • editor is named on the paper

published, author authenticated,

unreviewed paper

published, author authenticated,

reviews

(signed or unsigned)

slide19

peer-to-peer editing

  • authors ask a senior scientist to edit the paper
  • editor chooses 3 reviewers and asks them to openly review the paper
  • editor is named on the paper
  • review
  • text
  • numerical ratings
    • justification of claims
    • importance
    • originality

published, author authenticated,

reviewed paper

slide21

paper score: 86 %ile

  • paper evaluation function
  • (PEF)
  • arbitrary function that scores papers based on the available meta-information
  • simplest case: weighted average of review ratings
  • individuals or groups can define PEFs to prioritize the literature according to their needs

originality

importance

  • justification of claims
slide22

paper score: 86 %ile

originality

importance

  • justification of claims
slide23

paper score: 94 %ile

originality

importance

  • justification of claims
slide24

paper score: 98 %ile

ready to be

showcased in

Science or Nature

originality

importance

  • justification of claims

Questions?

paper evaluation functions pefs
Paper evaluation functions (PEFs)
  • weighted average of reviewer ratings
    • weighted by dimension
    • weighted by reviewer information
      • reviewer expertise factor
      • reviewer time investment
      • reviewer independence of authors
  • optionally normalized by error margin
    • like t values: more reviews  higher score
  • constant competition of alternative PEF “lenses” onto the literature
  • arbitrary algorithms possible
papernet
“Papernet”

Bayesian belief

propagation

pefs provide multiple lenses onto the literature
PEFs provide multiple“lenses onto the literature”

individual

scientist Joe

MRC Cognition and Brain Sciences Unit

PEF

Society for

Neuroscience

Public policythink tank

constantly evolving

and competing

plurality of PEFs

“ungamable”

evaluation system

Science

Magazine

Questions?

transitioning
Transitioning

Public website for open posting of digitally authenticated post-publication reviews

  • Pubmed-scale investment to develop collaborative software and install the system (Public funding. Involve Google?)
  • Papers published in the current system can be reviewed using the new system
  • Original reviewers can publish the reviews they wrote for a traditional journal
  • This provides a platform for continual online evaluation of the scientific literature
  • Tipping point reached when the evaluative signal becomes more reliable than journal prestige
  • Papers can then be published instantly without journals – as authenticated digital documents (like the reviews)
what can we do now
What can we do now?
  • Publish the reviews we write and receive online

Useful activism, not the solution.

  • View the problem as a grand challenge to cognitive and brain science

How to organize the collective cognitionof the scientific community?

  • Imagine how we want it to work,then talk and write about it…

futureofscipub.wordpress.com