1 / 21

Reproducibility Project: Cancer Biology

Reproducibility Project: Cancer Biology. Barriers to Replicability in the Process of Research Tim Errington Center for Open Science. Begley & Ellis, 2012. RP:P Key Features Systematic selection Preregistration Completely open Materials and feedback from original authors.

davidmsimmons
Download Presentation

Reproducibility Project: Cancer Biology

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Reproducibility Project: Cancer Biology Barriers to Replicability in the Process of Research Tim Errington Center for Open Science

  2. Begley & Ellis, 2012

  3. RP:P Key Features • Systematic selection • Preregistration • Completely open • Materials and feedback from original authors Open Science Collaboration, 2015

  4. Reproducibility Project: Cancer Biology Systematic sampling of 51 high-impact preclinical cancer biology papers Completely open prereg, process, data, materials: https://osf.io/e81xl/ Registered Reports with eLife Stage 1 review of protocols: Precommitment by researchers, reviewers, editor Stage 2 review of results: Confirming adherence to precommitments Primary Outcomes Meta-analytic summary of statistical outcomes (not shown today) Summary of process challenges for testing replicability

  5. Sampling Design 1st Review Data Collection 2nd Review Published Eligible if published 2010-2012 and had “cancer biology” keywords Excluded genomics, proteomics, high-throughput assays Selected highest impact by citations and readership 51 papers included 33 Nature family, 11 Cell family, 4 Science family, 3 others

  6. Sampling Design 1st Review Data Collection 2nd Review Published Top-Level Outcome Started Design Phase: 51 papers, 197 experiments Mean = 3.9, STD = 1.9; Range 1 to 11 Finished Design Phase: 36 papers, 114 experiments Mean = 3.2, STD = 1.6; Range 1 to 8

  7. Sampling Design 1st Review Data Collection 2nd Review Published Question: For how many papers/experiments was all data available in the paper or a repository? 0 of 51 papers (0%) 3 of 197 experiments (2%)

  8. Sampling Design 1st Review Data Collection 2nd Review Published Question: After contacting authors, how much original data was shared?

  9. Sampling Design 1st Review Data Collection 2nd Review Published Question: For how many papers/experiments could we design a full protocol based on the paper and no input from the original author(s)? 0 of 51 papers (0%) 0 of 197 experiments (0%)

  10. Sampling Design 1st Review Data Collection 2nd Review Published

  11. Sampling Design 1st Review Data Collection 2nd Review Published

  12. Sampling Design 1st Review Data Collection 2nd Review Published

  13. Sampling Design 1st Review Data Collection 2nd Review Published Resource Challenges in Assessing Replicability • Accessing materials • Recreating materials • Design adjustments • Unexpected events in data collection Illustration: Estimated cost per paper over time • Start: $25,000 • Stage 1 review: $36,429 • Data collection: $42,016 • Final: $54,442

  14. Sampling Design 1st Review Data Collection 2nd Review Published Question: How did the 32 Registered Reports fare in the Stage 1 review process?

  15. Sampling Design 1st Review Data Collection 2nd Review Published Top-Level Outcome Started Collection Phase: 29 papers, 89 experiments Finished Collection Phase: 23 papers, 52 experiments • 6 papers = no experiments • 5 papers = some experiments • 18 papers = all experiments

  16. Sampling Design 1st Review Data Collection 2nd Review Published

  17. Sampling Design 1st Review Data Collection 2nd Review Published

  18. Sampling Design 1st Review Data Collection 2nd Review Published Case studies: • Replication of Delmore et al., 2011 (Cell). Published replication: Aird et al., 2017 (eLife) • Replication of Kang et al., 2011(Nature). Published replication: TBD

  19. Sampling Design 1st Review Data Collection 2nd Review Published Question: How did the 18 Final Reports fare in the Stage 2 review process?

  20. Sampling Design 1st Review Data Collection 2nd Review Published Final summary Started: 51 papers, 197 experiments Finished Papers: 23 (some) / 18 (all) / 17 (accepted*) Experiments: 52 total Coming next: • Full report on process outcomes • Full report on meta-analysis of statistical outcomes

  21. Conclusion Testing replicability is impeded by process factors… • Lack of transparency and accessibility of methodology • Lack of sharing of original data • Lack of communication for obtaining needed information • Unexpected challenges with protocols during data collection • Resource challenges (cost, time, uncertainty) inflated by all the above Questions? tim@cos.io or nosek@cos.io

More Related