Are We All Publishing in the Journal of Irreproducible Results?

1. Are We All Publishing in the Journal of Irreproducible Results? Jeff Solka Ph.D. BINF 705 11/5/13 BINF705 FALL 2013 REPRODUCING SCIENTIFIC RESULTS

2. True Confessions I Drank the Kool-Aid Ph.D., Computational Sciences and Informatics (Computational Statistics), George Mason University, May 1995 Thesis: Matching Model Information Content to Data Information (Recipient of the Outstanding Dissertation in Statistics Award) M.S., Physics, Virginia Polytechnic Institute and State University, May 1989 M.S., Mathematics, James Madison University, May 1981 B.S., Mathematics and Chemistry, James Madison University, May 1978 BINF705 FALL 2013 REPRODUCING SCIENTIFIC RESULTS

3. Humor Let’s Start on a Light Note • Yes there really is a Journal of Irreproducible Results • http://www.jir.com BINF705 FALL 2013 REPRODUCING SCIENTIFIC RESULTS

4. Early Warnings Who Said This? Everyone should know that most cancer research is largely a fraud, and that the major cancer research organizations are derelict in their duties to the people who support them. BINF705 FALL 2013 REPRODUCING SCIENTIFIC RESULTS

5. Early Warnings Linus Carl Pauling http://www.nobelprize.org/nobel_prizes/chemistry/laureates/1954/pauling-facts.html Nobel Prize in Chemistry 1954 Prize motivation: "for his research into the nature of the chemical bond and its application to the elucidation of the structure of complex substances" BINF705 FALL 2013 REPRODUCING SCIENTIFIC RESULTS

6. It Happens in Physics The Millikan Oil Drop Experiment Robert A. Millikan (1868-1953) www2.chem.umd.edu/Courses/claroline/document/.../Lecture_5.ppt BINF705 FALL 2013 REPRODUCING SCIENTIFIC RESULTS

7. It Happens in Physics The Millikan Oil Drop Experiment1909-1913 Robert A. Millikan1923 Nobel Prize + + + + + + + + + + + + ++ qE _ mg _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ qE = weight – buoyant force = 4/3  r3 g ( oil - air) r obtained from field-free fall-time (Stokes’ Law): 4/3  r3 g ( oil - air) = 6   r v www2.chem.umd.edu/Courses/claroline/document/.../Lecture_5.ppt BINF705 FALL 2013 REPRODUCING SCIENTIFIC RESULTS

8. It Happens in Physics The Millikan Oil Drop Experiment • G. Holton (Harvard): Millikan used only the data that supported his assumption of integral charge on electron, but wrote that he published all data. • A. Franklin: Omitted drops were not bad, just not as precise. • D. Goodstein (CalTech): Defended Millikan as “using his scientific intuition;” followed standards of his time www2.chem.umd.edu/Courses/claroline/document/.../Lecture_5.ppt BINF705 FALL 2013 REPRODUCING SCIENTIFIC RESULTS

9. It Happens in Physics Did Millikan Go Wrong? • Is the sampling representative? • Are there unverified assumptions? • Were the right questions asked? • Are there systematic/subjective errors? • Is it reproducible? • Are there alternative explanations? What could/should Millikan have done? www2.chem.umd.edu/Courses/claroline/document/.../Lecture_5.ppt BINF705 FALL 2013 REPRODUCING SCIENTIFIC RESULTS

10. It Happens in Biology Perhaps a Result Closer to Home http://library.uthsc.edu/exhibits/stamps/StampsLarge/MendelAustria.gif http://www.nih.gov/about/director/ebiomed/mendel.htm BINF705 FALL 2013 REPRODUCING SCIENTIFIC RESULTS

11. It Happens in Medicine A Popular Medical Theory Out the Window http://www.scientificamerican.com/article.cfm?id=is-free-radical-theory-of-aging-dead BINF705 FALL 2013 REPRODUCING SCIENTIFIC RESULTS

12. What Should Prevent This? What are the Cross Checks Against Error or Deceit? www2.chem.umd.edu/Courses/claroline/document/.../Lecture_5.ppt (1) Peer review = review before work is done ->proposal (2) Referee system = review before work is published ->paper (3) Replication = test of repeatability after work is published BINF705 FALL 2013 REPRODUCING SCIENTIFIC RESULTS

13. Why is This Hard? Replication www2.chem.umd.edu/Courses/claroline/document/.../Lecture_5.ppt Why is replication hard? A. Recipe incomplete B. Resources unavailable Research costs money! C. Motivation lacking No credit for second experiment Not interesting enough D. Original data lacking BINF705 FALL 2013 REPRODUCING SCIENTIFIC RESULTS

14. Some Humor An Example from Chemistry http://pipeline.corante.com/archives/2009/02/27/your_paper_is_a_sack_of_raving_nonsense_thank_you.php BINF705 FALL 2013 REPRODUCING SCIENTIFIC RESULTS

15. Computer Science is Not Safe Reproducibility in Scientific Computing Appearing in http://www.davidhbailey.com/dhbtalks/dhb-reproducibility.pdf A December 2012 workshop on reproducibility in computing, held at Brown University in Rhode Island, USA, noted that Science is built upon the foundations of theory and experiment validated andimproved through open, transparent communication. With the increasingly central role of computation in scientic discovery this means communicatingall details of the computations needed for others to replicate the experiment. ... The “reproducible research" movement recognizes that traditionalscienticresearch and publication practices now fall short of this ideal, and encourages all those involved in the production of computational science ... to facilitate and practice really reproducible research. V. Stodden, D.H. Bailey, J. Borwein, R.J. LeVeque, W. Rider and W. Stein, “Setting the default to reproducible: Reproducibility in computational and experimental mathematics," http://www.davidhbailey.com/dhbpapers/icerm-report.pdf. BINF705 FALL 2013 REPRODUCING SCIENTIFIC RESULTS

16. Ok We are not Kidding Houston We Have a Problem http://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble BINF705 FALL 2013 REPRODUCING SCIENTIFIC RESULTS

17. It Hits Home for Us A Problem at the Highest Levels • http://www.nature.com/nature/focus/reproducibility/ • What about our drugs? • http://blogs.nature.com/news/2011/09/reliability_of_new_drug_target.html BINF705 FALL 2013 REPRODUCING SCIENTIFIC RESULTS

18. Is There Any Hope? Some Questions • How can we stop this? • How can we detect this? • Something is claimed by the scientific community as true but it actually is false. • How can we detect the reverse problem? • Something is claimed by the scientific community as false but it actually is true. BINF705 FALL 2013 REPRODUCING SCIENTIFIC RESULTS

19. Stopping It How Can We Stop This? • Enforcing duplication as a requirement for grant applications • Enforcing publication of secret sauce steps for independent verification • May require NDAs for the reviewers • Enforcing release of code and data • Enforcing stringent requirements in the case of pharmaceutical testing BINF705 FALL 2013 REPRODUCING SCIENTIFIC RESULTS

20. Detecting It How Can We Detect It? Is it detectable? Statistical arguments based on power. Detection may be a function of discipline. Can we argue based on the reputation of the investigator, the number of authors on the paper, … Other ideas BINF705 FALL 2013 REPRODUCING SCIENTIFIC RESULTS

21. Detecting It How can we detect the reverse problem? What nuggets may exist as unpublished manuscripts? The more existing theories that a paper contradicts the more revolutionary it is. The more existing theories that a paper contradicts the more likely that it is bunk. Is cold fusion or low energy nuclear reaction (LENR) true? Can one truly extract energy from a vacuum. Ideas??? BINF705 FALL 2013 REPRODUCING SCIENTIFIC RESULTS

22. Please No Tomatoes One Solution http://xkcd.com/435/ BINF705 FALL 2013 REPRODUCING SCIENTIFIC RESULTS