1 / 10

Dagstuhl Seminar

Dagstuhl Seminar: Reproducibility of Data-Oriented Experiments in e-Science January 25-29 2016 Dagstuhl, Germany. Dagstuhl Seminar. Dagstuhl Seminar.

lhe
Download Presentation

Dagstuhl Seminar

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Dagstuhl Seminar:Reproducibility of Data-Oriented Experiments in e-ScienceJanuary 25-29 2016Dagstuhl, Germany

  2. Dagstuhl Seminar

  3. Dagstuhl Seminar • Juliana Freire, Norbert Fuhr, and Andreas Rauber. Reproducibility of Data-Oriented Experiments in eScience. Dagstuhl Reports, 6(1), 2016. • http://drops.dagstuhl.de/opus/volltexte/2016/5817/ • Own experience:War Stories,SoA in CS, Tools, … • Working Groups

  4. Types of Reproducibility • The PRIMAD1 Model: which attributes can we “prime”? • Data • Parameters • Input data • Plattform • Implementation • Method • Research Objective • Actors • What do we gain by priming one or the other? [1] Juliana Freire, Norbert Fuhr, and Andreas Rauber. Reproducibility of Data-Oriented Experiments in eScience. DagstuhlReports, 6(1), 2016.http://drops.dagstuhl.de/opus/volltexte/2016/5817/

  5. Types of Reproducibility and Gains

  6. Reproducibility Papers • Consistency of results, not identity! • Reproducibility studies and papers • Not just re-running code / a virtual machine • When is a reproducibility paper worth the effort / worth being published? • Aim for reproducibility: for one’s own sake – and as Chairs of conference tracks, editor, reviewer, superviser, … • Review of reproducibility of submitted work (material provided) • Encouraging reproducibility studies • (Messages to stakeholders in Dagstuhl Report)

  7. Reproducibility Papers Transparency, openness, and reproducibility are vital features of science. Scientists embrace these features as disciplinary norms and values, and it follows that they should be integrated into daily research activities. These practices give confidence in the work; help research as a whole to be conducted at a higher standard and be undertaken more efficiently; provide verifiability and falsifiability; and encourage a community of mutual cooperation. They also lead to a valuable form of paper, namely, reports on evaluation and reproduction of prior work. Outcomes that others can build upon and use for their own research, whether a theoretical construct or a reproducible experimental result, form a foundation on which science can progress. Papers that are structured and presented in a manner that facilitates and encourages such post-publication evaluations benefit from increased impact, recognition, and citation rates. Experience in computing research has demonstrated that a range of straightforward mechanisms can be employed to encourage authors to produce reproducible work. These include: requiring an explicit commitment to an intended level of provision of reproducible materials as a routine part of each paper’s structure; requiring a detailed methods section; separating the refereeing of the paper’s scientific contribution and its technical process; and explicitly encouraging the creation and reuse of open resources (data, or code, or both). Message to Editors and Conference Chairs

  8. Reproducibility Papers The [insert name of journal/conference] encourages authors to provide their work in a way that enables reproduction of their outcomes. Just as you have benefited as an author from the work you cite in your paper, and the tools and resources that others have provided, your efforts will also assist the community, including your future collaborators, if you provide access to and understanding of the tools and resources that you have used and created while carrying out your project. We therefore [encourage/request that] authors include in their papers detailed explanations of how their work might be reproduced by others in the field, and to accompany their papers with links to data and source code if it is possible to do so. Authors can request separate reviewing of the reproducibility of their work, a category of publication that we explicitly acknowledge. In order to support these expectations authors are encouraged to include a detailed methods section in their paper that describes the techniques, tools, data resources, and code resources that enables readers to easily reproduce the work. Such a methods section is of greatest benefit to the reader when it is linked to materials stored in a trusted open repository, and these materials include illustrative or complete data, and code that can easily be re-used and understood. Message to Authors

  9. Reproducibility Papers • When is a Reproducibility paper worth being published?

  10. Dagstuhl Report • Juliana Freire, Norbert Fuhr, and Andreas Rauber. Reproducibility of Data-Oriented Experiments in eScience. Dagstuhl Reports, 6(1), 2016. • http://drops.dagstuhl.de/opus/volltexte/2016/5817/

More Related