Reproducible Research

1. Reproducible Research Jonathon LeFaive, University of Michigan Big Data Summer Institute June 18, 2019

2. Replicating vs. Reproducing 2019 Big Data Summer Institute

3. Replication • “Replication is the ultimate standard by which scientific claims are judged”—Roger Peng • Come to same conclusions with different data/protocols • Replicable = results are consistent and reliable 2019 Big Data Summer Institute

4. Reproduction • “An attainable minimum standard for assessing the value of scientific claims, particularly when full independent replication of a study is not feasible” –Roger Peng • Obtain same results with same data and same protocols • Reproducible = methods are consistent and reliable 2019 Big Data Summer Institute

5. Reproducible != True 2019 Big Data Summer Institute

6. Motivation for reproducibility 2019 Big Data Summer Institute

7. The reproducibility crisis 2019 Big Data Summer Institute

8. Why make my work reproducible? • Transparency • Save time • Keep track of complex projects http://science.sciencemag.org/content/334/6060/1226.full • Higher impact • More efficient collaboration 2019 Big Data Summer Institute

9. The Reproducibility Toolkit • Well-organized projects • Documentation • Version control • Capturing the computational environment • Automation 2019 Big Data Summer Institute

10. Organizing your projects 2019 Big Data Summer Institute

11. Exercise: Evaluate the structure of past projects • On your own: • Find a project you have worked on in the past • Sketch out the directory structure • (If applicable) sketch out the flow of information • Write down any naming conventions you used for files/folders • With a partner: • A collaborator wants to reproduce your project—what instructions would you give them? • What would you change about your organizational structure to make this easier? http://pgbovine.net/research-directory-structure.htm 2019 Big Data Summer Institute

12. Challenges of organization • Developing an intuitive directory structure • Coming up with good names for things • How long/when/where to keep intermediate data files? • Dealing with clutter of old stuff/temp files • Keeping track of dependencies • Managing backups/previous versions • Collaborating effectively with shared project content • Different local versions, determining structure of shared directory, content spread across computers, cloud apps, etc. 2019 Big Data Summer Institute

13. Tips for organizing your projects • Develop your own system and be consistent! • Think about where to put files you haven’t yet created • Separate raw from processed data • Separate code from data • Use file shortcuts to avoid unnecessary duplicates • File/folder names and paths should be self-explanatory • Code as verbs, data as nouns 2019 Big Data Summer Institute

14. Documentation 2019 Big Data Summer Institute

15. Documentation techniques • Including README files in project directories • Source code comments • Self-documenting code • Literate programming 2019 Big Data Summer Institute

16. README project-name/+- data/| +- README.md| +- genotypes.data| +- phenotypes.data+- scripts/| +- README.md| +- normalize-input.sh| +- run-association.sh+- README.md 2019 Big Data Summer Institute

17. Documenting code n = (icnt + gs - 1) / gs; uintceil_divide(uintx, uinty){return (x + y - 1) / y;}group_count = ceil_divide(item_count, max_group_size); /* Calculatesnumber of groupsbyceildividingitemcountbymaxgroupsize. */group_count = (item_count + max_group_size - 1) / max_group_size; group_count = (item_count + max_group_size - 1) / max_group_size; 2019 Big Data Summer Institute

18. Doxygen #' Adds together two numbers.#' #' @param x Left hand number.#' @param y Right hand number.#' @return The sum of x and y.#' @examples#' add(1, 1)#' add(10, 1)add <- function(x, y) {x + y} 2019 Big Data Summer Institute

19. “Literate programming is a methodology that combines a programming language with a documentation language, thereby making programs more robust, more portable, more easily maintained, and arguably more fun to write than programs that are written only in a high-level language.” --Donald Knuth 2019 Big Data Summer Institute

20. Literate programming • Literate programming allows you to encapsulate & share every aspect of your analysis in an interactive and descriptive way: • Data provenance • Dependencies • Code • Documentation • Tables/Figures/References etc. http://www.datacarpentry.org/rr-literate-programming/02-literate-programming/ 2019 Big Data Summer Institute

21. R Markdown • Literate programming language implemented in RStudio • Integrates standard R code with Markdown text formatting (+other stuff!) • knitr: Typesetting package to render your R Markdown document as a web page, PDF, Word document, etc. 2019 Big Data Summer Institute

22. Structure of R Markdown documents YAML-formatted header Markdown-formatted text https://blogs.uoregon.edu/rclub/2016/04/26/r-markdown-resources/ Code chunk 2019 Big Data Summer Institute

23. A rendered R Markdown document (HTML) Information from YAML header Text https://blogs.uoregon.edu/rclub/2016/04/26/r-markdown-resources/ Evaluated R code 2019 Big Data Summer Institute

24. R Notebooks • Like R Markdown documents, but you can evaluate code chunks independently, without needing to render the entire document 2019 Big Data Summer Institute

25. Create a new R Markdown document or an R Notebook Create a new document/notebook & try commands listed in tutorial 2019 Big Data Summer Institute

26. Jupyter notebooks • Similar to R Notebooks, a literate & interactive programming environment combining Markdown documentation with code chunks • popular with Python users • Supports many different kernels (e.g., you can run R code in Jupyter notebooks) 2019 Big Data Summer Institute

27. Version Control 2019 Big Data Summer Institute

28. Version control gone wrong 2019 Big Data Summer Institute

29. Version control system (VCS) • System for managing changes to files • No more duplicate files! • Full history of revisions is accessible • Revisions can be compared, restored, and merged • Multiple team members can edit files; VCS handles merging 2019 Big Data Summer Institute

30. Git: a flexible VCS • Git is a distributed version control system • i.e., does not require central server • Flexible branching design • Faster than non-distributed systems 2019 Big Data Summer Institute

31. Cloud-based VCS hosting platform • Central repository for storing Git-enabled projects • Issue tracking, wiki, package hosting, … • Extremely popular and community-oriented! • >36 million users, >100 million code repositories 2019 Big Data Summer Institute

32. Setting up Github • Create a GitHub account: https://github.com/join • Windows users: install GitHub Desktop https://desktop.github.com/ • This will install Git Shell on your PC—check that you have this in your programs • Mac/Linux users: type git in your terminal (Windows users use Git Shell)—should see usage info; if not, should be prompted to install • Detailed instructions:https://help.github.com/articles/set-up-git/ 2019 Big Data Summer Institute

33. Git terminology • A repositoryis a directory containing your project files and the Git metadata • A branchis a development arm of a project • A commit is a checkpoint for changes you have made • Your commits are added to a remote repository by pushing • You can grab someone else’s revisions by pulling commit branch repository 2019 Big Data Summer Institute

34. History tracking • Git tracks every committed change—you can access and revert to any commit in the repository’s history • No need to manually save old versions! 2019 Big Data Summer Institute

35. Basic Git commands git initgit add <source_file>git statusgit diff <file|directory>git commitgit log git remote add origin <github_url>git push –u origin master git pushgit pull 2019 Big Data Summer Institute

36. Git demo 2019 Big Data Summer Institute

37. Controlling the Computational Environment 2019 Big Data Summer Institute

38. Capturing the computational environment • Computational projects for big data can be extremely complex, and make use of multiple languages, libraries, and dependencies • Reproducibility encompasses the entire computational environment, not just your data & code • Debugging dependency issues can be time-consuming & impede reproducibility 2019 Big Data Summer Institute

39. Package managers name: my-project channels: - conda-forge - bioconda dependencies: - cyvcf2=0.8 - pyfaidx=0.5 - joblib=0.11 • Language specific • Pip, cget, Cargo, npm, etc. • Built-in (R, Julia, etc.) • System specific • apt, rpm, MacPorts, etc. • Conda 2019 Big Data Summer Institute

40. Conda environments • Conda is a cross-platform package manager, very popular with Python users • All dependencies (OS applications, Python libraries, R packages, etc.) can be specified in a YAML file conda env create --name "my-env" -f env.yml conda activate my-env conda deactivate • Provides an easy way for other users to reproduce your analyses with exactly the same dependencies 2019 Big Data Summer Institute

41. Containers (Singularity/Docker) • Frameworks for isolating computational environments & applications in standalone virtual “containers” • Containers include a barebones operating system and all dependencies specified by the creator • Containers can run on nearly any system, ensuring your code is widely reproducible 2019 Big Data Summer Institute

42. Conda & Singularity demo 2019 Big Data Summer Institute

43. What about hardware? • Conda environments & Docker containers only control the OS & software environment/dependencies • Hardware requirements are harder to control—e.g., not everyone can reproduce analyses that require 100s of CPUs • Best practices: • Document specific hardware environment you used (e.g., processor model, available RAM, required disk space) • Keep track of program performance (e.g., RAM usage, runtime, CPUs required) • Always provide your code, even if it’s not runnable on every system 2019 Big Data Summer Institute

44. Automation 2019 Big Data Summer Institute

45. Why automate your workflow? • Complex projects are often spread across several stages of data (pre)processing and analyses • Not only must each stage be reproducible, but also the flow of information between stages • Automation tools ensure complex workflows are reproduced in the proper order • When data or code changes, automation ensures all steps are followed in proper order and all downstream data is updated properly 2019 Big Data Summer Institute

46. Tools for automation • GNU make (build system) • Workflow engines • Ssnakemake, Nextflow, CWL, etc. • A series of “recipes” for controlling the flow of your analyses and managing data dependencies 2019 Big Data Summer Institute

47. Snakemake Example rulestep_one: output: "tmp/{letter}.txt"shell: "echo 'Hello' > {output}"rulestep_two:input:"tmp/a.txt", "tmp/b.txt", "tmp/c.txt"output:"out/merged.txt"shell:"cat {input} > out/merged.txt" 2019 Big Data Summer Institute

48. Snakemake demo 2019 Big Data Summer Institute

49. Summary • Starting with a mindset of reproducibility will pay off in the future! • Pay attention to project organization • Convey the ideas behind your code with documentation • Take advantage of version control systems • Capture environment and software dependencies • Automate whenever possible 2019 Big Data Summer Institute

50. Things you can do today • Think through how to effectively organize your projects and create a folder structure • Install Rstudio, Conda & Jupyter and try to reproduce some examples • Create a Github account and add your notes, code, etc. to a repository • Install Singularity or Docker and try to containerize your software • Install Snakemake (or other engine) and explore how you can port existing pipelines into a portable workflow 2019 Big Data Summer Institute