TVM & the Apache Software Foundation

1. TVM &the Apache Software Foundation Markus Weimer Member, Apache Software Foundation Architect, Microsoft ML Platform Why I am here Pays me to be here

2. Agenda

3. Source: https://www.apache.org/

4. History: apache httpd [1] http://news.netcraft.com/archives/2016/04/21/april-2016-web-server-survey.html

5. What is the ASF? -- A non-profit cooperation 100s Project Board Project Appoint Project 100s Officers (PMC Chairs) PMC Project Project 1000s Contributors Committers Elect Members Source for the numbers: http://www.infoworld.com/article/3079813/open-source-tools/the-apache-foundations-incredible-rise.html

6. Project Lifecycle

7. Agenda

8. Core Principles • Collaborative software development • Respectful, honest, technical-based interaction • Consistently high quality software • Commercial-friendly standard license • Faithful implementation of standards • Security as a mandatory feature • Collaborative software development • Respectful, honest, technical-based interaction • Consistently high quality software • Commercial-friendly standard license • Faithful implementation of standards • Security as a mandatory feature Community over code

9. Meritocracy and Roles

10. Authority in Apache

11. Consensus driven development

12. Voting • +1: I want this to happen • +0: Meh • -0: Ewh, really? • -1: No way. Veto (where possible) • Binding: The vote counts • E.g. PMC votes on release (majority vote) • Non-binding • E.g. User votes in a release Voting should be relatively rare. The goal is that the project proceeds by consensus, and its members negotiate outcomes with one another. The votes (+1,-1, …) are sometimes used in discussions to express opinions outside of a [VOTE] thread.

13. Communications

14. “If it didn’t happen on the mailing list, it didn’t happen”

15. Lesson: The Apache Process leads to better Software “If it isn’t on the mailing list, it didn’t happen” • The bad: • Higher latency per issue: Discussing things over email takes longer • This can be trained and gets get better over time. • The good: • Full visibility, no need for meetings to “bring everybody on the same page” • Documents the decision process, not just the outcome • Every developer (even the shy ones) have equal influence. • The phenomenal: • Quality goes up (code as well as discussion becomes part of your CV) • Throughput goes up (less blockage, uncertainty)

16. Agenda

17. Process History • As communities form, they make their own rules • In practice, people copy / merge the rules from past projects • REEF copied from Spark and Hadoop, which in turn spawned from Lucene

18. CTR vs. RTC Commit then review (CTR) Review then commit (RTC) Every change is reviewed by another committer. Every change is merged by another committer. Always releasable software Extremely clean commit log Used by every Big Data project in Apache C:\s\reef [master]> git log --oneline b5c807a [REEF-1373] Convert IClock to an interface 6b67524 [REEF-1420] Dispose IGroupCommClient/Network Service from IMRU tasks af8b908 [REEF-1369] Remove obsolete RuntimeClock.RegisterObserver f2b9b84 [REEF-1421] Transport Client inner thread is not canceled when the object is disposed 1a2f120 [REEF-1410] Validate Task constructor failure => FailedTask Event dbd628a [REEF-1414] Condition to EvaluatorExitLogger is inverted for RuntimeStop 414d4b4 [REEF-1392] Adding IObserver<ICloseEvent> for IMRU tasks b564736 [REEF-1345] Define IMRU task exceptions d5a671b [REEF-1403] Deadlock between ContextRuntime.StartTask and HeartBeatManager.OnNext(Alarm) f5ae659 [REEF-1306] Remove OnDriverReconnect from parameter in DriverConfiguration bf5caab [REEF-1396] Fix testFailureRestart to validate that the restarted Evaluators are received 4c0207e [REEF-1409] Upgrade HDP2.4 docker image to use 2.4.2 repository 947b18e [REEF-1398] Update version to 0.16.0-SNAPSHOT b495935 [REEF-1400] Update update_website.py script to include pom.xml file e03fa36 [REEF-1401] Fix Driver Restart to correctly use the right folder • Every committer does the code changes they see fit. • Other committers review and undo changes. • Stabilization happens during release. • Messy commit log

19. JIRAs, Code Reviews and commits • Design discussions happen on JIRA (yay!) • People are comfortable and encourage many open JIRAs • Example [MapReduce-279] (YARN) was open for 3.5 years. • Often, the committer closes issue (2nd best option) • Code reviews happen on GitHub • Commit messages link back to both. • Committer who does the merge also rebases to current `master` • Commits are squashed to create a linear commit history • “True history” is preserved on GitHub

20. Commit messages • Valid Markdown • Link to JIRA • Link to Pull Request • Useful one-line summary [REEF-1410] Validate Task constructor failure => FailedTask Event This addressed the issue by * Fixing heartbeat failure if task fails to start. * Adding a test to verify Task failure message. JIRA: [REEF-1410](https://issues.apache.org/jira/browse/REEF-1410) Pull Request: This closes #1019

21. Releases

22. Agenda

23. Next steps

24. Thanks for Your Time! • weimer@apache.org • Markus.Weimer@Microsoft.com • @markusweimer • http://markusweimer.com

25. Resources • https://www.apache.org/foundation/how-it-works.html • https://wiki.apache.org/hadoop/HowToContribute • https://wiki.apache.org/hadoop/HowToCommit • http://www.catb.org/~esr/faqs/smart-questions.html • https://issues.apache.org/jira/browse/HADOOP (YARN/MAPREDUCE/HDFS/...) • http://producingoss.com/ (Karl Fogel, Subversion) • http://www.infoworld.com/article/3079813/open-source-tools/the-apache-foundations-incredible-rise.html