Version Control with Git/Github & Scons Build System

1. Version Control with Git/Github & Scons Build System E.J. Brash Hall A/C Data Analysis Workshop December 18, 2013

2. Outline • Integration of Hall A/C software: Version management with Git and Github o Overview of git and github o Software development workflow o Examples of pushing/pulling updates o Issue tracking • Configuration of Hall A/C software: Development of a new build system – SCons o Overview of SCons o Current status of build/configure system

3. Hall A/C Coordinated Development with Git and Github We are now managing Hall A and Hall C software efforts with git and github. • Hall C analysis (hcana) is being developed within the Hall A framework (PODD) -> having an effective development management system is important to coordinate these efforts. • Hall C development has already spurred new efforts to improve PODD. • Ongoing Fortran/hcana comparisons in parallel with new development makes version control/management crucial to ensure meaningful comparisons. • Modular nature of experiments in Hall C (and Hall A) requires ongoing and continuous development of new analysis software, on an experiment-by- experiment basis -> version control is essential • Even experiments which use the same equipment often require different analysis configurations, due to different kinematics configurations, etc. •

4. Git – A distributed version control system • Git is DISTRIBUTED: o Rather than checking out the current tip of the source code, one actually “clones” the entire repository – no single point of failure. o The distributed nature of Git allows novel workflow models – more on this later. • Git’s branching model is advanced: o Multiple local branches that are independent of one another. o Creation, merging, and deletion of these branches is easy. o Again, this influences/improves workflow – allows for more experimentation with new ideas, creation of “feature-based” branches, etc.

5. Git Workflows • Subversion/CVS style workflow: o Just like with SVN/CVS, merging with the main repository will not be allowed if conflicts exist (relaed to merges from other developers).

6. Git Workflows • Integration Manager Workflow: o This is our current model for Hall A/C development o Integration Managers: • Hall A – Ole Hansen • Hall C – Steve Wood git@github.com:JeffersonLab/<repository-name>.git

7. Development Workflow using Github.com Online project hosting using git … many practical features which allow easy visualization of development path. • Developers use their own account on github.com to create “forks” of JeffersonLab repositories. • Changes are committed to a developer’s forked repository, and at some point may be merged with the main JeffersonLab repository, via a pull request to the integration manager. • The integration managers for the Jefferson Lab repositories govern this process (Ole Hansen in Hall A and Steve Wood in Hall C). • Integration manager model maintains limited write-access by developers/users to Jefferson Lab repositories -> important for comparison studies •

8. Git Workflows • Director and Lieutenants Workflow: o This could be our model as we move closer to start of running, and beyond o Lieutenants: each subsystem

9. Current JLab Git Projects • Hall A C++ Analyzer (Hall C submodule) • Hall C C++ Analyzer • Hall C Fortran engine • Hall C Replay x 2 (for comparisons) • Hall C Geant simulation of Compton Polarimeter • SIMC (Monte Carlo for Halls A and C) • SHMS Monte Carlo (for proposal development) • TreeSearch (Hall A TreeSearch track reconstruction)

10. Development under Git Documentation of git-based procedures to download and install Hall A/C software (source and replay) exists, and is continually updated (Wiki) • Easy to install and update code on JLab systems as well as on local (Mac and Linux) machines • git clone git@github.com:brash99/analyzer.git git branch –a o o Master remotes/origin/HEAD -> origin/master remotes/origin/Release-070 remotes/origin/Release-100 remotes/origin/Release-110 remotes/origin/Release-120 remotes/origin/Release-130 remotes/origin/Release-140 remotes/origin/Release-150 remotes/origin/master remotes/origin/scons_final_change remotes/origin/scons_link_develop remotes/origin/scons_link_develop_150 • • • • • • • • • • • • • git pull origin <branch-name> o

11. Example Network Graph $ git pull origin develop $ git checkout –b scons_link_develop_150 (make changes) $ git commit –a $ git push origin scons_link_develop_150 Analyzer Network

12. Pull Requests Users/developers who are “watching” development of the JeffersonLab/analyzer repository are notified by email of pull requests. Typically, other developers, as well as the integration manager, can make comments on the proposed pull request prior to it being accepted (or rejected).

13. Details of Pull Request

14. Keeping forks up-to-date $ git remote add upstream git@github.com:JeffersonLab/hcana.git $ git fetch upstream $ git merge upstream/develop $ git push origin <my_develop_branch>

15. Issue Tracking in Git X - Issues to be solved can be created at any time, and assigned to a particular developer When pull requests are made, can be associated with one or more issues, and issue can be closed (if appropriate) -

16. Where to get more info • Git: http://www.git-scm.com • GitHub: http://www.github.com • Code School: http://www.codeschool.com Tutorials on Git – free one is very useful! • Wiki: https://hallcweb.jlab.org/wiki/index.php/ROOT_Ana lyzer/Git

17. A new build/configure system - SCons • Traditionally, Hall A/C software has been built with “make” Platform/system/compiler dependent configuration handled within Makefiles (coupled with #ifdef statements within the code itself) o Dependency checking not included by default, and is based on timestamp. o Having an “autoconf”-like configuration is desirable, but GNU Autoconf is highly complex o Makefiles are platform-dependent, and incredibly cryptic – basically unreadable to non-experts – making changes and updates difficult o Libtool (management of libraries) not available for all platforms o • Is there something better out there?

18. A new build system - SCons • SCons is an open-source software construction tool Written entirely in Python – power of a real programming language in configuration and build scripts … plus, our students know and love Python! o Scripts are much more readable than Makefiles • Statistics for PODD: Makefiles-> 1115 lines, Scons->597 lines (442 basic) o Integrated functionality similar to Autoconf o Built-in support for C/C++, and easily extensible for other builders (ROOTCINT) o Built-in dependency-checking – based on MD5 signatures, and not timestamps – important for git o Designed from the ground up for cross-platform builds o Currently used by the JLab DAQ group for EVIO, and by Hall D for both online and offline build systems. o

19. Major Projects using SCons ASCEND - A system modeling package for engineering • Cantera – A toolkit for chemical kinetics and thermodynamics • CLAM – A framework to develop sophisticated audio analysis • FreeNOS – A microkernel operating system written in C++ • IntensityEngine – A platform for 3D games and virtual worlds • Lumiera – A professional video editor • Madagascar – Geophysical data processing • Nsound – C++ audio synthesis framework • openEHR – Electronic Health Record standard • V8 – Google’s open source Javascript engine • YafaRay – An open source raytracing engine •

20. SCons Scripts • SConstruct.py (analog to top level Makefile) o Controls build environment o Calls configuration scripts: • configure.py, linux32.py, linux64.py, darwin64.py • SConscript.py (analog to other Makefiles) o Builds ROOT dictionaries, libraries and main/standalone programs o At this point in time, developers would edit this file to add new *.C or *.h files -> eventually would like to use wildcards to avoid this

21. SCons commands SCons Make scons make scons –c make clean scons –j4 make –j4 scons debug=1 make DEBUG=1 scons standalone=1 make STANDALONE=1 scons cppcheck=1 No exist! scons checkheaders=1 No exist! Not necessary! make realclean Not necessary! make install

22. SCons – Current status Build and configuration scripts have been written, tested, and committed for both PODD and HCANA • Functionality verified with current version of SCons installed on JLab systems (v. 2.1.0) • Configuration checks for: • ROOT installation gcc/g++ compiler installation and functionality Platform-dependent compiler/linking flags (64/32 bit, Linux/MacOSX) Integration with cppcheck Checks for system header files added o o o o o Currently, we are maintaining the traditional Make system and SCons in parallel (for both PODD and HCANA) • Capability exists for integrating HCANA and PODD under SCons (and git) with modern IDE’s (XCode, Eclipse) … nice for students working on development! • Future work will focus on reorganization and auto-detection of code and header files to make new development easier. •