Remote Code Management

1. Remote Code Management Since late 2004, the source code management of the HIRLAM system is done remotely. How was this done? What are the possibilities and pitfalls? Toon Moene

2. In the beginning, there was COS • And all was well, because Cray had “nupdate”, a local source code management application. • HIRLAM, like most software maintained at ECMWF, was contained in “nupdate” program libraries. • However, ECMWF decided to switch from Cray to Fujitsu in 1995. • What to do ?

3. On the Fujitsu, we were free ... • ... to choose our own source code management system. • Like any good Unix-like system at that time it had SCCS (Source Code Control System). • However, because I suspected that we would have to change platforms again in the future, I argued for a free alternative for SCCS, namely RCS (Revision Control System). • During the All Staff Meeting in Madrid in 1995, the HIRLAM community accepted this choice. • Nevertheless, we kept the model of “one system manager, who has full control”. RCS didn't allow for any distributed management.

4. Remote Documentation Site • Between 1995 and 2004 the System Manager for the HIRLAM project (Gerard Cats) developed a documentation web site at KNMI. • So after 1995 we were familiar with the fact that documentation of the HIRLAM project, the releases and the release notes were not some files that were sent along with the source code, but on actively maintained web pages. • One of his students developed a Fortran code → .html tool that allowed the source to be viewable on this web site.

5. Remote Source Code Repository • From having the documentation of the source code online, it wasn't a large step to put the source code repository online itself. • As I had been working with the GNU Compiler Collection (GCC) in this way since the revolution of 1997 (“egcs”) I knew it could be done with CVS (Concurrent Versions System). • I asked my contacts at Cygnus/Red Hat to spell out to the group (Gerard Cats, Tomas Wilhelmsson, Jacob Weismann Poulsen) how to set up this remote repository and the access to it. • This went “on the air” in late 2004.

6. Switch to Subversion (SVN)‏ • In October 2005, after extensive preparations led by Daniel Berlin, the GCC repository switched from using CVS to Subversion (SVN). • Subversion was sold as (and indeed was) a CVS, but then done right. • It didn't have any large new features over CVS, but it was far easier to work with (especially on the server side). • This prompted a discussion in the HIRLAM system group whether we shouldn't switch too ... • ... which we did (work mostly done by Tomas Wilhelmsson); completed in early 2006.

7. How does it work – user side • To get the source code, you use the following command: toon@super:~/tmp$ svn co https://toon@svn.hirlam.org/trunk/harmonie Authentication realm: <https://svn.hirlam.org:443> Hirlam Subversion repository Password for 'toon': A harmonie/msms A harmonie/msms/Html.pm ... A harmonie/util/oulan/oulan_yomfthermo.h A harmonie/util/oulan/ext_lam_tovshirs.F U harmonie Checked out revision 8470. toon@super:~/tmp$ cd harmonie/ toon@super:~/tmp/harmonie$ svn up At revision 8470.

8. How does it work – server side • We chose to use svn-over-http(s) approach. • It has the following advantages: • 1. Uses HTTP protocol, standard ports (80 and 443); therefore • it works through most Institute's firewalls. • 2. Runs the server under standard HTTP server account. • 3. When using Apache as your HTTP server, you need version 2.0. • (can be run alongside a 1.3 setup – however, on a non-standard • port).

9. How does it work – access policy • All machines in the domains of the HIRLAM consortium's member states have default access to the repository at hirlam.org – for reading. • Write access to the repository is only granted to a few persons specifically configured on the server to have write access. The access is granted by username/password combination (as shown earlier). This is not restricted to the user's machine being in the aforementioned domains. • Example check-out command of the repository by someone who isn't interested in updating the code (note: no user name): svn co https://svn.hirlam.org/branches/harmonie-36h1.1

10. How does it work – update policy • The HIRLAM group decided to go for a “System” group – a small number of people who would be responsible for maintaining the repository. • This group is headed by the Project Leader for System Management. • All scientists who work on improving the HIRLAM system deliver their updates (which can be tested independently of the contents of the repository by using the script system around the Fortran code) to their “nearest” System group person, who will commit them to the repository. • Of course, it must be shown that the change was adequately tested.

11. How does it work – update policy • Outside the System Group, write access will be granted to persons who prepare large changes on development branches. • One famous example of the past few years in HIRLAM was the 'newsnow' branch – for implementing a new snow parametrization – it existed for several years. • The 'maintainer' of a branch is responsible for keeping it up to date with the other developments on the 'trunk'. • 'Phasing in' of this development then consists of one final merge of trunk into branch and then to merge the result back into trunk.

12. Phasing phased out • In this way, a massive phasing effort is not needed: the differences between a new development (on a branch) and the main development (on the trunk) are kept to a minimum by regular merging of the trunk into the branch. • Because the machine the source code repository resides on is reachable from all HIRLAM member states, there is no need for extended phasing visits.