Revision Control Systems

1. CS 294-73 (CCN 27241)Software Engineering for Scientific Computinghttp://www.cs.berkeley.edu/~colella/CS294Lecture 4: Development Tools

2. Revision Control Systems • Modern scientific computing is no longer a solo effort • Most interesting modeling questions that could be simulated by the heroic individual programming scientist have already been investigated • “Productivity language” that are meant to alleviate the complexity of programming high performance software have not delivered yet • Thus, coding is complicated and requires division of roles and responsibilities. • Working together on a common code is very error prone without some technology assistance. • Two tools to be discussed in this part • Concurrent Version System: cvs • Subversion: svn • Very similar in user interface

3. Concurrent Version System • Central Repository system. • There is one master version of the state of the code • Users have “check outs” or “working copy” of the master respository • Can access the master respository via several mechanisms • rsh connection • ssh connection • cvsserver • All user interaction is considered a client-side operation • Transactional protocol

4. SSH protocol interaction • Setting up keychains (ssh-keygen, ssh-add, ssh-agent) • Typing in your password for every single transaction with the central respository is a pain ssh-keygen >ssh-keygen –tdsa // you will be prompted for passphrase >sshanag.lbl.gov >mkdir .ssh :mkdir: .ssh: File exists >exit >cd .ssh >scpid_dsa.pub –l username anag.lbl.gov:.ssh/tempfile >ssh –l username anag.lbl.gov ;cd .ssh >catauthorized_keys2 tempfile>authorized_keys2 >exit ssh-add >ssh-add // provide passphrase • Try ssh again and see if your keychain is working.

5. Setting up for CVS repository work • Environment • CVSROOT=:ext:username@anag.lbl.gov:/usr/local/cvsroot • CVS_RSH=ssh • EDITOR=vi • Commands • >cvs checkout Chombo • >cvs add [filename|directory] • >cvs update • >cvs update –d // by default update only updates directories that were in this level of the repo when you did your checkout. -d says bring in any new directories. • M:Locally Modified or Merging, apparently without conflicts. This is the essence of concurrent version system • C: merge got confused, Conflicts remain. Your local file has been modified and >>> delimiters have been put in the file telling you where you need to pick a winner, and merge by hand. • U or P: Repo version has some updated file that you haven’t touched, your copy replaced.

6. Commands Cont. • >cvs diff • Show me the difference between my local file and the current repo version as of when I took an update. • >cvs log • Show a modification history of a file • >cvs stat • Show the repo info of my working copy of this file • Tags! • >cvs commit • Commit my local working copy of file(s) as the new repo version of these files • User prompted to use EDITOR to create a commit comment • >cvs remove // and watch the horror of ‘remove’ing directories • You can also use GUI-based tools that invoke these commands for you and help keep things straight

7. Tags • Creates a string to associate with a collection of files and their individual revisions • >cvs tag backpoint1 . • This is a “working copy” tag. You keep this info in your own local copy • >cvsrtag -D <date> <new_tag> [file | directory | module] • This is a repository tag. It modifies the repository to associate the new_tag with some revision of the [file|directory|module] • -D is a handy way of setting a tag of the repo at some point in time. • HEAD is a special repo tag reserved for the trunk (non-branched part of the repo) • We use this this special tag when working with branches

8. Branches • …. uhm, if nobody objects, I’ll leave this one for now. • Branches in CVS are a confusing issue • Better to work in Subversion if you are needing to work with complicated branches

9. Subversion: SVN • Central Repository system. • There is one master version of the state of the code • Users have “check outs” or “working copy” of the master respository • Can access the master respository via several mechanisms • rsh connection • ssh connection • svnserver • All user interaction is considered a client-side operation • Transactional protocol

10. Working with SVN • >svn checkout svn+ssh://anag.lbl.gov/usr/local/svnroot/Chombo4 MyLocalName • >svn update • Also a merging/concurrent process, as with CVS • No –d process here. Better, think of it as always applied • >svn log filename • >svn diff [filename|directory] • >svn add [filename|directory] • >svn commit [ |filename|directory] • >svn delete [filename|directory]

11. SVN Branches and Tags: copy • Tags and Branches and similar concepts are all handled by the universal copy command • >svncopyhttp://svn.example.com/repos/calc/trunkhttp://svn.example.com/repos/calc/branches/my-calc-branch -m "Creating a private branch of /calc/trunk.” • Here we are using the http protocol for svn. The same command using ssh protocol would be similar • svn copy svn+ssh://svn.example.com/repos/calc/trunksvn+ssh://svn.example.com/repos/calc/branches/my-calc-branch -m "Creating a private branch of /calc/trunk.” • Tags are just the same as branches. Everything is a copy • Subversion database is clever to not really copy everything. Copies are “shallow” until you modify a file

12. Merging • In svn, everything is just a copy. • Branches • Tags • Other SVN repos • Full Syntax • >svn merge left-tree@rev right-tree@rev working-set • users will almost never use fully specified version • But, it is good to know that the full format exists • Typical syntax • >svn merge svn+ssh://anag.lbl.gov/usr/local/svnroot/Chombo4 • Only one argument given here, and no rev specified, so….. • right-tree and working-set is assumed to be the current directory • The left-rev is assumed to be the most recent • You’ve done an >svn update

13. Branch and Merge • >export sroot=svn+ssh://usr/local/anag/svnroot • >svn copy $sroot/Chombo4/trunk $sroot/Chombo4/branches/GPU_Experiment • >svn checkout $sroot/Chombo4/branches/GPU_Experiment gpuExperiment • Edit and develop code in gpuExperiment • Other users can check out your branch and your updates and commits and add etc. operations are seen by each other • >svn merge $svnroot/Chombo4/trunk • Keep up to date with edits made to the main development trunk • Is the branch code suitable for the core trunk code ? • >cdtmp; svn checkout $svnroot/Chombo4/trunk trunk • >cd trunk; svn merge $sroot/Chombo4/branches/GPU_Experiment

14. GNU Make • A tricky bit of script parsing to manipulate files specialized to work well with compiling code • lots of features to let you do simple things simply. • complicated things without too much work. • almost impossible to figure out what is going wrong. • Main purpose: turn a set of source code into a library or executable. • Only two kinds of objects in a Makefile • Variables (lists of strings) • Rules • Only a few kinds of flow control • ifeq/ifneq/else/endif • No forms or looping available, no jumps, no recursion. • Most difficulties arising from make are related to • Non-trivial variable parsing of the makefile(s) • Rules can fire and trigger in non-obvious ways • The mysteries of regex

15. The Two type of Variables in GNU Make • Recursively Expanded Variables “=“ foo = $(bar) bar = $(ugh) ugh = Huh? all:;echo $(foo) > make all Huh? • Variable is executed at the time it is used in a command • = means build up a symbol table for this name • Notice $. Like in shell, there is the value ‘bar’ and the variable named ‘bar’

16. Good points: • Order doesn’t matter! Go and try it. • Can declare a variable as the composite of many other variables that can filled in by other parts of the Makefile • CFLAGS = $(DEBUG_FLAGS) $(OPT_FLAG) $(LIB_FLAGS) • Lets a makefile build up sophisticated variables when you don’t know all the suitable inputs, or what parts of the Makefile they will come from • >make all DIM=3 • Bad points: • No appending • # error, causes infinite loop • CFLAGS = $(CFLAGS) –c • Future = declarations can clobber what you specified • The last = declaration in the linear parsing of a Makefile is the only one that matters

17. Simply Expanded Variables “:=“ • Immediate mode variable. • The variable is assigned it’s value based on the current state of the Makefile parsing • No symbol chain is created. • Specific to GNU Make • Often just an easier to understand variable. • It acts like variables you know in other languages. • can use for appending • CFLAGS := $(CFLAGS) –c –e –mmx

18. Rules targets : prerequisites [TAB] recipe [TAB] recipe • prerequisites are also called “sources” • Simple example clobber.o : clobber.cppclobber.hconfig.h [TAB] g++ -c –oclobber.oclobber.cpp clob.ex : clobber.okillerApp.o [TAB] g++ -oclob.excobber.okillerApp.o

19. More powerful rules • Pattern Rules %.o : %.cpp $(CC) -c $(CFLAGS) $(CPPFLAGS) $< -o $@ #Gives a pattern that can turn a .cpp file into a .o file • Multitarget Rules %.f %.H : %.ChF • Suffix Rules • .c.o: • $(CC) -c $(CFLAGS) $(CPPFLAGS) -o $@ $<

20. Other Makefile commands • include • $(MAKE) • calling a makefile from inside a recipe • $(MAKELEVEL) can be looked at to see how deep the call stack is • export • send variables from this level of make to lower makelevels • subst • CFLAGs:= $(CFLAGS) $(substFALSE,,$(subst TRUE,-DCH_MPI $(mpicppflags),$(MPI))) • foreach • libincludes = $(foreachi,$(LibNames),-I$(CHOMBO_HOME)/src/$i)

21. What the “make” program does • Much mental confusion about make comes from thinking that the Makefileis the make program • Easy to see. It looks like a shell script. • Remember: Makefile is only Variables & Rules • make: • parses all of your Makefile • builds up variable chains (overriding variables defined on command line) • builds up rules database • Then looks at what target the user has specified • make then attempts to create a chain of rules from the files that exist to the targets specified. • recursive “=“ variables in source-target expressions are evaluated • Using the date stamp on files discovered in the chain make executes recipes to deliver the target. • “=“ variables are evaluated in recipes.

22. Demonstration of the pervasive Make ‘error’ FooBar = trendy F:= fashion vars: @echo $(FooBar) $(F) ifeq ($(F),fashion) FooBar=tragic endif F:= comedy >make vars tragic comedy >

23. USE_64?=TRUE #64bit pointers USE_MT?=TRUE #memory tracking USE_COMPLEX=TRUE #type for complexnumbers USE_TIMER?=TRUE #Chombo::Timer USE_HDF?=TRUE #HDF5 file i/o ?= special assignment, only if not already set. Chombo Makefiles Variables of interest DIM?=2 DEBUG?=TRUE # OPT=FALSE PRECISION?=DOUBLE MPI?=FALSE CXX?=g++ FC?=g77 MPICXX?=mpiCC

24. HDF5 http://www.hdfgroup.org/ftp/HDF5/current/src/building from the source code is the best option.  download the source filehdf5-1.8.7.tar.gzcd /usr/localsudo tar xzf hdf5-1.8.7.tar.gzcp -r hdf5-1.8.7 hdf5-1.8.7.parallelcd hdf5-1.8.7sudomkdir buildcd buildsudo ../configure --enable-production --prefix=/usr/local/hdf5-1.8.7 --with-default-api-version=v16sudo make allsudo make testsudo make installcd /usr/local/hdf5-1.8.7.parallelsudomkdir buildcd buildexport CC=mpic++sudo ../configure --enable-parallel --enable-production --prefix=/usr/local/hdf5-1.8.7.parallel --with-default-api-version=v16sudo make allsudo make testsudo make install your mpi compiler might have a different name on your system.

25. Writing out an HDF5 file from Chombo • Chombo/lib/src/AMRIO.H void WriteAMRHierarchyHDF5(const string& filename, const Vector<DisjointBoxLayout>& a_vectGrids, const Vector<LevelData<FArrayBox>* > & a_vectData, const Vector<string>& a_vectNames, const Box& a_domain, const Real& a_dx, const Real& a_dt, const Real& a_time, const Vector<int>& a_vectRatio, const int& a_numLevels); • There are several variations on this function for special uses. • Do not use the write[FAB|Level|etc] functions in your code. • Those are for debugging. To be discussed later.

26. Command Line Make >cvs checkout Chombo >cd Chombo/lib/mk >ln –s local/Make.defs.leopardMake.defs.local >cd .. >make –j4 all DIM=3 MPI=TRUE OPT=HIGH >ls libboxtools3d.Darwin.64.mpic++.gfortran.DEBUG.OPTHIGH.MPI.a libbasetools3d.Darwin.64.mpic++.gfortran.DEBUG.OPTHIGH.MPI.a .