OVAL - PowerPoint PPT Presentation

reya
slide1 n.
Skip this Video
Loading SlideShow in 5 Seconds..
OVAL PowerPoint Presentation
play fullscreen
1 / 13
Download Presentation
Presentation Description
148 Views
Download Presentation

OVAL

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. OVAL A testing tool for CMS software http://polywww.in2p3.fr/cms/software/oval David Chamont (CMS - LLR)

  2. General Description • Original goal : help to detect automatically the changes in the physics performances of the CMS software used for data production. • Current features of the tool : compile the provided testprograms, run them in specified runtime conditions (shell variables + auxiliary files), compare the output with references. • User interface is CVS-like, configuration files are XML-like. It is written in PERL and can be configured for any build system (scram, make). David Chamont (CMS - LLR)

  3. Outline • General description. • Configuration : OvalFile. • Commands : build/run/diff. • Results : log files. • Other features. • Summary. • Plans. David Chamont (CMS - LLR)

  4. Configuration : OvalFile <difflineexpr="^OVAL:"><diffnumberexpr="^OVAL NUM: .* is (.*)$" tolerance="5%"> <file name=".orcarc">MaxEvents = 500Random:Seeds = 0 3</file> <var name="FEDERATION"value="cmsuf01…"> <environment name="pt15"><var name="DATASET"value="eg_ele_pt15"><program name="Clusters.cpp"><program name="Electrons.cpp" args="-geo detailed"></environment> <environment name="flow"><var name="DATASET"value="jm_minbias"><program name="EnergyFlow.cpp"></environment> David Chamont (CMS - LLR)

  5. Commands : build/run/diff… cms038:ElectronPhoton> ls Makefile CVS Clusters.cpp Clusters.ref Electrons.cpp Electrons.ref EnergyFlow.cpp OvalFile David Chamont (CMS - LLR)


  6. Commands : …build/run/diff… cms038:ElectronPhoton> oval prod Clusters: build, run, diff. Electrons: build, run, diff (DIFFS). David Chamont (CMS - LLR)

  7. Commands : …build/run/diff cms038:ElectronPhoton> ls Makefile CVS Clusters.cpp Clusters.log Clusters.ref Electrons.cpp Electrons.log Electrons.ref EnergyFlow.cpp OvalFile electrons.hbook David Chamont (CMS - LLR)

  8. Results : log files… [oval build] ======================================== [oval build] make Electrons [oval build] ======================================== /usr/local/gcc-alt-2.95.2/bin/c++ -O2 -pthread -rdynamic -L/afs/cern.ch/user/c/chamont/ORCA_5/lib/Linux__2.2 -L/afs/cern.ch/cms/Releases/ORCA_5/lib/Linux__2.2 -L/afs/cern.ch/sw/lhcxx/specific/redhat61/gcc-2.95.2/3.6.2/lib -L/afs/cern.ch/cms/cmsim/cms122/lib/i386_linux22 -L/cern/2001/lib Electrons.o -o Electrons -lElectronFacilities -lEgammaH4Support -lEgammaL1Tools … -lmathlib -lblas -lkernlib -lpythia6152 -lz -lnsl -ldl -lg2c -lm David Chamont (CMS - LLR)

  9. Results : …log files… [oval run] ========================================== [oval run] LD_LIBRARY_PATH = [oval run] /afs/cern.ch/user/c/chamont/ORCA_5/lib/Linux__2.2 [oval run] :/afs/cern.ch/cms/Releases/ORCA_5/lib/Linux__2.2 [oval run] :/afs/cern.ch/sw/lhcxx/i386_linux22/Objectivity/5.2.1/lib [oval run] FEDERATION = cmsuf01::/cms/jet0900/jet0900.boot [oval run] DATASET = eg_ele_pt15 ... [oval run] .orcarc: [oval run] MaxEvents = 500 [oval run] Random:Seeds = 0 3 [oval run] ========================================= Welcome to COBRA ... David Chamont (CMS - LLR)

  10. Results : …log files [oval diff] ======================================== [oval diff] diff line : /^OVAL:/ [oval diff] diff number : /^OVAL NUM: .* is (.*)$/ ~5% [oval diff] ======================================== ref#1452!=log#2053 < OVAL: 12 electrons --- > OVAL: 11 electrons ref#1972!~ log#2592 (>5%) < OVAL NUM: energy mean is 29.7275 --- > OVAL NUM: energy mean is 27.4728 David Chamont (CMS - LLR)

  11. Other Features • Oval is not restrained to programs. For complex testing tasks, especially when databases are involved, one can also handle some scripts (no build step needed). • The same program or script can be executed several times with different arguments, and/or within different environments. • If there are many tests, one can organize them in a hierarchy of Unix directories, and create OvalFiles at any level. • The results of an oval command can be mailed to watchers as defined in the configuration. David Chamont (CMS - LLR)

  12. Summary • Scan the output, whatever it is. • Can be combine with any build system. • Very easy to use, except the tuning of regular expressions (more examples available soon). • No helper code provided. The users still have to write their test programs. • CMS use it for regression tests ; some developers run it daily on their code (kind of extreme programming !) • LCG plans to use it together with CppUnit. David Chamont (CMS - LLR)

  13. Plans • Improve infrastructure & support : • improve internal documentation, • add a tutorial, • progressively move to LCG savannah server. • React to users’ feedback • enable the users to provide their own diff command, • add a reuse mechanism for the environments, • automatic generation of trivial test programs for a given set of header files. David Chamont (CMS - LLR)