trilinos usage of cmake ctest cdash n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Trilinos Usage of CMake/CTest/CDash PowerPoint Presentation
Download Presentation
Trilinos Usage of CMake/CTest/CDash

Loading in 2 Seconds...

play fullscreen
1 / 26

Trilinos Usage of CMake/CTest/CDash - PowerPoint PPT Presentation


  • 216 Views
  • Uploaded on

2009-1098P. Trilinos Usage of CMake/CTest/CDash . Roscoe A. Bartlett http://www.cs.sandia.gov/~rabartl/ Department of Optimization & Uncertainty Estimation Sandia National Laboratories Kitware/SNL CMake/CTest/CDash Workshop May 7, 2009.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Trilinos Usage of CMake/CTest/CDash' - shada


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
trilinos usage of cmake ctest cdash

2009-1098P

Trilinos Usage of CMake/CTest/CDash

Roscoe A. Bartlett

http://www.cs.sandia.gov/~rabartl/

Department of Optimization & Uncertainty Estimation

Sandia National Laboratories

Kitware/SNL CMake/CTest/CDash Workshop

May 7, 2009

Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,for the United States Department of Energy under contract DE-AC04-94AL85000.

primary motivations for switching trilinos
Primary Motivations for Switching Trilinos
  • Native support for MS Windows (MS Visual Studio projects and self-extracting installers) and Mac (XCode project files and binary installers)
  • Improve robustness of our development and testing environments
    • Improve the stability of important Trilinos code
    • Support stronger collaborations with important customer codes (i.e. Almost Continuous Integration)
    • Better support R&D of deep stacks of vertically integrated algorithms

CMake Transition

  • Decrease the maintenance costs of the framework and testing infrastructure
    • Reduce maintenance cost by 1/2 to 1/3!
    • Allow Jim Willenbring and Brent Perschbacher to do more important things

Autotools

Total Effort

CMake

Time

gains initial looses switching to cmake for builds
Gains & (Initial) Looses Switching to CMake for Builds
  • What we gain:
    • Full dependency tracking of every kind possible on all platforms (i.e. header to object, object to library, library to executable, and build system files to all built files)
    • Support for shared libraries on a variety of platforms
    • Support for MS Windows (i.e. Visual Studio projects, Windows installers, etc.)
    • Simplified build system and easier maintenance (extremely easy to add new packages and maintain existing packages)
    • Improved mechanism for extending capabilities (as compared to M4 in autotools)
    • Ability to affect the development of the build tools with good existing collaborations
    • Significant ``in house'' knowledge-base
    • One hundred percent automated intra-package dependency tracking and handling (built into the prototype Trilinos/CMake build system)
  • What we lose:
    • CMake requires that all uses have 'cmake' installed on their machine when building from source and users will need to have at a very recent version of cmake. (However, cmake is very easy to build from source)
gains no losses switching to ctest cdash for testing
Gains (no losses) Switching to CTest/CDash for Testing
  • What we gain:
    • Test time-outs (this is a major maintenance issue for the current Perl-based test harness)
    • Memory testing with Valgrind and purify that is backed up by Kitware and a larger development community
    • Line coverage testing that is backed up by Kitware and a large development community
    • Support for selecting and excluding subsets of tests based on regular expressions or named keywords
    • Better integration with the build system (e.g. easier to support more advanced features like PBS batch systems and flexible testing control)
    • Better tracking of specific tests (i.e. each and every test can have a unique name that is easy to find)
  • What we lose:
    • Nothing!
design principles for trilinos cmake build system 1
Design Principles for Trilinos CMake Build System: #1
  • Make it exceedingly easy to define CMake files for new packages and to define libraries, tests, and examples in those packages.
  • Create a design for building individual package CMake files that automatically results in uniformity of how things are done.
    • Use standard macros to define every package's main features
    • Simplify maintenance.
  • Allow changes to logic and functionality that apply to all Trilinos packages without having to touch each individual Trilinos package's CMake files.
  • Provide 100% automatic intra-package dependencies handling
    • Avoid mistakes
    • Avoid duplication
    • Robustifies a number of important features.
  • Provide built-in automated support for as many critical software engineering practices a possible
    • Proper pre-checkin testing when continuous integration is being performed
design principles for trilinos cmake build system 2
Design Principles for Trilinos CMake Build System: #2
  • Avoid duplication of all kinds as much as possible.
    • Fundamental software maintenance issue.
  • The build system should be able to reproduce 100% update-to-date output by simply typing ‘make’.
    • E.g. Coping test input files to binary directory
  • Aggregate as much common functionality as possible to the top-level CMake files but allow individual CMake packages to refine the logic if they really need to.
  • Accept extra complexity at the global framework level to reduce it at the package level
    • Apply solid software engineering design principles to manage the complexity
    • Remove duplication
outline of cmake packagearch build system
Outline of CMake PackageArch Build System

Trilinos/

CMakeLists.txt # Top-level build file

cmake/ # Cmake support

TrilinosPackages.cmake # List of Trilinos packages

TrilinosTPLs.cmake # Lists of Trilinos TPLs

...

package_arch # Generic package architecture macros

PackageArchGlobalMacros.cmake

PackageMacros.cmake

PackageLibraryMacros.cmake

PackageAddExecutable.cmake

...

...

packages/

...

epetraext/

CMakeLists.txt # Top-level package build file

cmake/

Dependencies.cmake # Defines intra-package dependency and TPL lists

EpetraExt_config.h.in # Copied from ../src/ and hand modified

src/

CMakeLists.txt # Defines library sources and library(s)

test/

CMakeLists.txt

MatrixMatrix/

CMakeLists.txt # Define actual test executables and test runs

...

adding packages in trilinospackages cmake
Adding Packages in TrilinosPackages.cmake

SET( Trilinos_PACKAGES_AND_DIRS_AND_CATEGORIES

Teuchos teuchos PS

RTOp rtop PS

Epetra epetra PS

Triutils triutils PS

EpetraExt epetraext PS

Thyra thyra PS

RBGen rbgen SS

ForTrilinos ForTrilinos EX

PyTrilinos PyTrilinos SS

...

)

  • Adding a new Trilinos Package is a 1-line addition at the Framework Level!
  • NOTE: The packages must be listed in a order of strictly increasing dependences!
  • NOTE: If you get the ordering wrong, the automated dependency handling CMake scripts will automatically detect this and issue a very good error messages before the build is performed!
  • PS = Primary Stable Code, SS = Secondary Stable Code, EX = Experimental Code
defining a package s cmake build files
Defining a Package’s CMake Build Files

Trilinos/packges/

epetraext/CMakeLists.txt

  • Utility macros provide framework hooks into package functionality
    • Defines common behavior across all packages
    • Avoids duplication
    • Facilitates maintenance

INCLUDE(PackageMacros)

INCLUDE(AddOptionAndDefine)

#

# A) Define the package

#

PACKAGE(EpetraExt)

#

# B) Set up package-specific options

#

ADD_OPTION_AND_DEFINE(${PACKAGE_NAME}_BUILD_TRANSFORM

HAVE_TRANSFORM

"Enable transform functionality."

ON )

...

#

# C) Add the libraries, tests, and examples

#

ADD_SUBDIRECTORY(src)

PACKAGE_ADD_TEST_DIRECTORIES(test)

#

# D) Do standard postprocessing

#

PACKAGE_POSTPROCESS()

epetraext/cmake/Depencencies.cmake

SET(LIB_REQUIRED_DEP_PACKAGES Epetra Teuchos)

SET(LIB_OPTIONAL_DEP_PACKAGES Triutils)

SET(TEST_REQUIRED_DEP_PACKAGES)

SET(TEST_OPTIONAL_DEP_PACKAGES)

SET(LIB_REQUIRED_DEP_TPLS)

SET(LIB_OPTIONAL_DEP_TPLS)

SET(TEST_REQUIRED_DEP_TPLS)

SET(TEST_OPTIONAL_DEP_TPLS)

  • Intra-package dependencies !
    • Used in all intra-package dependency handing
    • All header-file paths and link libraries and directories are handled automatically
    • These dependencies can not be wrong! (i.e. the libs and execs would not build and link otherwise)
    • Optional dependencies create variables <PACKAGE>_ENABLE_<DEP> and HAVE_<UCPACKAGE>_<UCDEP> !
automatic intra package dependency handling
Automatic Intra-Package Dependency Handling

$ ./do-configure -D Trilinos_DUMP_PACKAGE_DEPENDENCIES:BOOL=ON

Configuring Trilinos build directory

Printing package dependencies ...

-- Teuchos_FORWARD_LIB_REQUIRED_DEP_PACKAGES='RTOp;EpetraExt;Isorropia;Thyra;Galeri;Amesos;Ifpack;Belos;RBGen'

-- Teuchos_FORWARD_LIB_OPTIONAL_DEP_PACKAGES='AztecOO;ML'

-- Epetra_FORWARD_LIB_REQUIRED_DEP_PACKAGES='Triutils;EpetraExt;Isorropia;AztecOO;Galeri;Amesos;Ifpack;Belos'

-- Epetra_FORWARD_LIB_OPTIONAL_DEP_PACKAGES='Thyra;ML;RBGen'

-- Zoltan_FORWARD_LIB_REQUIRED_DEP_PACKAGES='Isorropia'

...

-- EpetraExt_LIB_REQUIRED_DEP_PACKAGES='Epetra;Teuchos'

-- EpetraExt_LIB_OPTIONAL_DEP_PACKAGES='Triutils'

-- EpetraExt_FORWARD_LIB_OPTIONAL_DEP_PACKAGES='Isorropia;Thyra;Galeri;Amesos;ML'

-- EpetraExt_FORWARD_TEST_OPTIONAL_DEP_PACKAGES='Belos'

-- Isorropia_LIB_REQUIRED_DEP_PACKAGES='Teuchos;Epetra;Zoltan'

-- Isorropia_LIB_OPTIONAL_DEP_PACKAGES='EpetraExt'

-- Isorropia_FORWARD_LIB_OPTIONAL_DEP_PACKAGES='ML'

-- Thyra_LIB_REQUIRED_DEP_PACKAGES='RTOp;Teuchos'

-- Thyra_LIB_OPTIONAL_DEP_PACKAGES='EpetraExt;Epetra'

-- Thyra_FORWARD_LIB_REQUIRED_DEP_PACKAGES='Stratimikos'

...

-- Stratimikos_LIB_REQUIRED_DEP_PACKAGES='Thyra'

-- Stratimikos_LIB_OPTIONAL_DEP_PACKAGES='Amesos;AztecOO;Belos;Ifpack;ML'

-- Stratimikos_TEST_OPTIONAL_DEP_PACKAGES='Triutils'

adding a library in package src cmakelists txt
Adding a Library in PACKAGE/src/CMakeLists.txt
  • Dependent package header directories, libraries, and library link directories handled automatically!
  • Macros provide uniform behavior across all libraries across all packages!
  • Test-only libraries are supported with ‘TESTONLY’ flag (see NOX for an example)

INCLUDE(PackageLibraryMacros)

# A) Package-specific configuration options

PACKAGE_CONFIGURE_FILE(${PROJECT_NAME}_config.h)

# B) Define the header and source files (and directories)

INCLUDE_DIRECTORIES(${CMAKE_CURRENT_SOURCE_DIR})

SET(HEADERS

EpetraExt_ConfigDefs.h

...

)

SET(SOURCES

EpetraExt_ProductOperator.cpp

...

)

...

# C) Define the targets for package's library(s)

PACKAGE_ADD_LIBRARY(

epetraext

HEADERS ${HEADERS}

SOURCES ${SOURCES}

)

adding a test in package test cmakelists txt
Adding a Test in PACKAGE/test/CMakeLists.txt

epetraext/test/CMakeLists.txt

# Compile against epetra_test_err.h in all tests?

INCLUDE_DIRECTORIES(${CMAKE_CURRENT_SOURCE_DIR})

...

ADD_SUBDIRECTORY(MatrixMatrix)

  • Add common header include dirs
  • Just include the subdirectories

epetraext/test/MatrixMatrix/CMakeLists.txt

INCLUDE(PackageAddExecutableAndTest)

INCLUDE(PackageCopyFilesToBinaryDir)

PACKAGE_ADD_EXECUTABLE_AND_TEST(

MatrixMatrix_test

SOURCES cxx_main.cpp

COMM serial mpi

)

PACKAGE_COPY_FILES_TO_BINARY_DIR( EpetraExtMatrixMatrixCopyFiles

DEST_FILES

infileAB infileATBT infileAB2 infiles infileABT infileAB3 infileATB infileATB2

C.mtx C4x4.mtx C4x12x12x4.mtx C4x6.mtx C6x4.mtx C6x6.mtx mat6x4.mtx mat6x6.mtx

mat12x4.mtx mat4x12.mtx mat4x4.mtx mat4x6.mtx Y.mtx YTC.mtx Y_transp.mtx

roman roman2 romancase romancase2 cdt cdt_case cdt_d.mtx cdt_m.mtx cdt_tce.mtx

SOURCE_DIR ${${PACKAGE_NAME}_SOURCE_DIR}/test/MatrixMatrix

SOURCE_PREFIX "src_“

EXEDEPS MatrixMatrix_test

)

  • All header paths, link libraries etc are handled automatically!
  • Define executable and test in one shot!
  • 100% correct dependency tracking!
defining slightly more complex tests
Defining slightly more complex tests

thyra/test/operator_solve/CMakeLists.txt

INCLUDE(PackageAddExecutableAndTest)

PACKAGE_ADD_EXECUTABLE(

test_linear_op_with_solve

SOURCES test_linear_op_with_solve.cpp

COMM serial mpi

)

PACKAGE_ADD_TEST(

test_linear_op_with_solve

NAME test_linear_op_with_solve_n1_n2

ARGS "--n=1" "--n=2"

NUM_MPI_PROCS 1

COMM serial mpi

)

PACKAGE_ADD_TEST(

test_linear_op_with_solve

NAME test_linear_op_with_solve_n4

ARGS "--n=4"

NUM_MPI_PROCS 1

COMM serial mpi

XHOST s858352 s903186

)

  • Define test cases separately from executable if needed!
defining more sophisticated tests
Defining More Sophisticated Tests

Current Limits of built-in CTest tests:

  • Runs a single executable (or script) with a set of arguments
  • Pass/fail determined by return value (0), or regex fail or regex any pass on STDOUT
  • Runs test in current directory

What we would like:

  • Run multiple commands
  • Define pass/fail for each command in various ways (examine other output files, examine output from multiple commands, etc.)
  • Run tests in subdirectories created for the test

Solution:

  • Write user-defined CMake function PACKAGE_ADD_ADVANCED_TEST(...)
  • Write ‘cmake –P’ scripts
defining more sophisticated tests1
Defining More Sophisticated Tests

Define advanced test:

PACKAGE_ADD_ADVANCED_TEST(

<testName>

TEST_0 (EXEC <execTarget0> | CMND <cmndExec0>) ...

[TEST_1 [EXEC <execTarget1> | CMND <cmndExec1>] ...]

...

[TEST_N [EXEC <execTargetN> | CMND <cmndExecN>] ...]

[OVERALL_WORKING_DIRECTORY <overallWorkingDir>]

[FAIL_FAST]

[KEYWORDS <keyword1> <keyword2> ...]

[COMM [serial] [mpi]]

[OVERALL_NUM_MPI_PROCS <overallNumProcs>]

[HOST <host1> <host2> ...]

[XHOST <host1> <host2> ...]

[FINAL_PASS_REGULAR_EXPRESSION <regex> | FINAL_FAIL_REGULAR_EXPRESSION <regex>]

)

Where:

TEST_<i>

EXEC <execTarget> [NOEXEPREFIX] [NOEXESUFFIX] [ADD_DIR_TO_NAME]

| CMND <cmndExec>

[ARGS <arg1> <arg2> ... <argn>]

[MESSAGE "<message>"]

[WORKING_DIRECTORY <workingDir>]

[NUM_MPI_PROCS <numProcs>]

[OUTPUT_FILE <outputFile>]

[NO_ECHO_OUTPUT]]

[PASS_ANY

| PASS_REGULAR_EXPRESSION "<regex>"

| PASS_REGULAR_EXPRESSION_ALL "<regex1>" "<regex2>" ... "<regexn>"

| FAIL_REGULAR_EXPRESSION "<regex>"

| STANDARD_PASS_OUTPUT

]

defining more sophisticated tests2
Defining More Sophisticated Tests

PACKAGE_ADD_ADVANCED_TEST( ValidateParameters_compareTestOutput

COMM serial mpi

OVERALL_NUM_MPI_PROCS 1 # Should be 1,4,9

TEST_0 EXEC ml_Validate ARGS input1.xml

OUTPUT_FILE ValidateParameters_compareTestOutput.out

TEST_1 CMND ${PYTHON_EXECUTABLE}

ARGS ${PROJECT_SOURCE_DIR}/commonTools/test/utilities/compareTestOutput

evaluateCriteria baseline1.txt ValidateParameters_compareTestOutput.out

PASS_REGULAR_EXPRESSION "Test passed."

FAIL_FAST

)

defining more sophisticated tests3
Defining More Sophisticated Tests

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Advanced Test: ML_ValidateParameters_compareTestOutput

Running test commands: TEST_0, TEST_1

================================================================================

TEST_0

Running: "/Users/bmpersc/nightly/Trilinos.base/SERIAL_DEBUG/BUILD/packages/ml/test/ValidateParams/ML_ml_Validate.exe" "input1.xml"

Writing output to file "/Users/bmpersc/nightly/Trilinos.base/SERIAL_DEBUG/BUILD/packages/ml/test/ValidateParams/ValidateParameters_compareTestOutput.out"

--------------------------------------------------------------------------------

...

--------------------------------------------------------------------------------

TEST_0: Return code = 0

TEST_0: Pass criteria = Return code

TEST_0: Result = PASSED

================================================================================

TEST_1

...

defining more sophisticated tests4
Defining More Sophisticated Tests

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Advanced Test: ML_ValidateParameters_compareTestOutput

Running test commands: TEST_0, TEST_1

================================================================================

...

================================================================================

TEST_1

Running: "/usr/bin/python2.5" "/Users/bmpersc/nightly/Trilinos.base/SERIAL_DEBUG/Trilinos/commonTools/test/utilities/compareTestOutput" "evaluateCriteria" "baseline1.txt" "ValidateParameters_compareTestOutput.out"

--------------------------------------------------------------------------------

Test passed.

--------------------------------------------------------------------------------

TEST_1: Return code = 0

TEST_1: Pass criteria = Match REGEX {Test passed.}

TEST_1: Result = PASSED

================================================================================

OVERALL FINAL RESULT: TEST PASSED

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

http://trilinos-dev.sandia.gov/cdash/testDetails.php?test=182284&build=17561

trilinos cmake workshop outline
Trilinos CMake Workshop Outline

Automatic Intra-Package Dependency Handling

quickstart special configuration modes for trilinos
Quickstart: Special Configuration Modes for Trilinos
  • Configuring Trilinos to build all packages with all tests and examples:

-D Trilinos_ENABLE_ALL_PACKAGES:BOOL=ON

-D Trilinos_ENABLE_TESTS:BOOL=ON

  • Configuring a package(s) along with all of the packages it can use

-D Trilinos_ENABLE_Stratimikos:BOOL=ON

-D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=ON

-D Trilinos_ENABLE_TESTS:BOOL=ON

  • Configuring Trilinos to disable a package(s) and all packages it depends on:

-D Trilinos_ENABLE_Stratimikos:BOOL=ON

-D Trilinos_ENABLE_Amesos:BOOL=OFF

automatic intra package dependency handling1
Automatic Intra-Package Dependency Handling
  • Trilinos/cmake/python/data/TrilinosPackageDependenciesTable.html
  • http://gabriel.sandia.gov/TrilinosPackageDependenciesTable.html
package dependency structure for thyra

RTOp

Teuchos

Epetra

Triutils

Thyra

EpetraExt

Package Dependency Structure for Thyra

Required Dependence

Optional Dependence

example enabling a package and all optional packages
Example: Enabling a Package and All Optional Packages

$ ./do-configure –D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF \

-D Trilinos_ENABLE_Thyra:BOOL=ON \

-D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=ON

Configuring Trilinos build directory

...

Enabling all optional packages for current set of enabled packages ...

-- Setting Trilinos_ENABLE_EpetraExt=ON because Trilinos_ENABLE_Thyra=ON

-- Setting Trilinos_ENABLE_Epetra=ON because Trilinos_ENABLE_Thyra=ON

-- Setting Trilinos_ENABLE_Triutils=ON because Trilinos_ENABLE_EpetraExt=ON

Enabling all remaining required packages for the current set of enabled packages ...

-- Setting Trilinos_ENABLE_RTOp=ON because Trilinos_ENABLE_Thyra=ON

-- Setting Trilinos_ENABLE_Teuchos=ON because Trilinos_ENABLE_Thyra=ON

Enabling all optional intra-package enables that can be if both sets of packages are enabled ...

-- Setting EpetraExt_ENABLE_Triutils=ON since Trilinos_ENABLE_EpetraExt=ON AND Trilinos_ENABLE_Triutils=ON

-- Setting Thyra_ENABLE_EpetraExt=ON since Trilinos_ENABLE_Thyra=ON AND Trilinos_ENABLE_EpetraExt=ON

-- Setting Thyra_ENABLE_Epetra=ON since Trilinos_ENABLE_Thyra=ON AND Trilinos_ENABLE_Epetra=ON

Final set of enabled packages: Teuchos RTOp Epetra Triutils EpetraExt Thyra 6

dependency handling for pre checkin testing

RTOp

Teuchos

Epetra

Triutils

Thyra

EpetraExt

Lib Only

Libs & Tests

Dependency Handling for Pre-Checkin Testing

$ ./do-configure \

-D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF \

-D Trilinos_ENABLE_Epetra:BOOL=ON \

-D Trilinos_ENABLE_ALL_FORWARD_DEP_PACAKGES:BOOL=ON \

-D Trilinos_ENABLE_TESTS:BOOL=ON

dependency handling for pre checkin testing1

RTOp

Teuchos

Epetra

Triutils

Thyra

EpetraExt

Lib Only

Libs & Tests

Dependency Handling for Pre-Checkin Testing

$ ./do-configure \

-D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF \

-D Trilinos_ENABLE_RTOp:BOOL=ON \

-D Trilinos_ENABLE_ALL_FORWARD_DEP_PACAKGES:BOOL=ON \

-D Trilinos_ENABLE_TESTS:BOOL=ON

summary
Summary
  • There is a lot of functionality in the current tools CMake/CTest/CDash
  • CMake/CTest scripting gives great flexibility to extend functionality
  • Various Sandia projects duplicate effort in build, testing, and reporting tools
  • However, complex computational code projects require more functionality!

LET’S TALK ABOUT WHAT ELSE IS NEEDED!