Thyra For Developers

1. Thyra For Developers Roscoe A. Bartlett Department of Optimization & Uncertainty Estimation Sandia National Laboratories Trilinos Users Group Meeting (Developers Day), November 9th, 2006 Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,for the United States Department of Energy under contract DE-AC04-94AL85000.

2. Outline • What’s New with Thyra • Cross-Package issues • ???

3. Outline • What’s New with Thyra • Cross-Package issues • ???

4. What’s New with Thyra (for Developers) • Complete implicit linear operators • Handle classes • Operator/Vector adapters for Tpetra • Linear solver adapter for Belos • Stratimikos wrapper for linear solvers and preconditioners

5. Outline • What’s New with Thyra • Cross-Package issues • ???

6. Cross Package Issues related to Thyra • Structured output • Teuchos::Describable • Teuchos::VerboseObject • Teuchos::FancyOStream • Reporting timings • Teuchos::Time[Monitor] • Accepting parameters (hieratically) • Teuchos::ParameterList • Thyra interfaces and implementation depend on all of these! • These require inner package support for uniformity!

7. Structured Outputting : Teuchos::VerboseObject • Layered solvers will generate lots of output • Need to indent output for each solver layer (Teuchos::FancyOStream and OSTab) • Line prefixes with Teuchos::FancyOStream may help • Mostly handled as the Thyra Level! • Issues for non-Thyra code: • Always print to an std::ostream (or a Teuchos::FancyOStream), never std::cout or std::cerr directly! • Using Teuchos::VerboseObjectBase::getDefaultOStream() is aways better than using std::cerr or std::cout! • Don’t use if(procID ==0) to determine when to create output, let the std::ostream decide (i.e. use Teuchos::blackholestream or Teuchos::FancyOSteam to control process output) • Want the option to print from specific processes for debugging etc … • Don’t print your main ouput to a file!

8. Reporting of Timings : Teuchos::Time[Monitor] • Teuchos::TimeMonitor • Can be used to time almost any type of operation • Reports timing summary across processes • Naming of timers should be namespace! (with the package name) • e.g. “Belos::Solve …” instead of just “Solve …”” • Enhancements: See Bug 2107

9. The Significance of Parameter Lists and Outputting • In the future, most users will only interact with Trilinos solvers through setting parameters and looking at output! Example: Time Stepper => Nonlinear Solver => Linear Solver => Preconditioner <ParameterList> … <Parameter name=“Time Stepper” type=“string” value=“Rythmos::ImplicitBDFStepper”/> <ParameterList name=“Rythmos::SUNDIALS”> … </ParameterList> <ParameterList name=“Rythmos::ImplicitBDFStepper”> <Parameter name=“Nonlinear Solver” type=“string” value=“NOX”/> <ParameterList name=“TimeStepNewtonNonlinearSolver”> … </ParameterList> <ParameterList name=“NOX”> <Parameter name=“Linear Solver” type=“string” value=“Stratimikos”/> <ParameterList name=“Stratimikos”> <Parameter name="Linear Solver Type" type="string" value=“Belos"/> <ParameterList name="Linear Solver Types"> <ParameterList name=“Amesos”> … </ParameterList> <ParameterList name=“AztecOO”> … </ParameterList> <ParameterList name=“Belos”> … </ParameterList> </ParameterList> <Parameter name="Preconditioner Type" type="string" value="ML"/> <ParameterList name="Preconditioner Types"> <ParameterList name="Ifpack"> … </ParameterList> <ParameterList name="ML"> … </ParameterList> <ParameterList name=“Meros"> … </ParameterList> </ParameterList> </ParameterList> </ParameterList> </ParameterList> </ParameterList> Lin. Solver Time Stepper Nonlinear Solver Linear Solver Precond.

10. User Requirements for use of Parameter Lists • Allow specification of as many parameters as possible from an input file • i.e. XML format or some other format • Get back default values of parameters after a run for parameters not set by user • e.g. NOX returns the default values of parameters • Misspelling a parameter or sublist should result in a helpful error message and will not be silently ignored • Make it easy change choices and sub-parameters back and forth easily • Make it easy to find documentation and for the accepted parameter values

11. Developer Requirements for use of Parameter Lists • Make it easy to extract parameters and their values • Make it easy to validate parameters, sublists, and parameter values • Allow flexibility in how validation is done • Make it easy to avoid mistakes

12. Outline Developer Interaction with the Parameter Lists

13. Standardized Approach for Object Handling of Parameter Lists Namespace Teuchos { // Base class mix-in interface for objects that accept parameter lists class ParameterListAcceptor { public: // Set parameters from a parameter list and return with default values virtual void setParameterList( RCP<ParameterList> paramList ) = 0; // Get the parameter list that was set using setParameterList() virtual RCP<ParameterList> getParameterList() = 0; // Unset the parameter list that was set using setParameterList() virtual RCP<ParameterList> unsetParameterList ()=0 // Return a const parameter list of all of the valid parameters virtual RCP<const ParameterList > getValidParameters() const; ... }; } Teuchos::ParameterListAcceptor • Parameter list objects (not just parameter values) are “set” and remembered by the object! • Parameters may only be read later during a solve • Parameter list objects are non-const and can be augmented with default option values, output lists … • The “valid parameters” parameter list can be used for automatic documentation and for external validation is some cases … • Many Thyra interfaces inherit from Teuchos::ParameteListAcceptor: • e.g. Teuchos::LinearOpWithSolveFactoryBase, Teuchos::PreconditionerFactoryBase

14. Parameter List Validation namespace Teuchos { class ParameterList { public: ... void validateParameters( ParameterList const& validParamList ,int const depth = 1000 ,EValidateUsed const validateUsed = VALIDATE_USED_ENABLED ,EValidateDefaults const validateDefaults = VALIDATE_DEFAULTS_ENABLED ) const; ... }; } • A valid parameter list is compared against *this parameter list • Parameters and sublists can be validated to any level desired => Gives control over how validation is done

15. Recent Additions to the Teuchos::ParameterList class namespace Teuchos { class ParameterList { public: ... template<typename T> void set( string const& name ,T const& value ,std::string const& docString = "" ,RCP<const ParameterEntryValidator> const& validator = null ); ... }; } • Parameter Entry documentation: Optional (multi-line) documentation string • Parameter Entry Validators: Used in PL::validateParameters(…) the type and/or value of a parameter! namespace Teuchos { class ParameterEntryValidator { public: // Print documentation for this parameter. virtual void printDoc( std::string const& docString ,std::ostream & out ) const = 0; // Return an array of strings of valid values if applicable. virtual Teuchos::RefCountPtr<const Array<std::string> > validStringValues() const = 0; // Validate a parameter entry value and throw exception if validation fails. virtual void validate( ParameterEntry const& entry ,std::string const& paramName ,std::string const& sublistName ) const = 0; }; }

16. Constructing a “Valid” Parameter List // Setup static objects that define parameters const std::string AztecSolver_name = "Aztec Solver"; const Teuchos::RefCountPtr<Teuchos::StringToIntegralParameterEntryValidator<int> > aztecSolverValidator = Teuchos::rcp( new Teuchos::StringToIntegralParameterEntryValidator<int>( Teuchos::tuple<std::string>("CG","GMRES","CGS","TFQMR","BiCGStab","LU") ,Teuchos::tuple<int>(AZ_cg,AZ_gmres,AZ_cgs,AZ_tfqmr,AZ_bicgstab,AZ_lu) ,AztecSolver_name ) ); const std::string AztecSolver_default = "GMRES"; ... Teuchos::RefCountPtr<Teuchos::ParameterList> validAztecOOParams; ... // Create valid parameter list Teuchos::RefCountPtr<const Teuchos::ParameterList> getValidAztecOOParameters() { ... RefCountPtr<ParameterList> pl = validAztecOOParams; if(pl.get()) return pl; pl = validAztecOOParams = rcp(new ParameterList()); // pl->set( AztecSolver_name,AztecSolver_default ,"Type of linear solver algorithm to use." ,aztecSolverValidator ); .. // return pl; } See: Trilinos/packages/aztecoo/thyra/src/AztecOOParameterList.cpp

17. Reading from a User’s Parameter List // Setup static objects that define parameters const std::string AztecSolver_name = "Aztec Solver"; const Teuchos::RefCountPtr<Teuchos::StringToIntegralParameterEntryValidator<int> > aztecSolverValidator = Teuchos::rcp( new Teuchos::StringToIntegralParameterEntryValidator<int>( Teuchos::tuple<std::string>("CG","GMRES","CGS","TFQMR","BiCGStab","LU") ,Teuchos::tuple<int>(AZ_cg,AZ_gmres,AZ_cgs,AZ_tfqmr,AZ_bicgstab,AZ_lu) ,AztecSolver_name ) ); const std::string AztecSolver_default = “GMRES"; ... Teuchos::RefCountPtr<Teuchos::ParameterList> validAztecOOParams; ... // Read the parameters (initial validation is performed by client!) void setAztecOOParameters( Teuchos::ParameterList *pl ,AztecOO *solver ) { ... // Aztec Solver solver->SetAztecOption( AZ_solver ,aztecSolverValidator->getIntegralValue(*pl,AztecSolver_name,AztecSolver_default) ); ... #ifdef TEUCHOS_DEBUG // Check to make sure that I did not use the PL incorrectly! pl->validateParameters(*getValidAztecOOParameters()); #endif // TEUCHOS_DEBUG } See: Trilinos/packages/aztecoo/thyra/src/AztecOOParameterList.cpp

18. Outline User Interaction with the Parameter Lists

19. Parameter List and Sublist Structure : Stratimikos <ParameterList name=“Stratimikos”> <Parameter name="Linear Solver Type" type="string" value=“AztecOO"/> <Parameter name="Preconditioner Type" type="string" value="Ifpack"/> <ParameterList name="Linear Solver Types"> <ParameterList name="Amesos"> <Parameter name="Solver Type" type="string" value="Klu"/> <ParameterList name="Amesos Settings"> <Parameter name="MatrixProperty" type="string" value="general"/> ... <ParameterList name="Mumps"> ... </ParameterList> <ParameterList name="Superludist"> ... </ParameterList> </ParameterList> </ParameterList> <ParameterList name="AztecOO"> <ParameterList name="Forward Solve"> <Parameter name="Max Iterations" type="int" value="400"/> <Parameter name="Tolerance" type="double" value="1e-06"/> <ParameterList name="AztecOO Settings"> <Parameter name="Aztec Solver" type="string" value="GMRES"/> ... </ParameterList> </ParameterList> ... </ParameterList> <ParameterList name="Belos"> ... </ParameterList> </ParameterList> <ParameterList name="Preconditioner Types"> <ParameterList name="Ifpack"> <Parameter name="Prec Type" type="string" value="ILU"/> <Parameter name="Overlap" type="int" value="0"/> <ParameterList name="Ifpack Settings"> <Parameter name="amesos: solver type" type="string" value="Amesos_Klu"/> ... </ParameterList> </ParameterList> <ParameterList name="ML"> ... </ParameterList> </ParameterList> </ParameterList> Top level parameters Sublists passed on to package code! Linear Solvers Every parameter and sublist not in red is handled by Thyra code and is fully validated! Preconditioners See Doxygen documentation for Thyra::DefaultRealLinearSolverBuilder!

20. Automatically Generated Parameter List Documentation “Human readable” automatically generated documentation for Stratimikos Linear Solver Type : string = Amesos # Determines the type of linear solver that will be used. # The parameters for each solver type are specified in the sublist "Linear Solver Types" # Valid values: "Belos", "Amesos", "AztecOO" Preconditioner Type : string = ML # Determines the type of preconditioner that will be used. # This option is only meaningful for linear solvers that accept preconditioner factory objects! # The parameters for each preconditioner are specified in the sublist "Preconditioner Types" # Valid values: "None", "Ifpack", "ML" Linear Solver Types -> AztecOO -> Output Every RHS : bool = 0 # Determines if output is created for each individual RHS (true or 1) or if output # is just created for an entire set of RHSs (false or 0). Forward Solve -> Max Iterations : int = 400 # The maximum number of iterations the AztecOO solver is allowed to perform. Tolerance : double = 1e-06 # The tolerence used in the convergence check (see the convergence test # in the sublist "AztecOO Settings") AztecOO Settings -> Aztec Solver : string = GMRES # Type of linear solver algorithm to use. # Valid values: "CG", "GMRES", "CGS", "TFQMR", "BiCGStab", "LU" Convergence Test : string = r0 # The convergence test to use for terminating the iterative solver. # Valid values: "r0", "rhs", "Anorm", "no scaling", "sol" ... ... • Generated (hiearchially) by Thyra::DefaultRealLinearSolverBuilder::getValidParameters()->print(…) • See simple_stratimikos_example.exe --only-print-options --print-readable-format

21. Output of Default Parameter Values Example input parameter (sub)list to Stratimikos (use simple_stratimikos_example.exe) <ParameterList name="Stratimikos"> <Parameter name="Linear Solver Type" type="string" value="AztecOO" /> <Parameter name="Preconditioner Type" type="string" value="Ifpack" /> </ParameterList> Output (augmented) parameter (sub)list <ParameterList> <Parameter name="Linear Solver Type" type="string" value="AztecOO"/> <ParameterList name="Linear Solver Types"> <ParameterList name="AztecOO"> <ParameterList name="Forward Solve"> <ParameterList name="AztecOO Settings"> <Parameter isDefault="true" name="Aztec Solver" type="string" value="GMRES"/> <Parameter isDefault="true" name="Convergence Test" type="string" value="r0"/> ... </ParameterList> <Parameter isDefault="true" name="Max Iterations" type="int" value="400"/> <Parameter isDefault="true" name="Tolerance" type="double" value="1e-06"/> </ParameterList> <Parameter isDefault="true" name="Output Every RHS" type="bool" value="0"/> </ParameterList> </ParameterList> <Parameter name="Preconditioner Type" type="string" value="Ifpack"/> <ParameterList name="Preconditioner Types"> <ParameterList name="Ifpack"> <Parameter isDefault="true" name="Overlap" type="int" value="0"/> <Parameter isDefault="true" name="Prec Type" type="string" value="ILU"/> </ParameterList> </ParameterList> </ParameterList> => Ifpack is not showing its default parameters!

22. Why Parameter Validation is Important • Silently ignoring misspelled parameter names is unacceptable • Many users will have to wade through thousands of lines of out to find warnings • Makes it difficult to write testing software that ensures that parameters are indeed recognized • Almost every other kind of software will step and print error message • e.g. compilers, scripting languages, … • Example: User misspells “Aztec Solver” as “ztec Solver” … A thousand lines of output … Unused parameters: Statimikos -> Linear Solver Types -> AztecOO -> AztecOO Settings -> Aztec Solver : string = GMRES [default] … tens of parameters … … ztec Solver : string = GMRES [unused] … A hundred lines of output … Poor old user has to ask: • Was this option spelled correctly but just not read? • Was this option spelled incorrectly and therefore not used?

23. Error Messages for Improper Parameters/Sublists Example: User misspells “Aztec Solver” as “ztec Solver” <ParameterList> <Parameter name="Linear Solver Type" type="string" value="AztecOO"/> <ParameterList name="Linear Solver Types"> <ParameterList name="AztecOO"> <ParameterList name="Forward Solve"> <ParameterList name="AztecOO Settings"> <Parameter name="ztec Solver" type="string" value="GMRES"/> </ParameterList> </ParameterList> </ParameterList> </ParameterList> </ParameterList> Error message generated from PL::validateParameters(…) with exception: p=0: *** Caught standard exception of type 'N7Teuchos10Exceptions20InvalidParameterNameE' : /home/rabartl/PROJECTS/Trilinos.base/Trilinos/packages/teuchos/src/Teuchos_ParameterList.cpp:352: Throw test that evaluated to true: !validEntry Error, the parameter {name="ztec Solver",type="string",value="GMRES"} in the parameter (sub)list "DefaultRealLinearSolverBuilder->Linear Solver Types->AztecOO->Forward Solve->AztecOO Settings" was not found in the list of valid parameters! The valid parameters and types are: { "Aztec Preconditioner" : string = ilu "Aztec Solver" : string = GMRES … }

24. Error Messages for Improper Parameters/Sublists Example: User specifies the wrong type for “Aztec Solver” <ParameterList> <Parameter name="Linear Solver Type" type="string" value="AztecOO"/> <Parameter name="Preconditioner Type" type="string" value="Ifpack"/> <ParameterList name="Linear Solver Types"> <ParameterList name="AztecOO"> <ParameterList name="Forward Solve"> <ParameterList name="AztecOO Settings"> <Parameter name="Aztec Solver" type="int" value="GMRES"/> </ParameterList> </ParameterList> </ParameterList> </ParameterList> </ParameterList> Error message generated from PL::validateParameters(…) with exception: p=0: *** Caught standard exception of type 'N7Teuchos10Exceptions20InvalidParameterTypeE' : /home/rabartl/PROJECTS/Trilinos.base/Trilinos/packages/teuchos/src/Teuchos_StandardParameterEntryValidators.hpp:279: Throw test that evaluated to true: !validType Teuchos::StringToIntegralParameterEntryValidator::getIntegralValue(...): Error, the parameter {paramName="Aztec Solver",type="int"} in the sublist "DefaultRealLinearSolverBuilder->Linear Solver Types->AztecOO->Forward Solve->AztecOO Settings" has the wrong type. The correct type is "string"!

25. Error Messages for Improper Parameters/Sublists Example: User specifies the wrong value for “Aztec Solver” <ParameterList> <Parameter name="Linear Solver Type" type="string" value="AztecOO"/> <Parameter name="Preconditioner Type" type="string" value="Ifpack"/> <ParameterList name="Linear Solver Types"> <ParameterList name="AztecOO"> <ParameterList name="Forward Solve"> <ParameterList name="AztecOO Settings"> <Parameter name="Aztec Solver" type=“string" value="GMRESS"/> </ParameterList> </ParameterList> </ParameterList> </ParameterList> </ParameterList> Error message generated from PL::validateParameters(…) with exception: p=0: *** Caught standard exception of type 'N7Teuchos10Exceptions21InvalidParameterValueE' : /home/rabartl/PROJECTS/Trilinos.base/Trilinos/packages/teuchos/src/Teuchos_StandardParameterEntryValidators.hpp:260: Throw test that evaluated to true: itr == map_.end() Teuchos::StringToIntegralParameterEntryValidator::getIntegralValue("GMRESS",...): Error, the value "GMRESS" is not recognized for the parameter "Aztec Solver" in the sublist "". Valid selections include: "CG", "GMRES", "CGS", "TFQMR", "BiCGStab", "LU".

26. A few Comments on Parameter List Validation • PL::validateParameters(…) is only a tool for validation, not a “catch-all’ solution • How validation is performed is not important as long as it gets done! • Validation is ultimately the responsibility of the objects that accept and read from a parameter lists • Full validation may not occur until the underlying solver finishes • We should not have to (dramatically) change current parameter list structures • e.g. Ifpack, ML, Amesos, ... • 100% validation may not be possible in all cases

27. Parameter List Issues? • Handling multiple value types for a parameter and validation? • Accepting an string value or an enum value (e.g. Epetra_CombineMode) • In code as: Add, Zero, Insert, InsertAdd, Average, or AbsMax • In XML as: “Add”, “Zero”, “Insert”, “InsertAdd”, “Average”, or “AbsMax” • Allowing for implicit type conversions (e.g. “level-of-fill”) • Input as int, double, float …, Output as int, double, float • Solution => Teuchos::ParameterEntryValidator subclasases! • Handling and validation of non-“parameter” parameters? • Status tests in NOX or null-space vectors in ML • Adding an RCP-wrapped derived class object and getting back base class object? • Solution => Add specialized handling of RCP-wrapped objects for NOX PL class …

28. Parameter List Issues? • Preserving ordering of parameter and sublists? • Issue for Teuchos::ParameterList • Issue for Teuchos::XMLObject • Adding documentation and valid parameter value lists to the XML representation • Reading in sublists from another file (in the XML parsesr) • Broadcasting a parameter list? Non-parameter parameters? … • GUI for setting XML parameters? • Issues Ken mentioned to me: • Case sensitivity? • Accept “symmetric” for “Symmetric”? • Partial matching? • Accept “Sym” for “Symmetric”? • Error messages? • Suggesting correct spellings of parameters? • Suggest “Symmetric” when user inputs “symmetric”? • Any other issues?

29. Parameter List Summary • Consistency in the use and the look-and-feel of parameter lists is needed • User (should) expect parameter lists to be validated • User parameter lists and developer use of parameter lists must be validated! • Many tools for validation • PL::validateParameters( validParamList, ... ) • ParameterListValidator (StringToIntegralParameterListValiator, …)