The TSTT Interface for Mesh-Based Simulations

1. The TSTT Interface for Mesh-Based Simulations • A.C. Bauer, M.S. Shephard, E. Seol and J. Wan, • (acbauer@scorec.rpi.edu) • Scientific Computation Research Center • Rensselaer Polytechnic Institute, Troy, NY 12180 • Outline: • TSTT simulation infrastructure development • Structures to support the construction of adaptive loops • Mesh modification tools for adaptive mesh control • Adaptive loop for DOE accelerator modeling code • Adaptive loop for evolving geometry problem solved using commercial analysis engine

2. Terascale Simulation Tools & Technologies (TSTT) Center Goal: To enable high-fidelity calculations based on multiple coupled physical processes and multiple physical scales • Adaptive methods • Composite or hybrid solution strategies • High-order discretization strategies • Barrier: The lack of easy-to-use interoperable meshing, discretization and adaptive tools requires too much software expertise by application scientists • Organizations: • DOE Labs: Argonne, Brookhaven, Lawrence Livermore, Oak Ridge, Pacific Northwest, Sandia • Universities: Rensselaer Polytechnic Institute, SUNY Stony Brook, University of British Columbia • Project Managers: • David Brown: dlb@llnl.gov • Lori Freitag Diachin: diachin2@llnl.gov • Jim Glimm: glimm@ams.sunysb.edu • More information: www.tstt-scidac.org TSTT

3. TSTT Interoperability Goal To provideinterchangeable and interoperable access to different mesh management and discretization strategies • Ease experimentation with different technologies • Combine technologies together for hybrid solution techniques A different way of thinking about simulation code development Accomplishing this goal requires: • An abstract data model that encompasses a broad spectrum of mesh types and usage scenarios • A set of common interfaces • Implementation and data structure neutral • Small enough to encourage adoption • Flexible enough to support a broad range of functionality Geometry Information (Level A) Full Geometry Meshes (Level B) Mesh Components (Level C)

4. TSTT Simulation Infrastructures • 7 Interacting structures • Mathematical problem definition 1) Space/time domain *** 2) Physical and mathematical model attributes • First level of discretization 3) Mesh *** • Second level of discretization 4) Discretization operators acting on mathematical equations and degrees-of-freedom (dof) • Application of #4 over mesh entities 5) Algebraic system contributors • Solving for dof values 6) Algebraic system • Description of the physical parameters 7) Fields *** *** Main TSTT focus to date

5. Space/Time Domain Overall domain definition - geometric model and time domain • Want a “true definition” of the domain - not a specific approximation • Define other structures in relation to the domain definition • Problem attributes defined over portions of the domain • Mesh represents discretization of domain • Fields are discretizations of physical tensors defined over the domain • At SCOREC, direct geometric model access via topology • Shape independent abstraction • Need general combination of solids, surfaces and curves • Non-manifold topology representations known • Effectively supported by commercial solid modelers • Topology built from topology in modeling source • Geometric queries passed through modeler API - supports auto-meshing

6. Attributes • Problem definition attributes • Governing equations, material properties, loads, b.c.’s • Solution strategy attributes • Meshing parameters, PDE discretization, basis functions, etc. • Attributes applied to geometric model • Problem definition attributes: • Are tensors • Have distributions • Have relationships • Classification used to retrieve attributes

7. TSTT Mesh Data Model • Data model consists of entities and entity sets • Entity definition • Unique topology • Canonical ordering defines adjacency relationships • Entity set definition • Arbitrary grouping of TSTT mesh entities • There is a single “Root Set” • Relationship among entity sets • Contained-in • Hierarchical • Mesh Definition • Simple meshes are a set of connected, non-overlapping entities related through adjacency relationships • Meshes can contain collections of simple meshes

8. TSTT Mesh Interface Functionality • Core functionality • The smallest set of functions required to be considered TSTT compliant – extensions for implementation of advanced functionality • Provides basic access to vertex coordinates and adjacency information • Mesh loading and saving • Accessing global information such as the root set, geometric dimension, number of entities of a given type or topology • Primitive array access for entity sets • Global entity arrays for entity sets • Traversal functionality • Advanced traversal functions provide additional flexibility and performance • Single Entity Traversal • Iterator functionality (init, next, reset, end) • Query (topology, type, adjacency) • Workset Traversal • Block iterators allow access of entities in a user-defined array size • Array based query functions for topology, type, adjacency • Increases performance by reducing the number of calls through the interface • Interface is not as intuitive

9. TSTT Mesh Modification Operators • Simple mesh modification operations are also supported • Modifications • Changing vertex location • Add and delete entities • Caveats • No validity checks • Requires simple classification mechanisms against the geometric model • Intended to support higher-level functionality • Mesh quality improvement • Adaptive schemes with validity checking • Front tracking procedures • Basic mesh generation algorithms

10. Domain Discretization and Mesh Representation • Geometric model discretized into a mesh • Discretized geometric domain defined as “union” of the mesh entities • Maintain link to shape information • Support the field’s distributions used in equation discretization • Maintain the dof multiplying the distributions and support assembly of global system • Mesh topology provides flexible support for these functions • Mesh entities, Md, of dimension d • Mesh entities are constructed from lower dimensional mesh entities • e.g. A face is constructed from edges which are constructed from vertices • Entities are adjacent if one is used to construct the other • Typical complete mesh topological structures:

11. Mesh/Model Relationship • Relationship is termed “classification” • Mesh classification: unique association of a mesh entity, Midi, to a geometric model entity, Gjdj, where di≤dj is denoted by • Midi Gjdj indicates the left-hand entity (or set) represents a portion of the right-hand entity in the discretization Multiple Midi classified on a Gjdj • Boundary mesh entities are identifiedin terms of their classifications • Classification critical to supporting adaptive simulations and high level problem definitions • Reverse classification is the set of equal order mesh entities classified on a specific model entity

12. Simulation Fields • The discretization of the tensors used in the PDE in terms of distribution functions times degrees of freedom written over mesh entities • Field definition • Tensor qualification - order, symmetries, spatial dimension, coordinate system, constraints • Distribution functions - equations, mesh entities they act over, dof used • dof - mesh entities they are on, values after evaluation * Typical but not unique form

13. Fields Relationship With Other Simulation Structures DOF locations on associated entities • Relationship with the space/time domain • Tensor qualifiers specified over the space/time domain • Specifies the spatial domain dimension for the tensor • Constraints (e.g. boundary conditions, initial conditions, divergence free/solenoidal field, etc.) can be specified with attributes • Relationship with the mesh • Information used to discretize the tensor specified over the mesh • Distributions -- defined over mesh entities that are the same dimension as the tensor dimension • DOFS -- associated with mesh entities that are the dimension of the distribution mesh entities and lower 5 point FD stencil FV with 1 distribution per element C0 FEM quadratic distributions

14. Functionality of Field Library Locally adapted mesh • Load, save, create and delete functions • Point-wise interrogation functions using combinations of operators • Tensor operators • Addition • Multiplication • Transpose • Differential operators • Identity • Derivatives (x,y,z,t) • Gradient • Integral functions • Domain defined through TSTT entity sets and/or model entities • Uses point-wise interrogation operators for integration of general quantities • Mapping fields between meshes • Requires mesh information and above operations • Curl • Divergence • Laplacian Meshes sharing a boundary

15. TSTT Interface Implementations Implementations are software libraries that can be used through the TSTT interface • Implementations are required to contain a core functionality • Includes extensions allowing certain software requirements to be met • Multiple implementations can be used simultaneously • Tests to verify each implementations compliance with TSTT interface standard • Implementations of the TSTT Mesh Interface currently available at https://svn.scorec.rpi.edu/wsvn/TSTT • FMDB - http://www.scorec.rpi.edu/AOMD • FronTier - http://frontier.ams.sunysb.edu • GRUMMP - http://tetra.mech.ubc.ca/GRUMMP • MOAB - http://cubit.sandia.gov/MOAB • NWGrid - http://www.emsl.pnl.gov/nwgrid/index_nwgrid.html

16. SCOREC Involvement With TSTT • Development of interface standards • TSTT Mesh Interface • TSTT Geometry Interface • TSTT Fields Interface • Software development • FMDB (Flexible Mesh DataBase) library which is compliant with the TSTT Mesh Interface • FANS (Field Approximation for Numerical Simulations) library which is being developed in conjunction with development of the TSTT Fields Interface • Model library which is being developed in conjunction with development of the TSTT Geometry Interface • Notable uses of software • Development of adaptive loops for mesh-based simulations • Multiphysics simulations

17. Constructing Adaptive Solution Procedures • Components of an an adaptive mesh control loop • High level problem definition (domain and attributes) • Domain discretization (mesh generation) • Equation discretization • Solvers to deal with the large systems of equations • Error estimators/indicators • Correction indicators • Spatial and/or equation discretization enrichment methods • Implementation • Tightly coupled using a single set of structures • Advantage: Computationally efficient if done well • Disadvantage: Complex algorithm and code development • Loosely coupled building on existing components • Advantage: Ability to use existing analysis codes and adaptive tools and to mix and match tools • Disadvantage: Overhead of multiple structures and data conversion

18. Structures/Services for Adaptive Loops • Information needed by an adaptive loop for proper creation of input for a code • Overall Domain Definition - Geometric Model • Want a “true definition” of the domain - not a specific approximation • Physical Attributes • Sets of tensors needed to qualify the loads, material properties, boundary conditions and initial conditions • Domain Discretizations (meshes) • First piece of the two level discretization that is input to analysis codes • Simulation Fields Discretizations • The discretized versions of tensor variables defined in terms of distributions and degrees of freedom over the meshes as dictated by the PDE discretization process performed within the analysis code • The goal of the TSTT development is to provide all of the above information in a uniform manner

19. Adaptive Loop Driver Code Input and Output • Input • Domain definition • Mathematical problem attributes • Adaptive loop attributes/apriori mesh controls • Adaptive goal • Geometry approximation controls for mesh • Mesh size limits (memory, computation constraints) • a posteriori mesh controls • Field information • Error estimates • Convergence rates • Adaptive strategies • Model modifications through mesh motion • Output • Simulation code input • Adaptive loop convergence information • Mesh

20. Simulation Code Input and Output • Input in form manageable by simulation code • Sufficient mesh representation • Constructed from complete mesh representation from interface • Usually only nodes and elements • Boundary and possibly initial condiitons • Element “sides” and nodal boundary conditions from classification/reverse classification information • Tensor values from field and attributes • Other problem attributes • From complete problem definition • Ideally code input files do not change • Output • Computed field information • May include error estimates/indicators • Field information from visualization output may be sufficient • Ideally code output files do not change

21. Adaptive Mesh Control Need adaptive mesh control when • Discretization is inadequate - mesh refinement • Discretization computationally inefficient - mesh coarsening • Element shape becomes unacceptable – mesh improvement Two approaches for h-type mesh adaptation • Re-meshing • Define new mesh size field • Provide mesh generator with domain definition and mesh size field • Map any history dependent solution fields from the old to the new mesh • Local mesh modifications • Determine local operations needed to perform desired modifications • Execute the local modifications • Apply incremental history dependent solution field updates

22. Mesh Modification Operators • Single step operators • Face swap • Edge swap • Edge collapse • Region collapse • Edge split • Vertex motion • Curving interior mesh entity • Compound operators modify two or more entities to allow the modification of the problem entity • Examples of two step compound operators • Swap entity A first, and then swap entity B • Collapse entity A first, and then collapse entity B • Swap entity A first, and then collapse entity B • Swap entity A first, and then curve new entity B • Swap entity A first, and then reposition vertex B • Etc.

23. Accounting for Curved Domains During Refinement • Moving refinement vertices to boundary required during mesh modification (see IJNME paper, vol58 pp247-276, 2003 ) • Coarse initial mesh and the mesh after multiple refinement/coarsening • Operations to move refinement vertices x y z

24. Field Transfer on Adapted Meshes Edgemarked for collapse Zone updated by the operation Mesh before collapse Mesh after collapse Two approaches • Global • Computationally expensive • Procedures tend to diffuse information - accuracy loss • Local • Performed with each local mesh modification • Limited number of elements involved - efficient • No accuracy loss with some operations, others easier to control due to local nature

25. Adaptive Loop for Accelerator Design • Collaboration with SLAC (I. Malik, K. Ko and Z. Li) • Complex CAD geometry • Physics modeling by the Omega3P code from SLAC • Solves Maxwell’s equations in frequency domain • Determine power loss for specific modes • High level modeling accuracy needed • E.g., 0.1% error in frequency predictions • Adaptive mesh control needed to provide desired accuracy • Adaptive loop constructed using unmodified Omega3P TSTT

26. Providing the Adaptive Loop Components • Components added to construct adaptive loop • High level problem definition • Automatic mesh generation directly from CAD • a-posteriori error indication and convergence rate to define mesh size field • a-posteriori error indicator constructed from knowledge of the tensor and its discretized field • General mesh modification to construct requested mesh size field accounting for the CAD geometry • Approach to the coupling of the components • Interoperability procedures to provide information needed by components • Geometry - integration to multiple CAD systems • Mesh - integration to various mesh generation and modification procedures • Field - integration with different analysis procedures • Interoperability procedures used in this example from • TSTT • SCOREC • Simmetrix

27. Integration of Components

28. Operator Interfaces Using TSTT field operatorsallows easy construction of alternative error estimators Using geometry operatorsmeans alternate solid modelers can be inserted Using TSTT mesh operatorsmeans alternate mesh generatorsand mesh adaptation procedurescan be inserted Mesh adaptation based on local modification linkeddirectly to CAD Unaltered SLAC code Projection-based error estimatorused to construct new mesh sizefield given to mesh modification Error estimatorsfrom RPI and SLAC

29. Adaptive Results for Trispal Model Distribution of Wall-loss Level 1 Level 2 Level 0 Frequency ConvergenceQ Convergence

30. Metal Forming Simulation Large plastic deformation • Meshes often become invalid • Evolving geometry • Evolving contact Components of automated simulation • Commercial analysis engine (DEFORMTM) • Monitoring of mesh discretization errors and element shapes • Mesh model topology update • Construct mesh size field based on discretization errors and geometry approximation • General mesh modification to obtain the desired mesh size field • Adjust mesh size and shape to control geometric approximations • Local field transfer

31. Model Topology Update Geometric components in forming simulations • Workpiece • Dies • Die motions Model topology needs to be updated • Contact conditions change as simulation proceeds • Mesh updates require complete model topology • Simulation engine tracts only nodal contact • Must update model topology based on this information • Model update procedure • Maintain non-manifold model representation • Simulation contact information and “mesh geometry” • Uses initial classification to build topology and then corrects ambiguities • Mesh classified against updated model topology • Mesh modifications controlled • Attributes properly associated to mesh

32. Steering Link Problem • Adapted based on the error indicators on the effective strain

33. Summary of TSTT Work • TSTT Mesh 0.6 Interface complete • Working document describing the mesh and geometry data models as well as the overall vision • Implementations of TSTT Mesh 0.6 Interface available • https://svn.scorec.rpi.edu/wsvn/TSTT • FMDB - http://www.scorec.rpi.edu/AOMD • FronTier - http://frontier.ams.sunysb.edu • GRUMMP - http://tetra.mech.ubc.ca/GRUMMP • MOAB - http://cubit.sandia.gov/MOAB • NWGrid - http://www.emsl.pnl.gov/nwgrid/index_nwgrid.html • Preliminary interfaces for geometry, field data, and mesh/geometry data model manager • C, C++, and Fortran language interoperability through SIDL/Babel (CCA) • Analyzing performance ramifications of SIDL/Babel language interoperability tools (joint with the CCA) • TSTT Information • http://www.tstt-scidac.org

34. Closing Remarks on Adaptive Loops • Adaptive loops built from available components • Software for multiple sources combined to create functional adaptive loops that provide superior results • Initial SLAC adaptive loop constructed in less than 3 months • By people with little background with components • Included development of an error estimation and correction indication routines • Approach taken focused on optimizing time to getting adaptive results (not the computing resources needed for the simulation) • Did not alter Omega3P or DEFORM codes • Used meshing components and operators to create input files • Used field operators to extract results from output file