An Analysis of ASPECT Mantle Convection Simulator Performance and Benchmark Comparisons

1. An Analysis of ASPECT Mantle Convection SimulatorPerformanceand Benchmark Comparisons Eric M. Heien[1], Timo Heister [2], Wolfgang Bangerth[2], Louise H. Kellogg[1] 1) Department of Geology, University of California, Davis CA 95616 USA 2) Texas A&M University, College Station TX 77843 USA • Abstract • Due to the inaccessible nature of the deep earth, simulations are a key tool in understanding long term mantle dynamics. The associated technical challenges are substantial and include highly variable viscosities, fracture and shear zone deformation, and multiple interacting length and time scales. To better model, simulate and understand the processes involved, multiple simulators have been developed over the past several decades. ASPECT (Advanced Solver for Problems in Earth’s ConvecTion) is a recently developed mantle convection simulation code with support for many advanced features including adaptive mesh refinement, high performance solvers, strang parallel scalability and easy extensibility. • ASPECT is available at http://dealii.org/aspect/ Adaptive Mesh Refinement Already at t=5Myrs we see problems with low refinement. RL=3 loses the hot material plume near the surface and fails to resolve submerging flows between the upwellings. At this point, RL=5 and RL=7 are nearly indistinguishable. Fixed Mesh (Level = 7) Adaptive Mesh (Max Level 7) Using adaptive mesh refinement coarsens the mesh where high detail is not required. The effect on simulation results is examined below. The scaling for a shell model in 3D is similar to 2D, with strong scaling up to 1e5 DOF per core. The main issue with large scale simulations is the initialization time as shown above. By 256 cores initialization requires 30 minutes. Extrapolation indicates a simulation using 12,000 cores (4.8e8 DOF at optimal scaling) would take 1 day to initialize – longer than the allowed run time on Ranger. At t=60Myrs, RL=3 has obvious divergence from higher resolutions and fails to model secondary rising plumes. RL=5 and RL=7 still resolve features like small uprising plumes, but RL=5 loses some small scale features such as vortices and accurate heat modeling in the plume caps. Fixed Mesh (2.57e6 DOF) AMR (7.98e5 DOF) At t=50Myrs, there is little difference between the fixed mesh and AMR mesh though the AMR mesh has 1/3 the DOFs potentially allowing for faster solution. Scaling of 2D and 3D Simulations All simulations described here were run on the TACC Ranger system. This system consists of 16-way SMP compute nodes, each with 32GB of RAM and connected with an InfiniBand network. Effect of Mesh Refinement An under resolved mesh in the simulation can cause unphysical phenomenon. To examine this, we performed simulations with the same physical parameters and initial conditions using different mesh resolutions. The figures below show the initial mesh and simulation state with a spherical temperature perturbation. Fixed Mesh (2.57e6 DOF) AMR Mesh (8.81e5 DOF) By t=900Myrs, RL=3 is still divergent. RL=5 has lost feature symmetry with near-surface downwellings merging over time and creating further instability. RL=7 has maintained near perfect symmetry though apparent instabilities have appeared. At t=500Myrs, there is clear divergence though the basic structure is the same. Based on these results, it appears refinement levels of 3 or lower in ASPECT cause inaccurate modeling of mantle plumes and second order effects. AMR causes divergence of results but not significant change inplumes or similar phenomenon. Results of strong scaling tests for a 2D shell with 2.56e6 degrees of freedom are shown above. All aspects of computation scale well to 64 cores, or more specifically 5e4 to 1e5 degrees of freedom (DOF) per core. For reference, a 1Gyr simulation at RL=3 took 19 minutes on 16 cores, RL=5 took 134 minutes on 64 cores and RL=7 took 8 hours on 128 cores. Refinement Level = 3 (7.68e2 cells, 1.07e4 degrees of freedom) Refinement Level = 5 (1.23e4 cells, 1.62e5 degrees of freedom) Refinement Level = 7 (1.96e5 cells, 2.57e6 degrees of freedom)