Parallel computation of pollutant dispersion in industrial sites

1. Julien Montagnier Marc Buffat David Guibert • Parallel computation of pollutant dispersion in industrial sites

2. Motivation Observations • Numerical simulation of pollutant dispersion in industrial sites • Better evaluation of risk than with 1D model dispersion • Efficiency Navier Stokes solver to run parametric studies • Development of a parallel 3D Navier Stokes solver on unstructured meshes. 3D 1D

3. Numerical Methods Properties : • Finite volume on unstructured finite elements mesh. • Incompressible segregated solver with projection methods • Extension to variable density flow with projection on energy equation (Mach Uniformity through the coupled pressure and temperature correction algorithm, 2005 Nerinckx,)‏ Algorithm • Fixed point non linear iteration for each time step with : A (Wk+1 -Wk ) = F(Wk)‏ Parallelization : • Evaluation of fluxes and assembling part (RHS + matrix) parallelized using domain decomposition • Implicit upwind schemes >> Efficient solvers to solve large unstructured sparse linear systems of several millions of dofs. matrix RHS

4. Parallel Linear Solvers • Use of PETSC Krylov subspace iterative methods • Acceleration of convergence with different preconditioning methods (Hypre library) : • parallel ILU / AMG (Algebraic Multigrid Method)‏ • Many way of tunning AMG methods : • Coarsening schemes • Falgout • PMIS, (Parallel Maximal Independent Set) HMIS • Interpolation operation • Classical interpolation • FF, FF1 (De Sterck, Yang : Copper 2005 ; De Sterck 2006)

5. 3D Poisson Equation : Tetrahedral Mesh • Scale up • 1 >> 64 processors • 12,500 >> 400,000 dofs / proc • Speed up 1,000,000 dofs • P2CHPD IBM cluster, with Intel dual quad core processor nodes and Infiniband

6. Scale up results • Bring out 3 groups of preconditioning methods 1) ILU 2) AMG with high complexity coarsening schemes 3) AMG with low complexity coarsening schemes • Better AMG scale up with low complexity coarsen schemes • Krylov with AMG preconditioning + FF1 interpolation give the best scale up. • (500 x faster than ILU)‏

7. Beware the problem size ! • On the IBM cluster, scalability is good from 200,000 dofs / proc • With lower dofs, too much communication cause a loss in scalability

8. Speed Up on 1,000,000 dofs Speed up • PMIS-FF1 give the best results • On 32 processors • 10 % faster than PMIS FF, • 270 % faster than Falgout classical • 500 % faster than ILU • Efficiency collapse over 16 processors • (62,500 dofs / procs)‏ No. of procs

9. Real case study • PMIS – FF1 • Real geometry. • Application on meshes : 5 M of cells, • 30 M of dofs • Scalar transport equation

10. Parallel efficiency on Navier Stokes Problem • Assembling time : 30% total time • Parallelization of matrix assembling and RHS assembling perform well • Parallelization of linear solver perform well but depends on problem size

13. Conclusion • Objective : Build a new efficient parallel Navier Stokes solver • Laplacian equation : Low complexity scheme PMIS with FF1 interpolation gives the best results (speed up, scale up, simulation times)‏ • 500 times faster than ILU preconditioning methods • Navier Stokes problem on 5M cells mesh run in 6 hours on 64 processors. • Good speed up on 5M cells mesh up to 64 processors. • Communications in linear solver process limits speed up