High-Fidelity Terascale Simulations of Turbulent Combustion

1. High-Fidelity Terascale Simulations of Turbulent Combustion Jacqueline H. Chen, Evatt R. Hawkes, Ramanan Sankaran, James C. Sutherland, and Joseph C. Oefelein Combustion Research Facility Sandia National Laboratories Livermore CA Supported by Division of Chemical Sciences, Geosciences, and Biosciences, Office of Basic Energy Sciences, and DOE SciDAC Computing: LBNL NERSC, ORNL CCS/NLCF, PNNL MSCF, SNL HPCNP Infiniband test-bed, SNL CRF BES Opteron cluster Visualization: Kwan-Liu Ma and Hiroshi Akiba UC Davis

2. Outline • DNS of turbulent combustion – challenges and opportunities • Sandia S3D DNS capability – terascale simulations on Office of Science platforms • New combustion science to advance predictive models: • 3D simulations of turbulent jet flames with detailed chemistry. • Layering Large Eddy Simulation and DNS approaches. Vorticity fields in DNS of a turbulent jet flame (volume rendering by Kwan-Liu Ma and Hiroshi Akiba)

3. Turbulent combustion is a grand challenge! • Stiffness : wide range of length and time scales • turbulence • flame reaction zone • Chemical complexity • large number of species and reactions (100’s of species, thousands of reactions) • Multi-Physics complexity • multiphase (liquid spray, gas phase, soot, surface) • thermal radiation • acoustics ... • All these are tightly coupled Diesel Engine Autoignition, Soot Incandescence Chuck Mueller, Sandia National Laboratories

4. O(4) Range Continuum O(4) Several decades of relevant scales • Typical range of spatial scales • Scale of combustor: 10 – 100 cm • Energy containing eddies: 1 – 10 cm • Small-scale mixing of eddies: 0.1 – 10 mm • Diffusive-scales, flame thickness: 10 – 100 m • Molecular interactions, chemical reactions: 1 – 10 nm • Spatial and temporal dynamics inherently coupled • All scales are relevant and must be resolved or modeled Terascale computing: ~3 decades in scales (cold flow)

5. Increasing cost and resolution for fixed physical problem Combustion CFD Approaches to tackledifferent length scale ranges • Reynolds–Averaged Navier–Stokes (RANS) • Coarse meshes, full range of dynamic scales modeled • Empirical closure, bulk approximation, current engineering CFD • Large Eddy Simulation (LES) • Energetic scales resolved, sub-grid scale dynamics modeled • combustion closure required, captures large-scale unsteadiness • Direct Numerical Simulation (DNS) • No sub-grid modeling required but limited on range of scales • Building-block configurations, research tool

6. Oxidizer Fuel HO2 CH4 CH3O O Role of Direct Numerical Simulation (DNS) • A tool for fundamental studies of the micro-physics of turbulent reacting flows • A tool for the development and validation of reduced model descriptions used in macro-scale simulations of engineering-level systems • Physical insight into chemistry turbulence interactions • Full access to time resolved fields DNS Engineering-level CFD codes (RANS and future LES) Physical Models DNS Piston Engines

7. S3D code characteristics: Solves compressible reacting Navier-Stokes F90/F77, MPI, domain decomposition. Highly scalable and portable 8th order finite-difference spatial 4th order explicit RK integrator hierarchy of molecular transport models detailed chemistry multi-physics (sprays, radiation and soot) from SciDAC “Terascale Simulation of Turbulent Combustion” 70% parallel efficiency on 4096 processors on NERSC; 90% parallel efficiency on 5120 CrayXT3 processors; 75% parallel efficiency on 1000 CrayX1E processors Terascale computations => need scalar optimization customized to architecture CCS, NERSC consultants S3D MPP DNS capability at Sandia S3D is a state-of-the-art DNS code developed with 13 years of BES sponsorship. S3D scales up to 1000s of processors… and beyond?

8. S3D performance improvements on Seaborg Application Software Case Studies in FY05 for MICS (K. Roche) Using Xprofiler, IPM and working with David Skinner: • 8% improvement in performance by minimizing conversion between dimensional and nondimensional units for interfacing with Chemkin • interface to Mathematical Acceleration Subsystem (MASS) library for exponential and logarithmic functions resulted in 10% improvement. Another 5% due to using vectorized exp function in VMASS • Tabulating Gibbs energy as a function of temperature resulted in 8% savings • Remove excessive dynamic allocation calls resulted in 12% savings. • Eliminate MPI derived data types for communicating noncontiguous boundary data with neighbors 45% efficiency improvement

9. Outline • DNS of turbulent combustion – challenges and opportunities • Sandia S3D DNS capability – terascale simulations on Office of Science platforms • New combustion science to advance predictive models: • 3D simulations of turbulent jet flames with detailed chemistry • Layering Large Eddy Simulation and DNS approaches Vorticity fields in DNS of a turbulent jet flame (volume rendering by Kwan-Liu Ma and Hiroshi Akiba)

10. FY05 INCITE award: Direct simulation of a 3D turbulent nonpremixed CO/H2/air jet flame with detailed chemistry (350 million grids, 12 species) • Objectives: • Study fine-grained coupling between turbulent mixing and finite-rate chemical effects • Extinction-reignition dynamics • Mixing time scales • Reynolds number effect • Preferential diffusion • Flow unsteadiness • Test assumptions in subgrid mixing and reaction models z x y x INCITE run test case (1/2 resolved) 350 million grid points, Re=5000 Isoscalar surface of scalar dissipation rate (local mixing rate)

11. The effect of chemistry-turbulence interactions in turbulent nonpremixed jet flames: mixing of passive and reacting scalars Evatt Hawkes, Ramanan Sankaran, James Sutherland, and Jacqueline Chen Computational Platforms: NERSC Seaborg, ORNL CCS Phoenix Cray X1E, XT3 and PNNL MPP2

12. Air Mixing, Reaction Fuel Mixing, Reaction Air Description of runs- Temporally evolving non-premixed plane jet flames • A canonical flow with shear-driven turbulence x y Heat release iso-contours

13. Large computing allocations are enabling new science runs INCITE at NERSC, capability computing at NLCF ORNL, ERCAP at PNNL, NERSC Detailed H2/CO chemistry (17 d.o.f., Li et al. 2005) Parameters selected to maximize Re Case A: Re 6000 40 million grid points 480 processors on PNNL MPP2 ~ 1 TB data Case B: Re 8000 100 million grid points 1728 processors on Seaborg 192/240 processors on ORNL CCS Cray X1/E ~ 2.5 TB data INCITE calculation: Currently running on Seaborg Re 5000 350 million grid points ~4096 Seaborg processors 3 million hours total ~ 9TB raw data Two additional calculations, parametric study in Re Re 2500, 150 million grid points Re 10,000, 500 million grid points DNS data-sets of turbulent nonpremixed H2/CO flames (mixing, extinction/reignition)

14. Community data sets • New opportunities for numerical benchmarks – highly resolved LES and DNS • TNF workshop (1996-2004): International Collaboration of Experimental and Computational Researchers • Framework for detailedcomparison of measured and modeled results (http://www.ca.sandia.gov/TNF/abstract.html)

15. Non-premixed combustion concepts • Mixture fraction Z: the amount of fluid from the fuel stream in the mixture • Z is a conserved (passive) scalar – (no reactive source term) • Scalar dissipation, a measure of local molecular mixing rate:

16. Example development of the jet:40 million grid PNNL run • Left: Vorticity. Right: Simultaneous volume rendering of mixture fraction, scalar dissipation and OH radical. (Rendering by Hiroshi Akiba and Kwan-Liu Ma, UC Davis)

17. 100 million grid run on NERSC Seaborg and ORNL Cray X1: Re = 8000 Vorticity Scalar Dissipation

18. 100 million grid run on NERSC Seaborg and ORNL Cray X1: Re = 8000 HO2 dissipation OH dissipation

19. Mixing timescales • Models for molecular mixing are required in the PDF approach to turbulent combustion (Pope 1985), a sub-grid model used in RANS and LES approaches. • TNF workshop – CFD predictions are dependent on mixing timescale choice. • Models assume that scalar mixing timescales are identical for all scalars and determined by the turbulence timescale. • scalars with different diffusivities? • reactive scalars?

20. is assumed to be order unity in most models is assumed to be the same for all scalars Definitions • Mechanical time-scale: • Scalar time-scale: • Time-scale ratio:

21. Mixture fraction to mechanical timescale ratio • Confirmation that mixture fraction to mechanical time scale ratio is order unity. • Average value about 1.5, similar to values reported by experiments, simple chemistry DNS, and used successfully in models. Timescale Ratio rZ

22. Effect of diffusivity Increasing diffusivity rφ • Smaller, more highly diffusive species do have faster mixing timescales

23. Finite-rate chemistry effects on mixing rφ • HO2 and H2O2 have faster mixing times in the middle of the simulation, while OH and O are lower • Diffusivity trend does not appear to hold for HO2 and H2O2 versus O and OH. • What is going on?

24. Radical production and destruction in high dissipation regions Color scale: mass fraction (blue is low; red is high) White iso-contours: scalar dissipation rate OH • OH is consumed while HO2 is produced in high dissipation regions HO2

25. Dissipation of passive and reactive scalars • Blue: Z, Green: OH, Red: HO2 • Dissipation fields of Z and HO2 are co-incident and aligned with principal strain directions • OH dissipation occurs elsewhere, more in the centre of the jet • OH is reduced in the high dissipation regions, leading to longer mixing times, opposite for HO2 • At later times, mixing rates relax, OH returns and HO2 decreases

26. 100 million grid run on Seaborg / Cray X1 • We need to establish any Re dependence • Need parametrics at larger Re • Run in progress… but seems to be showing same effects

27. Conclusions - mixing timescales • New finding: detailed transport and chemistry effects can alter the observed mixing timescales • Models may need to incorporate these effects • a poor mixing model could lead to incorrectly predicting a stable flame when actually extinction occurs • This type of information cannot be determined any other way at present • ambiguities in a-posteriori model tests • too difficult to measure • need 3D and detailed chemistry to see this

28. Knowledge Discovery From Terascale Datasets • Challenge:Need interactive knowledge discovery software for terascale high-fidelity scientific data sets where computing pipeline is distributed geographically. • Solution:Intelligent visualization pipeline • Simultaneous cognitive visualization • Intelligent feature extraction/tracking • Scalable transparent data sharing and parallel I/O across platforms INCITE Award: DNS of a 3D turbulent flame (9 TB raw data - 350 million grids, 17 variables, 100,000 steps)

29. Advanced post-processing for extinction/re-ignition • Extinction viewed as radical or reaction-rate “hole” in a mixture-fraction isosurface • Stems from fast chemistry flame-sheet “flamelet” models of turbulent combustion • Holes can be expanding or contracting • Speed of the edge can be monitored – need algorithm for edge detection

30. Advanced post-processing for extinction/re-ignition • Holes could be tracked to determine any history effects • need parallel feature tracking

31. Outline • DNS of turbulent combustion – challenges and opportunities • Sandia S3D DNS capability – terascale simulations on Office of Science platforms • New combustion science to advance predictive models: • 2D DNS of HCCI combustion • 3D simulations of turbulent jet flames with detailed chemistry • Layering Large Eddy Simulation and DNS approaches Vorticity fields in DNS of a turbulent jet flame (volume rendering by Kwan-Liu Ma and Hiroshi Akiba)

32. Layering of LES and DNS approaches(the future...) Joseph C. Oefelein, Evatt R. Hawkes, Jacqueline H. Chen

33. DNS RANS-LES-DNS at High Re Swirl-Stabilized Premixed Combustion Azimuthal Velocity Component • LES gets lots more than RANS • High Re is too expensive for DNS (at present) • can afford 1000s now useful but… • would like 100000s and more LES RANS

34. Why LES? • LES treats the large scales directly • Large scales are geometry dependent • LES can couple directly with high Re experiments • But – LES needs sub-grid models

35. Applications Layering LES and DNS • Better BCs, ICs • Identify modelling questions • Identify relevant parameter space • Moderate Re: • Canonical problems • Cleverly designed experiments LES DNS High Re Experiments • Sub-grid models

36. Multiphysics capabilities and algorithmic framework – Oefelein • Theoretical framework (LES  DNS) • Fully-coupled conservation equations • Compressible, chemically reacting system • Real-fluid EOS, thermodynamics, transport • Detailed chemistry, multiphase flow, sprays • Dynamic subgrid-scale modeling • Numerical framework • Dual-time, all-Mach-number formulation • Generalized preconditioning methodology • Complex geometry, generalized coordinates • Massively-parallel (MPI), highly-scalable • Ported to all major platforms • Over 10 years development • Fine-grain scalability (NERSC IBM SP – Seaborg) • Grid size fixed, number of processors increased

37. Magnitude of vorticity (normalized) DNS (all scales) 40 million grid, 120,000 hours LES (large scales) 0.6 million grid, 600 hours

38. Summary • S3D is a state-of-the-art DNS capability for turbulent combustion simulations that scales to thousands of processors and is ported to all Office of Science platforms. • DOE supercomputing facilities are enabling new combustion science. • 3D DNS of detailed finite-rate chemistry effects in turbulent jets provides new insights and data for combustion modeling: • mixing of reactive scalars can be very different from conserved scalars • Layering LES and DNS approaches will open new avenues of exploration with great potential for the future

39. 100 million grid run on NERSC Seaborg and ORNL Cray X1: Re = 8000 HO2 dissipation OH dissipation

40. S3D performance on different architectures