Data Management Challenges in Petascale Simulations of Turbulent Combustion

1. Data Management Challenges in Petascale Simulations of Turbulent Combustion Jacqueline Chen, Evatt Hawkes, David Lignell and Chunsang Yoo Combustion Research Facility Sandia National Laboratories jhchen@sandia.gov Ramanan Sankaran Oak Ridge National Laboratory Supported by Division of Chemical Sciences, Geosciences, and Biosciences, Office of Basic Energy Sciences Computing: ORNL NLCF, NERSC, PNNL, SNL

2. Combustion and energy security • Combustion accounts for 3/4 of energy used in U.S. manufacturing • Ground transportation accounts for 2/3 of petroleum usage • Potential for improvement in thermal efficiency (30%45%) • Low temperature combustion (LTC) concepts for automobiles • Savings of 3 million barrels of oil per day (out of 20M) • Design improvements are difficult • Low hanging fruits have already been picked • Advanced concepts require combustion operating at the edge • Sound scientific understanding is necessary

3. Turbulent combustion is a grand challenge! • Stiffness : wide range of length and time scales • turbulence • flame reaction zone • Chemical complexity • large number of species and reactions (100’s of species, thousands of reactions) • Multi-Physics complexity • multiphase (liquid spray, gas phase, soot, surface) • thermal radiation • acoustics ... • All these are tightly coupled Diesel Engine Autoignition, Soot Incandescence Chuck Mueller, Sandia National Laboratories

4. Engineering CFD codes (RANS, LES) Physical Models DNS Direct Numerical Simulation (DNS) • DNS is a tool for fundamental studies of the micro-physics of turbulent reacting flows • Full access to time resolved fields • Physical insight into chemistry turbulence interactions • A tool for the development and validation of reduced model descriptions used in macro-scale simulations of engineering-level systems

5. DNS capability at Sandia S3D is a state-of-the-art DNS code developed with 13 years of BES sponsorship. • Solves compressible reacting Navier-Stokes equations. • High fidelity numerical methods. • 8th order finite-difference • 4th order explicit RK integrator • Hierarchy of molecular transport models • Detailed chemistry • Multi-physics (sprays, radiation and soot) • From SciDAC-TSTC (Terascale Simulation of Turbulent Combustion) • Fortran90 and MPI • Highly scalable and portable

6. S3D parallel performance S3D scales with 90% parallel efficiency on 10000 cores on CrayXT3 (ORNL)

7. Achieving quantitative predictability requires petascale computing • Petascale computers needed to achieve relevant parameters spaces for turbulent combustion: N ~ Re9/4 turbulence plus flame scales (3-4 decades of scales) • Relevant parameter regimes of real devices and laboratory-scale flames Re>15,000. • Terascale computing: Re~O(10,000), fully-developed turbulence • Turbulence-chemistry interactions requires transporting 20-80 species plus turbulence

8. DNS of canonical laboratory configurations • Lean premixed flames • Extinction and reignition in nonpremixed flames • Flame stabilization in autoigniting flames

9. DNS of lean premixed methane/air combustion Sankaran et al. 2006 • Goals • Better understanding of lean premixed combustion in natural-gas based stationary gas turbines • Model validation and development • Simulation details • Detailed CH4/Air chemistry (18 D.O.F.) • Slot burner configuration with mean shear • Spatially developing and statistically stationary simulation. • Better suited for model development • Parametric study: 3 simulations at increasing turbulence intensities • to understand the effect of turbulence on flame structure and burning speed. Temperature and heat release, Akiba and Ma

10. Parametric study with increasing Reynolds number • Small-scale intense turbulence leads to flame broadening in the preheat zone and increased flame shortening Heat release in a methane-air slot Bunsen flame, animation by H. Akiba and K. L. Ma

11. Fresh 1/4 Burned u’/SL=3 u’/SL=6 1/2 3/4 Effect on flame structure • Progress variable defined based on O2 mass fraction • Instantaneous slices shown on the left for Case 1 -Progress variable from 0 (blue) to 1 (red) in color - Heat release rate as line contours • Considerable influence on preheat zone • Reaction zone relatively intact

12. Turbulent nonpremixed combustion Fuel and air segregated Mixing limited Extinction Reignition Flame stabilization

13. Extinction and reignition in a CO/H2 jet flame Hawkes, Sankaran, Sutherland, Chen – 2006, DOE INCITE 2005, early user LCF /ORNL 2005 Burning Extinguished Understanding extinction/reignition in non-premixed combustion is key to flame stability and emission control in aircraft and power producing gas-turbines Discovered dominant reignition mode is due to engulfment of product gases, not flame propagation Scalar dissipation rate The largest ever simulations of combustion have been performed to advance this goal: • 500 million grid points • 11 species and 21 reactions • 16 DOF per grid point • 512 Cray X1E processors • 30 TB raw data • 2.5M hours on IBM SP NERSC (INCITE); 400K hours on Cray X1E (ORNL)

14. Air Air Mixing, Reaction Fuel Fuel Mixing, Reaction Air Air Description of runs- Temporally Evolving Non-premixed Plane Jet Flame Streamwise BC: periodic Spanwise BC: periodic • Jet develops temporally. • Shear-driven turbulence interacts with the flame. Initial condition Later time

15. Volume rendering of scalar dissipationYu, Ma, Chen, Chen, Hawkes, StorCloud demo at SC05. • Hardware accelerated parallel volume rendering (nearly interactive – read and render in 2 sec). • 4D time-varying terascale data • Multi-variate volume rendering • Intelligent visualization

16. Increasing Reynolds number Reynolds number effects on  Case H Case M Case L • Higher Re: • more fine-scale intermittent structure • higher fluctuations of 

17. Increasing Reynolds number results in more extinction Reynolds number effects on extinction Re=9000 Re=4500 Re=2500 20tj Simulation Time 40tj

18. How is extinction correlated with local mixing rates? Scalar Dissipation (mixing rate) Extinguished Regions

19. Mixture Fraction Isosurface Z=Z0 Burning >0 Extinguished <0 Flame edge Quantification of Extinction- Extinguished Flame Area • Reaction rate related to the conditional fine-grained surface-density of the stoichiometric surface • Isosurface extraction from volume data through triangulation • Data analysis on iso-surface and local normal vector • Identify flame holes • Flame edge analysis • Edge propagation speed • Study reignition mechanisms Analysis code has to be parallel, suitable for large data and reside within S3D Se

20. Joint PDF of edge speed and mixing rate • Color scale: Joint PDF, Black line: conditional mean speed. • First, mainly negative speeds, strong negative correlation with . • Then, broader PDF, with 2 branches • negatively correlating branch at very high  • positively correlating branch at low-intermediate  • Peak positive edge speed occurs at quite high !!! Simulation Time

21. Physical interpretation Burning Z=Zst 2 basic ideas for reignition: • Propagating edges along the stoichiometric contour. Extinguished

22. Tentative Interpretation Burning Z=Zst 2 basic ideas for reignition: • Propagating edges along the stoichiometric contour. O(SL) Extinguished

23. Tentative Interpretation Burning Z=Zst 2 basic ideas for reignition: • Turbulence folds burning flames onto non-burning areas. Extinguished

24. Tentative Interpretation Burning Z=Zst 2 basic ideas for reignition: • Turbulence folds burning flames onto non-burning areas. Extinguished

25. Burning Burning Z=Zst Z=Zst • Compressive strain leads to high  Extinguished Extinguished Tentative Interpretation • u’>>sL indicates laminar edge flame propagation unimportant. • Expect reignition by turbulent flame-folding. • To bring burning and non-burning surfaces together, compressive strain is required, leading to high dissipation. • Consistent with our result – but more work needed to confirm.

26. Understanding the effect of ignition on lift-off stabilization of n-heptane diesel jet requires petascale computing • How is combustion stabilized at the lift-off length in a n-heptane diesel jet? • Flame propagation • Autoignition (1st stage cool flame) • Is lift-off stabilization supported by premixed flame propagation into a cool flame mixture, or by second-stage autoignition? • Is lift-off scaling parameterized by ignition or by flame propagation? • How does turbulent mixing affect the transition from first-stage to second-stage, high temperature ignition? • How are soot precursors affected by ignition quality? High-speed Chemiluminescence Imaging in a combustion vessel near the lift-off length, Lyle Pickett,SNL

27. Air 1100K H2(65%)/N2(35%) 400K Stabilization of Lifted Autoigniting Flames • Stabilization mechanisms in lifted, vitiated flames • Hydrogen lifted flame • High flow velocity to lift off flame • Auto-ignition may occur below the base of the lifted flame due to high co-flow temperature • Approximately 1.2 million CPU hours per a simulation on Jaguar in NCCS • 9 species with 21 elementary reactions • 24 x 24 x 5.4mm3 domain size with 1600 x 1600 x 270 grid resolution ~0.7 billion grids • 6~8 flow-though times w/ Umean = 347 m/s • Total 3~4 simulations Liftoff height Highest heat release rate (green), YHO2(red), and vorticity (aqua) of hydrogen lifted jet flame at t = 0.3ms

28. Animation of hydroxyl radical in lifted hydrogen/air lifted flame

29. 50 m/s 80 m/s 110 m/s 140 m/s n-Heptane Lifted Autoigniting Flame • CPU hour estimation of n-heptane lifted flame • Skeletal n-heptane mechanism • 88 species; 384 reaction steps • 40 x 30 x 30 mm3 domain size • 2000 x 1200 x 1200 grid resolution (3 billion grids) • 12 flow-though time (2.4ms) calculation 200-300 m/s mean flow velocity • CPU hours • (2000 x 1200 x 1200 grid points) x (240,000 time steps) x (92 equations) x (3.0 x 10-9 hour/NgridNtimeNeq) 240 million CPU hours (on Jaguar in NCCS) • Requires petascale computer! Propane lifted flames in hot coflow (from Kim et. al, Proc. Combust. Inst. 31, 2007 to appear)

30. Community Data Sets • Precedents for comparison of measured and modeled results • TNF workshop http://www.ca.sandia.gov/TNF/abstract.html • Premixed flame workshops http://eetd.lbl.gov/aet/combustion/workshop/workshop.html • Addition of high-fidelity numerical benchmarks for model validation and development

31. Data Sharing What is the best model for sharing data, especially large data? • Reduced data on the web? • RANS, LES means, conditional means, slices, chunks • What is the basic set of data to make available? • Visiting research projects? • Giving out whole data? • What, if any, tools/procedures are needed to enable this? • Visualization • Data movement • Generic data-processing

32. Challenges of Petascale Computation: Mountains of Data HPSS storage facility at NERSC • Data storage: • Long term: where do we put 500TB? • Short term: scratch ~ 1TB, but need ~ 10TB! • Data movement: • Archive to scratch (~ 1 week to move 10TB) • HPC facility to local analysis cluster (longer) • Data processing: • Everything must be parallel, scalable. • IO speed, memory are the bottlenecks. • Visualization: • Parallel, Multi-variate • Multi-scale phenomena • Interpretation: • Physics are more complex • Wider range of scales, manual sifting is impossible. • Multi-scale analysis methods • Feature detection, growing, and tracking

33. S3D I/O Requirements • Jet simulation (Re=10,000) on X1E (20TF). • At the rate of one data dump every hour, I/O rate is 64GB/hour • On a Petaflop system, required I/O rate is 3.2TB/hour • To achieve 5% maximum overhead, I/O has to occur at 64TB/hour or 17GB/s • It will be useful to dump data more often than once an hour if higher I/O rates are available. I/O sizes of current terascale S3D combustion simulations

34. Data Movement and Workflow • 40TB of data generated last year (excluding prep runs and analysis output) • Geographically distributed HPC resources/archives • ORNL, NERSC, PNNL, Sandia (CA) • Need automatic handling of data • Kepler workflow for S3D, Ramanan Sankaran and Scott Klasky (ORNL) and Norbert Podhorszki (UC Davis) (SDM Center)

35. Parallel feature detection and tracking - gleaning insight from large data • Large and complex multiscale data • Interesting regions sparse • Temporal evolution of features

36. Interactive Scalable Feature Detection and Tracking W. Koegler 1996 IEEE Vis. • Feature definition: threshold on HO2 concentration • Feature detection and tracking: • Born • Merge • Die • Grow • Split • Feature statistics • Feature-based analysis DNS of Autoignition of H2/Air in Inhomogeneous Mixtures, Echekki and Chen, 2001

37. Feature Statistics Koegler 1996 IEEE Vis. Feature Graph time

38. Future of automated data discovery • Novel approach for data discovery: • Hierarchical features allow humans to frame hypotheses at different levels of detail. • Quantitative feature definitions – beyond static thresholds • Complex isosurfaces • Topological methods for scalar and vector definitions • Online distributed feature detection and tracking; computational steering of I/O and analysis • FastBit technology for efficient search and query • Frame hypotheses as parametric relationships between feature properties. • New rendering tools allow humans to visually and quantitatively review multiple features in context and provide training data to machine learning.

39. Scalable Interactive Combustion Analysis Toolkit Feature Detection/Tracking Demand-driven I/O Libraries Raw DNS Data Feature Definition Feature detection Demand-driven I/O preprocessor Feature-based analysis Feature tracking Analysis Libraries Flame surface statistics Chemical Analysis Conditional statistics Spectra – Fourier, wavelets Reduced order representation- POD, topology Averaging and filtering

40. Acknowledgments • Chunsang Yoo (SNL) • Ramanan Sankaran (ORNL) • Scott Klasky (ORNL) • Evatt Hawkes (SNL) • Mark Fahey (ORNL) • David Skinner (LBNL) • David Lignell (SNL) • Andrea Gruber (SINTEF, U. Trondheim) • Tianfeng Lu (Princeton U.) • Chung Law (Princeton U.) • Kwan-Liu Ma (U. C. Davis) • Hiroshi Akiba (U. C. Davis) • Hongfeng Yu (U. C. Davis) • Scott Klasky (ORNL) • Wendy Doyle (SNL) • John Wu (LBNL) • Arnaud Trouve (U. Maryland) • Hong Im (U. Michigan) • Chris Rutland (U. Wisconsin)

41. Contacts Jacqueline Chen Combustion Research Facility Sandia National Laboratories Livermore, CA 94550 jhchen@sandia.gov Ramanan Sankaran National Center for Computational Sciences Oak Ridge National Laboratory sankaranr@ornl.gov 41 Presenter_date

42. DNS of turbulent flame-wall interaction A. Gruber, R. Sankaran, E. R. Hawkes, and J. H. Chen - 2006 • DNS of a premixed flame interacting with turbulence in a wall-bounded channel flow • Detailed H2/air chemistry (14D.O.F.) • Inflow turbulence obtained from a separate simulation of inert channel flow • Cost: 100K hours on Cray X1E • Goals • Material failure from fluctuating thermal stress in micro-gas turbines • Study of spatial and temporal patterns of wall heat flux • Discovered correlation of near-wall coherent turbulence structures and exothermic radical recombination reactions with wall heat flux Wall heat fluxes in turbulent flame wall interaction at low Reynolds number

43. Flame stabilization in lifted autoigniting hydrogen flames • Goal: determine stabilization mechanisms in lifted, autoigniting hydrogen/air flames • Hydrogen lifted flame • Important submechanism for n-heptane • Comparison with experiment (Chung, Cabra) • Auto-ignition may occur below the base of the lifted flame due to high co-flow temperature • Approximately 1.5 million CrayXT3 CPU hours • 9 species; 21 elementary reaction steps (Li et al. 2006) • 24 x 20 x 3.6mm3 domain size; 1600 x 1000 x 240 grid resolution=~400M grids • 4 flow-though times; Umean = 347 m/s, Re=7000 • Parametric study with coflow temp, flow velocity, additives • Total 3~4 simulations (~3.0 million cpu-hrs on CrayXT3 ORNL 2006-2007) Heat release rate and vorticity in a H2/air lifted slot jet flame in heated coflow

44. Motivation Soot is a pollutant and lowers combustion efficiencies in diesel engines Soot radiation is the major heat transfer mode Goals Quantify soot formation and transport mechanisms in turbulent flames Approach DNS of turbulent sooting flames with detailed chemistry, transport, radiation 2–3 moment soot particle model with semi-empirical soot chemistry 3M cpu-hours on CrayXT3 at ORNL to observe slow soot processes Air Fuel Air fuel air DNS of non-premixed sooting ethylene flames D. O. Lignell, P. J. Smith, and J. H. Chen

45. DNS of stabilization in lifted auto-igniting flames (planned) Goal: Determine stabilization mechanisms in lifted, auto-igniting flames relevant to efficient, clean burning low-temperature compression ignition engines and flashback in aircraft gas turbine engines Hydrogen lifted flame • 300K hydrogen turbulent jet; 1100K air coflow jet • Important submechanism for n-heptane • Comparison with experiment (Chung, Cabra) • Auto-ignition may occur below the base of the lifted flame due to high co-flow temperature • Approximately 1.2 million CPU hours per simulation on Jaguar at NCCS - 9 species; 21 elementary reaction steps (Li et al. 2006) - 24x20x3.6 mm3 domain size; 1600x1000 x240 grid resolution= ~400M grids - 4–6 flow-though times; Umean = 350 m/s - Total 3~4 simulations (~3.0 million cpu-hours on CrayXT4 at ORNL 2006–2007) Heat release (gold) and vorticity in a lifted autoigniting H2/air jet flame

46. Increasing diffusivity Transport Effects on Mixing Timescale • Mixing time scale is key element of combustion models (transported PDF method) • Most models assume mixing time scale same as mechanical time scale for all species • Detailed transport and chemistry effects can alter the observed mixing timescales

47. Acknowledgments • Chunsang Yoo (SNL) • Ramanan Sankaran (ORNL) • Evatt Hawkes (SNL) • Mark Fahey (ORNL) • David Lignell (SNL) • NLCF Computational Combustion End Station (3.6M cpu-hours, 2006) • NERSC Incite Award 2005 (2.5M cpu-hours)