High fidelity terascale simulations of turbulent combustion
This presentation is the property of its rightful owner.
Sponsored Links
1 / 40

High-Fidelity Terascale Simulations of Turbulent Combustion PowerPoint PPT Presentation


  • 84 Views
  • Uploaded on
  • Presentation posted in: General

High-Fidelity Terascale Simulations of Turbulent Combustion. Jacqueline H. Chen, Evatt R. Hawkes, Ramanan Sankaran, James C. Sutherland, and Joseph C. Oefelein Combustion Research Facility Sandia National Laboratories Livermore CA Supported by

Download Presentation

High-Fidelity Terascale Simulations of Turbulent Combustion

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


High fidelity terascale simulations of turbulent combustion

High-Fidelity Terascale Simulations of Turbulent Combustion

Jacqueline H. Chen, Evatt R. Hawkes, Ramanan Sankaran, James C. Sutherland, and Joseph C. Oefelein

Combustion Research Facility

Sandia National Laboratories

Livermore CA

Supported by

Division of Chemical Sciences, Geosciences, and Biosciences,

Office of Basic Energy Sciences, and DOE SciDAC

Computing: LBNL NERSC, ORNL CCS/NLCF, PNNL MSCF,

SNL HPCNP Infiniband test-bed,

SNL CRF BES Opteron cluster

Visualization: Kwan-Liu Ma and Hiroshi Akiba UC Davis


Outline

Outline

  • DNS of turbulent combustion – challenges and opportunities

  • Sandia S3D DNS capability – terascale simulations on Office of Science platforms

  • New combustion science to advance predictive models:

    • 3D simulations of turbulent jet flames with detailed chemistry.

    • Layering Large Eddy Simulation and DNS approaches.

Vorticity fields in DNS of a turbulent jet flame

(volume rendering by

Kwan-Liu Ma and Hiroshi Akiba)


Turbulent combustion is a grand challenge

Turbulent combustion is a grand challenge!

  • Stiffness : wide range of length and time scales

    • turbulence

    • flame reaction zone

  • Chemical complexity

    • large number of species and reactions (100’s of species, thousands of reactions)

  • Multi-Physics complexity

    • multiphase (liquid spray, gas phase, soot, surface)

    • thermal radiation

    • acoustics ...

  • All these are tightly coupled

Diesel Engine Autoignition, Soot Incandescence

Chuck Mueller, Sandia National Laboratories


Several decades of relevant scales

O(4)

Range

Continuum

O(4)

Several decades of relevant scales

  • Typical range of spatial scales

    • Scale of combustor: 10 – 100 cm

    • Energy containing eddies: 1 – 10 cm

    • Small-scale mixing of eddies: 0.1 – 10 mm

    • Diffusive-scales, flame thickness: 10 – 100 m

    • Molecular interactions, chemical reactions: 1 – 10 nm

  • Spatial and temporal dynamics inherently coupled

  • All scales are relevant and must be resolved or modeled

Terascale computing:

~3 decades in scales

(cold flow)


Combustion cfd approaches to tackle different length scale ranges

Increasing cost

and resolution for

fixed physical problem

Combustion CFD Approaches to tackledifferent length scale ranges

  • Reynolds–Averaged Navier–Stokes (RANS)

    • Coarse meshes, full range of dynamic scales modeled

    • Empirical closure, bulk approximation, current engineering CFD

  • Large Eddy Simulation (LES)

    • Energetic scales resolved, sub-grid scale dynamics modeled

    • combustion closure required, captures large-scale unsteadiness

  • Direct Numerical Simulation (DNS)

    • No sub-grid modeling required but limited on range of scales

    • Building-block configurations, research tool


Role of direct numerical simulation dns

Oxidizer

Fuel

HO2

CH4

CH3O

O

Role of Direct Numerical Simulation (DNS)

  • A tool for fundamental studies of the micro-physics of turbulent reacting flows

  • A tool for the development and validation of reduced model descriptions used in macro-scale simulations of engineering-level systems

  • Physical insight into chemistry turbulence interactions

  • Full access to time resolved fields

DNS

Engineering-level

CFD codes

(RANS and

future LES)

Physical

Models

DNS

Piston Engines


S3d mpp dns capability at sandia

S3D code characteristics:

Solves compressible reacting Navier-Stokes

F90/F77, MPI, domain decomposition.

Highly scalable and portable

8th order finite-difference spatial

4th order explicit RK integrator

hierarchy of molecular transport models

detailed chemistry

multi-physics (sprays, radiation and soot) from SciDAC “Terascale Simulation of Turbulent Combustion”

70% parallel efficiency on 4096 processors on NERSC; 90% parallel efficiency on 5120 CrayXT3 processors; 75% parallel efficiency on 1000 CrayX1E processors

Terascale computations => need scalar optimization customized to architecture

CCS, NERSC consultants

S3D MPP DNS capability at Sandia

S3D is a state-of-the-art DNS code developed with

13 years of BES sponsorship.

S3D scales up to 1000s of processors… and beyond?


S3d performance improvements on seaborg

S3D performance improvements on Seaborg

Application Software Case Studies in FY05 for MICS (K. Roche)

Using Xprofiler, IPM and working with David Skinner:

  • 8% improvement in performance by minimizing conversion between dimensional and nondimensional units for interfacing with Chemkin

  • interface to Mathematical Acceleration Subsystem (MASS) library for exponential and logarithmic functions resulted in 10% improvement. Another 5% due to using vectorized exp function in VMASS

  • Tabulating Gibbs energy as a function of temperature resulted in 8% savings

  • Remove excessive dynamic allocation calls resulted in 12% savings.

  • Eliminate MPI derived data types for communicating noncontiguous boundary data with neighbors

45% efficiency improvement


Outline1

Outline

  • DNS of turbulent combustion – challenges and opportunities

  • Sandia S3D DNS capability – terascale simulations on Office of Science platforms

  • New combustion science to advance predictive models:

    • 3D simulations of turbulent jet flames with detailed chemistry

    • Layering Large Eddy Simulation and DNS approaches

Vorticity fields in DNS of a turbulent jet flame

(volume rendering by

Kwan-Liu Ma and Hiroshi Akiba)


High fidelity terascale simulations of turbulent combustion

FY05 INCITE award: Direct simulation of a 3D turbulent nonpremixed CO/H2/air jet flame with detailed chemistry (350 million grids, 12 species)

  • Objectives:

  • Study fine-grained coupling between turbulent mixing and finite-rate chemical effects

    • Extinction-reignition dynamics

    • Mixing time scales

    • Reynolds number effect

    • Preferential diffusion

    • Flow unsteadiness

  • Test assumptions in subgrid mixing and reaction models

z

x

y

x

INCITE run test case (1/2 resolved) 350 million grid points, Re=5000

Isoscalar surface of scalar dissipation rate (local mixing rate)


High fidelity terascale simulations of turbulent combustion

The effect of chemistry-turbulence interactions in turbulent nonpremixed jet flames: mixing of passive and reacting scalars

Evatt Hawkes, Ramanan Sankaran,

James Sutherland, and Jacqueline Chen

Computational Platforms: NERSC Seaborg, ORNL CCS Phoenix Cray X1E, XT3 and PNNL MPP2


Description of runs temporally evolving non premixed plane jet flames

Air

Mixing,

Reaction

Fuel

Mixing,

Reaction

Air

Description of runs- Temporally evolving non-premixed plane jet flames

  • A canonical flow with shear-driven turbulence

x

y

Heat release iso-contours


Dns data sets of turbulent nonpremixed h 2 co flames mixing extinction reignition

Large computing allocations are enabling new science runs

INCITE at NERSC, capability computing at NLCF ORNL, ERCAP at PNNL, NERSC

Detailed H2/CO chemistry (17 d.o.f., Li et al. 2005)

Parameters selected to maximize Re

Case A:

Re 6000

40 million grid points

480 processors on PNNL MPP2

~ 1 TB data

Case B:

Re 8000

100 million grid points

1728 processors on Seaborg

192/240 processors on ORNL CCS Cray X1/E

~ 2.5 TB data

INCITE calculation:

Currently running on Seaborg

Re 5000

350 million grid points

~4096 Seaborg processors

3 million hours total

~ 9TB raw data

Two additional calculations, parametric study in Re

Re 2500, 150 million grid points

Re 10,000, 500 million grid points

DNS data-sets of turbulent nonpremixed H2/CO flames (mixing, extinction/reignition)


Community data sets

Community data sets

  • New opportunities for numerical benchmarks – highly resolved LES and DNS

  • TNF workshop (1996-2004): International Collaboration of Experimental and Computational Researchers

    • Framework for detailedcomparison of measured and modeled results (http://www.ca.sandia.gov/TNF/abstract.html)


Non premixed combustion concepts

Non-premixed combustion concepts

  • Mixture fraction Z: the amount of fluid from the fuel stream in the mixture

  • Z is a conserved (passive) scalar – (no reactive source term)

  • Scalar dissipation, a measure of local molecular mixing rate:


Example development of the jet 40 million grid pnnl run

Example development of the jet:40 million grid PNNL run

  • Left: Vorticity. Right: Simultaneous volume rendering of mixture fraction, scalar dissipation and OH radical.

(Rendering by Hiroshi Akiba and Kwan-Liu Ma, UC Davis)


100 million grid run on nersc seaborg and ornl cray x1 re 8000

100 million grid run on NERSC Seaborg and ORNL Cray X1: Re = 8000

Vorticity

Scalar Dissipation


100 million grid run on nersc seaborg and ornl cray x1 re 80001

100 million grid run on NERSC Seaborg and ORNL Cray X1: Re = 8000

HO2 dissipation

OH dissipation


Mixing timescales

Mixing timescales

  • Models for molecular mixing are required in the PDF approach to turbulent combustion (Pope 1985), a sub-grid model used in RANS and LES approaches.

  • TNF workshop – CFD predictions are dependent on mixing timescale choice.

  • Models assume that scalar mixing timescales are identical for all scalars and determined by the turbulence timescale.

    • scalars with different diffusivities?

    • reactive scalars?


Definitions

is assumed to be order unity in most models

is assumed to be the same for all scalars

Definitions

  • Mechanical time-scale:

  • Scalar time-scale:

  • Time-scale ratio:


Mixture fraction to mechanical timescale ratio

Mixture fraction to mechanical timescale ratio

  • Confirmation that mixture fraction to mechanical time scale ratio is order unity.

  • Average value about 1.5, similar to values reported by experiments, simple chemistry DNS, and used successfully in models.

Timescale Ratio rZ


Effect of diffusivity

Effect of diffusivity

Increasing

diffusivity

  • Smaller, more highly diffusive species do have faster mixing timescales


Finite rate chemistry effects on mixing

Finite-rate chemistry effects on mixing

  • HO2 and H2O2 have faster mixing times in the middle of the simulation, while OH and O are lower

  • Diffusivity trend does not appear to hold for HO2 and H2O2 versus O and OH.

  • What is going on?


Radical production and destruction in high dissipation regions

Radical production and destruction in high dissipation regions

Color scale: mass fraction (blue is low; red is high)

White iso-contours: scalar dissipation rate

OH

  • OH is consumed while HO2 is produced in high dissipation regions

HO2


Dissipation of passive and reactive scalars

Dissipation of passive and reactive scalars

  • Blue: Z, Green: OH, Red: HO2

  • Dissipation fields of Z and HO2 are co-incident and aligned with principal strain directions

  • OH dissipation occurs elsewhere, more in the centre of the jet

  • OH is reduced in the high dissipation regions, leading to longer mixing times, opposite for HO2

  • At later times, mixing rates relax, OH returns and HO2 decreases


100 million grid run on seaborg cray x1

100 million grid run on Seaborg / Cray X1

  • We need to establish any Re dependence

  • Need parametrics at larger Re

  • Run in progress… but seems to be showing same effects


Conclusions mixing timescales

Conclusions - mixing timescales

  • New finding: detailed transport and chemistry effects can alter the observed mixing timescales

  • Models may need to incorporate these effects

    • a poor mixing model could lead to incorrectly predicting a stable flame when actually extinction occurs

  • This type of information cannot be determined any other way at present

    • ambiguities in a-posteriori model tests

    • too difficult to measure

    • need 3D and detailed chemistry to see this


Knowledge discovery from terascale datasets

Knowledge Discovery From Terascale Datasets

  • Challenge:Need interactive knowledge discovery software for terascale high-fidelity scientific data sets where computing pipeline is distributed geographically.

  • Solution:Intelligent visualization pipeline

    • Simultaneous cognitive visualization

    • Intelligent feature extraction/tracking

    • Scalable transparent data sharing and parallel I/O across platforms

INCITE Award: DNS of a 3D turbulent flame (9 TB raw data - 350 million grids, 17 variables, 100,000 steps)


Advanced post processing for extinction re ignition

Advanced post-processing for extinction/re-ignition

  • Extinction viewed as radical or reaction-rate “hole” in a mixture-fraction isosurface

  • Stems from fast chemistry flame-sheet “flamelet” models of turbulent combustion

  • Holes can be expanding or contracting

  • Speed of the edge can be monitored – need algorithm for edge detection


Advanced post processing for extinction re ignition1

Advanced post-processing for extinction/re-ignition

  • Holes could be tracked to determine any history effects

    • need parallel feature tracking


Outline2

Outline

  • DNS of turbulent combustion – challenges and opportunities

  • Sandia S3D DNS capability – terascale simulations on Office of Science platforms

  • New combustion science to advance predictive models:

    • 2D DNS of HCCI combustion

    • 3D simulations of turbulent jet flames with detailed chemistry

    • Layering Large Eddy Simulation and DNS approaches

Vorticity fields in DNS of a turbulent jet flame

(volume rendering by

Kwan-Liu Ma and Hiroshi Akiba)


Layering of les and dns approaches the future

Layering of LES and DNS approaches(the future...)

Joseph C. Oefelein, Evatt R. Hawkes, Jacqueline H. Chen


Rans les dns at high re

DNS

RANS-LES-DNS at High Re

Swirl-Stabilized Premixed Combustion

Azimuthal Velocity Component

  • LES gets lots more than RANS

  • High Re is too expensive for DNS (at present)

    • can afford 1000s now

      useful but…

    • would like 100000s and more

LES

RANS


Why les

Why LES?

  • LES treats the large scales directly

  • Large scales are geometry dependent

  • LES can couple directly with high Re experiments

  • But – LES needs sub-grid models


Layering les and dns

Applications

Layering LES and DNS

  • Better BCs, ICs

  • Identify modelling questions

  • Identify relevant parameter space

  • Moderate Re:

  • Canonical problems

  • Cleverly designed experiments

LES

DNS

High Re

Experiments

  • Sub-grid models


Multiphysics capabilities and algorithmic framework oefelein

Multiphysics capabilities and algorithmic framework – Oefelein

  • Theoretical framework (LES  DNS)

    • Fully-coupled conservation equations

    • Compressible, chemically reacting system

    • Real-fluid EOS, thermodynamics, transport

    • Detailed chemistry, multiphase flow, sprays

    • Dynamic subgrid-scale modeling

  • Numerical framework

    • Dual-time, all-Mach-number formulation

    • Generalized preconditioning methodology

    • Complex geometry, generalized coordinates

    • Massively-parallel (MPI), highly-scalable

    • Ported to all major platforms

  • Over 10 years development

  • Fine-grain scalability (NERSC IBM SP – Seaborg)

  • Grid size fixed, number of processors increased


Magnitude of vorticity normalized

Magnitude of vorticity (normalized)

DNS

(all scales)

40 million grid, 120,000 hours

LES

(large scales)

0.6 million grid, 600 hours


Summary

Summary

  • S3D is a state-of-the-art DNS capability for turbulent combustion simulations that scales to thousands of processors and is ported to all Office of Science platforms.

  • DOE supercomputing facilities are enabling new combustion science.

  • 3D DNS of detailed finite-rate chemistry effects in turbulent jets provides new insights and data for combustion modeling:

    • mixing of reactive scalars can be very different from conserved scalars

  • Layering LES and DNS approaches will open new avenues of exploration with great potential for the future


100 million grid run on nersc seaborg and ornl cray x1 re 80002

100 million grid run on NERSC Seaborg and ORNL Cray X1: Re = 8000

HO2 dissipation

OH dissipation


S3d performance on different architectures

S3D performance on different architectures


  • Login