large scale simulations of astrophysical turbulence n.
Skip this Video
Download Presentation
Large scale simulations of astrophysical turbulence

Loading in 2 Seconds...

play fullscreen
1 / 49

Large scale simulations of astrophysical turbulence - PowerPoint PPT Presentation

  • Uploaded on

Large scale simulations of astrophysical turbulence. Axel Brandenburg (Nordita, Copenhagen) Wolfgang Dobler (Univ. Calgary) Anders Johansen (MPIA, Heidelberg) Antony Mee (Univ. Newcastle) Nils Haugen (NTNU, Trondheim) etc. (...just google for Pencil Code ). Overview.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Large scale simulations of astrophysical turbulence' - jolie

Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
large scale simulations of astrophysical turbulence

Large scale simulations of astrophysical turbulence

Axel Brandenburg (Nordita, Copenhagen)

Wolfgang Dobler (Univ. Calgary)

Anders Johansen (MPIA, Heidelberg)

Antony Mee (Univ. Newcastle)

Nils Haugen (NTNU, Trondheim)


(...just google for Pencil Code)

  • History: as many versions as there are people??
  • Example of a cost effective MPI code
    • Ideal for linux clusters
    • Pencil formulation (advantages, headaches)
    • (Radiation: as a 3-step process)
  • How to manage the contributions of 20+ people
    • Development issues, cvs maintainence
  • Numerical issues
    • High-order schemes, tests
  • Peculiarities on big linux clusters
    • Online data processing/visualization
pencil code
Pencil Code
  • Started in Sept. 2001 with Wolfgang Dobler
  • High order (6th order in space, 3rd order in time)
  • Cache & memory efficient
  • MPI, can run PacxMPI (across countries!)
  • Maintained/developed by many people (CVS!)
  • Automatic validation (over night or any time)
  • Max resolution so far 10243 , 256 procs
range of applications
Range of applications
  • Isotropic turbulence
    • MHD (Haugen), passive scalar (Käpylä), cosmic rays (Snod, Mee)
  • Stratified layers
    • Convection, radiative transport (T. Heinemann)
  • Shearing box
    • MRI (Haugen), planetesimals, dust (A. Johansen), interstellar (A. Mee)
  • Sphere embedded in box
    • Fully convective stars (W. Dobler), geodynamo (D. McMillan)
  • Other applications and future plans
    • Homochirality (models of origins of life, with T. Multamäki)
    • Spherical coordinates
pencil formulation
Pencil formulation
  • In CRAY days: worked with full chunks f(nx,ny,nz,nvar)
    • Now, on SGI, nearly 100% cache misses
  • Instead work with f(nx,nvar), i.e. one nx-pencil
  • No cache misses, negligible work space, just 2N
    • Can keep all components of derivative tensors
  • Communication before sub-timestep
  • Then evaluate all derivatives, e.g. call curl(f,iA,B)
    • Vector potential A=f(:,:,:,iAx:iAz), B=B(nx,3)
a few headaches
A few headaches
  • All operations must be combined
    • Curl(curl), max5(smooth(divu)) must be in one go
    • out-of-pencil exceptions possible
  • rms and max values for monitoring
    • call max_name(b2,i_bmax,lsqrt=.true.)
    • call sum_name(b2,i_brms,lsqrt=.true.)
  • Similar routines for toroidal average, etc
  • Online analysis (spectra, slices, vectors)
cvs maintained
CVS maintained
  • pserver (password protected, port 2301)
    • non-public (ci/co, 21 people)
    • public (check-out only, 127 registered users)
  • Set of 15 test problems in the auto-test
    • Nightly auto-test (different machines, web)
  • Before check-in: run auto-test yourself
  • Mpi and nompi dummy module for single processor machine (or use lammpi on laptops)
switch modules
Switch modules
  • magnetic or nomagnetic (e.g. just hydro)
  • hydro or nohydro (e.g. kinematic dynamo)
  • density or nodensity (burgulence)
  • entropy or noentropy (e.g. isothermal)
  • radiation or noradiation (solar convection, discs)
  • dustvelocity or nodustvelocity (planetesimals)
  • Coagulation, reaction equations
  • Homochirality (reaction-diffusion-advection equations)
features problems
Features, problems
  • Namelist (can freely introduce new params)
  • Upgrades forgotten on no-modules (auto-test)
  • SGI namelist problem (see pencil FAQs)
high order schemes
High-order schemes
  • Alternative to spectral or compact schemes
    • Efficiently parallelized, no transpose necessary
    • No restriction on boundary conditions
    • Curvilinear coordinates possible (except for singularities)
  • 6th order central differences in space
  • Non-conservative scheme
    • Allows use of logarithmic density and entropy
    • Copes well with strong stratification and temperature contrasts
i high order spatial schemes
(i) High-order spatial schemes

Main advantage: low phase errors

ii high order temporal schemes
(ii) High-order temporal schemes

Main advantage: low amplitude errors

2N-RK3 scheme (Williamson 1980)

2nd order

3rd order

1st order

hydromagnetic turbulence and subgrid scale models
Hydromagnetic turbulence and subgrid scale models?
  • Want to shorten diffusive subrange
    • Waste of resources
  • Want to prolong inertial range
  • Smagorinsky (LES), hyperviscosity, …
    • Focus of essential physics (ie inertial range)
  • Reasons to be worried about hyperviscosity
    • Shallower spectra
    • Wrong amplitudes of resulting large scale fields
simulations at 512 3
Simulations at 5123

Biskamp & Müller (2000)





the bottleneck is a physical effect
The bottleneck: is a physical effect

compensated spectrum

Porter, Pouquet, & Woodward (1998) using PPM, 10243 meshpoints

Kaneda et al. (2003) on the Earth simulator, 40963 meshpoints

(dashed: Pencil-Code with 10243 )

bottleneck effect 1d vs 3d spectra
Bottleneck effect: 1D vs 3D spectra

Compensated spectra

(1D vs 3D)

hyperviscous smagorinsky normal
Hyperviscous, Smagorinsky, normal

height of bottleneck increased

Haugen & Brandenburg (PRE, astro-ph/0402301)

onset of bottleneck at same position

Inertial range unaffected by artificial diffusion

structure function exponents
Structure function exponents

agrees with She-Leveque

third moment

helical dynamo saturation with hyperdiffusivity
Helical dynamo saturation with hyperdiffusivity

for ordinary


ratio 125 instead of 5

mhd equations
MHD equations






Momentum and

Continuity eqns

vector potential
Vector potential
  • B=curlA, advantage: divB=0
  • J=curlB=curl(curlA) =curl2A
  • Not a disadvantage: consider Alfven waves



2nd der once

is better than

1st der twice!

wallclock time versus processor
Wallclock time versus processor #

nearly linear


100 Mb/s shows


1 - 10 Gb/s

no limitation

sensitivity to layout on linux clusters
Sensitivity to layout onLinux clusters

yprox x zproc

4 x 32  1 (speed)

8 x 16  3 times slower

16 x 8  17 times slower



100 Mbit

link only

24 procs

per hub

why this sensitivity to layout
Why this sensitivity to layout?

All processors need to communicate

with processors outside to group of 24

use exactly 4 columns
Use exactly 4 columns

Only 2 x 4 = 8 processors need to communicate outside the group of 24

 optimal use of speed ratio between 100 Mb ethernet switch and 1 Gb uplink

animation of energy spectra
Animation of energy spectra

Very long run at 5123 resolution

mri turbulence mri magnetorotational instability
MRI turbulenceMRI = magnetorotational instability


w/o hypervisc.

t = 600 = 20 orbits


w/o hypervisc.

Dt = 60 = 2 orbits

homochirality competition of left right
Homochirality: competition of left/right

Reaction-diffusion equation

  • Subgrid scale modeling can be unsafe (some problems)
    • shallower spectra, longer time scales, different saturation amplitudes (in helical dynamos)
  • High order schemes
    • Low phase and amplitude errors
    • Need less viscosity
  • 100 MB link close to bandwidth limit
  • Comparable to and now faster than Origin
  • 2x faster withGB switch
  • 100 MB switches with GB uplink +/- optimal
transfer equation par a llelization

Intrinsic Calculation

Ray direction

Transfer equation & parallelization


Analytic Solution:

the transfer equation parallelization

Analytic Solution:

Intrinsic Calculation

Ray direction

The Transfer Equation & Parallelization


current implementation
Current implementation
  • Plasma composed of H and He
  • Only hydrogen ionization
  • Only H- opacity, calculated analytically

No need for look-up tables

  • Ray directions determined by grid geometry

No interpolation is needed