Dey-Mittra ADI: An absolutely stable method for Maxwell’s Equations with Embedded Boundaries

1. Dey-Mittra ADI: An absolutely stable method for Maxwell’s Equations with Embedded Boundaries Travis Austin John Cary, David Smithe Tech-X Corporation Funded by US-DOE grants DE-FG02-05ER84172, DE-FC02-07ER41499, and FA9451-06-D-0115/002. ComPASS Meeting Boulder, CO Tuesday, October 6, 2009

2. Overview • Background • Motivation • Dey-Mittra Cut-Cell Stability • Alternating-Direction Implicit (ADI) Methods • Divergence-preserving • Tridiagonal Solves (Smithe, Cary, Carlsson, Ovtchinnikov) • Dey-Mittra ADI Method • Implementation • Frequency Extraction Results • Performance • Future Work

3. Background Maxwell’s Equations: Bz Ey Yee Method: Ey Ex Ez Ez Ez Bx By Ex Ey

4. Background Yee Method (Faraday’s Law): Courant-Friedrichs-Lewy Stability Condition:

5. MotivationDey-Mittra Cut-Cell Stability (a) Dey-Mittra Approach (b) Stairstep Approach - Only change Faraday update

6. MotivationDey-Mittra Cut-Cell Stability Dey-Mittra-Induced Stability Condition: Determination of fDM: • fractional value between 0 and 1 • based on stability derived from Gershgorin circle theorem • too small cut-cells make time step prohibitively small • an implicit method would overcome these restrictions

7. There are four 2nd-order accurate variants of the ADI algorithm, depending on the order of the operands: ZCZ is the first. We investigated the last … Alternating-Direction Implicit Methods Divergence-Preserving Form

8. The operator P+M is the curl operator, so for Yee-cell Of the four ADI combinations, only the last form, DP, can be algebraically manipulated to show that its final operation is equivalent to a finite-difference curl Thus it is divergence preserving, for source S. Alternating-Direction Implicit Methods Divergence-Preserving Form

9. Full details are in the paper D. N. Smithe, J. R. Cary, J. A. Carlsson, ”Divergence preservation in the ADI algorithms for electromagnetics,” J. of Comp. Physics 228, 7289 (2009). Alternating-Direction Implicit Methods Divergence-Preserving Form

10. Each domain does forward-solve over its domain Passes boundary data to single process global solve Receives data, then back-solves over its domain Alternating-Direction Implicit Methods Tridiagonal Solves   

11. Remedy is Concurrent Divide & Conquer In 2-D and 3-D there are multiple 1-D solves. Global solves are distributed across the processes. Alternating-Direction Implicit Methods Tridiagonal Solves 

12. For good scaling, the local backward solve must cover latency. N  # cells in process Longer 1-D dimension, N1/2 rather than N1/3, means more time to cover latency. Alternating-Direction Implicit Methods Tridiagonal Solves

13. There was good scaling as long as Ncells(1/Ndim) 64.  Implies typically good scaling for ADI in both 2-D and 3-D. On office linux cluster, needed Ncells(1/Ndim) 128 for good scaling  Implies good scaling for 2-D, and marginal scaling for 3-D. Alternating-Direction Implicit Methods Tridiagonal Solves (Ideal = dotted line)

14. Accelerator devices have boundaries which are nonconvex. This breaks each row tri-diagonal solve into several solves. Simply unit-fill the diagonals of rows for fully exterior field components, and set RHS source to zero. Dey-Mittra ADI Implementation

15. Dey-Mittra is a metallic cut-cell algorithm giving 2nd-order accurate global solutions. Modify Faraday’s law to use non-metallic electric line length and magnetic flux area. Dey-Mittra ADI Implementation dAnon-metallic dlnon-metallic

16. Division by small area limits algorithm.  Reduction in time step is significant, 0.5Dt for decent results, and 0.25Dt or even 0.10Dt for excellent results.  Throws away small cells, leading to occasional “pits” and “scratches” in geometry including particle creation/destruction surfaces. Using ADI can eliminate both these inconveniences. Dey-Mittra ADI Implementation

17. Technically, the stability of ADI requires that the two alternating-direction curl matrix operators, P and M, be anti-symmetric. The Dey-Mittra length/area factors appear to destroy this anti-symmetry. However, anti-symmetry can be recovered by solving for re-scaled fields. Then removing scale factors. Bottom-line: OK to use Dey-Mittra difference matrix even in non-anti-symmetric matrices. E.g., it’s still stable. (Even when matrix element  ) Dey-Mittra ADI Implementation

18. Testing modes of A6 magnetron. Dey-Mittra ADI Frequency Extraction Results Frequency (Hz) 1/Nx2

19. 2nd-order accuracy verified, even for dt that is 8 times the normal Courant limit. Dey-Mittra ADI Frequency Extraction Results

20. Dey-Mittra ADI Frequency Extraction Results • Method investigated in 3D for A15 cavity Magnetic Field (z-component) tf = 343.06 ps Time Steps = 2000 Electric Field (x-component) • Time Step = 2.0 DtCFL • fDM = 0.0000001 => all cells kept • in the simulation

21. Dey-Mittra ADI Performance: 2D • Simulation Parameters: • Single CPU Results • 2D A6 Magnetron Benchmark • 2500 (50x50) cells • GMRES w/ Jacobi preconditioner • 1.358 ns of simulation time • Explicit: 1280 time steps • ADI-1.0: 320 time steps • ADI-2.0: 160 time steps • ADI-4.0: 80 time steps • ADI-8.0: 40 time steps • ADI-16.0: 20 time steps • ADI-32.0: 10 time steps • ADI-64.0: 5 time steps • MaxIts raised at 8x CFL from 100 to 500 and then to 1000 at 32x CFL and then to 5000 at 64x CFL

22. Dey-Mittra ADI Performance: Strong Scaling

23. Full details are in the paper T. M. Austin, J. R. Cary, D. N. Smithe, C. Nieter, ”Alternating Direction Implicit Methods for FDTD using the Dey-Mittra Embedded Boundary Method,” accepted in The Open Plasma Physics Journal. Dey-Mittra ADI

24. Summary Motivation for ADI methods was discussed in the context of Dey-Mittra method. Efficient parallel tridiagonal solves were presented and performance was verified. Implementation of ADI for the Dey-Mittra method was introduced and results showed stability beyond CFL. Argument made for fast tridiagonal solves. Multiprocessor? GPUs?