A Case Study of Communication Optimizations on 3D Mesh Interconnects

1. Abhinav Bhatele, Eric Bohm, Laxmikant V. Kale Parallel Programming Laboratory A Case Study of Communication Optimizations on 3D Mesh Interconnects University of Illinois at Urbana-Champaign Euro-Par 2009

2. Outline • Motivation • Solution: Mapping of OpenAtom • Performance Benefits • Bigger Picture: • Resources Needed • Heuristic Solutions • Automatic Mapping Abhinav Bhatele @ Euro-Par 2009

3. OpenAtom • Ab-Initio Molecular Dynamics code • Consider electrostatic interactions between the nuclei and electrons • Calculate different energy terms • Divided into different phases with lot of communication Abhinav Bhatele @ Euro-Par 2009

4. OpenAtom on Blue Gene/L w32 = 32 water molecules with 70 Ry cutoff Runs on Blue Gene/L at IBM T J Watson Research Center, CO mode Abhinav Bhatele @ Euro-Par 2009

5. The problem lies in … Performance Analysis and Visualization Tool: Projections (part of Charm++) – Timeline View Abhinav Bhatele @ Euro-Par 2009

6. Solution – Topology Aware Mapping Abhinav Bhatele @ Euro-Par 2009

7. Processor Virtualization Programmer: Decomposes the computation into objects Runtime: Maps the computation on to the processors User View System View Abhinav Bhatele @ Euro-Par 2009

8. Benefits of Charm++ • Computation is divided into objects/chares/virtual processors (VPs) • Separates decomposition from mapping • VPs can be flexibly mapped to actual physical processors (PEs) Abhinav Bhatele @ Euro-Par 2009

9. Topology Manager API† • The application needs information such as • Dimensions of the partition • Rank to physical co-ordinates and vice-versa • TopoManager: a uniform API • On BG/L and BG/P: provides a wrapper for system calls • On XT3/4/5, there are no such system calls • Provides a clean and uniform interface to the application † http://charm.cs.uiuc.edu/~bhatele/phd/topomgr.htm Abhinav Bhatele @ Euro-Par 2009

10. Parallelization using Charm++ Eric Bohm, Glenn J. Martyna, Abhinav Bhatele, Sameer Kumar, Laxmikant V. Kale, John A. Gunnels, and Mark E. Tuckerman. Fine Grained Parallelization of the Car-Parrinelloab initio MD Method on Blue Gene/L. IBM J. of R. and D.: Applications of Massively Parallel Systems, 52(1/2):159-174, 2008. Abhinav Bhatele @ Euro-Par 2009

11. Mapping Challenge • Load Balancing: Multiple VPs per PE • Multiple groups of communicating objects • Intra-group communication • Inter-group communication • Conflicting communication requirements Abhinav Bhatele @ Euro-Par 2009

12. Topology Mapping of Chare Arrays RealSpace and GSpace have state-wise communication Paircalculator and GSpace have plane-wise communication Abhinav Bhatele @ Euro-Par 2009

13. Performance Improvements on BG/L w32 = 32 water molecules with 70 Ry cutoff Runs on Blue Gene/L at IBM T J Watson Research Center, CO mode, Year: 2006 Abhinav Bhatele @ Euro-Par 2009

14. Improved Timeline Views Abhinav Bhatele @ Euro-Par 2009

15. Results on Blue Gene/L GST_BIG = 64 Ge, 128 Sb and 256 Te molecules with 20 Ry cutoff Runs on Blue Gene/L at IBM T J Watson Research Center, CO mode Abhinav Bhatele @ Euro-Par 2009

16. Results on Blue Gene/P w256 = 256 water molecules with 70 Ry cutoff Runs on Blue Gene/P at Argonne National Laboratory, VN mode Abhinav Bhatele @ Euro-Par 2009

17. Results on Cray XT3 Runs on Cray XT3 (Bigben) at Pittsburgh Supercomputing Center, VN mode (with system reservation to obtain complete 3d mesh shapes) Abhinav Bhatele @ Euro-Par 2009

18. Performance Analysis w256M_70Ry on Blue Gene/L Performance Analysis and Visualization Tool: Projections – Idle time added across all processors Abhinav Bhatele @ Euro-Par 2009

19. Reduction in Communication Volume w256M_70Ry on Blue Gene/P Data obtained from Blue Gene/P’s Uniform Performance Counters Abhinav Bhatele @ Euro-Par 2009

20. Relative Performance Improvement w256M_70Ry Abhinav Bhatele @ Euro-Par 2009

21. Bigger picture • Different kinds of applications: • Computation bound • Communication bound • Latency tolerant • Latency sensitive • Technique: • Obtain processor topology and application communication graph • Heuristic Techniques for mapping Abhinav Bhatele @ Euro-Par 2009

22. Why does distance affect message latencies? • Consider a 3D mesh/torus interconnect • Message latencies can be modeled by (Lf/B) x D + L/B Lf = length of flit, B = bandwidth, D = hops, L = message size When (Lf * D) << L, first term is negligible But in presence of contention … Abhinav Bhatele @ Euro-Par 2009

23. Automatic Topology Aware Mapping • Many MPI applications exhibit a simple two-dimensional near-neighbor communication pattern • Examples: MILC, WRF, POP, Stencil, … Abhinav Bhatele @ Euro-Par 2009

24. E-mail: bhatele@illinois.edu Webpage: http://charm.cs.illinois.edu DOE Grant B341494 (CSAR), DOE Grant DE-FG05-08OR23332 (ORNL LCF) and NSF Grant ITR 0121357 Funding Acknowledgements: Shawn Brown and Chad Vizino (PSC) Glenn Martyna, Sameer Kumar, Fred Mintzer (IBM) Teragrid for running time on Bigben (XT3) ANL for running time on Blue Gene/P Abhinav Bhatele @ Euro-Par 2009