1 / 40

Partitioning, Load Balancing, and Ordering for Petascale Applications

arvid
Download Presentation

Partitioning, Load Balancing, and Ordering for Petascale Applications

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

    1. Partitioning, Load Balancing, and Ordering for Petascale Applications Erik Boman, Cedric Chevalier, Karen Devine, Bruce Hendrickson, Sandia National Labs Umit Çatalyürek, Ohio State University Michael Wolf, UIUC and Sandia

    3. Interesting Times Supercomputers becoming more complex Hierarchical: Nodes (CMP), processors, cores Heterogeneous: Accelerators, e.g., Cell, GPU How to deal with millions of cores (threads) Multicore impacts desktops to HPC Flops are cheap, data throughput/latency is key Rethink algorithms, software, libraries Programming model/environment uncertainty Is MPI enough? PGAS languages? Hybrid? How can apps use these computers efficiently? Data distribution increasingly important

    4. Partitioning and Load Balancing Assignment of application data to processors for parallel computation. Applied to grid points, elements, matrix rows, particles, ….

    5. Static Partitioning Static partitioning in an application: Data partition is computed. Data are distributed according to partition map. Application computes. Ideal partition: Processor idle time is minimized. Inter-processor communication costs are kept low.

    6. Dynamic Repartitioning (a.k.a. Dynamic Load Balancing) Dynamic repartitioning (load balancing) in an application: Data partition is computed. Data are distributed according to partition map. Application computes and, perhaps, adapts. Process repeats until the application is done. Ideal partition: Processor idle time is minimized. Inter-processor communication costs are kept low. Cost to redistribute data is also kept low.

    7. What makes a partition “good,” especially at petascale? Balanced work loads. Even small imbalances result in many wasted processors! 100,000 processors with one processor 5% over average workload is equivalent to 4760 idle processors and the rest perfectly balanced. Low interprocessor communication costs. Processor speeds increasing faster than network speeds. Partitions with minimal communication costs are critical. Scalable partitioning time and memory use. Scalability is especially important for dynamic partitioning. Low data redistribution costs for dynamic partitioning. (Umit will discuss dynamic repartitioning)

    8. Zoltan Toolkit No single method best in all cases Provide collection of algorithms: Zoltan Data-structure neutral interface Application callbacks Fully parallel Based on MPI Successfully used on up to 20K cores

    9. Partitioning Algorithms in the Zoltan Toolkit

    10. Recursive Coordinate Bisection: Developed by Berger & Bokhari (1987) for Adaptive Mesh Refinement. Idea: Divide work into two equal parts using a cutting plane orthogonal to a coordinate axis. Recursively cut the resulting subdomains. Geometric Partitioning

    11. Applications of Geometric Methods

    12. Graph Partitioning Represent problem as a weighted graph. Vertices = objects to be partitioned. Edges = dependencies between two objects. Weights = work load or amount of dependency. Partition graph so that … Parts have equal vertex weight. Weight of edges cut by part boundaries is small.

    13. Applications using Graph Partitioning

    14. Hypergraph Partitioning Alpert, Kahng, Hauck, Borriello, Çatalyürek, Aykanat, Karypis, et al. Hypergraph model: Vertices = objects to be partitioned. Hyperedges = dependencies between two or more objects. Partitioning goal: Assign equal vertex weight while minimizing hyperedge cut weight.

    15. Hypergraph Applications

    16. Hypergraph Partitioning: Advantages and Disadvantages Advantages: Communication volume reduced 30-38% on average over graph partitioning (Catalyurek & Aykanat). 5-15% reduction for mesh-based applications. More accurate communication model than graph partitioning. Better representation of highly connected and/or non-homogeneous systems. Greater applicability than graph model. Can represent rectangular systems and non-symmetric dependencies. Disadvantages: More expensive than graph partitioning.

    17. Performance Results Experiments on Sandia’s Thunderbird cluster. Dual 3.6 GHz Intel EM64T processors with 6 GB RAM. Infiniband network. Compare RCB, graph (ParMETIS) and hypergraph (Zoltan) methods. Measure … Amount of communication induced by the partition. Partitioning time.

    18. Test Data

    19. Communication Volume: Lower is Better

    20. Partitioning Time: Lower is better

    21. Aiming for Petascale Hierarchical partitioning in Zoltan v3. Partition for multicore/manycore architectures. Partition hierarchically with respect to chips and then cores. Similar to strategies for clusters of SMPs (Teresco, Faik). Treat core-level partitions as separate threads or MPI processes (application decides) Support 100Ks processors (millions of cores) Reduce collective communication operations during partitioning. Allow more localized partitioning on subsets of processors.

    22. Aiming for Petascale Reducing communication costs for applications. Reducing communication volume. Two-dimensional sparse matrix partitioning (Catalyurek, Bisseling, Boman, Wolf). Partitioning non-zeros of matrix rather than rows/columns. Reducing message latency. Minimize maximum number of neighboring parts (messages) Balancing both computation and communication (Pinar & Hendrickson); balance criterion is complex function of the partition instead of simple sum of object weights. Exploit hierarchical structure, topology Map parts onto processors to take advantage of network topology.

    23. Expanding scope of Zoltan Now supports three combinatorial problems Partitioning Ordering Coloring Focus on large problems, parallel scalability

    24. Sparse Matrix Ordering Fill-reducing ordering for Ax=b SPD case: Nested Dissection Provided in Zoltan via external libraries PT-Scotch and ParMetis PT-Scotch better for large #cores (Chevalier) CEA (France) solved 3D system of order 45 million in ~30 minutes on their TERA computer

    25. New Unsymmetric Method: HUND HUND = Hypergraph Unsymmetric Nested Dissection Grigori, Boman, Donfack, Davis, ‘08 Reduces fill in LU but allows pivoting Based on recursive bisection using hypergraphs Works directly on unsymmetric structure More robust than COLAMD Will implement in Zoltan, make available in SuperLU

    26. HUND Results

    27. Zoltan Integration into SciDAC ITAPS Dynamic services for meshes Trilinos Matrix partitioning via Isorropia PETSc Unstructured mesh partitioning in Sieve SuperLU Matrix ordering (planned) COMPASS/SLAC Accelerator modeling, PIC Wanted: More application collaborations

    28. Questions to (Potential) Users How can we make our software tools easier to use? What are your current bottlenecks? What new features do you need?

    29. For More Information… CSCAPES: http://www.cscapes.org Zoltan web page: http://www.cs.sandia.gov/Zoltan Download Zoltan v3 (open-source software). Read User’s Guide, try examples Zoltan tutorial Thursday 8:30 am

    30. Thanks S. Attaway (SNL) C. Aykanat (Bilkent U.) A. Bauer (RPI) R. Bisseling (Utrecht U.) D. Bozdag (Ohio St. U.) T. Davis (U. Florida) J. Faik (RPI) J. Flaherty (RPI) L. Grigori (INRIA) R. Heaphy (SNL) M. Heroux (SNL) D. Keyes (Columbia) K. Ko (SLAC) G. Kumfert (LLNL) L.-Q. Lee (SLAC) V. Leung (SNL) G. Lonsdale (NEC) X. Luo (RPI) L. Musson (SNL) S. Plimpton (SNL) L.A. Riesen (SNL) J. Shadid (SNL) M. Shephard (RPI) C. Silva (SNL) J. Teresco (Mount Holyoke) C. Vaughan (SNL)

    32. Geometric Repartitioning Implicitly achieves low data redistribution costs. For small changes in data, cuts move only slightly, resulting in little data redistribution. ANIMATE FOR PRESENTATION INSTEAD OF USING THREE PICTURES.ANIMATE FOR PRESENTATION INSTEAD OF USING THREE PICTURES.

    33. RCB Advantages and Disadvantages Advantages: Conceptually simple; fast and inexpensive. All processors can inexpensively know entire partition (e.g., for global search in contact detection). No connectivity info needed (e.g., particle methods). Good on specialized geometries. Disadvantages: No explicit control of communication costs. Mediocre partition quality. Can generate disconnected subdomains for complex geometries. Need coordinate information.

    34. Graph Partitioning: Advantages and Disadvantages Advantages: Highly successful model for mesh-based PDE problems. Explicit control of communication volume gives higher partition quality than geometric methods. Excellent software available. Serial: Chaco (SNL) Jostle (U. Greenwich) METIS (U. Minn.) Scotch (U. Bordeaux) Parallel: Zoltan (SNL) ParMETIS (U. Minn.) PJostle (U. Greenwich) PT-Scotch (U. Bordeaux) Disadvantages: More expensive than geometric methods. Edge-cut model only approximates communication volume.

    35. Graph Repartitioning Diffusive strategies (Cybenko, Hu, Blake, Walshaw, Schloegel, et al.) Shift work from highly loaded processors to less loaded neighbors. Local communication keeps data redistribution costs low. Multilevel partitioners that account for data redistribution costs in refining partitions (Schloegel, Karypis) Parameter weights application communication vs. redistribution communication.

    36. Hypergraph Repartitioning Augment hypergraph with data redistribution costs. Account for data’s current processor assignments. Weight dependencies by their size and frequency of use. Hypergraph partitioning then attempts to minimize total communication volume: Data redistribution volume + Application communication volume Total communication volume Lower total communication volume than geometric and graph repartitioning.

    37. Communication Volume: Lower is Better

    38. Partitioning Time: Lower is better

    39. Repartitioning Experiments Experiments with 64 parts on 64 processors. Dynamically adjust weights in data to simulate, say, adaptive mesh refinement. Repartition. Measure repartitioning time and total communication volume: Data redistribution volume + Application communication volume Total communication volume

    40. Repartitioning Results: Lower is Better

    41. Improving scalability of partitioning algorithms. Hybrid partitioners (for mesh-based apps.): Use inexpensive geometric methods for initial partitioning; refine with hypergraph/graph-based algorithms at boundaries. Use geometric information for fast coarsening in multilevel hypergraph/graph-based partitioners. Aiming for Petascale

More Related