P

Actual partition

proc = row

For a balanced partitioning, one could assume:

N

3

4

1

2

Assumed partition

3

4

1

2

Rows owned by 1,

In 2’s assumed partition

A2

AP

A1

A =

Scalable Conceptual Interfaces in hypre

Allison Baker

Center for Applied Scientific Computing, Lawrence Livermore National Laboratory

(joint work with Rob Falgout, Jim Jones, and Ulrike Meier Yang)

Overview

New Algorithm

What is hypre?

Goal: Generate neighborhood information in a scalable manner for large numbers of processors (P)

- A library of high-performance algorithms for solving large, sparse systems of linear equations on massively parallel computers

How? Assume the global partition!

Scalability: The key issue for large-scale computing

- The new algorithm is a kind of rendezvous algorithm that uses the concept of an assumed partition to answer queries about the global data distribution

- Good performance requires scalable algorithms and software

- Assumed Partition algorithm:
- Assume a global partition of data (N rows) that may be queried by any processor (with O(1) computation and storage cost)
- Reconcile assumed rows with actual rows – contact processors regarding rows owned in another’s assumed partition
- Use the assumed partition to determine send and receive processors

Parallel computing data is in distributed form

- Conceptual interface gets data from application code to hypre
- Each processor knows only its own piece of the linear system
- Problem: Solvers require “nearby” data from other processors and the interfaces must determine who owns this data efficiently

Goal: Scalable interfaces to solvers!

- LLNL’s new BlueGene/L machine > 100,000 processors!

After step 2, processors know who owns the data in their assumed partitions— now the assumed partition defines the rendezvous points

IJ Conceptual Interface

Description

- This assumed partition concept is applicable to all of hypre’s conceptual interfaces and a variety of situations in parallel codes

- hypre’s traditional linear-algebraic interface
- Matrix and right-hand side are defined in terms of row and column indices

New algorithm costs for P processors

Matrices are distributed across P processors by contiguous blocks of rows

Results

- Matrix-vector multiply requires some knowledge of the global partition

Comparison of the new assumed partition algorithm and the old algorithm for a 3D Laplacian operator with a 27-point stencil

Old method for determining neighborhood info

- Each processor owns ~64,000 rows
- Runs on LLNL’s MCR Linux cluster
- New algorithm has better scaling properties—This will be important for 100,000 processors!

- Each processor sends its range to all other processors
- All processors store the global partition and use it to determine who to receive data from (receive processors)
- Processors discover who to send data to (send processors) via a second communication

Old method costs for P processors

Not good enough!

As P increases,

the algorithm’s |cost increases!

What’s Next?

- Testing on more processors using BlueGene/L—16K processors coming soon!
- Adapting the assumed partition to the hypre conceptual interface for structured problems—more complicated!

For scalability, the computation, communications and storage costs should all depend on P logarithmically or better!

This work was performed under the auspices of the U.S. Department of Energy by University of California Lawrence Livermore National Laboratory under contract No. W-7405-Eng-48. UCRL-POST-209333