Collective Communication on Architectures that Support Simultaneous Communication over Multiple Link...
Download
1 / 37

Ernie Chan - PowerPoint PPT Presentation


  • 80 Views
  • Uploaded on

Collective Communication on Architectures that Support Simultaneous Communication over Multiple Links. Ernie Chan. Ernie Chan Robert van de Geijn Department of Computer Sciences The University of Texas at Austin. William Gropp Rajeev Thakur Mathematics and Computer Science Division

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Ernie Chan' - lareina-giles


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Collective Communication on Architectures that Support Simultaneous Communication over Multiple Links

Ernie Chan


Authors

Ernie Chan Simultaneous Communication over Multiple Links

Robert van de Geijn

Department of Computer Sciences

The University of Texas at Austin

William Gropp

Rajeev Thakur

Mathematics and Computer Science Division

Argonne National Laboratory

Authors


Testbed architecture
Testbed Architecture Simultaneous Communication over Multiple Links

  • IBM Blue Gene/L

    • 3D torus point-to-point interconnect network

    • One rack

      • 1024 dual-processor nodes

      • Two 8 x 8 x 8 midplanes

    • Special feature to send simultaneously

      • Use multiple calls to MPI_Isend


Outline
Outline Simultaneous Communication over Multiple Links

  • Testbed Architecture

  • Model of Parallel Computation

  • Sending Simultaneously

  • Collective Communication

  • Generalized Algorithms

  • Performance Results

  • Conclusion


Model of parallel computation
Model of Parallel Computation Simultaneous Communication over Multiple Links

  • Target Architectures

    • Distributed-memory parallel architectures

  • Indexing

    • p computational nodes

    • Indexed 0 … p - 1

  • Logically Fully Connected

    • A node can send directly to any other node


Model of parallel computation1
Model of Parallel Computation Simultaneous Communication over Multiple Links

  • Topology

    • N-dimensional torus

0

1

2

3

4

5

6

7

9

10

11

8

12

14

15

13


Model of parallel computation2
Model of Parallel Computation Simultaneous Communication over Multiple Links

  • Old Model of Communicating Between Nodes

    • Unidirectional sending or receiving


Model of parallel computation3
Model of Parallel Computation Simultaneous Communication over Multiple Links

  • Old Model of Communicating Between Nodes

    • Simultaneous sending and receiving


Model of parallel computation4
Model of Parallel Computation Simultaneous Communication over Multiple Links

  • Old Model of Communicating Between Nodes

    • Bidirectional exchange


Model of parallel computation5
Model of Parallel Computation Simultaneous Communication over Multiple Links

  • Communicating Between Nodes

    • A node can send or receive with 2N other nodes simultaneously along its 2N different links


Model of parallel computation6
Model of Parallel Computation Simultaneous Communication over Multiple Links

  • Communicating Between Nodes

    • Cannot perform bidirectional exchange on any link while sending or receiving simultaneously with multiple nodes


Model of parallel computation7
Model of Parallel Computation Simultaneous Communication over Multiple Links

  • Cost of Communication

    α + nβ

    • α: startup time, latency

    • n: number of bytes to communicate

    • β: per data transmission time, bandwidth


Outline1
Outline Simultaneous Communication over Multiple Links

  • Testbed Architecture

  • Model of Parallel Computation

  • Sending Simultaneously

  • Collective Communication

  • Generalized Algorithms

  • Performance Results

  • Conclusion


Sending simultaneously
Sending Simultaneously Simultaneous Communication over Multiple Links

  • Old Cost of Communication with Sends to Multiple Nodes

    • Cost to send to m separate nodes

      (α + nβ) m


Sending simultaneously1
Sending Simultaneously Simultaneous Communication over Multiple Links

  • New Cost of Communication with Simultaneous Sends

    (α + nβ) m

    can be replaced with

(α + nβ) + (α + nβ) (m - 1)


Sending simultaneously2
Sending Simultaneously Simultaneous Communication over Multiple Links

  • New Cost of Communication with Simultaneous Sends

    (α + nβ) m

    can be replaced with

(α + nβ) + (α + nβ) (m - 1) τ

Cost of one send

Cost of extra sends


Sending simultaneously3
Sending Simultaneously Simultaneous Communication over Multiple Links

  • New Cost of Communication with Simultaneous Sends

    (α + nβ) m

    can be replaced with

0 ≤τ ≤ 1

(α + nβ) + (α + nβ) (m - 1) τ

Cost of one send

Cost of extra sends


Sending simultaneously4
Sending Simultaneously Simultaneous Communication over Multiple Links

  • Benchmarking Sending Simultaneously

    • Logarithmic-Logarithmic timing graphs

    • Midplane – 512 nodes

    • Sending simultaneously with 1 – 6 neighbors

    • 8 bytes – 4 MB


Sending simultaneously5
Sending Simultaneously Simultaneous Communication over Multiple Links


Sending simultaneously6
Sending Simultaneously Simultaneous Communication over Multiple Links

  • Cost of Communication with Simultaneous Sends

    (α + nβ) (1 + (m - 1) τ)


Sending simultaneously7
Sending Simultaneously Simultaneous Communication over Multiple Links


Sending simultaneously8
Sending Simultaneously Simultaneous Communication over Multiple Links


Outline2
Outline Simultaneous Communication over Multiple Links

  • Testbed Architecture

  • Model of Parallel Computation

  • Sending Simultaneously

  • Collective Communication

  • Generalized Algorithms

  • Performance Results

  • Conclusion


Collective communication
Collective Communication Simultaneous Communication over Multiple Links

  • Broadcast (Bcast)

    • Motivating example

      Before After


Outline3
Outline Simultaneous Communication over Multiple Links

  • Testbed Architecture

  • Model of Parallel Computation

  • Sending Simultaneously

  • Collective Communication

  • Generalized Algorithms

  • Performance Results

  • Conclusion


Generalized algorithms
Generalized Algorithms Simultaneous Communication over Multiple Links

  • Short-Vector Algorithms

    • Minimum-Spanning Tree

  • Long-Vector Algorithms

    • Bucket Algorithm


Generalized algorithms1
Generalized Algorithms Simultaneous Communication over Multiple Links

  • Minimum-Spanning Tree


Generalized algorithms2
Generalized Algorithms Simultaneous Communication over Multiple Links

  • Minimum-Spanning Tree

    • Divide p nodes into N+1 partitions


Generalized algorithms3
Generalized Algorithms Simultaneous Communication over Multiple Links

  • Minimum-Spanning Tree

    • Disjointed partitions on N-dimensional mesh

0

1

2

3

4

5

6

7

9

10

11

8

12

14

15

13


Generalized algorithms4
Generalized Algorithms Simultaneous Communication over Multiple Links

  • Minimum-Spanning Tree

    • Divide dimensions by a decrementing counter from N+1

0

1

2

3

4

5

6

7

9

10

11

8

12

14

15

13


Generalized algorithms5
Generalized Algorithms Simultaneous Communication over Multiple Links

  • Minimum-Spanning Tree

    • Now divide into 2N+1 partitions

0

1

2

3

4

5

6

7

9

10

11

8

12

14

15

13


Outline4
Outline Simultaneous Communication over Multiple Links

  • Testbed Architecture

  • Model of Parallel Computation

  • Sending Simultaneously

  • Collective Communication

  • Generalized Algorithms

  • Performance Results

  • Conclusion


Performance results
Performance Results Simultaneous Communication over Multiple Links

Single point-to-point

communication


Performance results1
Performance Results Simultaneous Communication over Multiple Links

my-bcast-MST


Outline5
Outline Simultaneous Communication over Multiple Links

  • Testbed Architecture

  • Model of Parallel Computation

  • Sending Simultaneously

  • Collective Communication

  • Generalized Algorithms

  • Performance Results

  • Conclusion


Conclusion
Conclusion Simultaneous Communication over Multiple Links

  • IBM Blue Gene/L supports functionality of sending simultaneously

    • Benchmarking along with model checking verifies this claim

  • New generalized algorithms show clear performance gains


Conclusion1
Conclusion Simultaneous Communication over Multiple Links

  • Future Directions

    • Room for optimization to reduce implementation overhead

    • What if not using MPI_COMM_WORLD?

    • Possible new algorithm for Bucket Algorithm

  • Questions? [email protected]


ad