1 / 37

Ernie Chan

Collective Communication on Architectures that Support Simultaneous Communication over Multiple Links. Ernie Chan. Ernie Chan Robert van de Geijn Department of Computer Sciences The University of Texas at Austin. William Gropp Rajeev Thakur Mathematics and Computer Science Division

lareina-giles
Download Presentation

Ernie Chan

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Collective Communication on Architectures that Support Simultaneous Communication over Multiple Links Ernie Chan

  2. Ernie Chan Robert van de Geijn Department of Computer Sciences The University of Texas at Austin William Gropp Rajeev Thakur Mathematics and Computer Science Division Argonne National Laboratory Authors

  3. Testbed Architecture • IBM Blue Gene/L • 3D torus point-to-point interconnect network • One rack • 1024 dual-processor nodes • Two 8 x 8 x 8 midplanes • Special feature to send simultaneously • Use multiple calls to MPI_Isend

  4. Outline • Testbed Architecture • Model of Parallel Computation • Sending Simultaneously • Collective Communication • Generalized Algorithms • Performance Results • Conclusion

  5. Model of Parallel Computation • Target Architectures • Distributed-memory parallel architectures • Indexing • p computational nodes • Indexed 0 … p - 1 • Logically Fully Connected • A node can send directly to any other node

  6. Model of Parallel Computation • Topology • N-dimensional torus 0 1 2 3 4 5 6 7 9 10 11 8 12 14 15 13

  7. Model of Parallel Computation • Old Model of Communicating Between Nodes • Unidirectional sending or receiving

  8. Model of Parallel Computation • Old Model of Communicating Between Nodes • Simultaneous sending and receiving

  9. Model of Parallel Computation • Old Model of Communicating Between Nodes • Bidirectional exchange

  10. Model of Parallel Computation • Communicating Between Nodes • A node can send or receive with 2N other nodes simultaneously along its 2N different links

  11. Model of Parallel Computation • Communicating Between Nodes • Cannot perform bidirectional exchange on any link while sending or receiving simultaneously with multiple nodes

  12. Model of Parallel Computation • Cost of Communication α + nβ • α: startup time, latency • n: number of bytes to communicate • β: per data transmission time, bandwidth

  13. Outline • Testbed Architecture • Model of Parallel Computation • Sending Simultaneously • Collective Communication • Generalized Algorithms • Performance Results • Conclusion

  14. Sending Simultaneously • Old Cost of Communication with Sends to Multiple Nodes • Cost to send to m separate nodes (α + nβ) m

  15. Sending Simultaneously • New Cost of Communication with Simultaneous Sends (α + nβ) m can be replaced with (α + nβ) + (α + nβ) (m - 1)

  16. Sending Simultaneously • New Cost of Communication with Simultaneous Sends (α + nβ) m can be replaced with (α + nβ) + (α + nβ) (m - 1) τ Cost of one send Cost of extra sends

  17. Sending Simultaneously • New Cost of Communication with Simultaneous Sends (α + nβ) m can be replaced with 0 ≤τ ≤ 1 (α + nβ) + (α + nβ) (m - 1) τ Cost of one send Cost of extra sends

  18. Sending Simultaneously • Benchmarking Sending Simultaneously • Logarithmic-Logarithmic timing graphs • Midplane – 512 nodes • Sending simultaneously with 1 – 6 neighbors • 8 bytes – 4 MB

  19. Sending Simultaneously

  20. Sending Simultaneously • Cost of Communication with Simultaneous Sends (α + nβ) (1 + (m - 1) τ)

  21. Sending Simultaneously

  22. Sending Simultaneously

  23. Outline • Testbed Architecture • Model of Parallel Computation • Sending Simultaneously • Collective Communication • Generalized Algorithms • Performance Results • Conclusion

  24. Collective Communication • Broadcast (Bcast) • Motivating example Before After

  25. Outline • Testbed Architecture • Model of Parallel Computation • Sending Simultaneously • Collective Communication • Generalized Algorithms • Performance Results • Conclusion

  26. Generalized Algorithms • Short-Vector Algorithms • Minimum-Spanning Tree • Long-Vector Algorithms • Bucket Algorithm

  27. Generalized Algorithms • Minimum-Spanning Tree

  28. Generalized Algorithms • Minimum-Spanning Tree • Divide p nodes into N+1 partitions

  29. Generalized Algorithms • Minimum-Spanning Tree • Disjointed partitions on N-dimensional mesh 0 1 2 3 4 5 6 7 9 10 11 8 12 14 15 13

  30. Generalized Algorithms • Minimum-Spanning Tree • Divide dimensions by a decrementing counter from N+1 0 1 2 3 4 5 6 7 9 10 11 8 12 14 15 13

  31. Generalized Algorithms • Minimum-Spanning Tree • Now divide into 2N+1 partitions 0 1 2 3 4 5 6 7 9 10 11 8 12 14 15 13

  32. Outline • Testbed Architecture • Model of Parallel Computation • Sending Simultaneously • Collective Communication • Generalized Algorithms • Performance Results • Conclusion

  33. Performance Results Single point-to-point communication

  34. Performance Results my-bcast-MST

  35. Outline • Testbed Architecture • Model of Parallel Computation • Sending Simultaneously • Collective Communication • Generalized Algorithms • Performance Results • Conclusion

  36. Conclusion • IBM Blue Gene/L supports functionality of sending simultaneously • Benchmarking along with model checking verifies this claim • New generalized algorithms show clear performance gains

  37. Conclusion • Future Directions • Room for optimization to reduce implementation overhead • What if not using MPI_COMM_WORLD? • Possible new algorithm for Bucket Algorithm • Questions? echan@cs.utexas.edu

More Related