Exploring Memory Consistency for Massively Threaded Throughput-Oriented Processors

1. Exploring Memory Consistency for Massively Threaded Throughput-Oriented Processors Blake Hechtman Daniel J. Sorin

2. Executive Summary • Massively Threaded Throughput-Oriented Processors (MTTOPs) like GPUs are being integrated on chips with CPUs and being used for general purpose programming • Conventional wisdom favors weak consistency on MTTOPs • We implement a range of memory consistency models (SC, TSO and RMO) on MTTOPs • We show that strong consistency is viable for MTTOPs

3. What is an MTTOP? • Massively Threaded Throughput-Oriented • 4-16 core clusters • 8-64 threads wide SIMD • 64-128 deep SMT • Thousands of concurrent threads • Massively Threaded Throughput-Oriented • Sacrifice latency for throughput • Heavily banked caches and memories • Many cores, each of which is simple

4. Example MTTOP Cache Coherent Shared Memory Memory Controller Core Cluster Core Cluster Core Cluster Core Cluster Core Cluster L2 Bank Core Cluster Core Cluster L2 Bank Fetch Decode L2 Bank Core Cluster Core Cluster L2 Bank E E E E E E E L2 Bank Core Cluster Core Cluster L2 Bank L1 L2 Bank Core Cluster Core Cluster L2Bank Core Cluster Core Cluster Core Cluster Core Cluster

5. What is Memory Consistency? InitiallyA = B = 0 Thread 0 ST B= 1 LD r1, A Thread 1 ST A= 1 LD r2, B Sequential Consistency : {r1,r2} = 0,1; 1,0; 1,1 • Weak Consistency : {r1,r2} = 0,1; 1,0; 1,1; 0,0 • enables store buffering In this work, we explore hardware consistency models MTTOP hardware concurrency seems likely to be constrained by Sequential Consistency (SC)

6. (CPU) Memory Consistency Debate • Conclusion for CPUs: trading off ~10-40% performance for programmability • “Is SC + ILP = RC?” (Gniady ISCA99) But does this conclusion apply to MTTOPs?

7. Memory Consistency on MTTOPs • GPUs have undocumented hardware consistency models • Intel MIC uses x86-TSO for the full chip with directory cache coherence protocol • MTTOP programming languages provide weak ordering guarantees • OpenCL does not guarantee store visibility without a barrier or kernel completion • CUDA includes a memory fence that can enable global store visibility

8. MTTOP ConventionalWisdom • Highly parallel systems benefit from less ordering • Graphics doesn’t need ordering • Strong Consistency seems likely to limit MLP • Strong Consistency likely to suffer extra latencies Weak ordering helps CPUs, does it help MTTOPs? It depends on how MTTOPs differ from CPUs …

9. Diff 1: Ratio of Loads to Stores Weak Consistency reduces impact of store latency on performance CPUs MTTOPs Prior work shows CPUs perform 2-4 loads per store MTTOPs perform more loads per store store latency optimizations will not be as critical to MTTOP performance

10. Diff 2: Outstanding L1 cache misses Weak consistency enables more outstanding L1 misses per thread MTTOPs have more outstanding L1 cache misses  thread reordering enabled by weak consistency is less important to handle memory latency

11. Diff 3: Memory System Latencies Weak consistency enables reductions of store latencies MTTOP core cluster CPU core Fetch Decode Issue/Sel Fetch E E E E ROB Decode E E E E E E E L2 LSQ L2 1-2 cycles 5-20 cycles 100-500 cycles L1 L1 10-70 cycles 100-300 cycles 300-1000 cycles Mem Mem MTTOPs have longer memory latencies  small latency savings will not significantly improve performance

12. Diff 4: Frequency of Synchronization Weak consistency only re-orders memory ops between sync MTTOPs CPUs split problem into regions assign regions to threads do: work on local region synchronize CPU local region: ~private cache size MTTOP local region: ~private cache size/threads per cache MTTOPs have more threads to compute a problem  each thread will have fewer independent memory ops between syncs.

13. Diff 5: RAW Dependences Through Memory Weak consistency enables store to load forwarding CPUs MTTOPs • Coalescing for cache performance • Inlined function calls • Many architected registers • Few RAW dependencies through memory • Blocking for cache performance • Frequent function calls • Few architected registers • Many RAW dependencies through memory MTTOP algorithms have fewer RAW memory dependencies  there is little benefit to being able to read from a write buffer

14. MTTOP Differences & Their Impact • Other differences are mentioned in the paper • How much do these differences affect performance of memory consistency implementations on MTTOPs?

15. Memory Consistency Implementations Strongest Weakest TSO SC simple SC wb RMO Fetch Fetch Fetch Fetch Decode Decode Decode Decode E E E E E E E E E E E E E E E E E E E E E E E E E E E E FIFO WB FIFO WB CAM L1 L1 L1 L1 No write buffer Per-lane FIFO write buffer drained on FENCES Per-lane FIFO write buffer drained on LOADS Per-lane CAM for outstanding write addresses

16. Methodology • Modified gem5 to support SIMT cores running a modified version of the Alpha ISA • Looked at typical MTTOP workloads • Had to port workloads to run in system model • Ported Rodinia benchmarks • bfs, hotspot, kmeans, and nn • Handwritten benchmarks • dijkstra, 2dconv, and matrix_mul

17. Target MTTOP System

18. Results

19. Results

20. Conclusions • Strong Consistency should not be ruled out for MTTOPs on the basis of performance • Improving store performance with write buffers appears unnecessary • Graphics-like workloads may get significant MLP from load reordering (dijkstra, 2dconv) Conventional wisdom may be wrong about MTTOPs

21. Caveats and Limitations • Results do not necessarily apply to all possible MTTOPs or MTTOP software • Evaluation with writeback caches when current MTTOPs use write-through caches • Potential of future workloads to be more CPU-like

22. Thank you! Questions?