1 / 29

Deep Dive: Asynchronous Compute

Deep Dive: Asynchronous Compute

sldsrv
Download Presentation

Deep Dive: Asynchronous Compute

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Deep Dive: Asynchronous Compute Stephan Hodes Developer Technology Engineer, AMD Alex Dunn Developer Technology Engineer, NVIDIA

  2. Joint Session AMD Graphics Core Next (GCN) Compute Unit (CU) Wavefronts NVIDIA Maxwell, Pascal Streaming Multiprocessor (SM) Warps ● ● ● ● ● ● 2

  3. Terminology Asynchronous: Not independent, async work shares HW Work Pairing: Items of GPU work that execute simultaneously Async. Tax: Overhead cost associated with asynchronous compute 3

  4. Async Compute  More Performance 4

  5. Queue Fundamentals 3 Queue Types: ● Copy/DMA Queue ● Compute Queue ● Graphics Queue All run asynchronously! 3D 3D COMPUTE COMPUTE COPY COPY 5

  6. General Advice ● Always profile! Can make or break perf ● Maintain non-async paths Profile async on/off Some HW won’t support async ● ‘Member hyper-threading? Similar rules apply Avoid throttling shared HW resources 3D ● COMPUTE ● ● COPY ● ● 6

  7. Regime Pairing Good Pairing Poor Pairing Graphics Shadow Render (Geometry limited) Compute Light culling (ALU heavy) Graphics Compute G-Buffer (Bandwidth limited) SSAO (Bandwidth limited) (Technique pairing doesn’t have to be 1-to-1) 7

  8. - Red Flags Problem/Solution Format Topics: ● Resource Contention - ● Descriptor heaps - ● Synchronization models ● Avoiding “async-compute tax” 8

  9. Hardware Details - 4 SIMD per CU Up to 10 Wavefronts scheduled per SIMD Accomplish latency hiding Graphics and Compute can execute simultanesouly on same CU Graphics workloads usually have priority over Compute ● ● ● ● ● 9

  10. Resource Contention – Problem: Per SIMD resources are shared between Wavefronts SIMD executes Wavefronts (of different shaders) ● Occupancy limited by ●# of registers ●Amount of LDS ●Other limits may apply… ● Wavefronts contest for caches 10

  11. Resource Contention – ● Keep an eye on vector register (VGPR) count ● Beware of cache thrashing! Try limiting occupancy by allocating dummy LDS GCN VGPR Count <=24 28 32 36 40 48 64 84 <=128 >128 Max Waves/SIMD 10 9 8 7 6 5 4 3 2 1 ● 11

  12. Hardware Details - • Compute scheduled breadth first over SMs • Compute workloads have priority over graphics • Driver heuristic controls SM distribution Maxwell Pascal Static SM partitioning Dynamic SM partitioning Idle Graphics Compute SM Distribution 12 Time

  13. Descriptor Heap - Problem: HW only has one– applications can create many Switching descriptor heap could be a hazard (on current HW) ● GPU must drain work before switching heaps ● Applies to CBV/SRV/UAV and Sampler heaps ● (Redundant changes are filtered) ● D3D: Must call SetDescriptorHeap per CL! 13

  14. Descriptor Heap - Avoid hazard if total # descriptors (all heaps) < pool size Driver sub-allocates descriptor heaps from large pool Pool sizes (Kepler+): CBV/UAV/SRV = 1048576 Sampler = 2048 + 2032 static + 16 driver owned NB. [1048575|4095]  [0xFFFFF|0xFFF] (packed into 32-bit) ● ● ● 14

  15. Synchronization GPU synchronization models to consider: ● Fire-and-forget ● Handshake CPU also has a part to play ● ExecuteCommandLists (ECLs) schedules GPU work ● Gaps between ECLs on CPU can translate to GPU 15

  16. Fire-and-Forget (Sync.) Signal 0 GPU 0  ∞ Fn Good Pairing Wait ● Work beginning synchronized via fences 16

  17. Fire-and-Forget (Sync.) Signal 0 GPU 0  ∞ Fn Bad Good Wait ● Work beginning synchronized via fences ● But, some workloads vary frame-to-frame ● Variance leads to undesired work pairing ● Impacts overall frame time as bad pairing impacts performance 17

  18. CPU Latency (Sync.) Signal 0 GPU 0  ∞ Fn Good Pairing Wait ● Similar situation – CPU plays a role here 18

  19. CPU Latency (Sync.) Signal 0 GPU 0  ∞ Fn Good Bad Wait Latency ● Similar situation – CPU plays a role here ● Game introduces latency on the CPU between ECLs ● Latency translates to GPU ● Leads to undesired work pairing, etc… 19

  20. Handshake (Sync.) Wait Signal 0 1 GPU 0 1  ∞ Fn Wait Signal ● Synchronize begin and end of work pairing ● Ensures pairing determinism ● Might miss some asynchronous opportunity (HW manageable) ● Future proof your code! 20

  21. Synchronization - Advice CPU isn’t innocent, keep an eye on it Two GPU synchronization models: Fire-and-Forget  ●Cons: Undeterministic regime pairing ●Pros: Less synchronization == more immediate performance (best case scenario) Handshake  ●Cons: Additional synchronization might cost performance ●Pros: Regime pairing determinism (all the time) Synchronize for determinism (as well as correctness) ● ● 21

  22. Async. Tax Overhead cost associated with asynchronous compute ● Quantified by: [AC-Off(ms)] / [Serialized AC-On (ms)] % ●serialize manually via graphics API ● Can easily knock out AC gains! 22

  23. Async. Tax –Root Cause CPU: Additional CPU work organizing/scheduling async tasks Synchronization/ExecuteCommandLists overhead ● ● GPU: Synchronization overhead A Difference in work ordering between AC-On/Off Different shaders used between AC-On/Off paths Additional barriers (cross-queue synchronization) ● ● ● ● 23

  24. Async. Tax –Advice First: determine if CPU or GPU is the bottleneck (GPUView) CPU: Count API calls per frame, compare AC-On/Off for differences Measure differences through per-thread profiling GPU: Compare GPU cost of shaders for AC-On/Off Inspect difference contributors ● ● ● ● 24

  25. Tools ● API Timestamps: Time enable/disable async compute ● GpuView: (PTO) 25

  26. GPU View #1 COPY Using 3D, Compute, Copy Frame boundaries @ Flip Queue packets Compute overlapping graphics per-frame • GRAPHICS COMPUTE • • 26

  27. GPU View #2 - Markers NB. Open with, ctrl + e Description • Time: GPU accurate • DataSize: size in bytes of Data • Data: Event name emitted PIXBegin/EndEvent • Byte Array  ASCII/Unicode • Manual step  27

  28. GPU View #3 - Events CPU Timeline: ID3D12Fence::Signal • DxKrnl – SignalSynchronizationObjectFromCpu ID3D12Fence::Wait • DxKrnl – WaitForSynchronizationObjectFromCpu GPU Timeline: ID3D12CommandQueue::Signal • DxKrnl – SignalSynchronizationObjectFromGpu ID3D12CommandQueue::Wait • DxKrnl – WaitForSynchronizationObjectFromGpu 28

  29. Thanks \0 Questions? @AlexWDunn - adunn@nvidia.com Stephan.Hodes@amd.com 29

More Related