CENG 450 Computer Systems and Architecture Lecture 8

1. CENG 450Computer Systems and ArchitectureLecture 8 Amirali Baniasadi amirali@ece.uvic.ca

2. This Lecture • Review: Loop Unrolling • Dynamic Scheduling • Scoreboard

3. Simple Loop & Assembler Equivalent • x[i] & s are double/floating point type • R1 initially address of array element with the highest address • F2 contains the scalar value s • Register R2 is pre-computed so that 8(R2) is the last element to operate on • for (i=1000; i>0; i--) x[i] = x[i] + s; • Loop: LD F0, 0(R1) ;F0=array element • ADDD F4, F0, F2 ;add scalar in F2 • SD F4 , 0(R1) ;store result • SUBI R1, R1, #8 ;decrement pointer 8bytes (DW) • BNE R1, R2, Loop ;branch R1!=R2

4. Unscheduled Loop: LD F0, 0(R1) stall ADDD F4, F0, F2 stall stall SD F4, 0(R1) SUBI R1, R1, #8 stall BNE R1, R2, Loop stall 10 clock cycles Can we minimize? Scheduled Loop: LD F0, 0(R1) SUBI R1, R1, #8 ADDD F4, F0, F2 stall BNE R1, R2, Loop SD F4, 8(R1) 6 clock cycles 3 cycles: actual work; 3 cycles: overhead Can we minimize further? Where are the stalls? Schedule • Source instruction • Dependant instruction • Latency (clock cycles) • FP ALU op • Another FP ALU op • 3 • FP ALU op • Store double • 2 • Load double • FP ALU op • 1 • Load double • Store double • 0

5. LD F0, 0(R1) ADDD F4, F0, F2 SD F4 , 0(R1) SUBI R1, R1, #8 BNE R1, R2, Loop LD F0, 0(R1) ADDD F4, F0, F2 SD F4 , 0(R1) SUBI R1, R1, #8 BNE R1, R2, Loop LD F0, 0(R1) ADDD F4, F0, F2 SD F4 , 0(R1) SUBI R1, R1, #8 BNE R1, R2, Loop LD F0, 0(R1) ADDD F4, F0, F2 SD F4 , 0(R1) SUBI R1, R1, #8 BNE R1, R2, Loop Loop: LD F0, 0(R1) ADDD F4, F0, F2 SD F4, 0(R1) LD F6, -8(R1) ADDD F8, F6, F2 SD F8, -8(R1) LD F10, -16(R1) ADDD F12, F10, F2 SD F12, -16(R1) LD F14, -24(R1) ADDD F16, F14, F2 SD F16, -24(R1) SUBI R1, R1, #32 BNE R1, R2, Loop Loop Unrolling Eliminate Incr, Branch LD F0, 0(R1) ADDD F4, F0, F2 SD F4 , 0(R1) SUBI R1, R1, #8 BNE R1, R2, Loop LD F0, -8(R1) ADDD F4, F0, F2 SD F4 , -8(R1) SUBI R1, R1, #8 BNE R1, R2, Loop LD F0, -16(R1) ADDD F4, F0, F2 SD F4 , -16(R1) SUBI R1, R1, #8 BNE R1, R2, Loop LD F0, -24(R1) ADDD F4, F0, F2 SD F4 , -24(R1) SUBI R1, R1, #32 BNE R1, R2, Loop Four copies of loop Four iteration code Assumption: R1 is initially a multiple of 32 or number of loop iterations is a multiple of 4

6. Loop: LD F0, 0(R1) stall ADDD F4, F0, F2 stall stall SD F4, 0(R1) LD F6, -8(R1) stall ADDD F8, F6, F2 stall stall SD F8, -8(R1) LD F10, -16(R1) stall ADDD F12, F10, F2 stall stall SD F12, -16(R1) LD F14, -24(R1) stall ADDD F16, F14, F2 stall stall SD F16, -24(R1) SUBI R1, R1, #32 stall BNE R1, R2, Loop stall Loop Unroll & Schedule Loop: LD F0, 0(R1) LD F6, -8(R1) LD F10, -16(R1) LD F14, -24(R1) ADDD F4, F0, F2 ADDD F8, F6, F2 ADDD F12, F10, F2 ADDD F16, F14, F2 SD F4, 0(R1) SD F8, -8(R1) SD F12, -16(R1) SUBI R1, R1, #32 BNE R1, R2, Loop SD F16, 8(R1) Schedule No stalls! 14 clock cycles or 3.5 per iteration Can we minimize further? 28 clock cycles or 7 per iteration Can we minimize further?

7. Data Dependence • Data dependence • Indicates the possibility of a hazard • Determines the order in which results must be calculated • Sets upper bound on how much parallelism can be exploited • But, actual hazard & length of any stall is determined by pipeline • Dependence avoidance • Maintain the dependence but avoid hazard: Scheduling • Eliminate dependence by transforming the code

8. Data Dependencies • 1 Loop: LD F0, 0(R1) • 2 ADDD F4, F0, F2 • 3 SUBI R1, R1, 8 • 4 BNE R1, R2, Loop ;delayed branch • 5 SD F4, 8(R1) ;altered when move past SUBI

9. Name Dependencies • Two instructions use same name (register or memory location) but don’t exchange data • Anti-dependence (WAR if a hazard for HW) • Instruction j writes a register or memory location that instruction i reads from and instruction i is executed first • Output dependence (WAW if a hazard for HW) • Instruction i and instruction j write the same register or memory location; ordering between instructions must be preserved • How to remove name dependencies? • They are not true dependencies

10. Register Renaming WAW WAR 1 Loop: LD F0, 0(R1) 2 ADDD F4, F0, F2 3 SD F4, 0(R1) 4 LD F0, -8(R1) 5 ADDD F4, F0, F2 6 SD F4, -8(R1) 7 LD F0, -16(R1) 8 ADDD F4, F0, F2 9 SD F4, -16(R1) 10 LD F0, -24(R1) 11 ADDD F4,F0,F2 12 SD F4, -24(R1) 13 SUBI R1, R1, #32 14 BNE R1, R2, LOOP 1 Loop: LD F0, 0(R1) 2 ADDD F4, F0, F2 3 SD F4, 0(R1) 4 LD F6,-8(R1) 5 ADDD F8, F6, F2 6 SD F8, -8(R1) 7 LD F10,-16(R1) 8 ADDD F12, F10, F2 9 SD F12, -16(R1) 10 LD F14, -24(R1) 11 ADDD F16, F14,F2 12 SD F16, -24(R1) 13 SUBI R1, R1, #32 14 BNE R1, R2, LOOP • Name Dependencies are Hard for Memory Accesses • Does 100(R4) = 20(R6)? • From different loop iterations, does 20(R6) = 20(R6)? • Our example required compiler to know that if R1 doesn’t change then:0(R1) ≠ -8(R1) ≠ -16(R1) ≠ -24(R1)There were no dependencies between some loads and stores so they could be moved around No data is passed in F0, but can’t reuse F0 in cycle 4.

11. Control Dependencies • Example • if p1 {S1;}; • if p2 {S2;}; • S1 is control dependent on p1; S2 is control dependent on p2 but not on p1 • Two constraints • An instruction that is control dependent on a branch cannot be moved before the branch so that its execution is no longer controlled by the branch. • An instruction that is not control dependent on a branch cannot be moved to after the branch so that its execution is controlled by the branch. • Control dependencies relaxed to get parallelism-Loop unrolling

12. Dynamic Scheduling • Dynamic Scheduling: Hardware rearranges the order of instruction execution to reduce stalls • Disadvantages • Hardware much more complex • Key idea • Instructions execution in parallel (use available all executing units) • Allow instructions behind stall to proceed • Example • DIVD F0,F2,F4 • ADDD F10,F0,F8 • SUBD F12,F8,F14 • Out-of-order execution => out-of-order completion

13. Overview • In-order pipeline • 5 interlocked stages: IF, ID, EX, MEM, WB • Structural hazard: maximum of 1 instruction per stage • Unless stage is replicated (FP & integer EX) or “idle” (WB for stores) • Out-of-order pipeline • How does one instruction pass another without “killing”it? • Remember: only one instruction per-stage per-cycle • Must “buffer” instructions IF ID EX MEM WB

14. Instruction Buffer • Trick: instruction buffer (many names for this buffer) • Accumulate decoded instructions in buffer • Buffer sends instructions down rest of pipe out-of-order instruction buffer IF ID1 ID2 EX MEM WB

15. Scoreboard State/Steps instruction buffer • Confusion in community about which is which stage IF IS RO EX WB ID Structure Data Bus Registers EX EX EX Control/Status Scoreboard

16. Dynamic Scheduling: Scoreboard • Out-of-order execution divides ID stage: • 1.Issue—decode instructions, check for structural hazards • 2. Read Operands—wait until no data hazards, then read operands • Scoreboards allow instruction to execute whenever 1 & 2 hold, not waiting for prior instructions. • A scoreboard is a “data structure” that provides the information necessary for all pieces of the processor to work together. • Centralized control scheme • No bypassing • No elimination of WAR/WAW hazards • We will use In order issue, out of order execution, out of order commit ( also called completion) • First used in CDC6600.

17. Stages of Scoreboard Control • Issue—decode instructions & check for structural hazards (ID1) • If a functional unit for the instruction is free and no other active instruction has the same destination register (WAW), the scoreboard issues the instruction to the functional unit and updates its internal data structure. • If a structural or WAW hazard exists, then the instruction issue stalls, and no further instructions will issue until these hazards are cleared.

18. Stages of Scoreboard Control • Read Operands—wait until no data hazards, then read operands from registers (ID2) • A source operand is available if no earlier issued active instruction is going to write it, or if the register containing the operand is being written by a currently active functional unit. • When the source operands are available, the scoreboard tells the functional unit to proceed to read the operands from the registers and begin execution. • The scoreboard resolves RAW hazards dynamically in this step, and instructions may be sent into execution out of order.

19. Stages of Scoreboard Control • Execution—operate on operands (EX) • The functional unit begins execution upon receiving operands. When the result is ready, it notifies the scoreboard that it has completed execution. • Write result—finish execution (WB) • Once the scoreboard is aware that the functional unit has completed execution, the scoreboard checks for WAR hazards. If none, it writes results. If WAR, then it stalls the instruction. • Example: • DIVD F0, F2, F4 • ADDD F10, F0, F8 • SUBD F8, F8, F14 • Scoreboard would stall SUBD until ADDD reads operands

20. Scoreboard Data Structures • Instruction status • Which of 4 steps the instruction is in • Functional unit status • Busy Whether the unit is busy or not • Op Operation to perform in the unit (e.g., + or –) • Fi Destination register • Fj, Fk Source-register numbers • Qj, Qk Functional units producing source registers Fj, Fk • Rj, Rk ready bits for Fj, Fk • Register result status • Indicates which functional unit (if any) will write each register. • Blank when no pending instructions will write that register

21. Scoreboard Example LD F6, 34(R2) LD F2, 45(R3) MULT F0, F2, F4 SUBD F8, F6, F2 DIVD F10, F0, F6 ADDD F6, F8, F2 What are the hazards in this code? Latencies (clock cycles) LD 1 MULT 10 DIVD 40 ADDD, SUBD 2

22. Scoreboard Example

23. Scoreboard Example: Cycle 1 Issue LD #1 Shows in which cycle the operation occurred.

24. Scoreboard Example: Cycle 2 LD #2 can’t issue since integer unit is busy. MULT can’t issue because we require in-order issue.

25. Scoreboard Example: Cycle 3

26. Scoreboard Example: Cycle 4

27. Scoreboard Example: Cycle 5 Issue LD #2 since integer unit is now free

28. Scoreboard Example: Cycle 6 Issue MULT

29. Scoreboard Example: Cycle 7 MULT can’t read its operands (F2) because LD #2 hasn’t finished

30. Scoreboard Example: Cycle 8a DIVD issues. MULT and SUBD both waiting for F2

31. Scoreboard Example: Cycle 8b LD #2 writes F2

32. Scoreboard Example: Cycle 9 Now MULT and SUBD can both read F2 How can both instructions do this at the same time??

33. Scoreboard Example: Cycle 11 ADDD can’t start because Add unit is busy

34. Scoreboard Example: Cycle 12 SUBD finishes. DIVD waiting for F0

35. Scoreboard Example: Cycle 13 ADDD issues

36. Scoreboard Example: Cycle 14

37. Scoreboard Example: Cycle 15

38. Scoreboard Example: Cycle 16

39. Scoreboard Example: Cycle 17 ADDD can’t write because of DIVD RAW!

40. Scoreboard Example: Cycle 18 Nothing Happens!!

41. Scoreboard Example: Cycle 19 MULT completes execution

42. Scoreboard Example: Cycle 20 MULT writes

43. Scoreboard Example: Cycle 21 DIVD loads operands

44. Scoreboard Example: Cycle 22 Now ADDD can write since WAR removed

45. Scoreboard Example: Cycle 61 DIVD completes execution

46. Scoreboard Example: Cycle 62 DONE!!

47. Scoreboard • Operands for an instruction are read only when both operands are available in the register file • Scoreboard does not take advantage of forwarding • Instructions write to register file as soon as they are complete execution (assuming no WAR hazards) and do not wait for write slot • Reduced pipeline latency & benefits of forwarding • One additional cycle of latency as write result and read operand stages cannot overlap • Bus structure • Limited number of buses to register file represent structural hazards

48. Scoreboard • Limitations • No forwarding (RAW dependence handled through registers) • In-order issue for WAW/structural hazards limit scheduling flexibility • WAR stalls limit dynamic loop unrolling (no register unrolling) • Performance • 1.7X for FORTRAN programs • 2.5X for hand-coded assembly • Hardware • Scoreboard is cheap • Busses are not

49. Scoreboard Implications • Out-of-order completion => WAR, WAW hazards? • Solutions for WAR • Queue both the operation and copies of its operands • Read registers only during Read Operands stage • Solution for WAW & Structural Hazards • Must detect hazard: stall until the hazards are cleared • Need to have multiple instructions in execution phase • Multiple execution units or pipelined execution units • Scoreboard keeps track of dependencies, state or operations • Scoreboard replaces ID, EX, WB with 4 stages