College of Computer Science, Inner Mongolia University Hohhot, Inner Mongolia, P. R. China

1. Adaptive Subset Based Replacement Policy for High Performance CachingLiqiang He Yan Sun Chaozhong Zhang College of Computer Science, Inner Mongolia University Hohhot, Inner Mongolia, P. R. China JWAC-1: Cache Replacement Championship 2010-06-20 ISCA-2010

2. Inner Mongolia University Background • Cache Replacement Policy plays an important role in a cache design. • LRU policy is widely used in nowadays microprocessor • The LLC has poor locality due to the L1 already filters temporal locality • LRU causes thrashing when working set > cache size College of Computer Science JWAC-1: Cache Replacement Championship

3. Inner Mongolia University Possible solution • if working set > cache size, retain some working set [Qureshi, et al, ISCA’07] • record part of a longer cache access history How we do it? Grouping a cache set and keeping part of access history in each group. Inspired by the thread migration paper of Pierre at HPCA’04 L2 L2 L2 L2 L2 L2 C0 C1 Cn g0 g1 gn College of Computer Science JWAC-1: Cache Replacement Championship

4. Inner Mongolia University Overview Proposal: Subset Based Replacement Policy (SRP) SRP successfully reduces the misses through retaining part of longer history in the groups. But the static SRP does not suitable for different programs. To adapt the diversity of programs and the behavior changing inside a program, we propose Adaptive SRP policy (ASRP). ASRP obtains a 4.5 % of geometric average miss reduction over LRU. College of Computer Science JWAC-1: Cache Replacement Championship

5. Inner Mongolia University Outline • Introduction • Static Subset Based Replacement Policy • Adaptive Subset Based Replacement Policy • Summary College of Computer Science JWAC-1: Cache Replacement Championship

6. subset subset subset subset Inner Mongolia University Static Subset Based Replacement Policy Cache set Non-Active Active: Accept insertion Local LRU Stack College of Computer Science JWAC-1: Cache Replacement Championship

7. MRU LRU blocks in active subset a b c d Reference to ‘i’ a b c i Inner Mongolia University Insertion scheme in SRP Insertion only occurs in active subset Choose victim at LRU position. Do NOT promote to MRU College of Computer Science JWAC-1: Cache Replacement Championship

8. MRU LRU a b c d Reference to ‘c’ c a b d Inner Mongolia University Operation on cache hit in SRP hit in any (active or non-active) subset Move to local MRU position College of Computer Science JWAC-1: Cache Replacement Championship

9. Inner Mongolia University Changing of active subset When the misses in a set > a threshold X, change active subset Thus: A. force X consecutive misses only replacing the blocks in active subset B. assume N subsets, then a subset can change to active again ONLY after (N-1)*X misses C. a greater value of X, a longer time that blocks in non-active subsets can stay in a set College of Computer Science JWAC-1: Cache Replacement Championship

10. b4 b3 b2 b6 Inner Mongolia University Thrashing access pattern in SRP assume working set is 24 blocks, LLC is 16-way, 4 subsets, 4 blocks/subset b1 b2 b3 b4 b5 b6 b7 b8 b9 b10 b11 b12 b13 b14 b15 b16 b17 ….. b24 x = 6 Blocks in a set with SRP: b2b3b4b6b8b9b10b12 b14b15b16b18 b20b21b22b24 MRU b4 b10 b3 b9 Blocks in a set with LRU: b9 ….. b24 b2 b8 LRU b5 b6 b1 b11 b12 b7 When access b2b3b4b6b8again, SRP hits but LRU misses Subset 0 Subset 1 College of Computer Science JWAC-1: Cache Replacement Championship

11. Inner Mongolia University Case Study of thrashing workload Different static thresholds have different abilities to reduce misses College of Computer Science JWAC-1: Cache Replacement Championship

12. Inner Mongolia University Hardware implementation MRU LRU College of Computer Science JWAC-1: Cache Replacement Championship

13. Inner Mongolia University Results • SRP reduces misses for thrashing workloads • but increases for LRU-friendly ones. • Not exist a threshold that is suitable for all benchmarks College of Computer Science JWAC-1: Cache Replacement Championship

14. Inner Mongolia University Outline • Introduction • Static Subset Based Replacement Policy • Adaptive Subset Based Replacement Policy • Summary College of Computer Science JWAC-1: Cache Replacement Championship

15. Inner Mongolia University Adaptive SRP policy Different programs prefer different thresholds. In ASRP policy: • Victim selection and insertion policy are same as in SRP • ONLY difference: threshold is selected dynamically from a pool of values according to which one causes fewest misses. • The maximum threshold is 128 • Pick eight values: 20, 21, …, 27 • Apply the best threshold value to the cache College of Computer Science JWAC-1: Cache Replacement Championship

16. Thres-20-sets Thres-21-sets + + Thres-27-sets Follower Sets Inner Mongolia University ASRP policy via “Set Dueling” Divide the cache into two type: • Sampling sets (eight thresholds * 4sets/thres.) • Follower sets Eight counters misses to threshold X’s sampling sets:counter_x++ Counters decides threshold for Follower sets: counter with smallest value miss Cntr_0 Cntr_7 Eight thresholds College of Computer Science JWAC-1: Cache Replacement Championship

17. Inner Mongolia University Resetting mechanism To avoid the accumulative effect of a big value in a specific Cnrt_x Record the times of a same threshold is selected by the follower sets When the times > a threshold, reset all the Cntr_Xs Eight thresholds last_follow = Y ++ N -- Cntr_0 global_follow reset >? threshold Cntr_7 College of Computer Science JWAC-1: Cache Replacement Championship

18. Inner Mongolia University Budget • Totally 45K bits • only 70% of the budget used by LRU policy, and 35% of the total budget provided by this championship College of Computer Science JWAC-1: Cache Replacement Championship

19. Inner Mongolia University Results For 1MB 16-ways LLC. ASRP gets a geometric average speedup of 4.5% over LRU College of Computer Science JWAC-1: Cache Replacement Championship

20. Inner Mongolia University Analyze xalancbmk GemsFDTD The sampling mechanism does help ASRP to find the best thresholds for different programs College of Computer Science JWAC-1: Cache Replacement Championship

21. Inner Mongolia University Conclusion • Keeping part of working set in the cache helps reducing misses when the cache suffers a thrashing problem • The part of longer access history helps SRP more accurately capturing the frequently used blocks • Different programs and different phases of a program prefer different thresholds to contribute maximum hits to the cache • “Set Dueling” helps ASRP dynamically selecting a suitable threshold • The experiment results show the effectiveness of ASRP policy College of Computer Science JWAC-1: Cache Replacement Championship

22. Inner Mongolia University Thank you! Any question? College of Computer Science JWAC-1: Cache Replacement Championship

23. Inner Mongolia University Result on multi-core processor College of Computer Science JWAC-1: Cache Replacement Championship

24. Inner Mongolia University Case Study of LRU-friendly workload College of Computer Science JWAC-1: Cache Replacement Championship

25. Inner Mongolia University Explanation of active subset changing College of Computer Science JWAC-1: Cache Replacement Championship

26. Inner Mongolia University A simple example of SRP policy College of Computer Science JWAC-1: Cache Replacement Championship