Heracles: Improving Resource Efficiency at Scale

1. Heracles: Improving Resource Efficiency at Scale ISCA’15 Stanford University Google, Inc.

2. Outline • Introduction • Design • Isolation Mechanisms • Controllers • Evaluation • Conclusion

3. Motivation • Average server utilization in most datacenter is low, ranging between 10%~50%. • Difficult to consolidate the latency-critical services on a subset of highly utilized servers. • Increase the server utilization by launching best-effort tasks on the same server with a latency-critical job.

4. Motivation(Cont.) • Previous works tend to protect LC workloads, but reduce the opportunities for higher utilization through co-location.

5. Goal • Eliminate SLO violations at all levels of load for the LC job while maximizing the throughput for BE tasks.

6. Heracles • A real-time, feedback-based controller • Enables the safe co-location of best-effort(BE) tasks alongside a latency-critical(LC) service. • Ensures that LC jobs meet their target while maximizing the resources given to BE tasks.

7. Heracles(Cont.) • Four hardware and software isolation mechanisms. • Hardware: shared cache partitioning, fine-grained power/frequency setting. • Software: core isolation, network traffic control.

8. Isolation Mechanisms(Soft) • Core isolation • Pin workload to a set of core using cpusetcgroups. • Speed of (re)allocation: tens of milliseconds. • Network traffic • Limit the outgoing bandwidth of BE tasks using Linux traffic control. • No limit on LC job. • Take effect in less than hundreds of milliseconds.

9. Isolation Mechanisms(Hard) • LLC isolation • Cache Allocation Technology(CAT)in recent Intel chip. • Use way-partitioning to define non-overlapping partitions on LLC. • Take effect in a few milliseconds. • Implement software monitor to track the bandwidth usage of LC and BE jobs. • Scale down the # of cores for BE jobs if LC jobs does not receive sufficient bandwidth.

10. Isolation Mechanisms(Hard)(Cont.) • Power isolation • CPU frequency monitoring, Running Average Power Limit(RAPL), and per-core DVFS. • Take effect within a few milliseconds.

11. Design Approach • An optimization problem • Maximize utilization with the constraint that the SLO must be met. • Heracles • decomposes the high-dimensional optimization problem into many smaller and independent problem. • Decoupling interference sources. • Monitors latency, latency slack, and load. • Adjust the BE job allocation.

12. System Diagram

13. High-level Controller

14. Core & Memory Sub-controller

15. Max Load under SLO

16. Power and Network Sub-controller

17. Evaluation • Two sets of experiments • Co-locates LC applications with BE tasks on a single server. • Measuring end-to-end latency of Websearch on tens of servers. • BE tasks are also running. • Effective Machine Utilization(EMU) • LC throughput + BE throughput

18. Workloads • Three Google production LC workloads: • websearch • ml_cluster • Real-time text clustering using machine learning • memkeyval • In-memory key-value store • Run LC workloads with benchmarks that stress a single shared resource. • Stream-LLC, Stream-DRAM,cpu-pwr, iperf, brain, and streetview.

19. Latency of LC Applications

20. EMU

21. Shared Resource Utilization

22. Websearch in Cluster

23. Conclusion • Heracles • a heuristic feedback-based system that manages four isolation mechanisms to enable a latency-critical workload to be co-located with batch jobs without SLO violations. • Evaluation on real hardware demonstrates an average utilization of 90% across all evaluated scenarios without any SLO violations for the latency-critical job.