Energy-Efficient Solutions for 10Gbps Ethernet

1. Energy-Efficient Solutions for 10Gbps Ethernet YuryAudzevich, Alan Mujumdar, Philip Watts, Andrew W. Moore MSN 2012 workshop Friday, July 13th, 2012

2. Introduction Energy-efficiency of transmission systems is one of the key priorities with respect to the next generation of networking equipment. Open questions: What is the power contribution of the ‘lower layer’ transmission protocols? What is the power impact of the encoding blocks? Servers vs Network for a Google cluster (fat trees topology) (Energy proportional datacenter networks, Abtset al. ISCA 2010)

3. The focus of the research Which effect DC-balanced codes do have on the optical transmission system? …and in particular: what is • the effect on optical power requirement? • front-end power contribution, like PMA and PMD? • the power consumption of line coding block itself? ??? ???

4. 10Gb/s optical link simulations • Optical link – transmission system: • 219 bits PRBS is used as an input, the baud rate is adjustedafter encoding • Optical link parameters: • 100m Single Mode Fibre with parameters satisfying requirements for 10Gbps Ethernet over SMF • Optical link – receiving system: • Optical receiver with direct detector and AC coupling achieved using High Pass Filter • BER is calculated using the complementary error function

5. 10Gb/s optical link simulations (cont.) better better better better • The transmission system is relatively insensitive to the DC-balanced codec choice • Taking 100MHz HPF cut-off frequency and assuming 20dB link budget, the laser power requirement is lower for encoded sequences(0.3mW of savings) in comparison to PRBS

6. Physical Coding Sublayer – 8B10B (and 64B66B) 8B10B line code: Encoder/Decoder– implemented 3B4B and 5B6B codes plus disparity control checkfor DC-balance 64B66B line code: Encoder/Decoder – decoding from XGMII into 10GBASE-R format Scrambler/Descrambler – mixing of data to avoid long sequences of 0s/1s • Codecs were implemented in Verilog HDL and Synthesized using 90nm and 45nm technical process libraries • Industry standard estimation tools were used for power measurements

7. Early-days results – 8B10B PCS power 10Gb/s link results: obtained for 30microseconds simulation periods, with a symbol clock frequency of 625MHz for both 45nm and 90nm tech. process Inverse of energy-proportionality • IDLE sequencescosts MORE to encode • Low leakage 45nm library provides decrease in power by a factor of 2

8. Power estimates – 64B66B PCS 10Gb/s link results: Identical pattern sent for30microseconds ofsimulation periods, with a symbol clock frequency of 156.25 MHz 64B66B 10GBASE-SR / 10GBASE-LR (commonly used) 8B10B 10GBASE-LX4 (less common) • Power consumption of 64B66B codec is actually more data proportional • Scrambling & gearbox modules have a fixed power cost(~1.5-2.5 times larger power dissipation than the combination of encoder & decoder power)

9. Physical Medium Attachment – 8B10B and 64B66B PMA components are built using both CMOS and MCML logic families • CMOS designs were synthesized using standard cell libraries • MCML designs were built, optimized and analysed using HSPICE tools

10. PMA power – 8B10B and 64B66B 64B66B • MCML power is independent of the operating frequency but is strongly related to the optimization criteria • At high clock frequencies MCML designs become more power efficient than their CMOS counterparts • Even well power-optimized PMA designs may require5x-10x times higher power than the corresponding PCS blocks!

11. Implications • Our recent analysis of realistic trace data(10Gbps) showed average link utilization of only 8.79% - inconcordance with [1]. • The majority of the networks are overprovisioned to sustain peak loads and underutilized most of the time • Current implementations of Ethernet standards require continuous transmission of IDLE code words (even in the absence of MAC traffic) • So… • May be, we need a system that has good energy-proportionalityand can quickly restart • Sounds like we need a new MAC… [1] T. Benson, A. Akella, and D. A. Maltz. Network traffic characteristics of data centers in the wild. In Proceedings of ACMIMC '10, pp. 267-280, New York, USA, 2010.

12. Or an old energy-efficient MAC Remember this one? Ethernet – CSMA/CD Many features/ideas we don’t want, but one we do: • Preambles give clocks valuable re-sync. time and allow photonic systems to turn back on

13. Energy-efficient MAC • Where do we get energy-savings from: • Powering down thecodecs when no data is present • Using a synchronization preamble prior to data transmission for fast CDR Is the protocol going to make a difference? 89% 93% YES, It makes the difference 93% 87% With avg. Ethernet frame size of 1150bytesand 64bits of preamble, the effective energy-saving is ~93%

14. Key take-aways • Optimal laser power is independent of the DC-balanced codec chosen • Codec power consumption is not always data-proportional • Serialization/deserialization power dominates over all the other power groups • New MACs (off when idle) do save significant power • How do we test, build, trial? • regular NICs don’t help • Need something programmable but FAST… Thank you! QUESTIONS?