Deploying Safe User-Level Network Services with icTCP

1. Deploying Safe User-Level Network Services withicTCP Haryadi S. Gunawi Andrea C. Arpaci-Dusseau Remzi H. Arpaci-Dusseau The ADvanced Systems Laboratory (ADSL) Univ. of Wisconsin - Madison

2. Motivation • Vast number of proposed modifications to TCP/IP • Some adopted, others not deployed • TCP Vegas [SIGCOMM `94] • TCP Nice [OSDI `02] • Congestion Manager (CM) [SIGCOMM `99] • Heart of deployment problem: OS kernel • OS tend to be substantial and complex • Vendors dislike changing them when benefit is not imminent • A range of OS-es must implement the modifications • Transition of good research ideas into wide-spread use is slow • Emerging different network environments • Wireless (lossy network), Load-balancing (packet reordering) • Different extensions for different network environment • TCP only support some

3. f( ) f( ) in-kernel extensions Current TCP implementation TCP Applications TCP Reno

4. Information Control Inside to outside • Why in-kernel extensions? • Information and control only available within the kernel • Ex: TCP Vegas • Information: per-packet RTT and TCP States • Control: Congestion Window • Questions? • Can extensions be built in application layer? • What information and control need to be exposed? TCP Applications TCP f( )

5. Proposed Framework • icTCP (“information and control” TCP) • Address the problem of deployment • Slightly modified in-kernel TCP stack • Exports key pieces of information • Provides safe control to user-level libraries • Given information and control, extensions can be built in application layer • Evaluation • Converting TCP to icTCP requires a small amount of additional code • The resulting flow is TCP friendly • Minimal overhead • Implement 6 user-level extensions • Little complexity in implementing extensions at user-level

6. Deployable Flexible Composable Apps Apps App1 App2 lib-f( ) lib-f( ) lib-f lib-g lib-g( ) icTCP icTCP icTCP New services packaged as libraries and easily downloaded. Developers can tailor libraries to the their needs Services are used together for more powerful functionality Benefits • icTCP facilitates the development of user-level extensions

7. Outline • Motivation and Overview • icTCP Design and Implementation • Information • Control • 5 Axes of Evaluation • Conclusion

8. icTCP Design • Different TCP connections, different libraries • User-Libraries on top of icTCP • icTCP exposes Information and Control

9. Information • Which information should be exposed? • Too little  limited extensions • Too much  expanded interface • 2 types of Information: • Variables part of TCP specification (RFC 793) • cwnd, ssthresh • Message list and Ack list • More detailed information • History of recent packets and acknowledgements • Enabled by user-level services to save memory TCP Clients User Libraries

10. How and when to obtain information? • Polling • BSD socket interfaces • Unnecessary run-time overhead • Kernel-user space copy • When to obtain accurately? • TCP variables are updated: • upon receipt of an ACK • end of a round • Interrupt mechanism • icTCP notifies application when these events happen

11. Control • Allow internal TCP variables to be externally set in a safe manner • What variables and what values? • TCP-Friendly: don’t harm other competing standard flows • Philosophy: icTCP must be conservative • Control is allowed if not aggressive • Control 10 (virtual) variables that can be safely set TCP Clients User Libraries

12. Implementation of Safe-Control • Idea: Virtual variables • congestion window (cwnd)  virtual congestion window (vcwnd) • Manipulate policies through virtual variables • TCP Reno keeps running using the original variables • Safe Ranges • Enforce friendliness • 0 ≤ vcwnd ≤ cwnd • Without safe ranges, resulting flows not TCP-friendly • Choosing 10 variables, defining safe-ranges • Check safe-setting theoretically and empirically • Theoretically: Reno equation • congestion window: control how many packets in the network • vcwnd > cwnd : more packets in the network, thus more aggressive • Empirical Prove • Set virtual congestion window outside the safe ranges

13. Outline • Motivation and Overview • icTCP Design and Implementation • Framework Evaluation • Conclusion

14. 5 Axes of Evaluation How easy to convert TCP to icTCP? 1 How difficult to develop TCP extensions in this way? 5 2 Is icTCP friendly? 4 3 What types of extension can be built and deployed with icTCP? What are the overheads? Does it scale?

15. Converting TCP to icTCP 1 TCP to icTCP • icTCP in Linux 2.4.18 • Add 316 lines of code • Non-intrusive modifications

16. Unconstrained icTCP (allow variable setting to any value) Tuncons Reno Tuncons Runcons: TReno Reno TReno Reno Network Safety: Unconstrained icTCP Friendly? 2 TCP-Friendly: ~ 1 TCP-Unfriendly

17. Unconstrained icTCP Constrained icTCP (allow variable setting within the safe ranges) . Tcons Reno Tcons Rcons: TReno Reno TReno Reno Network Safety: Constrained icTCP Friendly? 2 TCP-Friendly TCP-Unfriendly • Safety  Safe Ranges are required!

18. 3 Scalability and Overhead Scale? Overhead? • What is the overhead? Does it scale? • Rate of getting info and setting variables • Different extensions, get/set at different rate • Two factors: • Per-ack or per-round interrupts • Need the message list and ack list • 3 synthetic libraries: • per-ack interrupt • per-round interrupt • per-round interrupt + gets message list

19. 3 Overhead Scale? Overhead? (96 conns) R1 R2 per-round 12% S per-round + msgList (8 conns) per-ack 30% 12 MB/s per-ack (4 conns) R3 R4 • Noticeable overhead, but not prohibitive

20. 4 TCP Extensions Range of Extensions? • Range of TCP extensions can be built on top of icTCP • Implement 3 sets case studies (6 extensions): • Congestion window: • TCP Vegas [Brakmo, SIGCOMM ’94] • TCP Nice [Venkataramani, OSDI ’02] • Congestion Manager (CM) [Balakhrisnan, SIGCOMM ’99] • Duplicate threshold • Reordering-Robust Ext [Zhang, ICNP ‘03] • Efficient Fast Retransmit [Tamura, LCN ‘98] • Retransmission Timeout • Eifel RTO [CCR ‘00]

21. f vcwnd per-round interrupt Vegas vcwnd TCP states msg list 4 User-level TCP Vegas Implementation Range of Extensions? • Using information and control is simple • Complexity is within the algo itself TCP Clients lib-Vegas icTCP TCP Vegas [Brakmo et.al., SIGCOMM ’94]

22. 4 Does it work? Range of Extensions? • TCP Reno: Send more packets until drop. • TCP Vegas: Maintain the “right” amount of extra data in the network • e.g. keep only 3 packets in the bottleneck queue Reno User-Level Lib. TCP Vegas In-kernel TCP Vegas • Same Behavior

23. lib-f TCP + f icTCP App1 App2 EFR RR dt < 3 dt > 3 Lossy Network Packet Reordering 4 icTCP Strengths Range of Extensions? • Implement less aggressive TCP variants at user-level • No need to push changes into the kernel • Ideally suited for tuning parameters whose optimal values depend upon the environment • Lossy Network  retransmit faster  Use Efficient Fast Retransmit • Packet reordering  postpone retransmission  Reordering Robust (RR) Extensions • Different environment, opposing solutions • TCP favors one solution • Correcting errors in parameter values • Eifel RTO [CCR ’00]: RTO prediction flaw when RTT drops • In newer Linux version, this prediction flaw is corrected with adding 2 new lines of code • Must wait vendors to correct this flaw

24. Apps lib-f Set existing variables with limits icTCP Apps lib-f Set existing vars to any values icTCP Enforcer 4 Other approaches Range of Extensions? • Can’t implement all extensions • Only allow 10 existing variables to be set • Conservative: Safety • STP (Self-Spreading Transport Protocols) [SOSP ’03] • Remotely upgrade other’s protocol stack • More TCP extensions can be deployed • Use separate module to enforce friendliness • Why we don’t use enforcer • Drawback: terminate non-conforming flows • icTCP modulates aggressive flows vs.

25. Ease of Development Difficult Develop? 5 • How difficult to develop TCP extensions in this way? • Complexity at user-level vs. in-kernel 1200(*) 438

26. Vegas Vegas RR RR Composability Difficult Develop? 5 • Composing multiple libraries • Vegas: good for small bottleneck queues • RR: good for reordering • Environment where both situations exist? • Vegas+RR: Better throughput TCP Clients icTCP

27. Conclusion • icTCP • Philosophy: Information and Control • Non-intrusive, simple, and TCP-friendly • Systems with icTCP reap benefits of user-level TCP extensions • Change TCP once now, no need to change it later

28. Questions? The ADvanced Systems Laboratory www.cs.wisc.edu/adsl

29. Related Work • Mogul et.al. [HotNets-2003] • Arbitrary state setting • Web100 and Net100 • Export a variety of per-connection statistics • Does not ensure network safety • STP [SOSP-2003] • Remotely upgrade other’s protocol stack • Invasive changes to the kernel • Additional machinery to ensure friendliness • InfoTCP/infokernel [SOSP-2003] • icTCP exposes more information • icTCP allows services to directly set cwnd inside of TCP • icTCP allows more variables to be controlled