Co* Projects : CoDNS, CoDeploy, CoMon

1. Co* Projects : CoDNS, CoDeploy, CoMon KyoungSoo Park, Vivek Pai and Larry Peterson Princeton University Co* Project 1st Asian PlanetLab Workshop

2. Background : CoDeeN • Academic Content Distribution Network • 100+ proxy servers on PlanetLab • Improve web performance & reliability • Running 24/7 since June 2003 • Roughly 3-4 million reqs/day aggregate • Highest-traffic project on PlanetLab Co* Project 1st Asian PlanetLab Workshop

3. Cache miss Response Request Cache Miss Cache hit Cache miss Response Cache hit Request Response How CoDeeN works CoDeeN Proxy Each CoDeeN proxy is a forward proxy, reverse proxy, & redirector Co* Project 1st Asian PlanetLab Workshop

4. CoDeeN as Project Factory • Deployment Issues • Reliability/Security • CoDNS • Reliable DNS service • CoDeploy • Large file distribution • Dynamic URL rewriting • CoMon • Monitoring infrastructure Co* Project 1st Asian PlanetLab Workshop

5. Local DNS Lookup Problems • CoDeeN experiences DNS problems • Local DNS lookup failures • 5+ seconds delay for cached records • Frequent & widely-distributed • Local DNS lookups important • LDNS cache hit rate: 80-90% • Unpredictable service • Directly affecting user-perceived latency • Random delay in web browsing • Critical in HTTP proxy, web crawlers and busy mail servers Co* Project 1st Asian PlanetLab Workshop

6. Experiment For Local Problems • Local name lookup every 6 seconds • “yyy.domain” on xxx.domain at Planetlab • “planetlab-2.cs.princeton.edu” for planetlab-1.cs.princeton.edu • Lookup should be handled locally • Failure criteria • 5+ seconds of latency • zero answer • Rolling average of the past 100 queries Co* Project 1st Asian PlanetLab Workshop

7. Expected DNS Behavior • planetlab3.flux.utah.edu • ricepl-1.cs.rice.pl Co* Project 1st Asian PlanetLab Workshop

8. planetlab1.cs.cornell.edu planetlab2.tamu.edu planetlab2.cs.uoregon.edu DNS Failure on Various Nodes Co* Project 1st Asian PlanetLab Workshop

9. Possible Causes • Packet loss • LDNS overloading • Cron jobs • Maintenance problems Co* Project 1st Asian PlanetLab Workshop

10. remote queries remote answer queries CoDNS CoDNS LDNS LDNS answer x CoDNS LDNS Solution? CoDNS CoDNS LDNS local node Co* Project 1st Asian PlanetLab Workshop

11. CoDNS • Cooperative name lookup scheme • If local server OK, use local server • When failures, ask a peer for the lookup • Selecting nearby peers • Liveness/Remote nameserver’s health • Send to improve cache locality • Remote request timeout • Dynamically adjusted to local server’s health • Exponentially backed off for each remote query Co* Project 1st Asian PlanetLab Workshop

12. Evaluation • Live traffic for one week for CoDeeN (20k - 30k) Co* Project 1st Asian PlanetLab Workshop

13. Finer-grained View • Live traffic for one day • Effectively flattens the spikes Co* Project 1st Asian PlanetLab Workshop

14. Availability • Add one ‘9’, from 99% to 99.9% Co* Project 1st Asian PlanetLab Workshop

15. For More Details • CoDNS: Improving DNS Performance and Reliability via Cooperative LookupsKyoungSoo Park, Vivek S. Pai, Larry Peterson and Zhe Wang [OSDI04] • Non-I2 nodes • CDN names lookup • CoDNS vs. TCP-based DNS Co* Project 1st Asian PlanetLab Workshop

16. CoDeploy • How to utilize the CDN infrastructure? • Large file(100’s MB) distribution • No custom protocol/software • Preserves HTTP semantics • Dynamic URL rewriting • Supports Akamai style CDN • CoralCDN [NSDI04] Co* Project 1st Asian PlanetLab Workshop

17. Large file distribution • BitTorrent • Bullet [SOSP03] • SplitStream [SOSP03] • FastReplica [USITS03] • Download managers(Windows update, Adobe download manger) Co* Project 1st Asian PlanetLab Workshop

18. Drawback • Availability • Only BitTorrent is deployed • But doesn’t really cache – peers are active downloaders • Complexity • Custom protocols • Origin site needs special server, format • Client software also custom Co* Project 1st Asian PlanetLab Workshop

19. file0-1 file1-2 file0-1 file file2-3 file4-5 file3-4 file4-5 Our Approach CDN CDN Agent CDN CDN Server CDN CDN Co* Project 1st Asian PlanetLab Workshop

20. egress ingress Low-Level HTTP stuff GET name/ranges Header: blah Header: blah HTTP/1.0 206 Partial Range: start-end/length Header: blah GET name Range: bytes ranges Header: blah HTTP/1.0 200 OK Content-length: piece length New-header: obj length Co* Project 1st Asian PlanetLab Workshop

21. file rewritten file file file Client CDN Server Client Server Dynamic URL Rewriting • http://www.host.com/path • http://proxy:port/capability/www.host.com/path Co* Project 1st Asian PlanetLab Workshop

22. Benefit • Transparently distributing the load • Avoiding slashdot effect • No setup at the browser • Cheap way of publishing popular contents • Easily customizable • Capability: access control Co* Project 1st Asian PlanetLab Workshop

23. CoMon • PlanetLab monitoring infrastructure • Node level statistics • Uptime, cpu/memory usage, DNS failure rate • Slice level statistics • # proc, network BW, cpu/memory usage • New slicestat • Provides slice level resource consumption statisics • e.g. curl http://localhost:3100/slicestat Co* Project 1st Asian PlanetLab Workshop

24. Co* Project 1st Asian PlanetLab Workshop

25. Co* Project 1st Asian PlanetLab Workshop

26. Co* Project 1st Asian PlanetLab Workshop

27. Co* Project 1st Asian PlanetLab Workshop

28. Direction • Which node is usable? • sshable, DNS lookup, cpu/memory, file descriptor availability • Which slice is going astray? • What are the typical experiments? • Automatic detection of overloading Co* Project 1st Asian PlanetLab Workshop

29. More info • Contact us! • princeton_codeen@slices.planet-lab.org • CoDeeN [USENIX04] • http://codeen.cs.princeton.edu/ • CoDNS [OSDI04] • http://codeen.cs.princeton.edu/codns/ • CoDeploy • http://codeen.cs.princeton.edu/codeploy/ • CoMon • http://codeen.cs.princeton.edu/comon/ Co* Project 1st Asian PlanetLab Workshop