The Edge is Not the End – Why Cloud and Mobile Make CDNs Obs

1. THE EDGE IS NOT THE END – WHY CLOUD AND MOBILE MAKE CDNS OBSOLETE BY PADDY GANTI

2. Recently Shane Lowry, our VP of Engineering, wrote a blog post on how the next disruption in application deliveryis about eliminatinghuman middleware. I wantedto providesome more context and alsosharesome data nuggets to expandon the facts laid out in that article. It’s no surprise that mobile adoption and the advent of cloudcomputing are the twobiggest disruptions we have seen in the Internet service deliveryspace. In this post, weconsider the implications for both client and server sidegiventhese disruptions.Wewillalso show that content sizes are increasing, devicediversityisexploding and the new choke point for application deliveryisnow the Radio Resource Controller (RRC) and Radio Access Network (RAN). These challenges dictate a solution spacethat'sdifferentfrom the previousapproacheswe have seen and InstartLogicisspecificallyfocusedhere. First, let’sstart by talking about the twokey disruptions – mobile and its impact on the client/device front; and cloud-basedcomputing and whatthatmeans for the back end.

3. MOBILE Mobile Data Growth in Exabytes Per Globally, mobile trafficis about 30% of all Internet activitytoday and isincreasingrapidly, with an additional 6% of activitygeneratedfromtablets. The Cisco Visual Networking Index (VNI) provides the following quantitative estimates in mobile data growth, which shows thatweexpect to see an 18x increase in 5 years (2011-2016). 20 15 10 5 0 2012 2013 2014 2015 2016 ExaBytes per Month

4. This growthisfueled by demand for better applications and more content (mostlyvideo) from a variety of mobile devices. The reasonthatthisgrowthdiffersfromwhatwe’veseenhistoricallyisthatpreviously desktops primarilyconsisted of Wintel-basedplatformsusingwiredline access to the Internet – which made iteasyto optimizefor a homogeneousworkload. Today’splethora of smart phones and tabletsmakesit an entirelydifferentballgame. Whileit'stempting to bundle all mobile growthintoa single bucket, in reality the demand for content emanatesfroma widevariety of devices. The varietystartswithplatforms. Let’sconsider the followingtreemap of Androiddevicesthat are out there (Androidowns 72% of the market, whileiOSaccounts for 26%). Device Model GT-I9100

5. Fromourown logs, wesee the following distribution of deviceplatforms: To add to that, wealsoneedto consider the screensize diversity, which ranges from 320x480 pixels (smart phones) all the way up to 1920x1080 pixels (HD displays). The bottom line isthat mobile data isgrowingexponentially and isbeingconsumed by a greatervariety of screens and deviceplatforms.

6. CLOUD SERVICE ADOPTION Interest over time While the client sideisexploding, on the server sideweseea trend towards cloudadoption. The manifestation of cloudcomputing for web pages is the presenceof a lot of third party components such as widgetsdoing A/B testing, providingfeed-back via beacons, and trackinguser behaviorapartfromprovidinganalytics. This increases the number of components on a given web page while not exactlycontributingmuchto the overallpayload. Wesawthatroughly 48% of the requests in http archive are classified as third-party. 2005 2007 2009 2011 2013 2015 With the explosion of mobile devices and consolidation of cloud services, and the perennial expectation thatcompute and network justkeepgettingbetter and faster, the logical conclusion isthatthis must meanthat the mobile web isfaster. But the reality isquitedifferent.

7. THE MOBILE WEB IS IN FACT GETTINGSLOWER OVER TIME Top 1K URLs speedIndex Whenwesayfaster, wemeanvisually/perceptuallyfaster. So the question thenboils down to – whatmetricshouldwechoosethat best correlateswithvisual perception of a page load? OnLoad isn't a good metric, since a page loadeventcanbeartificiallytriggered by sites evenwhen no visual content ispresent, and neitherisStart Render, whichcanbetriggeredafteronLoad. So wefinallysettled on Speed Index, whichis a WebPagetestmeasurement of how quickly the screenpaints (perceivedload time). The fasteryoupaintthe wholescreen, the lower the score. A Speed Index of lessthan a second is the holygrail in web performance. 10000 8000 + 6000 4000 2000 7/1/2012 1/1/2013 7/1/2013 1/1/2014 7/1/2014

8. Top 10K URLs speedIndex Top 100K URLs speedIndex Wetracked Speed Index for the top 1,000, top 10,000, and top 100,000 sites as cohorts to check if any apparent trend isuniform, or if itdiffers over the variousgroupings. Fromwhatwecansee, it's a uniform trend that mobile websites over the last 2 years are gettingslower not faster, despite all the advancesthat have been made. 12000 12000 9000 9000 6000 6000 3000 3000 (Note: the collection of data changed a bit in the middle, when the throughput of the mobile devicemeasurementwasaltered to use an emulated 3G network in June 2013. Howeverthese changes do not affect our conclusion in anymeaningfulway.) So whyis the Mobile Web gettingslower? 0 0 7/1/2012 1/1/2013 7/1/2013 1/1/2014 7/1/2014 7/1/2012 1/1/2013 7/1/2013 1/1/2014 7/1/2014

9. CONTENT IS GETTING FATTER The first fairlyobviousreasonis the growthin richer and more content-intensive web sites. To substantiatethis claim, wetooka look at the Page weightmetric. Top 1K URLs PageWeight Top 10K URLs Pageweight Top 100K URLs PageWeight 600000 600000 600000 500000 500000 500000 400000 400000 400000 300000 300000 300000 200000 200000 200000 7/1/2012 1/1/2013 7/1/2013 1/1/2014 7/1/2014 7/1/2012 1/1/2013 7/1/2013 1/1/2014 7/1/2014 7/1/2012 1/1/2013 7/1/2013 1/1/2014 7/1/2014 Median_byteVolume Median_byteVolume Median_byteVolume

10. As youcansee, the uniform trend across all cohortsis a markedincreasein page bytes. Nextwewanted to see if wecould pin thisincrease to particulartypes of web traffic, soweseparated out the Page weight data by content types: Top 100K – Size by Content Type Evolution Over Time 22500 20000 17500 15000 12500 Median_HtmlBytes 1500000 1250000 1000000 750000 Median_JsBytes 18000 14000 12000 10000 Top 10K – Size by Content Type Evolution Over Time Top 1K – Size by Content Type Evolution Over Time Median_CssBytes 22500 20000 17500 15000 12500 22500 20000 17500 15000 12500 250000 200000 150000 Median_HtmlBytes Median_HtmlBytes Median_ImgBytes 1200000 1000000 800000 600000 1500000 1250000 1000000 750000 Median_JsBytes Median_JsBytes Again the uniform trend shows that content sizes are bloatingacross all content types, rangingfrom a few percent in HTML to a near-doublingof Image bytes. 18000 14000 12000 10000 18000 14000 12000 10000 Median_CssBytes Median_CssBytes 250000 200000 150000 250000 200000 150000 Median_ImgBytes Median_ImgBytes

11. 3500 Page load Time (ms) 3000 A quantitative studyperformedby Mike Belshe (one of the creatorsof the SPDY protocol) on the impact of varyingbandwidth vs. latency on page load times for some of the mostpopular destinations on the Web showed the following: NETWORK LATENCY OF THE ACCESS MEDIUM 2500 2000 1500 1000 Page Load Time as bandwidth increases Lookingatthis graph, one would question anyprovider toutingbandwidthincreaseas a panacea for web page performance. 1 Mbps 2 Mbps 3 Mbps 4 Mbps 5 Mbps 6 Mbps 7 Mbps 8 Mbps 9 Mbps 10 Mbps 3500 Page load Time (ms) 3000 2500 Page Load Time as latency decreases 2000 1500 1000 200 ms 180 ms 160 ms 140 ms 120 ms 100 ms 80 ms 60 ms 40 ms 20 ms

12. “As youcanseefrom the data above, if users double theirbandwidthwithoutreducingtheir RTT significantly, the effect on Web Browsingwillbe a minimal improvement. However, decreasing RTT, regardless of currentbandwidthalwayshelpsmake web browsingfaster. To speed up the Internet at large, weshould look for more ways to bring down RTT. What if wecouldreduce cross-atlanticRTTsfrom 150ms to 100ms? This would have a largereffect on the speed of the internet thanincreasinga user’sbandwidthfrom 3.9Mbps to 10Mbps or even 1Gbps.” – Mike Belshe So weask, whatis the trend in RTTsacrossthe world? Let'sconsult an active measurementdatabasemaintained by Les Cotrell to seewhatthe trend isthere. Average RTT over Time 5000 South Asia S.E. Asia Russia Oceania North America Middle East Latin America Europe East Asia Central Asia Balkans Africa Avgrtt in ms to rest of the world 3750 2500 1250 0 1/1/2011 1/1/2012 1/1/2013 1/1/2014 7/1/2011 7/1/2012 7/1/2013 7/1/2014 Date/time of measurement

13. DSL Cable Fiber As youcansee, in the last couple of yearsthere has been a smallimprovementin RTTs, but by and large nothingmeaningful. Sincethe majority of e-commerce and hosting providers happen to be in the US, let'slook for FCC reports on latenciesacross DSL, Cable and Fiber. 70 60 In 2014, fiber-to-the-home services provided 24 ms round-trip latency on average, whilecable-based services averaged 31 ms, and DSL-based services averaged 48 ms. Compare thisto 2013, wherefiber-to-the-home services provided 18 ms round-trip latency on average, whilecable-based services averaged 26 ms, and DSL-based services averaged 43 ms. 50 40 Average Latency ( Milliseconds ) 30 20 10 0 1 Mbps 3 Mbps 6 Mbps 10 Mbps 15 Mbps 20 Mbps 24 Mbps 30 Mbps 40 Mbps 75 Mbps Advertised Speed ( Mbit/s )

14. Overalllatencyis not gettinganybetter – if anything, it'sgettingworse. The average RTT to Google isprettymuch the same as itwas in 2010, despite all the innovations brought to us by thisawesomecompany. An alternate study by M-Lab stresses this point of degradation in latency due to interconnections between providers. So far all the above data is desktop alone, solet's focus on latency numbers from AT&T:   To put thoselatencies in context, alsoconsider the bandwidthavailable by technology:

15. Sincewe are talking about mobile data, let’ssee the overallpatha packet has to traverse to get service over the internet: Technology 2G / 3G / 4G / Wi-Fi Sectors # of DirectionsPer Cell Site Sectors # of DirectionsPer Cell Site Wireline InternetBackbone Radio Access Backhaul Core Network Carriers# of SpectrumsIn Use As youcansee, it’s the confluence of a lot of technologies thathelpsbring information to yourfingertips.

16. While the middle mile wasthe bottleneck in the desktop world, in the mobile world the Radio Access Network (RAN) is the new bottleneck for mobile browsing. More specifically, let'stakea look at the capacity of a typicalcelltower: Each Sector Has 2 Carriers Typicallythesetowers are provisioned and operateat75% utilization, whichmeanswe have only 16.2Mbps to use. The averagevoice call takes 12Kbps, whichmeans a maximum of 1350 calls are supportedbeforedegrading. Add the average fat webpage to this mix and you are lookingat a maximum of 8 webpages holding the toweratcapacity. This is the new bottleneck in the whole mobile user experience, and thereis not much a user or content publishercan do about this – exceptsend the most important bits of the application in the first few packets. Major Market Cell Tower = 21.6 Mbps Capacity Each CarrierHas3.6 Mbps of Capacity

17. CONCLUSIONS • So we’vetalked about a lot of differentelements in this article. To summarize, wesawthat web content isgettingricherwhiledevicediversityisexploding, and thatwecannot pin ourhopes on fasterlanes, giventhat network access times have been stagnant for over a decade (and willlikely continue to beso in the near future). All these forces combine together to create a new pressure point on RAN congestion, whichisalreadyatcapacity. • While I have mostlydwelt on the problems in this post, the solution space for mobile web applications isto • makethingssmaller (withoutlosingquality of experience) • move themcloser to the user (I mean in the browser, not some server in the cloudgiven the RTT) • cache them as long as wecan (existing solutions do not) • loading the application resourcesintelligently (loading the mostsignificantresources first) • Soundseasyenough, yetitrequires a verydifferentapproach to application delivery – one thatweatInstartLogic, withour Software-Defined Application Deliveryplatform, are focused on.

18. REFERENCES • HTTP Archive • Cisco VNI • Ilya Grigorik's blog • Android Fragmentation • Why Mobile Apps are Slow • More Bandwidth Doesn't Matter Much • M-Lab Interconnection Study • FCC Broadband America • Netflix ISP Speed Index • PingER Project • High Performance Browser Networking • Bessemer Cloud