WP7 . Integration, Validation & Optimization

1. WP7. Integration, Validation & Optimization Fernando Goicolea Telefónica I+D fer@tid.es Last Review Madrid, October 4th 2012

2. Testbedoptimisation, System Validation and Lessons learned (T7.3)

3. Contents • Architecture and Testbed • Validation & Optimization Tests & Results • Subjective Validation Tests & Results • Lessons learned

4. Architecture and Testbed of the COAST Integrated Prototype (I) • Functional Architecture Information Overlay Content/Service Distributed Overlay Service/Network Infrastructure layer

5. Architecture and Testbed of the COAST Integrated Prototype (II) • Component Architecture • [3]. This architecture contains the basic component needed to support the validation plan and the scenarios defined in Deliverable D7.2 [4]

6. Architecture and Testbed of the COAST Integrated Prototype (III) • Testbed • Telefonica and Yahoopremises:

7. Content preparation for the COAST platform Evaluation • Testing the benefits of adaptation requires longer streaming experiments than e.g. 10 seconds, which is the duration of test samples typically used for codec evaluation. Consequently, a continuous video sequence of about 6 minutes duration with a lot of different scene content has been MVC encoded. • The same content is provided in different streams at the public website http://www.coast-fp7.eu/dash/mvc

8. Content preparation for the COAST platform Evaluation (II) • Seven MVC bitstreams at almost constant data rate (CBR) in a single MP4 file; • The extracted AVC Base View of the seven MVC streams; • Two different DASH presentations including all seven MVC representations, which differ in the segment size (2 seconds and 5 seconds); • Two different DASH presentations including all seven AVC representations.

9. Content preparation for the COAST platform Evaluation (III) • Screen shots of sample scene content: • Scenes, e.g. flying debris, dancers, moving cars, rotating machinery, flames (marked by red boxes), contain complex movements which also require higher bit rates, because prediction from previous frames is less efficient in such cases • Scene containing fine structures, (e.g. the ground station antenna marked by a green box) quite demanding because by the slow movement of the turning antenna some fine structures are uncovered which cannot be predicted from previous frames and thus have to be intra-coded

10. Content preparation for the COAST platform Evaluation (IV) • Explicit testing and voting has taken place at the scenes shown:

11. Description of subjective Tests • Based on ITU-R BT-500. Tests use Double Stimulus Continuous Quality Scale (DSCQS). • Each test cell consists of a coded video and its uncoded (original) version. The order of coded and original video is random. One pair of coded and original video sequences is repeated once before the subjects are asked to rate the quality of the two video sequences. • Instead of the original and the encoded video, various video clips without the COAST platform (directly streaming from the video server, demonstrating a number of lost frames and frozen video) and with the COAST platform (streaming from the COAST Tier-0 and Tier-1 caching overlay, switching between 3D and 2D) were played back. • We have used watching periods of 15-20 seconds in order to include in each watching period a reasonable number of DASH segments and test the behaviour of the COAST platform under different network load condition. TEST CELL 15-20 Secs

12. Description of subjective Tests (II) • Each subject to set their personal range from “bad” to “perfect” • Vote for each coded sequence was then calculated as: QLoss = Qoriginal– Qreceived • As the coded video is allowed to be rated better than the original one (as it may appear in cases where the received video is visually equal to the stored one), in the same sense the received video may in some cases be rated better than the original one. So, QLosswas clipped and then was matched to a standard Opinion Scale (OS) scale by calculating: QOS = 5 – QLoss/20 • Data was then statistically processed to obtain the Mean Opinion Scale (MOS) by averaging the votes of all participants. In addition the Standard Deviation and the 95% Confidence Interval (CI) were computed. . Using COAST Framework NotUsing COAST Framework

13. Description of subjective tests & results (III) • Experiments 2 and 2A: Video consumption using adaptation from an external Server (Coast Server) • Objective evaluate the adaptation provided by Video Retrieval Agent (VRA), streaming a video encoded with DASH from an external server under real traffic conditions. • Experiments 3 to 7: Video consumption under variable bandwidth conditions and adaptation using the Cache Overlay • Objective  evaluate the improvement of the PQoE derived from the content adaptation mechanism introduced by the COAST Video Retrieval Agent when the available bandwidth is limited and constant during video consumption • EX-3: unlimited bandwidth • EX-4: 2Mbps, No Adaptation, EX-4A: 2 Mbps, Adaptation • EX-5: 2Mbps, No Adaptation, EX-4A: 1.5 Mbps, Adaptation • EX-6: 2Mbps, No Adaptation, EX-4A: 1 Mbps, Adaptation • EX-7: 2Mbps, No Adaptation, EX-4A: 0.8 Mbps, Adaptation

14. Description of subjective tests & results (IV) • Experiments 8 to 10: Video live streaming using the P2P Overlay • Objective  functionalities of ToroStream (TS), the Peer-to-Peer (P2P) video streaming protocol developed to distribute streams of live captured video and designed around the principles of Network Coding (NC). • EX-8: 100 peers Distributed in 3 virtual machines with a pattern 40, 40, 20 • EX-9: Down to 70 peers 30 peer nodes were killed randomly, leaving 70 peers distributed in 3 virtual machines with a pattern 30, 30, 10. • EX-10: Down to 30 peers 40 peer nodes were killed randomly, leaving 30 peers

15. Results of Experiments • COAST Streaming Experience using Tier-1 Caches • Due to the unpredicted bandwidth conditions, there were many freezing frames and the overall quality was not very good. • Largest deviations between the participants due to the varying behaviour of the system (network was congested or less congested) • When adaptation algorithm was applied during EX-2A the PQoE has improved remarkably. As the system was following the network activity, the adaptation of the video resolution and switching between 3D and 2D view, minimized the number of lost frames (thus the “frozen frames” effect) and the video streaming was smoother. This has increased the PQoE for approximately 20% and the MOS for approximately 0.5 points.

16. Results of Experiments (II) • COAST Streaming Experience using Tier-1 Caches • COAST Tier-1 streaming with unlimited bandwidth -more than the maximum needs- conditions (EX-3) was always approaching an excellent mark (an average of 4.6 to 4.8 out of 5 with very small deviation between the votes). • From the results it seems that going down even to 1Mbps, enhanced with the adaptation option we may achieve a PQoS that is comparable (or in some cases better) than the PQoE that we achieve with the remote COAST web server.

17. Results of Experiments (III) • Similar results have been recorded with the “Dancers” video clip. Yet, here the MOS (and the PQoS) had a lower value especially for the experiments (EX-6, EX-7) with limited available bandwidth. • Fast moving scenes is more important as the frozen-frames effect is much more annoying. In parallel experiments that we had also done, it is shown that the video context is also important. For example, freezing the video streaming during a football or basketball match is more annoying than during a car racing video clip.

18. Results of Experiments (IV) • The overall MOS is higher. Moreover, limiting the bandwidth (thus increasing the frozen-frames) was less annoying. Similarly, the difference between the adapted and not-adapted video streaming was smaller. CONSIDERATIONS ON SEGMENT LENGTH • The 2 seconds segments were always performing better than the 5 seconds. This is because adaptation can cope more easily with congestion if segments are shorter(Once the application requests a certain segment, it has to wait for the completion of the transfer, even if the available rate decreases). • On the other hand, if the segment duration is further reduced, this will cause more HTTP requests; accordingly, control traffic increases and the additional bandwidth consumed for application signalling might be significant.

19. Results of Experiments (IV) • COAST Streaming Experience using Tier-0 Caches • Verify the functionalities of ToroStream, the Peer-to-Peer (P2P) video streaming protocol developed to distribute streams of live captured video and designed around the principles of Network Coding (NC). • Compare the PQoE between the infrastructure based on the COAST Tier-1 caching and the Tier-0 caching system. Measured Parameters: • Continuity Index (CI) of the ToroStream protocol is measured at the peer nodes. The CI represents the number of frames that could be successfully decoded prior to the relative playout deadline. • Buffering time

20. Results of Experiments (V) • COAST Streaming Experience using Tier-0 Caches Comparison with NextShare: A few peers whose input bandwidth is below the threshold required to recover the encoded video stream, totally fail to display the video. This number of nodes is at least 50 in case of the reference NextShare protocol, while in the proposed system it goes down to 25 nodes.

21. Results of Experiments (VI) • COAST Streaming Experience using Tier-0 Caches • EX-8, we have used 100 Tier-0/Peer nodes. result is voted as almost “Excellent”, directly comparable with EX-3, which is the experiment streaming the video clip from the Tier-1 caching system. Yet, it should be noticed that the peer nodes have been instantiated in the controlled environment of the Telefonica Data Centre. Thus the main difference between EX-3 and EX-8 is only the protocol implementation and not the network conditions. • EX-9, we randomly killed 30 nodes during the streaming session. Yet, the 70 remaining nodes are more than enough and they easily survive the network topology restructuring. Thus the result was among the “Perceptible but not annoying” and “Slightly Annoying”. • EX-10, we randomly killed 70 nodes during the streaming session. The 30 remaining nodes are enough to achieve merely a result of just below “Slightly Annoying”, while with the NextShare approach we do not even achieve continuity of the video streaming.

22. Results of Experiments (VII) • COAST Streaming Experience using Tier-0 Caches • In case of “Dancers”, the frozen frames were more annoying, while in case of the Living room they were less frustrating. • It should be noticed that in all cases the Tier-1 has performed better than Tier-0; yet this come with the extra cost of the network infrastructure deployment.

23. Search Engine Subjective Evaluation • Implemented as a geographically distributed prototype, hosted on Yahoo!’s infrastructure, and deployed on data centers in 3 continents. • Differentiating advantages of the COAST search engine, in comparison to a classical search engine, are related to the : • indexing of documents available on the COAST infrastructure, • indexing of documents that are not discovered through active crawling • the query response time.

24. Search Engine Subjective Evaluation (I) • Access to COAST-specific documents publishing frontend tool. The search engine analyses the content of these entries in order to index the Web page that contains a link to the published document Query result Query Result Presented in a separate tab (COAST Specific) Title Description Keywords

25. Search Engine Subjective Evaluation (IV) • Recent and popular document indexing: • DPI content (URLs with associated popularity), is first gathered through a hierarchical system of aggregators, and presented to the search engine through a standardized web service interface. • Search engine applies some filters to the URLs and indexes or updates the index with those URLs • It has been verified that hidden web, not discovered by active crawling, constitutes a significant part of available documents on the web. Yahoo! Toolbar (a data source that is similar to DPI to capture the URLs visited by users), up to 80% are not unknown to the Yahoo! search engine while half of them indeed contain useful information that may fulfil the information need of users. We observe that on average, 2.6 out of 10 documents in the top-10 results of one million random queries from the Yahoo! query logs are from the toolbar. • More importantly, according to our studies, the queries that benefit from passive crawling are those looking for tail content. • Content freshness can be controlled, here through a simple textbox located to the right of the search bar

26. Search Engine Subjective Evaluation (III) Freshness can be controlled

27. Search Engine Subjective Evaluation (IV) • Optimal Response Time: • Technique that optimizes the latency of results in a geographically distributed search engine by carefully choosing a master index for each document, selectively replicating documents and posting lists. • We showed that, with a budget of 1 million replicated documents, accounting for 14% of the local index of a search site, 60% of the queries could be resolved locally. • We also measured the latencies on our prototype and showed that the latency for queries resolved locally was 316 milliseconds..

28. COAST Network Awareness • Data used for the actual evaluation gathered by NEC from a real world testbed deployed ALTO in a large European operator as part of a COAST-OCEAN projects liaison initiative (50 clients distributed in 7 cities). • Aim of the experiment to ensure that the load on a particular CDN does not exceed a certain threshold. As long as the load on a CDN is kept, below such a threshold, it is expected that the users being served by that CDN receive a minimum Quality of Experience. • The scenario is such that a nationwide operator receives an end user request for content, and decides that this request is best served by one of several COAST caching nodes present in its network. • The ALTO server contains a static cost map with operational costs from a client location to the cache servers  Static Costs • The ALTO servers of each cache node also frequently convey their overall load on the ALTO server Dynamic Costs. • These overall costs are provided as ALTO maps to the clients (see equation).

29. COAST Network Awareness (II) • 50 end-hosts running on 50 PlanetLab nodes distributed over 7 cities. The ALTO server and the 2 CDN servers were deployed in 2 cities. • The 50 end-hosts would fetch data from one of the CDNs based on the overall costmap received from the ALTO server. The ALTO client installed on the end-hosts would periodically fetch the costmap and would therefore instruct which CDN server to contact. • To have realistic traffic source models, we model our traffic source based on the model of the number of YouTube video requests for a 24 hour period and we scale down the time period to 24 minutes

30. COAST Network Awareness (III) • Results: • Various combinations of static and dynamic load, and present the mean result (of 10 runs) for the extreme combinations scenarios (100%-static or 100%-dynamic) and an optimal combination (static = 70%, dynamic = 30%). • We can observe that one of the Cache servers is heavily loaded, wherein the other cache server is lightly loaded. • This mismatch is quite a realistic scenario and can occur due to a variety of reasons such as nodes in a certain region are watching a live streaming of a football match thereby inducing more load on that server.

31. COAST Network Awareness (III) • Results: • the other extreme scenario where only the dynamic cost is taken into account. • We can observe that the load is almost equally shared among the two cache servers. The downside is that the operational costs are higher than the case of the pure static scenario. This could imply that some network parameter such as Round-trip delay time, congestion is high.

32. COAST Network Awareness (III) • Results: • the optimal case for this test. In the optimal case (30%-Dynamic). • that the load-balancing takes place only between the 10th and 18th minute when the load is above a certain threshold on a single CDN. The operational costs increase only during this period and are similar to the operational costs for the 100%-static scenario during other periods. Note that the optimal solution performs load balancing only during heavy load scenarios..

33. Lessons Learnt

34. DASH protocol & Content adaptation • Variable quality/resolution but smoother streaming of a video is preferable as compared to a video with fixed quality/resolution that suffers from frozen frames. • PQoE between video clips with segments of different duration had a small deviation in all experiments. Segments with 2 seconds duration were always performing better than segments with 5 seconds duration. • Tens of applications in the market for HTTP streaming, each of which providing a different approach to adaptation. An in depth comparison of COAST solution with other commercial and experimental platforms for HTTP streaming would have been beneficial, in order to identify benefits and drawbacks of each implementation. • DASH support is coming out in the market only in the latest months • None of current product support 3D streaming • Open source implementations available do not include dynamic adaptation algorithms. • Difficult a comparison because performances of the adaptation model are dependent on the way in which the content is prepared (i.e. segment size and format) • It is particularly important to study and implement at client side efficient algorithms which can tune the bitrate because Video quality for OTT streaming in fact is not at the same level of what is offered today by satellite Digital TV or IPTV operators and it is a differientiating factor.

35. DASH protocol & Content adaptation • The adaptation algorithms must take into account: • type of the device used • the local network conditions • the CPU load in end devices • Main features of COAST content adaptation: • Open solution: latest standard specification for HTTP streaming support from MPEG (MPEG DASH) and W3C (HTML5) have been adopted • Efficiency: bitrate is automatically tuned according to local network conditions and terminal capabilities/context through two different algorithms seamlessly integrated • Flexibility: overall adaptation model is implemented partially in media framework (DALIA model for network status) and partially in the web browser through a JavaScript engine (control of device type, full screen mode, CPU load). • We demonstrated how the different actors in the video streaming industry (content providers and broadcasters, network operators, etc.) can leverage on COAST solution and either totally rely on embedded adaptation logic, either control it and tune it via a JavaScript interface, either override its decisions and implement a proprietary solution in the web browser.

36. DASH protocol & Content adaptation • We release public part of the code developed for the HTTP streaming client. This action was intended to promote the COAST technology as well as the adoption of the MPEG DASH standard. • it would have been much more beneficial to start discussion with community at SW design stage. In our case for example, the GStreamer community provided feedbacks on COAST solution after we released the SW, which would have been useful to design the COAST DASH client in a better way, more suitable to be integrated as a plugin of the official GStreamerSDK.

37. ExploitationPlans

38. Exploitation – STMCOAST relevance - DASH FOSS OUT DLNA DASH PG Demos/PR COAST DASH client PatentAdaptationLogic BCD StageFright STMediaFramework ST-Orly(STB platform) ST-Newman(TV platform) STE-Snowball(Mobile platform) CUSTOMERS

39. Exploitation – STMCOAST relevance – CEP & P2P DHT & P2P forcontentdiscovery / accelerated download Benchmarked on PlanetLab COAST CEP Headed Gateway WiFi Router Voice Multimedia Domotic CEP ST-xxx (HMG) PROTOTYPE

40. Yahoo! Exploitation Plans • Search is an important part of Yahoo’s business • Top content category in Europe • Goals • Exploit search improvements of COAST • Tighter relationship with ISPs • Improvements foreseen in WP3 • Algorithms: crawling, indexing, query processing • Infrastructure: • Scalability • Efficiency of communication, storage, and processing

41. Exploitation - NEC • NEC will exploit the DPI (WP4) technologies • Extending the use cases of its high performance packet inspection solutions (e.g., fast discovery of new content, prediction of their popularity observing the early content lifetime) • Opening new communication channels to additional global market players (e.g., search engines/WP3) • NEC will exploit the network-aware tools for content delivery/WP6 technologies • By offering more competitive solutions for cost- and network- control in highly decentralized CDNs (see next slide) • Differentiating its solutions with network-aware content surrogate selection as well as chunk-level content placement

42. Market trends NEC is targeting • Players involved in the Carrier CDN ecosystem • Global CDN Service Providers • e.g. Akamai, Limelight • Still leading the market at the expense of network operators • CDN Solution Vendors • e.g. Velocix, Jet-Stream, CoBlitz • Getting acquired by network equip. vendors • (Mobile) Network Equipment Vendors • e.g. ALU, Cisco, Juniper, NSN • Enriching products with CDN solutions • (Mobile) Network Operators • e.g. BT, FT, Telefonica • Looking for CDN solutions to enable them to sell CDN services • Cases and indicators: • Amazon launchesCloudFront CDN Service • AkamaiprovidesCloud Optimizationto Amazon EC2 • Cisco CDS deployed by BT in wholesaleContentConnect CDN • ALU acquiredVelocix, NSN partneredwithVerivue, • JuniperacquiredAkeena, VerivueacquiredCoBlitz • Ericsson Global partneredwithJet-Stream • L3 took over Netflix deal fromAkamai. NetflixtrafficcongestingComCast networks leading to Netflix/ComCastlitigation • Global CDN Market • ~30% CAGR according to iDATE/F&S • Target telcos revenues by 2013: ~50% of global CDN market • Recent Trends • ISPs unsatisfied with traffic revenue going down;global CDN providers get a free ride and help onlymarginally in optimizing traffic; ISPs getno revenue from content Carriers building their own highly distributed “Cloud” CDN: • in-network caching in access & metro network • traffic localized and controlled • CDN also offered to 3rd parties as a Service

43. Telefonica Exploitation Plans • Telefonica, as a large fixed/mobile network operator, is very interested in improving the QoS of content delivery and in the optimization of network resources • COAST results are applicable to at least three areas of Telefonica’s business: • The recently deployed Hybrid CDN, based on TID’s technology, can be improved with COAST content-aware functionalities • The content adaptation technologies that are being tested in COAST can be applied to the Imagenio VOD service, which in the future will also be available on computers and mobile devices • COAST network and context awareness features can improve the PQoS for Mobile Broadband users and reduce congestion problems

44. Synelixis Exploitation Plans • Strong interest in developing and enhancing value-added services and content centric networking IP using open platforms • Extensive testing of these services and validating in real testbeds. • Extend know-how & experience in innovative technologies for video caching and streaming • Promoting the company’s profile via collaboration with EU networking and A/V industrial giants (e.g. STM, NEC, Yahoo!, TID). • Promoting networking & caching A/V solutions in commercial projects & partnerships, where Synelixis products will form offerings towards the variety of the system vendors included therein

45. Exploitation - HHI • Fraunhofer HHI offers its R&D results in the form of IPs for • algorithms, • software and • hardware solutions. • HHI will exploit its authoring and transmission technologies based on MPEG-DASH, including 3D, MVC and FTV, for • future video media distribution systems and • hybrid media delivery. • Thus, the MPEG-DASH authoring tool – amongst other solutions developed in the COAST project – is a basis for exploitation in partnership with companies and SMEs.

46. Barcelona Media Exploitation Plans • Barcelona Media • Innovation center supporting R&D projects that link academia and industry • Focused on technologies related to communication media • Interests in COAST • Web search • Development of algorithms • Infrastructure improvements • Exploitation in COAST • Scientific publications • Intellectual property transferred to Yahoo!

47. Polito Exploitation Plans • POLITO is a research university interested in the balanced development of both theoretical and applied research • POLITO will exploit research results in the field of: • VLSI prototyping for DPI • advanced video streaming solutions using content based overlays and AVC/MVC coding standard • The COAST activities will promote: • Research collaborations with other academic institutions • Exploitation of research results by industrial partners • Creation of know how to be spent in the broad area of the future internet initiatives and projects

48. TUB Exploitation Plans TUB is generally open to an international perspective in the fields of both research and teaching: • Education andteachingat grade and post grade levels • Curricula, seminars/workshops, student projects, PhD/Master topics • Publications in internat. conferenceproceedings, journals • Knowledge acquired within COAST will be the foundation of further research (strategic exploitation) : • Context-aware networking solutions • Network awareness and adaptation in particular, i.e. the assessment of the technical context of communication, mobility • New collaborative projects with industry and academia

49. UCLA Exploitation Plans • UCLA is a government owned NOT for profit University. In general there is limited interest in direct commercial uses. • UCLA is Interested in Exploiting COAST results as follows: • Technology advancement that lead to publications in major International Journals and Conferences • Leverage of Coast technology in future projects and proposals funded by US agencies as well as private companies. • Foster a close collaboration with the lively European Research community. • Use coast as a mean to enable a constructive researchers and student exchange between UCLA and the Consortium Partners.

50. SNU Exploitation Plans • SNU is a research university interested in the balanced development of both theoretical and applied research • SNU will exploit research results in the field of: • Content-centric routing • Cache management • Interworking between IP networking and content networking • The COAST activities will promote: • SNU’s collaboration with other EU institutions • Exploitation of research results by industrial partners