Building media streaming apps and sites without plug-ins using MPEG-DASH

2. Building media streaming apps and sites without plug-ins using MPEG-DASH Daniel Schneider Senior Lead Program Manager, Microsoft Anthony Park Vice President, Engineering - Netflix 3-089

3. Key questions this talk aims to answer • What is MPEG-DASH and how does it help you to deliver a better video experience to your customers at a lower cost? • How does Windows enable video services to build MPEG-DASH clients? • Which tools are available to quickly get started developing an MPEG-DASH service on Windows?

4. Agenda • Introduction • The MPEG-DASH Standard • W3C Media Source Extensions • W3C Encrypted Media Extensions • Implementing an MPEG-DASH service in IE or a Windows Store app using JavaScript and HTML • Summary

5. Why standardize video APIs in browsers? Customers expect to be able to watch Netflix on any operating system or browser Common playback APIs reduce the burden of developers to have different implementations on different platforms Custom adaptive streaming algorithms can be written once in JavaScript and leveraged everywhere

6. Demo: Netflix in IE11 Previewon Windows 8.1 Preview

7. The Future of Netflix on Windows HTML5 Video in the Web Browser Supports multi-tasking scenarios, multi-tab, etc. Windows Store App Touch-first design, great for tablet and touch-screen hardware Provides deep integration with platform features (share contracts, search integration, etc.) Rich app experience (ex: semantic zoom) Dolby Digital 5.1 audio support

8. Moving from Plugins to HTML5 video Improved battery/CPU efficiency GPU-acceleration means less work for the CPU Full-quality 1080p now possible within the browser on low to mid-range PCs New browsers shying away from plug-in support Internet Explorer in the new Windows UI, as an example, doesn’t support plug-ins No more installing plug-ins! Removes a hurdle for new customers HTML5 video just works, no plug-ins necessary

9. Adaptive HTTP Streaming 480p • Manifest • 480, • 720 • 1080 720p 1080p 720p @ 01:03? 720p @ 01:06? 1080p @ 01:09?

10. Relevant standards and proposals • ISO/IEC 23009-1 “Dynamic Adaptive Streaming over HTTP (DASH)” • Specifies manifest and media segment formats • ISO/IEC 14496-12 “Part 12: ISO Base Media File Format, Edition 4” (aka MP4 container) • Specifies data structures for packaging media • ISO/IEC 23001-7 “Common Encryption in ISO Base Media File Format” • Specifies how to package protected content for consumption by multiple DRM systems • W3C Proposal “Media Source Extensions” • Specifies an HTML extension API enabling the implementation of adaptive streaming in the browser • W3C Proposal “Encrypted Media Extensions” • Specifies an HTML extension API enabling access to DRM implementations on the system from the browser Formats • Manifest • 480, • 720 • 1080 Web app or Windows Store app using JavaScript and HTML MSE and EME Client Trident / IE MF DRM

11. The MPEG-DASH standard • DASH stands for “Dynamic Adaptive Streaming over HTTP” • Specifies an adaptive media-streaming model • Control of the presentation lies exclusively with the client • Uses HTTP and standard web servers • Specifies formats for • Media Segments (media files) • Media Presentation Description (manifest)

12. DASH Profiles Full Profile • Set of restrictions on the offered Media Presentation (MPD & Segments) • Windows MSE implementation supports implementation of ISOBMFF based profiles: • On-Demand Profile • Segments are addressed as byte-range requests • Suitable for VOD only • Live Profile • Segments can be addressed by index number or time stamp • Can be used for Live and VOD delivery ISO Base media file format Main (MP4) MPEG-2 TS Main ISO BMFF On Demand ISO BMFF Live MPEG-2 TS Simple

13. Typical DASH ISOBMFF (MP4) file Index Segment (not for live) Media (or Data) Segment Media (or Data) Segment Initialization Segment

14. Typical DASH MP4 file PT: 0 PT: 3 PT: 6 Index Segment (not for live)

15. Typical DASH MP4 file Index Segment (not for live) Media (or Data) Segment Media (or Data) Segment Initialization Segment

16. DASH Manifest (MPD) high-level structure (live profile) • <MPD profiles="urn:mpeg:dash:profile:isoff-live:2011“> • <Period> • <ContentProtectionschemeIdUri="urn:mpeg:dash…" value="cenc"/> • <AdaptationSetmimeType=“video/mp4”> • <SegmentTemplate • media=“Level($Bandwidth$)/Fragment($Time$) • initialization="Level($Bandwidth$)/Fragments(init)> • <SegmentTimeline> • <S d=3 r=430 /> • </SegmentTimeline> • </SegmentTemplate> • <Representation bandwidth="6000000" width="1920" height="1080“/> • <Representation bandwidth="2962000" width="1280" height="720“/> • <Representation bandwidth="1427000" width="856" height="480“/> • </AdaptationSet> • <AdaptationSetmimeType=“audio/mp4”> • <SegmentTemplate … • <Representation bandwidth=“160000”/> • <Representation bandwidth=“64000”/> Media Presentation Description Live.XML 480p 720p • Manifest • 480, • 720 • 1080 1080p

17. DASH Manifest (MPD) high-level structure (on-demand profile) • <MPD profiles="urn:mpeg:dash:profile:isoff-on-demand:2011“/> • <Period> • <ContentProtectionschemeIdUri="urn:mpeg:da…" value="cenc"/> • <AdaptationSetmimeType=“video/mp4”> • <Representation bandwidth="6000000" width="1920" height="1080“> • <BaseURL>movie-high.mp4</BaseURL> • <SegmentBaseindexRange=“804-1123”> • <Initialization range=“0-803”> • </SegmentBase> • </Representation> • <Representation bandwidth="2962000" width="1280" height="720“> • <BaseURL>movie-medium.mp4</BaseURL> • <SegmentBase … • </Representation> • <Representation bandwidth="1427000" width=“856" height="480“> • <BaseURL>movie-low.mp4</BaseURL> • <SegmentBase … • </Representation> • </AdaptationSet> • … Media Presentation Description OD.XML 480p 720p • Manifest • 480, • 720 • 1080 1080p

18. W3C Media Source Extensions

19. W3C Media Source Extensions • W3C proposal extending the HTML standard • Defines an API to control media getting to the media element • Allows a web app to push init and media fragments into the video/audio tag • The main objects are MediaSource and SourceBuffer SourceBuffer audio MediaSource <video> sourceBuffers src SourceBuffer video

20. Setup Create MediaSource and set it as src to the video tag Add source buffers for audio and video Download and append initsegments to each of them

21. Playback Download and append data segment to each source buffer Keep downloading and appending as content plays Also keep removing to control your buffer

22. Switch streams Download and append init segment for new quality level Download and append data segments at the new quality level

23. Seek currentTime on video tag being set to new position Map currentTime to data segments and download these Append data segments for the new position

24. Demo: Simple streaming client using Media Source Extensions http://ie.microsoft.com/testdrive

25. W3C Encrypted Media Extensions

26. W3C Encrypted Media Extensions • W3C proposal extending the HTML standard • Defines an API that allows the use of a DRM system in a web app entirely in JavaScript • The main objects are MediaKeys and MediaKeySession • The entry points to the API are prefixed with MS MSMediaKeys MediaKeySession <video> msKeys

27. Protected Content In response to msneedKey event, create MSMediaKeys object and set it on the video tag Create a MediaKeySession using init data from msneedkeyevent In resonse to mskeymessage event acquire license from server and update mediaKeySession with response

28. Implementation of MSE and EME in Windows • MSE and EME can be used in IE and “Windows Store app using JavaScript and HTML” • Through MSE and EME, Windows can process/play DASH segments • Only ISO Base Media File Format (no MPEG2-TS) • Supported codecs: • In IE: H264, AAC • In HTML store apps: also Dolby Digital+ • PlayReady key system is pre-installed on all Windows versions with media capabilities and can be invoked via: • new MSMediaKeys(‘com.microsoft.playready’)

29. Implementing your MPEG-DASH streaming service in Internet Explorer or in a Windows Store app using JavaScript and HTML

30. Getting started: what you need • DASH sample app • (e.g. DASH.js) • A tool to inspect MP4 files • (e.g. MP4 Explorer) • Packaging tool/packaging pipeline • (e.g. MP4 box, Windows Azure Media Services) • Delivery Server • (e.g. IIS, Windows Azure Media Services)

31. DASH.JS • The DASH Industry Forum’s reference player. • A JavaScript implementation of a DASH client in the browser using MSE and EME • Player: http://dashif.org/reference/players/javascript/index.html • Source code: • https://github.com/Dash-Industry-Forum/dash.js

32. What are Media Services? Functional Components for Building Custom Media Workflows in the Cloud

33. Begin today! • Create a free Windows Azure trial account • Get $200 of Windows Azure free with your 1-month trial. • Scale your Media Services account to enable dynamic packaging and MPEG-DASH (On-Demand Streaming Reserved Units) • http://www.windowsazure.com/en-us/pricing/free-trial/ • Or use your MSDN account • Create Windows Azure Media Services subscription using your existing account • Use your credits to enable dynamic packaging and MPEG-DASH. • http://www.windowsazure.com/en-us/pricing/member-offers/msdn-benefits/

34. Things to note • When fetching segments / acquiring license via XHR • Observe Cross Origin Resource Sharing (CORS) • Host the player page on an http (not https) URL • For power efficiency • To avoid unnecessary disk caching, set xhr.responseType= ‘ms-stream’ or xhr.msCaching = ‘disabled’ • make use of the HTML Fullscreen API document.getElementById(“video”).msRequestFullscreen(); • Monitor download speed and playback quality to determine when to change bitrate

35. Monitoring playback quality • partial interface HTMLVideoElement { • readonly attribute VideoPlaybackQualityvideoPlaybackQuality; • }; • interface VideoPlaybackQuality { • readonly attribute DOMHighResTimeStampcreationTime; • readonly attribute unsigned long totalVideoFrames; • readonly attribute unsigned long droppedVideoFrames; • readonly attribute unsigned long corruptedVideoFrames; • readonly attribute double playbackJitter; • }; Attributes likely to change

36. Summary • MPEG-DASH provides standardization in the area of adaptive streaming • One format going to multiple clients • In IE and Windows Store apps using JavaScript and HTML, the MSE and EME APIs can be leveraged • Use the same code across app and site • Take full advantage of the hardware • Plugin-free in the browser • End-to-end tools are available today to kick-start development of a DASH service

37. Resources • DASH Specification (ISO 23009-1): • http://standards.iso.org/ittf/PubliclyAvailableStandards/index.html • Media Source Extensions Specification: • https://dvcs.w3.org/hg/html-media/raw-file/tip/media-source/media-source.html • Encrypted Media Extensions Specification: • https://dvcs.w3.org/hg/html-media/raw-file/tip/encrypted-media/encrypted-media.html • ISO Base Media File Format Specification (ISO 14496-12:2008/FDAM-3): • http://standards.iso.org/ittf/PubliclyAvailableStandards/index.html • Common Encryption Specification (ISO 23001-7): • http://www.3gpp.org/ftp/Inbox/LSs_from_external_bodies/ISO_IEC_JTC1_SG29_WG11/29n12313.zip • DASH.js: Reference Implementation of DASH in the browser by the DASH-IF. • https://github.com/Dash-Industry-Forum/dash.js/wiki • http://dashif.org/reference/players/javascript/0.2.3/index.html • Adaptive Streaming SDK + Player Framework • http://playerframework.codeplex.com/

38. Required Slide *delete this box when your slide is finalized Your MS Tag will be inserted here during the final scrub. Evaluate this session • Scan this QR codeto evaluate this session and be automatically entered in a drawing to win a prize!