Video Coding for Streaming Media Delivery on the Internet Conklin, G. et al, 2000

1. Video Coding for Streaming Media Delivery on the InternetConklin, G. et al, 2000 Jake Wires

2. Motivation • Original standards designed for constant bit rate encoding • Constant bit rate encoding does not account for the channel variability inherent in many network applications • Large demand for streaming multimedia—over 350,000 hours of live content broadcast per week

3. History of Streaming Media Technologies • VivoActive 1.0 • HTTP server-client transactions via TCP • H.263 video codec, G.723 audio codec • Multiplexing protocol to combine audio & video • Very simple

4. History: VivoActive • Problems • HTTP’s lack of flow control and TCP’s guaranteed delivery mechanisms resulted in large fluctuations of fragment delivery times • Necessitated large preroll buffer (5 – 20 sec) • Rebuffering (filling empty buffer) diminished user expereince • Session Control (i.e. f.f., rew.) difficult

5. History: RealAudio 1.0 • Introduced in March 1995 • Relied on Progressive Networks Architecture—a suite of dedicated network protocols • Used 2 connections: • TCP-based session control connection • UDP-based audio transmission connection

6. History: RealAudio 1.0 • Use of UDP-based transmission introduced possibility for lost, delayed, and out of order packets • Automatic Repeat-Request mechanism allowed clients to request retransmission of lost packets • Frame interleaving technique minimized perceptual damage • Send packets of interleaved (rather than continuous) audio frames • Dropped packet results in multiple small disruptions rather than single large disruption

7. History: RealVideo • RealVideo design features: • Produced compressed data that can be streamed at a fixed bit rate • Non-uniform nature of video clips (scene changes, transitions, etc.) results in unequal distribution of bits between frames when encoded at same level of distortion • To maintain fixed bit rate, encoder must introduce unequal distortion between frames, neglect encoding some frames, or both • Use of preroll buffer increases bit rate flexibility—only average rate must be kept constant • Variable-bit-rate (VBR) rate-control algorithm

8. History: RealVideo • Facilitated random access by inserting Intra frames • Employed unequal error protection • Large video frame size precluded simple interleaving techniques • Instead, RealVideo combined forward error correction codes for the most sensitive data with various error concealment mechanisms

9. History: RealSystem G2 • SureStream technology improved adaptive streaming capabilities: • A single SureStream file contains multiple streams of the original content • Different streams optimized for different channel capacities • Client of stream monitors connection performance, dynamically requests optimal stream from server • Used IETF standardized RTP and RTSP protocols

10. History: Network Strategies • Multicasting reduces amount of redundant data transferred • Streaming media delivery networks allow for multiple, geographically scattered servers (achieved either by splitting or caching) • Dedicated networks guarantee bandwidth • Multiple-access transmissions over Internet—requested info sent from several different locations simultaneously

11. Streaming Media • Three modes in which video information can be streamed: • Live broadcasting • On-demand streaming • Conferencing

12. Streaming Media: Live Broadcasting System Structure

13. Streaming Media: Live Broadcasting • Live video encoded on-the-fly, passed to server, which disperses compressed info to all splitters and/or clients • Splitters are additional servers that can be used to distribute server workload over a number of machines • Push splitting: initiated by server • Pull splitting: initiated by a client connecting to a local splitter; request propagated to nearest active splitter • Multiple-access splitting: lower-tier splitters can connect to multiple upper-tier splitters

14. Streaming Media: On-demand System Structure

15. Streaming Media: On-demand • Distribution system similar to live broadcasting system, except: • No direct connection between encoder and server • Remote proxy servers can cache popular files • Clients allowed to rewind, fast forward, etc. • Server proxy transfers only initiated by client

16. Video Coding Problems • Channel capacity fluctuations • Adaptive client-driven serving • Allow client to inform server of network performance • Congestion control • Error control: use bandwidth prescribed by congestion control algorithm to minimize effects of packet loss • Dynamic prediction • Combined with sufficient preroll buffer, a broad set of statistical techniques can (possibly) be used to throttle transfer rate to improve user experience

17. Video Coding Problems • Random access • Can provide random access by inserting I-frames at the boundaries of small video intervals • Reduces achievable compression rates • Processing resources • Diverse range of client machines • Live encoding often done on computer with limited resources

18. Video Coding Algorithms for Streaming • Original motion-compensated hybrid video coding algorithms produced single-rate encoded versions of input signals • Inadequate for many network applications • Need scalable coding algorithms

19. Scalable Streaming Techniques • Simulcast: produce several, redundant, independently coded streams, each optimized for specific channel conditions • Not suitable for splitting, where the combined representation of all streams should be as small as possible • Scalable coding • Based on spatio-temporal pyramid decomposition of source signal • Temporal, spatial, and Signal to Noise Ratio (quantizer size) scalability modes

20. Scalable Streaming Techniques • Multiple-description coding for use in multiple-access systems • Achieved by replacing scalar quantization with multiple-description quantization scheme

21. RealSystem 8 • Adaptive channel encoding of streaming content to match bit rate with bandwidth and loss statistics • Channel statistics not available until time of streaming, so server must manage adaptive strategies • Infeasible for server to encode on demand, so encoding techniques are required which allow server to efficiently choose between different bit rates

22. RealSystem 8 • SureStream components: • Adaptive Stream Management (ASM) • SureStream file format • Source and channel coding algorithms

23. RealSystem 8 • ASM • Allows clients to communicate which type of encoding should be ‘synthesized’ by server • File format plug-in uses ASM rules to produce various combinations of encoded streams from compressed media files • Rules expressed by fully programmable syntax • Client subscribes to appropriate rule based on channel characteristics • ASM is format-independent

24. RealSystem 8 • RealVideo 8 encoding module:

25. RealVideo 8 Algorithm • Input filters: remove noise and potential artifacts introduced by edits and conversions of video signal • Remove low-energy spatial noise • De-interlace analog signals to produce progressive frames without “jaggy”-shaped artifacts • Inverse telecine filter: remove redundant 5.97 frames introduced by NTSC video

26. RealVideo 8 Algorithm • Spatial resampler: downscale input frames to spatial resolutions suitable for streaming • Codecs: motion-compensated hybrid scheme • Rate control determines which frames to encode • Low bit rates: perceived quality impacted by relationship between framerate and frame quality • High bit rates: must maintain framerate, want to maintain appearance of uniform high quality • Parameterized preroll buffer size • Rate control important in providing resiliency and random access

27. RealVideo 8 Algorithm • CPU scalability control module: throttles complexity of different components of algorithm to maintain best quality level for encoding live presentations

28. RealVideo 8 Algorithm • Client-side video postprocessing: filters aimed at improving subjective quality of decoded signals • Frame rate upsampler: special temporal filter that attempts to interpolate intermediate frames • Especially useful when many original frames were regularly skipped to decrease bit rate of encoding

29. Pointers • Helix Community • “The Helix community is a collaborative effort among Real, independent developers, and leading companies to extend the Helix DNA™ platform, the first open multi-format platform for digital media creation, delivery and playback.” • https://helixcommunity.org/

30. Conclusion • Streaming multimedia poses many challenges to video encoders • Variability of channel performance • High server workloads • Need to adapt coded bit rate to match channel capacities; must be done efficiently and on the fly • “RealSystem 8 provides an open and extensible platform, capable of accommodating various future needs of streaming media infrastructure on the Internet, and in particular, new demands for improved video-coding techniques.”