Real time stream processing architecture for comcast ip video
1 / 20

Real-time Stream Processing Architecture for Comcast IP Video - PowerPoint PPT Presentation

  • Uploaded on

Real-time Stream Processing Architecture for Comcast IP Video. Strata Conference + Hadoop World 2013 Chris Lintz Gabriel Commeau. Agenda. Comcast VIPER Overview Architecture Overview Q & A. Comcast Video IP Engineering and Research (VIPER).

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about ' Real-time Stream Processing Architecture for Comcast IP Video' - aminia

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Real time stream processing architecture for comcast ip video

Real-time Stream Processing Architecture for Comcast IP Video

Strata Conference + HadoopWorld 2013

Chris Lintz

Gabriel Commeau

Agenda Video

  • Comcast VIPER Overview

  • Architecture Overview

  • Q & A

Comcast video ip engineering and research viper
Comcast Video IP Engineering and Research (VIPER) Video

Preparation Delivery Video Players







Video Players

Xbox Live




Why do we focus on real time
Why Do We Focus on Real-time? Video

  • Proactively diagnose issues

  • Form real-time intelligence

  • Help deliver best possible video experience


Prime Time

Video player analytics protocol
Video Player Analytics Protocol Video

  • Live and On Demand

  • JSON event objects

  • Key metrics

    • Bitrate

    • Frame rate

    • Fragments

    • Errors

We collect and use all data in accordance with best consumer

privacy practices and applicable laws

Flume data collection tier
Flume: Data collection Tier Video

  • Collect, aggregate and move large amounts of data

  • Distributed, scalable, reliable, customizable

  • Multi-tier architecture

Player sessions in real time
Player Sessions in Real-time Video

  • Sessions in Flume?

    • Technical issues: consistent hash and exactly-once semantics

    • Design goals

    • Separation of concerns

  • Session write-through rate?

Flume edge tier video player analytics end point
Flume Edge Tier: Video Player Analytics End Point Video

  • Analytics events over HTTPS

  • HTTP Source

  • Re-batch with inner sink and source

Flume mid tier processing and routing data
Flume Mid Tier: Processing and Routing Data Video

  • Video Player Event processing

    • Geo-location, asset metadata, validation, to-storm

  • Replication channel processor:

    • HDFS sink

    • Storm sink

Bridging flume to storm flume2storm connector
Bridging Flume to Storm: Flume2Storm Connector Video

  • Service discovery

  • Distributed, scalable and reliable

  • Low latency

Requirements for read writes from storm bolts
Requirements for Read/Writes from Storm Bolts Video

  • Functionality beyond key/value stores

  • Real-time and historic window queries

  • Speed of in-memory writes and durability of disk

Utilizing memsql for persistence
Utilizing VideoMemSQL for Persistence

  • Distributed in-memory SQL database

  • ACID, highly available, fault tolerant

  • Aggregators route queries to leaves

  • Leaves are auto-sharded

  • Solves our intense


Achievements in utilizing memsql
Achievements In Utilizing VideoMemSQL

  • Complex queries in milliseconds

  • Fault-tolerant Storm bolt state

  • Joins now available outside of Storm bolts

    • Foreign key shards

  • Complex data streams

    • Dynamic alters without locks or down time

    • JSON type

Wrapping up
Wrapping Up Video

  • Real-time at Comcast scale

    • Millions of video players

    • Horizontal scale everywhere

    • Aggregated metrics across US and complex analysis

    • Real-time API

  • Builds foundation

    • Advanced real-time analytics

    • Better platform for innovation

      • Alerts on complex objects

      • Supplemental real-time data back to clients

      • Popularity-based CDN