1 / 20

Managing 100TB of small files…

Prospero Media Storage. IGT – . Event. Managing 100TB of small files…. July 2011. Numbers. 70TB used space 700 million files 200GB and 250,000 files uploaded every day 1200Mbps bandwidth throughput in peak 180TB of data is being served out monthly

slone
Download Presentation

Managing 100TB of small files…

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Prospero Media Storage IGT – Event Managing 100TB of small files… July 2011

  2. Numbers • 70TBused space • 700 million files • 200GBand 250,000 files uploaded every day • 1200Mbpsbandwidth throughput in peak • 180TBof data is being served out monthly • 3700 Hits per second in peak • 40 storage node servers – 300TB raw space • $0.13 per GB

  3. Motivation • Web 2.0 content serving paradigm shift • Too many files • 12M users x 1 file = very long tail • Too many connections • 1M users + keepalive = 1M connections • Living with modern content in web 2.0 • 1 file x (thumbnail + iPhone + Mac) = 3 file copies

  4. Traditional Architecture HTTP IO IO IO IO Centralized Storage (NAS, SAN, DAS etc.)

  5. Traditional Architecture HTTP – TOO MANY CONNECTIONS IO IO IO IO Centralized Storage (NAS, SAN, DAS etc.)

  6. Traditional Architecture HTTP IO IO IO IO IO IO IO Centralized Storage (NAS, SAN, DAS etc.)

  7. Traditional Architecture HTTP IO IO IO IO IO IO IO Too much IO

  8. Traditional Architecture HTTP Cache IO IO IO IO IO IO IO Centralized Storage (NAS, SAN, DAS etc.)

  9. “There are only two hard things in Computer Science: cache invalidation and naming things”. -- Tim Bray quoting Phil Karlton

  10. Architecture goals • Symmetric identical server nodes • Simplified management and scaling • Linear scaling out • No functional / role servers • No single point of failure • No performance bottlenecks • Multiple datacenters support • DRP support • Geo load distribution

  11. Meet Prospero • Distributed Web content storage system • Full blown HTTP support • Runs on low cost commodity hardware • Adjustable file level replication controls redundancy policy for every content type • Provides dynamic image manipulation

  12. How do we do it?

  13. Designed to fail • Fallback for every operation • Geographical, machine, storage medium • Write never fails • All files will reach their destination • Journaling • Tracking all uploaded files • Pending jobs • Guaranteed file distribution

  14. How do we achieve this • Control the input • define the only unified API • Functional process isolation • every function deserves its own process by default • watchdogs • monitors • alerts

  15. get 37D815B5.jpg Go to 37 range servers Fallback if not found 2.static 6.static 0.static 4.static HTTP HTTP HTTP 20-3f 60-7f 00-1f 40-5f 1.static 3.static 7.static 5.static HTTP HTTP HTTP

  16. Fallback Example

  17. Node Architecture

  18. Real Life

  19. It’s all about performance • Non blocking IO, readiness notification (epoll) • Asynchronous file IO (AIO) • Zero copy (sendfile) • Memory maps • Inter-process binary protocols • UNIX socket • Minimize dynamic memory allocation • lighttpd memory footprint: 50MB

  20. Lessons learnt • Be symmetric • Control the input • Design to failure • Performance matters again • Simple is hard but a must

More Related