1 / 41

Wyatt Lloyd * Michael J. Freedman * Michael Kaminsky † David G. Andersen ‡

Don’t Settle for Eventual : Scalable Causal Consistency for Wide -Area Storage with COPS. Wyatt Lloyd * Michael J. Freedman * Michael Kaminsky † David G. Andersen ‡ * Princeton, † Intel Labs, ‡ CMU. Wide-Area Storage. Stores: Status Updates Likes Comments

kesler
Download Presentation

Wyatt Lloyd * Michael J. Freedman * Michael Kaminsky † David G. Andersen ‡

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Don’t Settle for Eventual: Scalable Causal Consistency for Wide-Area Storage with COPS Wyatt Lloyd* Michael J. Freedman* Michael Kaminsky† David G. Andersen‡ *Princeton, †Intel Labs, ‡CMU

  2. Wide-Area Storage Stores: Status Updates Likes Comments Photos Friends List Stores: Posts +1s Comments Photos Circles Stores: Tweets Favorites Following List

  3. Wide-Area Storage Serves Requests Quickly

  4. Inside the Datacenter Web Tier Storage Tier Web Tier Storage Tier Replication A-F A-F Remote DC G-L G-L M-R M-R S-Z S-Z

  5. Desired Properties: ALPS • Availability • Low Latency • Partition Tolerance • Scalability “Always On”

  6. Scalability A-Z A-Z Increase capacity and throughput in each datacenter A-C M-O M-O A-F A-C A-F P-S A-L G-L A-L D-F D-F P-S G-L G-J T-V M-Z M-Z M-R T-V M-R G-J W-Z K-L S-Z K-L S-Z W-Z

  7. Desired Property: Consistency • Restricts order/timing of operations • Stronger consistency: • Makes programming easier • Makes user experience better

  8. Consistency with ALPS Strong Sequential Causal Eventual Impossible [Brewer00, GilbertLynch02] Impossible [LiptonSandberg88, AttiyaWelch94] COPS Amazon LinkedIn Facebook/Apache Dynamo Voldemort Cassandra

  9. Causality By Example Causality ( ) Thread-of-Execution Gets-From Transitivity Remove boss from friends group Post to friends: “Time for a new job!” Friend reads post Friends Boss New Job!

  10. Causality Is Useful For Users: For Programmers: Friends Boss Photo Upload New Job! Add to album Employment Integrity Referential Integrity

  11. Conflicts in Causal K=1 K=1 K=1 K=2 K=2 K=2

  12. Conflicts in Causal Causal + Conflict Handling = Causal+ K=2 K=3 K=2 K=3 K=2 K=3

  13. Previous Causal+ Systems • Bayou ‘94, TACT ‘00, PRACTI ‘06 • Log-exchange based • Log is single serialization point • Implicitlycaptures and enforces causal order • Limits scalability OR • No cross-server causality

  14. Scalability Key Idea • Dependency metadata explicitly captures causality • Distributed verifications replace single serialization • Delay exposing replicated puts until all dependencies are satisfied in the datacenter

  15. COPS Key-Value Store All Data Causal+ Replication Local Datacenter Client Library All Data All Data

  16. Get Key-Value Store Local Datacenter Client Library get get

  17. Put put + ordering metadata Key-Value Store put after = ? Local Datacenter ? Client Library put put_after K:V Replication Q put after

  18. Dependencies • Dependencies are explicit metadata on values • Library tracks and attaches them to put_afters

  19. Dependencies • Dependencies are explicit metadata on values • Library tracks and attaches them to put_afters Client 1 put_after(Key,Val,deps) put(Key, Val) deps . . . Kversion version (Thread-Of-Execution Rule)

  20. Dependencies • Dependencies are explicit metadata on values • Library tracks and attaches them to put_afters Client 2 get(K) value get(K) value,version,deps' deps . . . Kversion L337 M195 deps' L337 M195 (Gets-From Rule) (Transitivity Rule)

  21. Causal+ Replication Key-Value Store put_after(K,V,deps) K:V,deps Replication Q put after

  22. Causal+ Replication dep_check(L337) put_after(K,V,deps) K:V,deps deps L337 M195 dep_check(M195) Exposing values after dep_checks return ensures causal+

  23. Basic COPS Summary • Serve operations locally, replicate in background • “Always On” • Partition keyspace onto many nodes • Scalability • Control replication with dependencies • Causal+ Consistency

  24. Gets Aren’t Enough My Operations Remote Datacenter You’re Fired!! Boss Remote Progress Remote Progress Remote Progress Boss Boss Boss Portugal! Portugal! Boss New Job! New Job! New Job!

  25. Gets Aren’t Enough My Operations Remote Datacenter You’re Fired!! Boss Remote Progress Remote Progress Remote Progress Boss Boss Boss Boss Boss Portugal! Portugal! New Job! New Job! Boss New Job! Portugal! Portugal! New Job! Boss

  26. Get Transactions • Provide consistent view of multiple keys • Snapshot of visible values • Keys can be spread across many servers • Takes at most 2 parallel rounds of gets • No locks, no blocking Low Latency

  27. Get Transactions My Operations Remote Datacenter Boss Could Get Remote Progress Remote Progress Remote Progress Remote Progress Remote Progress Boss Boss Boss Portugal! Boss Boss Boss Boss Boss Boss Portugal! New Job! Portugal! Portugal! New Job! New Job! Portugal! Portugal! New Job! Portugal! Never Boss Boss Portugal! Boss New Job!

  28. System So Far • ALPS and Causal+, but … • Proliferation of dependencies reduces efficiency • Results in lots of metadata • Requires lots of verification • We need to reduce metadata and dep_checks • Nearest dependencies • Dependency garbage collection

  29. Many Dependencies • Dependencies grow with client lifetime Put Put Get Get Put Put

  30. Nearest Dependencies • Transitively capture all ordering constraints

  31. The Nearest Are Few • Transitively capture all ordering constraints

  32. The Nearest Are Few • Only check nearest when replicating • COPS only tracks nearest • COPS-GT tracks non-nearest for transactions • Dependency garbage collection tames metadata in COPS-GT

  33. Extended COPS Summary • Get transactions • Provide consistent view of multiple keys • Nearest Dependencies • Reduce number of dep_checks • Reduce metadata in COPS

  34. Evaluation Questions • Overhead of get transactions? • Compare to previous causal+ systems? • Scale?

  35. Experimental Setup Local Datacenter COPS Servers Clients Replication Remote DC COPS N N N

  36. COPS & COPS-GTCompetitive for Expected Workloads All Put Workload – 4 Servers / Datacenter Low per-client write rates expected High per-client write rates result in 1000s of dependencies People tweeting 1000 times/sec People tweeting 1 time/sec

  37. COPS & COPS-GTCompetitive for Expected Workloads Varied Workloads – 4 Servers / Datacenter Pathological Expected Workload

  38. COPS Low Overhead vs. LOG • COPS – dependencies ≈ LOG • 1 server per datacenter only • COPS and LOG achieve very similar throughput • Nearest dependencies mean very little metadata • In this case dep_checks are function calls

  39. COPS Scales Out

  40. Conclusion • Novel Properties • First ALPS and causal+ consistent system in COPS • Lock free, low latency get transactions in COPS-GT • Novel techniques • Explicit dependency tracking and verification with decentralized replication • Optimizations to reduce metadata and checks • COPS achieves high throughput and scales out

More Related