1 / 40

From application requests to Virtual IOPs: Provisioned key-value storage with Libra

From application requests to Virtual IOPs: Provisioned key-value storage with Libra. David Shue * and Michael J. Freedman. (*now at Google). Shared Cloud. Tenant C. Tenant A. Tenant B. VM. VM. VM. VM. VM. VM. VM. VM. VM. Shared Cloud Storage. Tenant C. Tenant A. Tenant B. VM.

amanda
Download Presentation

From application requests to Virtual IOPs: Provisioned key-value storage with Libra

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue* and Michael J. Freedman (*now at Google)

  2. Shared Cloud Tenant C Tenant A Tenant B VM VM VM VM VM VM VM VM VM

  3. Shared Cloud Storage Tenant C Tenant A Tenant B VM VM VM VM VM VM VM VM VM Block Storage Key-Value Storage SQL Database

  4. Unpredictable Shared Cloud Storage Tenant C Tenant A Tenant B VM VM VM VM VM VM VM VM VM Key-Value Storage Block Storage Key-Value Storage SQL Database Disk IO-bound Tenants SSD-backed storage

  5. Provisioned Shared Key-Value Storage ReservationA ReservationB ReservationC Tenant C Tenant A Tenant B VM VM VM VM VM VM VM VM VM Application Requests GET/s PUT/s (1KB normalized) Shared Key-Value Storage IOPs BW Low-level IO SSD SSD SSD SSD SSD

  6. Libra Contributions • Libra IO Scheduler • Provisions low-level IOallocations for app-requestreservations w/ high utilization. • Supports arbitrary object distributions and workloads. • 2 key mechanisms • Track per-tenant app-request resource profiles. • Model IO resources with Virtual IOPs.

  7. Related Work

  8. Tenant B VM VM VM Provisioned Distributed Key-Value Storage Global Reservation Problem [Pisces OSD1 ’12] ReservationA ReservationB Tenant A Tenant B VM VM VM VM VM VM Shared Key-Value Storage Local Reservation Problem Storage Node 1 Storage Node N ...

  9. Tenant A Tenant B VM VM VM VM VM VM Provisioned Distributed Key-Value Storage ReservationA ReservationB demand data partitions Storage Node 1 Storage Node N ...

  10. Provisioned Distributed Key-Value Storage ReservationA ReservationA ReservationB ReservationB Storage Node 1 Storage Node N ...

  11. Provisioned Distributed Key-Value Storage ReservationB ReservationB ReservationA ReservationA ResA1 ResB1 ResAn ResBn Storage Node 1 Storage Node N ...

  12. Provisioned Local Key-Value Storage GET 1001100 GET K Key-Value Protocol Retrieve K Persistence Engine Read l337 Libra IO Scheduler IO Scheduler IO operation Physical Disk

  13. GET PUT Libra Design Persistence Engine Reservation Distribution Policy Provision IO allocations for tenant app-request reservations blah Libra IO Scheduler Determine the cost of a tenant request Libra Provisioning Policy How much IO to provision? How much IO to consume? DRR blah Physical Disk

  14. Provisioning App-request Reservations is Hard IO Amplification 1 KB PUT ≥ 1 KB Write Track tenant app-request resource profiles ReservationBn IO Interference Underestimate provisionable IO Variable IO throughput Non-linear IO Performance Model IO with Virtual IOPs Non-linear cost per KB

  15. H F O M V index V2 V3 V3 A O A index index V1 V0 index V3 Workload-dependent IO Amplification LevelDB (LSM-Tree) PUT K,V3 PUT FLUSH COMPACT

  16. H F O M V index V2 V3 V3 A O A index index V1 V0 index V3 Workload-dependent IO Amplification LevelDB (LSM-Tree) PUT K,V3 PUT FLUSH COMPACT

  17. H F O M V index V2 V3 V3 A O A index index V1 V0 index V3 Workload-dependent IO Amplification LevelDB (LSM-Tree) PUT K,V3 PUT FLUSH COMPACT

  18. H F O M V A G Z index index index index K Workload-dependent IO Amplification GET K Workload-dependent IO Amplification

  19. GET PUT IO IO Libra Tracks App-request IO Consumption to Determine IO Allocations Tenant A 500 IO/s Libra IO Scheduler blah Compute app-request IO profiles Track IO consumption Provision IO allocations Libra Provisioning Policy 5 5 GET EWMA per-request averages 80 100 x Per-GET 1 25 PUT = + 1 x 50 Per-PUT 6 FLUSH 0.5 PUT Tenant A 5 IO units 70 500 blah COMPACT FLUSH

  20. Unpredictable IO Interference 4 read/4 write tenants • Die-level parallelism, low latency IOPs • Shared-controller and bus contention • Erase-before-write overhead • FTL and read-modify-write garbage colleciton

  21. Unpredictable IO Interference Unpredictable IO Interference 21

  22. Libra Underestimates IO Capacity to Ensure Provisionable Throughput Provisionable IO throughput = floor(workloads) (18 Kop/s) Provisionable IO limit

  23. Libra Underestimates IO Capacity to Ensure Provisionable Throughput Provisionable IO throughput = floor(workloads) (18 Kop/s) Provisionable IO limit

  24. Libra Underestimates IO Capacity to Ensure Provisionable Throughput Provisionable IO throughput = floor(workloads) (18 Kop/s) Provisionable IO limit

  25. Non-linear IO Performance Max BW Max IOP/s IOP Cost = linear decrease until 6 KB, then constant IOP Cost = constant until 6 KB, then linear increase

  26. Non-linear IO Performance Non-linear IO Performance Max BW Max IOP/s

  27. Libra Uses Virtual IOPs to Model IO Resources • Unifies IO cost into a single metric • Captures non-linear IO performance • Provides IO insulation 2 equal-allocation tenants IO Insulation = 1/2 Max Read/Write

  28. Libra Uses Virtual IOPs to Model IO Resources 2 equal-allocation tenants IO Insulation = 1/2 Max Read/Write

  29. Libra Design Persistence Engine Update tenant VOP allocations Provision IO allocations for tenant app-request reservations blah Libra IO Scheduler Determine the cost of a tenant request Libra Provisioning Policy Charge tenant IOPs based on VOP cost Provision VOPs within provisionable limit blah Track app-request VOP consumption Physical Disk

  30. Evaluation • Does Libra's IO resource model achieve accurate resource allocations? • Does Libra's IO threshold make an acceptable tradeoff of performance for predictability in a real storage stack? • Can Libra ensure per-tenant app-request reservations while achieving high utilization?

  31. Libra Achieves Accurate IO Allocations Interference-free Ideal even Read 1 KB Throughput Ratio = Actual / Expected (IO Insulation)

  32. Libra Achieves Accurate IO Allocations Interference-free Ideal Write 1-256 KB Throughput Ratio = Actual / Expected (IO Insulation)

  33. Libra Achieves Accurate IO Allocations

  34. Libra Achieves Accurate IO Allocations

  35. Libra Achieves Accurate IO Allocations Min-Max Ratio = Min Throughput Ratio / Max Throughput Ratio

  36. Libra Trades-off Nominal IO Throughput For Predictability

  37. Libra Trades-off Nominal IO Throughput For Predictability Unprovisionable Throughput As a Percentage of Total Throughput < 10th percentile covered by SLA and higher-level policies

  38. Libra Achieves App-request Reservations 0.5x 1.5x Read Heavy Mixed Write Heavy Fully provisioned allocations Work-conserving consumption of unprovisioned resources

  39. Libra Achieves App-request Reservations Read Heavy Mixed Write Heavy

  40. Conclusion • Libra IO Scheduler • Provisions IOallocations for app-requestreservations w/ high utilization. • Supports arbitrary object distributions and workloads. • 2 key mechanisms • Track per-tenant app-request resource profiles. • Model IO resources with Virtual IOPs. • Evaluation • Achieves accurate low-level IO allocations. • Provisions the majority of IO resources over a wide range of workloads • Satisfies app-request reservations w/ high utilization.

More Related