1 / 32

Distributed SkyDrive: A Storage System based on Erasure Resilient Coding

Learn about the design of Distributed SkyDrive, an efficient storage system using Erasure Resilient Coding to ensure reliability and efficiency in data storage. Explore its features, layout, and future work in this comprehensive guide.

stephanie
Download Presentation

Distributed SkyDrive: A Storage System based on Erasure Resilient Coding

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

    1. Distributed SkyDrive: A StorageSystem based on ErasureResilientCoding Jian Liang Mentor: Jin Li, Li-wei He CCS Group, MSR 2006

    2. Outline Introduction Distributed Skydrive design Demo Future work

    3. Introduction Trend of enterprise storage Store more data with cheaper devices Software solution on cheap commodity (ATA) Replication based fail-over and load balance

    4. Replica-based Efforts GFS (Google) Blue (MSN Hotmail) Cosmos (MSN search) Strategy: replicate cheap disks + software = reliable RAID

    5. 3-Replica Layout

    6. Append Only Service PutBlob: write a variable size object (file/email/webpage) into storage GetBlob: read a object and associated metadata from storage given guid. ExpireBlob: Mark object to be removed QueryBlob: object lookup (search utility)

    7. Introduction to ERC ERC is linear All ERC blocks are linear combination of the coded data (encoding order doesnt matter) Systematic ERC Access to a fragment of original blocks without ERC decoding

    8. Distributed Storage With Erasure Coding

    9. Reliability

    10. Allocation of Frags To avoid correlated failure, each group of frags (Original & ERC fragments) should be placed onto different hard-drive, machine & racks. You can consider each as a group of 20 fragments, and devise a scheme that place frag on separate machine

    11. ERC Read ERC Read is potentially a distributed read If original frag is live & fresh, read there (single read) If original frag is dead or stale, do distributed read Read from any k frags that are live & fresh (tricks on stale frag) Do ERC decoding

    12. ERC Append

    13. Pros & Cons of ERC Storage Less space in ERC based storage E.g. 16/20 vs. 3-replica ERC 20/16 = 125% vs. 3-replica 3/1 = 300% 42% space cost of replica Challenges in ERC based Storage More I/O overhead, complexity in update E.g. 16/20 vs. 3-replica 5 vs. 3 writes, 15% more read assuming 1% of failure.

    14. D-Skydrive Storage Layout

    15. Fragment Structure

    16. Prototype Deployment

    17. Storage Node Distributed skydrive can also be used in corporate network for efficient data backup. In such a case, the skydrive will consist one or more index servers, and a number of fragment servers that are just hard drive in the peer users machine. Distributed skydrive provide reliability and efficiency to the data store. Distributed skydrive can also be used in corporate network for efficient data backup. In such a case, the skydrive will consist one or more index servers, and a number of fragment servers that are just hard drive in the peer users machine. Distributed skydrive provide reliability and efficiency to the data store.

    18. Append

    19. Read

    20. Range Check For Validity

    21. Consistency Problem

    22. Consistent Read during Write

    23. Fail Over

    24. Prototype Deployment

    25. Experiment Setup

    26. One Client R/W Throughput

    27. Throughput with File Size

    28. Demo: Photo-Share based on D-Skydrive Service Basic Functionality Create/Append/Recovery objects Multiply reader/writer Consistency test Failover Fresh -> Stale -> fresh

    29. Demo Setup

    30. Screenshot 1

    31. Screenshot 2

    32. Future Work Performance tuning on ERC input I/O optimization, cache/pre-fetch More measurement Distributed Index management

    33. Thank you !

More Related