236601 - Coding and Algorithms for Memories Lecture 13

1. 236601 - Coding and Algorithms for MemoriesLecture 13

2. Large Scale Storage Systems • Big Data Players: Facebook, Amazon, Google, Yahoo,… Cluster of machines running Hadoop at Yahoo! (Source: Yahoo!) • Failures are the norm 2

3. Node failures at Facebook Date XORingElephants: Novel Erasure Codes for Big Data M. Sathiamoorthy, M. Asteris, D. Papailiopoulos, A. G. Dimakis, R. Vadali, S. Chen, and D. Borthakur, VLDB 2013 3

4. Problem Setup • Disks are stored together in a group (rack) • Disk failures should be supported • Requirements: • Support as many disk failures as possible • And yet… • Optimal and fast recovery • Low complexity

5. Reed Solomon Codes • A code with parity check matrix of the form Where is a primitive element at some extension field and O() > n-1 Claim: Every sub-matrix of size dxd has full rank

6. Reed Solomon Codes • Advantages: • Support the maximum number of disk failures • Are very comment in practice and have relatively efficient encoding/decoding schemes • Disadvantages • Require to work over large fieldsSolution: EvenOdd Codes • Need to read all the disks in order to recover even a single disk failure – not efficient rebuildSolution: ZigZag Codes

7. The Repair Problem RS code • Facebook’s storage Scheme: • 10 data blocks • 4 parity blocks • Can tolerate any four disk failures 1 2 3 4 5 6 7 8 9 10 P1 P2 P3 P4 • A disk is lost – Repair job starts • Access, read, and transmit data of disks! • Overuse of system resources during single repair • Goal: Reduce repair cost in a single disk repair 7

8. ZigZag Codes • Designed by ItzhakTamo, Zhiying Wang, and JehoshuaBruck • The goal: construct codes correcting the max number of erasures and yet allow efficient reconstruction if only a single drive fails

9. ZigZag Codes • Lower bound: The min amount of data required to be read to recover a single drive failure • (n,k) code: n drives, k information, and n-k redundancy • M- size of a single drive in bits • For (n,n-2) code it is required to read at least 1/2 from the remaining drives, that is at least (1/2)(n-1)M bits • The last example is optimal • In general, for (n,n-r) code it required to read at least 1/r from the remaining drives (1/r)(n-1)M

10. ZigZag Codes • Example

11. Network Coding for Distributed Storage • Goal – show the following:In general, for (n,n-r) code it required to read at least 1/r from the remaining drives (1/r)(n-1)M • Network Coding for Distributed StorageDimakis, Godfrey, Wu, Wainwright, Ramchandran • File of size M is partitioned into k pieces of size M/k • The k pieces are encoded into n encoded pieces using an (n,k) MDS code

12. Network Coding for Distributed Storage • File of size M is partitioned into k pieces of size M/k • The k pieces are encoded into n encoded pieces using an (n,k)MDS code x1 x2 y1 x3 y2 x4

13. Network Coding for Distributed Storage • File of size M is partitioned into k pieces of size M/k • The k pieces are encoded into n encoded pieces using an (n,k)MDS code x1 β=? β x2 y1 x3 β y2 x5 x4

14. Network Coding for Distributed Storage • File of size M is partitioned into k pieces of size M/k • The k pieces are encoded into n encoded pieces using an (n,k)MDS code α=1 x1in x1out β=? ∞ ∞ α=1 x2in x2out ∞ β DC S ∞ α=1 x3in x3out β ∞ ∞ x5in x5out α=1 x4in x4out

15. ZigZag Codes • Example