1 / 20

A Row-Permutated Data Reorganization Algorithm for Growing Server-less VoD Systems

A Row-Permutated Data Reorganization Algorithm for Growing Server-less VoD Systems. Presented by Ho Tsz Kin. Agenda. Background Existing solutions Row-Permutated (RP) Algorithm Multi-RP Algorithm Performance Evaluation Conclusion. Background. Each node keeps balance video data blocks

lesley-good
Download Presentation

A Row-Permutated Data Reorganization Algorithm for Growing Server-less VoD Systems

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A Row-Permutated Data Reorganization Algorithm forGrowing Server-less VoD Systems Presented by Ho Tsz Kin

  2. Agenda • Background • Existing solutions • Row-Permutated (RP) Algorithm • Multi-RP Algorithm • Performance Evaluation • Conclusion

  3. Background • Each node keeps balance video data blocks • Nodes join the system • Data must be reorganized to utilize storage and streaming capacity n0 n1 n2 n3 n0 n1 n2 n3 n3 0 1 2 3 0 1 2 3 4 4 5 6 7 5 6 7 8 9 node n4 joins 8 9 10 11 10 11 12 13 14 12 13 14 15 15 16 17 18 19 16 17 18 19

  4. Background • Data reorganization • Require data block movement • Consume bandwidth • Should not disrupt services • Achieve storage and streaming balance

  5. Existing Solutions • Round-robin Reorganization • Round-robin placement policy • Advantages: Perfect storage and streaming balance • Drawbacks: Nearly all the data blocks must be reorganized n0 n1 n2 n3 n0 n1 n2 n3 n3 0 1 2 3 0 1 2 3 4 4 5 6 7 5 6 7 8 9 node n4 joins 8 9 10 11 10 11 12 13 14 12 13 14 15 15 16 17 18 19 16 17 18 19

  6. 0 Assign to each node with equal probability n0 n1 n2 n3 Existing Solutions • Randomized Reorganization • Randomized placement policy • Blocks are distributed to nodes randomly n0 n1 n2 n3 3 1 0 5 8 2 6 7 9 4 11 10 15 13 12 14 16 19 18 17

  7. Existing Solutions • Reorganization Algorithm • Number of nodes, N • Probability of residing in same node = • Probability of moving to new node = n0 n1 n2 n3 n4 P = P = 3 1 0 5 8 2 6 7 9 4 11 10 15 13 12 14 16 19 18 17

  8. Existing Solutions • Randomized Reorganization • Advantages: Block movement is minimized, achieve reasonable storage balance • Drawbacks: Streaming load is imbalance n0 n1 n2 n3 n0 n1 n2 n3 n4 3 1 0 5 3 1 0 5 4 8 2 6 7 8 2 6 7 11 node n4 joins 9 9 4 11 10 13 12 14 10 15 15 13 12 14 19 18 16 16 19 18 17 imbalance row 17

  9. Goal • Two extreme cases • Round-robin Reorganization • Overhead is maximum, balance streaming load • Randomized Reorganization • Overhead is minimum, imbalance streaming load • Two Goals: • Maintain balance streaming load but lower the overhead of round-robin reorganization • Allow controllable tradeoff between overhead and streaming load balance

  10. Row-Permutated (RP) Algorithm • Idea: the sequence of blocks within each row is not important in streaming load • Row-permutated placement policy • Streaming load is still balanced Both maintain balanced streaming load n0 n1 n2 n3 n0 n1 n2 n3 1 0 3 2 0 1 2 3 Round-robin Placement Row-Permutated Placement

  11. Row-Permutated (RP) Algorithm • Reorganization Algorithm • Reorganize one row per iteration • Identify overflow and underflow nodes • Overflow if more than 1 block • Underflow if no block • Move excess block from overflow nodes to underflow nodes n0 n1 n2 n3 n4 0 1 3 2 Underflow Node Overflow Node 7 4 5 6 8 10 9 11 Target row in this iteration Excess block 13 12 14 15 16 17 19 18

  12. Row-Permutated (RP) Algorithm • Perfect streaming and storage balance • Significantly lower down number of block movement during reorganization n0 n1 n2 n3 n0 n1 n2 n3 n4 0 1 3 2 4 0 1 3 2 4 9 8 7 5 6 7 5 6 node n4 joins 8 10 9 11 13 10 14 11 12 13 12 14 15 16 17 19 15 18 16 17 19 18

  13. Multi-RP Algorithm • Tradeoff between overhead and streaming balance • Control streaming balance by window size, w n0 n1 n2 n3 n0 n1 n2 n3 n4 0 1 2 3 11 13 10 15 w =2 5 4 6 7 12 14 11 9 10 8 16 18 12 13 14 15 17 19 16 20 18 22 Consider 2 rows 17 21 19 23

  14. Multi-RP Algorithm • Reorganization Algorithm • Reorganize w rows per iteration • Identify overflow and underflow nodes • Overflow if more than w blocks • Underflow if fewer than w blocks Overflow Nodes n0 n1 n2 n3 n4 11 13 10 15 w =2 12 14 16 18 Underflow Nodes 17 19

  15. Multi-RP Algorithm • In each overflow node • Choose row with largest number of block • Take blocks in this row as excess blocks • Move to underflow nodes • Contains smallest number of blocks in this row n0 n1 n2 n3 n4 n0 n1 n2 n3 n4 11 13 10 15 11 13 10 15 17 12 14 12 14 16 18 16 18 randomly 19 17 19

  16. Multi-RP Algorithm • Idea: Spread out blocks within row n0 n1 n2 n3 n4 n0 n1 n2 n3 n4 11 13 10 15 14 11 13 10 15 16 17 18 12 19 17 12 14 16 18 row with largest number of blocks 19 n0 n1 n2 n3 n4 11 13 10 15 17 14 12 16 18 19

  17. Performance Evaluation • Experiment Details • Number of data blocks = 4000 • Grow from 1 node to 200 nodes • Metrics • Data Reorganization Overhead • Number of block movement • Streaming Load Balance • Proportion of missing data block within one row, given that each node can only send out one block each round

  18. Data Reorganization Overhead

  19. Streaming load balance

  20. Conclusion • Identify the shortcomings of round-robin and randomized reorganization • RP and multi-RP reorganization are proposed • Perfect streaming load balance with lower overhead • Controllable tradeoff between overhead and streaming load balance

More Related