Distributed Storage Allocations for Optimal Delay

1. Distributed Storage Allocationsfor Optimal Delay • Derek Leong1, Alexandros G. Dimakis2, Tracey Ho1 • 1California Institute of Technology • 2University of Southern California • ISIT 2011 • 2011-08-02

2. Motivation How should we store a data object over a set of mobile nodes, subject to a given storage budget, so as to achieve the optimal recovery delay? When is coding beneficial? When will uncoded replication suffice?

3. Introduction • Network Model • Consider a network of n mobile storage nodes • Assume that the number of contacts between any given pair ofnodes follows a Poisson distribution with rate parameter ¸;the time between contacts is therefore described by an exponential distribution with mean 1/¸

4. Introduction • Storage Allocation • A source node creates a data object of unit size, and subsequently disseminates an encoded representation of it to other nodes for storage, subject to a given total storage budget T • At the end of the dissemination process, node 1 stores x1 amount of data, node 2 stores x2amount of data, and so on, such that

5. Introduction • Recovery by a Data Collector • At some time after the completion of the data dissemination process, a data collector node begins to recover the data object by contacting other nodes and accessing the coded data stored in them • We make the simplifying assumption that the stored data is instantaneously transmitted on contact • Let random variable D denote the recovery delay incurred by the data collector

6. Introduction • Objectives • We seek an allocation (x1;…;xn) of the given budgetT that produces the optimal recovery delay; specifically, we considertwo objectives involving the recovery delay D:(i) maximization of the probability of successful recovery by agiven deadline d, or recovery probability (ii) minimization of the expected recovery delay • Therefore, for each objective, we need to find (i)an optimal allocation of the given budget over the nodes, and (ii) an optimal coding scheme that jointly optimize the objective

7. Introduction • Objectives • Using an appropriate code, successful recovery occurs wheneverthe data collector accesses at least a unit amount of data(= size of the original data object) s 1 2 3 4 5 t2 t1 A. G. Dimakis, P. B. Godfrey, Y. Wu, M. J. Wainwright, and K. Ramchandran, “Network coding for distributed storage systems,” Trans. Inf. Theory, Sep 2010. A. Jiang, “Network coding for joint storage and transmission with minimum cost,” in Proc. ISIT, Jul 2006.

8. Introduction • Objectives • Therefore, assuming the use of an appropriate code (e.g. random linear coding, MDS code), we can express the recovery delay D as ,where is the set of all nodes contacted by thedata collector by time d

9. Maximizing Recovery Probability • Let W1, …, Wn be i.i.d. random variables denoting the times at which the data collector first contacts nodes 1, …, n, respectively, where • Observe that the data collector contacts each node by the specified deadline d> 0 independently with probability p¸ ; d given by • It follows that the probability of contacting exactly a subset r of the n nodes by time d is ; the recovery probability can therefore be obtained by summing over all subsets r that allow successful recovery: Wi~ Exponential(¸)

10. Recovery Probability: Related Work RECAP • Discussion between R. Karp, R. Kleinberg, C. Papadimitriou, E. Friedman, and others at UC Berkeley, 2005 • Found examples for the suboptimality of symmetric allocations • Conjectured that there exists a symmetric optimal allocation when the number of nodes n → ∞ • S. Jain, M. Demmer, R. Patra, K. Fall, “Using redundancy to cope with failures in a delay tolerant network,” SIGCOMM 2005 • Considered the allocation of a transmission budget over different routes in a DTN • Experimentally evaluated the performance of symmetric allocations along with other heuristics • Related theoretical claims and proofs incomplete/inaccurate

11. Recovery Probability: Illustrative Example RECAP n= 5 nodes, access probability p¸ ; d= 2/3, budget T = 7/3 RecoveryProbability A 7/157/15 7/157/157/15 0.79012 B 7/6 7/60 0 0 5 2 1 4 3 0.88889 C 2/3 2/3 1/31/31/3 C 0.90535

12. Recovery Probability: Optimal Symmetric Allocation RECAP • We are particularly interested in symmetric allocations because they are easy to describe and implement • Successful recovery for the symmetric allocation occursif and only if at least out of the m nonempty nodesare accessed • Therefore, the recovery probability of is given by D. Leong, A. G. Dimakis, and T. Ho, “Symmetric allocations for distributed storage,” in Proc. GLOBECOM, Dec 2010.

13. Recovery Probability: Optimal Symmetric Allocation RECAP The problem is nontrivial even when restricted to symmetric allocations… number of nonempty nodes in the symmetric allocation The recovery probability for the symmetric allocation is

14. Recovery Probability: Optimal Symmetric Allocation RECAP Maximal spreading (with coding) is optimal among symmetric allocations when the contact rate ¸ or recovery deadline d is sufficiently large: PROPOSITION 1 If , then either or is an optimal symmetric allocation. Minimal spreading (uncoded replication) is optimal among symmetric allocations when the contact rate ¸ or recovery deadline d is sufficiently small: PROPOSITION 2 If , then is an optimal symmetric allocation. D. Leong, A. G. Dimakis, and T. Ho, “Symmetric allocations for distributed storage,” in Proc. GLOBECOM, Dec 2010.

15. Recovery Probability: Optimal Symmetric Allocation RECAP other symmetric allocations may be optimal in the gap maximal spreading(with coding) is optimal among symmetric allocations When exactly, we observe numerically that minimal spreading is optimal among symmetric allocations for most values of T ;the optimal symmetric allocation changes continually over the intervalswhile is optimal for minimalspreading(uncoded replication) is optimal among symmetric allocations

16. Minimizing Expected Recovery Delay • By considering the derivative of wrtd, we obtain the following expression for the expected recovery delay: • CONJECTURE: A symmetric optimal allocation always exists for any n and T • Observe that given n, ¸, and T, the optimal allocation depends only on n and T, but not ¸; this is in contrast with the maximization offor which the optimal allocation depends on all parameters n, ¸, T, and d

17. Expected Delay: Related Work • T. Spyropoulos, K. Psounis, C. S. Raghavendra, “Spray and Wait: An efficient routing scheme for intermittently connected mobile networks,”ACM SIGCOMM Workshop on DTN 2005 • Spray a fixed number of uncoded replicas into the network, and wait for one of them to come into contact with the data collector • Showed that this fixed budget approach performs very well compared to other heuristics

18. Expected Delay: Optimal Symmetric Allocation Finding the optimal symmetric allocation… number of nonempty nodes in the symmetric allocation The expected recovery delay for the symmetric allocation is

19. Expected Delay: Optimal Symmetric Allocation • If T is an integer (i.e. ` = 1), then , which correspondsto minimal spreading (uncoded replication), is optimal • As the fractional part of T increases (i.e. ` increases), the amount of spreading (with coding) in the optimal symmetric allocation increases We are able to characterize the optimal symmetric allocation completely: • RESULT 1 • Suppose , where . • If , then is an optimal symmetricallocation. • If , then either or is an optimal symmetric allocation.

20. Expected Delay: Optimal Symmetric Allocation Proof Idea:Eliminating candidates for the optimal symmetric allocation… • We can show that an optimal m* can be found from among candidates: • For , where , the expected recovery delay is given by • Using a geometrical argument, we show that the choice ofminimizes the expected recovery delay among all ,where • To demonstrate the optimality of , i.e. , we apply the following bounds for the harmonic number Hn:

21. Simulation Study • We apply our theoretical insights to the design of a simple data dissemination and storage protocol for a delay tolerant network • Simulations allow us to capture the transient dynamics of the data dissemination process, and its interaction with the data recovery process • Our goal is to understand how different symmetric allocations perform under different circumstances: • Random waypoint mobility model vs real-world mobility traces • Low vs high mobility • Low vs high connectivity • Starting recovery immediately vs after some time

22. Simulation Study: Generalized Spray and Wait • Our protocol extends SPRAY AND WAIT by allowing nodes to storecoded packets that are each 1/ w the size of the original data object • Successful recovery occurs when the data collector accesses at leastw such packets (choosing w= 1 produces the original protocol) • Different symmetric allocations of the given budget T can be realized by changing the value of parameter w

23. Simulation Study: Random Waypoint: Key Observations • Number of wireless mobile nodesn= 100 • Plots show how the required wait time varies with the desired recovery probability PS • Each line represents a specific choice of parameter High-mobility scenario plots appear to be vertically scaled versions of the baseline scenario plots:speeding up of time Expected recovery delay performance is consistent with our analysis:minimal spreading is optimal in most plots Effect of increased connectivity appears less straightforward,e.g. phase transition not evident for recovery starting at time 0:data dissemination process impeded by greater interference? Recovery start time appears to have a limited impact on how different allocations perform relative to each other:most noticeable effect of starting recovery at time 0 is the reduced spread in performance Recovery probability performance is consistent with our analysis:phase transition in the optimal symmetric allocation is clearly discernable in most plots In the high recovery probability regime, maximal spreading (with coding) can lead to a significant reduction in the required wait time

24. Simulation Study: Mobility Traces: Key Observations • Number of wireless taxi cabs n= 100 • Plots show how the required wait time varies with the desired recovery probability PS • Each line represents a specific choice of parameter Despite nonideal conditions, many of our previous observations still apply here Plots show distinct “jumps” in wait times:reduced mobility of cabs at night In the high recovery probability regime, maximal spreading (with coding) can lead to a significant reduction in the required wait time

25. Summary: Theoretical Analysis • The optimal symmetric allocations are not the same for both objectives…(i)Maximization of Recovery Probability :For any budget T, there is a phase transition from a regime whereminimal spreading (uncoded replication) is optimal to a regime wheremaximal spreading (with coding) is optimal, as the access probability p(or the deadline d) increases(ii)Minimization of Expected Recovery Delay :With the averaging over both regimes, minimal spreading (uncoded replication) turns out to be optimal whenever the budget T is an integer; the amount of spreading in the optimal symmetric allocation increases with the fractional part of T • Performance gap between minimal spreading and maximal spreading can be quite substantial , e.g. for the required wait time in both the low and high recovery probability regimes

26. Summary: Simulation Study • Results of the simulation study are consistent with our analytical findings • Provides clear evidence that the choice of storage allocation can have a significant impact on the recovery delay performance • Shows how mobility, connectivity, and recovery start time may affect performance

27. Future Work • The simple contact model assumed here can be generalized to the case where a variable amount of data is transmitted during each contact between nodes • Allow nonuniform contact rates ¸i between the data collector and individual nodes

28. Thank you!

29. Additional Simulation Results

30. Simulation Study: Random Waypoint (Budget T= 5) • Number of wireless mobile nodesn= 100 • Plots show how the required wait time varies with the desired recovery probability PS • Each line represents a specific choice of parameter

31. Simulation Study: Random Waypoint (Budget T= 10) • Number of wireless mobile nodesn= 100 • Plots show how the required wait time varies with the desired recovery probability PS • Each line represents a specific choice of parameter

32. Simulation Study: Random Waypoint (Budget T= 20) • Number of wireless mobile nodesn= 100 • Plots show how the required wait time varies with the desired recovery probability PS • Each line represents a specific choice of parameter

33. Simulation Study: Mobility Traces (Budget T=5) • Number of wireless taxi cabs n= 100 • Plots show how the required wait time varies with the desired recovery probability PS • Each line represents a specific choice of parameter

34. Simulation Study: Mobility Traces (Budget T=10) • Number of wireless taxi cabs n= 100 • Plots show how the required wait time varies with the desired recovery probability PS • Each line represents a specific choice of parameter

35. Simulation Study: Mobility Traces (Budget T=20) • Number of wireless taxi cabs n= 100 • Plots show how the required wait time varies with the desired recovery probability PS • Each line represents a specific choice of parameter