1 / 30

Adaptive Content Management in Structured P2P Communities

Adaptive Content Management in Structured P2P Communities. Jussi Kangasharju Keith W. Ross David A. Turner. Content. Introduction Related Works Adaptive Algorithms Experimental Results Optimization Theory Conclusion. Introduction (1).

gzifa
Download Presentation

Adaptive Content Management in Structured P2P Communities

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Adaptive Content Management in Structured P2P Communities Jussi Kangasharju Keith W. Ross David A. Turner

  2. Content • Introduction • Related Works • Adaptive Algorithms • Experimental Results • Optimization Theory • Conclusion

  3. Introduction (1) • P2P file sharing is the dominant traffic type in the Internet • Two types of P2P system • Unstructured, e.g. KaZaA and Gnutella • Nodes are not organized into highly-structured overlays • Content is randomly assigned to nodes • Structured, e.g. CAN, Chord • Distributed hash table (DHT) substrates are used • Nodes are organized into highly-structured overlays • Keys are deterministically assigned to nodes

  4. Introduction (2) • Assume the system is DHT-based P2P file-sharing communities • P2P community: a collection of intermittently-connected nodes • Nodes: contribute storage, content and bandwidth to the rest of the community • A node in the community wants a file • Retrieve the file from the other nodes in the community • If the file is not found, the community retrieves the file from outside • The file will be cached and a copy will be forwarded to the requesting node

  5. Introduction (3) • Address the problem of content management in P2P file sharing communities • Propose algorithms to adaptively manage content • Minimize the average delay: the time from when a node makes a query for a file until the node receives the file in its entirety. • File transfer delay >> lookup delays • Intra-community file transfers occur at relatively fast rates as compared with file transfers into the community

  6. Introduction (4) • PROBLEM is equivalent to “adaptively managing content to maximize intra-community hit rates” • Replication: how should content be replicated to provide satisfactory hit rates • Replacement: how does a node determine to keep/evict the files • Contributions • Algorithms for dynamically replicating and replacing files in a P2P community • No a priori assumptions about file request rate or nodal up probabilities • Simple, adaptive and fully distributed • Analytical optimization theory to benchmark the adaptive replication algorithms • For complete-file replication • For the case when files are segmented and erasure codes are used

  7. Related Works • Squirrel [8] • Distributed, server-less, P2P web caching system • Built on top of the Pastry DHT substrate • Focus on the protocol design and implementation • Not address the issues of replication and file replacement • In [13] and [14], it studied optimal replication in an unstructured peer-to-peer network • Reduce random search times

  8. DHT Substrate • Node has access to the API of a DHT substrate • The substrate takes a file j as input and determines an ordered list of the up nodes • For a given value of K, (i1, i2,…,iK) • i1 is the first-place winner for file j

  9. LRU Algorithms (1) • Fundamental Problem: • “How can we adaptively add and remove replicas, in a distributed manner and as a function of evolving demand, to maximize the hit probability?” • Suppose X is a node that wants file j • Basic LRU Algorithm • X uses the substrate to determine i1, the first place winner for j • If i1 doesn’t have j, i1 retrieves j from outside the community and copies the file in storage • If i1 needs to make room for j, LRU replacement policy is used • i1 sends j to X • X does not put j in its storage

  10. LRU Algorithms (2) • Basic LRU Algorithm • A request can be a “miss” even when the file is cached in some up node within the community • Top-K LRU Algorithm • When i1 doesn’t have j, i1 determines i2,…,iK and pings each of these K-1 nodes to see if any of them have j • If so, i1 retrieves j from one of the nodes and puts a copy in its storage • Otherwise, i1 retrieves j from outside the community • The algorithm replicates content • Without any a priori knowledge of request patterns or nodal up probabilities • Fully distributed

  11. Observations • Top-K LRU algorithm is simple but its performance is significantly below the theoretical optimal • Observed that • LRU let unpopular file linger in nodes. Intuitively, if we do not store the less popular files, the popular files will have more replicas • Searching more than one node is needed to find files under the file-sharing system

  12. MFR Algorithm (1) • Most Frequently Requested (MFR) Algorithm • Has near optimal performance • Each node i maintains an estimate of aj(i), the local request rate for file j • aj(i) is the number of requests that node i has seen for file j divided by the amount of time node i has been up • Each node i stores the files with the highest aj(i) values, packing in as many files as possible

  13. MFR Algorithm (2) • MFR retrieval and replacement policy • Node i receives a request for file j, it updates aj(i) • If i doesn’t have j and MFR say it should, i retrieves j from the outside and puts j in its storage • If i needs to make room for j, MFR replacement policy is used • Searching more than one node is needed • “Ping” dynamics to influence aj(i) so that the number of replicas across all nodes become nearly optimal

  14. MFR Algorithm (3) • “Ping” the top-K winners in parallel • Retrieve the file from any node that has the file • Each “Ping” could be considered a request • Nodes update their request rate and manage their storage with MFR • However, this approach doesn’t give better performance • Sequentially request j from the top-K winners • Stop the sequential requests once j is found

  15. Experiment Results (1) • Run simulation experiments • 100 nodes and 10000 files • Request probabilities follow a Zipf distribution with parameters 0.8 and 1.2 • All file sizes are the same • Each node contributes the same amount of storage • Measure the hit performance of the algorithm

  16. Experiment Results (2) • LRU performs better than non-cooperative algorithm but significantly worse than the theoretical optimal

  17. Experiment Results (3)

  18. Experiment Results (4) • Using a K greater than 1 improves the hit probability • K beyond 5 gives insignificant improvement

  19. Experiment Results (5) • The number of replicas is changing over time, the graphs report the average values • The optimal scheme replicates the more popular files much more aggressively • The optimal scheme does not store the less popular files

  20. Experiment Results (6)

  21. Experiment Results (7) • The MFR algorithm is very close to optimal • Thus, the hit rates also are very close to optimal

  22. Analysis of MFR (1) • Analytical procedure for calculating the steady-state replica profile and hit probability for Top-K MFR for the case K=I • The results still serve as excellent approximations for when K is small • Assume • I is the number of nodes • J is the number of distinct files • pi is “up” probability of node i • Si is the amount of shared storage (in bytes) in node i • bj is the size (in bytes) of file j • qj is the request probability for file j • The request probability for the J files are known

  23. Analysis of MFR (2) • The procedure sequentially places copies of files into the nodes • Ti is the remaining unallocated storage • xij is equal to 1 if a copy of file j has been placed in node i • Initializes Ti=Si, xij=0 and vj=qj/bj • Find file j that has the largest value of vj • Sequentially examine the winning nodes for j until a node is found such that Ti>=bj and xij=0 • Set xij=1; • Set vj=vj(1-pi); • Set Ti=Ti-bj • If there is no node such that Ti>=bj and xij=0, remove file j from further consideration • Return to Step 1 if all files have not been removed from consideration

  24. Optimization Theory (1) • Analytical theory for optimal replication in P2P communities • Complete-File Replication (No Fragmentation) • File are segmented and erasure coded • No Fragmentation Subject to

  25. Optimization Theory (2) • The problem is NP-complete • Consider a special case • pi=p • nj=number of replicas for file j Subject to • The problem can be efficiently solved by dynamic programming

  26. Optimization Theory (3) • Upper bound on the performance of adaptive management algorithms for the case of erasures • File j is made up of Rj erasures • Any Mj of the Rj erasures are needed to reconstruct the file • Size of each erasure is bj/Mj • Assume homogenous “up” probabilities, pi=p • rth erasure of file j as erasure jr, r=1,…,Rj • njr is the number of erasures jr stored in the community of nodes

  27. Optimization Theory (4) 0-1 random variable which is 1 if any of the njr erasures jr is in some up node A hit for a request for file j if any Mj of the Rj erasures for file j are available

  28. Optimization Theory (5) • Theorem 2.2 of Boland et al [24], the function is Shur concave Subject to

  29. Optimization Theory (6) • Special case: No erasures • Rj=Mj=1 where • qj/bj plays a key role in influencing the number of replicas • It is upper bound on the true optimal because it is the optimal over continuous variables rather than integer variables

  30. Conclusion • Claim that structured/DHT-designs will potentially improve search and download performance • Proposed the Top-K MFR algorithm, which is simple, fully distributed, adaptive, near-optimal • Introduce an optimization methodology for bench-marking the performance of adaptive algorithms • The methodology can also be applied to designs that use erasures

More Related