Sharding and the Isis 2 DHT

1. Sharding and the Isis2 DHT Did you understand how the Isis2 distributed hash table works?

2. DHT Basics • Suppose we have a group containing 500 members, and decide to store data in shards of size 3. • This isn’t going to work: the shard size needs to divide evenly into the group size. • In Isis2 some shards can be a little too big, or too small. The value you specify is more of a target

3. DHT Basics • Suppose we have a group containing 500 members, and decide to store data in shards of size 3. • This isn’t going to work: the shard size needs to divide evenly into the group size. • In Isis2 some shards can be a little too big, or too small. The value you specify is more of a target You do get to specify a minimum size for the group as a whole, below which Isis2 temporarily disables the DHT functionality

4. DHT Basics • With a DHT storing data • Both Put and Get operations have costs roughly proportional to the time to do a remote procedure call: one RTT to each participant, issued in parallel • Like other DHTs, the Isis2 has costs proportional to the log of the group size. This relates to needing to route requests in a binary search manner: half way, then a quarter way, etc.

5. DHT Basics • With a DHT storing data • Both Put and Get operations have costs roughly proportional to the time to do a remote procedure call: one RTT to each participant, issued in parallel • Like other DHTs, the Isis2 has costs proportional to the log of the group size. This relates to needing to route requests in a binary search manner: half way, then a quarter way, etc. Isis2 offers a so-called “1-hop” DHT. No indirect routing occurs and none of these log(N) delays arise in this approach. Indirect routing is perceived as a problem with many other DHTs, like Chorus or Pastry, but doesn’t apply to the Isis2 version

6. DHT Basics • Within a group, shard membership • Counts off by rank: 0… NS-1, 0… NS-1, etc (where NS is the number of shards: the group size divided by the target shard size) • The first shard will be on left and includes members 0..S-1, where S is the shard size. The second shard will include members S..(2*S-1) • Membership is pretty random: you need to hash the address of the member onto a ring and then the shard is that member and the next S-1 along the edge

7. DHT Basics We use this scheme because it has low cost when a failure occurs. • Within a group, shard membership • Counts off by rank: 0… NS-1, 0… NS-1, etc (where NS is the number of shards: the group size divided by the target shard size) • The first shard will be on left and includes members 0..S-1, where S is the shard size. The second shard will include members S..(2*S-1) • Membership is pretty random: you need to hash the address of the member onto a ring and then the shard is that member and the next S-1 along the edge We ruled this approach out because “churn” after a failure is too expensive: large numbers of members might need to be reinitialized. This is how Chord and Pastry work, but not the way that the Isis2 DHT works.

8. When inserting an item… The key is first hashed by computing the hashcode modulo the number of shards. This gives the desired shard number. Then… • A multicast is sent to just the shard members. • The value is multicast to the entire group and those members that are in the matching shard retain it. Others ignore the multicast.

9. When inserting an item… The key is first hashed by computing the hashcode modulo the number of shards. This gives the desired shard number. Then… • A multicast is sent to just the shard members. • The value is multicast to the entire group and those members that are in the matching shard retain it. Others ignore the multicast.

10. When inserting a list of items The list of keys is first hashed by computing the hashcode item by item modulo the number of shards. This yields a list of shards. • Now each item is inserted in a separate parallel action, independently. • Now all items are inserted using a single multicast that goes to only the full set of shard members in the list

11. When inserting a list of items The list of keys is first hashed by computing the hashcode item by item modulo the number of shards. This yields a list of shards. • Now each item is inserted in a separate parallel action, independently. • Now all items are inserted using a single multicast that goes to only the full set of shard members in the list By using a single multicast, we guarantee all-or-nothing behavior. But this is a special protocol that only reaches the subset of members that are in the target shards. Moreover, it is an exceptionally fast protocol, like a parallel unicast.

12. Consistency To obtain strong consistency guarantees from the Isis2 DHT • The insert and get should be done using the DHTOrderedPut and DHTOrderedGet methods • An OrderedSend must be done targetting the entire group • The Isis2 system doesn’t offer consistency for DHT operations

13. Consistency To obtain strong consistency guarantees from the Isis2 DHT • The insert and get should be done using the DHTOrderedPut and DHTOrderedGet methods • An OrderedSend must be done targetting the entire group • The Isis2 system doesn’t offer consistency for DHT operations

14. DHT Fault Tolerance • True or False. The Isis2 DHT automatically masks faults, retrieving data redundantly in DHTGet and deduplicating to return just one key-value pair for any given key.

15. DHT Fault Tolerance • True. If DHTGet or DHTOrderedGet lack a response from some participant, the Isis2 DHT automatically retries. An exception is thrown if an entire shard fails, since that would prevent the system from getting even a single response for keys mapped to that shard. When using OrderedGet, the entire OrderedGet will be reissued if necessary, to ensure that the responses are collected along a consistent cut.