Towards Network Triangle Inequality Violation Aware Distributed Systems

1. Towards Network Triangle Inequality Violation Aware Distributed Systems

2. A C B AB + AC > BC > |AB – AC| Introduction • Many distributed systems rely on the neighbor selection mechanisms to construct overlay structures with good network performance. • Neighbor selection mechanisms often assume triangle inequality holds for the Internet delays in order to infer delays without measuring them.

3. A: 128.42.129.40 65 ms 330ms 520 ms B:76.194.27.220 C: 219.243.200.93 AB + AC < BC ! Network Triangle Inequality Violation • Real Internet delays violate triangle inequality in many cases. • Neighbor selection mechanisms make mistakes because of Triangle Inequality Violation (TIV).

4. What we do NOT know about TIV • Characteristics of TIVs for the Internet delays? • How do TIVs impact neighbor selection mechanisms? • Ways to reduce the impacts of TIVs?

5. Outline • Analyzing TIV characteristics • Understanding the impact of TIVs on neighbor selection mechanisms • TIV alert mechanism

6. Data Sets • DS2 data • RTTs among 4000 DNS servers • One DNS server per domain • Measured by the King tool • http://www.cs.rice.edu/~bozhang/ds2/ • Other data: • p2psim data, Meridian data, PlanetLab data

7. C TIV Severity B A 1- fraction of TIV Triangulation ratio of ABC = AB AC+BC TIV Severity Metric TIV severity: Sum of the triangulation ratios for all the TIVs (normalized by the network size)

8. 0 TIV severity 255 C1-C3 C1 C1-C2 C2-C3 C3 C2 C1 C2 C3 - Picture from PlanetLab.org Clustering Property • Can we predict TIV severity by clustering property? • Crossing cluster edges tend to cause more TIVs, but it is hard to predict TIV severity of an edge by this coarse-grain trend.

9. TIV Severity vs. Delay • Can we predict TIV severity by delay length? • Long edges tend to cause more TIVs. • Irregular relation between TIV severity and delay. • It is hard to predict the TIV severity of an edge just by its delay length.

10. nearest pair (average RTT: 6.08 ms) A B An Bn nearest-pair-edge random pair (average RTT: 156 ms) A B Ar Br random-pair-edge Proximity Property • Can we predict TIV severity by proximity property? • Close-by nodes do not necessarily have similar TIV severity characteristic.

11. Outline • Analyzing TIV characteristics • TIV is a complex phenomenon in the Internet, and it is hard to predict TIV by naïve heuristics. • Understanding the impact of TIVs on neighbor selection mechanisms • TIV alert mechanism

12. 20 ms 20 ms B 20 ms A d T (1-)d (1+)d Y (20, 25.3) 20ms 20ms (10,8) (30,8) 20ms X The Impact of TIVs on Neighbor Selection • Representative neighbor selection mechanisms Vivaldi: metric embedding Meridian: online probing • To reduce overhead: • Termination factor  • Limit the number of ring members

13. C 100ms 5ms A 5 ms B The Impacts of TIVs on Vivaldi • High error • Median absolute error: 20 ms for all the edges in the data set. • Coordinates oscillation • Median oscillation speed: 1.6ms/step • Large oscillation range: 170ms for a 20 ms edge!

14. 3ms N 6.5ms 4ms 6ms 25ms 2ms T 12ms B 11ms N A 6ms 18ms =0.5 The Impacts of TIVs on Meridian Misplacement: Given any two nodes A and T with delay d, because of TIV, the ring members within d delay of node A are not placed in the range (1-)d to (1+)d of node T. • Misplacement in ring construction happens on 12% of the ring members of all the nodes in the data set. • Meridian fails to find the nearest neighbor for 13% of the experiments even under idealized setting.

15. Outline • Analyzing TIV characteristics • Understanding the impact of TIVs on neighbor selection mechanisms • Vivaldi yields high error and rapid coordinate oscillation. • Meridian makes mistakes in ring construction and fails to find nearest neighbor even under idealized settings. • TIV alert mechanism

16. B A TIV Alert Mechanism • The edges causing severe TIVs are highly likely to be shrunk in when embedding them into a metric space. • Using the prediction ratio in metric embedding as a heuristic indicator of TIV severity.

17. TIV Alert Mechanism (cont.) Worst 20%: The top 20% edges with highest TIV severity • Identify edges causing severe TIVs with reasonable accuracy and recall rate. • Easy to get prediction ratios in Vivaldi and Meridian.

18. Experiment Methodology • Neighbor selection experiment methodology • Vivaldi: 32 random neighbor, 5D Euclidean space • Meridian: default setting (s = 2, =0.5, =1), no limitation on number of ring members. • Percentage penalty: • Aggregated over 5 runs

19. A Using TIV Alert in Vivaldi • Dynamic neighbor Vivaldi: • Identify the neighbors causing severe TIVs by prediction ratios and replace them by random neighbors • At each iteration, randomly sample another 32 neighbors, and from the 64 candidates, we remove the half with lowest prediction ratios.

20. A T Using TIV Alert in Meridian • Identify the edges causing severe TIVs by prediction ratios and fix the mistakes in ring construction and online query.

21. Conclusion • Analyzed the characteristics of TIVs based on the Internet delay measurement, and highlight the irregular behavior of TIVs. • Investigated the impacts of TIVs on two representative neighbor selection mechanisms. • Proposed a TIV alert mechanism that can identify edges causing severe TIVs. • TIV alert mechanism can provide TIV awareness in a variety of distributed systems.