Asynchronous Consensus

1. Asynchronous Consensus Ken Birman

2. Outline of talk • Reminder about models • Asynchronous consensus: Impossibility result • Solution to the problem • With an “oracle” that detects failures • Without oracles, using timeout • Big issues? Revisit from Byzantine agreement • Is this model realistic? In what ways is it “legitimate”? • Should we focus on impossibility, or “possibility”? • Asynchronous consensus in real world systems

3. Distributed Computing Models • Recall that we had two models • To reason about networks and applications we need to be precise about the setting in which our protocols run • But “real world” networks are very complex • They can drop packets, or reorder them • Intruders might be able to intercept and modify data • Timing is totally unpredictable

4. Asynchronous network model • Asynchronous because we lack clocks: • Network can arbitrarily delay a message • But we assume that messages are sequenced and retransmitted (arbitrary numbers of times), so they eventually get through. • “Free” to say: lossless, ordered • No value to assumptions about process speed • Failures in asynchronous model? • Usually, limited to process “crash” faults • If detectable, we call this “fail-stop” – but how to detect?

5. An asynchronous network Not causal!

6. An asynchronous network Time shrinks…

7. An asynchronous network Time shrinks… Time stretches…

8. Justification? • If we can do something in the asynchronous model, we can probably do it even better in a real network • Clocks, a-priori knowledge can only help… • But today we will focus on an impossibility result • By definition, impossibility in this model means “xxx can’t always be done”

9. Paradigms • Fundamental problems, the solution of which yields general insight into a broad class of questions • In distributed systems: • Agreement (on value proposed by a leader) • Consensus (everyone proposes a value… pick one) • Electing a leader • Atomic broadcast/multicast (send a message, reliably, to everyone who isn’t faulty, such that concurrent messages are delivered in the same order everywhere) • Deadlock detection, clock or process synchronization, taking a snapshot (“picture”) of the system state….

10. Consensus problem • Models distributed agreement • Comes in various forms (with subtle differences in the associated results)! • With a leader: leader gives an order, like “attack”, and non-faulty participants either attack or do nothing, despite some limited number of failures: Byzantine Agreement • Without a leader: participants have an initial vote; protocol runs and eventually all non-faulty participants chose the same outcome, and it is one of the initial votes (typically, 0 or 1): Fault-tolerant Consensus

11. Consensus problem P0 Q0 R1 P1 Q1 R1

12. Fault-tolerance • Goal: an algorithm tolerant of one failure • Failure: process crashes but this is not detectable • So the algorithm must work both in the face of arbitrary message delay caused by the network, and in the event of a single failure

13. If some process stays up… • Suppose we knew that P won’t fail • Then P could simply broadcast it’s input • All would “decide” upon this value • Solves the problem

14. If one process stays up • Indeed, suppose that P stays up only long enough to send one message • But there is only one failure • And we knew that P would “lead” • Then we can relay P’s message, using an all-to-all broadcast

15. Algorithm • P: broadcast my input • Q  P: on receiving P’s message for first time, broadcast a copy • Tolerates anything except failure of P in the first step, but we need to agree upon “P” before starting (ie P is the least ranked process, using alphabetic ranking)

16. Another algorithm • All processes start by broadcasting own value to all other processes • If we know that there is always exactly one failure, could wait until n-1 messages received, then using any deterministic rule • But doesn’t work if sometimes we have one failure, sometimes none

17. FLP result • Considers general case • Assumes an algorithm that can decide with zero or one failures • Proves that this algorithm can be prevented from reaching decision, indefinitely

18. Basic idea • Think of system state as a “configuration” • Configuration is v-valent if decision to pick v has become inevitable: all runs lead to v • If not 0-valent or 1-valent, configuration is bivalent • Initial configuration includes • At least one 0-valent: {0,0,0….0} • At least one 1-valent: {1,1,1…..1} • At least one bivalent: {0,0,…1,1}

19. Basic idea 0-valentconfigurations bi-valentconfigurations 1-valentconfigurations

20. Transitions between configurations • Configuration is a set of processes and messages • Applying a message to a process changes its state, hence it moves us to a new configuration • Because the system is asynchronous, can’t predict which of a set of concurrent messages will be delivered “next” • But because processes only communicate by messages, this is unimportant

21. Basic Lemma • Suppose that from some configuration C, the schedules 1, 2 lead to configurations C1 and C2, respectively. • If the sets of processes taking actions in 1 and 2, respectively, are disjoint than 2 can be applied to C1 and 1 to C2, and both lead to the same configuration C3

22. Basic Lemma C 2 1 C1 C2 2 1 C3

23. Main result • No consensus protocol is totally correct in spite of one fault • Note: Uses total in formal sense (guarantee of termination)

24. Basic FLP theorem • Suppose we are in a bivalent configuration now and later will enter a univalent configuration • We can draw a form of frontier, such that a single message to a single process triggers the transition from bivalent to univalent

25. Basic FLP theorem C e’ e bivalent D0 C1 univalent e’ e D1

26. Single step decides • They prove that any run that goes from a bivalent state to a univalent state has a single decision step, e • They show that it is always possible to schedule events so as to block such steps • Eventually, e can be scheduled but in a state where it no longer triggers a decision

27. Basic FLP theorem • They show that we can delay this “magic message” and cause the system to take at least one step, remaining in a new bivalent configuration • Uses the diamond-relation seen earlier • But this implies that in a bivalent state there are runs of indefinite length that remain bivalent • Proves the impossibility of fault-tolerant consensus

28. Notes on FLP • No failures actually occur in this run, just delayed messages • Result is purely abstract. What does it “mean”? • Says nothing about how probable this adversarial run might be, only that at least one such run exists

29. FLP intuition • Suppose that we start a system up with n processes • Run for a while… close to picking value associated with process “p” • Someone will do this for the first time, presumably on receiving some message from q • If we delay that message, and yet our protocol is “fault-tolerant”, it will somehow reconfigure • Now allow the delayed message to get through but delay some other message

30. Key insight • FLP is about forcing a system to attempt a form of reconfiguration • This takes time • Each “unfortunate” suspected failure causes such a reconfiguration

31. FLP and our first algorithm • P is the leader and is supposed to send its input to Q • Q “times out” and • Tells everyone that P has apparently failed • Then can disseminate its own value • If P wakes up, we re-admit it to the system but it is no longer considered least ranked • One can make such algorithms work… • But they can be attacked by delaying first P, then Q, then R, etc

32. FLP in the real world • Real systems are subject to this impossibility result • But in fact often are subject to even more severe limitations, such as inability to tolerate network partition failures • Also, asynchronous consensus may be too slow for our taste • And FLP attack is not probable in a real system • Requires a very smart adversary!

33. Chandra/Toueg • Showed that FLP applies to many problems, not just consensus • In particular, they show that FLP applies to group membership, reliable multicast • So these practical problems are impossible in asynchronous systems, in formal sense • But they also look at the weakest condition under which consensus can be solved

34. Chandra/Toueg Idea • Separate problem into • The consensus algorithm itself • A “failure detector:” a form of oracle that announces suspected failure • But it can change its mind • Question: what is the weakest oracle for which consensus is always solvable?

35. Sample properties • Completeness: detection of every crash • Strong completeness: Eventually, every process that crashes is permanently suspected by every correct process • Weak completeness: Eventually, every process that crashes is permanently suspected by some correct process

36. Sample properties • Accuracy: does it make mistakes? • Strong accuracy: No process is suspected before it crashes. • Weak accuracy: Some correct process is never suspected • Eventual strong accuracy: there is a time after which correct processes are not suspected by any correct process • Eventual weak accuracy: there is a time after which some correct process is not suspected by any correct process

37. A sampling of failure detectors

38. Perfect Detector? • Named Perfect, written P • Strong completeness and strong accuracy • Immediately detects all failures • Never makes mistakes

39. Example of a failure detector • The detector they call W: “eventually weak” • More commonly: W: “diamond-W” • Defined by two properties: • There is a time after which every process that crashes is suspected by some correct process • There is a time after which some correct process is never suspected by any correct process • Think: “we can eventually agree upon a leader.” If it crashes, “we eventually, accurately detect the crash”

40. W: Weakest failure detector • They show that W is the weakest failure detector for which consensus is guaranteed to be achieved • Algorithm is pretty simple • Rotate a token around a ring of processes • Decision can occur once token makes it around once without a change in failure-suspicion status for any process • Subsequently, as token is passed, each recipient learns the decision outcome

41. Rotating a token versus 2-phase commit Propose v… ack… Decide v “phase”

42. Rotating a token versus 2-phase commit • Their protocol is basically a 2-phase commit • But with n processes, 2PC requires 2(n-1) messages per phase, 3(n-1) total • Passing a token only requires n messages per phase, for 2n total (when nothing fails) • Tolerates f <  n/2  failures

43. Set of problems solvable in: Clock synchronization TRBnon-blocking atomic commitconsensusatomic broadcast reliablebroadcast Synchronous systems Asynchronous using P Asynchronous using W Asynchronous TRB: Byzantine Generals with only crash failures

44. Building systems with W • Unfortunately, this failure detector is not implementable • Using timeouts we can make mistakes at arbitrary times • But with long enough timeouts, could produce a close approximation to W

45. Would we want to? • Question: are we solving the right problem? • Pros and cons of asynchronous consensus • Think about an air traffic control application • Find one problem for which asynchronous consensus is a good match • Find one problem for which the match is poor

46. French ATC system (simplified) Onboard Radar X.500 Directory Controllers Air Traffic Database (flight plans, etc)

47. Potential applications • Maintaining replicated state within console clusters • Distributing radar data to participants • Distributing data over wide-area links within large geographic scale • Management and control (administration) of the overall system • Distributing security keys to prevent unauthorized action • Agreement when flight control handoffs occur

48. Broad conclusions? • The protocol seems unsuitable for high availability applications • If the core of the system must make progress, the agreement property itself is too strong • If a process becomes unresponsive might not want to wait for it to recover • Also, since we can’t implement any of these failure detectors, the whole issue is abstract… • Hence real systems don’t try to solve consensus as defined and used in these kinds of protocols!

49. Value of FLP/Consensus • A clear and elegant problem statement • Highlights limitations • Perhaps with clocks we can overcome them • More likely, we need a different notion of failure • “Crash failure” is too narrow, “unreachable” also treated as failure in many real systems • Caused much debate about real systems

50. Nature of debate • We’ll see many practical systems soon • Do they • Evade FLP in some way? • Are they subject to FLP? If so, what problem do they “solve”, given that consensus (and most problems reduce to consensus) is impossible to solve? • Or are they subject to even more stringent limitations? • Is fault-tolerant consensus even an issue in real systems?