Coordination and Agreement in Distributed Systems

1. Chapter 12 Coordination and agreement

2. outline • Introduction • Distributed mutual exclusion • Elections • Multicast communication • Consensus and related problems

3. introduction The aim of this chapter is to introduce some topics and algorithms related to the issue of how process coordinate their actions and agree on shared values in DS, despite failures

4. introduction Failure assumptions and failure detection: The real situation of the Internet • Network partition • Asymmetric routing • Intransitive connectivity

5. introduction Failure assumptions and failure detection (con.): Channel assumption: reliable Failed link will be repaired or circumvented Process assumption • crash failure without mention, otherwise arbitrary failure • Correct process: no crash failure and arbitrary failure

6. introduction Failure assumptions and failure detection (con.): Failure detectors: Is a service that processes queries about whether a particular process has failed, and it could be one of two types: • Unreliable failure detector • Reliable failure detector

7. introduction Failure assumptions and failure detection (con.): Unreliable failure detector: • Not accurate • May produce one of two values when given the identity of a process: Unsuspected or Suspected • A result of Unsuspected signifies that the detector has recently received evidence suggesting that the process has not failed • A result of Suspected signifies that the failure detector has some indication that the process may have failed

8. introduction Failure assumptions and failure detection (con.): Reliable failure detector: • Accurate • It answers processes’ queries with either a response of Unsuspected or Failed. • A result of Failed means that the detector has determined that the process has crashed.

9. Distributed mutual exclusion Mutual exclusion is required to prevent interference and ensure consistency when accessing the resources. Algorithms of mutual exclusion: Essential requirements for mutual exclusion • Safety • At most one process may execute in the critical section at a time • Liveness • Requests to enter and exit the critical section(CS) eventually succeed • Free from deadlock and starvation • Ordering • If one request to enter the CS happened-before another, then entry to the CS is granted in that order

10. Distributed mutual exclusion Algorithms of mutual exclusion (con.): Evaluate the performance of the algorithms • Bandwidth consumed • The number of messages sent in each entry and exit operation • Client Delay • Incurred by a process at each entry and exit operations • Throughput • Synchronization delay: delay between one process exiting the critical section and the next process entering it

11. Distributed mutual exclusion Algorithms of mutual exclusion (con.): Mutual exclusion algorithms: • The central server algorithm • A ring-based algorithm • An algorithm using multicast and logical clocks • Maekawa’s voting algorithm

12. Distributed mutual exclusion The central server algorithm The simplest way to achieve mutual exclusion is to establish a server that grants permission to enter the critical section. A process requests entry and waits for a reply. Conceptually, the reply is a token that grants permission. If no other process has the token, it can be granted. Otherwise the process must wait until the token is available. Other processes may have made prior requests for the token, in which case the most recent process must wait until previous requests are met.

13. Distributed mutual exclusion The central server algorithm • Meet safety and liveness, but not ordering • Performance • Bandwidth consumption • Enter(): A request message, a grant message • Exit(): a release message • Client Delay (no waiting processes) • Request message + grant message • Synchronization delay • A release message + a grant message

14. Server Queue of requests 4 2 3. Grant token 1. Request token 2. Release p p token 4 1 p p 3 2 Figure 12.2Server managing a mutual exclusion token for a set of processes

15. Distributed mutual exclusion A ring-based algorithm A simple way to arrange mutual exclusion is to arrange requests in a logical ring. Each process has a link to the next process. A token is passed around the ring. If the process receiving the ring needs access to the critical section, it enters the section, otherwise it passes the token to the next process. Figure 12.3 shows a ring-based algorithm graphically. It is self evident that safety, liveness and ordering requirements are met by this algorithm. Network bandwidth is consumed continuously, even when there are no requests.

16. Distributed mutual exclusion A ring-based algorithm • Meet safety and liveness, but not ordering • Performance • Bandwidth consumed • Continuously consume network bandwidth • Client Delay • Min: 0 message, when it has just received the token • Max: N messages, when it has just passed on the token • Synchronization delay • Min: 1 message, when processes enter CS one by one • Max: N message, when a process enter CS continuously and no other process enter CS

17. Figure 12.3A ring of processes transferring a mutual exclusion token

18. Distributed mutual exclusion An algorithm using multicast and logical clocks • A process enters CS if all other processes promise • Multicast + reply • Concurrence control • Lamportlogical clock: avoid dead-lock

19. 41 p 41 3 p Reply 1 34 Reply Reply 34 41 34 p 2 Figure 12.5Multicast synchronization

20. Distributed mutual exclusion An algorithm using multicast and logical clocks • Performance • Bandwidth • Enter(): N-1 multicast message, N-1 reply • Client delay: round-trip time • Synchronization delay: one message

21. Distributed mutual exclusion Maekawa’s voting algorithm A process enter CS when part of other processes promise • A voting set Vi with each process pi • Vi {p1, p2, …, pn} • pi Vi • Vi Vj  • | Vi |= K • Each process pj is contained in M of the voting sets Vi

22. Distributed mutual exclusion Maekawa’s voting algorithm • Performance • Bandwidth utilization • Better than 2(N-1) if N>4 • Client delay: a round trip time • Same as multicast algorithm • Synchronization: a round trip time • Worse than multicast algorithm’s

23. Elections An algorithm to choose a process to play a role is called an election algorithm. For example, a group of peers may select one of themselves to act as server for a mutual exclusion algorithm. A process that initiates a run of an election algorithm calls the election. Multiple elections could be called at the same time. A process that is engaged in an election algorithm at a particular time is a participant. At other times or when it is not engaged it is a non-participant

24. Elections Essential requirements for Elections • Safety A participant process pi has electedi = or electedi = P, where P is chosen as the non-crashed process at the end of the run with the largest identifier • Liveness • All processes piparticipate and eventually set electedi <>  or crash Election algorithms: • A ring-based Election algorithm • The bully algorithm

25. Multicast communication Challenges • Efficiency • Bandwidth utilization • Total transmission time • E.g. delivery tree in IP multicast • Delivery guarantees • Reliability • Ordering • Group management • Processes joining and leaving group at arbitrary times

26. Multicast communication Basic multicast • A correct process will eventually deliver the message, as long as the multicaster does not crash • Primitives: B-multicast / B-deliver • Different to IP multicast in the aspect of reliability Implementation scheme is based on reliable one-to-one send operation • To B-multicast(g, m): for each process pg, send(p, m); • On receive(m) at p: B-deliver(m) at p

27. Multicast communication Reliable multicast semantics • Integrity • A correct process delivers a message m at most once • Validity • If a correct process multicasts message m then it will eventually deliver m • Agreement • If a correct process delivers message m, then all other correct processes in group(m) will eventually deliver m • Atomicity: all or nothing • Different from basic multicast: it is not met if the multicaster fails when it is multicasting

28. Multicast communication Ordered multicast • FIFO ordering • If a correct process issues multicast(g,m) and then multicast(g,m`), then every correct process that delivers m` will deliver m before m` • Causal ordering • If multicast(g,m)multicast(g,m`), then any correct process that delivers m` will deliver m before m` • Total ordering • If a correct process delivers message m before it delivers m`, then any other correct process that delivers m` will deliver m before m`

29. Consensus and related problems • Make agreement in a distributed manner • Mutual exclusion: who can enter the critical region • Totally ordered multicast: the order of message delivery • Byzantine generals: attack or retreat? • Consensus problem • Agree on a value after one or more of the processes has proposed what the value should be • Consensus, byzantine general problem, interactive consistency, totally ordered multicast • Failure model • Process crash failure, process byzantine (arbitrary) failure

30. Consensus and related problems Requirements of a consensus algorithm • Termination Eventually each correct process sets its decision variable • Agreement If pi and pj are correct and have entered the decided state, then di = dj (i, j = 1,2, …, N) • Integrity If the correct processes all proposed the same value, then any correct process in the decided state has chosen that value

31. Consensus and related problems Consensus Process • Each process begins in an undecided state • A value is proposed from a set of values • Processes communicate with each other • The state of a decision variable is set Figure 12.17 shows a three processes engaged in a consensus algorithm. Two processes propose “proceed.” One proposes “abort,” but then crashes. The two remaining processes decide proceed.

32. Figure 12.17Consensus for three processes