Totally Ordered Broadcast

1. Totally Ordered Broadcast Algorithms for Distributed System Semester Project (Nam Thoai)

2. Totally Ordered Broadcast • Broadcast • Broadcast Service Qualities: • ordering : single-source FIFO, totally ordered, causally ordered • reliability • Implementing a broadcast service • Algorithm for Totally Ordered Broadcast • Modified Algorithm for Totally Ordered Broadcast

3. M Broadcast • Point-to-Point links: provides One-to-One communication – one processor sends a message on an incident link to a single other processor • Broadcast(multicast): provides one-to-all (one-to-many) communication – a processor sends a message to all processors • Broadcast (multicast) can be simulated by sending a number of point-to-point messages M

4. The Basic Service Specification (broadcast) • bc-sendi(m,qos): An input event of processor pi, which sends a message m to all processors, containing an indication of the sender; qos is a parameter describing the quality of service required. • bc-recvi(m,j,qos): An output event in which processor pi receives the message m previously broadcast by pj; qos, as above, describes the quality of service provided. • Basic broadcast service: each bc-recvi(m,j,qos) event is mapped to an earlier bc-sendi(m,qos) event, every message received was previously sent (Integrity), and every message that is sent is received once (Liveness) and only once (No Duplicates) at every processor.

5. Broadcast Service Qualities Broadcast properties can be organized along two axes: • Ordering: Do processors see all messages in the same order or just see the messages from a single processor in the order they were sent ? Does the order in which messages are received by the broadcast service preserve the happens-before relation ? • Reliability: Do all processors see the same set of messages even if failures occur in the underlying system? Do all processors see all the messages broadcast by a nonfaulty processor?

6. Ordering • Single-Source FIFO: For all messages m1 and m2 and all processors pi and pj, if pi sends m1 before sends m2, then m2 is not received at pj before m1 is. • Totally Ordered: For all messages m1 and m2 and all processors pi and pj, if m1 is received at pi before m2 is, then m2 is not received at pj before m1 is. • Causally Ordered: For all messages m1 and m2 and every processor pi, if m1happens before m2, then m2 is not received at pi before m1 is. Note: message m1 is said to happen before message m2 if either: • the bc-recv event for m1 happens before the bc-send event for m2, or • m1 and m2 are sent by the same processor and m1 is sent before m2.

7. Ordering • Causally Ordered implies Single-Source FIFO a a b a,b b,a b b,a b,a (1) causally ordered (totally ordered) (2) totally ordered (causally ordered, single-source ordered) a b,a b a,b (1) single-source FIFO (totally ordered, causally ordered)

8. Reliability In the presence of faulty processors, the Liveness property needs to be weakened. There are at most f faulty processors and the mapping  from bc-recv(m) events to bc-send(m) events. • Integrity: For each processor pi, 0  i  n-1, the restriction of  to bc-recvi events is well-defined. That is, every message received was previously sent - no message is received ‘out of thin air’. • No Duplicates: For each processor pi, 0  i  n-1, the restriction of  to bc-recvi events is one-to-one. That is, no message is received more than once at any single processor.

9. Pn-1 m1 m1 m1 . . . P0 P1 Pn-1-f Reliability • Nonfaulty Liveness: When restricted to bc-send and bc-recv events at nonfaulty processors,  is onto. That is, all messages broadcast by a nonfaulty processor are eventually received by all nonfaulty processors. • Faulty Liveness: If one nonfaulty processor has a bc-recv event that maps to a particular bc-send event of a faulty processor, then every nonfaulty processor has a bc-recv event that maps to the same bc-send event. That is, every messages sent by a faulty processor is either received by all nonfaulty processors or by none of them. Atomic broadcast is a reliable broadcast with total ordering; atomic broadcast is also called total broadcast. FIFO atomic broadcast is an atomic broadcast with single-source FIFO ordering. Causal atomic broadcast is an atomic broadcast with preserves causality. Pn-1 m1 m1 m1 . . . P0 P1 Pn-1-f

10. Implementing a Broadcast Service • Basic Broadcast Service Using the underlying point-to-point message system to send a copy of m to all processors. • Single-Source FIFO Ordering message(data, N) N N = N+1; bc-send(data, N) Message(data, n) is received if all message(i) (0  i < n) are received

11. b(2) a(1) c(3) a(1) a(1) a(1) a(1) b(2) a(1) b(2) b(2) b(2) b(2) b(2) c(3) c(3) c(3) c(3) a(1) c(3) c(3) Totally Ordered Broadcast • An Asymmetric Algorithm • Processor pi sends m using the basic broadcast service to a unique central site at processor pc. • Processor pc assigns a sequence number to each message and then sends it to all processors using the basic broadcast service. • Message(k) is received if all messages(i) (0  i < k) are received c a b   

12. Totally Ordered Broadcast • A Symmetric Algorithm • Based on assigning timestamps to messages. • Assume that the underlying communication system provides single-source FIFO broadcast. • Each processor maintains an increasing counter (timestamp), which is tagged on messages. • Each processor also maintains a vector with estimates of the timestamps of all other processes. Processor pi updates its entry for pj using the tags on messages received from pj and using special ‘timestamp update’ message sent by pj.

13. Totally Ordered Broadcast (Code for pi, 0  i  n-1) Initially ts[j]=0, 0  j  n-1, and pending is empty 1: when bc-sendi(m,to) occurs: // quality of service to means totally ordered 2: ts[i] := ts[i]+1 3: add m, ts[i], i to pending 4: enable bc-sendi(m, ts[i], ssf) // quality of service to means single-source FIFO 5: when bc-recvi(m, T, j, ssf), ji, occurs: // j indicates sender; ignore messages from self 6: ts[j] := T 7: add m, T, j to pending 8: if T > ts[i] then 9: ts[i] := T 10: enable bc-sendi(ts-up, T, ssf) // bcast timestamp update message 11: when bc-recvi(ts-up, T , j, ssf), ji, occurs: // ignore message from self 12: ts[j] := T 13: enable bc-recvi(m, j, to) when 14: m, T, j is the entry in pending with the smallest (T, j) 15: T  ts[k] for all k 16: result: remove m, T, j from pending

14. Totally Ordered Broadcast (implementation) (Code for pi, 0  i  n-1) Initially ts[j]=0, tn[j]=0 (0  j  n-1), N=0, and pending is empty 1: when bc-sendi(m,to) occurs: // quality of service to means totally ordered 2: ts[i] := ts[i]+1 N = N+1 3: add m, ts[i], N, i to pending 4: enable bc-sendi(m, ts[i], N, basic) // quality of service to means basic 5: when bc-recvi(m, T, N, j, basic), ji, occurs: // j indicates sender; ignore messages from self 6: ts[j] := T 7: add m, T, N, j to pending 8: if T > ts[i] then 9: ts[i] := T 10: enable bc-sendi(ts-up, T, basic) // bcast timestamp update message 11: when bc-recvi(ts-up, T , j, basic), ji, occurs: // ignore message from self 12: ts[j] := T 13: enable bc-recvi(m, j, to) when 14: m, T, N, j is the entry in pending with the smallest (T, j) 15: (T  ts[k] for all k) and (N = tn[j]+1) 16: result: remove m, T, j from pending, and tn[j] := tn[j] + 1