Replication techniques Primary-backup, RSM, Paxos

1. Replication techniquesPrimary-backup, RSM, Paxos Jinyang Li

2. Fault tolerance => replication • How to recover a single node from power failure? • Wait for reboot • Data is durable, but service is unavailable temporarily • Use multiple nodes to provide service • Another node takes over to provide service

3. Replicated state machine (RSM) • RSM is a general replication method • Lab 6: apply RSM to lock service • RSM Rules: • All replicas start in the same initial state • Every replica apply operations in the same order • All operations must be deterministic • All replicas end up in the same state

4. RSM opA opB opB opA • How to maintain a single order in the face of concurrent client requests? opA opA opB opB

5. opB opA RSM: primary/backup opA opB opA opB • Primary/backup: ensure a single order of ops: • Primary orders operations • Backups execute operations in order primary backup

6. Case study: Hypervisor [Bressoud and Schneider] • Goal: fault tolerant computing • Banks, NASA etc. need it • CPUs are most likely to fail due to complexity • Hypervisor: primary/backup replication • If primary fails, backup takes over • Caveat: assuming failure detection is perfect

7. Hypervisor replicates at VM-level • Why replicating at VM-level? • Hardware fault-tolerant machines are big in 80s • Software solution is more economical • Replicating at O/S level is messy (many interfaces) • Replicating at app level requires programmer efforts • Replicating at VM level has a cleaner interface (and no need to change O/S or app) • Primary and backup execute the same sequence of machine instructions

8. A Strawman design mem mem • Two identical machines • Same initial memory/disk contents • Start execute on both machines • Will they perform the same computation?

9. Strawman flaws • To see the same effect, operations must be deterministic • What are deterministic ops? • ADD, MUL etc. • Read time-of-day register, cycle counter, privilege level? • Read memory? • Read disk? • Interrupt timing? • External input devices (network, keyboard)

10. Hypervisor’s architecture Strawman replicates disks at both machines Problem: disks might not behave identically (e.g. fail at different sectors) mem mem SCSI bus primary • Hypervisor connects devices to • to both machines • Only primary reads/writes to devices • Primary sends read values to backup • Only primary handles interrupts from h/w • Primary sends interrupts to backup ethernet backup

11. Hypervisor executes in epochs • Challenge: must execute interrupts at the same point in instruction streams on both nodes • Strawman: execute one instruction at a time • Backup waits from primary to send interrupt at end of each instruction • Very slow…. • Hypervisor executes in epochs • CPU h/w interrupts every N instructions (so both nodes stop at the same point) • Primary delays all interrupts till end of an epoch • Primary sends all interrupts to backup

12. Hypervisor failover • If primary fails, backup must handle I/O • Suppose primary fails at epoch E+1 • In Epoch E, backup times out waiting for [end, E+1] • Backup delivers all buffered interrupts at the end of E • Backup starts epoch E+1 • Backup becomes primary at epoch E+2

13. Hypervisor failover • Backup does not know if primary executed I/O epoch E+1? • Relies on O/S to re-try the I/O • Device needs to support repeated ops • OK for disk writes/reads • OK for network (TCP will figure it out) • How about keyboard, printer, ATM cash machine?

14. Hypervisor implementation • Hypervisor needs to trap every non-deterministic instruction • Time-of-day register • HP TLB replacement • HP branch-and-link instruction • Memory-mapped I/O loads/stores • Performance penalty is reasonable • A factor of two slow down • How about its performance on modern hardware?

15. Caveats in Hypervisor • Hypervisor assumes failure detection is perfect • What if the network between primary/backup fails? • Primary is still running • Backup becomes a new primary • Two primaries at the same time! • Can timeouts detect failures correctly? • Pings from backup to primary are lost • Pings from backup to primary are delayed

16. Paxos: fault tolerant agreement • Paxos lets all nodes agree on the same value despite node failures, network failures and delays • Extremely useful: • e.g. Nodes agree that X is the primary • e.g. Nodes agree that Y is the last operation executed

17. Paxos: general approach • One (or more) node decides to be the leader • Leader proposes a value and solicits acceptance from others • Leader announces result or try again

18. Paxos requirement • Correctness (safety): • All nodes agree on the same value • The agreed value X has been proposed by some node • Fault-tolerance: • If less than N/2 nodes fail, the rest nodes should reach agreement eventually w.h.p • Liveness is not guaranteed

19. Why is agreement hard? • What if >1 nodes become leaders simultaneously? • What if there is a network partition? • What if a leader crashes in the middle of solicitation? • What if a leader crashes after deciding but before announcing results? • What if the new leader proposes different values than already decided value?

20. Paxos setup • Each node runs as a proposer, acceptor and learner • Proposer (leader) proposes a value and solicit acceptence from acceptors • Leader announces the chosen value to learners

21. Strawman • Designate a single node X as acceptor (e.g. one with smallest id) • Each proposer sends its value to X • X decides on one of the values • X announces its decision to all learners • Problem? • Failure of the single acceptor halts decision • Need multiple acceptors!

22. Strawman 2: multiple acceptors • Each proposer (leader) propose to all acceptors • Each acceptor accepts the first proposal it receives and rejects the rest • If the leader receives positive replies from a majority of acceptors, it chooses its own value • There is at most 1 majority, hence only a single value is chosen • Leader sends chosen value to all learners • Problem: • What if multiple leaders propose simultaneously so there is no majority accepting?

23. Paxos solution • Proposals are ordered by proposal # • Each acceptor may accept multiple proposals • If a proposal with value v is chosen, all higher proposals have value v

24. Paxos operation: node state • Each node maintains: • na, va: highest proposal # and its corresponding accepted value • nh: highest proposal # seen • myn: my proposal # in current Paxos

25. Paxos operation: 3P protocol • Phase 1 (Prepare) • A node decides to be leader (and propose) • Leader choose myn > nh • Leader sends <prepare, myn>to all nodes • Upon receiving <prepare, n> If n < nh reply <prepare-reject> Else nh = n reply <prepare-ok, na,va> This node will not accept any proposal lower than n

26. Paxos operation • Phase 2 (Accept): • If leader gets prepare-ok from a majority V = non-empty value corresponding to the highest na received If V= null, then leader can pick any V Send <accept, myn, V> to all nodes • If leader fails to get majority prepare-ok • Delay and restart Paxos • Upon receiving <accept, n, V> If n < nh reply with <accept-reject> else na = n; va = V; nh = n reply with <accept-ok>

27. Paxos operation • Phase 3 (Decide) • If leader gets accept-ok from a majority • Send <decide, va> to all nodes • If leader fails to get accept-ok from a majority • Delay and restart Paxos

28. Paxos operation: an example nh=N1:0 na = va = null nh=N2:0 na = va = null nh=N0:0 na = va = null Prepare,N1:1 Prepare,N1:1 nh= N1:1 na = null va = null nh: N1:1 na = null va = null ok, na= va=null ok, na =va=nulll Accept,N1:1,val1 Accept,N1:1,val1 nh=N1:1 na = N1:1 va = val1 nh=N1:1 na = N1:1 va = val1 ok ok Decide,val1 Decide,val1 N0 N1 N2

29. Paxos properties • When is the value V chosen? • When leader receives a majority prepare-ok and proposes V • When a majority nodes accept V • When the leader receives a majority accept-ok for value V

30. Understanding Paxos • What if more than one leader is active? • Suppose two leaders use different proposal number, N0:10, N1:11 • Can both leaders see a majority of prepare-ok?

31. Understanding Paxos • What if leader fails while sending accept? • What if a node fails after receiving accept? • If it doesn’t restart … • If it reboots … • What if a node fails after sending prepare-ok? • If it reboots …

32. Using Paxos for RSM • Fault-tolerant RSM requires consistent replica membership • Membership: <primary, backups> • RSM goes through a series of membership changes <vid-0, primary, backups><vid-1, primary, backups> .. • Use Paxos to agree on the <primary, backups> for a particular vid

33. Lab5: Using Paxos for RSM All nodes start with static config vid1:N1 vid1: N1 N2 joins A majority in vid1:N1 accept vid2: N1,N2 vid2: N1,N2 N3 joins A majority in vid2:N1,N2 accept vid3: N1,N2,N3 vid3: N1,N2, N3 N3 fails A majority in vid3:N1,N2,N3 accept vid4: N1,N2 vid4: N1,N2

34. Prepare, vid2, N3:1 oldview, vid2=N1,N2 N3 vid1: N1 N3 joins Lab5: Using Paxos for RSM vid1: N1 N1 vid2: N1,N2 vid1: N1 N2 vid2: N1,N2

35. Prepare, vid3, N3:1 Prepare, vid3, N3:1 Lab5: Using Paxos for RSM vid1: N1 N1 vid2: N1,N2 N3 vid1: N1 vid2: N1,N2 vid1: N1 N2 N3 joins vid2: N1,N2

36. Lab6: re-configurable RSM • Use RSM to replicate lock_server • Primary in each view assigns a viewstamp to each client requests • Viewstamp is a tuple (vid:seqno) • (0:0)(0:1)(0:2)(0:3)(1:0)(1:1)(1:2)(2:0)(2:1) • All replicas execute client requests in viewstamp order

37. Lab6: Viewstamp replication • To execute an op with viewstamp (vs), a replica must have executed all ops < vs • A newly joined replica need to transfer state to ensure its state reflect executions of all ops < vs

38. … Prepare, accept,decide etc. Largest vs (1:50) N2 vid1: N1 Transfer state to bring N2 up-to-date N2 joins Lab5: Using Paxos for RSM vid1: N1 N1 myvs:(1:50)