1 / 38

Fault Tolerance

Fault Tolerance. Chapter 7. Basic Concepts. System : A collection of components (incl. interconnections) that achieve a common task. Component : A software or hardware component or a set thereof (a subsystem). Failure : A deviation from the the specified behavior of a component.

zea
Download Presentation

Fault Tolerance

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Fault Tolerance Chapter 7

  2. Basic Concepts • System: A collection of components (incl. interconnections) that achieve a common task. • Component: • A software or hardware component or a set thereof (a subsystem). • Failure: A deviation from the the specified behavior of a component. Examples: 1) The program has crashed. 2) The disk is becoming slower. • Fault: The cause of a failure. Examples: 1) The crash of the program was because of a bug in the software. 2) The disk is slower because the R/W-head is overheated.

  3. Basic Concepts • Difference between fault and failure is very subtle. • It helps to introduce the notion of an observation unit. System Component 2 1 F1 2 Component 4 F2 Component 1 Component 3 • Observer 1 will treat F1 as a failure, and he will speak of F2 as a fault (since it was the cause for F1). Unit of observation of observer 1 is the whole system. • However, observer 2 will see F2 as a failure and will look at what fault did cause this failure. Unit of observation of observer 2 is component 1. • In other words: Observer 1 sees the system as a black box whereas observer 2 sees it as a glass box.

  4. Types of Failure in a Distributed System • Different types of failures.

  5. Basic Concepts • Failures may be: • - Permanent: they need recovery in order to be removed (e.g. OS crash  reboot). • - Transient: they disappear after some (short) period without recovery (e.g. dust on a disk). • Fault tolerance: • A system is said to be fault tolerant, if it is able to perform its function even in the case of • failures. • Examples: 1) Two-processor system. • 2) 747s have 4 engines but can fly with 3. • Redundancy: • Fault tolerance can be achieved using redundancy. • Different types of redundancy: • - Information redundancy: More information that help detect and correct failures; • e.g. CRC bits in packets. • - Time redundancy: Repetitions of operations in order to overcome (transient) failures; • e.g. retry access to a disk, redo a transaction. • - Functional redundancy: more functions (e.g. fault handling functions, monitors etc.) in order to detect/mask failures. • - Structural redundancy: more units (e.g. 2 WWW servers, 2 processors etc.) in order to mask • failures of any one of them.

  6. Quantitative Measures for Dependability • Dependability includes: • Reliability • Availability • Robustness • Trustworthiness • Security (see next chapter) • Safety • Maintainability • And perhaps more • Reliability: • Informally: The ability that a system can perform its functions according to its specification during a (specified long) time interval. Thus, the continuity of functioning is the main issue. • Examples: • reliable: An automobile that needs a repair each two years. • unreliable: An automobile that needs a repair after each month.

  7. Quantitative Measures for Dependability • Reliability: • Formally: • L: random variable for the lifetime of a component. (t >= 0) • F(t) = p(L <= t) is the distribution function of L (failure probability). • The reliability of the component is then: • R(t) = 1 – F(t) = p(L > t) (survivability) • Characteristic measures for reliability: • - Mean Time To Failure (MTTF): E[L] = (E[L] is also the life time expectation) • - Failure rate: a(t).Dt = p(L <= t+Dt | L > t) • Thus: • Special case: a(t) = a (constant) • Then: a = f(t)/R(t)  -a = -f(t)/1-F(t)  -a.t +c = ln(1-F(t))  F(t) = 1 – e-at +c • Since c = 0  F(t) = 1 – e-at is the exponential distribution! • Thus the failure rate is constant iff the lifetime is exponentially distributed.

  8. Quantitative Measures for Dependability • Reliability (continued): • - Residual lifetime expectation: r(t) = E(L| L > t) - t • Informally: After t units of time have elapsed, what is the expected rest period until the • next failure (in general until the next event). • Theorem: • Proof:

  9. Quantitative Measures for Dependability a down up b • Availability • . Informally: A component is available at time t, if it is able to perform its function at time t. • A highly reliable system is always highly available (in the perceived interval). • However, a highly available system, may be poorly reliable. • Example (for lowly reliable system that is highly available): Suppose you have a system that experiences one failure each hour. Each failure is repaired very quickly (e.g. after one second), and after that the system is available again. This system is clearly highly available, but the continuity of its service (i.e. its reliability) is very poor. • Formally: • To study the availability of a system, one has to consider, in addition to the failure rate, the (mean) repair rate b of the system (Mean Time To Repair: MTTR = 1/ b). • Here we make a useful assumption that the system is always in one of two states: up or down. • We define availability a time t as: a(t) = p(System is up at t) • The system switches from the up state to the down state with rate: a (constant) • The system switches from the down state to the up state with rate: b (constant) • This results in the following model: down

  10. Quantitative Measures for Dependability a up (1) down (2) b • Availability (continued) • Our goal is to find and expression a(t) = p(System is in state up at t) (= pup(t)). • Let p1(t) = pup(t), p2(t) = pdown(t) and let p12(Dt) and p21(Dt) be the transition probabilities from 1 • to 2 and vice versa. Clearly: p1(t) + p2(t) = 1 (since system is in either state at any time). [1] • Also:a = p12(Dt)/Dt and b = p21(Dt)/Dt (for Dt  0) • Consider state 1: p1(t+Dt) = p1(t).p11(Dt) + p2(t).p21(Dt) (*) • In words: To be in 1 at t+Dt means either the system was at 1 already and made a transition to 1 again, or it was in 2 and made a transition to 1. • Since p11(Dt) + p12(Dt) = 1 (system should make a transition), (*) results in: • p1(t+Dt) = p1(t).(1-p12(Dt)) + p2(t).p21(Dt)  • p1(t+Dt)-p1(t) = -p1(t).p12(Dt) + p2(t).p21(Dt)  (divide by Dt and set Dt to  0) • dp1(t)/dt = -p1(t).a + p2(t).b [2] • Consider state 2: Because of symmetry we get: dp2(t)/dt = -p2(t).b + p1(t).a [3] • With [1] and [2] we get: dp1(t)/dt + (a+ b).p1(t) = b this has the solution: • p1(t) = b/(a +b) + c.e-(a+b).t, suppose system starts in the up state  p1(0) = 1  • c = a/(a +b), hence p1(t) = b/(a +b) + a/(a +b).e-(a+b).t • For t  , p1(t) tends to b/(a +b), hence the availability is a(t) = b/(a +b) = MTTF/(MTTF+MTTR) • Or in other words: availability = expected uptime/(expected uptime+expected downtime)

  11. Failure Masking by Redundancy • Triple modular redundancy. • TMR is an example of structural redundancy. •  Voters filter out incorrect signals. • Voters are also “replicated” because they may fail, too.

  12. System Reliability • Problem: • Suppose you know the reliability of each component in your system, what is the reliability of your system? • Let S = {C1, …, Cn} be a system consisting of n components C1, …, Cn. • Let pi and Ri(t) be respectively the failure probability and the reliability of component i. • Typical cases: Serial structure, parallel structure, k-from-n structure. • Serial Structure: All components are needed in order for the system to work. •  System reliability: • In words: System survives t iff all components survive t. •  System’s failure probability:s(p1,…,pn) = 1 - • In words: System is intact iff all Ci are intact. With 1-pi is probability that Ci is • intact, it follows that 1-s = p(System is intact) = 1 2 n-1 n

  13. System Reliability 1 2 n Example: 3 servers each with failure probability p and reliability R(t) and all of them are needed to achieve a special task.  serial structure of three servers. 1) System failure probability: s(p) = 1 – (1-p)3 2) System reliability: RS(t) = R(t)3 Suppose R(t) = e-lt RS(t) = e-3lt In other words, the life time expectation of the system E[LS] = (with E[L] lifetime expectation of one server) Parallel Structure: At least one component is needed for the system to work.  System reliability:RS(t) = 1- In words: A component does not survives t with probability 1-Ri(t), hence the system does not survive t with probability which is clearly 1 – RS(t).  System failure probability: s(p1,…,pn) = In words: The system fails iff all components fail.

  14. System Reliability Example: 3 servers each with failure probability p and reliability R(t) and at least one of them is needed to achieve a special task.  parallel structure of three servers. 1) System failure probability: s(p) = p3 < p 2) System reliability: RS(t) = 1 – (1-R(t))3 Suppose R(t) = e-lt RS (t) = 1- (1-e-lt)3 The life time expectation of the system E[LS] = k-from-n Structure: At least k components are needed for the system to work. Example: TMR (2-from-3 Structure with voter) RS (t) = 3R(t)2 – 2R(t)3 FS(t) = 3F(t)2 – 2F(t)3 Thus ratio F(t)/FS(t) is high for small F(t)  Gain of TMR is huge. (However, in general lifetime of a TMR system is less then that of a single component)

  15. System Reliability TMR (continued) In fact, TMR is nothing but a combination of a serial and a parallel system (see figure below). 1) TMR failure probability: Let p be the failure probability of one component. Let p[i,j] be the failure probability of two serial components i and j.  p[i,j] = 1 – (1-p)2 for all i and j. Hence: s(p) = p[1,2].p[1,3].p[2,3] = (1 – (1-p)2)3 2) TMR reliability: Suppose R(t) = e-lt RS(t) = 3e-2lt - 2e-3lt  Attention: here we ignored the voter component needed in TMR 1 2 1 3 2 3

  16. Flat Groups versus Hierarchical Groups • Communication in a flat group. • Communication in a simple hierarchical group • Motivation: Replicated processes are organized into groups in order to achieve higher availability. • Main assumption: Processes are identical (and run in general on different machines). • Attention: Own reliability of processes cannot be augmented using replication.

  17. Agreement in Faulty Systems n = 4, m = 1 Agreement: (1, 2, ?, 4) n = 3, m = 1 No agreement is possible • Byzantine generals problem: n generals where n-m are loyal and m are traitors; the generals • try to achieve an agreement. • With m traitors at least 2m+1 loyal generals are needed for an agreement. • (2 army problem: 2 perfect generals (processes) but communication channel is impaired. •  No agreement is possible)

  18. RPC Semantics in Presence of Failures • Client cannot find server (e.g. server down): • Solution: Exception handling •  non transparent • Request to server is lost: • Solution: Resend request after timeout • - Message identifiers are needed to detect duplicates. • - Server is then stateful. • Reply from server is lost: • Solution: Resend request after timeout • - Client cannot be sure whether request or reply was lost. • - Operations that the server performed may be re-executed. •  not all operations are idempotent ! • Client crash: Server may be executing a request on behalf of a crashed client  orphan • Solution: find orphans and kill them (solutions are not ideal). • 1) Extermination: client logs requests, and after restart it kills any orphan. • 2) Reincarnation: Divide time into intervals, after restart, client broadcasts new epoch number. • Orphans are killed on receipt of such a message. • 3) Expiration: Server works T units of time and asks client for more time if needed. Client waits • T time units after restart (to be sure that potential orphans are gone). • Sever crash: see next slides

  19. Server Crashes (1) • A server in client-server communication • Normal case • Crash after execution • Crash before execution • Problem: From the point of view of the client (b) looks like (c), simply there is no reply. • Different RPC semantics: • At least once semantics: keep trying until reply has been received. • At most once semantics: never retry a request. • Exactly once semantics: in distributed systems generally unachievable.

  20. Server Crashes (2) • Different combinations of client and server strategies in the presence of server crashes. • P: Processing, C: Crash, M: Completion message • Server strategies: MP: Send completion message before processing • PM: Send completion message after processing • Thus: The number of executions of an operation depends on when the server crash has • occurred.

  21. Basic Reliable-Multicasting Schemes Problem: Sender overwhelmed with ACKs • A simple solution to reliable multicasting when all receivers are known and are assumed not to fail • Message transmission • Reporting feedback

  22. Feedback Control (for better Scalability) • Several receivers have scheduled a request for retransmission, but the first retransmission request leads to the suppression of others. • - Only negative ACKs (NACKs) are sent back to sender. • - Any receiver may suppress its NACK, if another receiver would require the same • message from the sender: • 1. A receiver (with a NACK wish) waits a random delay T. • 2. If after T it does not receive any NACK, it multicasts its own NACK. • 3. If it receives a NACK before T elapses, it suppresses its own scheduled NACK.

  23. Virtual Synchrony (1) • The logical organization of a distributed system to distinguish between message receipt and message delivery • Virtual synchrony: Deals with reliable multicast in presence of process failures. • Minimum requirement: If a sender sends a message m to a group G, m is either delivered to all • non-faulty processes in G or to none of them (reliable multicast). • Problem: What if the group membership changes during the transmission of m?

  24. Virtual Synchrony (2) • The principle of virtual synchronous multicast. • Process P3 crashes  Group membership changes. • 1) Partially sent multicast messages of P3 should be discarded. • 2) P3 should be removed from the group. • If the group membership changes voluntarily (without crashes), the system should deliver partially sent multicasts to all members or to none of them.

  25. Message Ordering (1) • Three communicating processes in the same group. • Typical message ordering policies for virtual synchrony: • No requirements: any ordering is valid • FIFO-ordering of multicasts: messages originating from the same process are delivered in the order the process has sent them. • Causally-ordered multicasts: messages that are causally related are delivered in the order of causality; but no requirement for concurrent messages. • Totally-ordered multicast: all messages appear in same order to all group members (orthogonal to FIFO, causal, or no requirement).

  26. Message Ordering (2) • Four processes in the same group with two different senders, and a possible delivery order of messages under FIFO-ordered multicasting • Totally-ordered FIFO would be: P2 is delivered message m3 and then message m1.

  27. Implementing Virtual Synchrony (1) • Six different versions of virtually synchronous reliable multicasting.

  28. Implementing Virtual Synchrony (2) • Process 4 notices that process 7 has crashed, sends a view change • Process 6 sends out all its unstable messages, followed by a flush message • Process 6 installs the new view when it has received a flush message from everyone else • Above protocol is implemented in Isis on top of TCP/IP. • Unstable message: A partially received message e.g. only 1 and 2 received it.

  29. Two-Phase Commit (1) • The finite state machine for the coordinator in 2PC. • The finite state machine for a participant. • Main goal: All-or-nothing property in presence of failures. • Failures: Coordinator’s or participant’s crash.

  30. Two-Phase Commit (2) • Actions taken by a participant P when residing in state READY and having contacted another participant Q. • Situations: • 1)Coordinator crash (or no response after timeout): • 1.1 If participant P is in state INIT  P makes a transition to state ABORT. • 1.2 If participant P is in state READY  P contacts another participant Q (see table above). 2) Participant crash (or no response from a participant after timeout): Since coordinator must be in the WAIT state, it makes a transition to the ABORT state.

  31. Two-Phase Commit (3) Actions by coordinator: write START_2PC to local log;multicast VOTE_REQUEST to all participants;while not all votes have been collected { wait for any incoming vote; if timeout { write GLOBAL_ABORT to local log; multicast GLOBAL_ABORT to all participants; exit; } record vote;}if all participants sent VOTE_COMMIT and coordinator votes COMMIT{ write GLOBAL_COMMIT to local log; multicast GLOBAL_COMMIT to all participants;} else { write GLOBAL_ABORT to local log; multicast GLOBAL_ABORT to all participants;} • Outline of the steps taken by the coordinator in a two phase commit protocol

  32. Two-Phase Commit (4) Actions by participant: write INIT to local log;wait for VOTE_REQUEST from coordinator;if timeout { write VOTE_ABORT to local log; exit;}if participant votes COMMIT { write VOTE_COMMIT to local log; send VOTE_COMMIT to coordinator; wait for DECISION from coordinator; if timeout { multicast DECISION_REQUEST to other participants; wait until DECISION is received; /* remain blocked */ write DECISION to local log; } if DECISION == GLOBAL_COMMIT write GLOBAL_COMMIT to local log; else if DECISION == GLOBAL_ABORT write GLOBAL_ABORT to local log;} else { write VOTE_ABORT to local log; send VOTE ABORT to coordinator;} DECISION must come from another participant or from coordinator after a potential recovery. • Steps taken by participant process in 2PC.

  33. Two-Phase Commit (5) Actions for handling decision requests: /* executed by separate thread */ while true { wait until any incoming DECISION_REQUEST is received; /* remain blocked */ read most recently recorded STATE from the local log; if STATE == GLOBAL_COMMIT send GLOBAL_COMMIT to requesting participant; else if STATE == INIT or STATE == GLOBAL_ABORT send GLOBAL_ABORT to requesting participant; else skip; /* other participant remains blocked */ • Steps taken for handling incoming decision requests.

  34. Three-Phase Commit • Finite state machine for the coordinator in 3PC • Finite state machine for a participant

  35. Recovery Stable Storage Disk 1 Disk 2 • Stable Storage: e.g. two disks • Crash after Disk 1 is updated: a’ is used (update begins with Disk 1) • Bad spot: get copy from the other disk

  36. Checkpointing (1) • A recovery line (consistent cut). • Recovery: • Backward recovery: back to an earlier fault-free state  checkpointing • Forward recovery: Switch to a new fault-free state (e.g. replication). • Problems with checkpointing: - Added overhead( message logging can help) • - Finding recovery line (see figure)

  37. Checkpointing (2) c11 c13 c12 c23 c22 c21 • The domino effect (independent checkpointing). • (c13, c23) and (c13, c22) inconsistent, since m resp. m’ received but not sent. •  Recovery from initial states is needed! • Solution: coordinated checkpointing, which always leads to a consistent cut: • 1. Coordinator multicasts CP_REQUEST to all processes. • 2. Processes do the checkpoint and defer any message traffic until CP_DONE has been received from coordinator (P1 does not send m). • 3. After ACK from all processes, coordinator multicasts CP_DONE.

  38. Message Logging • Incorrect message replay after recovery, leading to an orphan process (R) • Message logging: Messages since last checkpoint are logged and replayed after recovery. • Advantages: • + helps reduce checkpointing frequency (i.e. overhead). • + Message replay means that behavior of the process before and after recovery are the same. • Problem: orphan processes (see figure: R does work for Q and Q does not know that)

More Related