System Reliability Axioms: Understanding Failure Modes & Recovery Strategies

1. Operating System Reliability

2. Some Axioms • Some simple systems, designed from scratch, sometimes work • A complex system that works is invariably found to have evolved from a simple system that works • A complex system, designed from scratch never works

3. Failure-Mode Theorems • Complex systems usually operate in failure mode • A system should have safe behaviors when encountering failures • When a “fail-safe” system fails, it fails by failing to fail safe

4. Some definitions • Failure occurs when the system does not perform its services in the manner specified • Failures can be subtle (e.g., performance fault) • Fault is anomalous physical condition • Includes system specification/implementation mistakes • Error is part of system state that differs from its intended value

5. Classification of Failures • Process failures • System failures • Secondary storage failures • Communication medium failures

6. Process Failures • Examples • Computation results in incorrect outcome • System state deviates from specification • Process fails to progress • Errors leading to failure • Deadlock, timeout, protection violation • Bad input, consistency violation • Ignoring malicious behavior

7. System Failures • Processor fails to execute • Software error, hardware error (CPU, bus, etc.) • Fail-stop behavior assumed • Failure types • Amnesia • Partial-amnesia • Pause • Halting

8. Secondary Storage Failures • Stored data inaccessible • Parity error • Head crash • Contaminated medium • Reconstructable from archive + log, maybe • Mirrored disks (independent failure mode)

9. Communication Medium Failures • Site can’t communicate with another site • Causes • Switching node failure • Hardware failure • Software failure • Congestion • Link failure • Hardware • Implementation failure • Network partitions can result

10. Recovery • Restart process/processor • Reclaim resources • Undo/finish incomplete transactions • Concurrency makes things harder

11. Forward Error Recovery • Goal: To restore system from erroneous state to error-free state • If nature of error is completely known • Remove error from state • Proceed with execution from error-free state • Rarely possible to do

12. Backward Error Recovery • When error source unknown • Restore state to previous error-free state; restart • Independent of fault, errors causing fault • Problems • Performance penalty • No guarantee fault will not reoccur • Possible unrecoverable component of state • Recovery point: state used to replace error

13. Backward Error Recovery • Basic approaches • Operation-based • Logs • Update-in-place • Write-ahead-log • State-based

14. Update-in-Place • Every update to object also records the log • Name of object • Old and new states of object • Recoverable update operation implements as • Do, undo, redo operations

15. Write-ahead Log • Update-in-place has problem if crash occurs between update and log recorded to stable storage • Update object only after undo log recorded • Before committing updates, record both redo and undo logs • Expensive to write log to stable storage

16. State-Based Recovery • Save entire process state at recovery point • Recovery point called checkpoint • Rolling back process: restoring to checkpoint • Tradeoff: frequent checkpoints vs. completion delay • Shadow pages • Save unmodified page copy on stable storage • Update only volatile copy; discard on rollback

17. Concurrent Systems Recovery • Rollback issues • Orphan messages • Domino effect • Lost messages • Livelocks

18. Orphan Messages (a message prior to a checkpoint is sent to the future) x1 x2 X [ [ y1 m y2 Y [ [ z1 z2 Z [ [ [ recovery point

19. Domino Effect • Suppose Y rolls back to y2 • m is orphan message • Process Y must rollback to y1 • Suppose Z rolls back to z2 • Y rolls back to y1 • Now a message from future is sent to the past prior to a checkpoint • Forcing Z to roll back to z1

20. Lost Messages x1 X [ m z1 Z [ failure [ recovery point

21. Live Locks x1 X [ z1 Z [ repeated failure [ recovery point

22. Concurrent Recovery • Coordination required at either time of establishing checkpoints • Beginning of recovery

23. Checkpoint Assumptions • Communication via messages • Unreliable FIFO channels • Higher-level end-to-end protocols assumed • Subsumes rollback-caused message loss • No network partitions from communication failures

24. Checkpoint Algorithm Concepts • Permanent and tentative checkpoints • Saved on stable storage • Permanent: part of known consistent global checkpoint • Tentative: until successful termination of checkpoint algorithm • Rolls back only to permanent checkpoints

25. Synchronous Checkpoint Algorithms • Two-phase commit • Problems: • Message overhead for synchronizations • Synchronization delays • Costly when failures are rare

26. Asynchronous Checkpointing • Local checkpoints taken independently • Log all incoming messages on stable storage • Minimizes undone computation • Allows reprocessing of messages after rollback

27. Asynchronous Checkpointing Assumptions • Assumptions • Reliable FIFO communication channels • Infinite buffers • Event-driven computation • A process idle until message received • Processes message and change state • Sends zero or more messages • Can identify each event with monotonically increasing counter

28. Event-Driven Computation x1 x2 X y1 y2 Y z1 z2 Z

29. Asynchronous Checkpointing • Basic idea • Save states, messages sent at each event • Volatile logging • Each processor notes number of messages sent to others, and received from others • Use counters to determine orphan messages

30. Summary • Failures caused by errors • Can remove errors by forward/backward error recovery • Backward error-recovery more costly, more general • Synchronous checkpoints helpful, costly • Asynchronous checkpoints messier, domino effects