1 / 32

Distributed Systems

Distributed Systems. Topic 12: Recovery and Fault Tolerance Computer Science & Engineering Department The Chinese University of Hong Kong. Outline. 1 Introduction 2 Transaction Recovery 3 Fault Tolerance 4 Hierarchical and Group Masking of Faults 5 CORBA Fault Tolerance Service 6 Summary.

matthewford
Download Presentation

Distributed Systems

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Distributed Systems Topic 12: Recovery and Fault Tolerance Computer Science & Engineering Department The Chinese University of Hong Kong

  2. Outline 1 Introduction 2 Transaction Recovery 3 Fault Tolerance 4 Hierarchical and Group Masking of Faults 5 CORBA Fault Tolerance Service 6 Summary

  3. 1 Introduction • Fault tolerance: the survival attribute of systems. • Fault-tolerant applications: • transaction based • process control • Recovery aspects of distributed transactions. • The design of real time services. • Fail-stop vs Byzantine failure. • Masking failures in a service. • CORBA fault tolerance service.

  4. 1 Basic Approaches • Fault Detection: • Push Model: Server objects send heartbeat messages to Fault Manager. • Pull Model: Fault Manager polls (or pings) server objects through their is_alive() interface. • Data Recovery: • Checkpoint and rollback: Save the server object states. Roll back to checkpointed states at recovery. • Message logging and replay: Log all messages. Replay them at recovery.

  5. 2 Transaction Recovery • Recovery concerns data durability (permanent and volatile data) and failure atomicity. • A server keeps data in volatile memory and records committed data in a recovery file. • Recovery manager • save data items in permanent storage • Restore the server’s data items after a crash • reorganize the recovery file for better performance • reclaim storage space (in the recovery file)

  6. 2 Intentions List • An intentions list of a server is a list of data item names and values altered by a transaction. • The server uses the intentions list when a transaction commits or aborts. • When a server prepares to commit, it must have saved the intentions list in its recovery file. • The recovery files contain sufficient information to ensure the transaction is committed by all the servers.

  7. 2 Entries in Recovery File Type of entry Description of contents of entry Data item A value of a data item Transaction status Transaction identifier, transaction status (prepared, committed, aborted) and others Intentions list Transaction identifier and a sequence of intentions, each of which consists of <id of data item>, <position in recovery file of value of data item>

  8. 2.1 Logging • A log contains history of all the transactions performed by a server. • The recovery file contains a recent snapshot of the values of all the data items in the server followed by a history of transactions. • When a server is prepared to commit, the recover manager appends all the data items in its intentions list to the recovery file. • The recovery manager associates a unique identifier with each data item.

  9. 2.1 Log for Banking Service

  10. 2.1 Recovery by Logging • Recovery of data items • Recovery manager is responsible for restoring the server’s data items. • The most recent information is at the end of the log. • A recovery manager gets corresponding intentions list from the recovery file. • Reorganizing the recovery file • Checkpointing: the process of writing the current committed values (checkpoint) to a new recovery file. • Can be done periodically or right after recovery.

  11. 2.2 Shadow Versions • Shadow versions technique uses a map to locate versions of the server’s data items in a file called a version store. • The versions written by each transaction are shadows of the previous committed versions. • When prepared to commit, any changed data are appended to the version store. • When committing, a new map is made. When complete, new map replaces the old map.

  12. 2.2 Shadow Versions Example

  13. 2.2 Log and 2PC

  14. 2.3 Recovery of 2PC

  15. 3 Fault Tolerance • Two contrasting points on distributed systems: • The operation of a service depends on the correct operation of other services. • Joint execution of a set of servers is less likely to fail than any one of the individual components. • Designers of a service should specify its correct behavior and the way it may fail • Failure semantics: a description of the ways a service may fail. Can be used for its clients to mask its failures.

  16. 3 Characteristics of Faults Class of failure Subclass Description Omission failureA server omits to respond to a request Response failure Server responds incorrectly to a request Value failure Return wrong value State transition Has wrong effect on resources (for failure example, sets wrong values in data items) Timing failure Response not within a specified time interval Crash failure Repeated omission failure: a server repeatedly fails to respond to requests until it is restarted Amnesia-crash A server starts in its initial state, having forgotten its state at the time of the crash Pause-crash A server restarts in the state before the crash Halting-crash Server never restarts

  17. 3 Fail-Stop vs Byzantine Failures • A fail-stop server is one that fails cleanly. That is, it either functions, or else it crashes. • Byzantine failure behavior is used to describe the worse possible failure semantics of a server: it fails maliciously or arbitrarily. • Byzantine agreement is intended for correct behaviors within response time requirement in the presence of faulty hardware. • It depends on if messages can be authenticated.

  18. 3 Byzantine Generals

  19. 3 Byzantine Agreement Algorithms • Byzantine agreement algorithms send more messages and use more active servers. • When messages can be authenticated, 2N+1 servers are required to tolerate N bad servers. • When messages cannot be authenticated, 3N+1 servers are required. • With enough good servers, solutions require O(N2) messages with constant delay time. • Fortunately, the good news is ...

  20. 4 Hierarchical Masking of Faults • We describe two approaches to masking faults: hierarchical failure masking and group failure masking. • In hierarchical failure masking, a server of higher level tries to mask faults at lower-level. • When a lower-level failure cannot be masked, it is converted to a higher level exception. • Example: Server crash is masked in RR protocol by raising an exception to the client.

  21. 4 Group Failure Masking • A service can be made fault tolerant by implementing it by a group of servers. • A group is t-fault tolerant if it can tolerate up to t member failures. • For fail-stop failures, t+1 servers are needed. • For Byzantine failures, 2t+1 servers needed. • To ensure correctness, the server program must be deterministic, and each operation must be atomic w.r.t. other operations.

  22. 4 Group Failure Masking • A group can be closely synchronized or loosely synchronized. • In a closely synchronized group of servers: • All members execute requests immediately. • Server programs are both deterministic and atomic. • Suitable for real time system and Byzantine failures. • In a loosely synchronized group of servers: • One server (primary) performs requests, others (backup) log the requests and take over if needed. • Requires less resource but takes longer to recover.

  23. 5 CORBA Fault Tolerance Service Application Objects CORBAfacilities Object Request Broker CORBAservices Fault Tolerance

  24. 5 Outline of Fault Tolerant CORBA • Fault Tolerance Properties • Replication Styles, Membership Styles, Consistency Styles, Fault Monitoring Styles • Infrastructure-Controlled and Application-Controlled • Object Group References and Alternative Destinations • At-Most-Once Invocation (repeated requests detected) • Fault Detection and Notification • Checkpointing and Logging

  25. 5 Architectural Overview

  26. 5 Fault Detectors, FaultNotifier, Fault Analyzer, and ReplicationManager

  27. 5 Property Management interface PropertyManager { void set_default_properties(in Properties props) raises (InvalidProperty,UnsupportedProperty); Properties get_default_properties(); void remove_default_properties(in Properties props) raises (InvalidProperty,UnsupportedProperty); void set_type_properties(in TypeId type_id, in Properties overrides) raises (InvalidProperty,UnsupportedProperty); Properties get_type_properties(in TypeId type_id); void remove_type_properties(in TypeId type_id, in Properties props) raises (InvalidProperty, UnsupportedProperty); void set_properties_dynamically(in ObjectGroup object_group, in Properties overrides) raises(ObjectGroupNotFound, InvalidProperty, UnsupportedProperty); Properties get_properties(in ObjectGroup object_group) raises(ObjectGroupNotFound); };

  28. 5 ObjectGroupManager & GenericFactory // Specification of ObjectGroupManager Interface // which ReplicationManager Inherits interface ObjectGroupManager { ObjectGroup create_member(in ObjectGroup object_group, in Location the_location, in TypeId type_id …) ObjectGroup add_member(in ObjectGroup object_group, in Location the_location, in Object member); ObjectGroup remove_member(in ObjectGroup object_group,in Location the_location); ObjectGroup set_primary_member(in ObjectGroup object_group,in Location the_location); Locations locations_of_members(in ObjectGroup object_group); ObjectGroup get_object_group_ref(in ObjectGroup object_group); ObjectGroupId get_object_group_id(in ObjectGroup object_group); }; // Specification of GenericFactory Interface // which ReplicationManager Inherits and Application Objects Implement interface GenericFactory { typedef unsigned long long FactoryCreationId; Object create_object(in TypeId type_id, in Criteria the_criteria, out FactoryCreationId factory_creation_id); void delete_object(in FactoryCreationId factory_creation_id); };

  29. 5 Replication Management // Specification of ReplicationManager Interface interface ReplicationManager : PropertyManager, ObjectGroupManager, GenericFactory { void register_fault_notifier(in FaultNotifier fault_notifier); FaultNotifier get_fault_notifier() raises (InterfaceNotFound); };

  30. 5 Logging and Recovery // Specification of Checkpointable Interface // which Updateable and Application Objects Inherit interface Checkpointable { State get_state() raises(NoStateAvailable); void set_state(in State s) raises(InvalidState); }; // Specification of Updateable Interface // which Application Objects Inherit interface Updateable : Checkpointable { State get_update() raises(NoUpdateAvailable); void set_update(in State s) raises(InvalidUpdate); };

  31. 5 Fault Detection and Notification // Specification of PullMonitorable Interface which Application Objects Inherit interface PullMonitorable { boolean is_alive(); }; // Specification of FaultNotifier Interface interface FaultNotifier { typedef string ConsumerId; void push_structured_fault(in CosNotification::StructuredEvent event); CosNotifyFilter::Filter create_subscription_filter (in string constraint_grammar) raises (CosNotifyFilter::InvalidGrammar); ConsumerId connect_structured_fault_consumer( in CosNotifyComm::StructuredPushConsumer push_consumer, in CosNotifyFilter::Filter filter) ; void disconnect_consumer( in ConsumerId connection) raises(CosEventComm::Disconnected); … };

  32. 6 Summary • Transaction recovery • long-life application and data integrity • atomic commit protocol is the key • checkpoints and logging in a recovery file • Fault tolerance • real-time application • importance of fault semantics • primary-backup server for fail-stop failures • closely synchronized group for Byzantine failures • Emerging CORBA Fault Tolerance Service

More Related