1 / 26

CiFTS Coordinated Infrastructure for Fault Tolerant Systems

CiFTS Coordinated Infrastructure for Fault Tolerant Systems. Agenda. The Problem and the purpose The CIFTS framework The CIFTS team Call for Action. The Problem. No knowledge of this failure shared between System resources. MPI. detects “communication failure” with node X.

zazu
Download Presentation

CiFTS Coordinated Infrastructure for Fault Tolerant Systems

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. CiFTS Coordinated Infrastructure for Fault Tolerant Systems

  2. Agenda • The Problem and the purpose • The CIFTS framework • The CIFTS team • Call for Action

  3. The Problem No knowledge of this failure shared between System resources MPI detects “communication failure” with node X JS/RM continues to schedule jobs on same resources MPI Aborts! Application Aborts! More failures Cluster system software is agnostic of this MPI job failure Cluster system software is agnostic of this reason of MPI job failure Very less fault information sharing! No mechanism for global system knowledge!

  4. The Purpose JS/RM MPI Not launch jobs on node X until Further diagnosis detects “communication failure” with node X MPI shares this failure knowledge with the system Application Diagnostics Utility Checkpoints itself Runs scripts for root-causing Node X problem Application Application Checkpoints itself Checkpoints itself

  5. The Purpose JS/RM PVFS Launch jobs with NFS file system IO node failure. PVFS down Migrates existing jobs PVFS shares this information Application Checkpoints itself MPI-IO Prints a coherent error message Application Checkpoints itself

  6. PVFS Universal Logger Checkpoint Restart System Resource Manager/JS Fault Tolerant Backplane Event Analysis System Components Automatic Actions Diagnostics Tools Middleware Like MPI MPI-IO Linear Algebra Libraries Autonomics Libraries and Applications The CIFTS Framework Fault Tolerant Backplane

  7. Register with FTB Subscribe for events Publish events 1 2 3 A little deeper… Component Instance Component Instance 1 1 Distributed Fault Tolerant Backplane 2 2 3 3

  8. CIFTS API - some primitives • FTB_Init (IN FTB_comp_info_t *comp_info, OUT FTB_client_handle_t *client_handle, OUT char *error_msg) • FTB_Publish_event (IN FTB_client_handle_t handle, IN char *event_name, IN FTB_event_data_t *datadetails, OUT char *error_msg) • FTB_Create_mask (INOUT FTB_event_mask_t *evt_mask, IN char *field_name, IN char *field_val, OUT char *error_msg) • FTB_Subscribe (IN FTB_client_handle_t chandle, IN FTB_event_mask_t *event_mask, OUT FTB_subscribe_handle_t *shandle, OUT char *error_msg IN int (*callback)(OUT FTB_catch_event_info_t *, OUT void*), IN void *arg) • FTB_Poll_for_event (IN FTB_subscribe_handle_t shandle, OUT FTB_catch_event_info_t *catch_event, OUT char *error_msg); • FTB_Finalize (IN FTB_client_handle_t handle);

  9. Supported components BLCR FT-LA PVFS Fault Tolerant Backplane ROMIO ScaLAPACK Cobalt MVAPICH2 MPICH2 ZeptoOS OpenMPI LAMMPS LAM/MPI NWChem SWIM IPS

  10. Status Quo • Alpha version under works • Demos available on SC exhibit floor • Client API to be finalized by Q1’ FY08 • First release target : ? • Platforms supported : Linux clusters, IBM BGL • Discuss more with Pete

  11. CIFTS team • Argonne National Laboratory • Pete Beckman, Rinku Gupta, Ewing Lusk, Rob Ross, Rajeev Thakur • Indiana University • Andrew Lumsdaine & team • Lawrence Berkeley National Laboratory • Paul Hargrove • Oak Ridge National Laboratory • Al Geist, David Bernholdt • Ohio State University • D.K. Panda • University of Tennessee, Knoxville • Jack Dongarra

  12. Call for Action FT-LA Lustre GPFS Intel MLK BLCR IBRIX GFS ScaLAPACK Polyserv Fault Tolerant Backplane Panasas PVFS MAUI Cobalt ROMIO Condor MVAPICH2 MPICH-MX SGE LSF MPICH2 LAMMPS ZeptoOS Intel MPI PBS/Pro Global Arrays OpenMPI SWIM IPS NWChem SLURM Linux LAM/MPI Other Applications Scali MPI Star-CD LS-Dyna MM5 BLAST Eclipse Fluent

  13. Need more information? • SC’07 Exhibit floor • Demos and/or talks at ANL, ORNL and LBNL booth • CIFTS website • http://www.mcs.anl.gov/research/cifts/ • CIFTS wiki • http://wiki.mcs.anl.gov/cifts • CIFTS mailing list • cifts_discuss@googlegroups.com

  14. Do we need a slide on timeline? • Do we need to go into more details on the design?

  15. Backup

  16. CIFTS - The working view PVFS Universal Logger Checkpoint Restart System Resource Manager/JS Event Analysis System Components Automatic Actions Diagnostics Tools Bootstrap Server Middleware Like MPI MPI-IO Linear Algebra Libraries Autonomics Libraries and Applications

  17. FTB Internal Architecture Layers Component 1 Component n FTB Agent FTB Client API Client Library Linux BGL CRAY FTB Manager API Manager Library Manager Library Network Network Network Module1 Network Module2 Network Module1 Network Module2 Component software stack FTB Agent software stack

  18. What you need to know! Component 1 Component n FTB Agent Just the FTB Client API Client Library Linux BGL CRAY FTB Manager API Manager Library Manager Library Network Network Network Module1 Network Module2 Network Module1 Network Module2 Component software stack FTB Agent software stack

  19. Building a FTB-enabled sample component • List the events you may want to publish in an XML file (for convenience) • Use the API to make the component FTB-enabled • Publish and subscribe to events

  20. FTB-Enabled Component Development (Step1) STEP 1: Create an XML file, outlining the publishable events <ftb_component_details> <namespace>ftb.ftb_examples.watchdog<namespace> <publish_event> <event_name>WATCH_DOG_EVENT</event_name> <event_severity>Info</event_severity> <event_desc>This event is used by watchdog</event_desc> </publish_event> <publish_event> … </publish_event> </ftb_component_details>

  21. Developing a FTB-enabled component (Step 2) STEP 2: Enabling your FTB component! #include "libftb.h" #include "ftb_event_def.h" #include "ftb_throw_events.h" int main (int argc, char *argv[]) { strcpy(cinfo.comp_namespace, "FTB.FTB_EXAMPLES.Watchdog"); strcpy(cinfo.schema_ver, "0.5"); strcpy(cinfo.inst_name, "watchdog"); strcpy(cinfo.jobid,"watchdog-111"); strcpy(cinfo.catch_style,"FTB_POLLING_CATCH"); FTB_Init(&cinfo, &handle, err_msg); FTB_Register_publishable_events(handle, ftb_ftb_examples_watchdog_events, FTB_FTB_EXAMPLES_WATCHDOG_TOTAL_EVENTS, err_msg); FTB_Create_mask(&mask, "all", "init", err_msg); FTB_Subscribe(handle, &mask, &shandle, err_msg, NULL, NULL); FTB_Publish_event(handle, "WATCH_DOG_EVENT", publish_event_data, err_msg); FTB_Poll_for_event(shandle, &caught_event, err_msg); FTB_Finalize(handle); return 0; }

  22. Developing a FTB-enabled component (Step 2..contd) STEP 2: Enabling your FTB component! Creating your subscribe event mask Create a mask to catch all events • FTB_Create_mask(&mask, "all", "init", err_msg); Create a mask to catch “WATCH_DOG_EVENT” • FTB_Create_mask(&mask, "all", "init", err_msg); 2. FTB_Create_mask(&mask, "event_name", "WATCH_DOG_EVENT", err_msg); Create a mask to catch events of severity fatal • FTB_Create_mask(&mask, "all", "init", err_msg); 2. FTB_Create_mask(&mask, “severity”, ”FTB_FATAL", err_msg);

  23. Developing a FTB-enabled component (Step 3) STEP 3: Provide options to end user to compile your code with FTB • Modify configure.in and makefiles, so that you can compile your code • ./configure --with-ftb=<PATH to FTB install directory>

  24. Setting up FTB environment Compiling FTB • Download FTB • ./configure --with-platform=linux --with-bstrap-name=bucco • make • make install

  25. Using FTB Starting FTB • ./ftb_database_server • ./ftb_agent on all linux nodes • Run you component executables Connection Topology FTB Agent FTB Agent FTB Agent Agent contacts server FTB Agent Bootstrap DB server BS -Server provides parent address FTB Agent FTB Agent

  26. Open Issues • Policy management • Global knowledge of component prioritization for handling events • How can components announce their FT capabilities? • How can components request for action from other components? • How to we establish scoping of events?

More Related