Distributed Computing with DAFFIE

1. Distributed Computing with DAFFIE Glenn Bresnahan Boston University Scientific Computing and Visualization (http://scv.bu.edu/DAFFIE/Earlab-Oct02)

2. What is DAFFIE? DistributedApplicationsFrameworkForImmersiveEnvironments System for easily creating distributed and shared virtual environments Authored by Glenn Bresnahan, Erik Brisson, Robert Putnam Scientific Computing and Visualization

3. DAFFIE Messaging System • Application protocol • Client services • Client application library (API) • Message router (“event server”) Scientific Computing and Visualization

4. DAFFIE Message routing client client client proxy client client event server client client proxy client client Scientific Computing and Visualization

5. Application Architecture App thread App thread App thread … App thread Services (timing, streaming, file sharing) Event Messaging Message Transport Scientific Computing and Visualization

6. API - Control • id = event_join(server, &nc); • rc = event_leave(); • id = event_id(); • nc = event_nclient(); • millisec = event_clock(); • event_sleep(millisec); • event_register(class, species, instance); Scientific Computing and Visualization

7. Event headers information • EventType(event); • EventSize(event); • EventFrom(event); • EventTo(event); • EventSeq(event); Scientific Computing and Visualization

8. System Events – Join & Leave typedef struct { EVENT_HEAD head; int id; int nmember; } EVENT_JOIN; typedef struct { EVENT_HEAD head; int id; short detail; short exitcode; } EVENT_LEAVE Scientific Computing and Visualization

9. System Events – Identification typedef struct { EVENT_HEAD head; int tag; } EVENT_IDENTITY; typedef struct { EVENT_HEAD head; char host[MAXIDSTR]; char class[MAXIDSTR]; char species[MAXIDSTR]; char instance[MAXIDSTR]; int tag; } EVENT_IDENTIFICATION; Scientific Computing and Visualization

10. API – Send • rc = event_send(to, type, &event, size); • “to” is a client id or one of the following: • EVENT_BCAST_ALL • EVENT_BCAST_NOTME • EVENT_BCAST_SERVER • EVENT_BCAST_LOCAL • Macros • SEND(to, type, event); • BCAST(type, event); • event_drain(); Scientific Computing and Visualization

11. API - Receive • rc = event_receive(&event); • event_receive_type(n, type1, …); • event_select(on_off, et_low, et_high); • event_wait(); • event_wait_timeout(millisec); Scientific Computing and Visualization

12. API – Receive queue • event_scan_type(n, type1, …); • rc = event_scan(&event); • n = event_queued(); • event_flush(flag); • event_flush_sync(); • event_receive_enable(on_off); Scientific Computing and Visualization

13. API – Active messages • event_callback(type, func); • event_add_callback(type, func); • event_delete_calllback(type, func); • event_protect_type(on_off, et_low, et_high); Scientific Computing and Visualization

14. Client Services • Streaming • Data chunking • Bandwidth management • Dynamic tuning • File sharing • File servers • Conditional upload/download Scientific Computing and Visualization

15. Job Initiation & Monitoring • Rely on external authentication and job initiation (e.g. ssh, Globus) • Launch scripts (ad hoc) • cgl-video; visible.bu.edu:vgrab -compress -process; -:vswirl -steps 9999999 -np 4; -:vbroadcast; .:vdirect -object poolanim_grp ; • Agent initiation daemon • Logging daemon Scientific Computing and Visualization

16. Message Transports • TCP/IP • Single channel or striped • Shared memory • Myrinet • IP Multicast Scientific Computing and Visualization

17. Defining Upper Layer Protocol • Components • Event type ID (#define in a .h) • Data layout (compile-time) (C struct in a .h) • Data layout (run-time) (event_define) • Needed for canonical byte ordering • Types statically (compile time) defined • Protocol compiler Scientific Computing and Visualization

18. Message Routing • Message router (“event server”) • Multiple delivery service classes • Globally sequenced, reliable delivery • Client management • Message subscription • Multiprocessor, high bandwidth • Hierarchical servers • Proxy routers • Fan-in/fan-out • Tunneling • Direct routing for P2P Scientific Computing and Visualization

19. Messaging API • Multithreaded, non-blocking • Dynamic join and leave • Point-to-point and broadcast messages • Polled or callback (active messages) • Subscription based receive • Low level services • Registration, timing, synchronization, query Scientific Computing and Visualization

20. Service Discovery • Client registration/identification • Class (Viewer, Audio, Video, Agent) • Species (video: source, broadcaster, sink, processor) • Instance (video: stream-name) • ID • Hostname • Client location services • Service request/acknowledgement Scientific Computing and Visualization

21. Client Monitoring • Dynamic clients • JOIN, EXIT, IDENTIFICATION events • Event server monitoring • Heart beat • Latency monitor • Congestion control • Statistic generation • Client monitoring • Process level monitoring • Congestion management • Remote process termination Scientific Computing and Visualization

22. Leaky Example: Headers typedef struct { EVENT_HEAD head; char filename[200]; } EVENT_GETPARAM; typedef struct { EVENT_HEAD head; int logger, frame_size, sample_rate); } EVENT_START; typedef struct { EVENT_HEAD head; char msg[1000]; } EVENT_LOGINFO; Scientific Computing and Visualization

23. Leaky Example: Headers typedef struct { EVENT_HEAD head; int client; int size; float pressure[1]; } EVENT_ADVANCE; Scientific Computing and Visualization

24. Leaky Example: Headers #define ET_START 1001 #define ET_GETPARAM 1002 #define ET_LOGINFO 1003 #define ET_ADVANCE 1004 #define ET_STOP 1005 event_define(ET_START, “start”, 1, 3, EVENT_DATA_INT); event_define(ET_GETPARAM, “getparam”, 1, 200, EVENT_DATA_CHAR); event_define(ET_ADVANCE, “advance”, 3, 1, EVENT_DATA_INT, 1, EVENT_DATA_INT, 0, EVENT_DATA_FLOAT); Scientific Computing and Visualization

25. Leaky Example: Initialization id = event_join(“earlab.bu.edu”, &nc); printf(“Joined as client id %d of %d\n”, id, nc); event_register(“integrator”, “leaky”, “1.0”); event_receive_type(ET_GETPARAM, ET_START, ET_ADVANCE, ET_STOP); event_callback(ET_START, do_start); event_callback(ET_GETPARAM, do_getparam); … Scientific Computing and Visualization

26. Leaky Example: File Service GotParamFile = 0; efs_init(); efs_chdir_receive(“/tmp/earlab”); Scientific Computing and Visualization

27. Leaky Example: GetParameters do_getparams(EVENT_GETPARM *egp) { efs_download_cond(egp->filename, egp->filename, xfer_done, 0); return(1); } xfer_done(int reqid, int fsid, inr rc, char *file, int tag) { char path[1000]; strcpy(path, “/tmp/earlab/”); strcat(path, file); GotParamFile = 1; Leaky_GetParameters(path); ); Scientific Computing and Visualization

28. Leaky Example: Start do_start(EVENT_START *est) { Logger = est->logger; while (!GotParamFile) event_sleep(100); Leaky_Start(egp->frame_size, egp->sample_rate); return(1); } Scientific Computing and Visualization

29. Leaky Example: LogInfo void LogInfo(char *msg) { EVENT_LOGINFO einfo; strcpy(einfo->msg, msg); event_send(Logger, ET_LOGINFO, &einfo, sizeof(einfo)); } Scientific Computing and Visualization

30. Leaky Example: Advance do_advance(EVENT_ADVANCE *ein) { int bytes; EVENT_ADVANCE *eout; bytes = sizeof(EVENT_ADVANCE) + (ein->size-1)*sizeof(float); eout = (EVENT_ADVANCE *) malloc(bytes); Leaky_Advance(&ein->pressure, &eout->pressure); eout->client = event_clientid(); eout->size = ein->size; event_send(ein->client, ET_ADVANCE, eout, bytes); free(ein); return(1); } Scientific Computing and Visualization

31. DAFFIE Message routing client client client proxy client client event server client client proxy client client Scientific Computing and Visualization