1 / 45

The Ibis model as a paradigm for programming distributed systems

Explore the history of distributed systems, including clusters, grids, clouds, and networked world. Learn about the Ibis system and its application on smartphones.

dconnolly
Download Presentation

The Ibis model as a paradigm for programming distributed systems

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. The Ibis model as a paradigm for programming distributed systems Henri Balbal@cs.vu.nlVrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)

  2. Outline • History of distributed systems • Clusters, grids, clouds, networked world • Programming distributed systems • Driving applications • The Ibis system • Ibis on smartphones

  3. History of the Distributed World – Part I (1980s) • Networks of Workstations (NOWs) • Collections of Workstations (COWs) • Processor pools (Amoeba) • Condor pools • (Beowulf) clusters

  4. Amoeba processor pool (Zoo, 1994)

  5. History of the (more) Distributed World – Part II • Metacomputing (Smarr & Catlett, CACM 1992) • Flocking Condor (Epema, FGCS 1996) • Distributed ASCI Supercomputer (1996 – ?) • Grid Blueprint (Foster & Kesselmann 1998) • Desktop grids, SETI@home (1999)

  6. Design of DAS (1996) (slide from Andy’s ASCI’97 presentation) DAS-1 DAS-2 DAS-3 DAS-4

  7. DAS-3

  8. A real (heterogeneous) Grid

  9. History of the (Modern) Distributed World – Part III • Cloud computing • Infrastructure as a service • Virtualization • Mobile computing • Sensor networks • Smart phones • The Networked World

  10. Problem • How to write high-performance applications for real-world distributed systems? • How to integrate many different resources?

  11. Our approach • Study fundamental underlying problems • … hand-in-hand with realistic applications • … integrate solutions in one system: Ibis ! User Distributed Systems

  12. Fundamental problems • Performance – efficiency on wide-area systems • Heterogeneity – different systems & APIs • Malleability – resources come and go • Fault tolerance - crashes • Connectivity – firewalls, NAT, etc. • Security – very hard

  13. CONGRATULATIONS!! YOU HAVE WON2.5 MEURO!! Dear prof. Tanenbaum, You may not know me,but I’d like to give you2.5 Million Euro to do research. Please come to Brussels to collect the money. Yours sincerely, Mr. V.I. Person #1 Dutch Computer scientist Top security-expert Case study: spam filters

  14. Applications • Scientific applications • Imaging (VUMC, AMOLF) • Bioinformatics (sequence analysis, cell modeling) • Astronomy (data analysis challenge) • Multimedia content analysis • Games and model checking • Semantic web (distributed reasoning)

  15. Multimedia content analysis • Automatically extract information from images & video • Extract feature vectors from images • Describe properties (color, shape) • Data-parallel task on a cluster • Compute on consecutive images • Task-parallelism on a grid

  16. ‘Most Visionary Research’ award at AAAI 2007, (Frank Seinstra et al.) MMCA

  17. Games and Model Checking • Cansolve entire Awari game on wide-area DAS-3 • Needs 10G private optical network (StarPlane) • Distributed model checking has very similar communication pattern • Search huge state spaces, random work distribution, bulk asynchronous transfers • Can efficiently run DeVinE model checker on wide-area DAS-3, use up to 1 TB memory

  18. Awards Astronomy DACH 2008 - BS DACH 2008 - FT SCALE 2008 ISWC 2008 Multimedia Computing Semantic Web (van Harmelen et al.) AAAI-VC 2007

  19. Outline • History of distributed systems • Clusters, grids, clouds, networked world • Programming distributed systems • Driving applications • The Ibis system • Ibis on smartphones

  20. Ibis Philosophy • Real-world distributed applications should be developed and compiled on a local workstation, and simply be launched from there

  21. Ibis Approach • Virtual Machines (Java) deal with heterogeneity • Provide range of programming abstractions • Designed for dynamic/faulty environments • Easy deployment through middleware-independent programming interfaces • Modular and flexible: can replace Ibis components by external ones

  22. Ibis Design • Functionality from programming languages • High-Performance Application Programming System • Functionality fromoperating systems • Distributed Application Deployment System

  23. Ibis System

  24. Programming system • Programming models: • Message passing (RMI, MPJ) • Divide-and-conquer (Satin) • Master-worker (Maestro) • Jorus: (multimedia applications) • IPL (Ibis Portability Layer) • Java-centric “run-anywhere” library • Point-to-point, multicast, streaming, …. • Simple model (Join-Elect-Leave ) for tracking resources, supports malleability & fault-tolerance

  25. SmartSockets library • Detects connectivity problems • Tries to solve them automatically • With as little help from the user as possible • Integrates existing and several new solutions • Reverse connection setup, STUN, TCP splicing, SSH tunneling, smart addressing, etc. • Uses network of hubs as a side channel

  26. Example

  27. Example

  28. Deployment system • IbisDeploy GUI • JavaGAT: • Java Grid Application Toolkit • Make applications independent of underlying middleware • Zorilla P2P system • Jobs management, gossiping,clustering, flood scheduling

  29. Multimedia Content Analysis Ibis (Java) Client • Runs simultaneously on clusters (DAS-3, Japan, Australia), Desktop Grid, Amazon EC2 Cloud • Connectivity problems solved automatically by Ibis SmartSockets Servers Broker

  30. Connection management Standard sockets: only local VU machines can be reached due to firewalls problems With SmartSockets: run everywhere

  31. Ibis movie (part 1)

  32. Performance on 1 DAS-3 cluster • Relative speedups of Java/Ibis and C++/MPI • Using TCP or Myricom’s MX protocol • Sequential performance Java: 80% of C++

  33. Ibis Performance (wide area) • Wide-area DAS: • Frame-rate increases linearly with #clusters from 1 frame/sec to 4 frames/sec • World-wide experiments: 22 frames/sec

  34. Smart Phones • GSM + PC + GPS + camera + networks + …. • Location-aware • What if everyone always carries a smart phone (like a GSM now)? • Next wave in computing?

  35. Ibis on Smart Phones • Our focus: distributed smart phone applications • Applications running on multiple phones • Integration with distributed computing backbone • Use Android for development • Google’s open-source platform • Java-based

  36. Distributed applications • Disaster management (Katrina) • Use ad-hoc Wifi network when GSM network fails • Finding nearby people with certain skills • Bus drivers, CPR • Distributed decision support • Moving people to shelters (logistics) • Social networks • Similar issues • Find nearby friends, decide on restaurant

  37. Yes, on 23 Oct 2010, 3.48 pm atN 52°22.688´ E 004°53.990´ Haven’t we met before? Wild example • Track position => automatic diary of your life • Cross-comparisons between diaries

  38. eyeDentify • Object recognition on a G1 smartphone • Smartphone is a limited device: • Can run only 64 x 48 pixels (memory bound) • 1024 x 768 pixels would take 5 minutes • Distributed Ibis version: = + + 2.0 seconds 1024 x 768 pixels

  39. Ibis movie (part 2)

  40. Interdroid Novel Mobile Distributed Applications Data Management Context Sensitive Programming Models Distributed Communication

  41. Current work • Raven: API for Viable Episodic Networking • Decentralized synchronization API • Fine grained control over data sharing • Bluetooth support for ad-hoc communication • Discovery of devices using multiple networks • Context Aware Programming Models • Supporting distributed decision making • Representing and using context (location etc.) • Exploiting social relationships (Hyves, Facebook)

  42. Summary • It’s a wild (distributed) world

  43. Acknowledgements Niels Drost Ceriel Jacobs Roelof Kemp Timo van Kessel Thilo Kielmann Jason Maassen Rob van Nieuwpoort Nick Palmer Kees van Reeuwijk Frank J. Seinstra Kees Verstoep Gosia Wrzesinska

  44. Big Acknowledgement Andy

  45. Questions?

More Related