Sp3 1 high performance distributed computing
Download
1 / 14

SP3.1: High-Performance Distributed Computing - PowerPoint PPT Presentation


  • 106 Views
  • Uploaded on
  • Presentation posted in: General

SP3.1: High-Performance Distributed Computing. the Ibis Java-centric grid middleware. The KOALA grid scheduler. and. Henri Bal, Thilo Kielmann, Jason Maassen, Rob van Nieuwpoort, et al. Dick Epema Catalin Dumitrescu, Alex Iosup, Hashim Mohamed, Ozan Sonmez. TUDelft: KOALA.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha

Download Presentation

SP3.1: High-Performance Distributed Computing

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


SP3.1: High-Performance Distributed Computing

the Ibis Java-centric

grid middleware

The KOALA

grid scheduler

and

Henri Bal, Thilo Kielmann,

Jason Maassen,

Rob van Nieuwpoort, et al.

Dick Epema

Catalin Dumitrescu, Alex Iosup,

Hashim Mohamed, Ozan Sonmez


TUDelft: KOALA

  • KOALA is a multicluster/grid scheduler

  • Main goals of KOALA:

    • Load sharing of jobs across the sites in a grid:

      • Automatic resource selection

    • Co-allocation of jobs across the sites in a grid:

      • In order to use more resources

      • As dictated by the structure of applications (e.g., simulation/visualization)

  • KOALA has been released on the DAS in september 2005


KOALA: Scheduling

global queue

KOALA

local queues with local schedulers

load sharing

LS

LS

LS

co-allocation

clusters

global job

local jobs


VU: Ibis

  • Ibis: Java-centric grid middleware for distributed supercomputing

  • Satin: divide-and-conquer parallelism in grids

  • GAT: Grid Application Toolkit

  • Implemented several Java applications from

    • SP 1.3 (Medical/Vumc)

    • SP 1.6 (Telescience/AMOLF)

    • SP 2.1 (iPSE/ UvA)

    • SP 2.2 (AID/UvA)


Ibis: Grid’5000 experiments

  • Grid’5000: French computer scienceGrid with 2000 nodes at 9 sites

  • Used Grid’5000 for

    • Running Satin applications

    • Nqueens challenge (2nd Grid Plugtest) Ibis/Satin/GAT application running on 960 nodes

      at 6 sites, ~85% efficiency

    • Large-scale peer-to-peer experiments using Zorilla (Gnutella-like latency-based flooding of ads for joining a compution)


KOALA feature 1: the Runners

  • There are many ugly application types out there

  • No way they can all be supported by a single scheduler

  • Solution: runners (=interface modules)

  • Currently supported:

    • Any type of single-component job

    • MPI/DUROC jobs

    • Ibis jobs

    • HOC applications

runner


KOALA feature 2: the policies

  • Originally supported co-allocation policies:

    • Worst-Fit: balance job components across sites

    • Close-to-Files: take into account the locations of input files to minimize transfer times

  • Different application types require different ways of component placement

  • So:

    • Modular structure with pluggable policies

    • Take into account internal communication structure of applications


KOALA feature 3: support for HOCs

  • Higher-Order Components:

    • Pre-packaged software components with generic patterns of parallel behavior

    • Patterns: master-worker, pipelines, wavefront

  • Benefits:

    • Facilitates parallel programming in grids

    • Enables user-transparent scheduling in grids

  • Most important additional middleware:

    • Translation layer that builds a performance model from the HOC patterns and the user-supplied application parameters

  • Supported by KOALA (with Univ. of Münster)

  • Initial results: up to 50% reduction in runtimes


TUDelft: GrenchMark

  • GrenchMark is a flexible grid workload generator, submitter, and results analyzer

  • Main goals of GrenchMark:

    • Generic workload definition for many types of workloads and application characteristics

    • Grid workload generation

    • Submitting and replaying workloads in different grid settings

  • GrenchMark released in november 2005

  • GrenchMark used to test KOALA


KOALA future (1)

  • Support for more applications types, e.g.,

    • Workflows

    • Parameter sweep applications

  • Communication-aware and application-aware scheduling policies:

    • Take into account the communication pattern of applications when co-allocating

    • Also schedule bandwidth (in DAS3)

  • Better interface KOALA-local schedulers

    • KOALA is too nice


KOALA future (2)

  • Peer-to-peer structure instead of hierarchical grid scheduler

  • Support heterogeneity

    • DAS3

    • DAS2 + DAS3

    • PoC

    • DAS3 + Grid’5000


CPU’s

R

CPU’s

R

CPU’s

R

NOC

CPU’s

R

CPU’s

R

KOALA and

Ibis future

DAS-3


Conclusions

  • SP3.1 is well on track

  • SP3.1 has delivered reliable software tools for everybody to use:

    • KOALA/Grenchmark

    • Ibis/Satin

  • SP3.1 has a bright future

    • Still many research challenges

    • (Access to) great new heterogeneous testbeds


More information

  • Web sites:

    • www.st.ewi.tudelft.nl/koala:

      • general description

      • KOALA tutorial

      • papers

    • grenchmark.st.ewi.tudelft.nl:

      • general description

      • download

      • papers

    • www.cs.vu.nl/ibis:

      • Ibis distribution

      • documentation

      • papers


ad
  • Login