Sp3 1 high performance distributed computing
This presentation is the property of its rightful owner.
Sponsored Links
1 / 14

SP3.1: High-Performance Distributed Computing PowerPoint PPT Presentation


  • 87 Views
  • Uploaded on
  • Presentation posted in: General

SP3.1: High-Performance Distributed Computing. the Ibis Java-centric grid middleware. The KOALA grid scheduler. and. Henri Bal, Thilo Kielmann, Jason Maassen, Rob van Nieuwpoort, et al. Dick Epema Catalin Dumitrescu, Alex Iosup, Hashim Mohamed, Ozan Sonmez. TUDelft: KOALA.

Download Presentation

SP3.1: High-Performance Distributed Computing

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Sp3 1 high performance distributed computing

SP3.1: High-Performance Distributed Computing

the Ibis Java-centric

grid middleware

The KOALA

grid scheduler

and

Henri Bal, Thilo Kielmann,

Jason Maassen,

Rob van Nieuwpoort, et al.

Dick Epema

Catalin Dumitrescu, Alex Iosup,

Hashim Mohamed, Ozan Sonmez


Tudelft koala

TUDelft: KOALA

  • KOALA is a multicluster/grid scheduler

  • Main goals of KOALA:

    • Load sharing of jobs across the sites in a grid:

      • Automatic resource selection

    • Co-allocation of jobs across the sites in a grid:

      • In order to use more resources

      • As dictated by the structure of applications (e.g., simulation/visualization)

  • KOALA has been released on the DAS in september 2005


Koala scheduling

KOALA: Scheduling

global queue

KOALA

local queues with local schedulers

load sharing

LS

LS

LS

co-allocation

clusters

global job

local jobs


Vu ibis

VU: Ibis

  • Ibis: Java-centric grid middleware for distributed supercomputing

  • Satin: divide-and-conquer parallelism in grids

  • GAT: Grid Application Toolkit

  • Implemented several Java applications from

    • SP 1.3 (Medical/Vumc)

    • SP 1.6 (Telescience/AMOLF)

    • SP 2.1 (iPSE/ UvA)

    • SP 2.2 (AID/UvA)


Ibis grid 5000 experiments

Ibis: Grid’5000 experiments

  • Grid’5000: French computer scienceGrid with 2000 nodes at 9 sites

  • Used Grid’5000 for

    • Running Satin applications

    • Nqueens challenge (2nd Grid Plugtest) Ibis/Satin/GAT application running on 960 nodes

      at 6 sites, ~85% efficiency

    • Large-scale peer-to-peer experiments using Zorilla (Gnutella-like latency-based flooding of ads for joining a compution)


Koala feature 1 the runners

KOALA feature 1: the Runners

  • There are many ugly application types out there

  • No way they can all be supported by a single scheduler

  • Solution: runners (=interface modules)

  • Currently supported:

    • Any type of single-component job

    • MPI/DUROC jobs

    • Ibis jobs

    • HOC applications

runner


Koala feature 2 the policies

KOALA feature 2: the policies

  • Originally supported co-allocation policies:

    • Worst-Fit: balance job components across sites

    • Close-to-Files: take into account the locations of input files to minimize transfer times

  • Different application types require different ways of component placement

  • So:

    • Modular structure with pluggable policies

    • Take into account internal communication structure of applications


Koala feature 3 support for hocs

KOALA feature 3: support for HOCs

  • Higher-Order Components:

    • Pre-packaged software components with generic patterns of parallel behavior

    • Patterns: master-worker, pipelines, wavefront

  • Benefits:

    • Facilitates parallel programming in grids

    • Enables user-transparent scheduling in grids

  • Most important additional middleware:

    • Translation layer that builds a performance model from the HOC patterns and the user-supplied application parameters

  • Supported by KOALA (with Univ. of Münster)

  • Initial results: up to 50% reduction in runtimes


Tudelft grenchmark

TUDelft: GrenchMark

  • GrenchMark is a flexible grid workload generator, submitter, and results analyzer

  • Main goals of GrenchMark:

    • Generic workload definition for many types of workloads and application characteristics

    • Grid workload generation

    • Submitting and replaying workloads in different grid settings

  • GrenchMark released in november 2005

  • GrenchMark used to test KOALA


Koala future 1

KOALA future (1)

  • Support for more applications types, e.g.,

    • Workflows

    • Parameter sweep applications

  • Communication-aware and application-aware scheduling policies:

    • Take into account the communication pattern of applications when co-allocating

    • Also schedule bandwidth (in DAS3)

  • Better interface KOALA-local schedulers

    • KOALA is too nice


Koala future 2

KOALA future (2)

  • Peer-to-peer structure instead of hierarchical grid scheduler

  • Support heterogeneity

    • DAS3

    • DAS2 + DAS3

    • PoC

    • DAS3 + Grid’5000


D a s 3

CPU’s

R

CPU’s

R

CPU’s

R

NOC

CPU’s

R

CPU’s

R

KOALA and

Ibis future

DAS-3


Conclusions

Conclusions

  • SP3.1 is well on track

  • SP3.1 has delivered reliable software tools for everybody to use:

    • KOALA/Grenchmark

    • Ibis/Satin

  • SP3.1 has a bright future

    • Still many research challenges

    • (Access to) great new heterogeneous testbeds


More information

More information

  • Web sites:

    • www.st.ewi.tudelft.nl/koala:

      • general description

      • KOALA tutorial

      • papers

    • grenchmark.st.ewi.tudelft.nl:

      • general description

      • download

      • papers

    • www.cs.vu.nl/ibis:

      • Ibis distribution

      • documentation

      • papers


  • Login