Pegasus planning for execution in grids
This presentation is the property of its rightful owner.
Sponsored Links
1 / 17

Pegasus: Planning for Execution in Grids PowerPoint PPT Presentation


  • 87 Views
  • Uploaded on
  • Presentation posted in: General

Pegasus: Planning for Execution in Grids. Ewa Deelman, Carl Kesselman, Gaurang Mehta, Gurmeet Singh, Karan Vahi Information Sciences Institute University of Southern California. Pegasus Acknowledgement.

Download Presentation

Pegasus: Planning for Execution in Grids

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Pegasus planning for execution in grids

Pegasus: Planning for Execution in Grids

Ewa Deelman, Carl Kesselman, Gaurang Mehta, Gurmeet Singh, Karan Vahi

Information Sciences Institute

University of Southern California


Pegasus acknowledgement

Pegasus Acknowledgement

  • Ewa Deelman, Carl Kesselman, Saurabh Khurana, Gaurang Mehta, Sonal Patil, Gurmeet Singh, Mei-Hui Su, Karan Vahi (ISI)

  • James Blythe, Yolanda Gil (ISI)

  • http://pegasus.isi.edu

  • Research funded as part of the NSF GriPhyN, NVO and SCEC projects.


Grid applications

Grid Applications

  • Increasing in the level of complexity

  • Use of individual application components

  • Reuse of individual intermediate data products

  • Description of Data Products using Metadata Attributes

  • Execution environment is complex and very dynamic

    • Resources come and go

    • Data is replicated

    • Components can be found at various locations or staged in on demand

  • Separation between

    • the application description

    • the actual execution description


Pegasus planning for execution in grids

Abstract Workflow Generation

Concrete

Workflow Generation


Pegasus planning for execution in grids1

Pegasus:Planning for Execution in Grids

  • Maps from abstract to concrete workflow

    • Algorithmic and AI based techniques

  • Automatically locates physical locations for both components (transformations) and data

    • Use Globus RLS and the Transformation Catalog

  • Finds appropriate resources to execute

    • via Globus MDS

  • Reuses existing data products where applicable

  • Publishes newly derived data products

    • Chimera virtual data catalog


Pegasus planning for execution in grids

Chimera is developed at ANL

By I. Foster, M. Wilde, and J. Voeckler


Ligo scientific collaboration

LIGO Scientific Collaboration

  • Continuous gravitational waves are expected to be produced by a variety of celestial objects

  • Only a small fraction of potential sources are known

  • Need to perform blind searches, scanning the regions of the sky where we have no a priori information of the presence of a source

    • Wide area, wide frequency searches

  • Search is performed for potential sources of continuous periodic waves near the Galactic Center and the galactic core

  • The search is very compute and data intensive

  • LSC is using the occasion of SC2003 to initiate a month-long production run with science data collected during 8 weeks in the Spring of 2003


Pegasus planning for execution in grids

Additional resources used: Grid3 iVDGL resources

Thanks to everyone involved in standing up the tested and contributing the resources!


Ligo acknowledgements

LIGO Acknowledgements

  • Bruce Allen, Scott Koranda, Brian Moe, Xavier Siemens, University of Wisconsin Milwaukee, USA

  • Stuart Anderson, Kent Blackburn, Albert Lazzarini, Dan Kozak, Hari Pulapaka, Peter Shawhan, Caltech, USA

  • Steffen Grunewald, Yousuke Itoh, Maria Alessandra Papa, Albert Einstein Institute, Germany

  • Many Others involved in the Testbed

  • www.ligo.caltech.edu

  • www.lsc- group.phys.uwm.edu/lscdatagrid/

  • http://pandora.aei.mpg.de/merlin/

  • LIGO Laboratory operates under NSF cooperative agreement PHY-0107417


Montage

Montage

  • Montage (NASA and NVO)

    • Deliver science-grade custom mosaics on demand

    • Produce mosaics from a wide range of data sources (possibly in different spectra)

    • User-specified parameters of projection, coordinates, size, rotation and spatial sampling.

Mosaic created by Pegasus based Montage from a run of the M101 galaxy images on the Teragrid.


Small montage workflow

Small Montage Workflow


Montage acknowledgments

Montage Acknowledgments

  • Bruce Berriman, John Good, Anastasia Laity, Caltech/IPAC

  • Joseph C. Jacob, Daniel S. Katz, JPL

  • http://montage.ipac. caltech.edu/

  • Testbed for Montage: Condor pools at USC/ISI, UW Madison, and Teragrid resources at NCSA, PSC, and SDSC.

    Montage is funded by the National Aeronautics and Space Administration's Earth Science Technology Office, Computational Technologies Project, under Cooperative Agreement Number NCC5-626 between NASA and the California Institute of Technology.


Current system

Current System


Just in time planning

Just In-time planning

  • Partition Abstract workflow into partial workflows


Meta dagman

Meta-DAGMan


Other applications using pegasus

Other Applications Using Pegasus

  • Other GriPhyN applications:

    • High-energy physics: Atlas, CMS (many)

    • Astronomy: SDSS (Fermi Lab, ANL)

  • Astronomy:

    • Galaxy Morphology (NCSA, JHU, Fermi, many others, NVO-funded)

  • Biology

    • BLAST (ANL, PDQ-funded)

  • Neuroscience

    • Tomography (SDSC, NIH-funded)

  • http://pegasus.isi.edu

  • Funding by NSF GriPhyN, NSF NVO, NIH


  • Login