1 / 1

Clustering Workflows for Best Effort Systems Utilizing Pegasus

This paper discusses the implementation of automatic node clustering in Montage workflows for best effort systems, leveraging Pegasus on TeraGrid and USC resources. A structured approach is presented, showcasing the execution of over half a million tasks and the management of substantial data volumes, including CyberShake workflows with 100,000 nodes and LIGO workflows with 185,000 nodes. The research highlights the efficiency gains from employing level-based clustering techniques and the achievements of these high-complexity workflows in astrophysical data processing.

rae-fields
Download Presentation

Clustering Workflows for Best Effort Systems Utilizing Pegasus

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Workflow Task Clustering for Best Effort Systems with Pegasus pegasus.isi.edu Gurmeet Singh, Mei-Hui Su, Karan Vahi Ewa Deelman, Gaurang Mehta Information Sciences Institute University of Southern California Marina del Rey, CA 90292 Bruce Berriman, John Good Infrared Processing and Analysis Center California Institute of Technology Pasadena, CA 91125 Daniel S. Katz Center for Computation and Technology Louisiana State University Baton Rouge, LA 70803 A view of the Rho Oph dark cloud constructed with Montage from deep exposures made with the Two Micron All Sky Survey (2MASS) Extended Mission Automatic Node clustering The structure of a small Montage workflow Two clusters per level Two tasks per cluster 1 degree2 Montage On TeraGrid Level-based, clustering factor 5 No clustering SCEC CyberShake workflows run using Pegasus and DAGMan on the TeraGrid and USC resources Cumulatively, the workflows consisted of over half a million tasks and used over 2.5 CPU Years. The largest CyberShake workflow contained on the order of 100,000 nodes and accessed 10TB of data Support for LIGO on Open Science Grid LIGO Workflows: 185,000 nodes, 466,000 edges 10 TB of input data, 1 TB of output data.

More Related