1 / 8

Optimize Job Workflow with Efficient Data Management

Solve issues in job workflow by optimizing data staging, scheduling, and offloading activities, reducing wasted compute time, and enhancing resource usage and data availability. Automate job I/O activities for faster turnaround time and optimized resource allocation.

indiya
Download Presentation

Optimize Job Workflow with Efficient Data Management

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The stagesub tool Sudharshan S. Vazhkudai Computer Science Research Group CSMD Oak Ridge National Laboratory

  2. Problem Space • Users’ job workflow (stage, compute, offload) is mired by numerous issues • Staging and offloading are large data jobs prone to failure • Early staging and late offloading wastes scratch space • Delayed offloading renders result data vulnerable to purging • Compute time wasted on staging/offloading as part of the job

  3. Solution • Recognition that job input and output data needs to be managed more efficiently: in the way it is scheduled • Automatic scheduling of data staging/offloading activities so they can better coincide with computing • Results: • From a center standpoint: • Our techniques optimize resource usage and increase data and service availability • From a user job standpoint: • Our techniques reduce job turnaround time and optimize the usage of allocated time

  4. Coordination between staging, offloading and computation • Motivation • Lack of global coordination between the storage hierarchy and system software • Problems with manual and scripted staging • Human operational cost, wasted compute time/storage and increased wait time due to resubmissions • Our approach • Explicit specification of I/O activities alongside computation in a job script • Zero-charge Data Transfer Queue • Planning and Orchestration

  5. Specification of I/O Activities • Instrumented PBS job script: • #!/bin/bash • #PBS -N ldrdtest • #PBS -l nodes=4 • #PBS -l walltime=00:015:00 • #PBS -q batch • #PBS -A project-account • #PBS -M vazhkudaiss@ornl.gov • #STAGEIN [-d HH:MM:SS] hpssdir://hpss.ccs.ornl.gov/home/vazhkuda/inp [-k /spin/sys/.keytab/vazhkuda.kt] [-p 1217] [-l vazhkuda] file:///ccs/home/vazhkuda/SVLDRD/test/scratch • Multiple stagein directives: from more than one source • Your compute job • #STAGEOUT file:///ccs/home/vazhkuda/SVLDRD/test/scratch/out hpssdir:///home/vazhkuda/inp • Multiple stageout directives: to more than one source • Protocols supported: scp://, hpss://, gsiftp://, file://

  6. Data Transfer Queue • Motivation: • Queuing up and scheduling data transfers • Treats data transfers as “data jobs” • Enables efficient management of data transfers • Specifics: • Zero-charge data transfer queue • Queue operated with the same scheduling policy as compute queue • Queue is managed using access controls to avoid misuse • Added benefit: • Data transfer jobs can now be logged and charged, if need be!

  7. Job Queue Data Queue Orchestration • Parsing into individual stagin, compute and stageout jobs • Dependency setup and management using resource manager primitives: wrapper around qsub Head Node Compute Nodes Job Script 1. Stage Data 2. Compute Job 3. Offload Data Planner I/O Nodes 1 2 after 1 3 after 2

  8. Usage • module load stagesub • man stagesub • stagesub <pbs_script> • References: • Z. Zhang, C. Wang, S. S. Vazhkudai, X, Ma, G. Pike, J.W. Cobb, F. Mueller, “Optimizing Center Performance through Coordinated Data Staging, Scheduling and Recovery”, Proceedings of Supercomputing 2007 (SC07): Int'l Conference on High Performance Computing, Networking, Storage and Analysis, Reno, Nevada, November 2007. • H. Monti, A.R. Butt, S.S. Vazhkudai, “Just-in-time Staging of Large Input Data for Supercomputing Jobs”, Proceedings of Petascale Data Storage Workshop, Supercomputing 2008, Austin, Texas, November 2008.

More Related