1 / 4

Ganga: Introduction

Ganga: Introduction. Object Orientated Interactive Job Submission System Written in python Based on the concept of a job object Developed by Atlas + LHCb With dedicated (very helpful) support team Used widely - including outside HEP (e.g. biomedics)

soo
Download Presentation

Ganga: Introduction

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Ganga: Introduction • Object Orientated Interactive Job Submission System • Written in python • Based on the concept of a job object • Developed by Atlas + LHCb • With dedicated (very helpful) support team • Used widely - including outside HEP (e.g. biomedics) • Ensures that it is an experiment-neutral framework • All experiment specific extensions are plug-ins. • Very easy automated install • It just takes two commands and < 5 minutes! • Multiple backends • local, psub, condor as well as GRID (glite-wms-*) • A very powerful feature: run the same job as another process on the same machine or on the GRID just by changing its backend attribute • Both standalone and a framework • Is a complete ready to run system for submitting jobs • Also API on which additional features can be layered • Using local backend is great for development before switching to GRID • e.g. MINOS have developed a fault tolerant batch system Computing Lectures

  2. Ganga: The central concept: A job • A job is the central object • Jobs are created, configured, submitted and when complete, examined. • They are persistant • Saved automatically into a Registry • Quit Ganga at any time and come back later and resume • Creation • On creation each gets unique ID • This is a serial number that can be used to access the job • Configuration • At any time can configure all aspects - executable, environment, input and output sandboxes, backend • Cloning • Having configured one job it can be cloned to create others • Submission • Job gets submitted to the configured backend • Polling • A job poller runs in the background and monitors submitted jobs • It reports state changes. • When job is complete it retrieves output and places in the job’s output directory • Resubmit • If a job fails it can be resubmitted • Removal • Once a job is no longer needed it can be removed along with its input and output directories • Global summaries • Get one line per job summary either of all jobs or just a slice (by ID or some other attribute). Computing Lectures

  3. Ganga: Two Interfaces Command line • Example • To create job, configure, submit, check status and examine output:-my_job = Job()my_job.application = Executable(exe=File('~/somescript'), args=['1','2','3'])my_job.backend = LCG() my_job.submit()my_job.statusmy_job.peek(‘stdout’) • Ganga comes with ipython • help with objects - they list methods and state e.g.: help(my_job) • tab completion - just type start of data or function and hit tab e.g. my_job.p <tab> gives my_job.peek GUI Computing Lectures

  4. Ganga: Other Features • Subjobs • Have a master job and a dataset it is to operate on • Job Splitter creates subjobs each taking part of the dataset • Submitting the master job submits all the subjobs • Subjobs can also be accessed individually e.g. fix and resubmit errors. • When all subjobs complete Job Merger recombines the output • Trees • Initially job repository is a flat structure • Can create folder hierarchy each with own jobs • Can be used to perform global operations on collections of jobs • Templates • Configurable like jobs but cannot be submitted • Used to create preconfigured jobs • Authentication management • GRID certificate – will ask to re-authenticate when expired • AFS token • Configuration of all aspects • By resource file read on start up • Interactively during execution • Verbosity control • By level (CRITICAL, ERROR, WARNING, INFO, and DEBUG.) • By specific parts of GANGA Computing Lectures

More Related