1 / 23

PBS Job Management and Taskfarming

PBS Job Management and Taskfarming. Joachim Wagner 2008-07-24. Outline. Why do we need a cluster? Architecture Machines Software Job Management Running jobs Commands PBS Job discriptions Taskfarming Plans. Why do we need a cluster?. Resource conflicts

lewis
Download Presentation

PBS Job Management and Taskfarming

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. PBS Job Management and Taskfarming Joachim Wagner 2008-07-24

  2. Outline • Why do we need a cluster? • Architecture • Machines • Software • Job Management • Running jobs • Commands • PBS Job discriptions • Taskfarming • Plans

  3. Why do we need a cluster? • Resource conflicts • Waiting for colleague’s job to finish • Trouble, e.g. disk full • Medium-size jobs • Too big for desktop PC • Too small for ICHEC • Preparation of ICHEC runs • Learning

  4. Cluster Architecture maia.computing.dcu.ie Nodes SeparateNetwork SchoolNetwork LoginsHomeSoftware Job Queue

  5. Installed Software • OpenMPI • SRILM • MaTrEx, Moses, GIZA++ • XLE, Sicstus • Johnson & Charniak’s reranking parser • In progress: • LFG AA, incl. function labeller

  6. PBS Job Management User: JobDescription Job Submission Job Queue Job Scheduler Nodes are allocatedjob-exclusive for theduration of the job Job Execution

  7. PBS Job Management Commands • qsub myjob.pbs • submits a job • PBS description: shell script with #PBS commands(ignored by shell, see next slide) • qstat, qstat –f jobnumber • qdel jobnumber • pbsnodes –a • list all nodes with status and properties

  8. PBS Job Description Number of nodesCPUs/node Notification: end,begin and abort Maximum runtime

  9. Example: Memory-Intensive Job

  10. Node Properties • min4GB, min8GB: at least this much • mem4GB, mem8GB: exactly this much • long: please use this property for long jobs • will leave nodes that do not have this property available for other jobs • no limit enforced, but 24 h seems reasonable • Future: • 16 and 32 GB • CPU • local disk space (/tmp and swap)

  11. CPU-Intensive Jobs • Parallelisable, for example • Sentence by sentence processing • Cross-validation runs • Parameter search • Split into parts • Run each part on a different CPU

  12. Taskfarming TaskfarmingExecutable(n instances) PBS JobDescription childprocess reading 1 Master n-1 Worker MPICommunication Task file(.tfm):one taskper line Taskexecution

  13. Taskfarming Executable • If Instance ID == 0 • Run master code loop: • Read .tfm file (arg 1) • Send lines to worker • Exit if no more task and all worker finished • Else • Run worker loop: • Ask master for a task • Execute task • Exit if master has no more tasks

  14. Example: Taskfarming PBS File

  15. Example: Taskfarming TFM File

  16. Example: Taskfarming Helper Script run-package.sh

  17. Example: Taskfarming in Action 000 003 006 011 CPU 1 001 005 008 009 012 CPU 2 idle 002 004 007 010 CPU 3 Master: reads .tfm and distributes tasks CPU 4 time

  18. Example: Non-Terminating Task 000 003 006 (does not terminate) CPU 1 idle 001 005 008 009 011 CPU 2 idle 002 004 007 010 012 CPU 3 Master: reads .tfm and distributes tasks CPU 4 Killed atWalltimeLimit

  19. Estimating the PBS Walltime Parameter • Collect durations from test run • Usually high variance of execution time • Long sentences • Parameters • Don’t use #packages x avg. time per package • High risk (~50 %) that more time is needed • Instead: Random sampling with observed package durations: /home/jwagner/tools/walltime.py

  20. Effect of Task Size • Job will wait for last task to finish (or be killed when walltime limit is reached) • What if a task crashes? • Results are incomplete • Next tasks is executed • What if a task does not terminate? • Results are incomplete • Fewer CPUs available for remaining tasks • Overhead of starting tasks

  21. Considering Multiple CPUs per Node • 4 CPU cores per node • 8 GB node -> 2 GB per core • CPUs compete for RAM • Swapping of one task effects the 3 other tasks • Relatively slow CPUs in nodes • Compared to new desktop PCs • Optimise throughput of cluster / EUR • Not: throughput of node or CPU • Depends on application

  22. Plans • Fix sporadic errors of taskfarm.py • re-implentation in C • XML-RPC-based taskfarming • http-based • Run master on maia • Run workers also outside the cluster • Set parameters at runtime • Add more nodes (CNGL) • Install additional software

  23. Questions ? Contact: Joachim Wagner CNGL System Administrator jwagner@computing.dcu.ie (01) 700 6915

More Related