1 / 19

Using the High Performance Computing Cluster at TSL

Using the High Performance Computing Cluster at TSL. Graham Etherington TSL Bioinformatics Support Officer. What is the HPC Cluster?. Like a large collection high-memory computers. Controlled by management software (LSF) which allocates tasks to each computer (node). Access node. LSF.

kiora
Download Presentation

Using the High Performance Computing Cluster at TSL

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Using the High Performance Computing Cluster at TSL Graham Etherington TSL Bioinformatics Support Officer

  2. What is the HPC Cluster? • Like a large collection high-memory computers. • Controlled by management software (LSF) which allocates tasks to each computer (node). Access node LSF Hard disk storage

  3. HPC Node Resources • Shared with TGAC • Currently: • 29 x 128Gb nodes • 4 x 256Gb nodes • Planned: • 88 x 128Gb nodes • 9 x 256Gb nodes

  4. What does the HPC allow us to do? • Send computing tasks somewhere else • leave your PC free for other tasks. • Carry out memory-intensive jobs • 4GB RAM on PC, up to 256GB RAM on cluster. • Install programs once, use by everyone.

  5. Logging in to the HPC • Open a terminal window and type: $ sshhpc.tsl.ac.uk • You’ll be asked for your password • enter your current NBI password. When your NBI password changes, it will also change on the cluster. • The first time you log-in your account will be created.

  6. The log-in screen Displays information for: The number of jobs you have running. The number of jobs you have queued. The total number of jobs running on the cluster. The total number of jobs queued on the cluster.

  7. Task 1 • Start up your computers • Log on using the username on the yellow tag on your computer • Password – “Learning26” • Complete Task 1 in the tutorial.

  8. Where stuff is • Your home-directory /usr/users/sl/<username> shortcut: ~ • Data for everyone (blastdbs, indexes, reference sequences, external read sets, genome annotations,etc) /usr/users/sl/data/ • Programs /tsl/software/testing/

  9. Task 2 • Complete Task 2 in the tutorial

  10. Submitting jobs $ bsub-q TSL-Test128 -We 5 "echo hello” • Breaking it down: • bsub = submit the following job • -q = use the following queue on the cluster • -We = estimated run time of the job (in minutes) • “echo hello” – the task we want to run (in quotes) • What queues are available? • TSL-Test128 • TSL-Test256 • TSL-Prod128 • TSL-Prod256

  11. Submitting jobs • How do I run my own script? $ bsub-q TSL-Test128 -We 5 "source perl-5.16.3; perlSqrtTwoNumbers.pl 1 2” • ‘source’? What’s that about? • stipulate which version of a program to run (e.g. “source bwa-0.7.3”, “source bwa-0.7.4”, etc.) • often only one choice, but you still need to source it.

  12. Submitting jobs • Programs are always installed in the ‘Testing’ environment. • /tsl/software/testing/ • Strict directory-naming scheme • /tsl/software/testing/<program_name>/<version> • e.g. • /tsl/software/testing/bwa/0.7.3/ • /tsl/software/testing/bwa/0.7.4/

  13. Submitting jobs • How do I run an installed program? e.g. bsub -q TSL-Test128 -We 5 "source <program>-<version>; <program> [opts]” $ bsub-q TSL-Test128 -We 5 "source bwa-0.7.4; bwa index ref.fa” • Some things to note • hyphen between program and version • semi-colon after source

  14. Task 3 • Complete Task 3 in the tutorial.

  15. Submitting jobs $ bsub–q TSL-Test128 –We 10 “source bwa-0.7.4; bwamem -t 4 -c 5000 -O 5 -P -B 5 -U 10 ref.faleft.fqright.fq > map.sam” • Make life easier by using a bash script • store the commands and parameters for commonly used programs run_bwa.sh #! /bin/bash sourcebwa-0.7.4; bwamem -t 4 -c 5000 -O 5 -P -B 5 -U 10 $1 $2 $3 > $4; $ bsub–q TSL-Test128 –We 10 “./run_bwa.shref.faleft.fqright.fqmap.sam” • Edit the bash script to change the parameters.

  16. Task 4 • Complete Task 4 in the tutorial.

  17. Installing software • Remember: Strict directory-naming scheme • /tsl/software/testing/<program_name>/<version> • Inside each <version> there will be two sub-directories ‘src’ and ‘x86_64/bin/’ • ‘src’ is for source code and ‘x86_64/bin/’ for binaries (the actual executables). • After installation, a wrapper needs to be created. $ create-software-testing-wrapper <program_name>-<version>

  18. Which queue? • Testing • newly installed software. • a script you’re developing or running for the first time. • Production • ‘Stable’ software and scripts.

  19. Task 5 • Complete Task 5 in the tutorial

More Related