1 / 55

Introduction to the Linux Command Line for High-Performance Computing

Introduction to the Linux Command Line for High-Performance Computing. Dr. Charles J Antonelli LSAIT ARS October, 2014. Roadmap. Linux overview T he command shell N avigating the filesystem B asic commands & wildcarding S hell redirection & pipelining Editing text files P ermissions

thane
Download Presentation

Introduction to the Linux Command Line for High-Performance Computing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Introduction to theLinux Command Line for High-Performance Computing Dr. Charles J Antonelli LSAIT ARSOctober, 2014

  2. Roadmap • Linux overview • The command shell • Navigating the filesystem • Basic commands & wildcarding • Shell redirection & pipelining • Editing text files • Permissions • Processes cja 2014

  3. Poll • How many here to learn the command line • For HPC, specifically Flux, work? • For other computing cluster work? • For desktops/other Linux/Unix/Mac OS X? • Other? cja 2014

  4. Course Text William E Shotts, Jr.,“The Linux Command Line: A Complete Introduction,”No Starch Press, January 2012.Download Creative Commons Licensed version athttp://downloads.sourceforge.net/project/linuxcommand/TLCL/13.07/TLCL-13.07.pdf. cja 2014

  5. Linux Overview cja 2014

  6. A compute node 48 GB RAM 12 Intel cores Local disk Network cja 2014

  7. cja 2014 http://linuxforums.org.uk/index.php?action=media;sa=item;in=262

  8. The command shell cja 2014

  9. The command line • A basic way of interacting with a Linux system • Execute commands • Create files and directories • Edit file content • Access the web • Copy files to and from other hosts • Run HPC jobs • … do things you can’t do from the conventional point-and-click Graphical User Interface (GUI) cja 2014

  10. Why command line? • Linux was designed for the command line • You can create new Linux commands using the command line, without programming • Many systems provide only the command line, or poorly support a GUI interface • Such as most HPC systems • Many things can be accomplished only through the command line • Much systems administration & troubleshooting • You want to be cool cja 2014

  11. The command shell • The command shell is an application that reads command lines from the keyboard and passes them to the Linux operating system to be executed • When you login to a remote Linux system using a tool like ssh, you will automatically be connected to a shell • On your desktop, laptop, or tablet, you may have to find and execute a terminal emulator application to bring up a shell in a window cja 2014

  12. The command shell cja 2014 http://www.livefirelabs.com/208_global/images/unix-operating-system.gif

  13. Types of shells • Standard Linux shell (used in this course) • bash “Bourne-again” shell • Other shells • sh Bourne shell, original AT&T Unix shell • csh “C shell”, common on older Unix systems • tcsh “Enhanced C shell”, better version of csh • What shell am I using? echo $SHELL cja 2014

  14. Connecting via ssh • Terminal emulators • Linux and Mac OS X • Start Terminal • Use sshcommand • Windows • U-M PuTTY/WinSCP (U-M Blue Disc)https://www.itcs.umich.edu/bluedisc/ • PuTTYhttp://www.chiark.greenend.org.uk/~sgtatham/putty/ • SSH Secure Shell (deprecated) cja 2014

  15. Lab 0 Task: Start a local shell on Mac OS X • Login to Mac OS X with your uniqname and Kerberos password • If there is a Terminal icon in the Dock , double-click itOtherwise: • Bring up a Finder window:Click on the desktop, type Command-N • Start the Terminal Application:In the Finder window, click on Applications on the left, scroll down on the right until you find the Utilities folder, double-click on the Utilities folder, scroll down on the right until you find the Terminal application, double-click it • This creates a Terminal window with a shell running inside it • From a Terminal window, Command-N will start a new Terminal cja 2014

  16. Lab 0 cja 2014

  17. The shell prompt • The “~$ “ is the shell prompt • This means the shell is waiting for you to type something • Format can vary, usually ends with “$” , “%” or “#” • If $ or %, you have a normal shell • This shell has your privileges • If #, you have a so-called “root shell” • This shell has administrator privileges • You can do a great deal of irreversible damage cja 2014

  18. Typing into the shell • Basic input line editing commands • Backspace erases previous character • Left and right arrow move insertion point on the line • Control-U erases the line so you can start over • Enter executes the line you typed • Control-C interrupts whatever command you started and returns you to the shell prompt (usually) • Up and downarrow will access your command history • Type “exit” without the quotes to exit the shell • Click the red "close" icon at the top of the Terminal window to close it cja 2014

  19. Lab 1 Task: Enter some basic commands ~$ date ~$ id ~$ ps ~$ df -kh ~$ who ~$ top # type Control-C to exit cja 2014

  20. Interlude: Flux cja 2014

  21. Flux • Flux is a university-wideshared computational discovery / high-performance computing service. • Interdisciplinary • Provided by Advanced Research Computing at U-M (ARC) • Operated by CAEN HPC • Hardware procurement, software licensing, billing support by U-M ITS • Used across campus • Collaborative since 2010 • Advanced Research Computing at U-M (ARC) • College of Engineering’s IT Group (CAEN) • Information and Technology Services • Medical School • College of Literature, Science, and the Arts • School of Information http://arc.research.umich.edu/resources-services/flux/ cja 2014

  22. The Flux cluster Login nodes Compute nodes Data transfernode Storage … cja 2014

  23. A Flux node 48 GB RAM 12 Intel cores Local disk Network cja 2014

  24. Logging in to Flux • ssh flux-login.engin.umich.edu • MToken (or Software Token) required • You will be randomly connected a Flux login node • Currently flux-login1 or flux-login2 • Firewalls restrict access to flux-login.To connect successfully, either • Physically connect your ssh client platform to the U-M campus wired or MWireless network, or • Use VPN software on your client platform, or • Use ssh to login to an ITS login node (login.itd.umich.edu), and ssh to flux-login from there cja 2014

  25. A shell on a login node • Now you have a shell on the login node • Your shell prompt has changed to something like[flux-login1 ~]$ • Try those simple commands here! • Flux is for HPC work • The login nodes are reserved for copying data, editing and compiling programs and scripts, and short test runs • For HPC computation, you should run on the compute nodes • A batch job will always run on the compute nodes • An interactive batch job will get you a shell on a compute node cja 2014

  26. A shell on a compute node • Getting a shell on a compute node • Enter this command on the login node:[flux-login1 ~]$ ~cja/hpc100/getshell • After a short delay, you’ll get an interactive shell:qsub: waiting for job 11147004.nyx.engin.umich.edu to startqsub: job 11147004.nyx.engin.umich.edu ready[nyx5509 ~]$ • You can type commands into this shell for all of our in-class experiments • Type “exit” to exit this shell • After two hours, your interactive shell will be killed cja 2014

  27. Navigating the filesystem cja 2014

  28. Linux Filesystem Concepts • Files are stored in a directory • Directories may contain other directories as well as files • A hierarchy of these directories is called a directory tree • A directory tree (a connected graph with no cycles) has a single topmost root directory • A directory tree rooted at the system root directory “/” is called a filesystem cja 2014

  29. cja 2014 http://redhatlinux4u.blogspot.com/2011/11/file-hierarchy-system.html

  30. Linux Filesystem Concepts • A file is accessed using its path name • Absolute path name • /dir1/dir2/…/dirn/filename • /usr/X11R6/bin • Relative path name • current-working-directory/filename • bin • Every shell maintains a notion of a current working directory • Initialized at login to your home directory • Changedvia cd command • Two special directories • . refers to the current directory • .. refers to the current directory’s parent directory cja 2014

  31. Basic commands cja 2014

  32. Prerequisites Some fundamental commands: ~$ file file # what kind of file is file? ~$ cat file # display contents of text file ~$ less file # paginate text file ~$ man command # get info about command Exercise: figure out how to make the date command display the date in Coordinated Universal Time (UTC) cja 2014

  33. Lab 2 Task: navigate the file system Commands: ~$cd # make your home directory the current working directory ~$pwd # print working directory ~$ mkdirfoo # create directory foo ~$ cd foo # cd to the foo directory ~$ mkdir bar # create directory bar ~$ cd .. # cd back to the foo directory ~$ tree # display foo’s directory tree cja 2014

  34. Navigating the filesystem Some fundamental commands: ~$ pwd # print working directory ~$ cd dir# makedirthe current working directory ~$ cd # cd to your home dir ~$ cd ~cja# cd to cja’s home dir ~$ mkdirdir# create directory dir ~$ rmdirdir# remove (empty) directory dir ~$ tree # display dir tree cja 2014

  35. Listing info on files ls – list information about files ~$ ls # list contents of cur dir ~$ lsdir # list contents of dir ~$ ls –l # list details of files in cur dir including access, owner & group, size, and last-modified time ~$ ls–t # list newest files first ~$ ls –R dir # list all files in tree dir ~$ ls–ltdir # options can be combined cja 2014

  36. Working with files These commands manipulate files ~$ mv foo bar # rename file foo to bar ~$ cp foo bar # copy file foo to bar ~$ cp–r dir1 dir2 # copy dir tree dir1 to dir2 ~$ cp f1 f2 dir # copy f1 and f2 to directory dir ~$ mkdirdir # create empty directory dir ~$ rmdirdir # remove empty directory dir ~$ rmfile # remove file file ~$rm –r dir # remove directory tree dir cja 2014

  37. Lab 3 Exercise: Create a directory named tutorialin your home directory. In that directory, create a directory named sampleand a directory named test . Create a file named msgin directory testthat contains a copy of the file/etc/motd. Extra credit: Make the last-modified time of your copy identical to that of /etc/motd. Hint: look at the options of the copy command cja 2014

  38. Compressing and archiving These commands compress and archive files ~$ gzip foo # compress foo to foo.gz ~$ gunzip foo # uncompressfoo.gz to foo ~$ bzip2 foo# better compress foo to foo.bz2 ~$ bunzip2 foo# uncompressfoo.bz2 to foo ~$ tar cffoo.tar bar # archive subtree bar in file foo.tar ~$tar xffoo.tar# restore archive from file foo.tar ~$tar zcffoo.tgz bar # archive and compress ~$tar jcffoo.tjzbar # archive and compress better cja 2014

  39. Wildcards The shell accepts wildcarded arguments This is also called "shell globbing" Wildcards: ? Matches a single character * Matches zero or more characters [chars] Matches any of the chars [c1-c2] Matches chars ‘c1’ through ‘c2’ [^chars] Matches any but the chars ~$ ls foo.? # match files named foo.x, where x is any character ~$ echo *.[cs] # echo files that end in .c or .s ~$ mv [o-z]* save # move files starting with o through z to directory save ~$ echo [^A-Z]? # ??? cja 2014

  40. Shell redirection & pipelining cja 2014

  41. Shell redirection A Linux command can have its inputs and outputs redirected ~$ ls >foo # put list of files in current directory into file foo ~$ ls>>bar # add list of files in current directory to end of file foo ~$ sort <bar # sort lines from file bar ~$ sort <<EOF # sort lines entered at keyboardwhiskeybravotangoEOF ~$ wc -l </etc/motd >~/mycounts # count number of lines from file /etc/motd and put result in filemycounts in my home directory cja 2014

  42. More Linux commands More useful Linux tool commands ~$ grepstring # show lines of input containing string ~$ tail # show last few lines of input ~$ head # show first few lines of input ~$ sort # sort the input cja 2014

  43. Shell pipelining A Linux command can have its output connected to the input of another Linux command ~$ ls | wc –l # count files in current directory ~$ last | grep reboot# when did we reboot? Exercise: How many people are running MATLAB on your compute node? cja 2014

  44. Editing text files cja 2014

  45. Editing text files • Simple editor • nano or pico • "What you see is what you get” editor • Simple to learn if you want to get started quickly • Supported editors • vi or vim • emacs • Powerful but more complex • If you have time and inclination to become proficient, spend time here cja 2014

  46. Text files • Watch out for source code or data files written on Windows systems • Use these tools to analyze and convert source files to Linux format • file • dos2unix cja 2014

  47. Permissions cja 2014

  48. File Permissions • Three permission bits, aka mode bits • Files: Read, Write, Execute • Directories: List, Modify, Search • Three user classes • User (File Owner), File Group, Other • man chmod cja 2014

  49. File Permissions, examples -rw------- cjalsait 40 Oct 1 12:03 foo.bz2file readand write rights for the owner, no access for anyone else chmod u=rw,g=r,o= file -rw-r----- cjalsait 40 Oct 1 12:03 foo.bz2file read and write rights for the owner, read for members of the lsait group, no access and for others drwxr-x--x cjalsait 4096 Oct 1 12:15 barlist, modify, and search for the owner, list and search for group, and execute only for others cja 2014

  50. Processes cja 2014

More Related