Introduction to Linux

1. Introduction to Linux Data Analysis Group • 28th June 2019

2. The Data Analysis Group James Abbott Bioinformatics/Data Analysis Group Manager Background: Genomes, variants, interface and workflow development • Provide advice and assistance on • Experimental design • Programming • Statistics • Bioinformatics • Data analysis • Collaborate on projects • Funded through joint grant application • 'Pay-as-you-go' hourly rates • http://dag.compbio.dundee.ac.uk Marek Gierlinski Bioinformatician Background: Statistics, RNA-Seq, CHiP-Seq, Lapsed astrophysicist§ William Nicholson PRDA – Bioinformatics (co-supervised with Prof. Sara Brown) Background: Genomic data analysis, image processing and many things in between

3. What is Unix • Multi-user operating system • Acts as interface between user and hardware • Provides secure access to resources • Original release 1969, AT & T labs • Linux: Free Unix implementation by Linux Torvalds, 1991 • MacOS: Based upon BSD Unix implementation • A complex history...

4. Unix History (simplified... ) By Eraserhead1, Infinity0, Sav_vas - Levenez Unix History Diagram, Information on the history of IBM's AIX on ibm.com, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=1801948

5. Linux • Linux can be found running on • Servers • HPC systems • Desktops • Laptops • Mobile Devices • Today we will be connecting to a linux server over the network

6. Key to slides…. • Descriptive text • Example commands – black monospaced text: ls /tmp • Commands to try – red monospaced text • Prompts/STDOUT/STDERR –green monospaced text • e.g. -bash-4.1 $ echo $USER jabbott • [ENTER] – press enter key • [username] – your username, not '[username]'

7. Logging in

8. Requirements • Account credentials • An SSH client PuTTY Terminal and OpenSSH • An X11 server (optional, for GUI tools) Xming Xquartz XFree86

9. Finding Software on Training Machines • Software made available through ‘Apps Anywhere’ • Login with UoD credentials • Type name of software in ‘Search Apps’ field • Mouse-over package and click ‘Launch’

10. WINDOWS - starting Xming Start Xming Step through series of preferences dialogs Nothing obviously running but…

11. WINDOWS - PuTTY Start PuTTY: Enter ‘login.cluster.lifesci.dundee.ac.uk’ in ‘Host Name’ field Open ‘SSH’ menu, select ‘X11’ and check ‘Enable X11 Forwarding’ Select ‘Session’ (at top) • Normally: Type ‘compbio’ in ‘Saved Sessions’ field and click ’Save’ button • Don’t do this now – it won’t work on training machines • Click ‘Open’ button Acknowledge security alert Enter username followed by ‘Enter’ Enter password followed by ‘Enter’

12. MACOS/Linux • Start XQuartz:Finder -> Applications -> Utilities -> Xquartz • Open a terminal session • Finder -> Applications -> Utilities -> Terminal •  Terminal/Konsole icon in utilities group • In terminal window, type: ssh -X -l [username] login.cluster.lifesci.dundee.ac.uk Orssh -X [username]@login.cluster.lifesci.dundee.ac.uk Your username!

13. First time login… • LS27911:~ jabbott$ • The authenticity of host 'login.cluster.lifesci.dundee.ac.uk (::1)' can't be established. • ECDSA key fingerprint is SHA256:nJ2BGgjZcKLewMnQPEgoih4K+mTcCX+mLyGfbwDFCNg. • Are you sure you want to continue connecting (yes/no)? • Warning: Permanently added 'login.cluster.lifesci.dundee.ac.uk' (ECDSA) to the list of known hosts. • Password: • ningal:~ $ ssh -X -l jabbottlogin.cluster.lifesci.dundee.ac.uk yes

14. What am I looking at? • ningal:~ $ • This is the command prompt • Anything you type appears after the '$' character • Prompts can be customised • Common format: • LS27911:~ jabbott$ Current WorkingDirectory Hostname Username

15. Entering Commands

16. Running a Simple Command • Linux programs are run by typing commands • The command is the name of a program available on the computer • Try these as we go along… • Linux is case-sensitive...'date' is not the same as 'Date' • After typing the command, press to run it... ningal:~ $ Tue 27 Feb 2018 13:34:40 GMT • ls command lists contents of current working directory. date

17. Some example files… • Type the following, pressing ‘enter’ after each line: • ningal:~ $ • ningal:~ $ • ningal:~ $ • You should see something like: • /homes/jabbott/Intro_to_linux • ningal:~ $ • list1.txt list2.txt list3.txt cp –R /tmp/Intro_to_linux . cd Intro_to_linux pwd ls

18. Command Arguments • An argument to a command provides data for the command to use • Commonly a filename • Arguments follow the command name ningal:~ $ 4945 4950 35148 names.txt wcnames.txt

19. Command Options • Options modify the behaviour of a command • Single character options: '-’ ningal:~ $ • An option may require an argument ningal:~ $ • Multi-character options: '--’ ningal:~ $ date -R Tue, 27 Feb 2018 13:40:22 +0000 date -r names.txt Fri 23 Feb 2018 15:22:17 GMT Rscript –-version R scripting front-end version 3.3.3 (2017-03-06)

20. Combining Options and Arguments • Options (and their arguments) should precede command arguments ningal:~ $ wc -l names.txt • Some commands accept multiple arguments ningal:~ $ wc -l names.txtplaces.txt • Multiple single-character arguments can be combined ningal:~ $ wc -w -l names.txtplaces.txtningal:~ $ wc -wlnames.txtplaces.txt

21. Wildcards • Wildcards are special characters which will match range of characters in commands • Can be use to fine-tune arguments • *: Match zero or more of any character ningal:~ ls *.txtningal:~ wc *.txt • ?: Match any single character ningal:~ ls l?st.txt • []: Match specified range of characters ningal:~ ls l[io]st.txt • [!]: Match any character not specified ningal:~ ls list[!12].txt

22. Getting Help • Most commands offer some brief help explaining their usage: • -h, --help ningal:~ wc --help • More extensive documentation available in a man page ningal:~ man wc • Some more complex programs provide even more extensive documentation in ‘Texinfo’ format ningal:~ info coreutils 'wc invocation' • How to access this will be described in a ‘SEE ALSO’ section of the man page • If you don’t know the command you need: ningal:~ apropos word count • Man pages etc tend to be very formal, dry and not always easy to find what you want

23. Files and Directories

24. Files • Data is stored on disk in files. • Text files: data in plain text which can be viewed directly • Binary files: binary data in application specific format requiring software to access • The 'file' command can tell you what type of data a file contains... ningal:~ $ file names.txtnames.txt: ASCII text ningal:~ $ file lists.tarlists.tar: POSIX tar archive (GNU) • A text file can be viewed on the terminal ningal:~ $ cat list.txt • Don’t try to view a binary file…

25. File Naming • File names can contain pretty much any character • BUT... • Some characters may have special meaning • These need to be escaped (prefixed with '\'), or the name surrounded in quotes • Safest to stick to A-Za-z0-9._- • File names starting with '.' are hidden files • File names may include a suffix i.e. '.txt’ • Suffix is for the user's benefit - no meaning to the system

26. Directories • Act as containers to group related files together • Can contain many files, or directories • Directory naming follows same conventions as file naming • Directories are in fact special files… • Many commands which work on files also work on directories • Too many files in one directory problematic... • Think about a sensible way of arranging your files

27. The Linux Filesystem Layout • Filesystem has tree structure • Contained within root (/) directory • Directories within a directory are known as subdirectories • Root directory contains many subdirectories • Most are part of the OS itself • Certain directories will be available for data storage • /cluster mountpoint on UoD cluster

28. Paths • A file or directory can be addressed by it's path • A path lists sequence of directories to traverse to find file • Directories on path separated by '/' /tmp/mytempfile.txt • A path can be absolute or relative • Path can also refer to: • Current working directory (‘.’) • Parent of current working directory (‘..’)

29. Your Home Directory • Every user has a private directory known as 'home' directory • Normally found in '/homes/your_username' • This is your current working directory on logging in • Home directory can be referred to in paths by ‘~’ • Don't store large amounts of data here – these should go in /cluster/groupname

30. Navigating the Filesystem

31. ls - Many Options to Tailor Output

32. Disk usage reporting • Need to be aware of disk usage • May be restrictions due to: • Available space on filesystem • Disk quotas

33. Users and Permissions

34. Users and Groups • All users have a user id and group id • Your username identifies you to the system • You will also be a member of one or more groups • Groups allow management of a collection of users • Useful commands:

35. File and Directory Permissions • File permissions allow granular access control • Standard permissions: read, write execute • Permissions can be applied for a user (u), group (g) or other (o) users • Permissions can be seen with ls -l • type: - = file, d = directory, l = symlink • u: user (owner) permissions: rwx • g: group permissions: rwx • o: other permissions: rwx

36. What Permissions Allow

37. Changing Permissions • Change mode: chmod command • Permissions can be added (+) or removed (-) • Add read permissions for group:chmodg+r /path/to/file • Remove write permissions from others:chmod o-w path /path/to/file • Allow read, write and execute permissions for user:chmodu+rwx /path/to/file • Try these with some files in your directory, and see how the the permissions are changed with 'ls -l'

38. Access Control Lists • Standard permissions model useful, but doesn't fit all cases • Access Control Lists (ACLs) allow fine-grained permissions management • Different types according to technology in use: POSIX, Windows, NFS4 • All allow rules along the line of: 'Allow recursive read and write access to group "collaborators" to /my/important/data' 'Allow recursive read and write access to group "collaborators" to any new entries created in /my/important/data' 'Deny access to user fred to /my/important/data' • Can be more complex to get right • Coming soon to a filesystem near you…

39. Working with files

40. Basic File Operations

41. Viewing Text Files • cat shows file contents on display • If contents longer than screen, will scroll off top • less command shows one screen of text at a time • Interactive commands within less: • space: show next page • j/↑: scroll up one line • k/↓: scroll down one line • /searchterm: search forward for searchterm • q: quit • Remember: Don’t try to view a binary file with cat/less

42. Editing Text Files • Many editors available... • Some don't need X11 • Vi/emacs extremely powerful • But… • Gedit easier to get to grips with…

43. Compressing and Uncompressing Files • Files can be compressed to • save disk space • improve network transfer times • Most common: gzip, More efficient but slower: bzip2

44. Tar Archives • tar allows combining of many files and directories into single file • Optionally may be compressed with compress, gzip or bzip2 • -ftakes filename as argument

45. Managing Command Outputs

46. In, Out and Error Streams • Every linux process opens three streams • Associated with terminal by default • Standard input (0): input typed in terminal presented to process as input • Standard output (1): Output from process printed to terminal • Standard error (2): Errors encountered may also be printed to terminal • A well behaved Unix program is quiet… • These streams can be associated with other files if required: redirection

47. Redirection • Under bash shell: • These operators can also be combined myprogram < data > out.txt 2> err.txt • ./redirection.py– reads STDIN and writes to STDOUT • ./redirection.py < list1.txt > out.txt 2> err.txt

48. Pipes • Pipes allow output of one program to be sent to the input of another • Chain multiple commands together • Increased efficiency: save writing data to disk and rereading • | symbol denotes use of pipe on command line • Now try it: send the output of the date command to redirection.py ningal:~ date|./redirection.py

49. Process Management

50. Foreground and Background Jobs • Multiple jobs can be started from same shell • Job currently attached to terminal: foreground • Job detached from terminal: background • Jobs are started in foreground by default • Starting a job in the background: add ‘&’ mycommand & • Putting a running foreground job into the background: CTRL-Zandbg ningal:~ $ ./sleepy.py ^Z [1]+ Stopped ./sleepy.py ningal:~ $bg [1]+ ./sleepy.py &