Unix for Bioinformaticists: Unix Tools, Emacs, and Perl

1. Unix for Bioinformaticists: Unix Tools, Emacs, and Perl helpdesk at stat.rice.edu Aug 2004 Some slides are borrowed from Dr. Woely’s (BCM) presentation.

2. Do I Have to Know/Use Unix? • Simple answer: no. • Windows can do almost everything. • Complicated answer: yes, if you • are lazy (would like to automate things) • are good at reading manuals and writing scripts • want to make better use of your machine • are as poor as I am (can not afford pricy windows software) • especially if you will be a bioinformaticist

3. Why Unix Is Useful in Bioinformatics • Many tasks involve processing on large text based datasets. Unix tools in many cases are better than their windows counterparts. • You may need to use several tools to accomplish a task. Windows is not particularly good at gluing them. • When you need more CPU power, servers and clusters are usually *nix-based. • Many tools are available only under Unix-like systems.

4. Outline • Unix in general • Unix tools • Emacs • Perl

5. Unix Commands Single command: > sort –k1 file.txt Combine other commands: > sort –k1 file.txt | grep “Tag=Mouse” > output.txt Operate multiple files: > foreach file (*.txt) sort –k1 $file > $file:r_sorted.txt end

6. More commands > rename .html .htm *.html There are many such convenient tools. Scripts can be used if you can not find one, > foreach f (*.html) mv $f $f:r.htm end

7. More commands > wget -r -l1 --no-parent -A.tar.gz -Ppackages http://cran.r-project.org/src/contrib/PACKAGES.html download all .tar.gz files to packages directory, This command can do everything ‘teleport’ etc. under windows can do. > convert –rotate 90 file.jpg file.png Convert a .jpg file to .png format after rotating 90 degrees.

8. A shell script: lyx2pdf > lyx2pdf myfile.lyx #!/bin/csh set file = $1:r lyx --export latex $file.lyx latex $file.tex dvips -o $file.ps $file.dvi ps2pdf $file.ps

9. A Makefile %.html: %.tex latex2html -local_icons -no_subdir -split 0 $*.tex %.tex: %.lyx lyx2tex $*.lyx %.dvi: %.tex latex $*.tex %.ps: %.dvi dvips -o $*.ps $*.dvi %.pdf: %.ps ps2pdf $*.ps > make file.dvi > make file.ps > make file.pdf

10. A Perl Script #!/usr/bin/perl # read all the things at once undef $/; # read in the file and look for /* */ ($comm) = <> =~ /.*\/\*(.*)\*\//ms; # print comments print $comm, "\n";

11. crontab # do not forget to renew your library books 0 0 15 7 * mail bpeng@rice.edu %subject reminder Renew all the books! # backup your files to server every day at 6AM 6 * * * * /usr/local/bin/rsync -avz /home/bpeng thor.stat.rice.edu::backup > logfile

12. Graphviz > dot –Tps try.dot –o try.eps File: try.dot digraph G { A->B->C B->D->C }

13. Useful (and free) tools Servers: Apache, openssh, openldap Web: Mozilla/firefox, Konqueror, lynx Mail clients: Pine, Mutt, Mozilla/thunderbird, kmail, evolution Text processing: tetex/lyx, open office, koffice Languages: gcc, Perl, python, gmake, kdevelop Scientific libraries and tools: GNU Scientific Library, bioPython, bioPerl, R, Graphviz, gnuplot, octave Misc: VNC, wget,

14. Unix text-processing tools • Access to Unix • Mac OSX + developers kit • Linux • Stat and ruf/owlnet servers (Solaris) • Windows + cygwin • Tools - in contrast to Excel, faster, operate on larger files • Grep, Pipes, Sort, Comm, Diff, Join • Sed - regular expression substitution editor, replaced by perl in most contexts • Man - to list manual pages with options for most commands (if installed and concurrent version)

15. Grep • Grab lines that match a text phrase • Only the line that matches • Lines before or after the matched line • Lines that do not match • Piping multiple searches

16. GenBank Files

17. Grab the Locus, Definition and Keyword lines phase2.txt.out temp

18. Select Non-Human Definition Lines and Use Pipe kworley% grep -v Homo temp | grep DEF temp

19. Specify Lines to return grep -1 grep -B1 grep -A1

20. Sort • In dictionary (-d), month (-M), or numerical (-n) order • Ignore case (-f) • Specify output file (-o) • Specify the separator between fields (-t) • Unique lines only (-u) • Specify field on which to sort (-k POS1,[-POS2]), numbered starting from 0, can specify which character in the field (field.char) • Merge more than one sorted file (-m)

21. Comm • Select or reject lines in common between two sorted files • Options suppress printing of columns • comm [-123] file1 file2 • Column 1 is lines only in file 1 • Column 2 is lines only in file 2 • Column 3 is lines in both files

22. Diff • Compares two files (or sets of files in a directory) and output lines with differences • Compare as text (-a) • Ignore changes in white space (-b) or blank lines (-B), case difference (-i) • For directory comparisons • Report only files that differ not details (-q) • Compare subdirectories recursively (-r)

23. Join • Combines lines from two files based on a common field (-1 field -2 field) • Specify the fields from each file and the order to output (-o file_number.field file_number.field file_number.field)

24. What is Emacs? • A Unix text editor with additional functionality • Column functions • Settings for DNA mode • Settings for programming mode • Seamless integration with matlab, R, S-Plus, SAS etc.

25. Emacs Demonstrations • Search and replace • By query • All • New lines • Counting things • Column functions • Select • Kill • Copy • Paste

26. Query replace • Esc % • Replace phrase • With phrase • Designate carriage return with control Q control J • Y or N • ! To replace all

27. Starting File

28. Query Replace

29. End file

30. Rectangle functions • Mark, select rectangle • Control x r • r a • To register the rectangle as buffer a • k • To kill the rectangle • r i a • To insert previously registered rectangle a from buffer

31. Select Rectangle, Kill

32. Select Rectangle, Mark, Insert

33. What is Perl? • A general purpose programming language. • Invented to replace awk, sed, and sh. • A scripting language. • Practical Extraction and Reporting Language • Pathologically Eclectic Rubbish Lister “There is more than one way to do it” TIMTOWTDI

34. How to Use Perl • Perl “scripts” (programs) are text and are interpreted by the the perl program. • TIMTOWTDI: • You can put the script on the command line:>perl -e 'print "Hello, world!\n";' • You can pass it as an argument to perl:>perl my_program.pl • You can make the script self-executing:>my_program.pl

35. print, ", ', \n 'print "Hello, world!\n";' • In most programming languages, "print" means "display" or "output". • The single and double quote characters ( " ' ) are used to set apart blocks of "text". In this example, the single quote sets apart the perl script, and the double quotes sets apart the text to display. (Perl has others ways to quote.) • The backslash, '\', is used to change the meaning of a character, e.g. to generate special characters. \n means "start a new line" (e.g. the Carriage Return, or Return, or Enter.)

36. Example of a One Liner(Thanks to Dr. Wheeler) perl -nle '@f=split/\t/; print if ($f[2] > 95 );' blast_tbl_in.txt > blast_tbl_out.txt perl -nle '@f=split/\t/; print if ($f[2] > 95);' blast_tbl_in.txt >blast_tbl_out.txt

39. A One Liner: TIMTOWTDI • perl -nle '@f=split/\t/; print if ($f[2] > 95 );' blast_tbl_in.txt > blast_tbl_out1.txt • perl -ne '@f=split; print if ($f[2] > 95 );' blast_tbl_in.txt > blast_tbl_out2.txt • perl -ane 'print if ($F[2] > 95 );' blast_tbl_in.txt > blast_tbl_out3.txt

40. split, if, variables @f=split/\t/; print if ($f[2] > 95); • split is a function. It can be written with parens like in most languages, and takes UP TO three arguments:split( where_to_split, what_to_split, how_many_to_split) • split, like many Perl statements, uses defaults for missing arguments. • Special characters mark @whole_arrays, $array_members[1], %whole_hashes, $hash_members{'one'}, $simple_variables. • if acts like its common English meaning. It can go before a block or at the end of a statement (as above). • Perl converts between numbers and text. '>' is a numeric operator so 95 and $f[2] are treated as numbers. If gt replaced >, they would be treated as strings.

41. FASTA to XML perl -pi.bak -e's"^>(.*)$"</seq><title>\1</title><seq>";'test.fa

42. [localhost:~/test] steffen% ls test.fa test.fa.bak [localhost:~/test] steffen% perl -pi.bak -e 's"^>(.*)$"</seq><title>\1</title><seq>";' test.fa [localhost:~/test] steffen% ls test.fa test.fa.bak [localhost:~/test] steffen% more test.fa </seq><title>CSTAP1E0101A</title><seq> gttgcctgcgtcttcggxaacaacgtagttctcagGCCGCCCGACCAGGT ACTTTTTTGCTTTTTTTTTTTTTATTTTTTACAAATTATCAAAAGTTCTT GTGCTTTCAGGAGCGATTAACATTCTCATGGGCCATACCCTTGTCAGGTT TCATAAACTAAGTTAGATGGACCTGCTTGGTATTGTGGTGGAAGACCTCC AAGAAAACAAAGTCCCGGAATCTCAACGTCCTCTGTCTTCTGGCATTTCA TCTTCAAGAAACAATGTCTTATAGTTATTATTGCATGTTTTGGGAGGTTA AAGGGTAAAGTTTGTAATGCCTTGACTAAAAACTTCCAGTTGTTATGGTG cacaacaatttttggtatgctaacttatacttgtgcctaatccttaagga aaagaaagagccatatacctaaaactgactttatttttcaaaaggta </seq><title>CSTAP1E0102A</title><seq> tttttgctggcgaactatcaggagactacagxaactacttttcagtxcga actcacatcatcactggccgtcgttttacaacgtcgtgattgggaaaacc ctggcgttacccaacttaatcgccttgcagcacatccccctttcgccagc tggcgtaatagcgaagaggcccgcaccgatcgcccttcccaacagttgcg cagcctgaatggcgaatggcgcctgatgcggtattttctccttacgcttt caatgatgagcacttxtaaaggtctgx </seq><title>CSTAP1E0103A</title><seq> atttgagcagcatctattgaaaactaxcgxagxtcttcaggcgcgCCCAC CCGAGGTACTACCAAGCCAGTGTCCTGCCCGGTTTTAAGCCCTCGTCCTC TCCCTTCGCTCTCCTCCAAACTGAGCAGCATTAGTTCCACAAGCACAGAA GTTAAACGAAAAACTGTCTTGCTCCACGGTCTCCTACAGTAGAATGCTGG ATAATAATGCTTTCAGAAGCCACTTCTACAACCAGAACATTCTGACCACC ACAATCATCAGGTTTACACACACCCTACGAAACACTAGCGAGTTAACAAG actgatgaactacttgcagtcgaactccaatcattactggccgtcgtttt aa

43. Executing a Perl Script in a File $line = <>; $line =~ s">(.*)"<title>\1</title><seq>"; print $line; while( $line = <> ) { $line =~ s">(.*)"</seq><title>\1</title><seq>"; print $line; } print "</seq>\n";

44. File Reading, Binding, while $line = <>; • <> reads one line from the "current file" $line =~ s">(.*)"<title>\1</title><seq>"; • =~ makes the preceding string the "current line" (Binding) while( $line = <> ) { print $line; } • Repeats the statements between { and } while there is another line.

45. Self-executing Perl Scripts • You need to know the path to your Perl program:>which perl/usr/bin/perl • The first line of your script must be:#!/usr/bin/perl • Permissions need to allow execution >chmod 755 my_program.pl

46. FASTA to XML Fleshed Out #!/usr/bin/perl # # fasta2xml by David Steffen 6/2/2004 # - Converts fasta file to mini-xml format $inpfile = shift( @ARGV ); if( not( $inpfile =~ m/^(.*)\.fa$/ ) ) { die( "Input file, $inpfile, must be a fasta file and end in .fa\n" ); } $basefile = $1; open( INPFILE, $inpfile ) or die( "Can't open $inpfile: $!\n" ); $outfile = '>' . $basefile . '.xml'; open( OUTFILE, $outfile ) or die( "Can't open $outfile: $!\n" ); $line = <INPFILE>; $line =~ s">(.*)"<title>\1</title><seq>"; print OUTFILE $line; while( $line = <INPFILE> ) { $line =~ s">(.*)"</seq><title>\1</title><seq>"; print OUTFILE $line; } print OUTFILE "</seq>\n";

47. Running Other Programs from Perl $files = `ls`; The "backtic" (` `) characters execute the text in between as a command to the operating system, returning the output of that command (e.g. to the $files) variable. $error = system( "mv $file ${basefile}.abi" ); The system statement executes its argument as a command to the operating system, returning ERROR MESSAGES from that command. (Output is printed as usual.) There are other, subtle differences between ` ` and system.