Perl, Webservers (apache) and web applications (cgi-bin)

Perl, Webservers (apache) and web applications (cgi-bin)

Corrections • "Elements to be compared are passed into the subroutine as the package global variables $a and $b" (I said any variables could be used -- which is wrong). $a and $b are references (the equivalent of pointers in C) They are exempt from the usual restrictions on global variables when using "strict". • In chr1 -- lowercase letters are "regions of low complexity" which have been masked by repeatMasker

The final word on &func PG 114-115 The official name of a subroutine includes the & prefix. A subroutine may be called using the prefix, but the & is usually optional, so are the parentheses if the subroutine has been predeclared. (Note, however, that the & is not optional when you’re just naming the subroutine, such as when its used as an argument to defined or undef, or when you want to generate a reference to a named subroutine by saying $subref = \&name. Nor is the & optional when you want to do an indirect subroutine call with a subroutine name or reference using the &$subreff() or &{$subref}() constructs (see chap 4). Subroutines may be called recursively. If a subroutine is called using the & form, the argument list is optional, and if omitted, no @_ array is set up for the subroutine: the @_ array of the calling routine at the time of the call is visible to called subroutine instead. This is an efficiency mechanism that new users may wish to avoid.

Sorting by Multiple Fields #!/usr/bin/perl use strict; my $i = 0; my %score = ("tim" => 195, "tracy" => 205, "indy" => 30, "tina" => 195); # sort keys of hash, based on values of hash my @keys = keys %score; # Sorting just the "keys" descending by numerical score, then by key, ascending alphabetical my @winners = sort by_score_and_name @keys; foreach $i (@winners) { print "$i $score{$i}\n"; } sub by_score_and_name { $score{$b} <=> $score{$a} #can do multiple levels of sort -- if comparison or #result is 0 -- it will "short circuit" to the next one $a cmp $b; # and perl returns last evaluated expression } #note descending order AND ascending order #tracy 205 #tim 195 #tina 195 #indy 30

Hash of Hash #!/usr/bin/perl #simple example of hash of hash # m people X n genotypes $record{1}{SNP1} = "AA"; $record{1}{SNP2} = "AA"; $record{1}{SNP3} = "BB"; $record{1}{SNP4} = "AB"; $record{1}{SNP5} = "AA"; $record{2}{SNP1} = "BB"; $record{2}{SNP2} = "AB"; $record{2}{SNP3} = "BB"; $record{2}{SNP4} = "AB"; $record{2}{SNP5} = "AB";

Webserver • A webserver (apache) is a computer application that provides a service • the service is to provide text, images, and programs to clients (browsers IE, Safari, Firefox, Mozilla, Netscape, etc.)

clients server address: pdb.eng.uiowa.edu/~tabraun/biotech text images applications

Protocols • Needed so that the client/server may communicate • HTML – hypertext markup language • JPEG (jpg) – image format • others • cgi-bin (perl), Java, Java-script, PHP, Ruby, others • programming languages that are capable of generating information within these protocols • "server-side scripting languages" • Perl was never designed to be SSSL, however power and flexibility of Perl makes this possible • cgi – common gateway interface – a protocol for specifying how a server may "serve" perl to a client • Instead of static HTML, a perl script runs, and generates the HTML-equivalent "on the fly", or dynamically. • client accesses server • server executes program • output of program is sent by server/read by client • Thus, the perl script may provide different results depending on input and the state of the server (current data available, date, user specified inputs, etc) • Note that in many cases, the execution time of a "program" is much faster than the network delays – so output "appears" instantaneous

HTML example:http://pdb.eng.uiowa.edu/~tabraun/biotech/ <html>  <title>51:123 Bioinformatics Techniques</title> <font SIZE=+4> 51:123 Bioinformatics Techniques<br> <font SIZE=+2> Course Web Page <br> <hr> <br> <br> <font SIZE=+2> <b> <a href=BioinfoTechsSyllabus2_51_123.pdf>Syllabus</a></font></b><br> <br> <br> <font SIZE=-1> Announcements: <br><br> 9/8/2003 The text book is at the IMU.<br><br> 9/15/2003 For Assignment 3, try the <a href=upload.html> submission cgi</a> <br><br> 9/16/2003 Pace of the course material. I have received multiple statements of concern about the rate at which we are covering course material in the class. Due to the varied background of students in this class, this feedback is very valuable to me for trying to adjust the pace of the course. I had already planned on slowing the pace and including more examples over the next lectures. However, based on this feedback, it would appear that multiple students would have preferred this to occur sooner. If you have an opinion -- send me an e-mail. Please feel free to send me your feedback. This is the first time this course has been offered, so I (and other faculty in this area) am very interested in developing this course to fit the needs of students in this field. <br><br>

What does it mean? • What does it mean to be able to put a perl application (cgi-bin) "in" a web page? • Content of the page may be generated programatically or dynamically • Users can interact with the web page • content displayed depends on state of the program (day of the week, user input, etc.)

Simple Example http://texas.eng.uiowa.edu:8888/~tabraun/biotech/sequence.html

sequence.html <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN"> <HTML> <HEAD> <META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1"> </HEAD> <form action="http://texas.eng.uiowa.edu:8888/~tabraun/cgi-bin/translate.cgi" method="POST" enctype="multipart/form-data"> <P>Enter filename to upload: <input type="file" name="file"> <input type="submit"> </form> </HTML>

#!/usr/bin/perl -wT use strict; use CGI; use Fcntl qw( :DEFAULT :flock ); use constant BUFFER_SIZE => 16_000; use constant MAX_FILE_SIZE => 1_000_000; # my $seq = ""; my %aminos = ( "TTT", "F", "TTC", "F", "TTA", "L", "TTG", "L", "CTT", "L", "CTC", "L", "CTA", "L", "CTG", "L", "ATT", "I", "ATC", "I", "ATA", "I", "ATG", "M", "GTT", "V", "GTC", "V", "GTA", "V", "GTG", "V", "TCT", "S", "TCC", "S", "TCA", "S", "TCG", "S", "CCT", "P", "CCC", "P", "CCA", "P", "CCG", "P", "ACT", "T", "ACC", "T", "ACA", "T", "ACG", "T", "GCT", "A", "GCC", "A", "GCA", "A", "GCG", "A", "TAT", "Y", "TAC", "Y", "TAA", ".", "TAG", ".", "CAT", "H", "CAC", "H", "CAA", "Q", "CAG", "Q", "AAT", "N", "AAC", "N", "AAA", "K", "AAG", "K", "GAT", "D", "GAC", "D", "GAA", "E", "GAG", "E", "TGT", "C", "TGC", "C", "TGA", ".", "TGG", "W", "CGT", "R", "CGC", "R", "CGA", "R", "CGG", "R", "AGT", "S", "AGC", "S", "AGA", "R", "AGG", "R", "GGT", "G", "GGC", "G", "GGA", "G", "GGG", "G",); $CGI::DISABLE_UPLOADS = 0; $CGI::POST_MAX = MAX_FILE_SIZE; my $q = new CGI; $q->cgi_error and error( $q, "Error transferring file: " . $q->cgi_error ); my $file = $q->param( "file" ) || error( $q, "No file received." ); my $fh = $q->upload( "file" ); my $buffer = "";

binmode $fh; #binmode OUTPUT; print $q->header( "text/plain" ), "File received."; print "\n"; # process contents of file while ( read( $fh, $buffer, BUFFER_SIZE ) ) { print "$buffer"; $seq = $seq.$buffer; } &printOutput($seq); ############ Subroutines sub printOutput #pass in the sequence { my $sequence = ""; my $codon = ""; $sequence = $_[0]; $sequence =~s/\s+//g; #kill newlines? print "Amino acids:\n"; while(length($sequence) >0) { $sequence =~s/(...)|(..)|(.)//; $codon = $1; #print $q->print("|$codon| :$aminos{$codon}:"); print "$aminos{$codon}"; } print "\n"; } sub error { my( $q, $reason ) = @_; print $q->header( "text/html" ), $q->start_html( "Error" ), $q->h1( "Error" ), $q->p( "Your upload was not procesed because the following error ", "occured: " ), $q->p( $q->i( $reason ) ), $q->end_html; exit; }

HTML form http://pdb.eng.uiowa.edu/~tabraun/biotech/upload.html <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN"> <html> <head> <meta HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1"> </head> <form action="http://pdb.eng.uiowa.edu/~tabraun/cgi-bin/upload.cgi" method="POST" enctype="multipart/form-data"> <p>Enter filename to upload: <input type="file" name="file"> <p>Enter Assignment number (1, 2, 3, ...): <input type="text" name="assignment"> <p>Enter lastname: <input type="text" name="login"> <p>Enter name of file to upload (can re-name, or leave the same): <input type="text" name="filename"> <input type="submit"> </form> </html>

#!/usr/bin/perl -wT use strict; use CGI; use Fcntl qw( :DEFAULT :flock ); use constant UPLOAD_DIR => "/usr/local/apache/data/uploads"; use constant BUFFER_SIZE => 16_384; use constant MAX_FILE_SIZE => 1_048_576; # Limit each upload to 1 MB use constant MAX_DIR_SIZE => 100 * 1_048_576; # Limit total uploads to 100 MB use constant MAX_OPEN_TRIES => 100; $CGI::DISABLE_UPLOADS = 0; $CGI::POST_MAX = MAX_FILE_SIZE; my $q = new CGI; $q->cgi_error and error( $q, "Error transferring file: " . $q->cgi_error ); Modified Simple cgi-bin:CGI Programming with Perl, O'Reilly my $file = $q->param( "file" ) || error( $q, "No file received." ); my $filename = $q->param( "filename" ) || error( $q, "No filename entered." ); my $fh = $q->upload( "file" ); my $buffer = ""; if ( dir_size( UPLOAD_DIR ) + $ENV{CONTENT_LENGTH} > MAX_DIR_SIZE ) { error( $q, "Upload directory is full." ); } # Allow letters, digits, periods, underscores, dashes # Convert anything else to an underscore $filename =~ s/[^\w.-]/_/g; if ( $filename =~ /^(\w[\w.-]*)/ ) { $filename = $1; } else { error( $q, "Invalid file name; files must start with a letter or number." ); }

# Open output file, making sure the name is unique until ( sysopen OUTPUT, UPLOAD_DIR . "/$filename", O_CREAT | O_RDWR | O_EXCL ) { $filename =~ s/(\d*)(\.\w+)$/($1||0) + 1 . $2/e; $1 >= MAX_OPEN_TRIES and error( $q, "Unable to save your file." ); } # This is necessary for non-Unix systems; does nothing on Unix binmode $fh; binmode OUTPUT; # Write contents to output file while ( read( $fh, $buffer, BUFFER_SIZE ) ) { print OUTPUT $buffer; } close OUTPUT; print $q->header( "text/plain" ), "File received."; sub dir_size { my $dir = shift; my $dir_size = 0; # Loop through files and sum the sizes; doesn't descend down subdirs opendir DIR, $dir or die "Unable to open $dir: $!"; while ( readdir DIR ) { $dir_size += -s "$dir/$_"; } return $dir_size; } sub error { my( $q, $reason ) = @_; print $q->header( "text/html" ), $q->start_html( "Error" ), $q->h1( "Error" ), $q->p( "Your upload was not procesed because the following error ", "occured: " ), $q->p( $q->i( $reason ) ), $q->end_html; exit; }

#!/usr/bin/perl -wTuse strict;use CGI;use Fcntl qw( :DEFAULT :flock );use constant UPLOAD_DIR => "/home/tabraun/public_html/biotech/hw";use constant BUFFER_SIZE => 16_384;use constant MAX_FILE_SIZE => 1_048_576; # Limit each upload to 1 MBuse constant MAX_DIR_SIZE => 10 * 1_048_576; # Limit total uploads to 10 MBuse constant MAX_OPEN_TRIES => 100;$CGI::DISABLE_UPLOADS = 0;$CGI::POST_MAX = MAX_FILE_SIZE;my $q = new CGI;$q->cgi_error and error( $q, "Error transferring file: " . $q->cgi_error );my $file = $q->param( "file" ) || error( $q, "No file received." );my $login = $q->param( "login" ) || error( $q, "No login entered." );my $assignment = $q->param( "assignment" ) || error( $q, "No assignment entered." );my $filename = $q->param( "filename" ) || error( $q, "No filename entered." );my $fh = $q->upload( "file" );my $buffer = ""; if ( dir_size( UPLOAD_DIR ) + $ENV{CONTENT_LENGTH} > MAX_DIR_SIZE ) { error( $q, "Upload directory is full." );}# Allow letters, digits, periods, underscores, dashes# Convert anything else to an underscore$filename =~ s/[^\w.-]/_/g; if ( $filename =~ /^(\w[\w.-]*)/ ) { $filename = $1;}else { error( $q, "Invalid file name; files must start with a letter or number." );}$login =~ s/[^\w.-]/_/g;if ( $login =~ /^(\w[\w.-]*)/ ) { $login = $1;}else { error( $q, "Invalid login name; files must start with a letter or number." );}if ( $assignment =~ /^(\d+)/ ) { $assignment = $1;}else { error( $q, "Invalid login name; files must start with a letter or number." );} cgi-bin (Common Gateway Interface – Binary) – upload.cgi

mkdir "/home/tabraun/public_html/biotech/hw/$login";mkdir "/home/tabraun/public_html/biotech/hw/$login/$assignment";# Open output file, making sure the name is uniqueuntil ( sysopen OUTPUT, UPLOAD_DIR . "/$login/$assignment/$filename", O_CREAT | O_RDWR | O_EXCL ) { $filename =~ s/(\d*)(\.\w+)$/($1||0) + 1 . $2/e;$1 >= MAX_OPEN_TRIES and error( $q, "Unable to save your file $filename." );}# This is necessary for non-Unix systems; does nothing on Unixbinmode $fh;binmode OUTPUT;# Write contents to output filewhile ( read( $fh, $buffer, BUFFER_SIZE ) ) { print OUTPUT $buffer;}close OUTPUT;print $q->header( "text/plain" ), "File received."; sub dir_size { my $dir = shift; my $dir_size = 0; # Loop through files and sum the sizes; doesn't descend down subdirs opendir DIR, $dir or die "Unable to open $dir: $!"; while ( readdir DIR ) { $dir_size += -s "$dir/$_"; } return $dir_size;} sub error { my( $q, $reason ) = @_; print $q->header( "text/html" ), $q->start_html( "Error" ), $q->h1( "Error" ), $q->p( "Your upload was not procesed because the following error ", "occured: " ), $q->p( $q->i( $reason ) ), $q->end_html; exit;} upload.cgi continued

Server Configuration • Need a web server (apache) • Need perl • specify permissions (writing/reading/access) • location of scripts so web-server knows where to find them – apache configuration files • Other system issues • access through the firewall – web server is on port 80 • a firewall may be thought of as a set of rules that specifies the remote address space that may access the (local) server • and "ports" that may be accessed

Notes • Note that the cgi-bin generated HTML-formatted text was sent by the apache web server to the web-browser (client) • Used some "objects" – CGI.pm – that we don't know about yet. • Leads us to perl modules, objects, and eventually bioperl

Take Home Messages • Perl was originally designed for "text" processing • Powerful pattern matching, regular expressions, and flexibility of the language make it adaptable to other tasks (server-side scripting language) • You should know about Apache, cgi-bin programming, and Perl (but we will not do any cgi programming in this class) • Making tools and techniques available on the web is very common • (examples: Primer3, BLAST, Ensmart, local blast)

Another example http://genome.uiowa.edu/cgi-bin/distribution_info/index.cgi At: pdb: /home/httpd/cgi-bin/distribution_info Look at code VS "page-source" Why does code look like perl -- and not page-source?

Perl, Webservers (apache) and web applications (cgi-bin)