CISC3130: Redirection & Pipe

CISC3130: Redirection & Pipe Xiaolan Zhang Spring 2011

Agenda • Continue of last class: file system • Misc topics • Some productivity tips • Misc. commands • File manipulation utilities • Input/output redirection and pipe • Putting things together

Last class • Hierarchical file structure • Absolute pathname & relative pathname • /home/staff/zhang • zhang, staff/zhang, ../../etc/passwd “..”: refers to parent dir “.”: current directory “/”: root and seperator in file names “~”: home directory A tree of directories and files / lib dev bin home etc passwd cdrom tty24 staff zhang

Long listing • To get more information about each file [zhang@storm Demo]$ ls -al total 32 drwxr-xr-x 5 zhang staff 4096 2008-01-16 16:01 . drwxr-xr-x 41 zhang staff 4096 2008-01-16 16:01 .. drwxr-xr-x 2 zhang staff 4096 2008-01-16 15:55 CCodes -rw-r--r-- 1 zhang staff 38 2008-01-16 16:01 .HiddenFile -rw-r--r-- 1 zhang staff 53 2008-01-16 15:57 README drwxr-xr-x 2 zhang staff 4096 2008-01-16 15:55 SampleCodes drwxr-xr-x 4 zhang staff 4096 2008-01-16 15:56 ShellScriptes Total disc space taken in blocks (1024 Byte) User name of the owner and its group d means directory Who has permission to read/write the file

Long listing explained • Field 1 • 1st Character – File Type: First character specifies the type of the file. • - normal file, d directory, s socket file, l link file • Next 9 characters – File Permissions • Field 2 – Number of links • Second field specifies the number of links for that file • Field 3 – Owner: specifies owner of the file • Field 4 – Group: specifies the group of the file • Field 5 – Size: specifies the size of file. • Field 6 – Last modified date & time: • date and time of last modification of the file • Field 7 – File name

File permissions • Each file is associated with permission info. • Differentiate three type of users: owner user, users from same group as owner, others • Three type of access • Read (r): use “cat” to open a file to read, use “ls” to list files/directories under a directory • Write (w): modify the contents of the file, create/remove files from the directory • Execute (x): execute the file, or “cd” or “search” the directory for file • Trying to list other’s directory [zhang@storm ~]$ ls ../roche/ ls: cannot open directory ../roche/: Permission denied

Agenda • Continue of last class: file system • Misc. topics • Some productivity tips • Misc. commands • File manipulation utilities • Input/output redirection and pipe • Putting things together

Misc. Commands • Send a file to the printer: • lpr <fileName> • The file should be of format that the printer recognizes, e.g., text file, postscript file (.ps)! • which: show the full path name of a command $ which bash /bin/bash • How does shell find a command ? • Environment variable PATH stores a list of paths to search for programs: “set | grep PATH” or “echo $PATH”, “set” to show all variable settings • PATH=$PATH:$HOME/bin:. • Built-in commands: history, set, echo, etc.

Some useful tips • Bash stores the commands history • Use UP/DOWN arrow to browse them • Use “history” to show past commands • Repeat a previous command • !<command_no> • e.g., !239 • “!<any prefix of previous command> • E.g., !g++ • Search for a command • Type Ctrl-r, and then a string • Bash will search previous commands for a match • File name autocompletion: “tab” key

Customize your shell environment • To change your login shell [zhang@storm ~]$ chsh Changing shell for zhang. Password: New shell [/bin/bash]: /bin/bash Shell not changed. • Bash configuration files(in home dir, file name start with dot) • .bash_logout: A list of commands to be executed when leave shell • .bash_profile: a list of commands to be executed when log in • .bashrc: a list of commands to be executed when open a new shell • .bash_history: a list of commands you have been typing

Environment Variables • Setting environment variables PATH=$HOME/bin:$PATH • Environment variable PATH stores a list of paths that bash searches for programs • To see the current settings of an environment variable: set | grep PATH echo $PATH //set to show all variable settings • PATH=$PATH:$HOME/bin:. PS1="You rang? " export PATH PS1 • Can also set and export in one line: export PS1="Yes? “

Set environment variables • Setting environment variables PATH=$HOME/bin:$PATH • Environment variable PATH stores a list of paths that bash searches for programs • To see the current settings of an environment variable: set | grep PATH echo $PATH //set to show all variable settings • PATH=$PATH:$HOME/bin:. PS1="You rang? " export PATH PS1 • Can also set and export in one line: export PS1="Yes? “

Create customized command shorthand • Aliases • alias ls='ls –F’ • alias rm=‘rm –i’ ## so that you have to confirm the removal

Agenda • Continue of last class: file system • Misc. topics • Access online manual • Some productivity tips • Misc. commands • File manipulation utilities • Input/output redirection and pipe • Putting things together

What’s in a file ? • So far, we learnt that files are organized in a hierarchical directory structure • Each file has a name, resides under a directory, is associated with some admin info (permission, owner) • Contents of file: • Text (ASCII) file (such as your C/C++ source code) • Executable file (commands) • A link to other files, … • To check the type of file: “file <filename>”

Some useful file related utilities • Counting # of lines, words and characters in files • wc [option]…[FILE]…. • wc [OPTION]… --file0-from=F // to be discussed later • To search files for lines that match a pattern • grep [OPTIONS] PATTERN [FILE…] • Examples • grep “global warming” articles • -v option: lines that don’t match the pattern • Where did I define/access a variable named gNumOfOperations ? • grep gNumOfOperations *.[ch] • If no file is specified, search standard input for given pattern • For redirection/pipeline

Sort command • Sort lines of text file sort [OPTION]… [FILE]… • Many options to control sorting order • -r: reverse the normal order • -n: sort in numeric order • By default, the command sort in alphabetic order • -nr: sort in reverse numeric order • +n: sort starting at n+1-th field

Compare file contents • Suppose you carefully maintain diff. versions of your projects (so that you can undo some changes), and want to check what’s the difference. • cmp file1 file2: finds the first place where two files differ (in terms of line and character) • diff file1 file2: reports all lines that are different

Agenda • Continue of last class: file system • Misc. topics • Some productivity tips • Misc. commands • File manipulation utilities • Input/output redirection and pipe • Putting things together

Standard input, output, error • On startup, each program has three “files” created/opened for it • Standard input, by default is linked to keyboard • Standard output, by default is linked to terminal window • Standard error, by default linked to terminal window 0 1 2

Simple example • A very simple C program #include <stdio.h> main() { char yourName[256]; printf ("Your name ?\n"); if (fgets (yourName,256,stdin)==NULL) fprintf (stderr,"No input"); else printf("hello, %s\n", yourName); }

Input/Output Redirection • On command line, one can redirect these three files • To redirect standard output to a disk file: • $ command [ [ - ] option (s) ] [ option argument (s) ] [ command argument (s) ] > FILENAME • Execute the command, sending its standard output to specified file • Existing content of the file is deleted • For example: • ls –lt > InfoFilelist.txt • To append standard output to a file: use >> instead of > • grep “tax reform” *.txt > output • grep “fuel efficiency” *.txt >> output

Input/Output Redirection (cont’d) • To redirect standard error to a file, instead of keyboard $ command [ [ - ] option (s) ] [ option argument (s) ] [ command argument (s) ] 2> ERRORMSGS • Examples: [zhang@storm ~]$ ls abc ls: cannot access abc: No such file or directory [zhang@storm ~]$ ls abc 2> error [zhang@storm ~]$ more error ls: cannot access abc: No such file or directory

User > and 2> together • To split error messages from normal output [zhang@storm ~]$ ls research.tex abc ls: cannot access abc: No such file or directory research.tex [zhang@storm ~]$ ls research.tex abc 2> error > output [zhang@storm ~]$ cat error ls: cannot access abc: No such file or directory [zhang@storm ~]$ cat output research.tex [zhang@storm ~]$ • This is useful for running a command that might take long time to finish, or generates very long output …

More on redirection • To redirect both output and error to same file: • ./a.out < tt > dd 2> dd : does not work. Error output is not captured. • sort file.txt > dd 2>&1 • 2>&1: redirect error output to same place as standard output • grep numOfStudents 2>dd >&2 • >&2: redirect standard output to same place as error output • To discard output, redirect it to /dev/null • /dev/null: a special virtual file, “a black hole” • ./a.out > /dev/null 2>&1 • I don’t want to see the output or error message, nor do I want them saved to a file …

Input/Output Redirection (cont’d) • To read standard input from a file, instead of keyboard $ command [ [ - ] option (s) ] [ option argument (s) ] [ command argument (s) ] < FILENAME • Examples • mail zhang –s “Question” < proj1.cpp • ./a.out < values.txt //a.out is your program that reads integers from standard input and calculate the sum

Environment Variables • Use command env to show all environment variables and their values

Combining commands together • How many files are there under current directory ? ls > tmp wc –l < tmp rm tmp • Sort current online user by alphabetic order • Is some user login to the system now ? (using grep) Is file “tmp” listed ?

Pipe: getting rid of temporary file • Pipe: connect the output of one program to the input of another program • Any prog. that reads from standard input can read from pipe, similarly for the standard output • who am i | ./a.out | wc • The individual program/command knows nothing about redirection and pipe

Rule of composition • Pipe: one of the fundamental contributions of UNIX system • Design programs to be connected with other programs • Read/write simple, textual, stream-oriented formats • Read from standard input and write to standard output • Filter: program that takes a simple text stream on input and process it into another simple text stream on output

Command Pipeline: how ? • Pipe • an inter-process communication mechanism provided by kernel • Has a reading end and a writing end • Any data write to writing end can be read back from the reading end • Read/write pipe is no different from read/write files Writing end Reading end

Command Pipeline: how ? • Shell set things up • create a pipe, “start” two programs simultaneously, with their input/output redirected to the reading/ending end of pipe

The workings of shell • For each command line, shell creates a new child process to run the command • Sequential commands: e.g. date; who • Two commands are run in sequence • Pipelined commands: e.g. ls –l | wc • Two programs are load/execute simultaneously • Shell waits for the completion, and then display prompt to get next command …

Important concept: Process • Early computers run a job from starting to end • Multiprogramming was popularized later • To load multiple programs in memory and switch between them when one is waiting for I/O => increase CPU utilization • Timesharing: a variant of multiprogramming, in which each user has an online terminal (multiple users sharing the system)

Process • A process is an instance of a running program • It’s associated with a unique number, process-id. • OS stores its running state • A process is different from a program • wc, ls, a.out, … are programs, i.e., executable files • which program […] • When you run a program, you start a process to execute the program’s code • Multiple processes can run same program • At any time, there are multiple processes in the system • One of them is running, the rest is either waiting for I/O, or waiting to be scheduled

Access to Instructions The hard disk is too slow to provide instructions to CPU. So programs are first loaded into main memory, which is much faster. The CPU can then access the instructions more quickly.

Loading Program • Programs are stored in secondary storage (hard disks, CD-ROM, DVD) • To process data, CPU requires a working area, the Main Memory • Also called: RAM (random access memory), primary storage, and internal memory. • Before a program is run, it must first be copied from the slow secondary storage into fast main memory • Provides the CPU with fast access to instructions to execute.

ps command • To report a snapshot of current processes: ps • By default: report processes belonging to current user and associated with same terminal as invoker. • Example: [zhang@storm ~]$ ps PID TTY TIME CMD 15002 pts/2 00:00:00 bash 15535 pts/2 00:00:00 ps • List all processes: ps -e

BSD style output of ps Learn more about the command, using man ps [zhang@storm ~]$ ps axu USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 2112 672 ? Ss Jan17 0:11 init [3] root 2 0.0 0.0 0 0 ? S< Jan17 0:00 [kthreadd] root 3 0.0 0.0 0 0 ? S< Jan17 0:00 [migration/0] root 4 0.0 0.0 0 0 ? S< Jan17 0:00 [ksoftirqd/0] root 5 0.0 0.0 0 0 ? S< Jan17 0:00 [watchdog/0] root 6 0.0 0.0 0 0 ? S< Jan17 0:00 [migration/1] root 7 0.0 0.0 0 0 ? S< Jan17 0:00 [ksoftirqd/1] root 8 0.0 0.0 0 0 ? S< Jan17 0:00 [watchdog/1] root 9 0.0 0.0 0 0 ? S< Jan17 0:00 [migration/2]

Run program in background • To start some time-consuming job, and go on to do something else $ command [ [ - ] option (s) ] [ option argument (s) ] [ command argument (s) ] & • wc ch * > wc.out & • Shell starts a process to run the command, and does not wait for its completion, i.e., it goes back to reads and parses next command • Shell builtin command: wait • Kill a process: kill <processid>

Some useful commands • To let process keep running even after you log off (no hangup) • nohup COMMAND & • Output will be saved in nohup.out • To run your program with low priority • nice [OPTION] [COMMAND [ARG]...]

Some useful commands • To start programs at specified time (e.g. midnight) • at [-V] [-q queue] [-f file] [-mldv] timespec... • By default, read programs from standard input: [zhang@storm assignment]$ date Mon Jan 31 21:51:38 EST 2011 [zhang@storm assignment]$ at 10pm < todo job 15 at Mon Jan 31 22:00:00 2011 [zhang@storm assignment]$ more todo echo "HI!" ls | wc –l > temp

The Power of Pipe • Who is using the most CPU ? • ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10 • Find out how many subdirectories are there ? • Display the content of last edited file (under current directory)… • cat `ls –t | head -1`

Summary • We have looked at some basic “start up” topics. • Get around in the file system • Some basic files related commands • Introduction to shell • Redirection and pipe • Process concept

CISC3130: Redirection & Pipe