Processes

1. Processes • Process: instance of a program • has unique pid. • Process environment • Process id, parent-process-id, process-group-id • Opened files, • Working directory, • File creation mask • User ID, Group ID, • Resource limits, • Environment variables • Code env code

2. UNIX Process • New process created each time you execute a command. • Current process (parent) forks a new process (child) • Child created as a foreground (wrt parent) process: • parent forks new child • parent deactivated, waits for child to die • parent reactived upon death of child • Child created as a background process: • parent forks new child • parent immediately resumes activity

3. Processes • use “&” to execute a task in background • Example: $ sort infile > outfile & • ps - list processes. • jobs - list background processes. • ctrl-C (cancel foreground job) ctrl-Z (suspend foreground job) • bg - move (suspended) job into background. • fg pid - move background job to foreground. • kill pid - kill the process • -1 (kill process, and children) • -9 (kill process, may leave children alive)

4. Shell Process • Upon login: shell process created • Any command you type at prompt:new child of your shell process • What is your current shell?%echo $SHELL • How to switch to another shell?%bash just type shell name • How to switch login shell?%chsh user newshell (but won’t work here)

5. I/O Redirection • Redirection and Pipe “>“ redirects standard output (screen) to a file E.g. ls > dirlist “<“ redirects standard input (keyboard) from a file E.g. sort < infile > outfile “|” pipe output of program 1 to input of program 2(creates temporary file, automatically redirects) E.g. who | wc Or getit < in | check_it | process_it | format_id > out “>>“ appends output to a file E.g. ls -l >> oldfile

6. What’s going on? • What processes, programs, pipes, and files are used?%cat doc1 > out1; wc -l <out1%grep “root” /etc/passwd >OUT2 ; cat <IN >OUT &%cat doc1 | wc -l

7. What’s going on? % date ; who | wc Wed Sep 24 16:00:00 PDT 1997 4 24 182 | higher precedence than ; % (date ; who ) |wc 5 30 211

8. Shell Metacharacters > prog > file direct stdout to file >> prog>>file append stdout to file < prog<file take stdin from file | p1|p2 connect stdout of p1 to stdin to p2 * match string of 0 or more characters ? match any single character [ccc] match any single character from ccc ranges like 0-9 or a-e are legal ; command terminator & background process

9. Shell Metacharacters `…` run commands the output of … produces (…) run commands in … in a sub-shell $1, $2 arguments to shell file $var value of shell variable var \ \c take c literally (don’t evaluate) ‘…’ take … literally “…” take … literally, after $, `…`, \ interpreted # comment var=val assign variable var p1 && p2 run p1, if successful, run p2 p1 || p2 run p1, if unsuccessful, run p2

10. Unix File Systems • File: a collection of data • Just a sequence of bytes • no record or block structure required • Directory • A file that contains information on files • rooted tree file structure (inverted tree) • directories can contain both info on files and other directories.

11. Unix File System Road Map • Special files: /dev/* represents I/O devices. / /dev /etc /var /bin /tmp /usr /mnt /home passwd hosts …... spool adm include lib sue john etc bin 5bin printer mail(in) mail(out) uucp messages wtmp …

12. File Systems and the I-nodes • Each disk drive contains one or more file systems • Each file system occupies a number of cylinder groups. • Each file system has a superblock, an i-node table and files • The superblock is at a fixed location relative to the beginning of the file system. It contains a description of the file system. • One can find the location of the I-node table thru superblock. • Each entry of the I-node table is an I-node, which uniquely represents a file in the file system. • An I-node contains uid, gid, time of creation, modification, and access, permissions, location of the file on the disk. superblock I-node table file1 file2 free file3 free

13. Inodes • File - sequence of bytes (8 bits) • File conceptually has several components • name, contents, admin info • Inode stores • admin info • An Inode IS the file • directory hierarchy just provides convenient names for files • System’s internal name for a file is its i-number (the inode number) • ls -i lists i-number15444 file117777 file2

14. Directories • Directory is an ordinary file • can be read as ordinary files (by any program that reads text) • can’t be created or written as ordinary files (only system can) • od - octal dump • cat foohi therewelcome to unix • od -c myfile (output in decimal by byte pairs)0000000 h i t h e r e \n w e l c o m e 0000020 t o u n i x \n0000031 • 1st 7-digits are position in file, ordinal number of next character (in octal)

15. Inodes • od -b .0000000 4 ; . \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \00000020 277 ( . . \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \00000040 390 = b l a h \0 \0 \0 \0 \0 \0 \0 \0 \0 \00000060 • A filename is a link (links name in directory hierarchy to the inode, thus to the file contents)

16. Symbolic Links • Can have many links to the same file (inode) • rm - does not remove inode, removes directory entry (link) • Only when all links are gone is the file (inode) removed • ln -command for creating symbolic links • ln oldfile newfile (creates another link to the inode)

17. Permissions • Every file has a set of permissions associated with it • three types of permissions: read ( r), write (w), and execute (x) • three sets of permission: user, group, world. • In Unix system, users are identified by numbers:uid, gid ls -l -rwxr-xr-x 1 root 3743 Jan 4 1970 test user group others #links owner size (time of last mod) (file name) • Problem: how do you write a program which allows different users to access a set of files? E.g. the program “passwd”

18. Permissions (cont.) • Solution: use the set-uid bit • When a user execute a program with the set-uid bit set, the user assume the identity of the owner of the program. • For example ls -l /bin/passwd -rwsr-xr-x 1 root 8454 Jan 4 1994 /bin/passwd set-uid • Set-uid bit may break the security wall. (users can run the /bin/passwd and act like root) • Only special programs can be set-uid program, particularly if the owner is root.

19. Chmod • Change the access permissions of a file • chmod <permissions> <filename> • permissions can be specified as 3 octal digits, <user,group,others>, the three bits of an octal means r,w,x • Example: chmod 755 test • permissions can be specified as +x, or u+x, or g+r, … • “chmod +s test” sets the set-uid bit for file test. • If a directory has “x” in its permision, the dir is searchable, ie., one can do “ls” on the directory.

20. Pathnames • Every file and directory in the file system can be identified by a “full path name” (route from root to file) /home/sue/email/f1 • Relative path name • location relative to current directory . Current directory .. Parent directory • if cwd is /home/sue: ls email ls ./email cd .. ls ../fred / home sue fred email docs f2 f1

21. What’s Next? • QUESTION: How to create new commands? • ANSWER: Shell programming (shell scripts)! • But first, some details……………….

22. Some Details • cp [-ir…] file1 file2 cp [-ir…] file-list directory • i for interactive. prompt use whenever a file will be overwritten • r for recursive. copy a directory tree • ls [-alRF…] file-list • a for listing all files including the dot files • l for long format • R for recrusive. list the all subdirectories. • F for listing directories with a trailing / • date [+format] • %date ‘+%h %d, 19%y’ Oct 1, 1996

23. Some Details (cont.) • wc file-list • display the number of lines, words and charaters • more file-list • Browse through text files on page at a time. • head [-n …] file-list • Display the first n lines of the files (default=10) • tail [+n|-n| -f| …] • Display the last few lines in the files (default = 10) • Example: %tail +5 foo # display the last parf of foo starting from line 5 %tail -5 foo # display the last five lines of foo %tail +30 foo | head -15 | more #display line 30-45 of foo %tail -f foo # wait and display the new lines appended to foo

24. Some Details (cont.) • cut -c list file • cut -f list [-dChar] file • Cut out selected charaters or fields from each line of a file • Examples: %cut -c 1-5,15-20 foo # extract chars 1-5 and 5-20 from each line of foo. %cut -f 1,3 -d” “ moo # extract field 1 and 3 from each line of moo. • paste file1 file2 • Concatenate corresponding lines of the given input files • Example (reverse two fields of the file abc) %cut -f1 abc > abc1 %cut -f2 abc > abc2 %paste abc2 abc1 > xyz

25. Some Details (cont.) • grep, egrep, fgrep grep [-nv...] pattern [file-list] • Search the input text files for lines matcing the pattern %grep Unix doc.1 # Display the lines in doc.1 that contains “Unix” %grep -n Unix doc.* # Display the lines with line numbers %grep -v Unix doc.1 # Display the lines which do not contain “Unix” • sort [-tC…] [-o outfile] [field-list] [file-list] • sort the files %sort +1 list1 # sort list 1 starting from field 2 to the end of the line %sort +2-3 list2 # sort list2 on the third field %sort -n -o list4 list3 sort list3 numerically and place the output in list4

26. diff • diff file1 file 2 • Display different lines that are found when comparing two files • It prints a message that users ed-lide notation (a - append, c - change, d -delete) to describe how a group of lines has changed. • It also describes what changes need to be made to the first file to make it the same as the second file. • Example file1 file2 file3 apples apples oranges oranges oranges bananas bananas kumquats kiwis peaches

27. diff (cont.) %diff file1 file2 3c3 <bananas ----------------- >kumquats %diff file1 file3 1d0 <apples 3a3,4 >kiwis >peaches

28. comm file1 file2 • Takes two sorted text files and print common lines and lines which appear exclusively in one file on separate columns. • col1: lines only in file1, col 2: lines only in file2; col 3: comm file1 file2 %comm file1 file2 apple apple apple banana cantaloup banana grape grape cantaloup orange kiwi grape lemon kiwi lemon orange %comm -12 file1 file2 apple grape

29. tr [-csd…] pattern1 pattern2 • Translate input character to output character based on the input and output patterns • Example %tr ‘[A-Z]’ ‘[a-z]’ <in >out # xlate all letters to lower case. %tr -s ‘\012\011\040’ ‘\012\012\012’ < in > out # xlate blank, tab and newline chars to newline chars and squeeze (-s) consecutive newline char into one %tr -cs ‘[a-z][A-Z]’ ‘[\012*]’ < in > out # change all non-aplphbetic (-c) chars to newline chars and squeeze consecutive newlne char into one. %tr -d ‘\040’ < in > out # delete all blanks.

30. uniq [-cdu…] file-list • Display a file, removing all successive repeated lines • Example: file1: %uniq file1 apple apple banana banana banana apple apple banana %sort fruit | uniq -c apple 2 banana 3 %tr -cs ‘[a-z][A-Z]’ ‘[\012*]’ < fileA | sort | uniq # show a list of distinct words in fileA.

31. find <dir-name> <exp> • Recursively search the directory tree rooted at <pathname> and find all files whose names satisfy <exp> • There are many details in the expression. • Examples: %find . -name \*.doc -print # list all files names ending with .doc %find /etc/source -atime 2 -print # print the names of the files under /etc/source whose lst access time was 2 days ago. %find . -name “[a-z]*” -exec rm {} \; # remove the files under the curent directory whose names begin with a lower case letter. %find / \(-name a.out -o -name “*.o” \) -atime +7 -exec {} \; # remove the object and binary executable files under the root direory which have not be accessed more than 7 days.

32. cpio -i[cdv] -o[cBv] • System V file archive and backup progam • Example %find proj -print | cpio -ocBv > /dev/rmt8 # cpio get file names from stdin. “-o” create archive which is redirected to the tape device. %find proj -print | cpio -ocBv > proj.cpi # get file name from stdin and “-o” createsarchive which is redirected to proj.cpio %cpio -icdv “*.c” </dev/rmt8 # -i read from archive file from the tape device. -d creates directories as needed.

33. tar [options] [file-list] • key := c (create) | t (table of content) |r (append the file) | u (update the file) • options := v (verbose) | b (block) | f (file name follows) | m (use extraction time as the mod file) … • Create/extracting archive files for backup and transporting files %tar cvf proj.tar proj # create archive file proj.tar from file or dir proj %tar xvf proj.tar # extract files in proj.tar % tar tf proj.tar # list of the filenames in proj.tar without extracting data. %tar cf - proj | (cd /newproj/; tar xvpf -) # copy proj to the directory /newproj/. “p” to keep all the protection mode. • cp -r copies a dir tree but all the time info is gone. Tar preserve the time info. %tar cbf 20 proj.tar /usr/local/proj # avoid using full path names. When you extract the file, tar will insist to put fiels to /usr/local/proj.

34. uuencode & uudecode • Generate an ASCII encoded version of the give files • Example: %uuencode file.bin newfile.bin > file.bin.uu # encode file.bin and put the result in file.bin.uu %uudecode file.bin.uu # decode the file file.bin.uu and generate a new file newfile.bin • Sending a dir tree via email %tar cvf proj.tar proj %compress proj.tar # compress proj.tar to proj.tar.Z %uuencode proj.tar.Z proj.tar.Z | mail qli … at the receiving end, extract the mail and save it in xx %uudecode xx %zcat proj.tar.Z | %tar xvf -

35. sed [-n] | [-e] sed_cmd file_lists • A stream editor. It copies the lines from the file_list or stdin to stdout, editing the lines in the process. • Examples: %sed -n ‘/hello/p’ < input > output # copy the lines contains “hello”. -n suppress stdout so only the lines that matches are copied. %sed ‘5,7d’ file1 # delete lines 5 to 7 from file1. File1 is unchanged. %sed ‘s/Unix/UNIX/’ doc2 #replace the first occurrence of Unix in each line by UNIX. %sed ‘s/Unix/UNIX/g’ doc2 # replace all Unix by UNIX

36. awk [-f progfile] [-Fc] [prog] [files] • Pattern matching and stream editor. • Example: • Program awkexample: BEGIN { linetype=0} # initialization NR == 1 { print $1 “ “ $NF} # if it is the first line, print the last field /^$/ { print “This is an empty line” } /^Unix/ { printf(“Line starts with Unix\n %s\n”, $0); linetype=1; next;} /NonUnix$/ { printf(“End with NonUnix\n”); linetype=0; next;} linetype == 1 { print $0} END {printf(“%d lines processed\n”, NR);} # finishing it up

37. awk (cont.) • Test data file (awktest): Line1-field1 this is the las-field This line should not show Unix is simple and difficult Hello world is very simple Next blank line should show I line some other NonUnix This line should not show Bye

38. awk (cont.) • %awk -F” “ -f awkexample awktest Line1-field1 las-field This is an empty line Line starts with Unix Unix is simple and difficult Hello world is very simple Next blank line should show This is an empty line End with NonUnix This is an empty line 11 lines processed • Command line program: most of the awk commands can be used in he command line. %awk ‘{print $2 $1}’ # exchange field1 and field2

39. Other Commands • chown [-R] <new-owner> <file-list> • Only owner and root can chown. (Root only in some systems.Why?) • Example: %chown -R john testdir # john becomes the owner of all files under testdir. • kill [-signal] pid • Sending the specified singal to the process. The process can be programmed to catch most of the signals. • Examples: %kill -1 1 # poke the init process to reinitialize (reading /etc/ttytab) %kill -9 1234 # kill process 1234 • Signals can be spefified by name, e.g., HUP is 1, KILL is 9. • %kill -l # lists signal names

40. Other Commands (cont.) • du [-as….] [dir list] [file list] • Reports the allocated dispace for each file and/or directory specified • -a lists all files, -s lists the grand total of each dir given • Examples: %du -s # print the total disk space used by the files in and under the current dir. %du -s * # print the disk space used by each file and dir in the current dir.

41. make • A tool for maintaining programming projects • make [-isrntqpd] [-f file] [macro-definition] [targets] • It allows the users to specify dependencies among different source and binary files in his/her applications. • -i ignore error code returned by a command • -s silent mode • -r suppress built-in rules • -n no execute mode • -t touch target file • -q question before change • -p printout macro definitions and target descriptions • -d debug mode • -f alernative make file name

42. make (cont.) • Makefile: prog: x.o y.o z.o cc -o prog x.o y.o z.o -lm x.o: x.c def.h cc -c x.c y.o:y.c def.h cc -c y.c z.o:z.c cc -c z.c • make does depth-first search on the dependency graph x.c x.o def.h prog y.o y.c z.o z.c

43. Makefile format • a makefile containing explicit dependency lines in the following format: target1 [target2 …]: [dependency …] [; commands] [#commnets] [<tab> commands] [#comments] • each command line start with a tab character, and no other lines should start with a tab. • commands are bourne shell commands • a set of built-in rules are used by make, e.g.: .c.o: $(CC) $(CFLAGS) -c $< # if .o depends on .c and f.c is newer, compile f.c • Each command is executed by a different subshell.

44. make Macro • Syntax: Name=String. E.g. LIB=/users3/foo/lib • Predefined Macros for C CC=cc AS=as CFLAGS= -O -g LOADLIBS= • Built-in Macro (evaluated each time make reads a dependency line) $* - the basename (suffix removed) of the current target test.o : test.h cc -c $*.c # cc -c test.c $@ - the full target name $? - the list of dependencies that are newer than the target libops: ineract.o shed.o gen.o ar r $@ $? # put any .o files newers than libops into libops $$ - the $ sign.

45. make -- Suffix Rules • make uses some conventions to simplify the makefille. Example: prog: x.o y.o z.o cc -c …. # make finds files which can generate the .o files. Eg. x.c. If x.c is newer than x.o, x.c is compiled. • Suffix rules are predefined, generalized descriptions: .SUFFIXES: .o .c .s # define the suffix to be consdiered significant .c.o: $(CC) $(FLAGS) -c $< .s.o: $(AS) $(ASFLAGS) -o $@ $< • $< evaluates to whatever the dependecies triggered the rule. $* is similar to $< except that the suffix is removed. Both are used only in suffix rules

46. An Example of make MYPROG=/usr/local/myprog INCLUDE=$(MYPROG)/include BIN=$(MYPROG)/bin LIB=$(MYPROG)/lib CFLAGS= -g -I$(INCLUDE) .c.o: $(CC) $(CFLAGS) -c $*.c HEADERS=interface.h dbms.h SOURCE =driver.c interface.c dbms.c OBJECT=$(SOURCE:.c=.o) app: ($OBJECT) $(CC) -o app $(OBJECT) -l$(LIB) (continued)

47. make-example (cont.) print: @echo print source files # @ suppress the comman line printing @for file in $(SOURCE) \ do \ pr -n $$file; \ # $$ to make a $-sign for the shell command done clean: @rm -f *.o • Usage: make app make clean

48. What’s Next? • Shell scripts!

49. Guidelines for writing Unix Commands/Tools/Scripts • Standard command format • Recognize meta-characters (handle multiple files) • Standard I/O (stdin,stdout, stderr. If file arg is absent use std) • Keep messages and prompts to a minimum. • Provide verbose options • Input/output data should be text whenever possible. • Use dot files and environment variables for frequently used info. • Use standard library and tools to save coding effort.

50. Shell Script • Bourne Shell/Korn Shell • Topics: • pass arguments • global and local variables • macro • functions • Invoking a shell script $shell_script_file or $sh -options shell_script_file • the script file must have execute-permission.