COMP 400 Shell (Part 2)

1. COMP 400Shell (Part 2) George Vaughan Franklin University

2. Topics • VNC • Root • Special Characters in Shell • Useful Unix Commands for Scripting • computations in Shell Scripts • Flow Control in Shell Scripts * Note: Your Sys Admin book has a nice description of shell programming in the appendix

3. VNC • VNC is a free, handy tool for remotely managing Windows, Linux or Unix machines • Virtual Network Computing • URL: http://www.uk.research.att.com/vnc/ • Allows one to administer the Desktops of many computers on one computer. • Example: I can manage my Linux and Windows desktops (hence the machines themselves) from my laptop.

4. ROOT • ROOT is an overloaded term in Unix • ROOT user is the super-user. Root can see and do anything. (You must be root to add new users to the system) • ROOT directory is the parent of all UNIX directories (/tmp is under the root directory ‘/’)

5. Special Shell Characters • The Dot “.” Command • The Dot Command allows you to run a script in the current shell (i.e. a new shell is not created). • Note: using the dot command to run a script can affect the current shell. • Example (using 2 telnet sessions): $ ps -u gvaughan PID TTY TIME CMD 1434 pts/0 00:00:00 bash <= Login shell from telnet #1 12594 pts/2 00:00:00 bash <= Login shell from telnet #2 12668 pts/2 00:00:00 sleep <= sleep from script on telnet #2 12669 pts/0 00:00:00 ps <= ps on telnet #1

6. Special Shell Characters • “\” Line Continuation Character • allows one logical line in shell to fit on more than one physical line in a file • Example: ls | \ wc –l • will return a count of the files in the current directory. • This is equivalent to: ls | wc -l • Useful for long source lines - improves readability

7. Special Shell Characters • “;” Command Separator • Used to separate multiple commands on same physical line • Example: cd bin; ls • This is equivalent to: cd bin ls • Commands separated by semi-colon are executed serially. How is this different from the pipe? • 2 commands separated by a semi-colon are still executed in 2 distinct processes.

8. Special Shell Characters • “(…)” Sub Shell Construct • Allows one or more commands to be executed in a sub shell. • Behavior is similar to putting commands in a script. • Useful to protect current shell form side effects. • Useful for redirecting output from many commands • Examples: cd $HOME; (cd /bin; pwd); pwd (ls; pwd; ls) > trash #send output of all 3 commands to trash (ls; pwd; ls) & #run all cmds in background

9. Special Shell Characters • Example: >date; ps Mon May 5 21:16:38 EDT 2003 PID TTY TIME CMD 12325 pts/1 00:00:00 bash 12367 pts/1 00:00:00 ps • Why don’t we see ‘date’ in the ‘ps’ report? • Example: >(date; ps) Mon May 5 21:15:57 EDT 2003 PID TTY TIME CMD 12325 pts/1 00:00:00 bash 12361 pts/1 00:00:00 bash 12363 pts/1 00:00:00 ps • Notice the 2 instances of bash

10. Special Shell Characters • “&” Run command in background • Will run command in background. • Example (with 1 telnet session): >test01 & [1] 12820 <= process ID of script in background >ps PID TTY TIME CMD 1434 pts/0 00:00:00 bash <= login shell 12820 pts/0 00:00:00 bash <= script shell 12821 pts/0 00:00:00 sleep 12822 pts/0 00:00:00 ps

11. Special Shell Characters • “|” Pipe • Take the output of command 1 and use it as the input of command 2 • Example: ps -e | wc -l • the ‘ps’ and ‘wc’ commands are running concurrently in 2 distinct processes. • You can pipe multiple commands: who | awk '{print $1 }' | sort -u | wc -l • This reports the number of unique active logins on the system

12. Special Shell Characters • “>” Redirect Output • Sends output of command to file. • Will overwrite file • Examples: ls > trash (date;ps) > trash • “>>” Append Output • Appends output of command to end of file. • Example: ls >> trash

13. Special Shell Characters • “<“ Redirect input • Useful for reading command input from a file. • Example: • Use redirection to automatically edit a bunch of files. • Assume we have a file named control.vi • Inside the file control.vi we have the following vi commands (change bsh to ksh, write file, quit): :g/bsh/s//ksh :wq

14. Special Shell Characters • Redirection Example (continued) • We can now use the control.vi file to automatically edit another file such as myscript: vi myscript < control.vi • We can imagine how we might generalize this using a shell script with loops… • The sed command is very useful for this purpose (we will see sed described later…)

15. Special Shell Characters • We can also use input redirection to generate a list as arguments for a command. • Example: • Assume we have a file “trash” with the following contents: /etc /bin • We could use the file “trash” as input to “ls”: ls $(<trash) #(equivalent to ls /bin /etc) • ls will list the contents of /etc and /bin.

16. Special Shell Characters • “.” Current Directory (not the same as the dot command) • “.” is a shortcut for referencing the current directory • Example: ./myApp #execute the program myApp which exists #in the current directory. • “..” Parent Directory • “..” is a shortcut for referencing the parent directory • Example: ../myApp #execute the program myApp which exists #in the directory above.

17. Useful Unix Commands for Scripting • Unix has many commands – some are especially useful for scripts. • The following commands are feature rich – check man page.

18. Useful Unix Commands for Scripting • grep – prints any line in the input stream that contains a specified substring • When filtering output, think of grep as tool to make horizontal cuts (line selection) • Supports regular expressions with –e option • Example: Show me all of my processes: ps –ef | grep $LOGNAME • Useful for filtering out items of interest • “grep -v string” will match every line that doesn’t contain the string. • grep -i string (ignores case)

19. Useful Unix Commands for Scripting • awk, nawk – a filter like scripting language • When filtering output, think of ‘awk’ as tool to make horizontal cuts (line selection) and vertical cuts (field selection). • Quite powerful (it is a programming language) • Can be used to select and/or rearrange fields in input • Example: • ps –ef shows the following (only subset is shown): > ps -ef | more UID PID PPID C STIME TTY TIME CMD root 0 0 0 Dec 13 ? 0:01 sched root 1 0 0 Dec 13 ? 1:12 /etc/init - root 2 0 0 Dec 13 ? 0:00 pageout root 3 0 1 Dec 13 ? 529:56 fsflush

20. Useful Unix Commands for Scripting • awk, nawk (Cont.): • Assume we only want to see the command and the owner, in that order: >ps –ef | awk '{ print $9, $1 }' sched root /etc/init root pageout root fsflush root /usr/lib/utmpd root /usr/sbin/keyserv root /usr/lib/saf/sac root /usr/lib/devfsadm/devfseventd root • Notice the use of single quotes in AWK script (why?)

21. Useful Unix Commands for Scripting • awk, nawk (Cont.): • Another example: • Each line in the password file (/etc/passwd) is variable length and delimited by a colon ‘:’ pangh:x:118726:100:Hanbin Pang:/export/home/pangh:/bin/ksh (subset of file) • If I want to retrieve the user’s ID and name (field positions 1 and 5), I can use awk: >cat /etc/passwd | awk '{ FS=":"; print ($1, $5) }' vaughang George Vaughan(subset of output) • ‘FS’ in the example above is used to inform awk what character(s) are used as field separators (space is the default)

22. Useful Unix Commands for Scripting • cut- When filtering output, think of ‘cut’ as tool to make vertical cuts (column selection). • When can make vertical cuts based on character position or by field (awk is better for this task).

23. Useful Unix Commands for Scripting • Example of ‘cut’ by character position: • The command ‘ls –l’ can be used to show everyone's permissions to a file: >ls -l total 8 -rwxr-xr-x 1 vaughang faculty 415 Sep 6 09:32 dirsize -rwxr-xr-x 1 vaughang faculty 151 Sep 9 21:27 ex0280 drwxr-xr-x 2 vaughang faculty 512 Sep 6 00:06 junk -rw-r--r-- 1 vaughang faculty 20 Sep 6 00:07 trash2 • We can use cut based on character position to just display permissions for owner (info in columns 2 through 4): >ls -l | grep -v total | cut -c2-4 rwx rwx rwx rw- • In the example above, we used ‘grep –v’ to strip out the first line of ‘ls –l’ which contains information about the total number of disk blocks used by directory.

24. Useful Unix Commands for Scripting • Example of ‘cut’ by field position: • Each line in the password file (/etc/passwd) is variable length and delimited by a colon ‘:’ pangh:x:118726:100:Hanbin Pang:/export/home/pangh:/bin/ksh (subset of file) • If I want to retrieve the user’s ID and name (field positions 1 and 5), I can use cut: >cat /etc/passwd | cut -f1,5 –d’:’ vaughang:George Vaughan(subset of output) • The ‘-d’ option is used to tell ‘cut’ which character (or characters) serve as the delimiter (TAB is the default). • Note: awk is a more powerful tool for making vertical cuts based on fields.

25. Useful Unix Commands for Scripting • sort – sorts input. • can sort on any column (defaults to column 1). • the –u option will only show unique rows. • Example: • Count the number of active processes: ps -ef | wc –l • Count the number of unique process owners: ps -ef | awk '{ print $1 }' | sort -u | wc -l

26. Useful Unix Commands for Scripting • sed – stream editor • Used to edit input on the fly, powerful • Examples: • Don’t list processes owned by root: ps -ef | sed '/root/d' • Strip out blank lines: cat trash | sed '/^$/d' • Change each comma to a blank, globally cat trash | sed 's/,/ /g'

27. Useful Unix Commands for Scripting • find – recursively search directories for files of interest • Book provides a good overview • Example: • Count all files in the system that end in .html find / –name *.html –print | wc –l • date – print date and time • contains useful format control useful for creating file suffices: • Example: • Save a listing of my files as of today FILE_NAME=file_list_`date '+%m%d%y'` • Produces file name like: file_list_011203

28. Useful Unix Commands for Scripting • diff – show differences between 2 ASCII (text) files. • Output is cryptic, can be used to convert one file to another, allows SCCS to store deltas. • If files are the same, no output. • Example diff file_1 file_2 | wc –l #if zero, files are same • cat – dump the contents of a file to standard out: • Example: • Count the number of registered users on system by counting entries in password file: cat /etc/passwd | wc -l

29. Useful Unix Commands for Scripting • touch – change file timestamps. • useful for ‘marking a file’. • If file doesn’t exist, it is created with zero length. • ‘touched’ file can serve as a record when something happened. • Example: • touch the file ‘last_backup’ when backup completes: touch last_backup

30. Useful Unix Commands for Scripting • basename – strip of the path from a fully qualified file name. • Commands like ‘find’ produce lists of files with pre-appended paths. Use basename if you want just the filename. • Example: echo `basename vaughang/comp400/lecture_notes/notes1.pp` • produces the output ‘notes1.pp’

31. Useful Unix Commands for Scripting • which – tells you which copy of command you will execute if you execute the command. • Determined by your PATH variable. • Useful when you think you are using one copy when in fact you are using another copy. • Example: which ls • produce the following result: /bin/ls

32. Useful Unix Commands for Scripting • set –x • Used to turn on tracing in shell script. • Useful for debugging • set +x • Turns off shell tracing • Example: set –x date set +x date

33. Useful Unix Commands for Scripting • exit – terminate script and return integer for return code • Example: exit 2 • Will cause the script to terminate with a return code of 2. • Useful for error handling by other scripts. • Which shell variable will contain the return code from exit (see notes from previous lecture)?

34. Useful Unix Commands for Scripting • shift • The shift command will shift all command line arguments to the left by 1. • So, after using shift, $1=$2, $2=$3, $3=$4,… • shift can be used in a loop.

35. How Script Can Change Parent Shell Variables (“.” Command) • By default, shell variables are local to script. • How can we make a shell script change contents of parent shell? • With the “.” command: • example: . myScript • myScript will change variables in parent shell • why?

36. How Parent Shell Can Initialize Shell Variables In Script (“export” command) • When we use the export command on a shell variable, it becomes visible to subshells and scripts. • Exported variable is something like a global variable. • In the example below, $MY_VAR is set to 23 and then exported. $MY_VAR will be visible inside myScript export MY_VAR=23 myScript

37. Using “.” and “export” commands together • If we use a “.” command on a script and inside the script we “export” variables, then exported variables will not only be visible in parent shell but also in all future subshells. • Example: PATH variable in .profile • Notice that we always execute .profile with the “.” command (what would happen if we didn’t?)

38. A Note On Examples • Source for all examples in this and future lectures are on codd (einstein) at: /export/home/vaughang/public_html/comp400/examples • and on the web at: • http://cs.franklin.edu/~vaughang/comp400/examples/ • All shell examples are written in Korn Shell (ksh) although they should be quite compatible with Bash. • In all examples, the ‘greater-than’ symbol, ‘>’ will be used to indicate the shell prompt. So, if we execute the date command, >date Thu Sep 2 20:43:58 EDT 2004 we only type ‘date’ and not ‘>date’. • All scripts should be able to execute on codd (einstein) which is a Solaris machine, or any Linux distribution that supports Korn shell, located at /bin/ksh.

39. Flow Control in Shell Scripts • We will explore the following flow control constructs: • for • while • case

40. Flow Control in Shell Scripts • “for” flow construct • Format: for VAR in list do statement-body done • Loop for each element in list, assigning each element to $VAR per loop iteration.

41. 0001 #!/bin/ksh 0002 0003 ########### 0004 # File: ex0240 0005 # 0006 # Test each entry in arg 0007 # to see if file or dir 0008 ########### 0009 for i in $1/* 0010 do 0011 file $i 0012 done Line 9: For each file in dir specified by arg, determine file type. OUTPUT: >ex0240 $PWD ex0010: executable /bin/ksh script ex0020: executable /bin/ksh script ex0030: executable /bin/ksh script ex0040: executable /bin/ksh script Example ex0240

42. 0001 #!/bin/ksh 0002 0003 ########### 0004 # File: ex0250 0005 # 0006 # Test each entry in file 0007 # to see if file or dir 0008 ########### 0009 for i in $(<$1) 0010 do 0011 file $i 0012 done Line 9: Use the file specified by $1 as the list for the for loop OUPUT: >ls > trash >ex0250 trash ex0010: executable /bin/ksh script ex0020: executable /bin/ksh script ex0030: executable /bin/ksh script ex0040: executable /bin/ksh script ex0050: executable /bin/ksh script trash: ascii text Example 0250

43. Flow Control in Shell Scripts • “while” flow construct • Format: while [ expression ] do statement-body done • Keep looping while expression is true.

44. 0001 #!/bin/ksh 0002 0003 ########### 0004 # File: ex0260 0005 # 0006 # Create a counter 0007 ########### 0008 COUNTER=0 0009 while [ "$COUNTER" -lt 5 ] 0010 do 0011 COUNTER=$(( $COUNTER + 1 )) 0012 echo $COUNTER 0013 done Line 9: Loop as long as COUNTER is less than 5. Line 11:Math computations are performed inside $(( )) Line 11:Notice space after “$((“ and before “))” – they are necessary. OUTPUT: >ex0260 1 2 3 4 5 Flow Control in Shell Scripts

45. Flow Control in Shell Scripts • “case” flow construct • Format: case value in pattern1) command1 commandn;; patternn) command1 commandn;; esac

46. 0001: #!/bin/ksh 0002: 0003: ########### 0004: # File: ex0270 0005: # 0006: # translate money 0007: ########### 0008: COIN=$1 0009: case "$COIN" in 0010: 1) echo penny;; 0011: 5) echo nickel;; 0012: 10) echo dime;; 0013: 25) echo quarter;; 0014: *) echo not a coin!;; 0015: esac Line 9: Use ‘case’ to translate coin value to coin name. Output: $ ex0270 25 quarter $ ex0270 3 not a coin! Example ex0270

47. More Than 9 Command Line Arguments • As we have seen, a shell script can access command line arguments with shell variables such as $1, $2, $3, etc. • Shell however, will only allow us to access up to 9 command line arguments ($10 will not work). • The shift command is one way to address this, see: http://www.shu.ac.uk/schools/cms/teaching/ps/unix/params.html

48. More Than 9 Command Line Arguments • Another technique is to use a special shell variable, $* • $* contains all command line arguments. • We can use $* to loop through an indeterminate number of command line arguments…

49. 0001 #!/bin/ksh 0002 ########### 0003 # File: ex0280 0004 # 0005 # process more than 9 args 0006 ########### 0007 ARG_LIST="$*" 0008 for i in $ARG_LIST 0009 do 0010 printf "%03d " $i 0011 done 0012 printf "\n" Line 7: Store ‘n’ arguments in $ARG_LIST Line 8: Loop across all arguments Line 10: print each argument without a new line right justified zero padded Line 12: print a “new line” OUTPUT: ex0280 1 2 3 4 5 6 7 8 9 10 11 001 002 003 004 005 006 007 008 009 010 011 More Than 9 Command Line Arguments

50. Shell Arrays • Shell supports arrays • Setting Array Elements: MY_ARRAY[3]=34 MY_ARRAY[$COUNTER]=5 • Accessing Array Elements: echo ${MY_ARRAY[3]} echo ${MY_ARRAY[$COUNTER]}