CPS 393 Introduction to Unix and C

CPS 393Introduction to Unix and C START OF WEEK 3 (UNIX) 11/15/2014 Course material created by D. Woit Course material created by D. Woit 1

Comment on quotes • Enclosing characters in single quotes preserves the literal value of each character in quotes. • Enclosing characters in double quotes preserves the literal value of all characters within the quotes with the exception of ‘$’, ‘`’, ‘\’ and when history is enabled ‘!’. • Example: • varx=1 • echo $varx • 1 • echo “$varx” • 1 • echo ‘$varx’ • $varx

Cut, Paste Commands • cut extract specified input columns • paste combine input columns • both take input from stdin or files • Examples: • cut -c8 cutexample # outputs 8th column of each line of myfile • cut -c5-7,25- # outputs columns 5,6,7,25... of each line of stdin • cut -f2 table # outputs second field of each line of file table (default field separator for –f switch is tab) • cut -d' ' -f3 # outputs field 3, fields delimited by one space (here field separator is changed from tab to space) • paste file1 file2 # linei to stdout contains linei of f1 followed by a tab # followed by linei of f2 • cat –A shows tabs, newlines etc. • expand can convert tabs to spaces 11/15/2014 Course material created by D. Woit Course material created by D. Woit 3

HMWK • Create a file called stdnt-file with last-names in column 1, first-names in column 2, id numbers in column 3, userids in column 4.What combination of commands (and possibly temp files) can you use to create a file, stdnt-ids, that contains only the id numbers and userids from stdnt-file? • How can you combine commands (and possibly temp files) to create another file, stdnt1, that contains only the first and last students in stdnt-file? • How can you combine commands to find out exactly how long userid dwoit (or whoever you want) has been idle (has not been typing anything)? You will find the "w" command useful. It tells you who is logged in, and what they're doing. For an explanation do: man w. Note that "dwoit" (or whoever) will likely have multiple entries (one for each "terminal" connection), so your output may have multiple lines. 11/15/2014 Course material created by D. Woit Course material created by D. Woit 4

Transformers and Translators • sed (stream editor) • Example change string unix into UNIX • sed -e "s/unix/UNIX/" sedexample • -e says the string following is an edit command • the "s" in the string says substitute last string for first • But only its first occurrence in line • sedexample is name of file to edit • output to stdout (but could redirect with > or >>) • input is taken from stdin or files • Regular expressions (like in grep) can be used. 11/15/2014 Course material created by D. Woit Course material created by D. Woit 5

sed (stream editor) cont. • sed -e "s/dog/cat/g“ sedexample • changes all occurrences of "dog" to "cat" on each line (g=global) • sed -e "s/xx*/1234/" fx • changes first occurrence of x or xx or xxx etc on a line to 1234 in file fx • Character before * , i.e. x* can occur 0 or more times. • sed -e "1,5s/HI/hi there/" • changes "HI" to "hi there" on lines 1-5 of stdin • sed -e "/UNIX/ s/shell/Shell/" f1.txt • changes first "shell" to "Shell" only in lines containing the string "UNIX" • sed -e "/unix/ d"< myfile#deletes all lines containing string unix • sed -e "1,5d"< myfile delete lines 1-5 11/15/2014 Course material created by D. Woit Course material created by D. Woit 6

sed (stream editor) cont. • Sed commands can be submitted from a file e.g: • File sedfile has two lines: • s/shell/Shell/ • s/c language/ C Language/ • And can be used as sed –f sedfile infile • if we want to include '/' in the change, we use another character to delimit, such as ? Instead of /. • sed -e "s?his/her?their?g“ • sed -e "sXhis/herXtheirXg“ • Note: sed can be invoked in sequence e.g. change A to B and B to y • /home/dwoit> sed -e 's/A/B/' -e 's/B/y/‘ • A • Y 11/15/2014 Course material created by D. Woit Course material created by D. Woit 7

tr (translator ) • translates input, character by character, and writes to stdout given translation table specified by from-string and to-string • uses a 1:1 correspondence between from/to strings • tr ":" ";" <inf • changes all colons into semi-colons (uses file inf as input) • tr "x" "AX" <inf • changes all blanks to "A" and all "x" to "X“ • tr "A-Z" "a-z" <infile >outfile • changes all uppercase to lowercase letters (so it would make stdout a lowercase version of stdin) • tr "a-zA-Z" "A-Za-z" <infile • changes all upper to lower, and lower to upper • can use backslash escapes, \n (newline), \t (tab) etc • tr " " "\n“ • changes all blanks to newlines 11/15/2014 Course material created by D. Woit Course material created by D. Woit 8

tr (translator ) cont. • can use octal codes, if begin with a \ (see man ASCII or ascii command) • tr " x" "\012y" <inf • changes all blanks into newlines (\012 is octal newline) and all "x" to "y" • can use Character Classes (see man grep) • tr '[:lower:]' '[:upper:]' • tr -d "\n" <inf • removes all newline chars from inf (-d=delete) and displays on stdout • tr -d "a-z" <inf • removes all lower case letters from inf • tr -s " " " " <inf • (-s = squeeze) optionwilltake all repeated occurrences of char1 and replace by just char2. In example above char1=“ “ and char2=“ “ • i.e., all strings of 1 or more blanks are replaced by just one blank 11/15/2014 Course material created by D. Woit Course material created by D. Woit 9

tr (translator ) cont. • More examples with –s squeeze option: • tr -s "Ae" " Y" <inf • All strings of one or more "A"s are replaced by one space • All strings of one or more "e"s are replaced by one "Y" • tr -s '[:punct:][:blank:]' '\n' • squeezes all punctuation and horizontal whitespace into one newline (so it puts each word of input on its own line) • More information for tr command can be found in man pages 11/15/2014 Course material created by D. Woit Course material created by D. Woit 10

Pipes • A pipe | connects stdout of one command to stdin of another • e.g., ls | more # lists all files in current dir in alpha order, one screen at a time • ls -t | head # lists the 10 most recently modified files in this dir. ls -t ------------------> head ---------> screen • redirection is also possible : ls -t | head >outfile • see if string "IEEEtran" is anywhere in the first 10 lines of huge file bigfile.tex: • head < bigfile.tex | grep "IEEEtran" • big.file ------> head -------------> grep ------> screen • stdin stdout stdin stdout 11/15/2014 Course material created by D. Woit Course material created by D. Woit 11

Pipes cont. • last | head • see last few times someone has logged in (last's output is LONG) • pipes contribute to shell's usefulness and flexibility • no need to create new commands to perform a task– we can *combine* existing commands • existing commands to perform more complex tasks • no need for "temp" files 11/15/2014 Course material created by D. Woit Course material created by D. Woit 12

Tee command • saves the data passing through the pipe • writes stdin to stdout *and* to a given file • Example we want to change • head < bigfile.tex | grep " IEEEtran" • To • head < bigfile.tex |tee first10| grep "IEEEtran" • Does exactly same, but also saves first 10 lines of bigfile.tex in file first10 11/15/2014 Course material created by D. Woit Course material created by D. Woit 13

Example shell programs - #2 #!/bin/bash #source prof1 #prints names of a prof logged in #name supplied as command line argument who | grep $1 | cut -c1-8 See what happens if you use prof1 without an argument. How do you fix it? (hint: use quotes) who | grep “$1” | cut -c1-8 Corrected pgm prof1a will interpret lack of argument as a null string and print all users 11/15/2014 Course material created by D. Woit Course material created by D. Woit 15

HMWK • Use various UNIX commands and pipes to do the following (do not • use any temp files): • 1. List, in long form, all those files in your home directory that have read, write and execute perms for you, and whose name contains the string "txt". • 2. Do the above question again, but list in short form (just the file names.) • 3. List any lines in the first 10 lines of a file, myf, that contain the word "dog". (you should create different versions of myf to test your commands.) • 4. Find all lines of file, myf, that contain all the words "cat", "dog" and "mouse", in any order, and start with the letter "A". • 5. List all the files in your home dir that contain the string "so" somewhere within them. Next, just list the *number* of files in your home dir that contain the string "so" (you should use, among other commands, the command "wc -l", word-count (lines option), which you should look up in the man pages.) 11/15/2014 Course material created by D. Woit Course material created by D. Woit 16

HMWK cont. • 6. Use only ls, grep, cut, pipes to answer this question. What sequence of pipes and commands could you issue from your home dir to list, in alphabetical order, the absolute path names of all the directories in your file system (if you NEED to, use the sort command.) • 7. List the names of any subdirectories of your home directory that have rwx permissions for user and group, and no permissions for others--note: list just the directory names in short form (the names only.) • 8. List just names of all files/directories that have no extension; • 9. List just names of all files/directories that contain no vowels at all. 11/15/2014 Course material created by D. Woit Course material created by D. Woit 17

HMWK cont. • 10. Write a shell program called nw that takes either zero or one command line arguments. • nw with no arguments should print out the 10 newest (most recently modified) files/dirs in the current directory. • nw -n (nw with one argument, -n) prints out the n newest files/dirs in the current directory. • e.g., nw -3 prints out the 3 most recently modified files/dirs • If the argument is incorrect, then your program is allowed to do unpredictable things! • 11. List names in reverse alphabetical order, of directories that have "r" permissions for "other". (the "reverse" option of the sort command may be useful.) • 12. List in alphabetical order all files in the current directory that have been edited exactly 3 days ago. Do not list the same file more than once. (the uniq command will eliminate duplicate lines). 11/15/2014 Course material created by D. Woit Course material created by D. Woit 18

Foreground/Background Processing, • execute commands in background with & • /home/dwoit> find / -name firefox -print >find.out 2>/dev/null & • /home/dwoit> # free to do other work • /home/dwoit> grep "^the" bigfile.tex >grep.out 2>/dev/null & • How to list the active processes? • /home/dwoit> ps -l # lists processes, ids, status • Status can be R running, S suspended (waiting for an event to complete), D uninterruptable sleep, T stopped by job control signal • ... PID ... CMD • ... 2498 ... find • ... 1483 ... grep • ... 3398 ... ps 11/15/2014 Course material created by D. Woit Course material created by D. Woit 19

Foreground/Background Processing • Note: ps x gives shorter list or alternatively • ps –u user • A shell associates a job with each pipeline • /home/dwoit> jobs • [1] Running find ... & • [2] Running grep ... & • /home/dwoit> fg %1 # now in foreground--no prompt until done • ^z # suspends foreground process (ctrl-z) • /home/dwoit> bg %1 # starts find again in background • /home/dwoit> kill %1 or • /home/dwoit> kill 2498 # find cmd is killed (^c if fg process) • might need guaranteed kill: • kill -9 2498 11/15/2014 Course material created by D. Woit Course material created by D. Woit 20

Command History Editing for bash • !! re-execute last command issued • !-n re-execute current command minus n • !cmd re-execute last command that started with string "cmd" • ^str1^str2^ re-execute last cmd but substitute str2 for str1 • history (or fc -l) lists last commands (each is numbered) 11/15/2014 Course material created by D. Woit Course material created by D. Woit 21

Examples • /home/dwoit> ls • lab1.jav lab2.jav lab1.c393 lab2.c393 • /home/dwoit> !! • lab1.jav lab2.jav lab1.c393 lab2.c393 • /home/dwoit> find . -name "*.jav" • lab1.jav lab2.jav • /home/dwoit> ^jav^c393^ • lab1.c393 lab2.c393 • /home/dwoit> history • 8 ls • 9 ls • 10 find . -name "*.jav" -print • 11 find . -name "*.c393" -print • 12 history 11/15/2014 Course material created by D. Woit 22 Course material created by D. Woit

Examples cont. • /home/dwoit> !10 • lab1.jav lab2.jav • History in editing: • set -o vi makes command line editing use vi • set -o emacs makes command line editing use emacs • if set -o vi, then • esc-k brings back last command for editing (using vi) repeated "k"s move back in history, "j" move forward • if set -o emacs, then • up-arrow brings back last command for editing (using emacs) repeated "up-arrows"s move back in history, "down-arrows" move forward 11/15/2014 Course material created by D. Woit Course material created by D. Woit 23

HMWK • When the "find" command tries to look in a directory you don't have permission to read, it prints an error message on stderr (try this out to see what happens). If you redirect stderr to stdout, then "find" will print both on stdout and you can pipe all the output. Use find, pipes, grep and wc to print the *number* of directories that "find" cannot read in your filesystem. • -- • Do the same thing, but have the numberwritten into a file called num.no.read (using redirection). • -- • Repeat the previous question (output to file) but for the entire filesystem (i.e., start your "find" search at root)--but run the command in the background, or you will have to wait a long time for it to finish. If you want to logout before it's done, kill it first. 11/15/2014 Course material created by D. Woit Course material created by D. Woit 24

CPS 393 Introduction to Unix and C