Advanced unix
Download
1 / 62

Advanced UNIX - PowerPoint PPT Presentation


  • 205 Views
  • Updated On :

Advanced UNIX. 240-491 Special Topics in Comp. Eng. 1 Semester 2, 2000-2001. Objectives to discuss five useful filters: tr , grep , awk , sed , and find. 4. Filters (Part II, Sobell). 1. tr. format: tr [options] string1 [string2]

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Advanced UNIX' - medwin


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Advanced unix l.jpg
Advanced UNIX

240-491 Special Topics in Comp. Eng. 1Semester 2, 2000-2001

  • Objectives

    • to discuss five useful filters:tr, grep, awk, sed, and find

4. Filters(Part II, Sobell)


Slide2 l.jpg
1. tr

  • format:

    tr [options] string1 [string2]

  • trreads its standard input and translates each character in string1to the corresponding character in string2


Examples l.jpg
Examples

$ echo 12abc3def4 | tr ’abcdef’ ’xyzabc’12xyz3abc4

$ echo 12abc3de4 | tr ’[a-c][d-f]’ ’[x-z][a-c]’12xyz3abc4

$ cat foo.txt | tr ’[A-Z]’ ’[a-z]’


Slide4 l.jpg

$ tr ’\015’ ’ ’ < file1 > file2

  • \015 is carriage return

    $ cat mail.txt | tr -s ’ป’ ’ ’ > new-mail.txt

  • ป represents tab; could write \011

  • -s means remove duplicates of string2in output

    $ echo Can you read this? | tr -d ’aeiou’Cn y rd ths?


Rot13 text l.jpg
“rot13” Text

Popular in 1970-1980’s.

$ echo Gur chapuyvar bs gur wbxr vf ... | tr ’[N-Z][A-M][n-z][a-m]’ ’[A-M][N-Z][a-m][n-z]’The punchline of the joke is ...


2 grep l.jpg
2. grep

  • Format:

    grep [options] pattern [file-list]

  • Search one or more files, line by line, for a pattern (a regular expression). Actions taken depend on options.


Variants of grep l.jpg
Variants of grep

  • grep Uses basic RE pattern

  • fgrepFast grep. Pattern can only be an ordinary string.

  • egrepExtended grep. Pattern can use full REs.


Grep options l.jpg
grep options

  • -c print a count of matching lines

  • -i ignore case in pattern during search

  • -l list filenames with match

  • -n precede each matching line by a line number

  • -v print lines that do not match pattern


Examples9 l.jpg
Examples

File testa File testb File testcaaabb aaaaa AAAAAbbbcc bbbbb BBBBBff-ff ccccc CCCCCcccdd ddddd DDDDDdddaa

continued


Slide10 l.jpg

  • $ grep bb testaaaabbbbbcc

  • $ grep -v bb testaff-ffcccdddddaa

  • $ grep -n bb testa1: aaabb2: bbbcc

continued


Slide11 l.jpg

  • $ grep bb *testa: aaabbtesta: bbbcctestb: bbbbb

  • $ grep -i bb * $ grep -i BB * testa: aaabb testa: aaabb testa: bbbcc testa: bbbcc testb: bbbbb testb: bbbbb testc: BBBBB testc: BBBBB


Fancier patterns l.jpg
Fancier Patterns

  • $ grep ’fun..ion’ file

  • $ grep -n ’^#define’ file

  • $ grep ’^#de[a-z]*’ file

  • $ egrep ’while|if’ *.c

  • $ egrep ’[0-9]+’ *.c


3 awk l.jpg
3. awk

  • format:

    awk program file-list

    awk -f program-file file-list

  • awkis a pattern scanning and action processing language

  • The action language is very like C.


Overview l.jpg
Overview

3.1. Patterns & Actions

3.2. awkProcessing Cycle

3.3. How awkSees a Line

3.4. Pattern Expressions

3.5. ‘,’ Range Operator

continued


Slide15 l.jpg

3.6. Many Built-in Functions

3.7. BEGINand END

3.8. First awkProgram File: pre_header

3.9. Action Language

3.10. Associative Arrays


3 1 patterns actions l.jpg
3.1. Patterns & Actions

  • An awkprogram consists of:

    pattern {action}pattern {action} :


3 2 awk processing cycle l.jpg
3.2. awk Processing Cycle

1. Read next input line.

2. Apply all awkpatterns sequentially.

3. If a pattern matches, do its action.

4. Go to step (1).


Example l.jpg
Example

  • $ cat carsplym fury 77 73 2500chevy nova 79 60 3000ford mustang 65 45 10000volvo gl 78 102 9850ford ltd 83 15 10500chevy nova 80 50 3500fiat 600 65 115 450honda accord 81 30 6000ford thundbd 84 10 17000toyota tercel 82 180 750chevy impala 65 85 1550ford bronco 83 25 9500

continued


Slide19 l.jpg

  • $ awk ’/chevy/ {print}’ carschevy nova 79 60 3000chevy nova 80 50 3500chevy impala 65 85 1550

  • $ awk ’/chevy/’ carschevy nova 79 60 3000chevy nova 80 50 3500chevy impala 65 85 1550

  • $ awk ’/^h/’ carshonda accord 81 30 6000


3 3 how awk sees a line l.jpg
3.3. How awkSees a Line

  • awkviews each line as a record consisting of fields separated by spaces.

  • Each field is referred to by a variable called $<number>:

    • $1, $2, $3, etc.

    • $0 refers to the whole line (record)

  • The current line number is stored in NR

continued


Slide21 l.jpg


3 4 pattern expressions l.jpg
3.4. Pattern Expressions

  • Format:

    variable OP pattern

  • OPforms:

    • matching: ~ !~

    • ariithmetic: < <= == != >= >

    • boolean: && || !

continued


Slide23 l.jpg

  • $ awk ’$1 ~ /h/’ carschevy nova 79 60 3000chevy nova 80 50 3500honda accord 81 30 6000chevy impala 65 85 1550

  • $ awk ’$1 ~ /^h/’ carshonda accord 81 30 6000

continued


Slide24 l.jpg

  • $ awk ’$2 ~ /^[tm]/ {print $3, $2, “$” $5}’ cars65 mustang $1000084 thundbd $1700082 tercel $750

  • $ awk ’$3 ~ /5$/ {print $3, $1, “$” $5}’ cars65 ford $1000065 fiat $45065 chevy $1550

continued


Slide25 l.jpg

  • $ awk ’$3 == 65’ carsford mustang 65 45 10000fiat 600 65 115 450chevy impala 65 85 1550

  • $ awk ’$5 <= 3000’ carsplym fury 77 73 2500chevy nova 79 60 3000fiat 600 65 115 450toyota tercel 82 180 750chevy impala 65 85 1550

continued


Slide26 l.jpg

  • $ awk ’$5 >= “2000” && $5 < “9000”’ carsplym fury 77 73 2500chevy nova 79 60 3000chevy nova 80 50 3500fiat 600 65 115 450honda accord 81 30 6000toyota tercel 82 180 750

  • $ awk ’$5 >= 2000 && $5 < 9000’ carsplym fury 77 73 2500chevy nova 79 60 3000chevy nova 80 50 3500honda accord 81 30 6000


3 5 range operator l.jpg
3.5. ‘,’ Range Operator

  • Format:

    pattern1 , pattern2

  • Select a range of lines.

    • the first line of the range matches pattern1

    • the last line of the range matches pattern2

  • May return several groups of lines

continued


Slide28 l.jpg

  • $ awk ’/volvo/ , /fiat/’ carsvolvo gl 78 102 9850ford ltd 83 15 10500chevy nova 80 50 3500fiat 600 65 115 450

  • $ awk ’NR == 2 , NR ==4’ carschevy nova 79 60 3000ford mustang 65 45 10000volvo gl 78 102 9850

continued


Slide29 l.jpg

  • $ awk ’/chevy/ , /ford/’ carschevy nova 79 60 3000ford mustang 65 45 10000chevy nova 80 50 3500fiat 600 65 115 450honda accord 81 30 6000ford thundbd 84 10 17000chevy impala 65 85 1550ford bronco 83 25 9500

threegroups


3 6 many built in functions l.jpg
3.6. Many Built-in Functions

  • length(str) length of string strlength length of current line

  • split(strings, array, delimitor) split stringinto parts based on the delimitor, and place in array

    • split(“a bcd ef g1”, arr, “ “)

continued



3 7 begin and end l.jpg
3.7. BEGINand END

  • BEGIN {action} executed before first line is processed

  • END {action} executed after last line is processed

  • $ awk ’END {print NR, “cars for sale.”}’ cars12 cars for sale


3 8 first awk program file l.jpg
3.8. First awk Program File

  • $ cat pr_header## pr_header#BEGIN {print “Make Model Year Miles Price”print “---------------------------------” } {print}

continued


Slide34 l.jpg

  • $ awk -f pr_header carsMake Model Year Miles Price---------------------------------plym fury 77 73 2500chevy nova 79 60 3000 : :chevy impala 65 85 1550ford bronco 83 25 9500


Redirect out l.jpg
redirect_out

  • $ cat redirect_out/chevy/ {print > “chev.txt”}/ford/ {print > “ford.txt”}END {print “done.”}

  • $ awk -f redirect_out carsdone.$ cat chev.txtchevy nova 79 60 3000chevy nova 80 50 3500chevy impala 65 85 1550


3 9 action language l.jpg
3.9. Action Language

  • Very C like:

    • var = expr

    • if (cond) stat1 else stat2

    • while (cond) stat

    • for (expr1; cond; expr2) stat

    • printf “format” expr1, expr2, ...

    • { stat1 ; stat2; ... ; statN }

  • User-defined variables do not need to be declared

continued


Slide37 l.jpg


Price range l.jpg
price_range typed over several lines.

  • $ cat price_range{if ($5 <= 5000) $5 = “inexpensive”else if ($5 > 5000 && $5 < 10000) \ $5 = “please ask”else if ($5 >= 10000) $5 = “expensive”printf “%-10s %-8s 19%2d %5d %-12s\n”, \ $1, $2, $3, $4, $5}

continued


Slide39 l.jpg

  • $ awk -f price_range cars typed over several lines. plym fury 1977 73 inexpensivechevy nova 1979 60 inexpensive : :ford bronco 1983 25 please ask


Summary l.jpg
summary typed over several lines.

  • $ cat summaryBEGIN { yearsum = 0 ; costsum = 0 newcostsum = 0 ; newcnt = 0 } { yearsum += $3 ; costsum += $5 }$3 > 80 { newcostsum += $5 ; newcnt++ }END { printf “Avg. car age: %3.1f yrs\n”, \ 90 - (yearsum/NR) printf “Avg. car cost: $%7.2f\n”, \ costsum/NR printf “Avg. newer car cost: $7.2f\n”, \ newcostsum/newcnt }

continued


Slide41 l.jpg

  • $ awk -f summary cars typed over several lines. Avg. car age: 13.2 yrsAvg. car cost: $6216.67Avg. newer car cost: $8750.00


3 10 associative arrays l.jpg
3.10. Associative Arrays typed over several lines.

  • Arrays that use strings as indexes:

    • array[string] = value

  • Special for-loop for awkarrays:

    • for (elem in array) action

continued


Manuf l.jpg
manuf typed over several lines.

  • $ cat manuf {manuf[$1]++}END { for (name in manuf) \ print name, manuf[name] }

continued


Slide44 l.jpg

  • $ awk -f manuf cars typed over several lines. honda 1 fiat 1 volvo 1 ford 4 plym 1chevy 3toyota 1


Sorted output l.jpg
Sorted Output typed over several lines.

  • Sort by first column (i.e. by name):

    $ awk -f manuf cars | sort

  • Sort by second column (i.e. by number):

    $ awk -f manuf cars | sort +1


4 sed l.jpg
4. sed typed over several lines.

  • Format:

    sed ’list of ed commands’ file

  • Read lines one at a time from the input file

    • apply ed commands in order to each line

    • write edited line to stdout

  • ed is an old UNIX editor

    • vi without full-screen mode

    • did you think vi was tough :)


4 1 search and replace l.jpg
4.1. Search and Replace typed over several lines.

  • The ‘s’ command searches for a pattern (a regular expression), and replaces it with the new string:

    ’s/pattern/new-string/g’

    • ‘g’ means global (everywhere on line)


Examples48 l.jpg
Examples typed over several lines.

  • $ sed ’s/UNIX/UNIX(TM)/g’ file > new-file

  • $ sed ’s/^/ /’ file > new-file

    • put a tab at the start of every line (no g needed)

  • $ sed ’s/[ ][ ]*/\/g’ file > new-file

    • replace every sequence of blanks or tabs with a newline

    • this splits the input into 1 word/line

continued


Slide49 l.jpg

  • $who typed over several lines. ad tty1 Sep 29 07:14ron tty3 Sep 29 10:31td tty4 Sep 29 08:36$ who | sed ’s/ .* / /’ad 07:14ron 10:31td 08:36$

replace a blank and everything that follows it (as much as possible, including more blanks) up to the last blank


More information l.jpg
More Information typed over several lines.

  • sedcan use most ed commands, not just s

  • See the entry on sed in Sobell, p.680-691


5 find l.jpg
5. find typed over several lines.

  • Format:

    find starting-directory matching-conditions-and-actions

  • findsearches all the directories below the starting directory.

    • it carries out the specified actions on the files that match the specified conditions


Basic example l.jpg
Basic Example typed over several lines.

  • Assume we are in my home directory, and want to find the cars file (used in the awk examples):

    $ find . -name cars -print./teach/adv-unix/filters/cars$

starting

point

-name

condition

-print

action


5 1 some matching conditions l.jpg
5.1. Some Matching Conditions typed over several lines.

  • -name nm the filename is nm

  • -type tyty is a file type: f = file,d = directory, etc.

  • -user usr the file’s owner is usr

  • -group grp the file’s group owner is grp

continued


Slide54 l.jpg

  • -atime n typed over several lines. file was last accessed exactly n days ago

  • -mtime n file was last modified exactly n days ago

  • -size n file is exactly n 512-byte blocks long

  • Can use + or - to mean more or less.


5 2 example conditions l.jpg
5.2. Example Conditions typed over several lines.

  • -mtime +7 last modified more than 7 days ago

  • -size +100 larger than 50K

  • “And”ing conditions:

    -atime +60 -mtime +120

    • files last accessed more than 2 months ago and last modified more than 4 months ago

continued


Slide56 l.jpg

  • “Or”ing Conditions: typed over several lines.

    \( -mtime +7 -o -atime +30 \)

    • files last modified more than 7 days ago or last accedded more than 30 days ago

  • “Not”

    -name \*.dat \! -name gold.dat

    • all “.dat” files except gold.dat


5 3 some actions l.jpg
5.3. Some Actions typed over several lines.

  • -printdisplay pathname of matching file

  • -exec cmd execute cmd on file

  • -ok cmd prompt before executing cmd on file

  • Commands must end with \; and use {} to mean the matching file, e.g.:

    -ok rm {} \;


5 4 examples l.jpg
5.4. Examples typed over several lines.

  • $ find . -name \*.c -print

    • Starting from the current directory, display the pathnames of all the files ending in “.c”

  • $ find . \( -name core -o -name junk \) -print -ok rm {} \;

    • Print the pathnames of all the core and junk files in the current directory and below, and prompt to remove them.

continued


Slide59 l.jpg

  • $ find /usr -size +100 -mtime +30 typed over several lines. -exec ls -l {} \;

    • Display a long list of all the files under /usr larger than about 500K that have not been modified in a month.


5 5 problems with permissions l.jpg
5.5. Problems with Permissions typed over several lines.

  • A find over the entire filesystem will print many error messages when access is denied to other user’s directories.

  • These error messages (sent to stderr) can be redirected to /dev/null (a UNIX “black hole”).


Example61 l.jpg
Example typed over several lines.

  • Search for a file/directory called zip anywhere below the root directory:

    $ find / -name zip -printfind: /exports/tmp/code/4210341: Permission deniedfind: /exports/tmp/code/4210389: Permission deniedfind: /exports/home/suthon/private: Permission deniedfind: /exports/home/cj/mail: Permission denied : :

continued


Slide62 l.jpg


ad