Advanced UNIX - PowerPoint PPT Presentation

Advanced unix l.jpg
Download
1 / 62

  • 192 Views
  • Updated On :
  • Presentation posted in:

Advanced UNIX. 240-491 Special Topics in Comp. Eng. 1 Semester 2, 2000-2001. Objectives to discuss five useful filters: tr , grep , awk , sed , and find. 4. Filters (Part II, Sobell). 1. tr. format: tr [options] string1 [string2]

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

Advanced UNIX

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Advanced unix l.jpg

Advanced UNIX

240-491 Special Topics in Comp. Eng. 1Semester 2, 2000-2001

  • Objectives

    • to discuss five useful filters:tr, grep, awk, sed, and find

4. Filters(Part II, Sobell)


Slide2 l.jpg

1. tr

  • format:

    tr [options] string1 [string2]

  • trreads its standard input and translates each character in string1to the corresponding character in string2


Examples l.jpg

Examples

$ echo 12abc3def4 | tr ’abcdef’ ’xyzabc’12xyz3abc4

$ echo 12abc3de4 | tr ’[a-c][d-f]’ ’[x-z][a-c]’12xyz3abc4

$ cat foo.txt | tr ’[A-Z]’ ’[a-z]’


Slide4 l.jpg

$ tr ’\015’ ’ ’ < file1 > file2

  • \015 is carriage return

    $ cat mail.txt | tr -s ’ป’ ’ ’ > new-mail.txt

  • ป represents tab; could write \011

  • -s means remove duplicates of string2in output

    $ echo Can you read this? | tr -d ’aeiou’Cn y rd ths?


Rot13 text l.jpg

“rot13” Text

Popular in 1970-1980’s.

$ echo Gur chapuyvar bs gur wbxr vf ... |tr ’[N-Z][A-M][n-z][a-m]’ ’[A-M][N-Z][a-m][n-z]’The punchline of the joke is ...


2 grep l.jpg

2. grep

  • Format:

    grep [options] pattern [file-list]

  • Search one or more files, line by line, for a pattern (a regular expression). Actions taken depend on options.


Variants of grep l.jpg

Variants of grep

  • grepUses basic RE pattern

  • fgrepFast grep. Pattern can only bean ordinary string.

  • egrepExtended grep. Pattern can usefull REs.


Grep options l.jpg

grep options

  • -cprint a count of matching lines

  • -iignore case in pattern during search

  • -llist filenames with match

  • -nprecede each matching line by a line number

  • -vprint lines that do not match pattern


Examples9 l.jpg

Examples

File testaFile testbFile testcaaabbaaaaaAAAAAbbbccbbbbbBBBBBff-ffcccccCCCCCcccdddddddDDDDDdddaa

continued


Slide10 l.jpg

  • $ grep bb testaaaabbbbbcc

  • $ grep -v bb testaff-ffcccdddddaa

  • $ grep -n bb testa1: aaabb2: bbbcc

continued


Slide11 l.jpg

  • $ grep bb *testa: aaabbtesta: bbbcctestb: bbbbb

  • $ grep -i bb *$ grep -i BB * testa: aaabbtesta: aaabb testa: bbbcctesta: bbbcc testb: bbbbbtestb: bbbbb testc: BBBBBtestc: BBBBB


Fancier patterns l.jpg

Fancier Patterns

  • $ grep ’fun..ion’ file

  • $ grep -n ’^#define’ file

  • $ grep ’^#de[a-z]*’ file

  • $ egrep ’while|if’ *.c

  • $ egrep ’[0-9]+’ *.c


3 awk l.jpg

3. awk

  • format:

    awk program file-list

    awk -f program-file file-list

  • awkis a pattern scanning and action processing language

  • The action language is very like C.


Overview l.jpg

Overview

3.1. Patterns & Actions

3.2. awkProcessing Cycle

3.3. How awkSees a Line

3.4. Pattern Expressions

3.5. ‘,’ Range Operator

continued


Slide15 l.jpg

3.6. Many Built-in Functions

3.7. BEGINand END

3.8. First awkProgram File: pre_header

3.9. Action Language

3.10. Associative Arrays


3 1 patterns actions l.jpg

3.1. Patterns & Actions

  • An awkprogram consists of:

    pattern {action}pattern {action}:


3 2 awk processing cycle l.jpg

3.2. awk Processing Cycle

1. Read next input line.

2. Apply all awkpatterns sequentially.

3. If a pattern matches, do its action.

4. Go to step (1).


Example l.jpg

Example

  • $ cat carsplymfury77732500chevynova79603000fordmustang654510000volvogl781029850fordltd831510500chevynova80503500fiat60065115450hondaaccord81306000fordthundbd841017000toyotatercel82180750chevyimpala65851550fordbronco83259500

continued


Slide19 l.jpg

  • $ awk ’/chevy/ {print}’ carschevynova79603000chevynova80503500chevyimpala65851550

  • $ awk ’/chevy/’ carschevynova79603000chevynova80503500chevyimpala65851550

  • $ awk ’/^h/’ carshondaaccord81306000


3 3 how awk sees a line l.jpg

3.3. How awkSees a Line

  • awkviews each line as a record consisting of fields separated by spaces.

  • Each field is referred to by a variable called $<number>:

    • $1, $2, $3, etc.

    • $0 refers to the whole line (record)

  • The current line number is stored in NR

continued


Slide21 l.jpg

  • $ awk ’{print $3, $1}’ cars77 plym79 chevy65 ford :83 ford

  • $ awk ’/chevy/ {print $3, $1}’ cars79 chevy80 chevy65 chevy


3 4 pattern expressions l.jpg

3.4. Pattern Expressions

  • Format:

    variable OP pattern

  • OPforms:

    • matching:~!~

    • ariithmetic:< <= == != >= >

    • boolean:&&||!

continued


Slide23 l.jpg

  • $ awk ’$1 ~ /h/’ carschevynova79603000chevynova80503500hondaaccord81306000chevyimpala65851550

  • $ awk ’$1 ~ /^h/’ carshondaaccord81306000

continued


Slide24 l.jpg

  • $ awk ’$2 ~ /^[tm]/ {print $3, $2, “$” $5}’ cars65 mustang $1000084 thundbd $1700082 tercel $750

  • $ awk ’$3 ~ /5$/ {print $3, $1, “$” $5}’ cars65 ford $1000065 fiat $45065 chevy $1550

continued


Slide25 l.jpg

  • $ awk ’$3 == 65’ carsfordmustang654510000fiat60065115450chevyimpala65851550

  • $ awk ’$5 <= 3000’ carsplymfury77732500chevynova79603000fiat60065115450toyotatercel82180750chevyimpala65851550

continued


Slide26 l.jpg

  • $ awk ’$5 >= “2000” && $5 < “9000”’ carsplymfury77732500chevynova79603000chevynova80503500fiat60065115450hondaaccord81306000toyotatercel82180750

  • $ awk ’$5 >= 2000 && $5 < 9000’ carsplymfury77732500chevynova79603000chevynova80503500hondaaccord81306000


3 5 range operator l.jpg

3.5. ‘,’ Range Operator

  • Format:

    pattern1 , pattern2

  • Select a range of lines.

    • the first line of the range matches pattern1

    • the last line of the range matches pattern2

  • May return several groups of lines

continued


Slide28 l.jpg

  • $ awk ’/volvo/ , /fiat/’ carsvolvogl781029850fordltd831510500chevynova80503500fiat60065115450

  • $ awk ’NR == 2 , NR ==4’ carschevynova79603000fordmustang654510000volvogl781029850

continued


Slide29 l.jpg

  • $ awk ’/chevy/ , /ford/’ carschevynova79603000fordmustang654510000chevynova80503500fiat60065115450hondaaccord81306000fordthundbd841017000chevyimpala65851550fordbronco83259500

threegroups


3 6 many built in functions l.jpg

3.6. Many Built-in Functions

  • length(str)length of string strlengthlength of current line

  • split(strings, array, delimitor)split stringinto parts based on the delimitor, and place in array

    • split(“a bcd ef g1”, arr, “ “)

continued


Slide31 l.jpg

  • $ awk ’length > 23 {print NR}’ cars3910


3 7 begin and end l.jpg

3.7. BEGINand END

  • BEGIN {action}executed before first lineis processed

  • END {action}executed after last lineis processed

  • $ awk ’END {print NR, “cars for sale.”}’ cars12 cars for sale


3 8 first awk program file l.jpg

3.8. First awk Program File

  • $ cat pr_header## pr_header#BEGIN {print “Make Model Year Miles Price”print “---------------------------------” } {print}

continued


Slide34 l.jpg

  • $ awk -f pr_header carsMake Model Year Miles Price---------------------------------plymfury 77 732500chevynova 79 603000::chevyimpala 65 851550fordbronco 83 259500


Redirect out l.jpg

redirect_out

  • $ cat redirect_out/chevy/ {print > “chev.txt”}/ford/ {print > “ford.txt”}END {print “done.”}

  • $ awk -f redirect_out carsdone.$ cat chev.txtchevynova79603000chevynova80503500chevyimpala65851550


3 9 action language l.jpg

3.9. Action Language

  • Very C like:

    • var = expr

    • if (cond) stat1 else stat2

    • while (cond) stat

    • for (expr1; cond; expr2) stat

    • printf “format” expr1, expr2, ...

    • { stat1 ; stat2; ... ; statN }

  • User-defined variables do not need to be declared

continued


Slide37 l.jpg

  • Long statements, conditions, expressions may need to be typed over several lines.

  • Use ‘\’ to hide newline:

    if ($3 > 2000 && \ $3 < 3000) print $3


Price range l.jpg

price_range

  • $ cat price_range{if ($5 <= 5000) $5 = “inexpensive”else if ($5 > 5000 && $5 < 10000) \$5 = “please ask”else if ($5 >= 10000) $5 = “expensive”printf “%-10s %-8s 19%2d %5d %-12s\n”, \$1, $2, $3, $4, $5}

continued


Slide39 l.jpg

  • $ awk -f price_range carsplymfury197773inexpensivechevynova197960inexpensive::fordbronco198325please ask


Summary l.jpg

summary

  • $ cat summaryBEGIN { yearsum = 0 ; costsum = 0 newcostsum = 0 ; newcnt = 0 } { yearsum += $3 ; costsum += $5 }$3 > 80 { newcostsum += $5 ; newcnt++ }END { printf “Avg. car age: %3.1f yrs\n”, \90 - (yearsum/NR) printf “Avg. car cost: $%7.2f\n”, \costsum/NR printf “Avg. newer car cost: $7.2f\n”, \newcostsum/newcnt }

continued


Slide41 l.jpg

  • $ awk -f summary carsAvg. car age: 13.2 yrsAvg. car cost: $6216.67Avg. newer car cost: $8750.00


3 10 associative arrays l.jpg

3.10. Associative Arrays

  • Arrays that use strings as indexes:

    • array[string] = value

  • Special for-loop for awkarrays:

    • for (elem in array) action

continued


Manuf l.jpg

manuf

  • $ cat manuf {manuf[$1]++}END { for (name in manuf) \print name, manuf[name] }

continued


Slide44 l.jpg

  • $ awk -f manuf carshonda 1 fiat 1 volvo 1 ford 4 plym 1chevy 3toyota 1


Sorted output l.jpg

Sorted Output

  • Sort by first column (i.e. by name):

    $ awk -f manuf cars | sort

  • Sort by second column (i.e. by number):

    $ awk -f manuf cars | sort +1


4 sed l.jpg

4. sed

  • Format:

    sed ’list of ed commands’ file

  • Read lines one at a time from the input file

    • apply ed commands in order to each line

    • write edited line to stdout

  • ed is an old UNIX editor

    • vi without full-screen mode

    • did you think vi was tough :)


4 1 search and replace l.jpg

4.1. Search and Replace

  • The ‘s’ command searches for a pattern (a regular expression), and replaces it with the new string:

    ’s/pattern/new-string/g’

    • ‘g’ means global (everywhere on line)


Examples48 l.jpg

Examples

  • $ sed ’s/UNIX/UNIX(TM)/g’ file > new-file

  • $ sed ’s/^//’ file > new-file

    • put a tab at the start of every line (no g needed)

  • $ sed ’s/[ ][ ]*/\/g’ file > new-file

    • replace every sequence of blanks or tabs with a newline

    • this splits the input into 1 word/line

continued


Slide49 l.jpg

  • $whoad tty1 Sep 29 07:14ron tty3 Sep 29 10:31td tty4 Sep 29 08:36$ who | sed ’s/ .* / /’ad 07:14ron 10:31td 08:36$

replace a blank and everything that follows it (as much as possible, including more blanks) up to the last blank


More information l.jpg

More Information

  • sedcan use most ed commands, not just s

  • See the entry on sed in Sobell, p.680-691


5 find l.jpg

5. find

  • Format:

    find starting-directory matching-conditions-and-actions

  • findsearches all the directories below the starting directory.

    • it carries out the specified actions on the files that match the specified conditions


Basic example l.jpg

Basic Example

  • Assume we are in my home directory, and want to find the cars file (used in the awk examples):

    $ find . -name cars -print./teach/adv-unix/filters/cars$

starting

point

-name

condition

-print

action


5 1 some matching conditions l.jpg

5.1. Some Matching Conditions

  • -name nmthe filename is nm

  • -type tyty is a file type: f = file,d = directory, etc.

  • -user usrthe file’s owner is usr

  • -group grpthe file’s group owner is grp

continued


Slide54 l.jpg

  • -atime nfile was last accessedexactly n days ago

  • -mtime nfile was last modifiedexactly n days ago

  • -size nfile is exactly n 512-byte blocks long

  • Can use + or - to mean more or less.


5 2 example conditions l.jpg

5.2. Example Conditions

  • -mtime +7last modified more than7 days ago

  • -size +100larger than 50K

  • “And”ing conditions:

    -atime +60 -mtime +120

    • files last accessed more than 2 months ago and last modified more than 4 months ago

continued


Slide56 l.jpg

  • “Or”ing Conditions:

    \( -mtime +7 -o -atime +30 \)

    • files last modified more than 7 days ago or last accedded more than 30 days ago

  • “Not”

    -name \*.dat \! -name gold.dat

    • all “.dat” files except gold.dat


5 3 some actions l.jpg

5.3. Some Actions

  • -printdisplay pathname of matching file

  • -exec cmdexecute cmd on file

  • -ok cmdprompt before executing cmd on file

  • Commands must end with \; and use {} to mean the matching file, e.g.:

    -ok rm {} \;


5 4 examples l.jpg

5.4. Examples

  • $ find . -name \*.c -print

    • Starting from the current directory, display the pathnames of all the files ending in “.c”

  • $ find . \( -name core -o -name junk \) -print -ok rm {} \;

    • Print the pathnames of all the core and junk files in the current directory and below, and prompt to remove them.

continued


Slide59 l.jpg

  • $ find /usr -size +100 -mtime +30-exec ls -l {} \;

    • Display a long list of all the files under /usr larger than about 500K that have not been modified in a month.


5 5 problems with permissions l.jpg

5.5. Problems with Permissions

  • A find over the entire filesystem will print many error messages when access is denied to other user’s directories.

  • These error messages (sent to stderr) can be redirected to /dev/null (a UNIX “black hole”).


Example61 l.jpg

Example

  • Search for a file/directory called zip anywhere below the root directory:

    $ find / -name zip -printfind: /exports/tmp/code/4210341: Permission deniedfind: /exports/tmp/code/4210389: Permission deniedfind: /exports/home/suthon/private: Permission deniedfind: /exports/home/cj/mail: Permission denied::

continued


Slide62 l.jpg

  • Redirect standard errors to the black hole using 2>

    $ find / -name zip -print 2> /dev/null/exports/home/s4110068/project/zip/exports/home/s4110316/project/zip/exports/home/s4110316/zip/exports/home/s4110316/zip/zip$


  • Login