1 / 28

CSE4251 The Unix Programming Environment

CSE4251 The Unix Programming Environment. Lecture 10. awk. Recap. Regular expressions (lec-5) symbols & rules to describe text patterns/filters Unix commands/utilities that support regular expressions grep ( fgrep , egrep ) - search a file for a string or regular expression

Download Presentation

CSE4251 The Unix Programming Environment

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. CSE4251 The Unix Programming Environment Lecture 10 • awk

  2. Recap • Regular expressions (lec-5) • symbols & rules to describe text patterns/filters • Unix commands/utilities that support regular expressions • grep(fgrep, egrep)- search a file for a string or regular expression • sed - stream editor • awk (nawk) - pattern scanning and processing language note: there are some minor differences between the regular expressions supported by these programs

  3. awk history • The name AWK • Initials of designers: Alfred V. Alo, Peter J. Weinberger, and Brian W. Kernighan. • Appear 1977, stable release 1985 • In BSD, OS X: bawk or nawk. • GNU/Linux : gawk $ which awk /bin/awk $ ls -l /bin/*awk lrwxrwxrwx. 1 root root4 Jul 2 2013 /bin/awk -> gawk -rwxr-xr-x. 1 root root 382456 Jul 4 2012 /bin/gawk

  4. awk basics • basic function: • search files for lines that contains certain patterns, • do actions on those lines • basic command format: $ awk ‘{action}’ file.txt $ awk‘/pattern/{action}’ file.txt $ awk‘/pattern1/{action1} /pattern2/{action2}’ file.txt

  5. gamefile northwest NW Charles Main 3.0 .98 3 34 western WE Sharon Gray 5.3 .97 5 23 southwest SW Lewis Dalsass 2.7 .8 2 18 southern SO Suan Chin 5.1 .95 4 15 southeast SE Patricia Heme 4.0 .7 4 17 eastern EA TB Savage 4.4 .84 5 20 northeast NE AM Main Jr. 5.1 .94 3 13 north NO Margot Webber 4.5 .89 5 9 central CT Ann Stephens 5.7 .94 5 13

  6. awk basics • basic command examples: $ awk ‘{print}’ gamefile #print all lines (no pattern constraints) $ awk '{print}' gamefile northwest NW Charles Main 3.0 .98 3 34 western WE Sharon Gray 5.3 .97 5 23 southwest SW Lewis Dalsass 2.7 .8 2 18 southern SO Suan Chin 5.1 .95 4 15 southeast SE Patricia Heme 4.0 .7 4 17 eastern EA TB Savage 4.4 .84 5 20 northeast NE AM Main Jr. 5.1 .94 3 13 north NO Margot Webber 4.5 .89 5 9 central CT Ann Stephens 5.7 .94 5 13

  7. awk basics • basic command examples: $ awk ‘{print $1}’ gamefile #print 1st field in all lines $ awk '{print $1}' gamefile northwest western southwest southern southeast eastern northeast north central

  8. awk basics • basic command examples: $ awk ‘/north/{print $1}’ gamefile #print 1st field in lines containing north $ awk '/north/{print $1}' gamefile northwest northeast north

  9. awk basics • basic command examples: $ awk ‘/north/{print $1} /west/{print}’ gamefile $ awk '/north/{print $1} /west/{print}' gamefile northwest northwest NW Charles Main 3.0 .98 3 34 western WE Sharon Gray 5.3 .97 5 23 southwest SW Lewis Dalsass 2.7 .8 2 18 northeast north

  10. More concepts • A line is called a record • text separated by delimiter is called field • default delimiter is space • FS: input field separator (delimiter) • default is space • two ways to change default delimiter • change via –F • change via setting FS

  11. More concepts • Change delimiter to “:” via –F • Change delimiter to “:” via setting FS $ awk -F: '/north/{print $1}' gamefile northwest NW Charles Main 3.0 .98 3 34 northeast NE AM Main Jr. 5.1 .94 3 13 north NO Margot Webber 4.5 .89 5 9 $ awk '{FS=":"} /north/{print $1}' gamefile northwest northeast NE AM Main Jr. 5.1 .94 3 13 north NO Margot Webber 4.5 .89 5 9

  12. More concepts • $0, $1, ... etc • $0 : the whole line • $1 : the first field in a line • NR : Number of record • also the line number • NF : number of fields in a line

  13. More concepts • E.g., print line number, first field, and number of fields in the line; connect each output filed with “-” $ awk '{print NR "-" $1 "-" NF}' gamefile 1-northwest-8 2-western-8 3-southwest-8 4-southern-8 5-southeast-8 6-eastern-8 7-northeast-9 8-north-8 9-central-8

  14. Another example • E.g., print line number, employ first name, and number of fields in the line; connect each output filed with “---” $ cat employees.txt Tom Jones 4424 5/12/66 543354 Mary Adams 5346 11/4/63 28765 Sally Chang 1654 7/22/54 650000 Billy Black 1683 9/23/44 336500 $ awk '{print NR "---" $1 "---" NF}' employees.txt 1---Tom---5 2---Mary---5 3---Sally---5 4---Billy---5

  15. awkcomparison expression • Conditional expression • condition? exp1 : exp2 • Logical operation • &&, ||, !

  16. More examples • awk ‘$3<4000’ employees.txt • print lines where $3 is less than 4000 • awk‘/Tom/{print “Hello, “ $1}’ employees • find the line containing Tom, then print “Hello Tom“ $ awk '$3<3000' employees.txt Sally Chang 1654 7/22/54 650000 Billy Black 1683 9/23/44 336500 $ awk '/Tom/{print "Hello, " $1}' employees.txt Hello, Tom

  17. More examples • awk‘/ly/{ print $1}’ employees • print the names that contain ly • awk ‘$1 !~ /ly/{ print $1}’ employees • print the names that dose not contain ly $ awk '/ly/{print $1}' employees.txt Sally Billy $ awk '$1 !~ /ly$/{print $1}' employees.txt Tom Mary

  18. More examples • Conditional expression $ cat needmax.txt 1 2 3 5 6 3 7 2 $ awk '{max=($1>$2)? $1 : $2; print max}' needmax.txt 2 5 6 7

  19. More examples • awk'$8 > 10 && $8 < 17' gamefile • awk ‘$7==5{print $7+5}’ gamefile $ awk '$8 > 10 && $8 < 17' gamefile southern SO Suan Chin 5.1 .95 4 15 central CT Ann Stephens 5.7 .94 5 13 $ awk '$7==5{print $7+5}' gamefile 10 10 10 10

  20. Math operators • awk '/southern/{print $5 + 10.56}' gamefile • awk '/southern/{print $8 - 10}' gamefile • awk '/southern/{print $8 / 2}' gamefile • awk '/southern/{print $8 * 2}' gamefile • awk '/northeast/ {print $8 % 3}' gamefile

  21. More operators • assignment operators: =, +=, -=, *=, /=, %=, ^= • increment and decrement: ++, -- • awk '$3 == "Chris"{ $3 = "Christian"; print}' gamefile • if a line’s 3rd field is “Chris”, change it to Christian and print out the line • awk‘{$7^=2; print $7}’ gamefile • square the 7th field and print out the 7thfield

  22. More operators • awk‘{x=1; y=x++; print x, y}’ gamefile $ awk '{x=1;y=x++; print x,y}' gamefile 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1

  23. BEGIN pattern • BEGIN pattern is followed by an action block that is executed before processing any lines from the input file. • can run an awk command without file $ awk 'BEGIN{print "Hello" }' Hello

  24. BEGIN pattern • Change delimiter to “:” via setting FS $ awk '{FS=":"} /north/{print $1}' gamefile northwest northeast NE AM Main Jr. 5.1 .94 3 13 north NO Margot Webber 4.5 .89 5 9 $ awk 'BEGIN{FS=":"} /north/{print $1}' gamefile northwest NW Charles Main 3.0 .98 3 34 northeast NE AM Main Jr. 5.1 .94 3 13 north NO Margot Webber 4.5 .89 5 9

  25. END pattern • END pattern allows actions to be executed after processing all lines in the input file • can run an awk command without file $ awk 'END{ print "The number of records is " NR}' employees.txt The number of records is 4

  26. Can use redirection & pipe in actions • awk'$8 > 10 && $8 < 17 {print}' gamefile • awk '$8 > 10 && $8 < 17 {print > “tmp.out”}' gamefile $ $ awk '$8 > 10 && $8 < 17 {print}' gamefile southern SO Suan Chin 5.1 .95 4 15 central CT Ann Stephens 5.7 .94 5 13 $ awk '$8 > 10 && $8 < 17 {print > "tmp.out"}' gamefile $ cat tmp.out southern SO Suan Chin 5.1 .95 4 15 central CT Ann Stephens 5.7 .94 5 13

  27. awkin script • awk –fawk.scriptsomefile $ cat awk.script #file: awk_first /Tom/{print "Tom's birthday is " $4} /Mary/{print NR, $0} #print line number /^Sally/{print "Hi, Sally. " $1 " has salary of $" $5 "."} $ awk -f awk.script employees.txt Tom's birthday is 5/12/66 2 Mary Adams 5346 11/4/63 28765 Hi, Sally. Sally has salary of $650000.

  28. More to explore • conditional statement • loop • arrays • user-defined functions. Chapter 6 of Unix Shells by Example [Online Version] ,  4th Edition by Ellie Quigley

More Related