Unix Utilities (sort/uniq)

1. CS465 – Unix Unix Utilities (sort/uniq)

2. The sort command • Sorts lines • Default behavior: Do a case-sensitive, ascii-alphabetic line sort, starting at the beginning of each line • Can use sort options to sort on different fields and in different ways.

3. sort options • Format: $ sort [options][files] • Options: +n skip n fields before sorting -- older method (i.e. sort from field n+1 to end of line) -kx sort from field x to end of line (new method) +n -m sort from field n+1 to field m -- older method -kx,y sort from field x to field y (new method) -kx,x -ky,y sort on field x, then on field y

4. sort options • Format: $ sort [options][files] • Options: • -b ignore leading whitespace • -d dictionary order (blanks and alphabetic chars only) • -f ignore case (upper/lower considered same) • -n sort in numeric order • -o file output to named file • -r sort in reverse (descending) order • -tc separate fields using c (default is whitespace)

5. sort examples $ sort +1 list1 # sort list1 starting from field 2 to the end of the line $ sort –k2,3 list2 # sort list2 based upon the second and third fields together $ sort –k3,3 –k5,5 list3 # sort list3 on the third field, then the fifth field

6. sort examples $ ls -l | sort -k9 -r # sort long listing of current directory in reverse filename (field 9) order $ sort –k3 -o slist2 list2 # sort list2, starting with the third field, and output the results to slist2 $ sort -k2 -b list3 > slist3 # sort list3, starting with field 2, and ignoring blanks, and place the output in slist3

7. $ sort -k2 sortfile.txt bruce 1 david 10 edward 12 albert 2 chris 20 $ $ sort -n -k2 sortfile.txt bruce 1 albert 2 david 10 edward 12 chris 20 $ sort examples $ sort sortfile.txt albert 2 bruce 1 chris 20 david 10 edward 12 $

8. Review sort examples on handout Handout

9. uniq command • Removes duplicate lines from a file: $ cat ab.txt aaa aaa bbb bbb $ uniq ab.txt aaa bbb • Duplicate lines in file must be adjacent, so uniq is often used with sort: $ sort ab.txt | uniq > ab-uniq.txt

10. Using sort with uniq • $ cat fruit • apple • banana • banana • apple • banana • $ $ uniq fruit apple banana apple $ • $ sort fruit | uniq • apple • banana • $

11. uniq options -c print each line once, along with a count of occurences of each -d print duplicate lines once (and don’t print any unique lines) -fN do not compare the first N fields (skip fields) -u print ONLY unique lines (discard ALL duplicates)

12. uniq examples $ uniq -d names Pam Sue $ $ cat names Bill Pam Pam Ron Sue Sue $ $ uniq -u names Bill Ron $ $ uniq –c names 1 Bill 2 Pam 1 Ron 2 Sue $ $ uniq names Bill Pam Ron Sue $

13. uniq examples $ sort –k2 names | uniq –f1 Bill Jones Dave Smith $ $ cat names Bill Jones Pam Smith Sue Smith Paul Jones Dave Smith Ron Smith $ $ sort –k2 names | uniq –f1 -c 2 Bill Jones 4 Dave Smith $ $ sort –k2 names Bill Jones Paul Jones Pam Smith Sue Smith Dave Smith Ron Smith $