Unix filters
Download
1 / 36

Unix Filters - PowerPoint PPT Presentation


  • 569 Views
  • Updated On :

Unix Filters. Commands with a dual purpose. Objectives. Reinforce Directory and File Commands Review Redirection Introduce Filters: head tail more less sort uniq wc tee cut Introduce Pipes. Filters. Filter commands – Unix commands that serve dual purposes: standalone

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Unix Filters' - jayme


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Unix filters l.jpg

Unix Filters

Commands with a dual purpose


Objectives l.jpg
Objectives

  • Reinforce Directory and File Commands

  • Review Redirection

  • Introduce Filters:

    • head

    • tail

    • more

    • less

    • sort

    • uniq

    • wc

    • tee

    • cut

  • Introduce Pipes


Filters l.jpg
Filters

  • Filter commands – Unix commands that serve dual purposes:

    • standalone

    • used with other commands and pipes

  • Why are filters important?

    • The output from a command may be overwhelming, e.g. recall the output from the:

      # command to list all of the files in the /home directory

      ls –l /home

      filters may be a way of reducing output to only the pertinent data

    • We may wish to organize the data presented.

    • We may wish to reformat the data presented.


Definition of a filter l.jpg
Definition of a Filter

  • A filter is a program that:

    • receives input from the standard output (stdout) of a Unix command, typically as text;

    • transforms that output, according to a parameterized set of user guidelines; and, finally,

    • writes that output to the standard output (stdout) or possibly to another filter (through redirection).


Filter commands l.jpg
Filter Commands

  • Filter commands include:

    • head

    • tail

    • more

    • less

    • sort

    • uniq

    • wc

    • tee

    • cut

    • grep*

    • awk*

      * to be covered later


Filters can be used with or without pipes l.jpg
Filters can be used with or without Pipes

  • Examples:

    • with: ls | wc –l

      counts the number of files in the directory

    • without wc –l abc.txt

      Counts the number of lines in the file abc.txt


Pipes and filters l.jpg
Pipes and Filters

pipe

Unix Command

filter

output

  • Example: ls | wc –l

  • This particular command:

    • gets a list of the files in the directory

    • sends this list through the pipe

    • the filter counts the lines of the data received

  • The result is equivalent to: The number of files in the directory


What do pipes do l.jpg
What do pipes do?

Using the example:

ls | wc –l

to do this without a pipe would require something like this:

ls >tempfile # list contents of dir, redirect to tempfile

# count the lines in tempfile (i.e. tell me the number of files)

wc –l tempfile

rmtempfile # delete the tempfile


Using cat with pipes l.jpg
Using cat with pipes

  • Avoid using cat with pipes unless the cat command will transform the data through the use of options!

  • Good:

    cat –n abc | more # the cat adds line numbers to the listing of file abc

  • Bad:

    catabc | more # the cat does not alter the listing at all. In this case simply use more as a command as in:

    moreabc


Redirection examples l.jpg
Redirection Examples

cat <abc >def3

cat >abc

cat >>abc

cat abc1

cat <abc2

1.This assumes the file abc does exist.

2.This example is merely a demonstration of redirecting from a file. It is the equivalent of: cat abc

3. This is equivalent to: cp abc def


Redirection l.jpg
Redirection

  • Redirection deals with changing the source or destination of the standard input and output.

  • Standard (I/0):

    • input : the keyboard

      • redirect is the less than (<) symbol

    • output: the computer screen

      • redirect is:

        • greater than symbol (>) to create a new output file (if file exists writes over it)

        • two greater than symbols (>>) append to existing file or if not an existing file create a new one


Pipes versus redirection l.jpg
Pipes versus Redirection

  • Pipes (|) send output from a Unix command to another Unix command (filter)

  • Redirection:

    • output (>, >>) send output of Unix command to a device (usually a file) other than the standard output (the screen)

    • input (<) provide input to a Unix command from a device (usually a file) other than the standard input (the keyboard)


Pipes versus redirection13 l.jpg
Pipes versus Redirection

std out

std in

no redirection

Unix command

<

>

redirection (<,>)

device to/from

a Unix command

disk

disk

output

input

Unix command

Unix command

pipe (|)

Unix Command to

Unix Command


Slide14 l.jpg
head

  • head outputs the first part of files

  • Format: head [options] filename

  • Selected options:

    • -cn : print the first n characters of the file

    • -n : print the first n lines the default is 10


Head examples l.jpg
head Examples

  • Command:

    head –c5 abc # displays first 5 characters (includes whitespace)

    headabc # displays first 10 lines of abc

  • Pipe

    ls | head -15 #displays first 15 lines of file listing


Slide16 l.jpg
tail

  • tail outputs the last part of files

  • Format: tail [options] filename

  • Selected options:

    • -cn : print the last n characters of the file

    • -n : print the last n lines the default is 10


Tail examples l.jpg
tail Examples

  • Command:

    tail –c5 abc # displays last 5 characters (includes whitespace)

    tail abc # displays last 10 lines of abc

  • Pipe

    ls | tail -15 #displays last 15 lines of file listing

    headabc | tail -5 # prints lines 6 through 10 of file abc


Slide18 l.jpg
more

  • more displays files on a page-by-page basis

  • Format: more [options] filename

  • Options (selected):

    • -n : number of lines to show at a time

    • -d :prompt to continue

    • +/pattern : begin at pattern

  • Commands to use while in more:

    • space : advance a page at a time

    • return : advance a line at a time

    • q : quit more


More examples l.jpg
more Examples

  • Command:

    moreabc # lists file abc one page/line at a time

  • Pipe:

    ls –l | more # lists directory contents one page/line at a time


Slide20 l.jpg
less

  • less displays files on a page-by-page basis

  • Format: less [options] filename

  • Options (selected):

    • -E : quit at EOF (does not require the q)

    • -N : show line numbers

    • -ppattern : begin at pattern

  • Commands to use while in less:

    • h : help (try it, you’ll like it)

    • space : advance a page at a time

    • return : advance a line at a time

    • q : quit less


Less examples l.jpg
less Examples

  • Command:

    less abc # lists file abc one page/line at a time

  • Pipe:

    ls –l | less # lists directory contents one page/line at a time


Slide22 l.jpg
sort

  • sort sorts lines of text [files]

  • usually (but not always) used with redirection

  • Format: sort [options]

  • Options (selected):

    • -b : ignore leading blanks

    • -f : ignore case

    • -r : sort descending rather than ascending

    • -M : sort using month order (i.e. Jan comes before Feb)

    • -n : sort numerical rather than by string

    • -u : unique, eliminate duplicate lines

    • -tchar : use char as field separator (instead of

      whitespace)


Sort anomaly l.jpg
sort Anomaly

  • Caution: If multiple, adjacent delimiters are found, only the first delimiter is treated as a delimiter! This includes the default delimiter (whitespace)

  • For example:


Slide24 l.jpg
sort

  • When using fields to sort, you may also use characters within a field.

  • Examples:

    • sort –t: k+5,5 /etc/passwd (uses colon as field delimiter and does the 5th field only)

    • sort –k 1.2 shuffled (sorts on 2nd character (and beyond) in first field)

    • sort –nu myfile (sorts starting with first field, values in numerical order, removing duplicates)


Sort fields l.jpg
Sort Fields

  • Examples:

  • sort –k 2 filename # sorts on last name (Smith, Ye, etc…)

  • sort –t, -k 2 filename # sorts on first name, month, and day (field2 of comma delimited fields)

  • sort –t: -k 1.8 filename # sorts on 2nd letter of last name (this assumes that the 2nd letter of the last name is in the 8th column of the first field. There is no colon (:) delimiter in the data making the entire line the 1st field


Slide26 l.jpg
sort

  • When sorting on multiple fields, you may also use the –k pos1[,pos2] options.

  • If you wish to indicate the format of the key to be sorted, place the format value adjacent to the field number.

  • Examples:


Slide27 l.jpg
uniq

  • uniq removes duplicate adjacent lines from a file

  • To ensure that all of the data is unique you should sort the file first (recall that sort also can produce unique values with the –u option)

  • Format: uniq [options] filename

  • Options (selected):

    • -c : count the instances

    • -d : print only the duplicates (once)

    • -u : print only the unique lines

    • -n : ignore first n fields


Uniq examples l.jpg
uniq Examples

  • Command:

    uniq –u abc # only prints out the unique lines in file abc assuming duplicates are next to each other (without the –u, uniq will print out all of the lines, but only once, the 2nd or more duplicate lines will not print out)

    uniq –c abc # only prints out the count of unique lines assuming duplicates are next to each other

  • Pipe:

    sortabc | uniq –c # put abc in order and tell me how many lines are unique


Slide29 l.jpg
wc

  • wcprints the number of newlines, words, and characters (bytes) in files

  • Format: wc [options] filename

  • Options:

    • -c : print the number of characters in the file

    • -w : print the number of words in a file

    • -l : print the number of lines in a file

  • If no options given, wc prints newlines, words, and bytes in files in that order


Slide30 l.jpg
cut

  • cut removes columns or fields from a line

  • Format: cut [options] filename

  • Options:

    *lists may be specified using integers separated by commas, ranges are separated by hyphens (-) Examples: 1,5-7 means units 1,5,6,7 and 1,5- means units 1,5 and beyond to end of line


Slide31 l.jpg
cut

  • Examples:

    cut -d: -f5 /etc/passwd (print only the 5th field, delimited by colon(:), of the /etc/passwd file)

    cut –d” “ –f2 shuffled (print only the 2nd field, delimited by a space, from file shuffled)

    cut –c4-8 shuffled (print only the columns 4 through 8 of each line in shuffled)


Slide32 l.jpg
tees

  • A filter that permits redirection to a file and standard output (screen)

  • Requires a pipe

  • Format:

    cmd| tee outfile

    where:

    • cmd is any Unix command that produces standard output

    • outfile is the file receiving the output data


Tee examples l.jpg
tee Examples

  • Filter only:

    ls –al ~ | tee dirlist # displays a long listing of all the files in your home directory and write the same to the dirlist file.

    ls –al ~ | tee dirlist | sort –r # displays a long listing of all the files in your home directory in reverse order, but saves the listing in order to the file dirlist.


Pipes and tee l.jpg
Pipes and tee

tee

disk

Unix Command

pipe

screen



Unix filters36 l.jpg

Unix Filters

Commands with a dual purpose


ad