korn shell script writing hpc n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Korn Shell Script Writing & HPC PowerPoint Presentation
Download Presentation
Korn Shell Script Writing & HPC

Loading in 2 Seconds...

play fullscreen
1 / 174

Korn Shell Script Writing & HPC - PowerPoint PPT Presentation


  • 209 Views
  • Uploaded on

Korn Shell Script Writing & HPC. Science & Technology Support Group High Performance Computing OSC (Ohio Supercomputer Center) 1224 Kinnear Road Columbus, OH 43212. Introduction What is a shell script? Script Programming Language Why use shell scripts? Korn shell scripts References

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Korn Shell Script Writing & HPC' - orlando-finley


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
korn shell script writing hpc

Korn Shell Script Writing & HPC

Science & Technology Support Group

High Performance Computing

OSC (Ohio Supercomputer Center)

1224 Kinnear Road

Columbus, OH 43212

table of contents
Introduction

What is a shell script?

Script Programming Language

Why use shell scripts?

Korn shell scripts

References

The Evolution of a Script

Sampling of Script Features

A simple script

A more versatile Script

Creating Files within a Script

A documented Script

Script Basics

Creating a shell script

Executing a shell script

Echoing script commands

Script Variables

User-defined Variables

Storing a Value into a Variable

Assignment Operator =

Read Command

Command Substitution

Null Variables

Special Shell Variables

Positional Parameters

Referencing Variables

Parameter Substitution

Exercises

Decision-Making statements

Flow Control

Exit Status

If-Then-Else command

Conditional Tests

Logical Operators

Illustrative Examples

Case statement

Exercises

Table of Contents
table of contents1
Looping Statements

for Statement

select Statement

while Statement

Until Statement

Exercises

Working with Integers

Arithmetic Operations

Operator Precedence

Numerical Bases

String vs Integer Variables

Exercises

Script I/O

Shell Redirection Operators

"Here-is" Files

print

read

Quoting

Exercises

Communicating with Running Scripts

Sending Signals

How Scripts Receive Signals

Script Functions

Why Use Functions?

Syntax & Use

Auto-Loading

Function Definition Location

Command Precedence

HPC Designed Scripts

Timing Loops

Debugging Options

Batch Jobs

Performance Tool Preprocessing

Multiple Tools runs

Multiple Tool Reports

File Renaming

Table of Contents
introduction
Introduction
  • What is a shell script?
  • Script Programming Language
  • Why use shell scripts?
  • Korn Shell scripts
  • References
what is a shell script
What is a shell script?
  • A shell script is a collection of Unix commands entered into a file. When the file (script) is executed, the commands are executed in order.
  • Running a shell script is EXACTLY equivalent to interactively entering each command
  • For example, say a user always does the same four steps before printing out a file: making the file owner- writable, using vi to check/change the file, printing it out, and then making the file unwritable (for safety reasons)
  • An interactive session of a user following this procedure for the file comp.txt is shown on the next page
interactive editing and printing
Interactive editing and printing
  • $ chmod u+w comp.txt
  • $ vi comp.txt
  • Here is a table summarizing OSC computers as of today.
  • Supercomputer # Processors Proc Speed Ideal Calc.
  • ------------ ----------- --------- -------------
  • Itanium Cluster 124 900 MHz 223.2 GFLOPS
  • Cray SV-1 16 300 MHz 19.2 GFLOPS
  • Origin 2000 64 300 MHz 38.4 GFLOPS
  • IA-32 Intel 128 550 MHz 70.4 GFLOPS
  • Sun Fire 6800 24 900 MHz 43.2 GLOPS
  • "comp.txt" 11 lines, 535 characters
  • $ lpr -Pps1 comp.txt
  • $ chmod u-w comp.txt
making the viprint script
Making the viprint script
  • To make the previous editing and printing procedure into a script, all the user has to do is type the same commands they use interactively into a file
  • Say the file is called viprint, it looks like this:
  • $ cat viprint
  • chmod u+w comp.txt
  • vi comp.txt
  • lpr -Pps1 comp.txt
  • chmod u-w comp.txt
  • After viprint is made executable, whenever the user wants to execute these four commands they simply type viprint as if it was a “normal” Unix command
  • $ viprint
script programming language
Script Programming Language
  • The real power is writing shell script comes from using special shell commands and variables that essentially make your script act like a program
  • In this course, you will learn about the programming statements which can be used in scripts, including:
    • Initializing and using your own script variables. Both single valued and arrays
    • Performing operations with variables
    • Decision-making statements (if-then-else, etc)
    • Looping statements (for,etc)
    • Defining and using functions in scripts
    • Script I/O commands and procedures
  • Above bullet reads like a “table of contents” for a text on a “traditional” programming languages such as Fortran or C, although …
  • Script programming has less capabilities and commands
why use shell scripts
Why use shell scripts?
  • Convenience: Run a number of Unix commands by typing a single script name
  • Command log: Script can permanently contain a detailed set of commands with numerous options. (If interactive, complex command is gone)
  • Usefulness: Write a script with programming commands to give the script a number of capabilities/options
  • Take “full” use of what the shell has built-in to offer you
  • Considering a script as a program, no compiler, loader, or libraries needed to make it executable
  • Portability: If a script works in the shell on your machine it will work on the same shell on a different machine
  • Easy to debug: Small number of programming statements which are relatively simple
  • Easy to modify, expand, and customize: Just type in the new commands
  • Run-type interaction: Can send signals to running scripts
korn shell scripts
Korn shell scripts
  • Several types of Unix shells exist, each with their own script programming languages:
    • Bourne (sh), Born Again shell (bash), Korn shell
    • C shell (csh), TENEX C shell (tcsh)
  • Features you would expect from a well-written shell:
    • Convenient interactive features: command history, command-line editing, and filename completion
    • Powerful scripting capabilities including reliability and efficiency
  • This course will discuss Korn shell scripting
    • It has all the convenient interactive features
    • It is backward compatible with the Bourne shell (the first shell)
      • Most scripts are sh
      • Start-up sequence for ksh same as sh only enhanced
      • Unix utilities and administration commands depend on sh
    • It does not have the scripting bugs that the C shell has
korn shell scripts1
Korn shell scripts
  • What if I already use a different shell on my machine?
    • Could switch to Korn shell (use the chsh command if you can)
    • Keep your existing shell as your interactive shell and write your scripts as Korn shell scripts (Easily done: one line needed in the script)
references
References
  • Internet URLs
    • nacphy.physics.orst.edu/rubin/melanie/node144.html (Tutorial)
    • gonzo.tamu.edu/csh.whynot.html (Bugs in C shell scripting)
  • Usenet Newsgroups
    • comp.unix.shell
    • comp.unix.questions
  • Anonymous ftp
    • athos.rutgers.edu/~patra/vi.html (shell-101.BetaA.Z)
    • rtfm.mit.edu:/pub/usenet/news.answers/unix-faq/faq (Unix FAQ)
  • OSC Unix-Series workshops (oscinfo.osc.edu)
    • Basic Unix
    • Intermediate Unix
    • Introduction to Perl
the evolution of a script
The Evolution of a Script
  • Sampling of Scripting Features
  • A Simple Script
  • A More Versatile Script
  • Creating Files within a Script
  • A Commented Script
sampling of scripting features
Sampling of Scripting Features
  • Next few slides will act as a course overview as we take a simple script and keep extending its capabilities
  • The more advanced scripts will demonstrate some of the capabilities of Korn Shell scripting
  • Don’t worry about understanding all the details at this point. A complete explanation of the scripting commands used is given in later chapters
a simple script
A Simple Script
  • In the script called triangle, a C program is compiled and then executed. The program calculates the area of the triangle and reads in the lengths of the sides from a file and outputs the result to another file.
  • Here is an example of a user running the script:
  • $ cat triangle
  • cc -n32 -cckr -O3 -LNO -Ofast -TARG:madd=on -o area area.c
  • area < sides.in > area.dat
  • cat area.dat
  • rm area
  • $ ls -l triangle
  • -rwxr--r-- 1 dje appl 108 May 3 12:09 triangle
  • $ cat sides.in
  • 8 3 7
  • $ triangle
  • Area of Triangle is 10.392305
a simple script1
A Simple Script
  • Limitation of the simple triangle script
    • Script only works for the one area program. What if you want to run other triangle-related programs with this same script ?
a more versatile script
A More Versatile Script
  • Scripts can take arguments on the command line when they are run. The first argument is passed to a variable with a special name: $1. We can use this feature to make the triangle script work for any program.
  • Here is the resulting new script in action:
  • $ cat triangle
  • cc -n32 -cckr -O3 -LNO -Ofast -TARG:madd=on -o $1$1.c
  • $1 < sides.in > $1.dat
  • cat $1.dat
  • rm $1
  • $ triangle area
  • Area of Triangle is 10.392305
  • $ triangle perim
  • Perimeter of Triangle is 18.000000
a more versatile script1
A More Versatile Script
  • Limitation of versatile script:
    • Scripts should be as independent of possible of “outside” existing files (for portability and ease of use). Since the sides.in file is so short, we should create the file within the script
creating files within a script
Creating Files Within a Script
  • The easiest method for creating a small “internal” file is to use “Here-is” Unix redirection. After it is used, this temporary file is removed:
  • $ cat triangle
  • if [[ $2 = debug ]]; then
  • cc -g -o $1 $1.c
  • print "$1 meant to be debugged"
  • else
  • cc -n32 -cckr -O3 -LNO -Ofast -TARG:madd=on -o $1 $1.c
  • cat << EOF > sides.in
  • 8 3 7
  • EOF
  • $1 < sides.in > $1.dat
  • cat $1.dat
  • rm $1 sides.in
  • fi
  • $ triangle perim
  • Perimeter of Triangle is 18.000000
a commented script
A Commented Script
  • A bad script writing style has been shown in all incarnations of the triangle script: the lack of comments. Comments are critical to another user (or you) in understanding what the script does.
  • The special “comment” on the first line of script actually specifies the shell to be used for the script commands. This is required for users whose interactive and script shells differ.
  • On the next page is the timing script just discussed but now with proper commenting
a commented script1
A Commented Script
  • $ cat triangle
  • #!/bin/ksh
  • # Compiling C code found in first parameter
  • cc -n32 -cckr -O3 -LNO -Ofast -TARG:madd=on -o $1 $1.c
  • # Using redirection to make input file
  • cat << EOF > sides.in
  • 8 3 7
  • EOF
  • # Using while loop to run the executable 17 times
  • let i=0
  • while (( i < 17 )); do
  • $1 < sides.in >> $1.dat
  • let i=i+1
  • done
  • # Checking output
  • cat $1.dat
  • # Clean up files no longer needed
  • rm $1 sides.in
script basics
Script Basics
  • Creating a Shell Script
  • Executing a Shell Script
  • Echoing Script Commands
creating a shell script
Creating a Shell Script
  • A shell script is an executable file which is executed by the shell line-by-line. It can contain the following:
    • UNIX commands
    • shell programming statements
    • comments
  • Create using editor of choice
  • Can include a #! construct in first line of script to override login shell
    • #!/bin/ksh uses Korn shell to execute script
    • #!/bin/csh uses C shell to execute script
    • Recommend always put this in every Korn shell script. Makes it clear what the file is. "file" Unix command will even identify it as a Korn Shell script
  • Name should be short and descriptive
executing a shell script
Executing a Shell Script

There are 3 ways to execute a shell script:

1."dot" method

$ . scriptname

2."just the name" method

$ scriptname

3.in the background

$ scriptname &

executing a shell script1
Executing a Shell Script
  • Method 1 runs the command as if you typed them in on the command line
    • No ksh subprocess created
  • Note that methods 2 and 3 require:
    • execute permission for scriptname

chmod +x scriptname

    • current directory (.) must be in PATH or else must use ./scriptname
executing a shell script2
Executing a Shell Script
  • Demonstration of execution methods:
  • $ ps
  • PID TTY TIME CMD
  • 31612 ttyq5 0:00 sh
  • 31631 ttyq5 0:00 ps
  • $ ls -l showsh
  • -rw-r--r-- 1 dje appl 43 Nov 25 1997 showsh
  • $ cat showsh
  • print Executing the shell script showsh
  • ps
  • $ . showsh
  • Executing the shell script showsh
  • PID TTY TIME CMD
  • 31612 ttyq5 0:00 sh
  • 31628 ttyq5 0:00 ps
executing a shell script3
Executing a Shell Script
  • $ chmod u+x showsh
  • $ showsh
  • Executing the shell script showsh
  • PID TTY TIME CMD
  • 31578 ttyq5 0:00 ps
  • 31612 ttyq5 0:00 sh
  • 31635 ttyq5 0:00 sh
echoing script commands
Echoing Script commands
  • For understanding or debugging scripts it is often convenient to see the script commands shown on the monitor along with the output they produce.
  • Accomplished using the set -x command at the beginning of the file
  • To turn off the echoing, make the command set +x (counter-intuitive?)
  • Also useful in batch job files and - on occasion - interactively
  • On the next page a demonstration of the power of set -x is shown
echoing script commands1
Echoing Script commands
  • $ see
  • /homea/dje/ksh_script
  • boba.osc.edu:0.0
  • a_opt.txt cba_opt.txt item.set.3.txt pid.txt
  • b_opt.txt fant4.txt item.set.4.txt showsh.txt
  • c_opt.txt item.intmod.txt no_opt.txt trap.txt
  • $ cat see
  • #!/bin/ksh
  • pwd
  • echo $DISPLAY
  • ls *.txt
  • $ cat see2
  • #!/bin/ksh
  • set -x
  • pwd
  • echo $DISPLAY
  • ls *.txt
echoing script commands2
Echoing Script commands
  • $ see2
  • + pwd
  • /homea/dje/ksh_script
  • + echo boba.osc.edu:0.0
  • boba.osc.edu:0.0
  • + ls a_opt.txt b_opt.txt c_opt.txt cba_opt.txt fant4.txt item.intmod.txt item.set.3.txt item.set.4.txt no_opt.txt pid.txt showsh.txt trap.txt
  • a_opt.txt cba_opt.txt item.set.3.txt pid.txt
  • b_opt.txt fant4.txt item.set.4.txt showsh.txt
  • c_opt.txt item.intmod.txt no_opt.txt trap.txt
script variables
Script Variables
  • User-defined Variables
  • Storing a Value into a Variable
    • The assignment operator =
    • The read command
    • Command Substitution
  • Null Variables
  • Special Shell Variables
  • Positional Parameters
  • Referencing Variables
  • Parameter Substitution
  • Exercises
user defined variables
User-defined Variables
  • The Korn shell includes the following types of variables:
    • User-defined variables
    • Special shell variables
  • User-defined variables can be initialized, used and changed from the command line or from within a shell script.
    • Not "declared" like in a programming language. Just type the name of the variable
    • Korn shell scripts use string and integer variables
  • A variable name can consist of the following:
    • letters, digits, and the underscore character
    • first character of a variable name must be a letter or an underscore character
  • A variable can be made read-only: once it is assigned a value, the value cannot be changed
    • "Archaic" method: readonly variable_name
    • Alternate method: typeset -r variable_name
storing a value into a variable
Storing a Value into a Variable
  • There are several methods for variables to obtain values.
  • The assignment operator =
  • Enter the name that you have chosen for the variable followed by an equal sign and then the value that you want to store in the variable.
  • To use the a variable's value, put the $ operator in front of it's name. (Read the $ as "contents of")
  • The typeset command can also be used for variable assignment. More up-to-date procedure, typeset has many options.
  • To assign a value to an integer variable, the assignment command must be proceeded by the word let
  • On the next page, a script is shown that demonstrates all these approaches
storing a value into a variable1
Storing a Value into a Variable
  • $ cat assign
  • street=Elm
  • print street <-- Most common error when using variables
  • print $street
  • my_name="John Smith"
  • echo $my_name
  • neph=Chris Sellgren
  • print $neph
  • typeset card=ace; print $card
  • let num=57; echo $num
  • let num=num+3; print $num
  • $ assign
  • street
  • Elm
  • John Smith
  • assign2.ksh[9]: Sellgren: not found
  • ace
  • 57
  • 60
storing a value into a variable2
Storing a Value into a Variable
  • Second method: read in a value for a variable
  • So far have used print and echo (archaic) commands for printing out all the strings following them and then a carriage return
  • There is a complimentary command read which input a value into variables. The user type in as many strings as there are variables on the read line, and then hits carriage return
  • On the next page is a demo script indicating how the read command works.
storing a value into a variable3
Storing a Value into a Variable
  • $ cat input
  • print Please enter a ship name
  • read ship
  • print Please enter your full name
  • read fname lname
  • print Captain $lname, welcome aboard the $ship
  • echo May I call you $fname?
  • print Enter an integer; read num
  • let num=2*num
  • echo Twice your number is $num
  • $ input
  • Please enter a ship name
  • Redoubt
  • Please enter your full name
  • Horatio Hornblower
  • Captain Hornblower, welcome aboard the Redoubt
  • May I call you Horatio?
  • Enter an integer
  • 27
  • Twice your number is 54
storing a value into a variable4
Storing a Value into a Variable
  • Use command substitution. The string(s) that a Unix command returns may be put into a script variable
  • The following syntax is as follows:
  • var_name=$(UNIX command)
  • As with all the script commands, command substitution can be done interactively as well
  • On the next page is a demo script illustrating this technique
storing a value into a variable5
Storing a Value into a Variable
  • 136:oscb$ cat command
  • dir=$(pwd)
  • print I am in the directory $dir
  • weekday=$(date +%A)
  • print It is $weekday
  • files=$(ls -lt | wc -l)
  • print There a $files files in this directory
  • me=$(whoami)
  • print I am $me
  • computer=$(hostname)
  • print My computer is called $computer
  • logged_on=$(who | cut -f1 -d' ')
  • print These people are also on $computer: $logged_on
  • 137:oscb$ command
  • I am in the directory /home/dje
  • It is Tuesday
  • There a 277 files in this directory
  • I am dje
  • My computer is called oscb
  • These people are also on oscb: osu2796 opr006 paul ysu039 osu2796 jimg mmittal jimg mmittal osu2224 osu2224 srb cls038 osu1938 jeff dje
null variables
Null Variables

A variable can be set to a null value, even if previously assigned, using any of the following methods. There should be no spaces preceding or following the equal sign. If the contents of null character variables are printed out, a blank appears. If the contents of a “nullled” integer variable is outputted, a 0 is displayed.

$ name=

$ name=''

$ name="”

$ let num=

$ unset varname

null variables1
Null Variables

All variables that don't exist are assumed null unless set -o nounset is used. Then the shell will indicate an error when an undefined variable is encountered

$ unset name

$ set -o nounset

$ print $name

ksh: name: parameter not set

$ set +o nounset

special shell variables
Special Shell Variables
  • In addition to your own user-defined variables, other variables defined by the shell for the user are called special shell variables and are most useful
  • These variables are set and updated automatically by the shell. Their values cannot be changed, but they may be referenced.
  • The special shell variables of most use are identified by the following symbols:

$#

$-

$?

$$

$!

$0

$*

special shell variables1
Special Shell Variables
  • The variable $# contains the number of arguments typed on the command line.

$ cat numargs

print The number of arguments is $#

$ numargs

The number of arguments is 0

$ numargs 1 2 3 4 5

The number of arguments is 5

$ numargs "Hello World"

The number of arguments is 1

special shell variables2
Special Shell Variables
  • The variable $- contains the shell flags (options) of the current shell.

$ print $-

isum

$ set +u

$ echo $-

ism

special shell variables3
Special Shell Variables
  • The variable $? contains the exit status of the last command. If the exit status is 0, the command ran successfully. If it is non-zero, there was a problem
  • $ ls

file1 data account.txt

$ rm file1

$ print $?

0

$ rm dataa

dataa: No such file or directory

$ print $?

2

  • We will see exit status again. It will be used to define true and false. Also, the user can set the exit status in their script by using the exit command
special shell variables4
Special Shell Variables
  • The variable $$ contains the Process ID (PID) of the current shell process.
  • $ cat pid
  • #!/bin/ksh
  • ps
  • print $$
  • $ pid
  • PID TTY TIME CMD
  • 34738 ttyq5 0:00 ps
  • 34744 ttyq5 0:00 sh
  • 34765 ttyq5 0:00 pid
  • 34765
  • $ . pid
  • PID TTY TIME CMD
  • 34743 ttyq5 0:00 ps
  • 34744 ttyq5 0:00 sh
  • 34744
special shell variables5
Special Shell Variables
  • The variable $! contains the process ID number of the last command sent to the background.
  • $ sleeper &
  • [1] 29144
  • $ ps
  • PID TTY TIME CMD
  • 29144 ttyq5 0:00 sleeper
  • 29150 ttyq5 0:00 sh
  • 29160 ttyq5 0:00 ps
  • 29167 ttyq5 0:00 sleep
  • $ compress moby_dick &
  • [2] 29136
  • $ print $!
  • 29136
special shell variables6
Special Shell Variables
  • The variable $0 contains the name of the script currently being execute
  • $cat old_script
  • print The name of this script is $0
  • $old_script
  • The name of this script is old_script
  • $mv old_script new_script
  • $new_script
  • The name of this script is new_script
  • The variable $* contains all the arguments on the command line.

$ cat args

print The arguments are: $*

$ args bob dave

The arguments are: bob dave

positional parameters
Positional Parameters
  • For a script, the command line arguments are also called positional parameters. Each positional parameters refers to one individual argument on the command line. The ten positional parameters available to the script writer are referenced as follows:

$1 $2 $3 $4 $5 $6 $7 $8 $9

  • The parameter $1 contains the first argument, $2 the second argument, and so on. Consider the following script:

$ cat parms

print Arg 1 is: $1

print Arg 2 is: $2

print Arg 3 is: $3

print Arg 4 is: $4

print Arg 5 is: $5

positional parameters1
Positional Parameters

$ parms 2001 A Space Odyssey

Arg 1 is: 2001

Arg 2 is: A

Arg 3 is: Space

Arg 4 is: Odyssey

Arg 5 is:

$ parms "Space, the final frontier"

Arg 1 is: Space, the final frontier

Arg 2 is:

Arg 3 is:

Arg 4 is:

Arg 5 is:

positional parameters2
Positional Parameters

Use the set command to change positional parameters. It replaces existing positional parameters with new values

$cat newpos

print starting args are $*

print number of args is $#

print arg 1 is $1

print arg 2 is $2

set NCC 1701 Enterprise

print new args are $*

print number of args is $#

print arg 1 is $1

print arg 2 is $2

print arg 3 is $3

positional parameters3
Positional Parameters

Here is an example of the use of the newpos script:

$ newpos 76 trombones

starting args are 76 trombones

number of args is 2

arg 1 is 76

arg 2 is trombones

new args are NCC 1701 Enterprise

number of args is 3

arg 1 is NCC

arg 2 is 1701

arg 3 is Enterprise

positional parameters4
Positional Parameters
  • What if there are more than 10 positional parameters?

$ cat ten_args

print arg 10 is $10

$ ten_args a b c d e f g h i j

arg 10 is a0

  • Use the shift command to perform a shift of the positional parameters n positions to the left.

(Default n=1)

  • The shell variables $# and $* also change when the shift command is used (along with the values of the positional parameters.
  • Once a default shift is performed, the first parameter is discarded unless it is saved in a user-defined variable
positional parameters5
Positional Parameters

Examples of shift usage:

$ cat shift_it

print $#: $0: $*

shift

print $#: $0: $*

shift

print $#: $0: $*

shift 3

print $#: $0 $*

$ shift_it It was the best of times, it was the worst of times.

12: shift_it: It was the best of times, it was the worst of times

11: shift_it: was the best of times, it was the worst of times

10: shift_it: the best of times, it was the worst of times

7: shift_it: times, it was the worst of times

$ cat ten_args

arg1=$1

shift

print $arg1 $*

$ ten_args 1 2 3 4 5 6 7 8 9 10

1 2 3 4 5 6 7 8 9 10

referencing variables
Referencing Variables
  • So far we have seen that the syntax for referencing a variable (obtaining its contents) is $var_name
  • An alternate syntax which has more capabilities is ${var_name} This syntax:
    • Allows a script to easily use more than 10 positional parameters
    • Allows for more general use of a variable contents
    • Allows for the number of characters in the variable’s string contents to be determined
    • Allows for parameter substitution: replacement of the value of a variable with an different value based on specific conditions and events.
referencing variables1
Referencing Variables
  • Greater than 10 positional parameters:
  • $ cat moreargs
  • #!/bin/ksh
  • print arg 10 is $10
  • print arg 10 is ${10}
  • print arg 11 is $11
  • print arg 11 is ${11}
  • $ moreargs what rough beast, its time come at last, slouches toward Bethlehem
  • arg 10 is what0
  • arg 10 is toward
  • arg 11 is what1
  • arg 11 is Bethlehem
referencing variables2
Referencing Variables
  • More general use
  • $ cat copy
  • set -x
  • file=lone
  • cp $file $filewolf
  • cp $file ${file}wolf
  • ls lone*
  • $ copy
  • + file=lone
  • + cp lone
  • Insufficient arguments
  • Usage: cp [-aDfirRp] [-b size] [-tP -[e size]] f1 f2
  • cp [-aDfirRp] [-b size] [-tP -[e size]] f1 ... f2 dir
  • + cp lone lonewolf
  • + ls lone lonewolf
  • lone lonewolf
  • Length of a Variable’s contents
  • $ cat length
  • book="Pillars of the Earth"
  • print length of book= ${#book}
  • $ length
  • length of book= 20
parameter substitution
Parameter Substitution
  • Misnamed: should be variable substitution
  • Four syntaxes for parameter substitution
  • ${var_name: - value}
  • ${var_name: = value}
  • ${var_name: + value}
  • ${var_name: ? value}
  • The meanings and typical usages of these four methods are shown on the following pages
parameter substitution1
Parameter Substitution

${parameter:-value}

  • If parameter exists and isn't null, return its present contents; otherwise return value.
  • Purpose: returning a default value if the variable is undefined.
  • If parameter is null, the value returned is temporary. The null value of the variable is not permanently altered
  • Interactive Examples:

$ count=""

$ print You have ${count:-5} cards

You have 5 cards

$ print $count

$ FOO=/usr/local/bin/xx

$ print edit file ${FOO:-$HOME/xx}

edit file /usr/local/bin/xx

$ print $FOO

/usr/local/bin/xx

parameter substitution2
Parameter Substitution
  • As the name implies, this particular syntax of parameter substitution can be used positional parameters.
  • Consider the following script that searches for a file:
  • $ cat fnddemo
  • #!/bin/ksh
  • set -x
  • find ${2:-$HOME} -name $1 -print
  • $ fnddemo bk.txt.Z .
  • + find . -name bk.txt.Z -print
  • ./bk.txt.Z
  • $ fnddemo dale11.ps
  • + find /homea/dje -name dale11.ps -print
  • /homea/dje/dale11.ps
parameter substitution3
Parameter Substitution

${parameter:=value}

  • If parameter exists and isn't null, return its present contents; otherwise set it to value and then return value
  • Purpose: Setting a variable to a default value if it is undefined.
  • This form of parameter substitution cannot be used with positional parameters because their values can only be changed with the set command
  • Interactive Example:

$ count=""

$ print You have ${count:=5} cards

You have 5 cards

$ print $count

5

parameter substitution4
Parameter Substitution

${parameter:+value}

  • This substitution format acts in an opposite manner to the previous two.
  • If parameter exists and is not null, return value; otherwise return null.
  • Purpose: Testing for the existence of a variable
  • Interactive Example:
  • $ cat trace
  • set -x
  • trace=$1; print ${trace:+"Trace mode on"}
  • $ trace y
  • + trace=y
  • + print Trace mode on
  • Trace mode on
  • $ trace2
  • + trace=
  • + print
  • $
parameter substitution5
Parameter Substitution

${parameter:?message}

  • If parameter exists and isn't null, return its contents; otherwise print
  • scriptname: parameter: message
  • and abort the current script. Omitting message produces the default system message.
  • Purpose: Catching errors that result from variables being undefined which need to be.
  • Examples:
  • $ cat fnd2
  • find ${2:-$HOME} -name ${1:?"No file specified"} -print
  • $ fnd2
  • fnd2: 1: No file specified
  • $ cat fnd3
  • find ${2:-$HOME} -name ${1:?} -print
  • $ fnd3
  • fnd3: 1: parameter null or not set
parameter substitution6
Parameter Substitution
  • NOTE: In all forms, the colon (:) is optional. If omitted, the operator tests for existence only, i.e., change "exists and isn't null" to "exists" in each definition.
advanced pattern matching
Advanced Pattern Matching
  • Korn shell enhancements to the set of pattern-matching operators:

OperatorMeaning

${variable#pattern} If the pattern matches the beginning of the variable's value, delete the shortest part that matches and return the rest.

${variable##pattern} If the pattern matches the beginning of the variable's value, delete the longest part that matches and return the rest.

${variable%pattern} If the pattern matches the end of the variable's value, delete the shortest part that matches and return the rest.

${variable%%pattern} If the pattern matches the end of the variable's value, delete the longest part that matches and return the rest.

Mnemonic: # matches the front because number signs precede numbers; % matches the rear because percent signs follow numbers.

advanced pattern matching1
Advanced Pattern Matching
  • Typical use of the above pattern-matching operators is stripping off components of pathnames such as directory prefixes and filename suffixes.
  • Example: Assume the variable TEST has the value /home/job/src/long.file.name then:

ExpressionResult

${TEST##/*/} long.file.name

${TEST#/*/} job/src/long.file.name

${TEST} /home/job/src/long.file.name

${TEST%.*} /home/job/src/long.file

${TEST%%.*} /home/job/src/long

exercises
Exercises

1. Write a script called lsc which executes the command ls -C. To execute this command you must give the full path name to your lsc script. Make the lsc shell script executable and run it.

2.Write a script called b which changes your current directory to /bin, displays the current directory, and then executes the lsc script created above. Make the b script executable and run it. What is your current directory when the b script is finished executing? Why?

3.Write a script called ex_on, which turn on execute permission on the first argument on the command line.

4.Modify ex_on to turn on execute permission on all arguments on the command line.

5.Write a script called 12 that prints the twelfth argument on the command line.

exercises1
Exercises

6. Suppose you keep a file of your home coin collection that keeps track of how many coins you have in a given category. Lines in this file look like:

62 U.S. proofs

11 U.S. pennies (1850-1908)

36 U.S. pennies (1909-1950)

9 U.S. nickels (1861-1938)

  • Write a script that prints the N types of coins of which you have the most. The default for N is 10. The program should take one argument for the name of the input file and an optional argument for how many lines to print.
decision making commands
Flow Control

Exit Status II

if/then/else Command

Condition Tests

Logical Operators

Negation Operator

Illustrative Examples

Case Statement

Exercises

Decision Making Commands
flow control
Flow Control
  • Flow control gives a programmer the power to specify that only certain blocks of shell commands are executed
  • Korn shell supports the following decision making constructs:

if/else Execute a list of statements if a certain condition is/is not true

case Execute one of several lists of statements depending on the value of a variable

exit status ii
Exit Status II
  • For decision making, we will look at the exit status in a different way then we did before
  • The constructs if, while, until and the logical AND (&&) and OR (||) operators use exit status to make logical decisions:

A 0 value is a logical "true" (success)

A nonzero is a logical "false" (failure)

  • There are built-in true and false commands which you can use. The true command returns an exit status of 0, the false command returns an exit status of 1.
exit status of a script
Exit Status of a Script
  • A script, like any other process, sets an exit status when it finishes executing. Shell scripts will finish in one of the following ways:

Abort - If the script aborts due to an internal error, the exit status is that of the last command (the one that aborted the script).

End - If the script runs to completion, the exit status is that of the last command in the script

Exit - If the script encounters and exit command, the exit status is that set by that command

    • Syntax for the exit command is exit [num]. When the exit command is encountered the script ends right there and the exit status is set to num
if else
if / else
  • Simplest type of decision making construct is the conditional embodied in Korn shell's if statement.
  • Syntax:

if condition

then

statements

[elif condition <-- can use multiple elif clauses

then statements . . .]

[else

statements]

fi <-- must be by itself on the final line of the construct

if else1
if / else
  • The if statement uses an exit status to determine whether or not to execute the commands.
  • Statements are executed only if the given condition is true.
  • If one or more elifs are used, the last else clause is an "if all else fails" part.
condition tests
Condition Tests
  • The if construct can only test exit status but that doesn't limit you to checking only whether commands ran properly or not.
  • Using the [[ ]] operator, many different attributes can be tested:
    • string comparisons
    • file attributes
    • arithmetic conditions
  • [[ condition ]] just returns an exit status that tells whether condition is true or not
  • [[ ]] surround expressions that include various types of relational operators.
condition tests strings
Condition Tests: Strings

String comparison operators:

Relational Operator True if ...

str = pat str matches pat.

str != pat str does not match pat.

str1 < str2 str1 is less than str2.*

str1 > str2 str1 is greater than str2.*

-n str stris not null (has length greater than 0).

-z str str is null (has length 0).

*based on the ASCII value of their characters

str refers to an expression with a string value, and pat refers to a pattern that can contain wildcards.

string comparison examples
String Comparison Examples
  • $ cat compstr
  • x=castle; y=cast*
  • if [[ $x = $y ]]
  • then
  • print $x matches $y
  • else
  • print $x does not match $y
  • fi
  • $ compstr
  • castle matches cast*
  • $ cat compstr1
  • x=hello; y=helper
  • if [[ $x > $y ]]
  • then
  • print "$x is > $y"
  • else
  • print "$x is not > $y"
  • fi
  • $ compstr1
  • hello is not > helper
string comparison examples1
String Comparison Examples
  • $ cat compstr2
  • if [[ -n $1 ]]
  • then
  • print First argument is $1
  • fi
  • if [[ -z $1 ]]
  • then
  • print Where is the first argument?
  • fi
  • $ compstr2 Logan
  • First argument is Logan
  • $ compstr2
  • Where is the first argument?
condition tests files
Condition Tests: Files

File attribute checking:

Relational Operator True if ...

-a file file exists

-d file file is a directory

-f file file is a regular file

-r file there is read permission on file

-s file file is not empty

-w file you have write permission on file

-x file you have execute permission on file, or directory search permission if it is a directory

-O file you own file

-G file your group id is same as file's

file1 -nt file2 file1 is newer than file2

file1 -ot file2 file1 is older than file2

There are 21 such operators in total.

file checking demo directory
File Checking Demo Directory
  • Consider a directory that contains the following files and subdirectories
  • $ls -lrt
  • -rwxr--r-- 1 dje appl 5 Jun 27 2000 file
  • -rw-r--r-- 1 dje appl 5 Jun 27 2000 my_prog
  • -rw-r--r-- 1 dje appl 5 Jun 27 2000 my_scr.1
  • drwxrwsr-x 2 dje appl 512 Jul 3 2000 scripts
  • -rw-rw-r-- 1 dje appl 5 Sep 4 2000 slide1.ps
  • drwxr-sr-x 2 dje appl 512 May 18 11:54 probs
  • drwxr-sr-x 2 dje appl 512 May 18 11:54 solutions
  • -rw-r--r-- 1 dje appl 0 May 18 11:54 court.msg
  • -rw-r--r-- 1 dje appl 0 May 18 11:55 heel
  • -rw-r--r-- 1 dje appl 0 May 18 11:57 greed
  • -rwxr-xr-x 1 dje appl 80 May 18 12:01 file.c
  • -rwxr-xr-x 1 dje appl 80 May 18 12:01 file.f90
  • -rw-r--r-- 1 dje appl 0 May 18 12:04 typescript
file checking examples
File Checking Examples
  • $ cat fexist
  • if [[ -a $1 ]] then
  • print File: $1 does exist
  • else
  • print File: $1 does not exist
  • fi
  • $ fexist sort.c
  • File: sort.c does not exist
  • $ cat ffull
  • if [[ -s $1 ]] then
  • print $1 has something in it
  • else
  • print $1 is empty
  • fi
  • $ ffull heel
  • heel is empty
  • $ cat fx
  • if [[ -x $1 ]] then
  • print I can execute $1
  • else
  • print I cannot execute $1
  • fi
  • $ fx file
  • I can execute file
condition tests integers
Condition Tests: Integers
  • Need to use a new test operator for integer comparisons
  • (( … )) instead of [[ … ]]

Relational OperatorMeaning

< less than

> greater than

<= less than or equal

>= greater than or equal

== equal

!=not equal

integer comparison examples
Integer Comparison Examples
  • $ cat intcomp
  • if (( $1 < $2 )) then
  • print $1 is less than $2
  • fi
  • if (( $1 <= $2 )) then
  • print $1 is less than or equal to $2
  • fi
  • if (( $1 == $2 )) then
  • print $1 is equal to $2
  • fi
  • if (( $1 > $2 )) then
  • print $1 is greater than $2
  • fi
  • if (( $1 >= $2 )) then
  • print $1 is greater than or equal to $2
  • fi
  • if (( $1 != $2 )) then
  • print $1 not equal to $2
  • fi
integer comparison examples1
Integer Comparison Examples
  • $ intcomp 3 89
  • 3 is less than 89
  • 3 is less than or equal to 89
  • 3 not equal to 89
  • $ intcomp 13 13
  • 13 is less than or equal to 13
  • 13 is equal to 13
  • 13 is greater than or equal to 13
logical operators
Logical Operators
  • Two special symbols allow the user to combine relational tests into more complex conditions

&& is a logical AND

|| is a logical OR

  • How they work:

statement1 && statement2

statement2 will execute only if statement1 was successful (returned an exit status of 0)

statement1 || statement2

statement2 will execute only if statement1 was not successful (returned a nonzero exit status).

logical operators1
Logical Operators
  • The logical operators && and || are commonly used with if/else commands
  • Examples:
  • $ ls -l
  • -r-xr-xr-x 1 dje appl 365 May 18 15:13 intcomp
  • -rw-r--r-- 1 dje appl 209 May 18 15:12 length.txt
  • $ cat setw
  • #!/bin/ksh
  • if [[ -f $1 && -x $1 ]] then
  • print You are getting write permission for $1
  • chmod u+w $1
  • fi
  • $ setw length.txt
  • $ setw intcomp
  • You are getting write permission for intcomp
  • $ ls -l
  • -rwxr-xr-x 1 dje appl 365 May 18 15:13 intcomp
  • -rw-r--r-- 1 dje appl 209 May 18 15:12 length.txt
logical operators2
Logical Operators
  • $ cat gone
  • #!/bin/ksh
  • if [[ $1 = quit || $1 = bye ]] then
  • print You have closed correctly
  • else
  • print Improper closing word
  • exit 11
  • fi
  • $ gone bye
  • You have closed correctly
  • $ echo $?
  • 0
  • $ gone quit
  • You have closed correctly
  • $ echo $?
  • 0
  • $ gone end
  • Improper closing word
  • $ echo $?
  • 11
the negation operator
The Negation Operator
  • If an exclamation point ! is put before any of the conditions discussed, it reverses the “truth or falseness” of the condition.
  • [[ ! -d probs ]] returns true (exit status 0) if probs is not a directory
  • [[ ! Bob = ?o? ]] returns false (exit status 1) even though Bob matches the pattern
combining different types of conditions
Combining Different Types of Conditions
  • If you want to combine an integer comparison with file or string comparison within the same condition, you must do two things:
    • Use [[ … ]] for the test operator
    • Use the integer relational operators shown below

Relational OperatorComparison

-ltless than

-leless than or equal

-eqequal

-gegreater than or equal

-gtgreater than

-nenot equal

combining conditionals example
Combining Conditionals Example
  • $ cat fileint
  • #!/bin/ksh
  • let num=$2
  • if [[ $1 < tact || num -ge 3 ]]; then
  • print Pippin
  • else
  • print Merry
  • fi
  • $ fileint ace 45
  • Pippin
  • $ fileint vice 2
  • Merry
illustrative examples
Illustrative Examples

Test operator run interactively

$ [[ -z "" ]]

$ print $?

0

$ [[ -z foo ]]

$ print $?

1

Treating integers strings as strings: compressed format

$ if [[ 6 > 57 ]] ; then print huh?;fi

huh?

slide91
Illustrative Examples

The following script sets user execute permission on an ordinary, file that is not a directory:

if [[ -f $1 && ! -d $1 ]]; then

chmod u+x $1

fi

illustrative examples1
Illustrative Examples
  • The following script removes the first file if it's older than the second file and the variable KEY is non-null:

if [[ $1 -ot $2 && -n $KEY ]]

then

/bin/rm $1

fi

illustrative examples2
Illustrative Examples

The following script compares two files and if no differences are found, removes the second file. Also helpful USAGE programming style:

USAGE="Usage:\t $0 file1 file2"

if [[ $# -ne 2 ]]; then

print ${USAGE}

exit 1

fi

diff $1 $2 > /dev/null

if [[ $? -eq 0 ]]; then

/bin/rm $2

print $2 removed, $1 kept

else

print $1 and $2 differ

fi

case command
Case command
  • Provides a multiple choice decision structure. When executed one of several statement blocks will be run.
  • Replacement for the “elif-ladder”
  • Lets you test strings against patterns that can contain wildcard characters.
  • Syntax:

case expression in

pattern1)

statements ;;

pattern2)

statements ;;

...

esac

case operation
Case Operation
  • If expression matches one of the patterns, its corresponding statements are executed
  • If there are several patterns separated by pipes, the expression can match any of them in order for the associated statements to be run ( | acts as logical or )
  • Patterns are checked in order for a match; if none is found, nothing happens
  • When the double colons (;;) are encountered at the end of the one of the statement lists, the case terminates
case examples
Case Examples

Here's a simple script which moves C and Fortran source files to one directory and object code files to another:

case $file in

*.c|*.f)

/bin/mv $file ${HOME}/src ;;

*.o )

/bin/mv $file ${HOME}/obj;;

* )

print $file not moved ;; (Default value or USAGE)

esac

(Could have also used *.[cf] wildcard construct above)

case examples1
Case Examples
  • The case statement is often used to specify options for a shell script.
  • Here is a shell script called dt_fmat that allows the user to enter options that affect the way the date is displayed:

case $1 in

-d) print -n Date:

date +"%a %h %d" ;;

-t) print -n Time:

date +"%T" ;;

-w) print -n Weekday:

date +"%a" ;;

-y) print -n Year: 20

date +"%y" ;;

-m) print -n Month:

date +"%h" ;;

*) print -n Date:

date +"%a %h %d"

print -n Time:

date +"%T" ;;

esac

case usage
Case Usage
  • $ date
  • Sat May 19 12:01:57 EDT 2001
  • $ dt_fmt -d
  • Date:Sat May 19
  • $ dt_fmt -t
  • Time:12:02:05
  • $ dt_fmt -w
  • Weekday:Sat
  • $ dt_fmt -y
  • Year: 2001
  • $ dt_fmt -m
  • Month:May
  • $ dt_fmt
  • Date:Sat May 19
  • Time:12:02:54
elif ladder vs case
“Elif-Ladder” vs Case
  • The following script shows and “elif-ladder” and a case statement that do the exact same thing. Note how the case statement has a simpler syntax
  • if [[ $1 = cat || $1 = dog ]]; then
  • print You have a normal pet
  • elif [[ -z $1 ]]; then
  • print You have no pets
  • else
  • print You have a wierd pet
  • fi
  • case $1 in
  • cat|dog) print You have a normal pet ;;
  • '') print You have no pets ;;
  • *) print You have a wierd pet ;;
  • esac
  • Advantage of “elif-ladder”: not limited to string comparisons
exercises2
Exercises

1.Write a script called char that checks a single character on the command line, c. If the character is a digit, digit is displayed. If the character is an upper or lowercase alphabetic character, letter is displayed. Otherwise, other is displayed. Have the script print an error message if the argument c is more than one character in length.

2.Write a script called mycopy that copies a source file to a directory. Add a check to see if the source file exists. If the source file does not exist, print an error message.

3.Write a script called mydir that prints the message File is a directory if the file is a directory.

exercises3
Exercises

4.Write a script called ver that accepts one argument on the command line indicating the name of a file. The file is copied to another file with the same name with the addition of the extension .v2. Also, the line #Version 2 is added to the top of the file.

5.Execute the ver script on itself, creating a new version of the file called ver.v2.

6.Rewrite ver.v2 to accept a possible second argument. If two arguments are entered, the file specified by the first argument is copied to a file with the name of the second argument. If no second argument is entered, the file is copied to another file with the

same name, adding the extension .v2. In either case, the line #Version2 is added to the top of the file.

looping commands
Looping Commands
  • for
  • select
  • while
  • until
  • Exercises
for loop command
for Loop Command
  • The for loop allows a section of commands to be run a fixed number of times
  • During each time through an iteration, the loop variable is set to a different value.
  • Syntax:

for name [in list]

do

statements that can use $name

done

where

  • name is the loop variable which can be called anything
  • list is a list of strings (defaults to $*) separated by spaces
  • Operation:name is set to each string in list, in order, for each iteration; the number of iterations equals the number of names in list.
for loop examples
for Loop Examples

Outputting a Phrase

$ cat simple

for word in This is a test

do

print $word

done

$ simple

This

is

a

test <-- Omitting the in This is a test phrase

and instead running the script as

$ simple This is a test

would yield the same output.

for loop examples1
for Loop Examples

The list can be the contents of a local variable:

Check to see who is logged on the machines listed in the variable SYSTEMS="myrtlegull sandy newport daytona":

for sys in $SYSTEMS

do

finger @$sys

print

done

for loop examples2
for Loop Examples

list can contain shell wildcards and command substitution as well:

$ cat file1

for i in *

do

print $i

done

$ file1

file1

file2

$ cat file2

for i in $(ls)

do

print $i

done

$ file2

file1

file2

for loop examples3
for Loop Examples
  • Can explicitly make the list all the command line arguments (also notice the alternate syntax):
  • $ cat upper
  • #!/bin/ksh
  • for bug in $* ; do
  • typeset -u day=$bug
  • print $day
  • done
  • $ upper Robson Bond Green
  • ROBSON
  • BOND
  • GREEN
select command
select Command
  • select allows you to generate simple interactive ASCII menus easily.
  • Syntax:

select name [in list]

do

statements that can use $name

done

  • This is the same syntax as the for loop except for the keyword select. As with for, in list defaults to $* if omitted.
select operation
select Operation
  • Each item in list is displayed as the choice in the menu with a number preceding it. At the bottom of the menu the PS3 prompt is displayed and script is waiting for the user to select a choice by entering its number. (A default value for the PS3 prompt is provided by the system but the user can change it)
  • The selected choice is stored in name and the selected number in a system-predefined variable called REPLY
  • The statements in the body of the do loop are run
  • Repeats the process forever; exit loop with break statement (or user can issue ctrl-d)
select example
select Example

The following script termselect allows you to select a terminal setting:

PS3='terminal? '

select term in vt100 vt220 xterm

do

if [[ -n $term ]]; then

TERM=$term

print TERM is $term

else

print invalid choice

fi

done

select example1
select Example

$ termselect

1) vt100

2) vt220

3) xterm

terminal? 4

invalid choice

terminal? 3

TERM is xterm

select example2
select Example
  • $ cat travel
  • select i in Past Present Future; do
  • print You have picked choice $REPLY
  • print "You will go to the $i"
  • break
  • done
  • $ travel
  • 1) Past
  • 2) Present
  • 3) Future
  • #? 1
  • You have picked choice 1
  • You will go to the Past
  • $ travel
  • 1) Past
  • 2) Present
  • 3) Future
  • #? 3
  • You have picked choice 3
  • You will go to the Future
  • $
while command
while Command
  • Allows a section of code to be run repetitively while a certain condition holds true.
  • Syntax:

while condition

do

statements ...

done

  • Operation: while conditionremains true (exit status 0) the body of the while loop is run
  • [[ ]]operator can be used here as with if to specify the condition
  • Beware of creating an infinite loop (condition must become false at some point).
  • If condition is false when a while loop is first encountered, the loop will not be executed even once
while example
while Example

Example: print out positional parameters:

$ cat args

while [[ $# -ne 0 ]]; do

print $1

shift

done

$ args There are 7 “deadly sins”

There

are

7

deadly sins

while example1
while Example

Example: Using a while as a counting loop:

  • $ cat whileint
  • print $#: $3
  • let i=1
  • while (( i <= 5 )); do
  • shift
  • let i=i+1
  • done
  • print $#: $3
  • $ whileint 1 3 5 7 9 11 13 15 17 19
  • 10: 5
  • 5: 15
until command
until Command
  • Allows a section of code to be run repetitively until a certain condition becomes true.
  • Syntax:

until condition

do

statements ...

done

  • Operation: while conditionremains false (exit status nonzero) the body of the until loop is run
  • [[ ]]operator can be used here as with if to specify the condition
  • Typically, condition is false when an until loop is first encountered; thus the loop will not be executed at least once
  • Just about any until can be converted to a while by simply negating the condition
until example
until Example
  • Classic example: checking user input
  • $ cat prompt
  • #!/bin/ksh
  • until [[ $ans == no || $ans == yes ]]; do
  • print -n 'Continue? (yes or no): '
  • read ans
  • done
  • $ prompt
  • Continue? (yes or no): y
  • Continue? (yes or no): sure
  • Continue? (yes or no): ok
  • Continue? (yes or no): yes
  • $ prompt
  • Continue? (yes or no): n
  • Continue? (yes or no): nope
  • Continue? (yes or no): no way
  • Continue? (yes or no): no
  • $
exercises4
Exercises

1.Write a script called lis that uses a for loop to display all files and directories in the current directory.

2. Write a script called dirs that displays only the names of the directory files in your current directory.

3. Using the select command, write a script that creates a menu of the abbreviations for all the months of the year. If a certain month abbreviation is chosen, the script should print out the full name of the month and and number of days in it. Change the PS3 prompt to something appropriate.

4. Using a while loop, write a script that displays a guessing game. The player is to guess a number between 0 and 59. If the guess is too low, the message “Higher” is printed. If the guess is too high the message “Lower” is printed. Guesses continues in until the player guesses the correct number. The number the user is trying to guess is taken from the seconds of the date command.

exercises5
Exercises

5. Modify the script from problem 4 to keep track of the number of guesses it takes for the user to discover the correct number. Have the script print the total number of guesses that it took.

6. Rewrite the first while example script shown in this chapter (printing out the positional parameters) using the until statement instead.

working with integer variables
Working with Integer Variables
  • Arithmetic Operations
  • Operator Precedence
  • Numerical Bases
  • String Variables vs Integer Variables
  • Exercises
arithmetic operations
Arithmetic Operations
  • So far we have seen how to give values to integer variables and compare them (in two different ways!)
  • In this section, integers will be used in arithmetic operations
  • Below is a list of the most common operations performed with integers
  • ArithmeticOperatorMeaning

+ addition

- subtraction

* multiplication

/ division with truncation

% modulus division

  • The shell interprets symbols surrounded by $(( and )) as arithmetic expressions.
  • Variables in arithmetic expressions do not need to be preceded by dollar signs.
arithmetic examples
Arithmetic Examples
  • Demonstration of the integer divisions:
  • $ cat intop
  • let x=31
  • let y=8
  • print The integer division of $x by $y is $((x/y))
  • print The modulus division of $x by $y is $((x%y))
  • $ intop
  • The integer division of 31 by 8 is 3
  • The modulus division of 31 by 8 is 7
  • Parentheses can give operations higher precedence:
  • $ cat paren
  • #!/bin/ksh
  • let x=$(( $1 + $2 * $3 - $4 ))
  • print $x
  • let x=$(( ($1 + $2) * ($3 - $4) ))
  • print $x
  • $ paren 2 4 6 3
  • 23
  • 18
different numerical bases
Different Numerical Bases
  • Possible to represent integers in alternate bases:
  • $ cat base
  • #!/bin/ksh
  • print -n $(( 57 ))
  • print -n ,
  • print -n $(( 16#39))
  • print -n ,
  • print -n $(( 8#071))
  • print -n ,
  • print -n $(( 2#00111001))
  • print
  • $ base
  • 57,57,57,57
  • $
string variables vs integer variables
String Variables vs Integer Variables
  • Arithmetic does not work with string variables:
  • $ cat intstr
  • print As integers ...
  • if (( $1 < $2 )) then
  • print $1 is less than $2
  • fi
  • let x=$2+4; print $2+4 is $x
  • print As strings ...
  • if [[ $1 > $2 ]] then
  • print $1 is greater than $2
  • fi
  • x=$2+4; print $2+4 is $x
  • $ intstr 3 15
  • As integers ...
  • 3 is less than 15
  • 15+4 is 19
  • As strings ...
  • 3 is greater than 15
  • 15+4 is 15+4
exercises6
Exercises

1. Write a script which finds the sum of all the positional parameters.

2. Write a script to determine if the first positional parameter is prime.

3. Write a script to perform the factorial of the first positional parameter

4. Write a script that will output positive and negative integers.

5. Write a script that will convert the first positional parameter from base 10 to base 5.

6. Write a script that reads in a time in seconds and outputs as hours:minutes:sec.

script input output
Input / Output Redirection

“Here-is” documents

print

read

Quoting

Exercises

Script Input / Output
input output redirection
Input / Output Redirection
  • The basic I/O redirectors >, < and | are good enough for most tasks.
  • The Korn shell supports 16 redirectors in total.
  • Here are some other useful ones:

Redirector Function

>> file Direct stdout to file, appending to file if it already exists

>| file force stdout to file even if noclobber is set

<< label “here-is” document (see below)

n> file direct file descriptor n to file

n<file set file as file descriptor n

>&n duplicate stdout to file descriptor n

<&n duplicate stdin from file descriptor n

input output redirection1
Input / Output Redirection

Example:

This script called start which can be used to start a different script (entered as positional parameters) in the background and save both stdout and stderr in a single log file.

$* > logfile 2>&1 &

If we want to do the same as above but to the screen as well:

$* 2>&1 | tee logfile &

The command tee takes its stdin and copies it to stdout and to the file given as argument.

here is documents
“Here-is” documents
  • Special form of inline input redirection
  • Syntax:

command << label

input for command

....

label

  • The shell performs command and variable substitution where appropriate unless label is surrounded by quotes
print
print
  • We have already seen many examples of the print command.
  • Here are some useful escape sequences and options; enclose escape sequences in quotes (either double or single):

Escape Sequence Character printed

\a bell

\b backspace

\r return

\t tab

Option Function

-n omit the final newline

-r ignore any escape sequence

-s print to history file

-un print to file descriptor n

print example
print Example
  • $ cat escape
  • #!/bin/ksh
  • print "Call me \t Ishmael"
  • print -r "Call me \t Ishmael"
  • $ escape
  • Call me Ishmael
  • Call me \t Ishmael
  • $ cat print.u3
  • #!/bin/ksh
  • print -u3 "My errors:" 3>errors
  • $ ls errors
  • Cannot access errors: No such file or directory
  • $ print.u3
  • $ cat errors
  • My errors:
slide132
read
  • We have seen simple examples of read.
  • Shell scripts with read are useful for certain tasks such as reading in lines from a file
  • Suppose we have the file /etc/nicks :

myrtle beach_bum

killian brewster

daytona speedster

...

which is a list of hostnames followed by nicknames. We want to assign the variable NICKNAME to the appropriate nickname depending on which machine you log into.

slide133
read
  • The following code will do it:

machine=$(uname -n)

while read host nick; do

if [[ $host = $machine ]]; then

NICKNAME=$nick

print NICKNAME is $NICKNAME

break

fi

done < /etc/nicks <-- read exits with status 1 when there is

nothing left to read

quoting
Quoting
  • A source of confusion for novice and experienced Korn shell script writers is when to use single quotes ‘ ‘ and when to use double quotes “ “
  • Throughout the workshop we have learned that certain symbols have special meanings: $name indicates the contents of the variable name, $(command) indicates the output of a command, etc.
  • Single quotes are strong quotes: they prevent the special meaning symbols from performing their tasks. The symbols are treated just like normal characters.
  • Double quotes are weak quotes: they allow the following actions to take place:
    • variable and positional parameter substitution
    • command substitution
    • arithmetic calculations
quoting demonstration
Quoting Demonstration
  • $ cat quote
  • #!/bin/ksh
  • money=franc
  • print "Nonsense words ${money} $1 $(date +%T) $((3*75))"
  • print 'Nonsense words ${money} $1 $(date +%T) $((3*75))'
  • $ quote raven
  • Nonsense words franc raven 16:23:30 225
  • Nonsense words ${money} $1 $(date +%T) $((3*75))
exercises7
Exercises

1.Write a shell script tfile which creates a test file called junk. Do not use a cp or mv command to accomplish this. File junk contains the following:

This is a test

abcdefghijklmnopqrstuvwxyz

0123456789

end of test

2. Create three files called data0, data1, and data2 which contain the month, day, and year respectively of your birthday. Write a single script that reads from all three of these files and then outputs your birthday to the screen

signal handling
Signal Handling
  • We want to have control over our shell scripts even when they are interrupted by signals.
  • Signals come in the form of control keys, the kill command and user-initiated events like logging out.
  • Programs can be set up to "trap" specific signals and process them in their own way.
  • Syntax of trap command:

trap command sig1 sig2 ...

when any of sig1, sig2, etc., are received, run command, then resume execution

signal handling1
Signal Handling
  • After command finishes, the script resumes execution just after the command that was interrupted.
  • sig1, sig2, ... can be given by name or number (name is more portable).
  • trap without arguments lists traps that have been set
signal handling2
Signal Handling

Simple example - shell script loop :

trap 'print You hit ctrl-c!' INT

while true: do

sleep 60

done

ctrl-c will not kill this program but rather print the message and start another sleep; type ctrl-\ to stop it (then rm core).

signal handling3
Signal Handling

Enhancing the script to handle background execution (trap the kill command):

trap 'print You hit ctrl-c!' INT

trap 'print No kill I!' TERM

while true: do

sleep 60

done

if run as loop &, kill %loop won't stop it; need kill -KILL %loop

Note: kill -l will give the numbers associated with given signals (defined in

/usr/include/sys/signal.h).

signal handling4
Signal Handling

Examples:

  • If any of the signals listed (0, 1, 2, 3, or 15) are received by the program, file temp$$ will automatically be removed. Without the exit command, program execution would

continue from the point where the signal was received:

trap 'rm -f temp$$; exit 0' 0 1 2 3 15

  • The command listed for trap is null. This specifies that the signals listed are to be ignored, and the program will try to continue where it left off:

trap "" 0 1 2 3 15

  • Using the null command (:) specifies that the signals listed are to be ignored for the next command only:

trap : 0 1 2 3

signal handling5
Signal Handling
  • What if you want a program running in the background to continue even after you log out?

Shell sends the HUP (hangup) signal to all of your background jobs when you log out.

  • Use the command nohup to put a job into the background which will ignore HUP and QUIT(ctrl-\) signals:

nohup $* > logfile 2>&1 &

signal handling6
Signal Handling

Most common signals:

SignalNameNote

1 HUP Hangup

2 INT ctrl-c

3 QUIT ctrl-\

9 KILL Cannot be trapped

15 TERM default kill signal

24 TSTP ctrl-z

script functions
Script Functions
  • Why Use Functions?
  • Syntax & Use
  • Auto-Loading
  • Function Definition Location
  • Command Precedence
  • Exercises
functions
Functions
  • Improves shell's programmability
    • already in memory (unless autoloaded)
    • modular programing
  • Syntax:

function functname {

shell commands

}

or,

functname () {

shell commands

}

functions1
Functions
  • Delete a function definition with

$ unset -f functname

  • Display all defined functions:

$ functions

functions2
Functions
  • Two important differences between functions and shell scripts run by name (Method 2):
    • functions do not run in a subprocess; behave more like a script run by Method 1
    • functions have precedence over scripts
  • Where to put them?
    • enter on command line (no longer available if you log out and log back in)
    • .profile (only available in login shell)
    • $ENV file (available to all child shells)
    • in $FPATH directory (Korn shell will use autoload feature)
  • Function names should begin with a letter or underscore
  • The closing brace (}) must appear on a line by itself
functions3
Functions

Example: Run a program and view the output

$ myrun ()

> {

> cd $WORKDIR

> a.out < data.in

> more data.out

> /bin/rm data.in

> mv data.out data.in

> }

exercises8
Exercises

1.Write a script which sets the PAGER environment variable. The script should accept input values from:

  • command-line arguments:
  • -p pager or -ppager
  • shell environment variable PAGER
  • default setting: more -s

Include a usage line. (Hint: You should try using the pattern matching operators introduced in this section)

2. Write a script that implements a simple version of the shell's built-in whence command. It should check each of the directories on your PATH for a command given as an argument, and print the full pathname if the command is found. Only files that are executable by you (and are not directories!) should be returned.

exercises9
Exercises

3. Design functions pushd and popd which implement a "stack" of directories (as in the C shell).

  • The stack is just a list of directories, stored in an environment variable we'll call DIRSTACK.
  • pushd should "push" a specified directory onto the stack, i.e., prepend the directory name to DIRSTACK.
  • popd "pops" a directory off the stack, i.e., cds to the top directory on the stack and then removes that directory from the stack.
  • Your functions should:
    • Allow the use of "." to specify the current directory.
    • Return an error message if pushd is invoked without an argument, or if the specified directory does not exist.
    • popd should return an appropriate message and do nothing if the stack is empty.
  • Why are these implemented as functions rather than scripts?
  • What modifications (if any) should you make to your ~/.profile in order to use them?
exercises10
Exercises

4. Write a script that will change the suffix of all the files in the present directory from .old to .new

hpc designed scripts
HPC Designed Scripts
  • Repeating Program Execution
  • Debugging Options
  • Batch Jobs
  • Performance Tool Preprocessing
  • Multiple Tool Runs
  • Multiple Tool Reports
  • File Renaming
repeating program execution
Repeating Program Execution
  • It is often desired to run a program several times. Even a serial programs may have to run repeatedly to debug intermittent problems, find logical errors, etc.
  • In HPC parallel processing arena, there are even more reasons to write a script that will execute code many times:
    • Each run is with a different number of processors so that the traditional speedup graph data can be collected (and compared with Ahmdal’s Law)
    • To perform repeated timing of the entire or partial code.
      • Several parallel machines do not allow dedicated use of processors
      • A severe load imbalance may come and go depending on how concurrently the various processors actually work
      • Over synchronized program’s times can vary due to the order of arrival of the processors at barriers
    • To gather data from program output for statistical analysis. One can also gather timing data for similar reasons.
    • To confirm that a parallel program is preventing race conditions which can arise when one processor is interrupted by another when the former is trying to read, update, and write back to a shared memory location
  • Repeated runs is easily done in our introductory triangle script by using a while loop. Here the script will run the timearea code which times how long the program takes to calculate the area
timing script
Timing Script
  • $ cat triangle
  • cc -n32 -cckr -O3 -LNO -Ofast -TARG:madd=on -o $1 $1.c
  • cat << EOF > sides.in
  • 8 3 7
  • EOF
  • let i=0
  • while (( i < $2 )); do
  • $1 < sides.in >> $1.dat
  • let i=i+1
  • done
  • cat $1.dat
  • rm $1 sides.in
timing script usage output
Timing Script Usage & Output
  • $ triangle timearea 17
  • 1.215377
  • 1.227504
  • 1.197454
  • 1.193690
  • 1.220033
  • 1.190733
  • 1.194406
  • 1.221519
  • 1.201336
  • 1.187614
  • 1.197026
  • 1.220640
  • 1.217389
  • 1.218919
  • 1.192417
  • 1.218107
  • 1.228289
debugging options
Debugging Options
  • To use debugging software (dbx, Totalview, Xbrowse, etc) programs need to be compiled with certain options.
  • The debugging option is traditionally –g but has been improved lately to include various levels of optimizations while the debugging is being performed (G0,G1,…)
  • For all debugging options, the compiler will make a symbol table which connects a program identifier (variable name, etc.) with a memory address. This allows the debugger to track the values of any variables as the program is executed line by line
  • On next page we show a variation of the introductory triangle script in which a decision is made- through the use of a second positional parameter- if debugging will be implemented
debugging script
Debugging Script
  • $ cat triangle
  • if [[ $2 = debug ]]; then
  • cc -g -o $1 $1.c
  • print "$1 ready to be debugged"
  • else
  • cc -n32 -cckr -O3 -LNO -Ofast -TARG:madd=on -o $1 $1.c
  • $1 < sides.in > $1.dat
  • cat $1.dat
  • rm $1
  • fi
  • $ triangle area
  • Area of Triangle is 10.392305
  • $ triangle area debug
  • area ready to be debugged
batch jobs
Batch Jobs
  • There are a number of popular batch processing software that are heavily used on HPC machines: NQE, PBS, and LSF to name three.
  • At this point in the course, we now realize that the “batch files” we are used to writing are actually just shell scripts. The header lines have the syntax and are treated as comments if the batch script is not submitted to the batch software.
  • But since these batch scripts have designed purpose – to run the UNIX commands in them and send a log file back-we can tailor them to reflect it.
  • For example, it is recommended that the shell command echoing feature be turned on so that the commands appear in the log file preceding their output. An example is shown on the next slides.
a batch script
A Batch Script
  • Here is a simple batch job (file sp_gravity.job):

#QSUB -r grav

#QSUB –lT 3000

#QSUB –lM 10Mw

#QSUB -oe

set -x # echo commands

cd gravity/batch

ls -lrt

f77 -o spg sp_gravity.f

./spg < region.rsp

ls –lrt

a log file grav o3613
A Log file (grav.o3613)

+ ls -lrt

total 1512

-rw-r----- 1 dje G-541 46098 May 18 12:22 sp_gravity.f

-rw-r----- 1 dje G-541 637236 May 18 12:22 newbedrx.cray

-rwxr----- 1 dje G-541 299 May 18 12:31 region.prm

-rw-r--r-- 1 dje G-541 16 May 18 12:31 region.rsp

-rw-r--r-- 1 dje G-541 125 May 18 12:55 sp_gravity.job

+ f77 -o spg sp_gravity.f

sp_gravity.f:

MAIN xy2glqr:

"sp_gravity.f", line 346: Warning: local variable "zero" never used

gauleg:

+ ./spg

+ 0< region.rsp

Parameter file:

Gravity output file:

region.out

3-D ORTHOGONAL COLLOCATION ON FINITE ELEMENTS **********

(GAUSS-LEGENDRE QUADRATURE)

...

Integration over source region in progress . . .

log file cont
Log file (cont.)

+ ls -lrt

total 1712

-rw-r----- 1 dje G-541 46098 May 18 12:22 sp_gravity.f

-rw-r----- 1 dje G-541 637236 May 18 12:22 newbedrx.cray

-rwxr----- 1 dje G-541 299 May 18 12:31 region.prm

-rw-r--r-- 1 dje G-541 16 May 18 12:31 region.rsp

-rw-r--r-- 1 dje G-541 125 May 18 12:55 sp_gravity.job

-rwxr-xr-x 1 dje G-541 67228 May 18 12:58 spg

-rw-r--r-- 1 dje G-541 2128 May 18 12:58 region.coeff

-rw-r--r-- 1 dje G-541 14502 May 18 13:00 region.out

qsub command with options
qsub Command with options
  • The realization that a batch job is really just a normal shell script, is clear when the script only consists of the Unix commands.
  • The “header” section is replaced by options to the qsub command:

qsub –r grav –lT 3000 –lM 10Mw –eo < grav.ksh

  • Where grav.ksh is the remaining Unix commands:

set -x

cd gravity/batch

ls -lrt

f77 -o spg sp_gravity.f

./spg < region.rsp

ls –lrt

  • This is the alternative ( not recommended) method for running a batch job. It will return the same log file shown on the previous slides.
advanced batch scripting
Advanced Batch Scripting
  • Since a batch script is – just that – a script, we can using any part of the scripting language in it.
  • Consider our “classic” compile and run script we have been using in the previous slides. If the compilation is a failure, there is no point in proceeding with the script’s commands since there will be no executable to work with.
  • In the advanced batch script shown on the next slide we use the if/then shell command to check to see if there were no errors produced in the compilation. If there was, we exit the script (and the job,BTW).
  • The log file returned after submitting this script is also shown. An error was deliberately inserted in the source code.
error checking batch script
Error-Checking Batch Script

set -x

cd SV1/dje/gravity

f90 -o spg sp_gravity.f

if [[ $? -ne 0 ]]; then

exit 33

fi

./spg < region.rsp

error checking log file
Error-Checking Log File

+ cd SV1/dje/gravity

+ f90 -o spg sp_gravity.f

do i=1,

^

f90-197 f90: ERROR XY2GLQR, File = sp_gravity.f, Line = 408, Column = 14

Unexpected syntax: "operand" was expected but found "EOS".

Cray Fortran: Version 3.5.0.4 (u15f51p42138g41109a18)

Cray Fortran: Mon Jun 10, 2002 17:13:34

Cray Fortran: Compile time: 0.3487 seconds

Cray Fortran: 1130 source lines

Cray Fortran: 1 errors, 0 warnings, 0 other messages, 0 ansi

Cray Fortran: "explain f90-message number" gives more information about each message

+ [[ 1 -ne 0 ]]

+ exit 33

performance tool preprocessing
Performance Tool Preprocessing
  • Almost all of the sophisticated software tools that exist on HPC machines to profile your code and aid in serial, vector, and parallel optimization require the user to run the code and generate a raw data file. The tool subsequently analyzes the raw data under the user control (usually interactively)
  • For some HPC software tools the time to gather the raw data can be long, especially if the code is complex with many functions and subroutines. Or the syntax required might be different than ever other optimization tool on that machine.
  • Because of these reasons, it is often useful to perform the generation of raw data as a shell script.
  • The script shown on the next slide, generates the raw data file for the prof tool on the Cray SV1.
using the prof preprocessing script
Using the prof preprocessing script

oscb:$ cat profraw.ksh

#! /bin/ksh

f90 –G1 –o $1 –lprof $1.f

./$1 < $1.in

# Creates the prof.data file

prof –x –m prof.data $1 > $1.prof.raw

# User can pick the raw output filename

rm $1

oscb:$ profraw.ksh poisson

oscb:$ profview poisson.prof.raw

multiple tool runs
Multiple Tool Runs
  • For HPC tools that access Hardware Counters on the actual vectorizing processors or commodity Microprocessors (Pentiums, Itantiums, Ultra-Sparcs, MIPS R12K, etc.) there are too many counter chip to gather all the information with just one run of an optimization tool.
  • Therefore, to access all the hardware use information, your code must be run several times. For Cray machines, 4 runs of hpm and perf are required. For Microprocessor-based MPP machines there a typically 64 counters only two of which can be read out in a single run of the appropriate tool (ssrun for the Origin 2K, collect for the Sun Fire 6800, etc.)
  • Compiling and running an executable, with different options chosen to access various counters is an ideal task for shell programming.
  • In the next slide, a script is shown that gathers all the hardware use info for Cray machines with the hpm and perf tools installed
total hw ksh
total_hw.ksh

#! /bin/ksh

# First parameter is the source code “prefix”

cc –O4 -TARG:madd=on -o $1 $1.c

hpm –g0 ./$1 > $1.hpm

hpm –g1 ./$1 >> $1.hpm

hpm –g2 ./$1 >> $1.hpm

hpm –g3 ./$1 >> $1.hpm

# Recompile with perf options

cc –F –lperf –o $1 –O4 –TARG:madd=on $1.c

typeset –x PERF_GROUP=0 PERF_DATA=p0.raw ./$1

typeset –x PERF_GROUP=1 PERF_DATA=p1.raw ./$1

typeset –x PERF_GROUP=2 PERF_DATA=p2.raw ./$1

typeset –x PERF_GROUP=3 PERF_DATA=p3.raw ./$1

cat p0.raw p1.raw p2.raw p3.raw >perf.data

rm $1

saving multiple tool reports
Saving Multiple Tool Reports
  • A powerful feature of HPC Optimization software tools is that they provide information on a variety of performance topics with just one run of the tool. A certain tool might have a report of subroutine time-profiling, a caller-callee tree, a summary report, etc.
  • When interactively using such a tool, a user can pick which report to read, and then pick another and so on.
  • It is often desirable to just save a select set of the possible reports. This can be done by running the tool several times but with a different option each time. The option determines what report will be saved in that run. Again, a natural job for a shell script.
  • In the next slide, a script is shown that saves several reports from the flowview tool used on Cray PVP machines. (12 reports are available from this tool)
flowview report selection script
flowview report selection script

oscb:$ cat flowchoice.ksh

#! /bin/ksh

# Script assumes raw data file flow.data exists

# First parameter is prefix of the source code

pre=$1

shift

for opt in $x

do

flowview –L$opt >> pre.flow

done

oscb$ flowchoice.ksh neuralnet m h u k

renaming files
Renaming Files
  • Specifically, renaming the prefixes or suffixes of a large number of files. Many HPC parallel programs will produce hundreds of files which contain data at certain time steps, or spatial extent. Data visualization files are one example of this type of output.
  • Say the user needs to put a new suffix or prefix on a large set of files in order that they not be overwritten by a subsequent data set.
  • After this course, one’s mind should naturally think “ I bet a script could do this for me”. You would win that bet.
  • On the next slide is shown a script that will change the suffix of all files whose name has the syntax strain001.txt, strain002.txt to strain500.txt to the same prefix but with “txt” replaced by “old”. The script can easily be made more general purpose as well.
renaming script
Renaming script

oscb$ cat suffix.ksh

#set –x

for file in $*

do

mv $file ${file%.*}.old

done

oscb$ suffix.ksh strain*.txt