Untitled Document

Chapter - 13  Shell Scripting

In all our previous chapters we have seen the different commands and used them on files to get or manipulate the file results .If we frequently need these commands and their operations on files then best thing is store the combination of these commands in a file and the file itself executed as a shell script or shell program . Though its not mandatory , we normally use the .sh or.ksh extension for shell scripts . This makes it easy to match them with wild-cards .

Shell scripts are executed in a separate child process, and this sub-shell need not be of the same type as your login shell .Even though your login shell is Bourne . We can use Korn shell to run our script even though our login shell is Bourne .

First Shell Script -

Lets write a simple script which will print the Today's date , Your Loging Shell and Your Login id . Open your vi editor and write the following code ..

   #!/bin/ksh 
   #This is a sample script print Today's date , Your Loging Shell and Your Login id
   echo " Welcome user `whoami` "
   echo "Your Login Shell is : $SHELL"
   echo " Today's Date is : `date` "

Now Lets understand the script line by line -

The First line is called as interpreter line . It always begins with #! and is followed by the pathname of the shell to be used for running the script .Here we are using Korn Shell .The second line begining with (#) which is a comment character which can be placed anywhere in a line , the Shell ingnores all characters placed on its right .However this doesn't apply to the first line which also begins with a # .

The last three lines are Just printing the output to the terminal using echo statement and command substitution technique .

To run the script we need to make it executable and then invoke the command -

chmod 711 first.shell.script.ksh

We mentioned that shell scripts are executed by a child shell . Actually , the child shell reads and executes each statement in sequence . You can explicitly spwan a child of your choice with the script name as argument

ksh first.shell.script.ksh

When used in this way , the korn shell opens the file but ignores the interpreter line .The script doesn't need to have execute permission either . We will make it a practice to use the interpreter line in all our scripts .

This is a simple script which doesn't take any input or argument from user .Lets create a script which is more interactive and ask some input and arguments from user .

read : Interactive Script -

The read statement in shell is used for taking input from the user to make script interactive .Input supplied through the standard input is read into these variables . Invoke the following statement to read the input -

read fname

The script pasuses here to take input from the keyboard . Whatever you entered is stored in the variable fname . Since this is a form of assignment , no $ is used before name . The following script takes two inputs - first as a string to be searched in a file and second is a filename where we want to search the string

   #!/bin/ksh 
   #The script search the server log for a particular string
   echo " Enter The string to be searched : "
   read strname
   echo " Enter The File name to be looked For :"
   read fname
   echo " You have entered String : $strname and Filename as :$fname "
   grep "$strname" "$fname"

Command Line Arguments -

When arguments are specified with a shell script , they are assigned to certain special "Variables" - Rather we called it as Positional Parameters .

The First argument is read by the shell into the parameters $1 , the second argument into $2 , and so on . We can't really call them shell variables because all variable names start with a letter .In addiition to these positional parameters , there are few other special parameters used by the shell . Their significance is noted below -

  • $*-- It stores the complete set of positional parameters as a single string
  • $# It is set to the number of arguments specified . This lets you design scripts that check whether the right number of arguments have been entered .
  • $0 -- Holds the name of the script itself .

Lets use these positional parametrs in our above script and check its output

   #!/bin/ksh 
   #The script search the server log for a particular string
   echo " You are running Script : $0"
   echo " The number of arguments specified is $#"
   echo " The Arguments are : $* "
   grep "$1" "$2"

When we specified arguments in this way , the first word ( the script name ) is stored in $0 , the second word ( the first argument) to $1 and second word ( the second argument) to $2 . We can store the more positional parameter in this way up to $9 .

When we use multiword string to represent a single command line argument , we must enclosed in quotes . Like if you want to search Mangesh Pande , use "Mangesh Pande" first.prog.ksh .

Here is the summary of the Positional parameters which we may need or help in our scripts.

Parameter Operation
$1 , $2 ..... Storing Command Line Arguments
$# Number of arguments specified in command line
$0 Stores the Name of executed shell script
$* Complete set of positional Parameters as a single script
$@ Each Quoted string treated as a separate argument
$? Exit Status of last command
$! PID of last background job
$$ pid of current shell

exit : Status of Command -

How does shell knows that whether a command or a script completed and executed successfully ? Shell has the command exit () which has numeric vaules decides whether the command is succeded or failed .The two very common exit values run with numeric values are -

  • exit 0 - Used when command execution went fine
  • exit 1 - used when command execution fails or something went wrong

We don't need to put these exit status command at the end of every shell script because shell understand when script exeution is complete .

Every command returns its exit status through exit command to the caller which is nothing but your shell . A command is said to return a true exit status ( a zero 0 value ) if it executes successfully and false ( non zero value (1)) if it fails . The shell offers a variable ($?) and a command (test) that evaluates a command's exit status .

The parameter $? stores the exit status of the last command . It has the value 0 if the command succeeds and a non-zero value if it fails . This parameter is set by exit''s argument.

To find out whether a command executed successfully or not simply use echo $? after the command . 0 indicates success , other vallues point to failure .

Conditional Execution : Logical Operators && and | | -

The Shell provides you two operators that allows conditional execution - the && and | | which has following syntax .

  • cmd1 && cmd2
  • cmd1 || cmd2

The && delimits two commands , the command cmd2 executed only when command cmd1 succeeds . We can use it with grep in this way 

grep "mangesh pande " emp.csv && echo " Pattern found"

The | | operator plays and inverse role , the second command is executed only when the first fails .

grep "mohan pande " emp.csv | | echo " Pattern Not found"

We can make use of these simple conditional operators for making simple decisions . To make the complex decision we have to use if statement which is our next topic of discussion .

The if Statement ( Condition) :

The if statement is used for decision making on the fulfillment of certain conditions . In shell we have following forms of if statement which are much similar to other languages .

   if command is successfull 
   then
   execute command or statements
   else
   execute command or statements
   fi

Form -2

   if command is successfull 
   then
   execute command or statements
   fi

Form -3

   if command is successfull 
   then
   execute command or statements
   elseif command is successfull
   then ..........
   else .......
   fi

In general , by looking at the syntax the form-1 executes the statement inside then only when if conditions become true otherwise it will execute else statement . Its not mandatory to have else part in every if statement form-2 - executes only when if statements is true . You must have noticed that every if statement is closed with corresponding fi and we will encounter an error if fi is not present .

In shell - we can directly pass the commands to if statement , shell determines it exit status and depending on that it performs operation in if else statement . Lets see an example how it works .

   #!/bin/ksh 
   #The script search the server log for a particular string using if else statement
   echo " You are running Script : $0"
   echo " The number of arguments specified is $#"
   echo " The Arguments are : $* "
   if grep "$1" "$2"
   then
   echo " Pattern found"
   else
   echo " Pattern not found"
   fi

There is one more form of if statement which is test - which we will discuss next . This is used for evaluation of expression .

test and [ ] To evalautes exprression :

When we use if to evaluate expression , we need to use test statement because true or false values returned by expression can't be directly handle by if . test uses certain conditions on its right and returns either a true or false status , which is used by if for making decision . tests works in three ways

  • Compare numbers
  • Compare two strings or a single one for a null value
  • Checks a file attributes

The test doesn't display any output but it simply set the parameter $? . Lets how test work in all of the above three comparison

Numeric expression :

The numerical comparison has different set of operators that we have summarise in below table . The operator beging with hyphen (-) and a two letter string . The operators are mnemonic -eq, -ne , -gt etc ( refer the table ) . Numeric comparison in the shell is confined to integer only; decimal values are truncated . To illustrate the numerical expression - lets see assign some values to variable and numerically compare them .

   i=3 ; j=8.3 , k=6 
   test $i -eq $k ; echo $? ------------- Not equal
   1
   test $i -lt $k ; echo $? -------------- True
   0
   test $k -gt $j ; echo $? ------------------- 8.3 is not greater than 6
   1

from the third example of test we can clearly say that numerical comparison is strictly restricted to integers only . In the next example we will use the test and the script will check if user has given correct argument or not ( i.e. 2 inputs in our example) if not then it will show you the usage and if correct arguments are provided then only it will run future commands .Save this script as test.ksh

   #!/bin/ksh 
   #The script search the server log for a particular string using if else statement
   echo " You are running Script : $0"
   echo " The number of arguments specified is $#"
   echo " The Arguments are : $* "
   if test $# -lt 2 ; then
   echo "Usage : $0 "
   elseif test $# -eq 2 ; then
   if grep "$1" "$2"
   then
   echo " Pattern found"
   else
   echo " Pattern not found"
   fi   
Options Operation
-eq Equal to
-ne Not Equal to
-gt Greater Than
-ge Greater than or equal to
-lt Less Than
-le Less Than or equal to

Shorthand for test :

test is so widely used that fortunately there exists a shorthand method of executing it . A pair of rectangular bracket enclosing the expression can replace it .Thus the following two forms are equivalent .

test $i -eq $j

[ $i -eq $j ]

Note that we must provide whitespace around the operators ( like -eq ) , their operands ( like $i) and inside the [ and ] . The second form is easier to handle and will be used henceforth . But don't forget to use whitespace here !

String Comparison: 

Like nummerical comparison we also have a String comparison which compares the strings . Lets see below example -

   #!/bin/ksh 
   # Script to check user input for null values 
   if [ $# -eq 0 ] ; then 
      echo "Enter the string to be searched"
      read pname 
   if [ -z "$pname" ] ; then     # -z checks for a null string
      echo " You have not entered the string" ; exit 1
   fi
   echo " Enter the filename to be used : \c"
   read flname 
   if [ ! -n "$flname" ] ; then  # ! -n is the same as -z 
      echo "You have not entered the filename" ; exit 2
   fi
   test.ksh "$pname" "$flname"   # Runs the script that will do the job 
   else 
   test.ksh $*                   # We'll change $* to "$@" soon 
 fi 
The table below summarises the different comparison operations that we can run on string .

Options Operation
s1 = s2 String s1 = s2
s1 != s2 String s1 is not equal to s2
-n strng String strng is not a null string
-z strng String strng is a null string
strng String strng is assigned and not null s
s1 == s2 String s1 = s2 ( Korn and Bash only )

File Tests :

test can be used to test the various file attributes like its type ( directory or symbolic link ) or its permission (read , write and execute ). The table below summarise the facilties of file tests .

   [ -f emp.csv ] ; echo $? ------------------------- An ordinary file 
   0 ----------- yes
   [ -x emp.csv ] ; echo $? -------------------------- An executable file
   echo " The number of arguments specified is $#"
   1 ------------- No
   [ ! -w emp.csv ] | | echo " False that file is not writable "     

The ! negates a test so [ ! -w emp.csv ] negates [ -w emp.csv ] . Lets create a script using these features which will reads the user given file and tells us whether files exists , whether it is readable or writable .

   #!/bin/ksh 
   #filetest.ksh : To check file attributes
   if [ ! -e $1 ] ; then
   echo "File Does not exist "
   elseif [ ! -r $1 ] ; then
   echo " File is not readable "
   elseif [ ! -w $1 ] ; then
   echo " File is not writable "
   else
   echo " File is both readable and writable"
   fi   
Options Operation
-f file file exists and is a regular file
-r file file exists and is readable
-w file file exists and is writable
-x file file exists and is executable
-d files files exists and is a directory
-s file file exists and has a size greater than zero
-e file file exists ( Korn and Bash only )
-u file file exists and has SUID bit set
-k file file exists and has sticky bit set
-L file file exists and is a symbolic link ( Korn and Bash only )
f1 -nt f2 f1 is newer than f2 ( Korn and Bash only )
f1 -ot f2 f1 is older thanf2 ( Korn and Bash only )
f1 -ef f2 f1 is linked to f2 ( Korn and Bash only )

Case Statements :

The case statement is the second form of conditional statement offereed by shell . The general syntax of the case statement is as follows :

   case expression in 
   pattern 1 ) Command 1 ;;
   pattern 2 ) Command 2 ;;
   pattern 3 ) Command 3 ;;
   ...........
   esac

case first matches expression with pattern1 . If the match succeeds , then it executes command1 ( which may be one or more command ) .If the match fails , then pattern2 is matched and so on .Each command list is terminated with pair of semicolons ( ;; ) and entire construct is closed with esac ( reverse of case ) .

Lets write a menu driven script which takes values from 1 to 3 and perform some action depending upon the number keyed in . The three menu choices are displayed with a multi line echo statement .

   #!/bin/ksh 
   #usrmenu.ksh :: Menu Driven Program using Switch case statement
   echo " ..... MENU....."
   echo " 1. List The Files from the directory "
   echo " 2. User Logged in to the system "
   echo " 3. today's Date "
   echo " Enter Your choice "
   read option
   case "$option" in
   1) ls -l ;;
   2) who ;;
   3) date ;;
   *) echo "Invalid option"
   esac   

We can even use command substitution in case statement like this -

case ` who | grep "mangesh" in `

Matching Multiple Patterns : 

case can be also be used for matching multiple patterns . For instance , the expression y | Y can be used to match y in both upper and lowercase .

   echo " Do you wish to continue ? ( y/n ) : " 
   read ans
   case "$ans" in
   y | Y ) ;;
   n | N ) exit ;;
   esac

We can even use the wil cards in case statement for matching characters in case statement - we will see an example how it can be used 

expr : Computation and String handling -

The bourne shell doesn't have any computing features though it can check whether an integer is greater than another .Bourn shell use external expr command for computatuion . It performs following operations .

  • Performs arithematic operations on integers
  • Manipulates String

If we use Korn or Bash we have better ways of handling these strings , but we need to understand this if we have to debug someone's script which has expr .

Computation -

expr can perform the four basic operations as well as the modulus ( remainder ) functions :

   a=5 b=6 
   expr 6+5
   11
   expr $b-$a
   1
   expr $a \* $b --------------------- * ( Aestrik ) has to be escaped
   30
   expr $b / $a
   1 ---------------------------------- Decimal output truncated
   expr 15%4
   3

The operand +,-,* etc must be enclosed on either side by whitespace .The multiplication operand (*) has to be escaped to prevent the shell interpreting as filename metacharacter . expr can handle only integers , division yields only the integer part .

expr can be used with command substitution to assign a variable .

   i=8 j=2 ; k= `expr $i + $j ` 
   echo "$k"
   10

String Handling :

For manipulating strings , expr uses two expressions separated by a colon . The string to be worked upon is placed on the left of the : and a regular expression is placed on its right . Depending on the combination of the expression , expr can perform three important string functions :

  • Deterine the length of the string
  • Extract a sub string
  • Locate the position of a character in a string

Length of a String

The regular expression .* signifies to expr that it has to print the number of characters matching the pattern . i.e the length of the entire string

   expr "ijklmnopqrstu" : '.*' ------------------------ space on either side of : is required 
   13

This feature is useful in validating data entry e.g if a string entered by the user should not be more than 15 characters long then we can use above logic in if statement and do the validation .

Extracting a Subtring

expr can extract a string enclosed by the escaped characters \ ( and \ ) . If you wish to extract the 2 - digit year from a 4 - digit string , you must create a pattern group and extract in this way :

   str=2015 
   expr "$str" : '..\(..\)

It signifies that the first two characters in the value of $str have to be ignored and two characters have to be extracted from the third character position .

locating position of a character : 

expr can also return the location of the first occurence of a character inside a string . to locate the position of the character p in the string value of $str , we have to count the number of characters which are not p ([^p]*) followed by a d

str="ijklmnopqrstu" ; expr "$str" : '[^p]*p'

while loop -

Shell offers three types of loops - while , until and for . All of them repeat the instruction set enclosed by certain keywords as often as their control command permits .

Most of us are familiar with while loop and its usage . It repetaedly perform a set of instructions until the control command return a true exit status . The general syntax of this command is as follows

   while condition is true 
   do
   commands
   done

The commands enclosed by do and done are executed repeatedly as long as condition remains true .

We will write a simple program which will write some details in a text file keyed by the user . The program will continue to run until user says no for further operations . Lets see how the code looks and works with while loop .

   #!/bin/ksh 
   #loop.ksh -- Program with while loop
   opt=y #-------we must declare this to enter first time in whiile loop
   while [ "$opt" = "y" ]
   do
   echo "enter the empid and Name : "
   read id name
   echo "$id | $name " >> Emp.txt
   echo " Enter any more ( y / n ) "
   read opt
   case $opt in
   y* | Y* ) opt=y ;; # Also accepts yes, YES etc .
   n*|N*) opt=n ;; # Also accepts no ,No etc
   *) answer=y ;; # any other reply means y
   esac
   done

wait in while loops -

Sometime we explicitly want to pause aur program or loop for few seconds or minutes to get input from some other location or for execution purpose . We can do this by putting the sleep command . In the above script if you want to make a pause of few seconds before the loop asks for second input - here is the command to do that .

sleep 60 -------------- Sleeps for 60 seconds

Setting up an Infinite loop -

You can set up the infinite while loop in this way

   while true ; do 
   df -t
   sleep 120
   done

For loop -

The shell's for loop use different structure than used in other programming language .There is no three -part structure used typically in other programming language . Instead it uses list , here is the syntax how for loop in shell works -

   for variable in list ; 
   do
   commands
   done

The loop body also uses the keywords do and done , but the additional parameters here are variable and list .Each whitespace separated word in list is assigned to variable in turn and commands are executed until list is exhausted . Lets see a simple example which will give us more clear understanding of things :

   for file in tes1.html test2.html test3.html test4.html ; do 
   cp $file ${file}.bak
   echo $file copied to $file.bak
   done
   do
   commands
   done

The list here comprises a series of character strings separated by whitespace . Each item in the list is assigned to the variable file . file first get the value test1.html , then test2.html nas so on .Each file is copied with a.bak extension and the completion message dispalyed after every file is copied .

Some other sources of the list -

for is important loop in Shell scripting - so it is good to understand the loop thoroughly . Here are some more sources that we can use in list of for loop -

List from variables -

 We can use a series of variables in the command line .They are evaluated by the shell before executing the loop :

for var in $PATH $HOME $EMAIL ; do echo $var ; done

We have to provide the semicolons at the right places if you want to enter the entire loop in a single line .The ouput represnts the three environment variables of shell .

List from command substitution - 

We can also use the command substitution to create the list . In our first for loop example we have provided four files as list - now lets store the filenames in a file list.txt and we will use this file in for loop as command substitution . Here is the example for it -

   for file in `cat list.txt ` ; 
   do echo $file
   done 

This is the best way when the list is large . It is also a good arrangement because you can change the list without having to change the script .

List from wild-cards - 

for loop permits wild cards and interpret them as filenames . Lets write the first for loop example with wild cards . for loop prints all the files having .html extension from current directory .

   for file in *.html ; 
   do echo $file
   done 

List from positional Parameters - 

for is alos used to process positional paramters that are assigned from command line argument . The following example uses the shell $@ parameter to represent all command line arguments . Exceute the script by passing three arguments , one should be mutliword string . We will use the emp.csv file as an example .

   #!/bin/ksh 
   # Script with for loop using positional parameters
   for param in "$@" ; 
   do grep "$param" emp.csv | | echo "Pattern $param not found"
   done

basename : Changing filename Extensions -

basename is yet another external commands - which is most effective when used with for loop . When basename is used with two arguments , it strips off the second argument from the first argument

basenmae list.txt txt ---------------------------- txt will be stripped off

We can now use this option when you want to change the .html file to .txt using for loop

   for file in *.html ; do 
   org_name=`basename $file html `
   mv $file ${org_name}txt 
   done

This will convert all the .html files to .txt in a single stroke .

set and shift : Manipulating Positional Parameters -

 Some Unix / Linux commands like date produce single -line output . We also pass command output through filters like grep and head to produce a single line .Recall that we had to use an external command (cut)to extract a field from the date output

date | cut -d " " -f1

To overcome this ; shell has an internal command to do this job - the set statement . set assigns its arguments to the positional parameters $1 , $2 and so on . This feature is especially useful for picking up individual fields from the output of a program . But before we do that , let's use set to convert its argument to positional parameter .

set 9971 2878 8800

This assigns the value 9971 to the positional parameter $1 , 2878 to $2 and 8800 to $3 . It also sets $# and $* . You can verify by echoing each parameter

   echo " \$1 is $1 , \$2 is $2 and \$3 is $3 
   echo " The $# arguments are $* "

We will now use command substitution to extract individual fields from hte date output and without using cut

   set `date` 
   echo " $* "
   echo " The date today is $2 $3 , $6 "

The day of the week is avialble in $1 .

shift : Shifting Arguments to left - 

shift transfers the contents of positional parameter to its immediate lower numbered one . This is done as many times as the statement is called. When called once , $2 becomes $1 , $3 becomes $2 and so on . Try the below positional parameter that were previously filled up with date .

   set `date` 
   echo " $@ "
   echo $1 $2 $3
   shift
   echo $1 $2 $3
   shift 2 ------------------------- Shift 2 places
   echo $1 $2 $3

Note that the contents of the leftmost parameter , $1 are lost every time shift is invoked . In this way , we can access $10 by first shifting it and converting it to $9.

set -- Helps Command substitution -

we often need to use set with command substitution . There is a small problem thhough, especially when the output of the command begins with a :

   set `ls -l test.txt` 
   -rw-r--r-- : bad option(s)   

Since the permission string begins with a - (regualr files) , set interprets it as an option and finds it to be a "bad"one .set creates another problem when its arguments evaluate to null string . Consider this command :

set `grep PPP /etc/passwd`

If the PPP can't be located in the file , set will operate with no arguments and puzzle the user by dispalying all variables on the terminal (its default output ) ! . The solution to both these problems lies in the use of -- ( two hyphens ) immediately after set :

   set -- `ls -l test.txt` 
   set -- `grep PPP /etc/passwd`   

set now understands that the arguments following -- are not to betreated as option . The two hyphens also direct set to suppress its default behavior if its arguments evaluate to a null string .

set -x : Debugging Shell Scripts -

Apart from assigning values to positional parametetrs , set serves as a useful debugging tool withbits -x option . When used inside ascript ( or even at the $ prompt ) , it echoes each statement on the terminal , precceded by a + as it is executed . modify any previous script to turn on the set option by placing the following statement at the begining of the script .

       #!/bin/ksh 
       #********* Source Code From Website - Mangadaku - visit us at -http://mangadaku.com/ *****      
       # To determine whether given number is odd or even
       set -x
       echo " Enter The Number "
       read num
       
       num_mod=`expr $num%2`
       if [[ $num_mod -eq 0 ]]; then
       
       echo " The Number $num is an even number "
       else 
       
       echo " The number $num is an odd number "
       fi
       
OUTPUT : 
       +echo " Enter The Number "
        Enter The Number
       +read num 
        5
       +num_mod=`expr 5%2`
       +if [[ 5 -eq 0 ]]
       +echo " The number 5 is an odd number "
       The number 5 is an odd number 

set +x turns off set -x and you can place latter statement at the end of the script . This is what we will see when we invoke the script < >>

This is an ideal tool to use if you have trouble fiinding out why scripts don't work in the manner expected . Note how the shell prints each statement as it is being executed , affixing a + to each .

 

The HERE Documents ( << ) -

There are occassions when the data your program reads is fixed and fairly limited . The shell uses the << symbol to read data from the same file containing the script . This is referred to as a here document , signifyig that the data is here rather than in a separate file . Any command using standard input can also take input from a here document .

Lets see a simple example of mailx command using here document .

   mailx mangesh << MARK 
   Your prorgram for printing the invoices has been completed 
   on `date` . Please check the printout .
   MARK 

The here document symbol ( << ) is followed by two lines of data and a delimiter ( the string MARK ) . The shell treats every line following the command and delimited by MARK as input to the command . mangesh at the other end will see the two line of message text with the date inserted by command substitution . The word MARK itself doesn't show up . When this sequence is placed insdie a script , execution is faster because mailx doesn't have to read an external file

Here Document with interactive Programms -

We have written and seen many programms which takes input from user . Often its a same input that is keyed in , in response to series of questions passed by the command .Lets see below script whcih asks for 2 arguments , the word or string to be searched and file from where user want to search string . We can make this script to work non interactively by supplying the inputs through a here document :

   $ test.ksh << END
   > director
   > emp.csv
   > END 
   Enter the pattern to be searched : Enter the file to be used : Searching for director from file emp.csv 
   9876 | Jai Sharma      | director  |production | 12/03/65|7000
   2365 | barun sengupta  |director   |personnel  | 11/05/55|8500

Even though the prompts are displayed in a single line , the important thing is that the script worked . We have been able to run an interactive script noninteractively !

Untitled Document Scroll To Top Untitled Document