9. Shell scripts

What actually makes a shell script:

  • Put a group of commands in a file, then make the file executable.

  • In the executable file, apply the shell syntax to add variables, conditionals, iterations.

  • Create functions for the frequently used blocks.

  • Utilize awk, grep, and sed languages with regular expressions for string parsing and substitution.

9.1. Shell scripting exercises

  • Create a directory for shell scripting exercises.

  • A shell script and executable permissions.

  • if and case statements

  • Loops with while and until conditional statements.

  • for loop.

  • Functions in shell scripts.

  • awk programming language.

  • grep line search filter and regular expressions.

  • String editor sed

References: for details, consult Bash Guide for Beginners


9.2. Creating a shell script

SSH to the LXC container and create a new directory bash_scripts where you will run the shell scripting exercises.

mkdir  bash_scripts
cd bash_scripts

Note, it is recommended to create a new script file for every exercise; make them executable; give them names with extension .sh – it is just a convention rather than a must. The first line of a bash script starts with

#!/bin/bash

All non-executable comments in a script are prepended with #

File content:

# For example, we list all the files in the current directory
ls -la
ls -l /etc  # Comment. Here, we list files in /etc directory

Script scr1.sh is like a calculator:

File content:

#!/bin/bash
echo "I will work out X*Y"
echo "Enter X"
read X
echo "Enter Y"
read Y
echo "X*Y = $X*$Y = $[X*Y]"

Make the script executable and run

chmod 755 scr1.sh
./scr1.sh

9.3. if conditional statements

File content:

#!/bin/bash
    X=10
    Y=5
    if [ "$X" -gt "$Y" ]; then
            echo "$X is greater than $Y"
    elif [ "$X" -lt "$Y"]; then
            echo "$X is less than $Y"
    else
            echo "$X is equal to $Y"
    fi
  • The test square brackets [ ] mean test for a true or a false statement.

File content:

if [ condition ]
then
  perform a task at the condition = true
else
  perform a task at the condition = false
fi

For example

File content:

#!/bin/bash

echo 'Enter the parameter value:'
read parameter

if [ "$parameter" == 'this' ]
then
   echo $parameter
else
   echo 'Does not match'
fi

There are various cases for [ ] test in bash. For example, if directory DIR1 exists:

   if [-d "DIR1" ]

If file ttx.txt exists:

   if [-e "ttx.txt" ]

You can run command man to see various predefined cases for tests:

man test

9.4. case conditional statements

File content:

#!/bin/bash
case $1 in
    --test|-t)
            echo "you used the --test option"
            exit 0
    ;;
    --help|-h)
            echo "Usage:"
            echo "        myprog.sh [--test|--help|--version]"
            exit 0
    ;;
    --version|-v)
            echo "myprog.sh version 0.0.1"
            exit 0
    ;;
    *)
            echo "No such option $1"
            echo "Usage:"
            echo "        myprog.sh [--test|--help|--version]"
            exit 1
    ;;
esac
echo "You typed \"$1\" on the command-line"

Note, always watch for correct syntax of case statement:

File content:

case string
in
  regex1)
  commands1
   ;;
  regex2)
  commands2
   ;;
 ........
esac

Where regex is a regular expression to match the string. To catch all remaining strings, use *) at the end.


9.5. Looping with while and until statements

Script scr2.sh:

File content:

#!/bin/bash
N=1
while [ "$N" -le "10" ]
do
        echo "Number $N"
        N=$[N+1]
done

Script scr3.sh

File content:

#!/bin/bash
N=1
until [ "$N" -gt "10" ]
do
    echo "Number $N"; N=$[N+1]
done

Note, common mistakes in shell scripting are usually due to incorrect syntax. For example, there should be no spaces before and after operator “=”

File content:

N=1          # correct
N =1         # error
N= 1         # error
N=$[N+1]     # correct
N =$[N+1]    # error
N= $[N+1]    # error

9.6. Looping with for statement

Script scr4.sh

File content:

#!/bin/bash
for i in red white blue
do
        echo "$i is a color"
done

Script backup-lots.sh

File content:

#!/bin/bash
for i in 0 1 2 3 4 5 6 7 8 9 ; do
        cp $1 $1.BAK-$i
done

The enumeration in the loop can be represented by a conscize statement, {0..9}:

File content:

#!/bin/bash
for i in {0..9} ; do
        cp $1 $1.BAK-$i
done

Now create a file important_data with some numbers in it and then run

./backup-lots.sh important_data

which will copy the file 10 times with 10 different extensions. As you can see, the variable $1 has a special meaning – it is the first argument on the command-line. Note, watch for correct syntax:

File content:

for i in {0..9}
do
....
done

another correct alternative:

File content:

for i in {0..9}  ; do
....
done

Now modify the script:

File content:

#!/bin/bash
if [ "$1" = "" ] ; then
        echo "Usage: backup-lots.sh <filename>"
        exit
fi
for i in {0..9} ; do
        NEW_FILE=$1.BAK-$i
        if [ -e $NEW_FILE ]; then
                echo "backup-lots.sh: **warning** $NEW_FILE"
                echo "                already exists - skipping"
        else
                cp $1 $NEW_FILE
        fi
done

Looping over glob expressions

File content:

#!/bin/bash
for i in *.txt ; do
        echo "found a file:" $i
done

Note, the script above loops over all *.txt files in the current directory so you need to create several files with .txt extension in this exercise.

File content:

#!/bin/bash
for i in /usr/share/*/*.txt ; do
        echo "found a file:" $i
done

Breaking out of the loops and continuing

File content:

#!/bin/bash
for i in {0..9} ; do
        NEW_FILE=$1.BAK-$i
        if [ -e $NEW_FILE ]; then
                echo "backup-lots.sh: **error** $NEW_FILE"
                echo "                already exists - exitting"
                break
        else
                cp $1 $NEW_FILE
        fi
done

which causes program execution to continue on the line after the done.

The continue statement is useful for terminating the current iteration of the loop.

File content:

#!/bin/bash
for i in {0..9} ; do
        NEW_FILE=$1.BAK-$i
        if [ -e $NEW_FILE ] ; then
                echo "backup-lots.sh: **warning** $NEW_FILE"
                echo "                already exists - skipping"
                continue
        fi
        cp $1 $NEW_FILE
done

9.7. functions

Function definitions provide a way to group statement blocks into one.

File content:

#!/bin/bash
function usage ()
{
        echo "Usage:"
        echo "        myprog.sh [--test|--help|--version]"
}
case $1 in
        --test|-t)
                echo "you used the --test option"
                exit 0
        ;;
        --help|-h)
                usage
        ;;
        --version|-v)
                echo "myprog.sh version 0.0.2"
                exit 0
        ;;
        -*)
                echo "Error: no such option $1"
                usage
                exit 1
        ;;
esac
echo "You typed \"$1\" on the command-line"

Note, watch for syntax:

File content:

function usage ()
{
   command1
   command2; command3
   ......
}

The word function in a function is optional. That is, the following will work as well:

File content:

usage ()
{
   command1
   command2; command3
   ......
}

9.8. Using quotes

Single forward quotes ' protect the enclosed text from the shell.

echo 'error $?'
echo 'shell name $0'

Double quotes " allow all shell interpretations to take place inside them.

echo "error $?" #gives the error code of the last command
echo "shell name $0" #gives the current shell name

Command substitution

X=`expr 100 + 50 '*' 3`
echo $X

Assigning command output to a variable:

FSIZE=`wc -l /etc/profile`

same as

FSIZE=$(wc -l /etc/profile)

9.9. Introduction to awk

The basic function of awk is to search files for lines or other text units containing one or more patterns. When a line matches one of the patterns, special actions are performed on that line.

Display user names from /etc/passwd (field 1):

awk -F: '{ print $1 }' /etc/passwd

Where F is the field separator in the passwd file. The fields are separated by :
Default field separator is a blank space. Awk scans the input file and splits each input line into fields.
Similarly:

cat /etc/passwd | awk -F: '{ print $1 }'

Display user names home directories and login shell (fields 1 and 7), and store them in a separate file, users.txt

awk -F: '{ print $1, $6, $7 }' /etc/passwd > users.txt

or

cat /etc/passwd | awk -F: '{ print $1, $6, $7 }' > users.txt

` Default field separator is empty space. To print users (field 1) from just created file users.txt:

awk '{ print $1 }' users.txt

Recommended tutorial: The GNU awk programming language


9.10. Introduction to grep

grep is used to search files or standard input for lines containing required patterns.

We’ll work with a text file, list.txt, containing the following text:

File content:

Check the inode list today
reboot the machine tomorrow
  Reboot it again in a week
Call Tech support in case of emergency.
tel: 834

Oop 0
Oops 1
Oopss 12
Oopsss 123
Oopssss  1234
End

To get the line containing string “inode” in file list.txt:

grep inode  list.txt

To get the line containing “inode lis ” in file list.txt:

grep "inode lis " list.txt

It should give you nothing as there is no string ” lis “

To search for the line containing “inode list” in all the files in current directory:

grep "inode list" *

Syntax of grep:

grep [options] regex [files]

where regex are regular expressions.

Using regular expressions
Regular expressions: Literals (plain text or literal text), metacharacters (special meaning characters). When you construct regular expressions, you use metacharacters and literals to specify three basic ideas about your input text: position anchors, groups, ranges and quantity modifiers.

Anchors:

^  -match at the beginning of a line
$  -match at the end of a line
grep '^Call' list.txt
grep '^  Reboot' list.txt
grep 'today$' list.txt

Count the number of empty lines:

grep -c '^$' list.txt

Display all lines containing only word End by itself:

grep '^End$' list.txt

Groups and ranges:

[abc]   -matches any single character from a,b or c
[a-e]   -matches any single charcter from among the range a-e
[^abc]  -inverse match, matches a single character not among a,b, or c.
[^a-e]  -inverse match, matches a single character not from the range a-e
\< word\>    -matches word (option -w can be used instead)
. (single dot) -matches any single character among a new line
\      -turn off the special meaning of the character that follows
grep '[Rr]eboot' list.txt
grep '\<[Rr]eboot\>' list.txt

Display all lines from file list.txt which contain thre adjucent digits:

grep '[0-9][0-9][0-9]' list.txt

Display the lines with four or more characters in the line:

grep '....' list.txt

Display all non-blank lines from file list.txt:

grep '.' list.txt

Display all lines that contain a period:

grep '\.' list.txt

Modifiers:

*  -matches zero or more instance of the preceding single character
?  -matches zero or one  instance of the preceding regex (implies 'grep -E' option).
+  -matches one or more  instance of the preceding regex (implies 'grep -E' option).
\{n,m\} -matches a range of occurrences of the single character or regex that precedes this construct; \{n\} matches n occurences;
\{n,\} matches at least n occurences.
|  -matches either the regex specified before or after the vertical bar (implies 'grep -E' option).

Display all lines from list.txt that contain Oop, Oops, Oopss, and so on:

grep 'Oops*' list.txt

Display all lines from list.txt that contain Oops, Oopss, and so on:

grep 'Oopss*' list.txt

Display all lines from list.txt that contain two or more adjacent digits:

grep '[0-9][0-9][0-9]*' list.txt

Display all lines from list.txt that contain ‘3’ or ‘34’ number combination:

grep -E '34?' list.txt

Display all lines from list.txt containing at least one digit:

grep -E '[0-9]+' list.txt

Display all lines from list.txt containing sss and ssss:

grep 's\{3,4\}' list.txt

Display all lines from list.txt containing any three, four or five digit numbers:

grep '[0-9]\{3,5\}' list.txt

Display all lines from list.txt containing “Reboot”, “reboot” or “support” strings:

grep -E '[Rr]eboot|support' list.txt

Display all lines from list.txt containing any letter (no empty lines):

grep '[A-Za-z]' list.txt

Display all lines from list.txt containing any non alpha-numeric and space symbol:

grep '[^ 0-9A-Za-z]' list.txt

Display all lines from list.txt containing uppercase letter, followed by zero or more lowercase letters:

grep '[A-Z][a-z]' list.txt

Display all lines from list.txt containing 3 digit telephone number:

grep 'tel: [0-9]\{3\}' list.txt

Recommended tutorial on Regular expressions and grep


9.11. Introduction to sed

String editor, sed, is used for editing lines in a file or a stream; output is going to the standard output and can be re-directed to a new file. Syntax:

sed [options] 'command1' [files]
sed [options] -e 'command1' [-e command2 ...] [files]
sed [options] -f script [files]

Delete lines from 3 through 5 in file list.txt:

sed '3,5d' list.txt

Delete lines that contain “O” at the beginning of the line:

sed '/^O/d' list.txt

Translate capital C,R,O into small c,r,o:

sed 'y/CRO/cro/' list.txt

Delete ampty lines:

sed '/^$/d' list.txt

Replace string Oop with Wee for the first occurence on a line

sed 's/Oop/Wee/' list.txt

Remove ss string (replace with empty entry)for the first occurence on a line:

sed 's/ss//' list.txt

Remove ss string for all occurences on a line:

sed 's/ss//g' list.txt

Substitute a single space for any number of spaces wherever they occur on the line:

sed 's/ */ /g' list.txt

Substitute underscore for any number of spaces wherever they occur on the line:

sed 's/ */_/g' list.txt

More examples: Useful one-line scripts for sed

Recommended tutorial: The GNU sed stream editor