Shell scripts

8. Shell scripts#

What actually makes a shell script:

Put a group of commands in a file, then make the file executable.
In the executable file, apply the shell syntax to add variables, conditionals, iterations.
Create functions for the frequently used blocks.
Utilize awk, grep, and sed languages with regular expressions for string parsing and substitution.

8.1. Shell scripting exercises#

Create a directory for shell scripting exercises.
A shell script and executable permissions.
if and case statements
Loops with while and until conditional statements.
for loop.
Functions in shell scripts.
awk programming language.
grep line search filter and regular expressions.
String editor sed
Advantages and disadvantages of shell scripts.

References: for details, consult Bash Guide for Beginners

8.2. Creating a shell script#

ssh to your lxc container.

Create a new directory bash_scripts where you will run the shell scripting exercises.

mkdir  bash_scripts
cd bash_scripts

Note, it is recommended to create a new script file for every exercise; make them executable; give them names with extension .sh – it is just a convention rather than a must. The first line of a bash script starts with

#!/bin/bash

All non-executable comments in a script are prepended with #

File content:

# For example, we list all the files in the current directory
ls -la
ls -l /etc  # Comment. Here, we list files in /etc directory

8.3. Variables, read, echo, assignment, substitution.#

Shell scripts contain variables. Variables are created by assignment operator, for example:

DIR=/home/hostadm

Variable value is referenced by $ sign:

echo $DIR

Variables can be updated via assignment operator:

DIR=$DIR/KVM
echo $DIR

Variables also can be read from stdo (keyboard) by using command read.

Script scr1.sh is like a calculator:

File content:

#!/bin/bash
echo "I will work out X*Y"
echo "Enter X"
read X
echo "Enter Y"
read Y
echo "X*Y = $X*$Y = $[X*Y]"

Make the script executable and run

chmod 755 scr1.sh
./scr1.sh

8.4. `if` conditional statements#

File content:

#!/bin/bash
    X=10
    Y=5
    if [ "$X" -gt "$Y" ]; then
            echo "$X is greater than $Y"
    elif [ "$X" -lt "$Y"]; then
            echo "$X is less than $Y"
    else
            echo "$X is equal to $Y"
    fi

The test square brackets [ ] mean test for a true or a false statement.

File content:

if [ condition ]
then
  perform a task at the condition = true
else
  perform a task at the condition = false
fi

For example

File content:

#!/bin/bash

echo 'Enter the parameter value:'
read parameter

if [ "$parameter" == 'this' ]
then
   echo $parameter
else
   echo 'Does not match'
fi

There are various cases for [ ] test in bash. For example, if directory DIR1 exists:

   if [-d "DIR1" ]

If file ttx.txt exists:

   if [-e "ttx.txt" ]

You can run command man to see various predefined cases for tests:

man test

8.5. `case` conditional statements#

File content:

#!/bin/bash
case $1 in
    --test|-t)
            echo "you used the --test option"
            exit 0
    ;;
    --help|-h)
            echo "Usage:"
            echo "        myprog.sh [--test|--help|--version]"
            exit 0
    ;;
    --version|-v)
            echo "myprog.sh version 0.0.1"
            exit 0
    ;;
    *)
            echo "No such option $1"
            echo "Usage:"
            echo "        myprog.sh [--test|--help|--version]"
            exit 1
    ;;
esac
echo "You typed \"$1\" on the command-line"

Note, always watch for correct syntax of case statement:

File content:

case string
in
  regex1)
  commands1
   ;;
  regex2)
  commands2
   ;;
 ........
esac

Where regex is a regular expression to match the string. To catch all remaining strings, use *) at the end.

8.6. Variable `$1`#

Variables $1, $2, $3, … take the first, second, third, … inpit parameters for the script. For example,

script.sh --help

Where $1 will pickup string --help as the value inside script.sh

8.7. Looping with `while` and `until` statements#

Script scr2.sh:

File content:

#!/bin/bash
N=1
while [ "$N" -le "10" ]
do
        echo "Number $N"
        N=$[N+1]
done

Script scr3.sh

File content:

#!/bin/bash
N=1
until [ "$N" -gt "10" ]
do
    echo "Number $N"; N=$[N+1]
done

Note, common mistakes in shell scripting are usually due to incorrect syntax. For example, there should be no spaces before and after operator “=”

File content:

N=1          # correct
N =1         # error
N= 1         # error
N=$[N+1]     # correct
N =$[N+1]    # error
N= $[N+1]    # error

8.8. Looping with `for` statement#

Script scr4.sh

File content:

#!/bin/bash
for i in red white blue
do
        echo "$i is a color"
done

Script backup-lots.sh

File content:

#!/bin/bash
for i in 0 1 2 3 4 5 6 7 8 9 ; do
        cp $1 $1.BAK-$i
done

The enumeration in the loop can be represented by a conscize statement, {0..9}:

File content:

#!/bin/bash
for i in {0..9} ; do
        cp $1 $1.BAK-$i
done

Now create a file important_data with some numbers in it and then run

./backup-lots.sh important_data

which will copy the file 10 times with 10 different extensions. As you can see, the variable $1 has a special meaning – it is the first argument on the command-line. Note, watch for correct syntax:

File content:

for i in {0..9}
do
....
done

another correct alternative:

File content:

for i in {0..9}  ; do
....
done

Now modify the script:

File content:

#!/bin/bash
if [ "$1" = "" ] ; then
        echo "Usage: backup-lots.sh <filename>"
        exit
fi
for i in {0..9} ; do
        NEW_FILE=$1.BAK-$i
        if [ -e $NEW_FILE ]; then
                echo "backup-lots.sh: **warning** $NEW_FILE"
                echo "                already exists - skipping"
        else
                cp $1 $NEW_FILE
        fi
done

Looping over glob expressions

File content:

#!/bin/bash
for i in *.txt ; do
        echo "found a file:" $i
done

Note, the script above loops over all *.txt files in the current directory so you need to create several files with .txt extension in this exercise.

File content:

#!/bin/bash
for i in /usr/share/*/*.txt ; do
        echo "found a file:" $i
done

Breaking out of the loops and continuing

File content:

#!/bin/bash
for i in {0..9} ; do
        NEW_FILE=$1.BAK-$i
        if [ -e $NEW_FILE ]; then
                echo "backup-lots.sh: **error** $NEW_FILE"
                echo "                already exists - exitting"
                break
        else
                cp $1 $NEW_FILE
        fi
done

break command causes program execution to continue on the line after the done.

The continue statement in the loop block is useful for terminating the current iteration of the loop.

File content:

#!/bin/bash
for i in {0..9} ; do
        NEW_FILE=$1.BAK-$i
        if [ -e $NEW_FILE ] ; then
                echo "backup-lots.sh: **warning** $NEW_FILE"
                echo "                already exists - skipping"
                continue
        fi
        cp $1 $NEW_FILE
done

8.9. functions#

Function definitions provide a way to group statement blocks into one.

File content:

#!/bin/bash
function usage ()
{
        echo "Usage:"
        echo "        myprog.sh [--test|--help|--version]"
}
case $1 in
        --test|-t)
                echo "you used the --test option"
                exit 0
        ;;
        --help|-h)
                usage
        ;;
        --version|-v)
                echo "myprog.sh version 0.0.2"
                exit 0
        ;;
        -*)
                echo "Error: no such option $1"
                usage
                exit 1
        ;;
esac
echo "You typed \"$1\" on the command-line"

Note, watch for syntax:

File content:

function usage ()
{
   command1
   command2; command3
   ......
}

The word function in a function is optional. That is, the following will work as well:

File content:

usage ()
{
   command1
   command2; command3
   ......
}

8.10. Using quotes#

Single forward quotes ' protect the enclosed text from the shell.

echo 'error $?'
echo 'shell name $0'

Double quotes " allow all shell interpretations to take place inside them.

echo "error $?" #gives the error code of the last command
echo "shell name $0" #gives the current shell name

Command substitution

X=`expr 100 + 50 '*' 3`
echo $X

Assigning command output to a variable:

FSIZE=`wc -l /etc/profile`

same as

FSIZE=$(wc -l /etc/profile)

8.11. Introduction to `awk`#

The basic function of awk is to search files for lines or other text units containing one or more patterns. When a line matches one of the patterns, special actions are performed on that line.

Display user names from /etc/passwd (field 1):

awk -F: '{ print $1 }' /etc/passwd

Where F is the field separator in the passwd file. The fields are separated by :
Default field separator is a blank space. Awk scans the input file and splits each input line into fields.
Similarly:

cat /etc/passwd | awk -F: '{ print $1 }'

Display user names home directories and login shell (fields 1 and 7), and store them in a separate file, users.txt

awk -F: '{ print $1, $6, $7 }' /etc/passwd > users.txt

or

cat /etc/passwd | awk -F: '{ print $1, $6, $7 }' > users.txt

` Default field separator is empty space. To print users (field 1) from just created file users.txt:

awk '{ print $1 }' users.txt

8.12. Introduction to `grep`#

grep is used to search files or standard input for lines containing required patterns.

We’ll work with a text file, list.txt, containing the following text:

File content:

Check the inode list today
reboot the machine tomorrow
  Reboot it again in a week
Call Tech support in case of emergency.
tel: 834

Oop 0
Oops 1
Oopss 12
Oopsss 123
Oopssss  1234
End

To get the line containing string “inode” in file list.txt:

grep inode  list.txt

To get the line containing “inode lis “ in file list.txt:

grep "inode lis " list.txt

It should give you nothing as there is no string “ lis “

To search for the line containing “inode list” in all the files in current directory:

grep "inode list" *

Syntax of grep:

grep [options] regex [files]

where regex are regular expressions.

Using regular expressions
Regular expressions: Literals (plain text or literal text), metacharacters (special meaning characters). When you construct regular expressions, you use metacharacters and literals to specify three basic ideas about your input text: position anchors, groups, ranges and quantity modifiers.

Anchors:

^  -match at the beginning of a line
$  -match at the end of a line

grep '^Call' list.txt
grep '^  Reboot' list.txt
grep 'today$' list.txt

Count the number of empty lines:

grep -c '^$' list.txt

Display all lines containing only word End by itself:

grep '^End$' list.txt

Groups and ranges:

[abc]   -matches any single character from a,b or c
[a-e]   -matches any single charcter from among the range a-e
[^abc]  -inverse match, matches a single character not among a,b, or c.
[^a-e]  -inverse match, matches a single character not from the range a-e
\< word\>    -matches word (option -w can be used instead)
. (single dot) -matches any single character among a new line
\      -turn off the special meaning of the character that follows

grep '[Rr]eboot' list.txt
grep '\<[Rr]eboot\>' list.txt

Display all lines from file list.txt which contain thre adjucent digits:

grep '[0-9][0-9][0-9]' list.txt

Display the lines with four or more characters in the line:

grep '....' list.txt

Display all non-blank lines from file list.txt:

grep '.' list.txt

Display all lines that contain a period:

grep '\.' list.txt

Modifiers:

*  -matches zero or more instance of the preceding single character
?  -matches zero or one  instance of the preceding regex (implies 'grep -E' option).
+  -matches one or more  instance of the preceding regex (implies 'grep -E' option).
\{n,m\} -matches a range of occurrences of the single character or regex that precedes this construct; \{n\} matches n occurences;
\{n,\} matches at least n occurences.
|  -matches either the regex specified before or after the vertical bar (implies 'grep -E' option).

Display all lines from list.txt that contain Oop, Oops, Oopss, and so on:

grep 'Oops*' list.txt

Display all lines from list.txt that contain Oops, Oopss, and so on:

grep 'Oopss*' list.txt

Display all lines from list.txt that contain two or more adjacent digits:

grep '[0-9][0-9][0-9]*' list.txt

Display all lines from list.txt that contain ‘3’ or ‘34’ number combination:

grep -E '34?' list.txt

Display all lines from list.txt containing at least one digit:

grep -E '[0-9]+' list.txt

Display all lines from list.txt containing sss and ssss:

grep 's\{3,4\}' list.txt

Display all lines from list.txt containing any three, four or five digit numbers:

grep '[0-9]\{3,5\}' list.txt

Display all lines from list.txt containing “Reboot”, “reboot” or “support” strings:

grep -E '[Rr]eboot|support' list.txt

Display all lines from list.txt containing any letter (no empty lines):

grep '[A-Za-z]' list.txt

Display all lines from list.txt containing any non alpha-numeric and space symbol:

grep '[^ 0-9A-Za-z]' list.txt

Display all lines from list.txt containing uppercase letter, followed by zero or more lowercase letters:

grep '[A-Z][a-z]*' list.txt

Display all lines from list.txt containing 3 digit telephone number:

grep 'tel: [0-9]\{3\}' list.txt

8.13. Introduction to `sed`#

String editor, sed, is used for editing lines in a file or a stream; output is going to the standard output and can be re-directed to a new file. Syntax:

sed [options] 'command1' [files]
sed [options] -e 'command1' [-e command2 ...] [files]
sed [options] -f script [files]

Delete lines from 3 through 5 in file list.txt:

sed '3,5d' list.txt

Delete lines that contain “O” at the beginning of the line:

sed '/^O/d' list.txt

Translate capital C,R,O into small c,r,o:

sed 'y/CRO/cro/' list.txt

Delete ampty lines:

sed '/^$/d' list.txt

Replace string Oop with Wee for the first occurence on a line

sed 's/Oop/Wee/' list.txt

Remove ss string (replace with empty entry)for the first occurence on a line:

sed 's/ss//' list.txt

Remove ss string for all occurences on a line:

sed 's/ss//g' list.txt

Substitute a single space for any number of spaces wherever they occur on the line:

sed 's/ */ /g' list.txt

Substitute underscore for any number of spaces wherever they occur on the line:

sed 's/ */_/g' list.txt

More examples: Useful one-line scripts for sed

Shell scripts

Contents

8. Shell scripts#

8.1. Shell scripting exercises#

8.2. Creating a shell script#

8.3. Variables, read, echo, assignment, substitution.#

8.4. if conditional statements#

8.5. case conditional statements#

8.6. Variable $1#

8.7. Looping with while and until statements#

8.8. Looping with for statement#

8.9. functions#

8.10. Using quotes#

8.11. Introduction to awk#

8.12. Introduction to grep#

8.13. Introduction to sed#

8.4. `if` conditional statements#

8.5. `case` conditional statements#

8.6. Variable `$1`#

8.7. Looping with `while` and `until` statements#

8.8. Looping with `for` statement#

8.11. Introduction to `awk`#

8.12. Introduction to `grep`#

8.13. Introduction to `sed`#