Shell Scripts

6. Shell Scripts#

What actually makes a shell script:

Put a group of commands in a file, then make the file executable.
In the executable file, apply the shell syntax to add variables, conditionals, iterations.
Create functions for the frequently used blocks.
Utilize awk, grep, and sed languages with regular expressions for string parsing and substitution.

6.1. Shell Scripting Exercises#

Create a directory for shell scripting exercises.
A shell script and executable permissions.
if and case statements.
Loops with while and until conditional statements.
for loop.
Functions in shell scripts.
awk programming language.
grep line search filter and regular expressions.
String editor sed.
Advantages and disadvantages of shell scripts.

References: for details, consult the Bash Guide for Beginners

6.2. Creating a Shell Script#

SSH into your LXC container.

Create a new directory named bash_scripts where you will run the shell scripting exercises:

mkdir bash_scripts
cd bash_scripts

Note: It is recommended to create a new script file for every exercise, make them executable, and give them names ending with the .sh extension (this is a convention rather than a strict requirement). The first line of a Bash script starts with a shebang:

#!/bin/bash

All non-executable comments in a script are prepended with a hash symbol (#):

File content:

# For example, we list all the files in the current directory
ls -la
ls -l /etc  # Comment. Here, we list files in the /etc directory

6.3. Variables: Read, Echo, Assignment, and Substitution#

Shell scripts contain variables. Variables are created using an assignment operator, for example:

DIR=/home/hostadm

Variable values are referenced using the $ sign:

echo $DIR

Variables can be updated via the assignment operator:

DIR=$DIR/KVM
echo $DIR

Variables can also be read from standard output (or keyboard input) by using the read command.

For example, the script scr1.sh acts like a calculator:

File content:

#!/bin/bash
echo "I will work out X*Y"
echo "Enter X"
read X
echo "Enter Y"
read Y
echo "X*Y = $X*$Y = $[X*Y]"

Make the script executable and run it:

chmod 755 scr1.sh
./scr1.sh

6.4. Variable `$1`#

The variables $1, $2, $3, and so on, capture the first, second, third, and subsequent input parameters passed to the script. For example:

script.sh --help

Here, $1 captures the string --help as a value inside script.sh.

6.5. Variables with Commands#

We can assign a command to a variable and then execute it within the shell. For example, define the variable cmd and execute it:

cmd="ls /etc"

eval $cmd

Example of a more complex statement: tar the /etc directory and copy the resulting tar file into a new directory named DIR1:

my_dir=etc
cmd="tar -cf ${my_dir}.tar /${my_dir}; mkdir DIR1; cp ${my_dir}.tar DIR1"
eval $cmd

6.6. `if` Conditional Statements#

The test square brackets [ ] evaluate whether a statement is true or false.

File content:

if [ condition ]
then
  perform a task if condition = true
else
  perform a task if condition = false
fi

For example:

File content:

#!/bin/bash

echo 'Enter the parameter value:'
read parameter

if [ "$parameter" == 'this' ]
then
   echo $parameter
else
   echo 'Does not match'
fi

Another example using nested conditions:

File content:

#!/bin/bash
    X=10
    Y=5
    if [ "$X" -gt "$Y" ]; then
            echo "$X is greater than $Y"
    elif [ "$X" -lt "$Y" ]; then
            echo "$X is less than $Y"
    else
            echo "$X is equal to $Y"
    fi

There are various conditional tests available in Bash. For example, to check if the directory DIR1 exists:

   if [ -d "DIR1" ]

To check if the file ttx.txt exists:

   if [ -e "ttx.txt" ]

You can run the man command to view various predefined test options:

man test

6.7. Conditional Command Sequences#

Execute command2 only if command1 completes successfully:

command1 && command2

Example:

mkdir testdir
ls testdir && echo "testdir exists"

Alternatively:

[ -d testdir ] && echo "testdir exists"

Conversely, execute command2 only if command1 fails:

command1 || command2

Example:

ls testdir_2 || echo "testdir_2 doesn't exist"

Alternatively:

[ -d testdir_2 ] || echo "testdir_2 doesn't exist"

6.8. `case` Conditional Statements#

File content:

case $variable in
  regex1)
    commands1
    ;;
  regex2)
    commands2
    ;;
  *)
    commands3
    ;;
esac

Where regex is a regular expression used to match the variable’s value. Use *) at the end to catch all remaining unmatched strings.

File content:

#!/bin/bash
case $1 in
    --test|-t)
            echo "you used the --test option"
            exit 0
    ;;
    --help|-h)
            echo "Usage:"
            echo "        myprog.sh [--test|--help|--version]"
            exit 0
    ;;
    --version|-v)
            echo "myprog.sh version 0.0.1"
            exit 0
    ;;
    *)
            echo "No such option $1"
            echo "Usage:"
            echo "        myprog.sh [--test|--help|--version]"
            exit 1
    ;;
esac
echo "You typed "$1" on the command-line"

Note: Always pay close attention to the correct syntax of case statements.

6.9. Looping with `while` and `until` Statements#

Script scr2.sh:

File content:

#!/bin/bash
N=1
while [ "$N" -le "10" ]
do
        echo "Number $N"
        N=$[N+1]
done

Script scr3.sh:

File content:

#!/bin/bash
N=1
until [ "$N" -gt "10" ]
do
    echo "Number $N"
    N=$[N+1]
done

Note: Common mistakes in shell scripting are usually due to incorrect syntax. For instance, there must be no spaces before and after the = assignment operator:

File content:

N=1          # correct
N =1         # error
N= 1         # error
N=$[N+1]     # correct
N =$[N+1]    # error
N= $[N+1]    # error

6.10. Looping with the `for` Statement#

Script scr4.sh:

File content:

#!/bin/bash
for i in red white blue
do
        echo "$i is a color"
done

Script backup-lots.sh:

File content:

#!/bin/bash
for i in 0 1 2 3 4 5 6 7 8 9 ; do
        cp $1 $1.BAK-$i
done

The enumeration in the loop can be represented more concisely using brace expansion, {0..9}:

File content:

#!/bin/bash
for i in {0..9} ; do
        cp $1 $1.BAK-$i
done

Now create a file named important_data with some numbers in it, and then run:

./backup-lots.sh important_data

This will copy the file 10 times with 10 different extensions. As you can see, the variable $1 has a special meaning: it represents the first argument passed on the command line. Watch out for correct syntax:

File content:

for i in {0..9}
do
....
done

Another correct alternative:

File content:

for i in {0..9}  ; do
....
done

Now modify the script:

File content:

#!/bin/bash
if [ "$1" = "" ] ; then
        echo "Usage: backup-lots.sh <filename>"
        exit
fi
for i in {0..9} ; do
        NEW_FILE=$1.BAK-$i
        if [ -e $NEW_FILE ]; then
                echo "backup-lots.sh: **warning** $NEW_FILE"
                echo "                already exists - skipping"
        else
                cp $1 $NEW_FILE
        fi
done

Looping over glob expressions:

File content:

#!/bin/bash
for i in *.txt ; do
        echo "found a file:" $i
done

Note that the script above loops over all .txt files in the current directory, so you will need to create several files with a .txt extension to test this exercise.

File content:

#!/bin/bash
for i in /usr/share/*/*.txt ; do
        echo "found a file:" $i
done

Breaking out of loops and continuing:

File content:

#!/bin/bash
for i in {0..9} ; do
        NEW_FILE=$1.BAK-$i
        if [ -e $NEW_FILE ]; then
                echo "backup-lots.sh: **error** $NEW_FILE"
                echo "                already exists - exiting"
                break
        else
                cp $1 $NEW_FILE
        fi
done

The break command causes program execution to continue on the line immediately following the done statement.

The continue statement is useful for terminating the current iteration of a loop and proceeding to the next:

File content:

#!/bin/bash
for i in {0..9} ; do
        NEW_FILE=$1.BAK-$i
        if [ -e $NEW_FILE ] ; then
                echo "backup-lots.sh: **warning** $NEW_FILE"
                echo "                already exists - skipping"
                continue
        fi
        cp $1 $NEW_FILE
done

6.11. Functions#

Function definitions provide a way to group statement blocks into a single reusable unit:

File content:

function usage ()
{
   command1
   command2; command3
   ......
}

The word function is optional. The following syntax works as well:

File content:

usage ()
{
   command1
   command2; command3
   ......
}

For example, create a script named myprog.sh with the following content:

File content:

#!/bin/bash
function usage ()
{
        echo "Usage:"
        echo "        myprog.sh [--test|--help|--version]"
}
case $1 in
        --test|-t)
                echo "you used the --test option"
                exit 0
        ;;
        --help|-h)
                usage
        ;;
        --version|-v)
                echo "myprog.sh version 0.0.2"
                exit 0
        ;;
        -*)
                echo "Error: no such option $1"
                usage
                exit 1
        ;;
esac
echo "You typed "$1" on the command-line"

6.12. Using Quotes#

Single forward quotes (') protect the enclosed text from shell interpretation:

echo 'error $?'
echo 'shell name $0'

Double quotes (") allow all shell interpretations and variable expansions to take place:

echo "error $?" # outputs the error code of the last command
echo "shell name $0" # outputs the current shell name

Command substitution:

X=`expr 100 + 50 '*' 3`
echo $X

Assigning command output to a variable:

FSIZE=`wc -l /etc/profile`

which is equivalent to:

FSIZE=$(wc -l /etc/profile)

6.13. Introduction to `awk`#

The basic function of awk is to search files for lines or text units containing one or more patterns. When a line matches a pattern, specified actions are performed on it. awk is very frequently used to extract specific columns from text based on matching criteria.

awk reads data from a file or a pipe using the following syntax:

awk 'EXPRESSION { PROGRAM }' file(s)

or

COMMAND | awk 'EXPRESSION { PROGRAM }'

To display usernames from /etc/passwd (field 1):

awk -F: '{ print $1 }' /etc/passwd

Where -F: specifies the field separator in the passwd file. The fields are separated by colons (:).
The default field separator in awk is a space or tab. awk scans the input file and splits each input line into fields accordingly.
Similarly:

cat /etc/passwd | awk -F: '{ print $1 }'

To display usernames, home directories, and login shells (fields 1, 6, and 7) and store them in a separate file named users.txt:

awk -F: '{ print $1, $6, $7 }' /etc/passwd > users.txt

or

cat /etc/passwd | awk -F: '{ print $1, $6, $7 }' > users.txt

Since the default field separator becomes an empty space when reading users.txt, we can print usernames from it using:

awk '{ print $1 }' users.txt

To extract only the columns of lines containing a specific pattern, such as hostadm:

awk '/hostadm/ { print $1 }' users.txt

We can use regular expressions inside the selection blocks (/ /). For example, to view permissions on all files in the /etc directory with a .conf extension:

ls -l /etc | awk '/.*\.conf/ { print $1, $9 }'

In the print command block, we can include custom strings for formatted output:

ls -l /etc | awk '/.*\.conf/ { print "Permission:", $1, "File:",  $9 }'

Adding a conditional statement, such as printing files larger than 5000 bytes:

ls -l /etc | awk '/.*\.conf/  { if ( $5 > 5000 ) print "Permission:", $1, "Size", $5, "File:", $9 }'

BEGIN and END blocks allow you to print statements before and after the main awk processing output:

ls -l /etc | awk 'BEGIN { print "Conf Files > 5K found:\n" } /.*\.conf/ { if ( $5 > 5000 ) print "Permission:", $1, "Size", $5, "File:", $9 } END { print "\n ---- The end -------" }'

6.14. Introduction to `grep`#

grep is used to search files or standard input for lines containing required patterns.

We will work with a text file named list.txt containing the following content:

File content:

Check the inode list today
reboot the machine tomorrow
  Reboot it again in a week
Call Tech support in case of emergency.
tel: 834

Oop 0
Oops 1
Oopss 12
Oopsss 123
Oopssss  1234
End

To find lines containing the string “inode” in list.txt:

grep inode list.txt

To find lines containing “inode lis “ in list.txt:

grep "inode lis " list.txt

This will return nothing because that exact string does not exist in the file.

To search for lines containing “inode list” across all files in the current directory:

grep "inode list" *

The syntax for grep is:

grep [options] regex [files]

where regex represents regular expressions.

6.14.1. Using Regular Expressions#

Regular expressions consist of literals (plain text) and metacharacters (special control characters). They allow you to specify three core concepts: position anchors, groups and ranges, and quantity modifiers.

Anchors:

^  - matches at the beginning of a line
$  - matches at the end of a line

grep '^Call' list.txt
grep '^  Reboot' list.txt
grep 'today$' list.txt

To count the number of empty lines:

grep -c '^$' list.txt

To display all lines containing only the word “End” by itself:

grep '^End$' list.txt

Groups and ranges:

[abc]   - matches any single character from a, b, or c
[a-e]   - matches any single character within the range a to e
[^abc]  - inverse match; matches any single character except a, b, or c
[^a-e]  - inverse match; matches any single character outside the range a to e
\<word\> - matches the exact word (option -w can also be used)
.       - matches any single character except a newline
\       - turns off the special meaning of the following character

grep '[Rr]eboot' list.txt
grep '\<[Rr]eboot\>' list.txt

Display all lines from list.txt containing three adjacent digits:

grep '[0-9][0-9][0-9]' list.txt

Display lines containing four or more consecutive characters:

grep '....' list.txt

Display all non-blank lines from list.txt:

grep '.' list.txt

Display all lines containing a literal period:

grep '\.' list.txt

Modifiers:

*        - matches zero or more instances of the preceding single character
?        - matches zero or one instance of the preceding regex (implies the 'grep -E' option)
+        - matches one or more instances of the preceding regex (implies the 'grep -E' option)
\{n,m\}  - matches a range of occurrences; \{n\} matches exactly n occurrences; \{n,\} matches at least n occurrences
|        - matches either the regex specified before or after the vertical bar (implies the 'grep -E' option)

Display all lines from list.txt containing “Oop”, “Oops”, “Oopss”, etc.:

grep 'Oops*' list.txt

Display all lines containing “Oops”, “Oopss”, etc.:

grep 'Oopss*' list.txt

Display lines containing two or more adjacent digits:

grep '[0-9][0-9][0-9]*' list.txt

Display lines containing the number ‘3’ or the combination ‘34’:

grep -E '34?' list.txt

Display lines containing at least one digit:

grep -E '[0-9]+' list.txt

Display lines containing “sss” and “ssss”:

grep 's\{3,4\}' list.txt

Display lines containing any 3, 4, or 5-digit numbers:

grep '[0-9]\{3,5\}' list.txt

Display lines containing “Reboot”, “reboot”, or “support”:

grep -E '[Rr]eboot|support' list.txt

Display lines containing any letter (no empty lines):

grep '[A-Za-z]' list.txt

Display lines containing any non-alphanumeric and non-space symbol:

grep '[^ 0-9A-Za-z]' list.txt

Display lines containing an uppercase letter followed by zero or more lowercase letters:

grep '[A-Z][a-z]*' list.txt

Display lines containing a 3-digit telephone number:

grep 'tel: [0-9]\{3\}' list.txt

6.15. Introduction to `sed`#

The stream editor, sed, is used for editing lines in a file or a stream. Its output goes to standard output and can be redirected to a new file. Syntax:

sed [options] 'command1' [files]
sed [options] -e 'command1' [-e command2 ...] [files]
sed [options] -f script [files]

To print lines 3 through 5 from list.txt:

sed -n '3,5p' list.txt

To delete lines 3 through 5 from list.txt:

sed '3,5d' list.txt

We can use regular expressions enclosed in / / to target specific blocks for commands.

For example, to delete lines starting with “O”:

sed '/^O/d' list.txt

To delete empty lines:

sed '/^$/d' list.txt

To translate capital letters C, R, and O into lowercase c, r, and o:

sed 'y/CRO/cro/' list.txt

To replace the first occurrence of “Oop” with “Wee” on each line:

sed 's/Oop/Wee/' list.txt

To remove the string “ss” (replace with an empty entry) for the first occurrence on a line:

sed 's/ss//' list.txt

To remove all occurrences of “ss” on a line:

sed 's/ss//g' list.txt

To substitute a single space for any sequence of multiple spaces on a line:

sed 's/ */ /g' list.txt

To substitute an underscore for any sequence of multiple spaces:

sed 's/ */_/g' list.txt

To make sed modify the original file content directly, use the -i (in-place) option. For example, to apply changes to list2.txt:

cp list.txt list2.txt
sed -i 's/ */_/g' list2.txt

More examples: Useful one-line scripts for sed

6.16. Printing Lines Containing a Matching Pattern Using `grep`, `awk`, and `sed`#

The following three commands all print out the exact same matching line from list.txt:

grep Reboot list.txt

awk '/Reboot/ { print $0 }' list.txt

sed -n '/Reboot/p' list.txt

Shell Scripts

Contents

6. Shell Scripts#

6.1. Shell Scripting Exercises#

6.2. Creating a Shell Script#

6.3. Variables: Read, Echo, Assignment, and Substitution#

6.4. Variable $1#

6.5. Variables with Commands#

6.6. if Conditional Statements#

6.7. Conditional Command Sequences#

6.8. case Conditional Statements#

6.9. Looping with while and until Statements#

6.10. Looping with the for Statement#

6.11. Functions#

6.12. Using Quotes#

6.13. Introduction to awk#

6.14. Introduction to grep#

6.14.1. Using Regular Expressions#

6.15. Introduction to sed#

6.16. Printing Lines Containing a Matching Pattern Using grep, awk, and sed#

6.4. Variable `$1`#

6.6. `if` Conditional Statements#

6.8. `case` Conditional Statements#

6.9. Looping with `while` and `until` Statements#

6.10. Looping with the `for` Statement#

6.13. Introduction to `awk`#

6.14. Introduction to `grep`#

6.15. Introduction to `sed`#

6.16. Printing Lines Containing a Matching Pattern Using `grep`, `awk`, and `sed`#