Shell scripting exercises
Contents
8. Shell scripting exercises¶
Create a directory for shell scripting exercises.
A shell script and executable permissions.
ifandcasestatementsLoops with
whileanduntilconditional statements.forloop.Functions in shell scripts.
awkprogramming language.grepline search filter and regular expressions.String editor
sed
References: for details, consult Bash Guide for Beginners
8.1. Creating a shell script¶
Create a new VM, scripts by cloning kvm1:
virt-clone -o kvm1 -n scripts -f KVM/scripts.qcow2
Fix the hostname and machine id.
Create a new directory bash_scripts where you will run the shell scripting exercises.
mkdir bash_scripts
cd bash_scripts
Note, it is recommended to create a new script file for every exercise; make them executable; give them names with extension .sh – it is just a convention rather than a must. The first line of a bash script starts with
#!/bin/bash
All non-executable comments in a script are prepended with #
File content:
# For example, we list all the files in the current directory
ls -la
ls -l /etc # Comment. Here, we list files in /etc directory
Script scr1.sh is like a calculator:
File content:
#!/bin/bash
echo "I will work out X*Y"
echo "Enter X"
read X
echo "Enter Y"
read Y
echo "X*Y = $X*$Y = $[X*Y]"
Make the script executable and run
chmod 755 scr1.sh
./scr1.sh
8.2. if and case statements¶
File content:
#!/bin/bash
X=10
Y=5
if [ "$X" -gt "$Y" ]; then
echo "$X is greater than $Y"
elif [ "$X" -lt "$Y"]; then
echo "$X is less than $Y"
else
echo "$X is equal to $Y"
fi
File content:
#!/bin/bash
case $1 in
--test|-t)
echo "you used the --test option"
exit 0
;;
--help|-h)
echo "Usage:"
echo " myprog.sh [--test|--help|--version]"
exit 0
;;
--version|-v)
echo "myprog.sh version 0.0.1"
exit 0
;;
*)
echo "No such option $1"
echo "Usage:"
echo " myprog.sh [--test|--help|--version]"
exit 1
;;
esac
echo "You typed \"$1\" on the command-line"
Note, always watch for correct syntax of case statement:
File content:
case string
in
regex1)
commands1
;;
regex2)
commands2
;;
........
esac
Where regex is a regular expression to match the string. To catch all remaining strings, use *) at the end.
8.3. Looping with while and until statements¶
Script scr2.sh:
File content:
#!/bin/bash
N=1
while [ "$N" -le "10" ]
do
echo "Number $N"
N=$[N+1]
done
Script scr3.sh
File content:
#!/bin/bash
N=1
until [ "$N" -gt "10" ]
do
echo "Number $N"; N=$[N+1]
done
Note, common mistakes in shell scripting are usually due to incorrect syntax. For example, there should be no spaces before and after operator “=”
File content:
N=1 # correct
N =1 # error
N= 1 # error
N=$[N+1] # correct
N =$[N+1] # error
N= $[N+1] # error
8.4. Looping with for statement¶
Script scr4.sh
File content:
#!/bin/bash
for i in red white blue
do
echo "$i is a color"
done
Script backup-lots.sh
File content:
#!/bin/bash
for i in 0 1 2 3 4 5 6 7 8 9 ; do
cp $1 $1.BAK-$i
done
Now create a file important_data with some numbers in it and then run
./backup-lots.sh important_data
which will copy the file 10 times with 10 different extensions. As you can see, the variable $1 has a special meaning – it is the first argument on the command-line. Note, watch for correct syntax:
File content:
for i in 0 1 2 3 4 5 6 7 8 9
do
....
done
another correct alternative:
File content:
for i in 0 1 2 3 4 5 6 7 8 9 ; do
....
done
Now modify the script:
File content:
#!/bin/bash
if [ "$1" = "" ] ; then
echo "Usage: backup-lots.sh <filename>"
exit
fi
for i in 0 1 2 3 4 5 6 7 8 9 ; do
NEW_FILE=$1.BAK-$i
if [ -e $NEW_FILE ]; then
echo "backup-lots.sh: **warning** $NEW_FILE"
echo " already exists - skipping"
else
cp $1 $NEW_FILE
fi
done
Looping over glob expressions
File content:
#!/bin/bash
for i in *.txt ; do
echo "found a file:" $i
done
Note, the script above loops over all *.txt files in the current directory so you need to create several files with .txt extension in this exercise.
File content:
#!/bin/bash
for i in /usr/share/*/*.txt ; do
echo "found a file:" $i
done
Breaking out of the loops and continuing
File content:
#!/bin/bash
for i in 0 1 2 3 4 5 6 7 8 9 ; do
NEW_FILE=$1.BAK-$i
if [ -e $NEW_FILE ]; then
echo "backup-lots.sh: **error** $NEW_FILE"
echo " already exists - exitting"
break
else
cp $1 $NEW_FILE
fi
done
which causes program execution to continue on the line after the done.
The continue statement is useful for terminating the current iteration of the loop.
File content:
#!/bin/bash
for i in 0 1 2 3 4 5 6 7 8 9 ; do
NEW_FILE=$1.BAK-$i
if [ -e $NEW_FILE ] ; then
echo "backup-lots.sh: **warning** $NEW_FILE"
echo " already exists - skipping"
continue
fi
cp $1 $NEW_FILE
done
8.5. functions¶
Function definitions provide a way to group statement blocks into one.
File content:
#!/bin/bash
function usage ()
{
echo "Usage:"
echo " myprog.sh [--test|--help|--version]"
}
case $1 in
--test|-t)
echo "you used the --test option"
exit 0
;;
--help|-h)
usage
;;
--version|-v)
echo "myprog.sh version 0.0.2"
exit 0
;;
-*)
echo "Error: no such option $1"
usage
exit 1
;;
esac
echo "You typed \"$1\" on the command-line"
Note, watch for syntax:
File content:
function usage ()
{
command1
command2; command3
......
}
The word function in a function is optional. That is, the following will work as well:
File content:
usage ()
{
command1
command2; command3
......
}
8.6. Using quotes¶
Single forward quotes ' protect the enclosed text from the shell.
echo 'error $?'
echo 'shell name $0'
Double quotes " allow all shell interpretations to take place inside them.
echo "error $?" #gives the error code of the last command
echo "shell name $0" #gives the current shell name
Command substitution
X=`expr 100 + 50 '*' 3`
echo $X
Assigning command output to a variable:
FSIZE=`wc -l /etc/profile`
same as
FSIZE=$(wc -l /etc/profile)
8.7. Introduction to awk¶
The basic function of awk is to search files for lines or other text units containing one or more patterns. When a line matches one of the patterns, special actions are performed on that line.
Display user names from /etc/passwd (field 1):
awk -F: '{ print $1 }' /etc/passwd
Where F is the field separator in the passwd file. The fields are separated by :
Default field separator is a blank space. Awk scans the input file and splits each input line into fields.
Similarly:
cat /etc/passwd | awk -F: '{ print $1 }'
Display user names home directories and login shell (fields 1 and 7), and store them in a separate file, users.txt
awk -F: '{ print $1, $6, $7 }' /etc/passwd > users.txt
or
cat /etc/passwd | awk -F: '{ print $1, $6, $7 }' > users.txt
` Default field separator is empty space. To print users (field 1) from just created file users.txt:
awk '{ print $1 }' users.txt
Recommended tutorial: The GNU awk programming language
8.8. Introduction to grep¶
grep is used to search files or standard input for lines containing required patterns.
We’ll work with a text file, list.txt, containing the following text:
File content:
Check the inode list today
reboot the machine tomorrow
Reboot it again in a week
Call Tech support in case of emergency.
tel: 834
Oop 0
Oops 1
Oopss 12
Oopsss 123
Oopssss 1234
End
To get the line containing string “inode” in file list.txt:
grep inode list.txt
To get the line containing “inode lis ” in file list.txt:
grep "inode lis " list.txt
It should give you nothing as there is no string ” lis “
To search for the line containing “inode list” in all the files in current directory:
grep "inode list" *
Syntax of grep:
grep [options] regex [files]
where regex are regular expressions.
Using regular expressions
Regular expressions: Literals (plain text or literal text),
metacharacters (special meaning characters).
When you construct regular expressions, you use metacharacters and literals to specify three basic ideas about your input text: position anchors, groups, ranges and quantity modifiers.
Anchors:
^ -match at the beginning of a line
$ -match at the end of a line
grep '^Call' list.txt
grep '^ Reboot' list.txt
grep 'today$' list.txt
Count the number of empty lines:
grep -c '^$' list.txt
Display all lines containing only word End by itself:
grep '^End$' list.txt
Groups and ranges:
[abc] -matches any single character from a,b or c
[a-e] -matches any single charcter from among the range a-e
[^abc] -inverse match, matches a single character not among a,b, or c.
[^a-e] -inverse match, matches a single character not from the range a-e
\< word\> -matches word
. (single dot) -matches any single character among a new line
\ -turn off the special meaning of the character that follows
grep '[Rr]eboot' list.txt
grep '\<[Rr]eboot\>' list.txt
Display all lines from file list.txt which contain thre adjucent digits:
grep '[0-9][0-9][0-9]' list.txt
Display the lines with four or more characters in the line:
grep '....' list.txt
Display all non-blank lines from file list.txt:
grep '.' list.txt
Display all lines that contain a period:
grep '\.' list.txt
Modifiers:
* -matches zero or more instance of the preceding single character
? -matches zero or one instance of the preceding regex (implies 'grep -E' option).
+ -matches one or more instance of the preceding regex (implies 'grep -E' option).
\{n,m\} -matches a range of occurrences of the single character or regex that precedes this construct; \{n\} matches n occurences;
\{n,\} matches at least n occurences.
| -matches either the regex specified before or after the vertical bar (implies 'grep -E' option).
Display all lines from list.txt that contain Oop, Oops, Oopss, and so on:
grep 'Oops*' list.txt
Display all lines from list.txt that contain Oops, Oopss, and so on:
grep 'Oopss*' list.txt
Display all lines from list.txt that contain two or more adjacent digits:
grep '[0-9][0-9][0-9]*' list.txt
Display all lines from list.txt that contain ‘3’ or ‘34’ number combination:
grep -E '34?' list.txt
Display all lines from list.txt containing at least one digit:
grep -E '[0-9]+' list.txt
Display all lines from list.txt containing sss and ssss:
grep 's\{3,4\}' list.txt
Display all lines from list.txt containing any three, four or five digit numbers:
grep '\<[0-9]\{3,5\}\>' list.txt
Display all lines from list.txt containing “Reboot”, “reboot” or “support” strings:
grep -E '[Rr]eboot|support' list.txt
Display all lines from list.txt containing any letter (no empty lines):
grep '[A-Za-z]' list.txt
Display all lines from list.txt containing any non alpha-numeric and space symbol:
grep '[^ 0-9A-Za-z]' list.txt
Display all lines from list.txt containing uppercase letter, followed by zero or more lowercase letters:
grep '[A-Z][a-z]' list.txt
Display all lines from list.txt containing 3 digit telephone number:
grep 'tel: [0-9]\{3\}' list.txt
Recommended tutorial on Regular expressions and grep
8.9. Introduction to sed¶
String editor, sed, is used for editing lines in a file or a stream; output is going to the standard output and can be re-directed to a new file.
Syntax:
sed [options] 'command1' [files]
sed [options] -e 'command1' [-e command2 ...] [files]
sed [options] -f script [files]
Delete lines from 3 through 5 in file list.txt:
sed '3,5d' list.txt
Delete lines that contain “O” at the beginning of the line:
sed '/^O/d' list.txt
Translate capital C,R,O into small c,r,o:
sed 'y/CRO/cro/' list.txt
Delete ampty lines:
sed '/^$/d' list.txt
Replace string Oop with Wee for the first occurence on a line
sed 's/Oop/Wee/' list.txt
Remove ss string (replace with empty entry)for the first occurence on a line:
sed 's/ss//' list.txt
Remove ss string for all occurences on a line:
sed 's/ss//g' list.txt
Substitute a single space for any number of spaces wherever they occur on the line:
sed 's/ */ /g' list.txt
Substitute underscore for any number of spaces wherever they occur on the line:
sed 's/ */_/g' list.txt