Shell scripts
Contents
8. Shell scripts¶
What actually makes a shell script:
Put a group of commands in a file, then make the file executable.
In the executable file, apply the shell syntax to add variables, conditionals, iterations.
Create functions for the frequently used blocks.
Utilize
awk,grep, andsedlanguages with regular expressions for string parsing and substitution.
8.1. Shell scripting exercises¶
Create a directory for shell scripting exercises.
A shell script and executable permissions.
ifandcasestatementsLoops with
whileanduntilconditional statements.forloop.Functions in shell scripts.
awkprogramming language.grepline search filter and regular expressions.String editor
sedAdvantages and disadvantages of shell scripts.
References: for details, consult Bash Guide for Beginners
8.2. Creating a shell script¶
ssh to your lxc container.
Create a new directory bash_scripts where you will run the shell scripting exercises.
mkdir bash_scripts
cd bash_scripts
Note, it is recommended to create a new script file for every exercise; make them executable; give them names with extension .sh – it is just a convention rather than a must. The first line of a bash script starts with
#!/bin/bash
All non-executable comments in a script are prepended with #
File content:
# For example, we list all the files in the current directory
ls -la
ls -l /etc # Comment. Here, we list files in /etc directory
8.3. Variables, read, echo, assignment, substitution.¶
Shell scripts contain variables. Variables are created by assignment operator, for example:
DIR=/home/hostadm
Variable value is referenced by $ sign:
echo $DIR
Variables can be updated via assignment operator:
DIR=$DIR/KVM
echo $DIR
Variables also can be read from stdo (keyboard) by using command read.
Script scr1.sh is like a calculator:
File content:
#!/bin/bash
echo "I will work out X*Y"
echo "Enter X"
read X
echo "Enter Y"
read Y
echo "X*Y = $X*$Y = $[X*Y]"
Make the script executable and run
chmod 755 scr1.sh
./scr1.sh
8.4. if conditional statements¶
File content:
#!/bin/bash
X=10
Y=5
if [ "$X" -gt "$Y" ]; then
echo "$X is greater than $Y"
elif [ "$X" -lt "$Y"]; then
echo "$X is less than $Y"
else
echo "$X is equal to $Y"
fi
The test square brackets [ ] mean test for a true or a false statement.
File content:
if [ condition ]
then
perform a task at the condition = true
else
perform a task at the condition = false
fi
For example
File content:
#!/bin/bash
echo 'Enter the parameter value:'
read parameter
if [ "$parameter" == 'this' ]
then
echo $parameter
else
echo 'Does not match'
fi
There are various cases for [ ] test in bash. For example, if directory DIR1 exists:
if [-d "DIR1" ]
If file ttx.txt exists:
if [-e "ttx.txt" ]
You can run command man to see various predefined cases for tests:
man test
8.5. case conditional statements¶
File content:
#!/bin/bash
case $1 in
--test|-t)
echo "you used the --test option"
exit 0
;;
--help|-h)
echo "Usage:"
echo " myprog.sh [--test|--help|--version]"
exit 0
;;
--version|-v)
echo "myprog.sh version 0.0.1"
exit 0
;;
*)
echo "No such option $1"
echo "Usage:"
echo " myprog.sh [--test|--help|--version]"
exit 1
;;
esac
echo "You typed \"$1\" on the command-line"
Note, always watch for correct syntax of case statement:
File content:
case string
in
regex1)
commands1
;;
regex2)
commands2
;;
........
esac
Where regex is a regular expression to match the string. To catch all remaining strings, use *) at the end.
8.6. Variable $1¶
Variables $1, $2, $3, … take the first, second, third, … inpit parameters for the script. For example,
script.sh --help
Where $1 will pickup string --help as the value inside script.sh
8.7. Looping with while and until statements¶
Script scr2.sh:
File content:
#!/bin/bash
N=1
while [ "$N" -le "10" ]
do
echo "Number $N"
N=$[N+1]
done
Script scr3.sh
File content:
#!/bin/bash
N=1
until [ "$N" -gt "10" ]
do
echo "Number $N"; N=$[N+1]
done
Note, common mistakes in shell scripting are usually due to incorrect syntax. For example, there should be no spaces before and after operator “=”
File content:
N=1 # correct
N =1 # error
N= 1 # error
N=$[N+1] # correct
N =$[N+1] # error
N= $[N+1] # error
8.8. Looping with for statement¶
Script scr4.sh
File content:
#!/bin/bash
for i in red white blue
do
echo "$i is a color"
done
Script backup-lots.sh
File content:
#!/bin/bash
for i in 0 1 2 3 4 5 6 7 8 9 ; do
cp $1 $1.BAK-$i
done
The enumeration in the loop can be represented by a conscize statement, {0..9}:
File content:
#!/bin/bash
for i in {0..9} ; do
cp $1 $1.BAK-$i
done
Now create a file important_data with some numbers in it and then run
./backup-lots.sh important_data
which will copy the file 10 times with 10 different extensions. As you can see, the variable $1 has a special meaning – it is the first argument on the command-line. Note, watch for correct syntax:
File content:
for i in {0..9}
do
....
done
another correct alternative:
File content:
for i in {0..9} ; do
....
done
Now modify the script:
File content:
#!/bin/bash
if [ "$1" = "" ] ; then
echo "Usage: backup-lots.sh <filename>"
exit
fi
for i in {0..9} ; do
NEW_FILE=$1.BAK-$i
if [ -e $NEW_FILE ]; then
echo "backup-lots.sh: **warning** $NEW_FILE"
echo " already exists - skipping"
else
cp $1 $NEW_FILE
fi
done
Looping over glob expressions
File content:
#!/bin/bash
for i in *.txt ; do
echo "found a file:" $i
done
Note, the script above loops over all *.txt files in the current directory so you need to create several files with .txt extension in this exercise.
File content:
#!/bin/bash
for i in /usr/share/*/*.txt ; do
echo "found a file:" $i
done
Breaking out of the loops and continuing
File content:
#!/bin/bash
for i in {0..9} ; do
NEW_FILE=$1.BAK-$i
if [ -e $NEW_FILE ]; then
echo "backup-lots.sh: **error** $NEW_FILE"
echo " already exists - exitting"
break
else
cp $1 $NEW_FILE
fi
done
break command causes program execution to continue on the line after the done.
The continue statement in the loop block is useful for terminating the current iteration of the loop.
File content:
#!/bin/bash
for i in {0..9} ; do
NEW_FILE=$1.BAK-$i
if [ -e $NEW_FILE ] ; then
echo "backup-lots.sh: **warning** $NEW_FILE"
echo " already exists - skipping"
continue
fi
cp $1 $NEW_FILE
done
8.9. functions¶
Function definitions provide a way to group statement blocks into one.
File content:
#!/bin/bash
function usage ()
{
echo "Usage:"
echo " myprog.sh [--test|--help|--version]"
}
case $1 in
--test|-t)
echo "you used the --test option"
exit 0
;;
--help|-h)
usage
;;
--version|-v)
echo "myprog.sh version 0.0.2"
exit 0
;;
-*)
echo "Error: no such option $1"
usage
exit 1
;;
esac
echo "You typed \"$1\" on the command-line"
Note, watch for syntax:
File content:
function usage ()
{
command1
command2; command3
......
}
The word function in a function is optional. That is, the following will work as well:
File content:
usage ()
{
command1
command2; command3
......
}
8.10. Using quotes¶
Single forward quotes ' protect the enclosed text from the shell.
echo 'error $?'
echo 'shell name $0'
Double quotes " allow all shell interpretations to take place inside them.
echo "error $?" #gives the error code of the last command
echo "shell name $0" #gives the current shell name
Command substitution
X=`expr 100 + 50 '*' 3`
echo $X
Assigning command output to a variable:
FSIZE=`wc -l /etc/profile`
same as
FSIZE=$(wc -l /etc/profile)
8.11. Introduction to awk¶
The basic function of awk is to search files for lines or other text units containing one or more patterns. When a line matches one of the patterns, special actions are performed on that line.
Display user names from /etc/passwd (field 1):
awk -F: '{ print $1 }' /etc/passwd
Where F is the field separator in the passwd file. The fields are separated by :
Default field separator is a blank space. Awk scans the input file and splits each input line into fields.
Similarly:
cat /etc/passwd | awk -F: '{ print $1 }'
Display user names home directories and login shell (fields 1 and 7), and store them in a separate file, users.txt
awk -F: '{ print $1, $6, $7 }' /etc/passwd > users.txt
or
cat /etc/passwd | awk -F: '{ print $1, $6, $7 }' > users.txt
` Default field separator is empty space. To print users (field 1) from just created file users.txt:
awk '{ print $1 }' users.txt
Recommended tutorial: The GNU awk programming language
8.12. Introduction to grep¶
grep is used to search files or standard input for lines containing required patterns.
We’ll work with a text file, list.txt, containing the following text:
File content:
Check the inode list today
reboot the machine tomorrow
Reboot it again in a week
Call Tech support in case of emergency.
tel: 834
Oop 0
Oops 1
Oopss 12
Oopsss 123
Oopssss 1234
End
To get the line containing string “inode” in file list.txt:
grep inode list.txt
To get the line containing “inode lis ” in file list.txt:
grep "inode lis " list.txt
It should give you nothing as there is no string ” lis “
To search for the line containing “inode list” in all the files in current directory:
grep "inode list" *
Syntax of grep:
grep [options] regex [files]
where regex are regular expressions.
Using regular expressions
Regular expressions: Literals (plain text or literal text),
metacharacters (special meaning characters).
When you construct regular expressions, you use metacharacters and literals to specify three basic ideas about your input text: position anchors, groups, ranges and quantity modifiers.
Anchors:
^ -match at the beginning of a line
$ -match at the end of a line
grep '^Call' list.txt
grep '^ Reboot' list.txt
grep 'today$' list.txt
Count the number of empty lines:
grep -c '^$' list.txt
Display all lines containing only word End by itself:
grep '^End$' list.txt
Groups and ranges:
[abc] -matches any single character from a,b or c
[a-e] -matches any single charcter from among the range a-e
[^abc] -inverse match, matches a single character not among a,b, or c.
[^a-e] -inverse match, matches a single character not from the range a-e
\< word\> -matches word (option -w can be used instead)
. (single dot) -matches any single character among a new line
\ -turn off the special meaning of the character that follows
grep '[Rr]eboot' list.txt
grep '\<[Rr]eboot\>' list.txt
Display all lines from file list.txt which contain thre adjucent digits:
grep '[0-9][0-9][0-9]' list.txt
Display the lines with four or more characters in the line:
grep '....' list.txt
Display all non-blank lines from file list.txt:
grep '.' list.txt
Display all lines that contain a period:
grep '\.' list.txt
Modifiers:
* -matches zero or more instance of the preceding single character
? -matches zero or one instance of the preceding regex (implies 'grep -E' option).
+ -matches one or more instance of the preceding regex (implies 'grep -E' option).
\{n,m\} -matches a range of occurrences of the single character or regex that precedes this construct; \{n\} matches n occurences;
\{n,\} matches at least n occurences.
| -matches either the regex specified before or after the vertical bar (implies 'grep -E' option).
Display all lines from list.txt that contain Oop, Oops, Oopss, and so on:
grep 'Oops*' list.txt
Display all lines from list.txt that contain Oops, Oopss, and so on:
grep 'Oopss*' list.txt
Display all lines from list.txt that contain two or more adjacent digits:
grep '[0-9][0-9][0-9]*' list.txt
Display all lines from list.txt that contain ‘3’ or ‘34’ number combination:
grep -E '34?' list.txt
Display all lines from list.txt containing at least one digit:
grep -E '[0-9]+' list.txt
Display all lines from list.txt containing sss and ssss:
grep 's\{3,4\}' list.txt
Display all lines from list.txt containing any three, four or five digit numbers:
grep '[0-9]\{3,5\}' list.txt
Display all lines from list.txt containing “Reboot”, “reboot” or “support” strings:
grep -E '[Rr]eboot|support' list.txt
Display all lines from list.txt containing any letter (no empty lines):
grep '[A-Za-z]' list.txt
Display all lines from list.txt containing any non alpha-numeric and space symbol:
grep '[^ 0-9A-Za-z]' list.txt
Display all lines from list.txt containing uppercase letter, followed by zero or more lowercase letters:
grep '[A-Z][a-z]*' list.txt
Display all lines from list.txt containing 3 digit telephone number:
grep 'tel: [0-9]\{3\}' list.txt
Recommended tutorial on Regular expressions and grep
8.13. Introduction to sed¶
String editor, sed, is used for editing lines in a file or a stream; output is going to the standard output and can be re-directed to a new file.
Syntax:
sed [options] 'command1' [files]
sed [options] -e 'command1' [-e command2 ...] [files]
sed [options] -f script [files]
Delete lines from 3 through 5 in file list.txt:
sed '3,5d' list.txt
Delete lines that contain “O” at the beginning of the line:
sed '/^O/d' list.txt
Translate capital C,R,O into small c,r,o:
sed 'y/CRO/cro/' list.txt
Delete ampty lines:
sed '/^$/d' list.txt
Replace string Oop with Wee for the first occurence on a line
sed 's/Oop/Wee/' list.txt
Remove ss string (replace with empty entry)for the first occurence on a line:
sed 's/ss//' list.txt
Remove ss string for all occurences on a line:
sed 's/ss//g' list.txt
Substitute a single space for any number of spaces wherever they occur on the line:
sed 's/ */ /g' list.txt
Substitute underscore for any number of spaces wherever they occur on the line:
sed 's/ */_/g' list.txt