{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Shell scripts\n",
"\n",
"What actually makes a shell script:\n",
"\n",
"- Put a group of commands in a file, then make the file executable.\n",
"- In the executable file, apply the shell syntax to add variables, conditionals, iterations.\n",
"- Create functions for the frequently used blocks. \n",
"- Utilize ```awk```, ```grep```, and ```sed``` languages with regular expressions for string parsing and substitution. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Shell scripting exercises\n",
"\n",
"\n",
"- Create a directory for shell scripting exercises.\n",
"- A shell script and executable permissions.\n",
"- `if` and `case` statements\n",
"- Loops with `while` and `until` conditional statements.\n",
"- `for` loop.\n",
"- Functions in shell scripts.\n",
"- `awk` programming language.\n",
"- `grep` line search filter and regular expressions.\n",
"- String editor `sed`\n",
"- Advantages and disadvantages of shell scripts.\n",
"\n",
"[References: for details, consult Bash Guide for Beginners ](http://linuxcourse.rutgers.edu/2024/html/documents/Bash-Beginners-Guide)\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Creating a shell script\n",
"\n",
"ssh to your lxc container.\n",
"\n",
"Create a new directory `bash_scripts` where you will run the shell scripting exercises.\n",
"```bash \n",
"mkdir bash_scripts\n",
"cd bash_scripts\n",
"```\n",
"\n",
"Note, it is recommended to create a new script file for every exercise; make them executable; give them names with extension `.sh` -- it is just a convention rather than a must. The first line of a bash script starts with shebang\n",
"```bash\n",
"#!/bin/bash\n",
"```\n",
"All non-executable comments in a script are prepended with #\n",
"```{admonition} File content:\n",
" # For example, we list all the files in the current directory\n",
" ls -la\n",
" ls -l /etc # Comment. Here, we list files in /etc directory\n",
"```\n",
"\n",
"## Variables, read, echo, assignment, substitution.\n",
"Shell scripts contain variables. Variables are created by assignment operator, for example:\n",
"```bash\n",
"DIR=/home/hostadm\n",
"```\n",
"Variable value is referenced by ```$``` sign:\n",
"```bash\n",
"echo $DIR\n",
"```\n",
"Variables can be updated via assignment operator:\n",
"```bash\n",
"DIR=$DIR/KVM\n",
"echo $DIR\n",
"```\n",
"\n",
"Variables also can be read from stdo (keyboard) by using command ```read```.\n",
"\n",
"\n",
"Script `scr1.sh` is like a calculator:\n",
"```{admonition} File content:\n",
" #!/bin/bash\n",
" echo \"I will work out X*Y\"\n",
" echo \"Enter X\"\n",
" read X\n",
" echo \"Enter Y\"\n",
" read Y\n",
" echo \"X*Y = $X*$Y = $[X*Y]\"\n",
"```\n",
"Make the script executable and run\n",
"```bash\n",
"chmod 755 scr1.sh\n",
"./scr1.sh\n",
"```\n",
"\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Variable `$1`\n",
"\n",
"Variables `$1`, `$2`, `$3`, ... take the first, second, third, ... inpit parameters for the script. For example,\n",
"\n",
"```bash\n",
"\n",
"script.sh --help\n",
"```\n",
"\n",
"Where $1 will pickup string ```--help``` as the value inside ```script.sh```\n",
"\n",
"***"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Variables with commands\n",
"\n",
"We can assign a command to a variable, then execute it in the shell.\n",
"For example define variable ```cmd``` and execute it:\n",
"\n",
"```bash\n",
"cmd=\"ls /etc\"\n",
"\n",
"```\n",
"```\n",
"eval $cmd\n",
"```\n",
"\n",
"Example of a more complex statement: tar directory /etc and copy the tar file into new directory DIR1:\n",
"```bash\n",
"my_dir=etc\n",
"cmd=\"tar -cf ${my_dir}.tar /${my_dir}; mkdir DIR1; cp ${my_dir}.tar DIR1\"\n",
"eval $cmd\n",
"```\n",
"\n",
"***\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## ```if``` conditional statements\n",
"\n",
"\n",
"- The test square brackets [ ] mean test for a true or a false statement.\n",
"\n",
"```{admonition} File content:\n",
" if [ condition ]\n",
" then\n",
" perform a task at the condition = true\n",
" else\n",
" perform a task at the condition = false\n",
" fi\n",
"```\n",
"For example\n",
"\n",
"```{admonition} File content:\n",
" #!/bin/bash\n",
" \n",
" echo 'Enter the parameter value:'\n",
" read parameter\n",
" \n",
" if [ \"$parameter\" == 'this' ]\n",
" then\n",
" echo $parameter\n",
" else\n",
" echo 'Does not match'\n",
" fi\n",
"```\n",
"Another example with nested conditions\n",
"\n",
"```{admonition} File content:\n",
" #!/bin/bash\n",
" X=10\n",
" Y=5\n",
" if [ \"$X\" -gt \"$Y\" ]; then\n",
" echo \"$X is greater than $Y\"\n",
" elif [ \"$X\" -lt \"$Y\"]; then\n",
" echo \"$X is less than $Y\"\n",
" else\n",
" echo \"$X is equal to $Y\"\n",
" fi\n",
"```\n",
"\n",
"\n",
"\n",
"There are various cases for [ ] test in bash. For example, if\n",
" directory DIR1 exists:\n",
"```\n",
" if [-d \"DIR1\" ]\n",
"```\n",
"If file ttx.txt exists:\n",
"```\n",
" if [-e \"ttx.txt\" ]\n",
"```\n",
"\n",
"You can run command man to see various predefined cases for tests:\n",
"```bash\n",
"man test\n",
"```\n",
"\n",
"***\n",
"## Conditional command sequence\n",
"\n",
"Execute ```command2``` if ```command1``` completes successfully:\n",
"\n",
"```command1``` && ```command2```\n",
"\n",
"Example:\n",
"```bash\n",
"mkdir testdir\n",
"ls testdir && echo \"testdir exists\"\n",
"```\n",
"Similarly:\n",
"```bash\n",
"[-d testdir ] && echo \"testdir exists\"\n",
"```\n",
"\n",
"In opposite case, execute ```command2``` if ```command1``` fails:\n",
"\n",
"```command1``` || ```command2```\n",
"\n",
"Example:\n",
"```bash\n",
"ls testdir_2 || echo \"testir_2 doesn't exist\"\n",
"```\n",
"Similarly:\n",
"```bash\n",
"[-d testdir_2 ] || echo \"testdir_2 doesn't exists\"\n",
"```\n",
"\n",
"***\n",
"\n",
"## ```case``` conditional statements\n",
"\n",
"```{admonition} File content:\n",
" case $variable in\n",
" regex1)\n",
" commands1\n",
" ;;\n",
" regex2)\n",
" commands2\n",
" ;;\n",
" *)\n",
" commands3\n",
" ;;\n",
" esac\n",
"```\n",
"Where regex is a regular expression to match the variable value. To catch all remaining strings, use *) at the end.\n",
"\n",
"\n",
"```{admonition} File content:\n",
" #!/bin/bash\n",
" case $1 in\n",
" --test|-t)\n",
" echo \"you used the --test option\"\n",
" exit 0\n",
" ;;\n",
" --help|-h)\n",
" echo \"Usage:\"\n",
" echo \" myprog.sh [--test|--help|--version]\"\n",
" exit 0\n",
" ;;\n",
" --version|-v)\n",
" echo \"myprog.sh version 0.0.1\"\n",
" exit 0\n",
" ;;\n",
" *)\n",
" echo \"No such option $1\"\n",
" echo \"Usage:\"\n",
" echo \" myprog.sh [--test|--help|--version]\"\n",
" exit 1\n",
" ;;\n",
" esac\n",
" echo \"You typed \\\"$1\\\" on the command-line\"\n",
"```\n",
"\n",
"Note, always watch for correct syntax of case statement.\n",
"\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Looping with ```while``` and ```until``` statements\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Script `scr2.sh`:\n",
"```{admonition} File content:\n",
" #!/bin/bash\n",
" N=1\n",
" while [ \"$N\" -le \"10\" ]\n",
" do\n",
" echo \"Number $N\"\n",
" N=$[N+1]\n",
" done\n",
"```\n",
"Script `scr3.sh`\n",
"\n",
"```{admonition} File content:\n",
" #!/bin/bash\n",
" N=1\n",
" until [ \"$N\" -gt \"10\" ]\n",
" do\n",
" echo \"Number $N\"; N=$[N+1]\n",
" done\n",
"```\n",
"Note, common mistakes in shell scripting are usually due to incorrect syntax. For example, there should be no spaces before and after operator \"=\"\n",
"```{admonition} File content:\n",
" N=1 # correct\n",
" N =1 # error\n",
" N= 1 # error\n",
" N=$[N+1] # correct\n",
" N =$[N+1] # error\n",
" N= $[N+1] # error\n",
"```\n",
"\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {
"jp-MarkdownHeadingCollapsed": true
},
"source": [
"## Looping with ```for``` statement\n",
"\n",
"Script `scr4.sh`\n",
"```{admonition} File content:\n",
" #!/bin/bash\n",
" for i in red white blue\n",
" do\n",
" echo \"$i is a color\"\n",
" done\n",
"```\n",
"Script `backup-lots.sh`\n",
"```{admonition} File content:\n",
" #!/bin/bash\n",
" for i in 0 1 2 3 4 5 6 7 8 9 ; do\n",
" cp $1 $1.BAK-$i\n",
" done\n",
"```\n",
"\n",
"The enumeration in the loop can be represented by a conscize statement, ```{0..9}```:\n",
"```{admonition} File content:\n",
" #!/bin/bash\n",
" for i in {0..9} ; do\n",
" cp $1 $1.BAK-$i\n",
" done\n",
"```\n",
"\n",
"Now create a file `important_data` with some numbers in it and then run \n",
"```bash\n",
"./backup-lots.sh important_data\n",
"```\n",
"which will copy the file 10 times with 10 different extensions. As you can see, the variable `$1` has a special meaning -- it is the first argument on the command-line. Note, watch for correct syntax:\n",
"```{admonition} File content:\n",
" for i in {0..9}\n",
" do\n",
" ....\n",
" done\n",
"```\n",
"another correct alternative:\n",
"```{admonition} File content:\n",
" for i in {0..9} ; do\n",
" ....\n",
" done\n",
"```\n",
"\n",
"Now modify the script:\n",
"```{admonition} File content:\n",
" #!/bin/bash\n",
" if [ \"$1\" = \"\" ] ; then\n",
" echo \"Usage: backup-lots.sh \"\n",
" exit\n",
" fi\n",
" for i in {0..9} ; do\n",
" NEW_FILE=$1.BAK-$i\n",
" if [ -e $NEW_FILE ]; then\n",
" echo \"backup-lots.sh: **warning** $NEW_FILE\"\n",
" echo \" already exists - skipping\"\n",
" else\n",
" cp $1 $NEW_FILE\n",
" fi\n",
" done\n",
"```\n",
"Looping over glob expressions\n",
"```{admonition} File content:\n",
" #!/bin/bash\n",
" for i in *.txt ; do\n",
" echo \"found a file:\" $i\n",
" done\n",
"```\n",
"Note, the script above loops over all `*.txt` files in the current directory so you need to create several files with `.txt` extension in this exercise.\n",
"```{admonition} File content:\n",
" #!/bin/bash\n",
" for i in /usr/share/*/*.txt ; do\n",
" echo \"found a file:\" $i\n",
" done\n",
"```\n",
"\n",
"Breaking out of the loops and continuing\n",
"```{admonition} File content:\n",
" #!/bin/bash\n",
" for i in {0..9} ; do\n",
" NEW_FILE=$1.BAK-$i\n",
" if [ -e $NEW_FILE ]; then\n",
" echo \"backup-lots.sh: **error** $NEW_FILE\"\n",
" echo \" already exists - exitting\"\n",
" break\n",
" else\n",
" cp $1 $NEW_FILE\n",
" fi\n",
" done\n",
"```\n",
"```break``` command causes program execution to continue on the line after the ```done```.\n",
"\n",
"The ```continue``` statement in the loop block is useful for terminating the current iteration of the loop.\n",
"```{admonition} File content:\n",
" #!/bin/bash\n",
" for i in {0..9} ; do\n",
" NEW_FILE=$1.BAK-$i\n",
" if [ -e $NEW_FILE ] ; then\n",
" echo \"backup-lots.sh: **warning** $NEW_FILE\"\n",
" echo \" already exists - skipping\"\n",
" continue\n",
" fi\n",
" cp $1 $NEW_FILE\n",
" done\n",
"```\n",
"\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {
"jp-MarkdownHeadingCollapsed": true
},
"source": [
"## functions\n",
"\n",
"Function definitions provide a way to group statement blocks into one.\n",
"```{admonition} File content:\n",
" function usage ()\n",
" {\n",
" command1\n",
" command2; command3\n",
" ......\n",
" }\n",
"```\n",
"The word function in a function is optional. That is, the following will work as well:\n",
"```{admonition} File content:\n",
" usage ()\n",
" {\n",
" command1\n",
" command2; command3\n",
" ......\n",
" }\n",
"```\n",
"\n",
"For example, create script ```myprog.sh``` with the content below:\n",
"```{admonition} File content:\n",
" #!/bin/bash\n",
" function usage ()\n",
" {\n",
" echo \"Usage:\"\n",
" echo \" myprog.sh [--test|--help|--version]\"\n",
" }\n",
" case $1 in\n",
" --test|-t)\n",
" echo \"you used the --test option\"\n",
" exit 0\n",
" ;;\n",
" --help|-h)\n",
" usage\n",
" ;;\n",
" --version|-v)\n",
" echo \"myprog.sh version 0.0.2\"\n",
" exit 0\n",
" ;;\n",
" -*)\n",
" echo \"Error: no such option $1\"\n",
" usage\n",
" exit 1\n",
" ;;\n",
" esac\n",
" echo \"You typed \\\"$1\\\" on the command-line\"\n",
"```\n",
"\n",
"\n",
"\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {
"jp-MarkdownHeadingCollapsed": true
},
"source": [
"## Using quotes\n",
"\n",
"\n",
"\n",
"Single forward quotes `'` protect the enclosed text from the shell.\n",
"```bash\n",
"echo 'error $?'\n",
"echo 'shell name $0'\n",
"```\n",
"\n",
"Double quotes `\"` allow all shell interpretations to take place inside them.\n",
"```bash\n",
"echo \"error $?\" #gives the error code of the last command\n",
"echo \"shell name $0\" #gives the current shell name\n",
"```\n",
"\n",
"Command substitution\n",
"```bash\n",
"X=`expr 100 + 50 '*' 3`\n",
"echo $X\n",
"```\n",
"\n",
"Assigning command output to a variable:\n",
"```bash\n",
"FSIZE=`wc -l /etc/profile`\n",
"```\n",
"same as\n",
"```bash\n",
"FSIZE=$(wc -l /etc/profile)\n",
"```\n",
"\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {
"jp-MarkdownHeadingCollapsed": true
},
"source": [
"## Introduction to ```awk```\n",
"\n",
"\n",
"The basic function of ```awk``` is to search files for lines or other text units containing one or more patterns. When a line matches one of the patterns, special actions are performed on that line.\n",
"Very often, ```awk``` is used to extract certain ```columns``` from text based on matching criteria. \n",
"\n",
"Command ```awk``` reads data from a file or pipe. Syntax:\n",
"\n",
"```awk 'EXPRESSION { PROGRAM }' file(s)```\n",
"\n",
"and \n",
"\n",
"```COMMAND | awk 'EXPRESSION { PROGRAM }'```\n",
"\n",
"Display user names from `/etc/passwd` (field 1):\n",
"```bash\n",
"awk -F: '{ print $1 }' /etc/passwd\n",
"```\n",
"Where `F` is the field separator in the passwd file. The fields are separated by `:`\n",
"
\n",
"The default field separator is a blank ot tab space. Awk scans the input file and splits each input line into fields. \n",
"
\n",
"Similarly:\n",
"```bash\n",
"cat /etc/passwd | awk -F: '{ print $1 }'\n",
"```\n",
"\n",
"Display user names home directories and login shell (fields 1 and 7), and store them in a separate file, `users.txt`\n",
"```bash\n",
"awk -F: '{ print $1, $6, $7 }' /etc/passwd > users.txt\n",
"```\n",
"or\n",
"```bash\n",
"cat /etc/passwd | awk -F: '{ print $1, $6, $7 }' > users.txt\n",
"```\n",
"`\n",
"Default field separator is empty space. To print users (field 1) from just created file users.txt:\n",
"```bash\n",
"awk '{ print $1 }' users.txt\n",
"```\n",
"\n",
"To get only the columns of the line containing a pattern, for example ```hostadm```:\n",
"```bash\n",
"awk '/hostadm/ { print $1 }' users.txt\n",
"```\n",
"\n",
"In the select block ```/ /``` we can use regular expressions. For example, to see the permissions on all files in ```/etc``` directory with extension ```.conf```:\n",
"```bash\n",
"ls -l /etc | awk '/.*\\.conf/ { print $1, $9 }'\n",
"```\n",
"\n",
"In the command block ```print```, we can include strings for formatted output, for example:\n",
"```bash\n",
"ls -l /etc | awk '/.*\\.conf/ { print \"Permission:\", $1, \"File:\", $9 }'\n",
"```\n",
"\n",
"Adding a conditional statement, for example to print for the file size > 5000 bytes:\n",
"```bash\n",
"ls -l /etc | awk '/.*\\.conf/ { if ( $5 > 5000 ) print \"Permission:\", $1, \"Size\", $5, \"File:\", $9 }'\n",
"```\n",
"\n",
"```BEGIN``` and ```END``` blocks are to print statements before and after the ```awk``` command output, for example:\n",
"\n",
"\n",
"```bash\n",
"ls -l /etc | awk \\\n",
"'BEGIN { print\"Conf Files > 5K found:\\n\" } \\\n",
"/.*\\.conf/ { if ( $5 > 5000 ) print \"Permission:\", $1, \"Size\", $5, \"File:\", $9 } \\\n",
"END { print \"\\n ---- The end -------\" }'\n",
"```\n",
"\n",
"\n",
"[Recommended tutorial: The GNU awk programming language](https://linuxcourse.rutgers.edu/2024/html/documents/Bash-Beginners-Guide/chap_06.html)\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Introduction to ```grep```\n",
"\n",
"\n",
"\n",
"```grep``` is used to search files or standard input for ```lines``` containing required patterns.\n",
"\n",
"We'll work with a text file, `list.txt`, containing the following text:\n",
"\n",
"```{admonition} File content:\n",
" Check the inode list today\n",
" reboot the machine tomorrow\n",
" Reboot it again in a week\n",
" Call Tech support in case of emergency.\n",
" tel: 834\n",
"\n",
" Oop 0\n",
" Oops 1\n",
" Oopss 12\n",
" Oopsss 123\n",
" Oopssss 1234\n",
" End\n",
"```\n",
"To get the line containing string \"inode\" in file `list.txt`:\n",
"```bash\n",
"grep inode list.txt\n",
"```\n",
"To get the line containing \"inode lis \" in file `list.txt`:\n",
"```bash\n",
"grep \"inode lis \" list.txt\n",
"```\n",
"It should give you nothing as there is no string \" lis \"\n",
"\n",
"To search for the line containing \"inode list\" in all the files in current directory:\n",
"```bash\n",
"grep \"inode list\" *\n",
"```\n",
"\n",
"Syntax of `grep`:\n",
"```bash\n",
"grep [options] regex [files]\n",
"```\n",
"where regex are regular expressions.\n",
"\n",
"Using regular expressions\n",
"
\n",
"Regular expressions: Literals (plain text or literal text),\n",
"metacharacters (special meaning characters).\n",
"When you construct regular expressions, you use metacharacters and literals to specify three basic ideas about your input text: `position anchors`, `groups`, `ranges` and `quantity modifiers`.\n",
"```{admonition} Anchors: \n",
" ^ -match at the beginning of a line\n",
" $ -match at the end of a line\n",
"```\n",
"```bash\n",
"grep '^Call' list.txt\n",
"grep '^ Reboot' list.txt\n",
"grep 'today$' list.txt\n",
"```\n",
"\n",
"Count the number of empty lines:\n",
"```bash\n",
"grep -c '^$' list.txt\n",
"```\n",
"Display all lines containing only word End by itself:\n",
"```bash\n",
"grep '^End$' list.txt\n",
"```\n",
"```{admonition} Groups and ranges: \n",
" [abc] -matches any single character from a,b or c\n",
" [a-e] -matches any single charcter from among the range a-e\n",
" [^abc] -inverse match, matches a single character not among a,b, or c.\n",
" [^a-e] -inverse match, matches a single character not from the range a-e\n",
" \\< word\\> -matches word (option -w can be used instead)\n",
" . (single dot) -matches any single character among a new line\n",
" \\ -turn off the special meaning of the character that follows\n",
"```\n",
"```bash\n",
"grep '[Rr]eboot' list.txt\n",
"grep '\\<[Rr]eboot\\>' list.txt\n",
"```\n",
"Display all lines from file list.txt which contain thre adjucent digits:\n",
"```bash\n",
"grep '[0-9][0-9][0-9]' list.txt\n",
"```\n",
"Display the lines with four or more characters in the line:\n",
"```bash\n",
"grep '....' list.txt\n",
"```\n",
"Display all non-blank lines from file list.txt:\n",
"```bash\n",
"grep '.' list.txt\n",
"```\n",
"Display all lines that contain a period:\n",
"```bash\n",
"grep '\\.' list.txt\n",
"```\n",
"```{admonition} Modifiers: \n",
" * -matches zero or more instance of the preceding single character\n",
" ? -matches zero or one instance of the preceding regex (implies 'grep -E' option).\n",
" + -matches one or more instance of the preceding regex (implies 'grep -E' option).\n",
" \\{n,m\\} -matches a range of occurrences of the single character or regex that precedes this construct; \\{n\\} matches n occurences;\n",
" \\{n,\\} matches at least n occurences.\n",
" | -matches either the regex specified before or after the vertical bar (implies 'grep -E' option).\n",
"```\n",
"Display all lines from `list.txt` that contain Oop, Oops, Oopss, and so on:\n",
"```bash\n",
"grep 'Oops*' list.txt\n",
"```\n",
"Display all lines from `list.txt` that contain Oops, Oopss, and so on:\n",
"```bash\n",
"grep 'Oopss*' list.txt\n",
"``` \n",
"Display all lines from `list.txt` that contain two or more adjacent digits:\n",
"```bash\n",
"grep '[0-9][0-9][0-9]*' list.txt\n",
"```\n",
"Display all lines from list.txt that contain '3' or '34' number combination:\n",
"```bash\n",
"grep -E '34?' list.txt\n",
"```\n",
"Display all lines from `list.txt` containing at least one digit:\n",
"```bash\n",
"grep -E '[0-9]+' list.txt\n",
"```\n",
"Display all lines from `list.txt` containing sss and ssss:\n",
"```bash\n",
"grep 's\\{3,4\\}' list.txt\n",
"```\n",
"Display all lines from `list.txt` containing any three, four or five digit numbers:\n",
"```bash\n",
"grep '[0-9]\\{3,5\\}' list.txt\n",
"```\n",
"Display all lines from `list.txt` containing \"Reboot\", \"reboot\" or \"support\" strings:\n",
"```bash\n",
"grep -E '[Rr]eboot|support' list.txt\n",
"```\n",
"Display all lines from list.txt containing any letter (no empty lines):\n",
"```bash\n",
"grep '[A-Za-z]' list.txt\n",
"```\n",
"Display all lines from `list.txt` containing any non alpha-numeric and space symbol:\n",
"```bash\n",
"grep '[^ 0-9A-Za-z]' list.txt\n",
"```\n",
"Display all lines from list.txt containing uppercase letter, followed by zero or more lowercase letters:\n",
"```bash\n",
"grep '[A-Z][a-z]*' list.txt\n",
"```\n",
"Display all lines from list.txt containing 3 digit telephone number:\n",
"```bash\n",
"grep 'tel: [0-9]\\{3\\}' list.txt\n",
"```\n",
"\n",
"[Recommended tutorial on Regular expressions and grep](http://linuxcourse.rutgers.edu/2024/html/documents/Bash-Beginners-Guide/chap_04.html)\n",
"\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Introduction to ```sed```\n",
"\n",
"\n",
"\n",
"String editor, ```sed```, is used for editing lines in a file or a stream; output is going to the standard output and can be re-directed to a new file.\n",
"Syntax:\n",
"```bash\n",
"sed [options] 'command1' [files]\n",
"sed [options] -e 'command1' [-e command2 ...] [files]\n",
"sed [options] -f script [files]\n",
"```\n",
"Print lines from 3 through 5 in file ```list.txt```\n",
"```bash\n",
"sed -n '3,5p' list.txt\n",
"```\n",
"Delete lines from 3 through 5 in file ```list.txt```:\n",
"```bash\n",
"sed '3,5d' list.txt\n",
"```\n",
"We can use the regular expressions for pattern matching of a block in ```/ /```to perform a command on.\n",
"\n",
"For example, to delete lines that contain \"O\" at the beginning of the line:\n",
"```bash\n",
"sed '/^O/d' list.txt\n",
"```\n",
"Delete ampty lines:\n",
"```bash\n",
"sed '/^$/d' list.txt\n",
"```\n",
"Translate capital C,R,O into small c,r,o:\n",
"```bash\n",
"sed 'y/CRO/cro/' list.txt\n",
"```\n",
"Replace string Oop with Wee for the first occurence on a line\n",
"```bash\n",
"sed 's/Oop/Wee/' list.txt\n",
"```\n",
"Remove ss string (replace with empty entry)for the first occurence on a line:\n",
"```bash\n",
"sed 's/ss//' list.txt\n",
"```\n",
"Remove ss string for all occurences on a line:\n",
"```bash\n",
"sed 's/ss//g' list.txt\n",
"```\n",
"Substitute a single space for any number of spaces wherever they occur on the line:\n",
"```bash\n",
"sed 's/ */ /g' list.txt\n",
"```\n",
"Substitute underscore for any number of spaces wherever they occur on the line:\n",
"```bash\n",
"sed 's/ */_/g' list.txt\n",
"```\n",
"To have ```sed``` to modify the original content of a file, use option ```-i```, which mean in-line. \n",
"For example to apply the changes to file list2.txt:\n",
"```bash\n",
"cp list.txt list2.txt\n",
"sed -i 's/ */_/g' list2.txt\n",
"```\n",
"\n",
"[More examples: Useful one-line scripts for sed](http://linuxcourse.rutgers.edu/2024/html/documents/sed1line.txt)\n",
"\n",
"[Recommended tutorial: The GNU sed stream editor](http://linuxcourse.rutgers.edu/2024/html/documents/Bash-Beginners-Guide/chap_05.html)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Printing the lines containing a matching pattern by ```grep```, ```awk```, and ```sed```\n",
"\n",
"The three commands below print out the same line from file list.txt:\n",
"\n",
"```bash\n",
"grep Reboot list.txt\n",
"\n",
"awk '/Reboot/ { print $0 }' list.txt\n",
"\n",
"sed -n '/Reboot/p' list.txt\n",
"```"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.13.11"
}
},
"nbformat": 4,
"nbformat_minor": 4
}