{ "cells": [ { "cell_type": "code", "execution_count": 3, "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [ "remove_cell", "remove-cell" ] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "<>:539: SyntaxWarning: invalid escape sequence '\\.'\n", "<>:539: SyntaxWarning: invalid escape sequence '\\.'\n", "/tmp/ipykernel_299857/3427652635.py:539: SyntaxWarning: invalid escape sequence '\\.'\n", " ls -l /etc | awk '/.*\\.conf/ { print $1, $9 }'\n" ] } ], "source": [ "my_notebook_text = \"\"\"\n", "# Shell scripts\n", "\n", "What actually makes a shell script:\n", "\n", "- Put a group of commands in a file, then make the file executable.\n", "- In the executable file, apply the shell syntax to add variables, conditionals, iterations.\n", "- Create functions for the frequently used blocks. \n", "- Utilize ```awk```, ```grep```, and ```sed``` languages with regular expressions for string parsing and substitution. \n", "\n", "## Shell scripting exercises\n", "\n", "\n", "- Create a directory forM shell scripting exercises.\n", "- A shell script and executable permissions.\n", "- `if` and `case` statements\n", "- Loops with `while` and `until` conditional statements.\n", "- `for` loop.\n", "- Functions in shell scripts.\n", "- `awk` programming language.\n", "- `grep` line search filter and regular expressions.\n", "- String editor `sed`\n", "- Advantages and disadvantages of shell scripts.\n", "\n", "[References: for details, consult Bash Guide for Beginners ](http://linuxcourse.rutgers.edu/2024/html/documents/Bash-Beginners-Guide)\n", "
\n", "\n", "## Creating a shell script\n", "\n", "ssh to your lxc container.\n", "\n", "Create a new directory `bash_scripts` where you will run the shell scripting exercises.\n", "```bash \n", "mkdir bash_scripts\n", "cd bash_scripts\n", "```\n", "\n", "Note, it is recommended to create a new script file for every exercise; make them executable; give them names with extension `.sh` -- it is just a convention rather than a must. The first line of a bash script starts with shebang\n", "```bash\n", "#!/bin/bash\n", "```\n", "All non-executable comments in a script are prepended with #\n", "```{admonition} File content:\n", " # For example, we list all the files in the current directory\n", " ls -la\n", " ls -l /etc # Comment. Here, we list files in /etc directory\n", "```\n", "\n", "## Variables, read, echo, assignment, substitution.\n", "Shell scripts contain variables. Variables are created by assignment operator, for example:\n", "```bash\n", "DIR=/home/hostadm\n", "```\n", "Variable value is referenced by ```$``` sign:\n", "```bash\n", "echo $DIR\n", "```\n", "Variables can be updated via assignment operator:\n", "```bash\n", "DIR=$DIR/KVM\n", "echo $DIR\n", "```\n", "\n", "Variables also can be read from stdo (keyboard) by using command ```read```.\n", "\n", "\n", "Script `scr1.sh` is like a calculator:\n", "```{admonition} File content:\n", " #!/bin/bash\n", " echo \"I will work out X*Y\"\n", " echo \"Enter X\"\n", " read X\n", " echo \"Enter Y\"\n", " read Y\n", " echo \"X*Y = $X*$Y = $[X*Y]\"\n", "```\n", "Make the script executable and run\n", "```bash\n", "chmod 755 scr1.sh\n", "./scr1.sh\n", "```\n", "\n", "
\n", "\n", "## Variable `$1`\n", "\n", "Variables `$1`, `$2`, `$3`, ... take the first, second, third, ... inpit parameters for the script. For example,\n", "\n", "```bash\n", "\n", "script.sh --help\n", "```\n", "\n", "Where $1 will pickup string ```--help``` as the value inside ```script.sh```\n", "\n", "***\n", "\n", "## Variables with commands\n", "\n", "We can assign a command to a variable, then execute it in the shell.\n", "For example define variable ```cmd``` and execute it:\n", "\n", "```bash\n", "cmd=\"ls /etc\"\n", "\n", "```\n", "```\n", "eval $cmd\n", "```\n", "\n", "Example of a more complex statement: tar directory /etc and copy the tar file into new directory DIR1:\n", "```bash\n", "my_dir=etc\n", "cmd=\"tar -cf ${my_dir}.tar /${my_dir}; mkdir DIR1; cp ${my_dir}.tar DIR1\"\n", "eval $cmd\n", "```\n", "\n", "***\n", "\n", "\n", "## ```if``` conditional statements\n", "\n", "\n", "- The test square brackets [ ] mean test for a true or a false statement.\n", "\n", "```{admonition} File content:\n", " if [ condition ]\n", " then\n", " perform a task at the condition = true\n", " else\n", " perform a task at the condition = false\n", " fi\n", "```\n", "For example\n", "\n", "```{admonition} File content:\n", " #!/bin/bash\n", " \n", " echo 'Enter the parameter value:'\n", " read parameter\n", " \n", " if [ \"$parameter\" == 'this' ]\n", " then\n", " echo $parameter\n", " else\n", " echo 'Does not match'\n", " fi\n", "```\n", "Another example with nested conditions\n", "\n", "```{admonition} File content:\n", " #!/bin/bash\n", " X=10\n", " Y=5\n", " if [ \"$X\" -gt \"$Y\" ]; then\n", " echo \"$X is greater than $Y\"\n", " elif [ \"$X\" -lt \"$Y\"]; then\n", " echo \"$X is less than $Y\"\n", " else\n", " echo \"$X is equal to $Y\"\n", " fi\n", "```\n", "\n", "\n", "\n", "There are various cases for [ ] test in bash. For example, if\n", " directory DIR1 exists:\n", "```\n", " if [ -d \"DIR1\" ]\n", "```\n", "If file ttx.txt exists:\n", "```\n", " if [ -e \"ttx.txt\" ]\n", "```\n", "\n", "You can run command man to see various predefined cases for tests:\n", "```bash\n", "man test\n", "```\n", "\n", "***\n", "## Conditional command sequence\n", "\n", "Execute ```command2``` if ```command1``` completes successfully:\n", "\n", "```command1``` && ```command2```\n", "\n", "Example:\n", "```bash\n", "mkdir testdir\n", "ls testdir && echo \"testdir exists\"\n", "```\n", "Similarly:\n", "```bash\n", "[ -d testdir ] && echo \"testdir exists\"\n", "```\n", "\n", "In opposite case, execute ```command2``` if ```command1``` fails:\n", "\n", "```command1``` || ```command2```\n", "\n", "Example:\n", "```bash\n", "ls testdir_2 || echo \"testir_2 doesn't exist\"\n", "```\n", "Similarly:\n", "```bash\n", "[ -d testdir_2 ] || echo \"testdir_2 doesn't exists\"\n", "```\n", "\n", "***\n", "\n", "## ```case``` conditional statements\n", "\n", "```{admonition} File content:\n", " case $variable in\n", " regex1)\n", " commands1\n", " ;;\n", " regex2)\n", " commands2\n", " ;;\n", " *)\n", " commands3\n", " ;;\n", " esac\n", "```\n", "Where regex is a regular expression to match the variable value. To catch all remaining strings, use *) at the end.\n", "\n", "\n", "```{admonition} File content:\n", " #!/bin/bash\n", " case $1 in\n", " --test|-t)\n", " echo \"you used the --test option\"\n", " exit 0\n", " ;;\n", " --help|-h)\n", " echo \"Usage:\"\n", " echo \" myprog.sh [--test|--help|--version]\"\n", " exit 0\n", " ;;\n", " --version|-v)\n", " echo \"myprog.sh version 0.0.1\"\n", " exit 0\n", " ;;\n", " *)\n", " echo \"No such option $1\"\n", " echo \"Usage:\"\n", " echo \" myprog.sh [--test|--help|--version]\"\n", " exit 1\n", " ;;\n", " esac\n", " echo \"You typed \\\"$1\\\" on the command-line\"\n", "```\n", "\n", "Note, always watch for correct syntax of case statement.\n", "\n", "
\n", "\n", "## Looping with ```while``` and ```until``` statements\n", "\n", "\n", "Script `scr2.sh`:\n", "```{admonition} File content:\n", " #!/bin/bash\n", " N=1\n", " while [ \"$N\" -le \"10\" ]\n", " do\n", " echo \"Number $N\"\n", " N=$[N+1]\n", " done\n", "```\n", "Script `scr3.sh`\n", "\n", "```{admonition} File content:\n", " #!/bin/bash\n", " N=1\n", " until [ \"$N\" -gt \"10\" ]\n", " do\n", " echo \"Number $N\"; N=$[N+1]\n", " done\n", "```\n", "Note, common mistakes in shell scripting are usually due to incorrect syntax. For example, there should be no spaces before and after operator \"=\"\n", "```{admonition} File content:\n", " N=1 # correct\n", " N =1 # error\n", " N= 1 # error\n", " N=$[N+1] # correct\n", " N =$[N+1] # error\n", " N= $[N+1] # error\n", "```\n", "\n", "
\n", "\n", "## Looping with ```for``` statement\n", "\n", "Script `scr4.sh`\n", "```{admonition} File content:\n", " #!/bin/bash\n", " for i in red white blue\n", " do\n", " echo \"$i is a color\"\n", " done\n", "```\n", "Script `backup-lots.sh`\n", "```{admonition} File content:\n", " #!/bin/bash\n", " for i in 0 1 2 3 4 5 6 7 8 9 ; do\n", " cp $1 $1.BAK-$i\n", " done\n", "```\n", "\n", "The enumeration in the loop can be represented by a conscize statement, ```{0..9}```:\n", "```{admonition} File content:\n", " #!/bin/bash\n", " for i in {0..9} ; do\n", " cp $1 $1.BAK-$i\n", " done\n", "```\n", "\n", "Now create a file `important_data` with some numbers in it and then run \n", "```bash\n", "./backup-lots.sh important_data\n", "```\n", "which will copy the file 10 times with 10 different extensions. As you can see, the variable `$1` has a special meaning -- it is the first argument on the command-line. Note, watch for correct syntax:\n", "```{admonition} File content:\n", " for i in {0..9}\n", " do\n", " ....\n", " done\n", "```\n", "another correct alternative:\n", "```{admonition} File content:\n", " for i in {0..9} ; do\n", " ....\n", " done\n", "```\n", "\n", "Now modify the script:\n", "```{admonition} File content:\n", " #!/bin/bash\n", " if [ \"$1\" = \"\" ] ; then\n", " echo \"Usage: backup-lots.sh \"\n", " exit\n", " fi\n", " for i in {0..9} ; do\n", " NEW_FILE=$1.BAK-$i\n", " if [ -e $NEW_FILE ]; then\n", " echo \"backup-lots.sh: **warning** $NEW_FILE\"\n", " echo \" already exists - skipping\"\n", " else\n", " cp $1 $NEW_FILE\n", " fi\n", " done\n", "```\n", "Looping over glob expressions\n", "```{admonition} File content:\n", " #!/bin/bash\n", " for i in *.txt ; do\n", " echo \"found a file:\" $i\n", " done\n", "```\n", "Note, the script above loops over all `*.txt` files in the current directory so you need to create several files with `.txt` extension in this exercise.\n", "```{admonition} File content:\n", " #!/bin/bash\n", " for i in /usr/share/*/*.txt ; do\n", " echo \"found a file:\" $i\n", " done\n", "```\n", "\n", "Breaking out of the loops and continuing\n", "```{admonition} File content:\n", " #!/bin/bash\n", " for i in {0..9} ; do\n", " NEW_FILE=$1.BAK-$i\n", " if [ -e $NEW_FILE ]; then\n", " echo \"backup-lots.sh: **error** $NEW_FILE\"\n", " echo \" already exists - exitting\"\n", " break\n", " else\n", " cp $1 $NEW_FILE\n", " fi\n", " done\n", "```\n", "```break``` command causes program execution to continue on the line after the ```done```.\n", "\n", "The ```continue``` statement in the loop block is useful for terminating the current iteration of the loop.\n", "```{admonition} File content:\n", " #!/bin/bash\n", " for i in {0..9} ; do\n", " NEW_FILE=$1.BAK-$i\n", " if [ -e $NEW_FILE ] ; then\n", " echo \"backup-lots.sh: **warning** $NEW_FILE\"\n", " echo \" already exists - skipping\"\n", " continue\n", " fi\n", " cp $1 $NEW_FILE\n", " done\n", "```\n", "\n", "
\n", "\n", "## functions\n", "\n", "Function definitions provide a way to group statement blocks into one.\n", "```{admonition} File content:\n", " function usage ()\n", " {\n", " command1\n", " command2; command3\n", " ......\n", " }\n", "```\n", "The word function in a function is optional. That is, the following will work as well:\n", "```{admonition} File content:\n", " usage ()\n", " {\n", " command1\n", " command2; command3\n", " ......\n", " }\n", "```\n", "\n", "For example, create script ```myprog.sh``` with the content below:\n", "```{admonition} File content:\n", " #!/bin/bash\n", " function usage ()\n", " {\n", " echo \"Usage:\"\n", " echo \" myprog.sh [--test|--help|--version]\"\n", " }\n", " case $1 in\n", " --test|-t)\n", " echo \"you used the --test option\"\n", " exit 0\n", " ;;\n", " --help|-h)\n", " usage\n", " ;;\n", " --version|-v)\n", " echo \"myprog.sh version 0.0.2\"\n", " exit 0\n", " ;;\n", " -*)\n", " echo \"Error: no such option $1\"\n", " usage\n", " exit 1\n", " ;;\n", " esac\n", " echo \"You typed \\\"$1\\\" on the command-line\"\n", "```\n", "\n", "\n", "\n", "
\n", "\n", "## Using quotes\n", "\n", "\n", "\n", "Single forward quotes `'` protect the enclosed text from the shell.\n", "```bash\n", "echo 'error $?'\n", "echo 'shell name $0'\n", "```\n", "\n", "Double quotes `\"` allow all shell interpretations to take place inside them.\n", "```bash\n", "echo \"error $?\" #gives the error code of the last command\n", "echo \"shell name $0\" #gives the current shell name\n", "```\n", "\n", "Command substitution\n", "```bash\n", "X=`expr 100 + 50 '*' 3`\n", "echo $X\n", "```\n", "\n", "Assigning command output to a variable:\n", "```bash\n", "FSIZE=`wc -l /etc/profile`\n", "```\n", "same as\n", "```bash\n", "FSIZE=$(wc -l /etc/profile)\n", "```\n", "\n", "
\n", "\n", "## Introduction to ```awk```\n", "\n", "\n", "The basic function of ```awk``` is to search files for lines or other text units containing one or more patterns. When a line matches one of the patterns, special actions are performed on that line.\n", "Very often, ```awk``` is used to extract certain ```columns``` from text based on matching criteria. \n", "\n", "Command ```awk``` reads data from a file or pipe. Syntax:\n", "\n", "```awk 'EXPRESSION { PROGRAM }' file(s)```\n", "\n", "and \n", "\n", "```COMMAND | awk 'EXPRESSION { PROGRAM }'```\n", "\n", "Display user names from `/etc/passwd` (field 1):\n", "```bash\n", "awk -F: '{ print $1 }' /etc/passwd\n", "```\n", "Where `F` is the field separator in the passwd file. The fields are separated by `:`\n", "
\n", "The default field separator is a blank ot tab space. Awk scans the input file and splits each input line into fields. \n", "
\n", "Similarly:\n", "```bash\n", "cat /etc/passwd | awk -F: '{ print $1 }'\n", "```\n", "\n", "Display user names home directories and login shell (fields 1 and 7), and store them in a separate file, `users.txt`\n", "```bash\n", "awk -F: '{ print $1, $6, $7 }' /etc/passwd > users.txt\n", "```\n", "or\n", "```bash\n", "cat /etc/passwd | awk -F: '{ print $1, $6, $7 }' > users.txt\n", "```\n", "\n", "Default field separator is empty space. To print users (field 1) from just created file users.txt:\n", "```bash\n", "awk '{ print $1 }' users.txt\n", "```\n", "\n", "To get only the columns of the line containing a pattern, for example ```hostadm```:\n", "```bash\n", "awk '/hostadm/ { print $1 }' users.txt\n", "```\n", "\n", "In the select block ```/ /``` we can use regular expressions. For example, to see the permissions on all files in ```/etc``` directory with extension ```.conf```:\n", "```bash\n", "ls -l /etc | awk '/.*\\.conf/ { print $1, $9 }'\n", "```\n", "\n", "In the command block ```print```, we can include strings for formatted output, for example:\n", "```bash\n", "ls -l /etc | awk '/.*\\.conf/ { print \"Permission:\", $1, \"File:\", $9 }'\n", "```\n", "\n", "Adding a conditional statement, for example to print for the file size > 5000 bytes:\n", "```bash\n", "ls -l /etc | awk '/.*\\.conf/ { if ( $5 > 5000 ) print \"Permission:\", $1, \"Size\", $5, \"File:\", $9 }'\n", "```\n", "\n", "```BEGIN``` and ```END``` blocks are to print statements before and after the ```awk``` command output, for example:\n", "\n", "\n", "```bash\n", "ls -l /etc | awk \\\n", "'BEGIN { print\"Conf Files > 5K found:\\n\" } \\\n", "/.*\\.conf/ { if ( $5 > 5000 ) print \"Permission:\", $1, \"Size\", $5, \"File:\", $9 } \\\n", "END { print \"\\n ---- The end -------\" }'\n", "```\n", "\n", "\n", "[Recommended tutorial: The GNU awk programming language](https://linuxcourse.rutgers.edu/2024/html/documents/Bash-Beginners-Guide/chap_06.html)\n", "
\n", "\n", "## Introduction to ```grep```\n", "\n", "\n", "\n", "```grep``` is used to search files or standard input for ```lines``` containing required patterns.\n", "\n", "We'll work with a text file, `list.txt`, containing the following text:\n", "\n", "```{admonition} File content:\n", " Check the inode list today\n", " reboot the machine tomorrow\n", " Reboot it again in a week\n", " Call Tech support in case of emergency.\n", " tel: 834\n", "\n", " Oop 0\n", " Oops 1\n", " Oopss 12\n", " Oopsss 123\n", " Oopssss 1234\n", " End\n", "```\n", "To get the line containing string \"inode\" in file `list.txt`:\n", "```bash\n", "grep inode list.txt\n", "```\n", "To get the line containing \"inode lis \" in file `list.txt`:\n", "```bash\n", "grep \"inode lis \" list.txt\n", "```\n", "It should give you nothing as there is no string \" lis \"\n", "\n", "To search for the line containing \"inode list\" in all the files in current directory:\n", "```bash\n", "grep \"inode list\" *\n", "```\n", "\n", "Syntax of `grep`:\n", "```bash\n", "grep [options] regex [files]\n", "```\n", "where regex are regular expressions.\n", "\n", "Using regular expressions\n", "
\n", "Regular expressions: Literals (plain text or literal text),\n", "metacharacters (special meaning characters).\n", "When you construct regular expressions, you use metacharacters and literals to specify three basic ideas about your input text: `position anchors`, `groups`, `ranges` and `quantity modifiers`.\n", "```{admonition} Anchors: \n", " ^ -match at the beginning of a line\n", " $ -match at the end of a line\n", "```\n", "```bash\n", "grep '^Call' list.txt\n", "grep '^ Reboot' list.txt\n", "grep 'today$' list.txt\n", "```\n", "\n", "Count the number of empty lines:\n", "```bash\n", "grep -c '^$' list.txt\n", "```\n", "Display all lines containing only word End by itself:\n", "```bash\n", "grep '^End$' list.txt\n", "```\n", "```{admonition} Groups and ranges: \n", " [abc] -matches any single character from a,b or c\n", " [a-e] -matches any single charcter from among the range a-e\n", " [^abc] -inverse match, matches a single character not among a,b, or c.\n", " [^a-e] -inverse match, matches a single character not from the range a-e\n", " \\< word\\> -matches word (option -w can be used instead)\n", " . (single dot) -matches any single character among a new line\n", " \\ -turn off the special meaning of the character that follows\n", "```\n", "```bash\n", "grep '[Rr]eboot' list.txt\n", "grep '\\<[Rr]eboot\\>' list.txt\n", "```\n", "Display all lines from file list.txt which contain thre adjucent digits:\n", "```bash\n", "grep '[0-9][0-9][0-9]' list.txt\n", "```\n", "Display the lines with four or more characters in the line:\n", "```bash\n", "grep '....' list.txt\n", "```\n", "Display all non-blank lines from file list.txt:\n", "```bash\n", "grep '.' list.txt\n", "```\n", "Display all lines that contain a period:\n", "```bash\n", "grep '\\.' list.txt\n", "```\n", "```{admonition} Modifiers: \n", " * -matches zero or more instance of the preceding single character\n", " ? -matches zero or one instance of the preceding regex (implies 'grep -E' option).\n", " + -matches one or more instance of the preceding regex (implies 'grep -E' option).\n", " \\{n,m\\} -matches a range of occurrences of the single character or regex that precedes this construct; \\{n\\} matches n occurences;\n", " \\{n,\\} matches at least n occurences.\n", " | -matches either the regex specified before or after the vertical bar (implies 'grep -E' option).\n", "```\n", "Display all lines from `list.txt` that contain Oop, Oops, Oopss, and so on:\n", "```bash\n", "grep 'Oops*' list.txt\n", "```\n", "Display all lines from `list.txt` that contain Oops, Oopss, and so on:\n", "```bash\n", "grep 'Oopss*' list.txt\n", "``` \n", "Display all lines from `list.txt` that contain two or more adjacent digits:\n", "```bash\n", "grep '[0-9][0-9][0-9]*' list.txt\n", "```\n", "Display all lines from list.txt that contain '3' or '34' number combination:\n", "```bash\n", "grep -E '34?' list.txt\n", "```\n", "Display all lines from `list.txt` containing at least one digit:\n", "```bash\n", "grep -E '[0-9]+' list.txt\n", "```\n", "Display all lines from `list.txt` containing sss and ssss:\n", "```bash\n", "grep 's\\{3,4\\}' list.txt\n", "```\n", "Display all lines from `list.txt` containing any three, four or five digit numbers:\n", "```bash\n", "grep '[0-9]\\{3,5\\}' list.txt\n", "```\n", "Display all lines from `list.txt` containing \"Reboot\", \"reboot\" or \"support\" strings:\n", "```bash\n", "grep -E '[Rr]eboot|support' list.txt\n", "```\n", "Display all lines from list.txt containing any letter (no empty lines):\n", "```bash\n", "grep '[A-Za-z]' list.txt\n", "```\n", "Display all lines from `list.txt` containing any non alpha-numeric and space symbol:\n", "```bash\n", "grep '[^ 0-9A-Za-z]' list.txt\n", "```\n", "Display all lines from list.txt containing uppercase letter, followed by zero or more lowercase letters:\n", "```bash\n", "grep '[A-Z][a-z]*' list.txt\n", "```\n", "Display all lines from list.txt containing 3 digit telephone number:\n", "```bash\n", "grep 'tel: [0-9]\\{3\\}' list.txt\n", "```\n", "\n", "[Recommended tutorial on Regular expressions and grep](http://linuxcourse.rutgers.edu/2024/html/documents/Bash-Beginners-Guide/chap_04.html)\n", "\n", "
\n", "\n", "## Introduction to ```sed```\n", "\n", "\n", "\n", "String editor, ```sed```, is used for editing lines in a file or a stream; output is going to the standard output and can be re-directed to a new file.\n", "Syntax:\n", "```bash\n", "sed [options] 'command1' [files]\n", "sed [options] -e 'command1' [-e command2 ...] [files]\n", "sed [options] -f script [files]\n", "```\n", "Print lines from 3 through 5 in file ```list.txt```\n", "```bash\n", "sed -n '3,5p' list.txt\n", "```\n", "Delete lines from 3 through 5 in file ```list.txt```:\n", "```bash\n", "sed '3,5d' list.txt\n", "```\n", "We can use the regular expressions for pattern matching of a block in ```/ /```to perform a command on.\n", "\n", "For example, to delete lines that contain \"O\" at the beginning of the line:\n", "```bash\n", "sed '/^O/d' list.txt\n", "```\n", "Delete ampty lines:\n", "```bash\n", "sed '/^$/d' list.txt\n", "```\n", "Translate capital C,R,O into small c,r,o:\n", "```bash\n", "sed 'y/CRO/cro/' list.txt\n", "```\n", "Replace string Oop with Wee for the first occurence on a line\n", "```bash\n", "sed 's/Oop/Wee/' list.txt\n", "```\n", "Remove ss string (replace with empty entry)for the first occurence on a line:\n", "```bash\n", "sed 's/ss//' list.txt\n", "```\n", "Remove ss string for all occurences on a line:\n", "```bash\n", "sed 's/ss//g' list.txt\n", "```\n", "Substitute a single space for any number of spaces wherever they occur on the line:\n", "```bash\n", "sed 's/ */ /g' list.txt\n", "```\n", "Substitute underscore for any number of spaces wherever they occur on the line:\n", "```bash\n", "sed 's/ */_/g' list.txt\n", "```\n", "To have ```sed``` to modify the original content of a file, use option ```-i```, which mean in-line. \n", "For example to apply the changes to file list2.txt:\n", "```bash\n", "cp list.txt list2.txt\n", "sed -i 's/ */_/g' list2.txt\n", "```\n", "\n", "[More examples: Useful one-line scripts for sed](http://linuxcourse.rutgers.edu/2024/html/documents/sed1line.txt)\n", "\n", "[Recommended tutorial: The GNU sed stream editor](http://linuxcourse.rutgers.edu/2024/html/documents/Bash-Beginners-Guide/chap_05.html)\n", "\n", "## Printing the lines containing a matching pattern by ```grep```, ```awk```, and ```sed```\n", "\n", "The three commands below print out the same line from file list.txt:\n", "\n", "```bash\n", "grep Reboot list.txt\n", "\n", "awk '/Reboot/ { print $0 }' list.txt\n", "\n", "sed -n '/Reboot/p' list.txt\n", "```\n", "\n", "\"\"\"" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [ "remove_cell", "remove-cell" ] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Both GOOGLE_API_KEY and GEMINI_API_KEY are set. Using GOOGLE_API_KEY.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "# Shell Scripts\n", "\n", "What actually makes a shell script:\n", "\n", "- Put a group of commands in a file, then make the file executable.\n", "- In the executable file, apply the shell syntax to add variables, conditionals, iterations.\n", "- Create functions for the frequently used blocks. \n", "- Utilize ```awk```, ```grep```, and ```sed``` languages with regular expressions for string parsing and substitution. \n", "\n", "## Shell Scripting Exercises\n", "\n", "\n", "- Create a directory for shell scripting exercises.\n", "- A shell script and executable permissions.\n", "- `if` and `case` statements.\n", "- Loops with `while` and `until` conditional statements.\n", "- `for` loop.\n", "- Functions in shell scripts.\n", "- `awk` programming language.\n", "- `grep` line search filter and regular expressions.\n", "- String editor `sed`.\n", "- Advantages and disadvantages of shell scripts.\n", "\n", "[References: for details, consult the Bash Guide for Beginners](http://linuxcourse.rutgers.edu/2024/html/documents/Bash-Beginners-Guide)\n", "
\n", "\n", "## Creating a Shell Script\n", "\n", "SSH into your LXC container.\n", "\n", "Create a new directory named `bash_scripts` where you will run the shell scripting exercises:\n", "```bash \n", "mkdir bash_scripts\n", "cd bash_scripts\n", "```\n", "\n", "Note: It is recommended to create a new script file for every exercise, make them executable, and give them names ending with the `.sh` extension (this is a convention rather than a strict requirement). The first line of a Bash script starts with a shebang:\n", "```bash\n", "#!/bin/bash\n", "```\n", "All non-executable comments in a script are prepended with a hash symbol (`#`):\n", "```{admonition} File content:\n", " # For example, we list all the files in the current directory\n", " ls -la\n", " ls -l /etc # Comment. Here, we list files in the /etc directory\n", "```\n", "\n", "## Variables: Read, Echo, Assignment, and Substitution\n", "\n", "Shell scripts contain variables. Variables are created using an assignment operator, for example:\n", "```bash\n", "DIR=/home/hostadm\n", "```\n", "Variable values are referenced using the `$` sign:\n", "```bash\n", "echo $DIR\n", "```\n", "Variables can be updated via the assignment operator:\n", "```bash\n", "DIR=$DIR/KVM\n", "echo $DIR\n", "```\n", "\n", "Variables can also be read from standard output (or keyboard input) by using the `read` command.\n", "\n", "For example, the script `scr1.sh` acts like a calculator:\n", "```{admonition} File content:\n", " #!/bin/bash\n", " echo \"I will work out X*Y\"\n", " echo \"Enter X\"\n", " read X\n", " echo \"Enter Y\"\n", " read Y\n", " echo \"X*Y = $X*$Y = $[X*Y]\"\n", "```\n", "Make the script executable and run it:\n", "```bash\n", "chmod 755 scr1.sh\n", "./scr1.sh\n", "```\n", "\n", "
\n", "\n", "## Variable `$1`\n", "\n", "The variables `$1`, `$2`, `$3`, and so on, capture the first, second, third, and subsequent input parameters passed to the script. For example:\n", "\n", "```bash\n", "script.sh --help\n", "```\n", "\n", "Here, `$1` captures the string `--help` as a value inside `script.sh`.\n", "\n", "***\n", "\n", "## Variables with Commands\n", "\n", "We can assign a command to a variable and then execute it within the shell.\n", "For example, define the variable `cmd` and execute it:\n", "\n", "```bash\n", "cmd=\"ls /etc\"\n", "```\n", "```bash\n", "eval $cmd\n", "```\n", "\n", "Example of a more complex statement: tar the `/etc` directory and copy the resulting tar file into a new directory named `DIR1`:\n", "```bash\n", "my_dir=etc\n", "cmd=\"tar -cf ${my_dir}.tar /${my_dir}; mkdir DIR1; cp ${my_dir}.tar DIR1\"\n", "eval $cmd\n", "```\n", "\n", "***\n", "\n", "\n", "## `if` Conditional Statements\n", "\n", "\n", "- The test square brackets `[ ]` evaluate whether a statement is true or false.\n", "\n", "```{admonition} File content:\n", " if [ condition ]\n", " then\n", " perform a task if condition = true\n", " else\n", " perform a task if condition = false\n", " fi\n", "```\n", "For example:\n", "\n", "```{admonition} File content:\n", " #!/bin/bash\n", "\n", " echo 'Enter the parameter value:'\n", " read parameter\n", "\n", " if [ \"$parameter\" == 'this' ]\n", " then\n", " echo $parameter\n", " else\n", " echo 'Does not match'\n", " fi\n", "```\n", "Another example using nested conditions:\n", "\n", "```{admonition} File content:\n", " #!/bin/bash\n", " X=10\n", " Y=5\n", " if [ \"$X\" -gt \"$Y\" ]; then\n", " echo \"$X is greater than $Y\"\n", " elif [ \"$X\" -lt \"$Y\" ]; then\n", " echo \"$X is less than $Y\"\n", " else\n", " echo \"$X is equal to $Y\"\n", " fi\n", "```\n", "\n", "There are various conditional tests available in Bash. For example, to check if the directory `DIR1` exists:\n", "```bash\n", " if [ -d \"DIR1\" ]\n", "```\n", "To check if the file `ttx.txt` exists:\n", "```bash\n", " if [ -e \"ttx.txt\" ]\n", "```\n", "\n", "You can run the `man` command to view various predefined test options:\n", "```bash\n", "man test\n", "```\n", "\n", "***\n", "## Conditional Command Sequences\n", "\n", "Execute `command2` only if `command1` completes successfully:\n", "\n", "```bash\n", "command1 && command2\n", "```\n", "\n", "Example:\n", "```bash\n", "mkdir testdir\n", "ls testdir && echo \"testdir exists\"\n", "```\n", "Alternatively:\n", "```bash\n", "[ -d testdir ] && echo \"testdir exists\"\n", "```\n", "\n", "Conversely, execute `command2` only if `command1` fails:\n", "\n", "```bash\n", "command1 || command2\n", "```\n", "\n", "Example:\n", "```bash\n", "ls testdir_2 || echo \"testdir_2 doesn't exist\"\n", "```\n", "Alternatively:\n", "```bash\n", "[ -d testdir_2 ] || echo \"testdir_2 doesn't exist\"\n", "```\n", "\n", "***\n", "\n", "## `case` Conditional Statements\n", "\n", "```{admonition} File content:\n", " case $variable in\n", " regex1)\n", " commands1\n", " ;;\n", " regex2)\n", " commands2\n", " ;;\n", " *)\n", " commands3\n", " ;;\n", " esac\n", "```\n", "Where `regex` is a regular expression used to match the variable's value. Use `*)` at the end to catch all remaining unmatched strings.\n", "\n", "```{admonition} File content:\n", " #!/bin/bash\n", " case $1 in\n", " --test|-t)\n", " echo \"you used the --test option\"\n", " exit 0\n", " ;;\n", " --help|-h)\n", " echo \"Usage:\"\n", " echo \" myprog.sh [--test|--help|--version]\"\n", " exit 0\n", " ;;\n", " --version|-v)\n", " echo \"myprog.sh version 0.0.1\"\n", " exit 0\n", " ;;\n", " *)\n", " echo \"No such option $1\"\n", " echo \"Usage:\"\n", " echo \" myprog.sh [--test|--help|--version]\"\n", " exit 1\n", " ;;\n", " esac\n", " echo \"You typed \"$1\" on the command-line\"\n", "```\n", "\n", "Note: Always pay close attention to the correct syntax of `case` statements.\n", "\n", "
\n", "\n", "## Looping with `while` and `until` Statements\n", "\n", "\n", "Script `scr2.sh`:\n", "```{admonition} File content:\n", " #!/bin/bash\n", " N=1\n", " while [ \"$N\" -le \"10\" ]\n", " do\n", " echo \"Number $N\"\n", " N=$[N+1]\n", " done\n", "```\n", "Script `scr3.sh`:\n", "\n", "```{admonition} File content:\n", " #!/bin/bash\n", " N=1\n", " until [ \"$N\" -gt \"10\" ]\n", " do\n", " echo \"Number $N\"\n", " N=$[N+1]\n", " done\n", "```\n", "Note: Common mistakes in shell scripting are usually due to incorrect syntax. For instance, there must be no spaces before and after the `=` assignment operator:\n", "```{admonition} File content:\n", " N=1 # correct\n", " N =1 # error\n", " N= 1 # error\n", " N=$[N+1] # correct\n", " N =$[N+1] # error\n", " N= $[N+1] # error\n", "```\n", "\n", "
\n", "\n", "## Looping with the `for` Statement\n", "\n", "Script `scr4.sh`:\n", "```{admonition} File content:\n", " #!/bin/bash\n", " for i in red white blue\n", " do\n", " echo \"$i is a color\"\n", " done\n", "```\n", "Script `backup-lots.sh`:\n", "```{admonition} File content:\n", " #!/bin/bash\n", " for i in 0 1 2 3 4 5 6 7 8 9 ; do\n", " cp $1 $1.BAK-$i\n", " done\n", "```\n", "\n", "The enumeration in the loop can be represented more concisely using brace expansion, `{0..9}`:\n", "```{admonition} File content:\n", " #!/bin/bash\n", " for i in {0..9} ; do\n", " cp $1 $1.BAK-$i\n", " done\n", "```\n", "\n", "Now create a file named `important_data` with some numbers in it, and then run:\n", "```bash\n", "./backup-lots.sh important_data\n", "```\n", "This will copy the file 10 times with 10 different extensions. As you can see, the variable `$1` has a special meaning: it represents the first argument passed on the command line. Watch out for correct syntax:\n", "```{admonition} File content:\n", " for i in {0..9}\n", " do\n", " ....\n", " done\n", "```\n", "Another correct alternative:\n", "```{admonition} File content:\n", " for i in {0..9} ; do\n", " ....\n", " done\n", "```\n", "\n", "Now modify the script:\n", "```{admonition} File content:\n", " #!/bin/bash\n", " if [ \"$1\" = \"\" ] ; then\n", " echo \"Usage: backup-lots.sh \"\n", " exit\n", " fi\n", " for i in {0..9} ; do\n", " NEW_FILE=$1.BAK-$i\n", " if [ -e $NEW_FILE ]; then\n", " echo \"backup-lots.sh: **warning** $NEW_FILE\"\n", " echo \" already exists - skipping\"\n", " else\n", " cp $1 $NEW_FILE\n", " fi\n", " done\n", "```\n", "Looping over glob expressions:\n", "```{admonition} File content:\n", " #!/bin/bash\n", " for i in *.txt ; do\n", " echo \"found a file:\" $i\n", " done\n", "```\n", "Note that the script above loops over all `.txt` files in the current directory, so you will need to create several files with a `.txt` extension to test this exercise.\n", "```{admonition} File content:\n", " #!/bin/bash\n", " for i in /usr/share/*/*.txt ; do\n", " echo \"found a file:\" $i\n", " done\n", "```\n", "\n", "Breaking out of loops and continuing:\n", "```{admonition} File content:\n", " #!/bin/bash\n", " for i in {0..9} ; do\n", " NEW_FILE=$1.BAK-$i\n", " if [ -e $NEW_FILE ]; then\n", " echo \"backup-lots.sh: **error** $NEW_FILE\"\n", " echo \" already exists - exiting\"\n", " break\n", " else\n", " cp $1 $NEW_FILE\n", " fi\n", " done\n", "```\n", "The `break` command causes program execution to continue on the line immediately following the `done` statement.\n", "\n", "The `continue` statement is useful for terminating the current iteration of a loop and proceeding to the next:\n", "```{admonition} File content:\n", " #!/bin/bash\n", " for i in {0..9} ; do\n", " NEW_FILE=$1.BAK-$i\n", " if [ -e $NEW_FILE ] ; then\n", " echo \"backup-lots.sh: **warning** $NEW_FILE\"\n", " echo \" already exists - skipping\"\n", " continue\n", " fi\n", " cp $1 $NEW_FILE\n", " done\n", "```\n", "\n", "
\n", "\n", "## Functions\n", "\n", "Function definitions provide a way to group statement blocks into a single reusable unit:\n", "```{admonition} File content:\n", " function usage ()\n", " {\n", " command1\n", " command2; command3\n", " ......\n", " }\n", "```\n", "The word `function` is optional. The following syntax works as well:\n", "```{admonition} File content:\n", " usage ()\n", " {\n", " command1\n", " command2; command3\n", " ......\n", " }\n", "```\n", "\n", "For example, create a script named `myprog.sh` with the following content:\n", "```{admonition} File content:\n", " #!/bin/bash\n", " function usage ()\n", " {\n", " echo \"Usage:\"\n", " echo \" myprog.sh [--test|--help|--version]\"\n", " }\n", " case $1 in\n", " --test|-t)\n", " echo \"you used the --test option\"\n", " exit 0\n", " ;;\n", " --help|-h)\n", " usage\n", " ;;\n", " --version|-v)\n", " echo \"myprog.sh version 0.0.2\"\n", " exit 0\n", " ;;\n", " -*)\n", " echo \"Error: no such option $1\"\n", " usage\n", " exit 1\n", " ;;\n", " esac\n", " echo \"You typed \"$1\" on the command-line\"\n", "```\n", "\n", "
\n", "\n", "## Using Quotes\n", "\n", "Single forward quotes (`'`) protect the enclosed text from shell interpretation:\n", "```bash\n", "echo 'error $?'\n", "echo 'shell name $0'\n", "```\n", "\n", "Double quotes (`\"`) allow all shell interpretations and variable expansions to take place:\n", "```bash\n", "echo \"error $?\" # outputs the error code of the last command\n", "echo \"shell name $0\" # outputs the current shell name\n", "```\n", "\n", "Command substitution:\n", "```bash\n", "X=`expr 100 + 50 '*' 3`\n", "echo $X\n", "```\n", "\n", "Assigning command output to a variable:\n", "```bash\n", "FSIZE=`wc -l /etc/profile`\n", "```\n", "which is equivalent to:\n", "```bash\n", "FSIZE=$(wc -l /etc/profile)\n", "```\n", "\n", "
\n", "\n", "## Introduction to `awk`\n", "\n", "The basic function of `awk` is to search files for lines or text units containing one or more patterns. When a line matches a pattern, specified actions are performed on it. \n", "`awk` is very frequently used to extract specific columns from text based on matching criteria.\n", "\n", "`awk` reads data from a file or a pipe using the following syntax:\n", "\n", "```bash\n", "awk 'EXPRESSION { PROGRAM }' file(s)\n", "```\n", "or\n", "```bash\n", "COMMAND | awk 'EXPRESSION { PROGRAM }'\n", "```\n", "\n", "To display usernames from `/etc/passwd` (field 1):\n", "```bash\n", "awk -F: '{ print $1 }' /etc/passwd\n", "```\n", "Where `-F:` specifies the field separator in the passwd file. The fields are separated by colons (`:`).\n", "
\n", "The default field separator in `awk` is a space or tab. `awk` scans the input file and splits each input line into fields accordingly.\n", "
\n", "Similarly:\n", "```bash\n", "cat /etc/passwd | awk -F: '{ print $1 }'\n", "```\n", "\n", "To display usernames, home directories, and login shells (fields 1, 6, and 7) and store them in a separate file named `users.txt`:\n", "```bash\n", "awk -F: '{ print $1, $6, $7 }' /etc/passwd > users.txt\n", "```\n", "or\n", "```bash\n", "cat /etc/passwd | awk -F: '{ print $1, $6, $7 }' > users.txt\n", "```\n", "\n", "Since the default field separator becomes an empty space when reading `users.txt`, we can print usernames from it using:\n", "```bash\n", "awk '{ print $1 }' users.txt\n", "```\n", "\n", "To extract only the columns of lines containing a specific pattern, such as `hostadm`:\n", "```bash\n", "awk '/hostadm/ { print $1 }' users.txt\n", "```\n", "\n", "We can use regular expressions inside the selection blocks (`/ /`). For example, to view permissions on all files in the `/etc` directory with a `.conf` extension:\n", "```bash\n", "ls -l /etc | awk '/.*\\.conf/ { print $1, $9 }'\n", "```\n", "\n", "In the `print` command block, we can include custom strings for formatted output:\n", "```bash\n", "ls -l /etc | awk '/.*\\.conf/ { print \"Permission:\", $1, \"File:\", $9 }'\n", "```\n", "\n", "Adding a conditional statement, such as printing files larger than 5000 bytes:\n", "```bash\n", "ls -l /etc | awk '/.*\\.conf/ { if ( $5 > 5000 ) print \"Permission:\", $1, \"Size\", $5, \"File:\", $9 }'\n", "```\n", "\n", "`BEGIN` and `END` blocks allow you to print statements before and after the main `awk` processing output:\n", "```bash\n", "ls -l /etc | awk 'BEGIN { print \"Conf Files > 5K found:\\n\" } /.*\\.conf/ { if ( $5 > 5000 ) print \"Permission:\", $1, \"Size\", $5, \"File:\", $9 } END { print \"\\n ---- The end -------\" }'\n", "```\n", "\n", "[Recommended tutorial: The GNU awk programming language](https://linuxcourse.rutgers.edu/2024/html/documents/Bash-Beginners-Guide/chap_06.html)\n", "
\n", "\n", "## Introduction to `grep`\n", "\n", "`grep` is used to search files or standard input for lines containing required patterns.\n", "\n", "We will work with a text file named `list.txt` containing the following content:\n", "\n", "```{admonition} File content:\n", " Check the inode list today\n", " reboot the machine tomorrow\n", " Reboot it again in a week\n", " Call Tech support in case of emergency.\n", " tel: 834\n", "\n", " Oop 0\n", " Oops 1\n", " Oopss 12\n", " Oopsss 123\n", " Oopssss 1234\n", " End\n", "```\n", "\n", "To find lines containing the string \"inode\" in `list.txt`:\n", "```bash\n", "grep inode list.txt\n", "```\n", "To find lines containing \"inode lis \" in `list.txt`:\n", "```bash\n", "grep \"inode lis \" list.txt\n", "```\n", "This will return nothing because that exact string does not exist in the file.\n", "\n", "To search for lines containing \"inode list\" across all files in the current directory:\n", "```bash\n", "grep \"inode list\" *\n", "```\n", "\n", "The syntax for `grep` is:\n", "```bash\n", "grep [options] regex [files]\n", "```\n", "where `regex` represents regular expressions.\n", "\n", "### Using Regular Expressions\n", "\n", "Regular expressions consist of literals (plain text) and metacharacters (special control characters). They allow you to specify three core concepts: `position anchors`, `groups and ranges`, and `quantity modifiers`.\n", "\n", "```{admonition} Anchors: \n", " ^ - matches at the beginning of a line\n", " $ - matches at the end of a line\n", "```\n", "```bash\n", "grep '^Call' list.txt\n", "grep '^ Reboot' list.txt\n", "grep 'today$' list.txt\n", "```\n", "\n", "To count the number of empty lines:\n", "```bash\n", "grep -c '^$' list.txt\n", "```\n", "To display all lines containing only the word \"End\" by itself:\n", "```bash\n", "grep '^End$' list.txt\n", "```\n", "\n", "```{admonition} Groups and ranges: \n", " [abc] - matches any single character from a, b, or c\n", " [a-e] - matches any single character within the range a to e\n", " [^abc] - inverse match; matches any single character except a, b, or c\n", " [^a-e] - inverse match; matches any single character outside the range a to e\n", " \\ - matches the exact word (option -w can also be used)\n", " . - matches any single character except a newline\n", " \\ - turns off the special meaning of the following character\n", "```\n", "```bash\n", "grep '[Rr]eboot' list.txt\n", "grep '\\<[Rr]eboot\\>' list.txt\n", "```\n", "\n", "Display all lines from `list.txt` containing three adjacent digits:\n", "```bash\n", "grep '[0-9][0-9][0-9]' list.txt\n", "```\n", "Display lines containing four or more consecutive characters:\n", "```bash\n", "grep '....' list.txt\n", "```\n", "Display all non-blank lines from `list.txt`:\n", "```bash\n", "grep '.' list.txt\n", "```\n", "Display all lines containing a literal period:\n", "```bash\n", "grep '\\.' list.txt\n", "```\n", "\n", "```{admonition} Modifiers: \n", " * - matches zero or more instances of the preceding single character\n", " ? - matches zero or one instance of the preceding regex (implies the 'grep -E' option)\n", " + - matches one or more instances of the preceding regex (implies the 'grep -E' option)\n", " \\{n,m\\} - matches a range of occurrences; \\{n\\} matches exactly n occurrences; \\{n,\\} matches at least n occurrences\n", " | - matches either the regex specified before or after the vertical bar (implies the 'grep -E' option)\n", "```\n", "\n", "Display all lines from `list.txt` containing \"Oop\", \"Oops\", \"Oopss\", etc.:\n", "```bash\n", "grep 'Oops*' list.txt\n", "```\n", "Display all lines containing \"Oops\", \"Oopss\", etc.:\n", "```bash\n", "grep 'Oopss*' list.txt\n", "``` \n", "Display lines containing two or more adjacent digits:\n", "```bash\n", "grep '[0-9][0-9][0-9]*' list.txt\n", "```\n", "Display lines containing the number '3' or the combination '34':\n", "```bash\n", "grep -E '34?' list.txt\n", "```\n", "Display lines containing at least one digit:\n", "```bash\n", "grep -E '[0-9]+' list.txt\n", "```\n", "Display lines containing \"sss\" and \"ssss\":\n", "```bash\n", "grep 's\\{3,4\\}' list.txt\n", "```\n", "Display lines containing any 3, 4, or 5-digit numbers:\n", "```bash\n", "grep '[0-9]\\{3,5\\}' list.txt\n", "```\n", "Display lines containing \"Reboot\", \"reboot\", or \"support\":\n", "```bash\n", "grep -E '[Rr]eboot|support' list.txt\n", "```\n", "Display lines containing any letter (no empty lines):\n", "```bash\n", "grep '[A-Za-z]' list.txt\n", "```\n", "Display lines containing any non-alphanumeric and non-space symbol:\n", "```bash\n", "grep '[^ 0-9A-Za-z]' list.txt\n", "```\n", "Display lines containing an uppercase letter followed by zero or more lowercase letters:\n", "```bash\n", "grep '[A-Z][a-z]*' list.txt\n", "```\n", "Display lines containing a 3-digit telephone number:\n", "```bash\n", "grep 'tel: [0-9]\\{3\\}' list.txt\n", "```\n", "\n", "[Recommended tutorial on Regular expressions and grep](http://linuxcourse.rutgers.edu/2024/html/documents/Bash-Beginners-Guide/chap_04.html)\n", "\n", "
\n", "\n", "## Introduction to `sed`\n", "\n", "The stream editor, `sed`, is used for editing lines in a file or a stream. Its output goes to standard output and can be redirected to a new file.\n", "Syntax:\n", "```bash\n", "sed [options] 'command1' [files]\n", "sed [options] -e 'command1' [-e command2 ...] [files]\n", "sed [options] -f script [files]\n", "```\n", "\n", "To print lines 3 through 5 from `list.txt`:\n", "```bash\n", "sed -n '3,5p' list.txt\n", "```\n", "To delete lines 3 through 5 from `list.txt`:\n", "```bash\n", "sed '3,5d' list.txt\n", "```\n", "\n", "We can use regular expressions enclosed in `/ /` to target specific blocks for commands.\n", "\n", "For example, to delete lines starting with \"O\":\n", "```bash\n", "sed '/^O/d' list.txt\n", "```\n", "To delete empty lines:\n", "```bash\n", "sed '/^$/d' list.txt\n", "```\n", "To translate capital letters `C`, `R`, and `O` into lowercase `c`, `r`, and `o`:\n", "```bash\n", "sed 'y/CRO/cro/' list.txt\n", "```\n", "To replace the first occurrence of \"Oop\" with \"Wee\" on each line:\n", "```bash\n", "sed 's/Oop/Wee/' list.txt\n", "```\n", "To remove the string \"ss\" (replace with an empty entry) for the first occurrence on a line:\n", "```bash\n", "sed 's/ss//' list.txt\n", "```\n", "To remove all occurrences of \"ss\" on a line:\n", "```bash\n", "sed 's/ss//g' list.txt\n", "```\n", "To substitute a single space for any sequence of multiple spaces on a line:\n", "```bash\n", "sed 's/ */ /g' list.txt\n", "```\n", "To substitute an underscore for any sequence of multiple spaces:\n", "```bash\n", "sed 's/ */_/g' list.txt\n", "```\n", "\n", "To make `sed` modify the original file content directly, use the `-i` (in-place) option.\n", "For example, to apply changes to `list2.txt`:\n", "```bash\n", "cp list.txt list2.txt\n", "sed -i 's/ */_/g' list2.txt\n", "```\n", "\n", "[More examples: Useful one-line scripts for sed](http://linuxcourse.rutgers.edu/2024/html/documents/sed1line.txt)\n", "\n", "[Recommended tutorial: The GNU sed stream editor](http://linuxcourse.rutgers.edu/2024/html/documents/Bash-Beginners-Guide/chap_05.html)\n", "\n", "## Printing Lines Containing a Matching Pattern Using `grep`, `awk`, and `sed`\n", "\n", "The following three commands all print out the exact same matching line from `list.txt`:\n", "\n", "```bash\n", "grep Reboot list.txt\n", "\n", "awk '/Reboot/ { print $0 }' list.txt\n", "\n", "sed -n '/Reboot/p' list.txt\n", "```\n" ] } ], "source": [ "import os\n", "import time\n", "import base64 \n", "from google import genai\n", "from google.genai.errors import APIError\n", "\n", "os.environ[\"GEMINI_API_KEY\"] = \"AQ.Ab8RN6K9G5S-h6IWgpUgfmHbIk7b8iZlWTuYptMFqOT4Ph6FnQ\"\n", "client = genai.Client()\n", "\n", "# Use triple quotes \"\" \" so you can paste long paragraphs easily!\n", "\n", "prompt_instructions_simple = f\"Rewrite this text for a university course. Make it clear and write WCAG 2.1 AA compliant alt text descriptions for any charts mentioned: {my_notebook_text}\"\n", " \n", "\n", "\n", "\n", "prompt_instructions_detailed5 = (\n", " \"You are processing raw course text meant for Jupyter Notebook markdown cells.\\n\\n\"\n", " \n", " \"CRITICAL RULE: Do NOT delete, alter, or remove any original image links, image paths, \"\n", " \"or image tags (e.g., lines starting with `![...](...)` or containing ``). \"\n", " \"Leave them exactly where they are in the original text sequence.\\n\\n\"\n", " \n", " \"INSTRUCTION:\\n\"\n", " \"Rewrite the text to make it clear and university-appropriate.Make it clear and write WCAG 2.1 AA compliant. Use the following strict sequence \"\n", " \"when dealing with image placements:\\n\\n\"\n", " \n", " \"1. First, insert the NEW Text-Based Structured Alternative ABOVE the image:\\n\"\n", " \" - For numerical data charts or grid plots: Provide a clean Markdown Data Table.\\n\"\n", " \" - For hardware architecture, network layouts, or OS Process lifecycles (like fork(), process states, execution threads): \"\n", " \"Provide a high-level text flowchart inside a preformatted ```text code block.\\n\"\n", " \" - For Terminal Commands or Pipelines (like stdin/stdout piping, shell interactions): \"\n", " \"Provide a vertical box-and-arrow sequence map illustrating data flow from one utility to the next.\\n\"\n", " \" - Keep diagram characters perfectly aligned and compatible with static HTML page exports.\\n\\n\"\n", " \" - CRITICAL RULE FOR VIRTUAL DESKTOP LOGIN CONTEXT:\\n\"\n", " \" If the text or diagram is about logging into the Virtual Desktop or using Ravada, \"\n", " \"the text diagram sequence MUST start exactly with this workflow path:\\n\"\n", " \" [Web access to Ravada portal with authentication] ===> [Start VM] ===> [Access VM via SSH with provided port]\\n\\n\"\n", " \n", " \"2. Second, place the ORIGINAL untouched image link exactly as it was.\\n\\n\"\n", " \n", " \"3. Third, immediately BELOW the original image link, add the Comprehensive Detailed Description:\\n\"\n", " \"Inside this caption block, you MUST start with the exact text '**Accessible Image Description:**'. Place it always under the image.\"\n", " \" - Overview: One sentence describing what the visual shows at a high level.\\n\"\n", " \" - Setup: Explain the primary components, labels, axes, or code pathways without hyper-granular specs.\\n\"\n", " \" - Key Takeaways: Core conceptual trends or execution results shown.\\n\\n\"\n", "\n", " f\"Course Text:\\n{my_notebook_text}\"\n", ")\n", "prompt_instructions = prompt_instructions_detailed5\n", "\n", "# Trying the 2.5-flash model which often has lower server congestion\n", "try:\n", " response = client.models.generate_content(\n", " model='models/gemini-3.5-flash-lite',\n", " contents= prompt_instructions,\n", " )\n", "\n", " #print(response.text)\n", "\n", " output_text = response.text\n", " print(output_text)\n", " \n", " # --- THE MAGIC TRICK ---\n", " # This forces Jupyter to create a new cell below and paste the AI response inside it!\n", " #get_ipython().set_next_input(response.text)\n", " \n", "except APIError as e:\n", " if e.code == 503:\n", " print(\"Servers are still busy. Waiting 5 seconds before a quick automatic retry...\")\n", " time.sleep(5)\n", " # Automatic second attempt\n", " response = client.models.generate_content(\n", " model='models/gemini-3.5-flash-lite',\n", " contents= prompt_instructions,\n", " )\n", " print(response.text)\n", " # --- THE MAGIC TRICK ---\n", " # This forces Jupyter to create a new cell below and paste the AI response inside it!\n", " #get_ipython().set_next_input(response.text)\n", " else:\n", " raise e" ] }, { "cell_type": "markdown", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "# Shell Scripts\n", "\n", "What actually makes a shell script:\n", "\n", "- Put a group of commands in a file, then make the file executable.\n", "- In the executable file, apply the shell syntax to add variables, conditionals, iterations.\n", "- Create functions for the frequently used blocks. \n", "- Utilize ```awk```, ```grep```, and ```sed``` languages with regular expressions for string parsing and substitution. \n", "\n", "## Shell Scripting Exercises\n", "\n", "\n", "- Create a directory for shell scripting exercises.\n", "- A shell script and executable permissions.\n", "- `if` and `case` statements.\n", "- Loops with `while` and `until` conditional statements.\n", "- `for` loop.\n", "- Functions in shell scripts.\n", "- `awk` programming language.\n", "- `grep` line search filter and regular expressions.\n", "- String editor `sed`.\n", "- Advantages and disadvantages of shell scripts.\n", "\n", "[References: for details, consult the Bash Guide for Beginners](http://linuxcourse.rutgers.edu/2024/html/documents/Bash-Beginners-Guide)\n", "
\n", "\n", "## Creating a Shell Script\n", "\n", "SSH into your LXC container.\n", "\n", "Create a new directory named `bash_scripts` where you will run the shell scripting exercises:\n", "```bash \n", "mkdir bash_scripts\n", "cd bash_scripts\n", "```\n", "\n", "Note: It is recommended to create a new script file for every exercise, make them executable, and give them names ending with the `.sh` extension (this is a convention rather than a strict requirement). The first line of a Bash script starts with a shebang:\n", "```bash\n", "#!/bin/bash\n", "```\n", "All non-executable comments in a script are prepended with a hash symbol (`#`):\n", "```{admonition} File content:\n", " # For example, we list all the files in the current directory\n", " ls -la\n", " ls -l /etc # Comment. Here, we list files in the /etc directory\n", "```\n", "\n", "## Variables: Read, Echo, Assignment, and Substitution\n", "\n", "Shell scripts contain variables. Variables are created using an assignment operator, for example:\n", "```bash\n", "DIR=/home/hostadm\n", "```\n", "Variable values are referenced using the `$` sign:\n", "```bash\n", "echo $DIR\n", "```\n", "Variables can be updated via the assignment operator:\n", "```bash\n", "DIR=$DIR/KVM\n", "echo $DIR\n", "```\n", "\n", "Variables can also be read from standard output (or keyboard input) by using the `read` command.\n", "\n", "For example, the script `scr1.sh` acts like a calculator:\n", "```{admonition} File content:\n", " #!/bin/bash\n", " echo \"I will work out X*Y\"\n", " echo \"Enter X\"\n", " read X\n", " echo \"Enter Y\"\n", " read Y\n", " echo \"X*Y = $X*$Y = $[X*Y]\"\n", "```\n", "Make the script executable and run it:\n", "```bash\n", "chmod 755 scr1.sh\n", "./scr1.sh\n", "```\n", "\n", "
\n", "\n", "## Variable `$1`\n", "\n", "The variables `$1`, `$2`, `$3`, and so on, capture the first, second, third, and subsequent input parameters passed to the script. For example:\n", "\n", "```bash\n", "script.sh --help\n", "```\n", "\n", "Here, `$1` captures the string `--help` as a value inside `script.sh`.\n", "\n", "***\n", "\n", "## Variables with Commands\n", "\n", "We can assign a command to a variable and then execute it within the shell.\n", "For example, define the variable `cmd` and execute it:\n", "\n", "```bash\n", "cmd=\"ls /etc\"\n", "```\n", "```bash\n", "eval $cmd\n", "```\n", "\n", "Example of a more complex statement: tar the `/etc` directory and copy the resulting tar file into a new directory named `DIR1`:\n", "```bash\n", "my_dir=etc\n", "cmd=\"tar -cf ${my_dir}.tar /${my_dir}; mkdir DIR1; cp ${my_dir}.tar DIR1\"\n", "eval $cmd\n", "```\n", "\n", "***\n", "\n", "\n", "## `if` Conditional Statements\n", "\n", "\n", "- The test square brackets `[ ]` evaluate whether a statement is true or false.\n", "\n", "```{admonition} File content:\n", " if [ condition ]\n", " then\n", " perform a task if condition = true\n", " else\n", " perform a task if condition = false\n", " fi\n", "```\n", "For example:\n", "\n", "```{admonition} File content:\n", " #!/bin/bash\n", "\n", " echo 'Enter the parameter value:'\n", " read parameter\n", "\n", " if [ \"$parameter\" == 'this' ]\n", " then\n", " echo $parameter\n", " else\n", " echo 'Does not match'\n", " fi\n", "```\n", "Another example using nested conditions:\n", "\n", "```{admonition} File content:\n", " #!/bin/bash\n", " X=10\n", " Y=5\n", " if [ \"$X\" -gt \"$Y\" ]; then\n", " echo \"$X is greater than $Y\"\n", " elif [ \"$X\" -lt \"$Y\" ]; then\n", " echo \"$X is less than $Y\"\n", " else\n", " echo \"$X is equal to $Y\"\n", " fi\n", "```\n", "\n", "There are various conditional tests available in Bash. For example, to check if the directory `DIR1` exists:\n", "```bash\n", " if [ -d \"DIR1\" ]\n", "```\n", "To check if the file `ttx.txt` exists:\n", "```bash\n", " if [ -e \"ttx.txt\" ]\n", "```\n", "\n", "You can run the `man` command to view various predefined test options:\n", "```bash\n", "man test\n", "```\n", "\n", "***\n", "## Conditional Command Sequences\n", "\n", "Execute `command2` only if `command1` completes successfully:\n", "\n", "```bash\n", "command1 && command2\n", "```\n", "\n", "Example:\n", "```bash\n", "mkdir testdir\n", "ls testdir && echo \"testdir exists\"\n", "```\n", "Alternatively:\n", "```bash\n", "[ -d testdir ] && echo \"testdir exists\"\n", "```\n", "\n", "Conversely, execute `command2` only if `command1` fails:\n", "\n", "```bash\n", "command1 || command2\n", "```\n", "\n", "Example:\n", "```bash\n", "ls testdir_2 || echo \"testdir_2 doesn't exist\"\n", "```\n", "Alternatively:\n", "```bash\n", "[ -d testdir_2 ] || echo \"testdir_2 doesn't exist\"\n", "```\n", "\n", "***\n", "\n", "## `case` Conditional Statements\n", "\n", "```{admonition} File content:\n", " case $variable in\n", " regex1)\n", " commands1\n", " ;;\n", " regex2)\n", " commands2\n", " ;;\n", " *)\n", " commands3\n", " ;;\n", " esac\n", "```\n", "Where `regex` is a regular expression used to match the variable's value. Use `*)` at the end to catch all remaining unmatched strings.\n", "\n", "```{admonition} File content:\n", " #!/bin/bash\n", " case $1 in\n", " --test|-t)\n", " echo \"you used the --test option\"\n", " exit 0\n", " ;;\n", " --help|-h)\n", " echo \"Usage:\"\n", " echo \" myprog.sh [--test|--help|--version]\"\n", " exit 0\n", " ;;\n", " --version|-v)\n", " echo \"myprog.sh version 0.0.1\"\n", " exit 0\n", " ;;\n", " *)\n", " echo \"No such option $1\"\n", " echo \"Usage:\"\n", " echo \" myprog.sh [--test|--help|--version]\"\n", " exit 1\n", " ;;\n", " esac\n", " echo \"You typed \"$1\" on the command-line\"\n", "```\n", "\n", "Note: Always pay close attention to the correct syntax of `case` statements.\n", "\n", "
\n", "\n", "## Looping with `while` and `until` Statements\n", "\n", "\n", "Script `scr2.sh`:\n", "```{admonition} File content:\n", " #!/bin/bash\n", " N=1\n", " while [ \"$N\" -le \"10\" ]\n", " do\n", " echo \"Number $N\"\n", " N=$[N+1]\n", " done\n", "```\n", "Script `scr3.sh`:\n", "\n", "```{admonition} File content:\n", " #!/bin/bash\n", " N=1\n", " until [ \"$N\" -gt \"10\" ]\n", " do\n", " echo \"Number $N\"\n", " N=$[N+1]\n", " done\n", "```\n", "Note: Common mistakes in shell scripting are usually due to incorrect syntax. For instance, there must be no spaces before and after the `=` assignment operator:\n", "```{admonition} File content:\n", " N=1 # correct\n", " N =1 # error\n", " N= 1 # error\n", " N=$[N+1] # correct\n", " N =$[N+1] # error\n", " N= $[N+1] # error\n", "```\n", "\n", "
\n", "\n", "## Looping with the `for` Statement\n", "\n", "Script `scr4.sh`:\n", "```{admonition} File content:\n", " #!/bin/bash\n", " for i in red white blue\n", " do\n", " echo \"$i is a color\"\n", " done\n", "```\n", "Script `backup-lots.sh`:\n", "```{admonition} File content:\n", " #!/bin/bash\n", " for i in 0 1 2 3 4 5 6 7 8 9 ; do\n", " cp $1 $1.BAK-$i\n", " done\n", "```\n", "\n", "The enumeration in the loop can be represented more concisely using brace expansion, `{0..9}`:\n", "```{admonition} File content:\n", " #!/bin/bash\n", " for i in {0..9} ; do\n", " cp $1 $1.BAK-$i\n", " done\n", "```\n", "\n", "Now create a file named `important_data` with some numbers in it, and then run:\n", "```bash\n", "./backup-lots.sh important_data\n", "```\n", "This will copy the file 10 times with 10 different extensions. As you can see, the variable `$1` has a special meaning: it represents the first argument passed on the command line. Watch out for correct syntax:\n", "```{admonition} File content:\n", " for i in {0..9}\n", " do\n", " ....\n", " done\n", "```\n", "Another correct alternative:\n", "```{admonition} File content:\n", " for i in {0..9} ; do\n", " ....\n", " done\n", "```\n", "\n", "Now modify the script:\n", "```{admonition} File content:\n", " #!/bin/bash\n", " if [ \"$1\" = \"\" ] ; then\n", " echo \"Usage: backup-lots.sh \"\n", " exit\n", " fi\n", " for i in {0..9} ; do\n", " NEW_FILE=$1.BAK-$i\n", " if [ -e $NEW_FILE ]; then\n", " echo \"backup-lots.sh: **warning** $NEW_FILE\"\n", " echo \" already exists - skipping\"\n", " else\n", " cp $1 $NEW_FILE\n", " fi\n", " done\n", "```\n", "Looping over glob expressions:\n", "```{admonition} File content:\n", " #!/bin/bash\n", " for i in *.txt ; do\n", " echo \"found a file:\" $i\n", " done\n", "```\n", "Note that the script above loops over all `.txt` files in the current directory, so you will need to create several files with a `.txt` extension to test this exercise.\n", "```{admonition} File content:\n", " #!/bin/bash\n", " for i in /usr/share/*/*.txt ; do\n", " echo \"found a file:\" $i\n", " done\n", "```\n", "\n", "Breaking out of loops and continuing:\n", "```{admonition} File content:\n", " #!/bin/bash\n", " for i in {0..9} ; do\n", " NEW_FILE=$1.BAK-$i\n", " if [ -e $NEW_FILE ]; then\n", " echo \"backup-lots.sh: **error** $NEW_FILE\"\n", " echo \" already exists - exiting\"\n", " break\n", " else\n", " cp $1 $NEW_FILE\n", " fi\n", " done\n", "```\n", "The `break` command causes program execution to continue on the line immediately following the `done` statement.\n", "\n", "The `continue` statement is useful for terminating the current iteration of a loop and proceeding to the next:\n", "```{admonition} File content:\n", " #!/bin/bash\n", " for i in {0..9} ; do\n", " NEW_FILE=$1.BAK-$i\n", " if [ -e $NEW_FILE ] ; then\n", " echo \"backup-lots.sh: **warning** $NEW_FILE\"\n", " echo \" already exists - skipping\"\n", " continue\n", " fi\n", " cp $1 $NEW_FILE\n", " done\n", "```\n", "\n", "
\n", "\n", "## Functions\n", "\n", "Function definitions provide a way to group statement blocks into a single reusable unit:\n", "```{admonition} File content:\n", " function usage ()\n", " {\n", " command1\n", " command2; command3\n", " ......\n", " }\n", "```\n", "The word `function` is optional. The following syntax works as well:\n", "```{admonition} File content:\n", " usage ()\n", " {\n", " command1\n", " command2; command3\n", " ......\n", " }\n", "```\n", "\n", "For example, create a script named `myprog.sh` with the following content:\n", "```{admonition} File content:\n", " #!/bin/bash\n", " function usage ()\n", " {\n", " echo \"Usage:\"\n", " echo \" myprog.sh [--test|--help|--version]\"\n", " }\n", " case $1 in\n", " --test|-t)\n", " echo \"you used the --test option\"\n", " exit 0\n", " ;;\n", " --help|-h)\n", " usage\n", " ;;\n", " --version|-v)\n", " echo \"myprog.sh version 0.0.2\"\n", " exit 0\n", " ;;\n", " -*)\n", " echo \"Error: no such option $1\"\n", " usage\n", " exit 1\n", " ;;\n", " esac\n", " echo \"You typed \"$1\" on the command-line\"\n", "```\n", "\n", "
\n", "\n", "## Using Quotes\n", "\n", "Single forward quotes (`'`) protect the enclosed text from shell interpretation:\n", "```bash\n", "echo 'error $?'\n", "echo 'shell name $0'\n", "```\n", "\n", "Double quotes (`\"`) allow all shell interpretations and variable expansions to take place:\n", "```bash\n", "echo \"error $?\" # outputs the error code of the last command\n", "echo \"shell name $0\" # outputs the current shell name\n", "```\n", "\n", "Command substitution:\n", "```bash\n", "X=`expr 100 + 50 '*' 3`\n", "echo $X\n", "```\n", "\n", "Assigning command output to a variable:\n", "```bash\n", "FSIZE=`wc -l /etc/profile`\n", "```\n", "which is equivalent to:\n", "```bash\n", "FSIZE=$(wc -l /etc/profile)\n", "```\n", "\n", "
\n", "\n", "## Introduction to `awk`\n", "\n", "The basic function of `awk` is to search files for lines or text units containing one or more patterns. When a line matches a pattern, specified actions are performed on it. \n", "`awk` is very frequently used to extract specific columns from text based on matching criteria.\n", "\n", "`awk` reads data from a file or a pipe using the following syntax:\n", "\n", "```bash\n", "awk 'EXPRESSION { PROGRAM }' file(s)\n", "```\n", "or\n", "```bash\n", "COMMAND | awk 'EXPRESSION { PROGRAM }'\n", "```\n", "\n", "To display usernames from `/etc/passwd` (field 1):\n", "```bash\n", "awk -F: '{ print $1 }' /etc/passwd\n", "```\n", "Where `-F:` specifies the field separator in the passwd file. The fields are separated by colons (`:`).\n", "
\n", "The default field separator in `awk` is a space or tab. `awk` scans the input file and splits each input line into fields accordingly.\n", "
\n", "Similarly:\n", "```bash\n", "cat /etc/passwd | awk -F: '{ print $1 }'\n", "```\n", "\n", "To display usernames, home directories, and login shells (fields 1, 6, and 7) and store them in a separate file named `users.txt`:\n", "```bash\n", "awk -F: '{ print $1, $6, $7 }' /etc/passwd > users.txt\n", "```\n", "or\n", "```bash\n", "cat /etc/passwd | awk -F: '{ print $1, $6, $7 }' > users.txt\n", "```\n", "\n", "Since the default field separator becomes an empty space when reading `users.txt`, we can print usernames from it using:\n", "```bash\n", "awk '{ print $1 }' users.txt\n", "```\n", "\n", "To extract only the columns of lines containing a specific pattern, such as `hostadm`:\n", "```bash\n", "awk '/hostadm/ { print $1 }' users.txt\n", "```\n", "\n", "We can use regular expressions inside the selection blocks (`/ /`). For example, to view permissions on all files in the `/etc` directory with a `.conf` extension:\n", "```bash\n", "ls -l /etc | awk '/.*\\.conf/ { print $1, $9 }'\n", "```\n", "\n", "In the `print` command block, we can include custom strings for formatted output:\n", "```bash\n", "ls -l /etc | awk '/.*\\.conf/ { print \"Permission:\", $1, \"File:\", $9 }'\n", "```\n", "\n", "Adding a conditional statement, such as printing files larger than 5000 bytes:\n", "```bash\n", "ls -l /etc | awk '/.*\\.conf/ { if ( $5 > 5000 ) print \"Permission:\", $1, \"Size\", $5, \"File:\", $9 }'\n", "```\n", "\n", "`BEGIN` and `END` blocks allow you to print statements before and after the main `awk` processing output:\n", "```bash\n", "ls -l /etc | awk 'BEGIN { print \"Conf Files > 5K found:\\n\" } /.*\\.conf/ { if ( $5 > 5000 ) print \"Permission:\", $1, \"Size\", $5, \"File:\", $9 } END { print \"\\n ---- The end -------\" }'\n", "```\n", "\n", "[Recommended tutorial: The GNU awk programming language](https://linuxcourse.rutgers.edu/2024/html/documents/Bash-Beginners-Guide/chap_06.html)\n", "
\n", "\n", "## Introduction to `grep`\n", "\n", "`grep` is used to search files or standard input for lines containing required patterns.\n", "\n", "We will work with a text file named `list.txt` containing the following content:\n", "\n", "```{admonition} File content:\n", " Check the inode list today\n", " reboot the machine tomorrow\n", " Reboot it again in a week\n", " Call Tech support in case of emergency.\n", " tel: 834\n", "\n", " Oop 0\n", " Oops 1\n", " Oopss 12\n", " Oopsss 123\n", " Oopssss 1234\n", " End\n", "```\n", "\n", "To find lines containing the string \"inode\" in `list.txt`:\n", "```bash\n", "grep inode list.txt\n", "```\n", "To find lines containing \"inode lis \" in `list.txt`:\n", "```bash\n", "grep \"inode lis \" list.txt\n", "```\n", "This will return nothing because that exact string does not exist in the file.\n", "\n", "To search for lines containing \"inode list\" across all files in the current directory:\n", "```bash\n", "grep \"inode list\" *\n", "```\n", "\n", "The syntax for `grep` is:\n", "```bash\n", "grep [options] regex [files]\n", "```\n", "where `regex` represents regular expressions.\n", "\n", "### Using Regular Expressions\n", "\n", "Regular expressions consist of literals (plain text) and metacharacters (special control characters). They allow you to specify three core concepts: `position anchors`, `groups and ranges`, and `quantity modifiers`.\n", "\n", "```{admonition} Anchors: \n", " ^ - matches at the beginning of a line\n", " $ - matches at the end of a line\n", "```\n", "```bash\n", "grep '^Call' list.txt\n", "grep '^ Reboot' list.txt\n", "grep 'today$' list.txt\n", "```\n", "\n", "To count the number of empty lines:\n", "```bash\n", "grep -c '^$' list.txt\n", "```\n", "To display all lines containing only the word \"End\" by itself:\n", "```bash\n", "grep '^End$' list.txt\n", "```\n", "\n", "```{admonition} Groups and ranges: \n", " [abc] - matches any single character from a, b, or c\n", " [a-e] - matches any single character within the range a to e\n", " [^abc] - inverse match; matches any single character except a, b, or c\n", " [^a-e] - inverse match; matches any single character outside the range a to e\n", " \\ - matches the exact word (option -w can also be used)\n", " . - matches any single character except a newline\n", " \\ - turns off the special meaning of the following character\n", "```\n", "```bash\n", "grep '[Rr]eboot' list.txt\n", "grep '\\<[Rr]eboot\\>' list.txt\n", "```\n", "\n", "Display all lines from `list.txt` containing three adjacent digits:\n", "```bash\n", "grep '[0-9][0-9][0-9]' list.txt\n", "```\n", "Display lines containing four or more consecutive characters:\n", "```bash\n", "grep '....' list.txt\n", "```\n", "Display all non-blank lines from `list.txt`:\n", "```bash\n", "grep '.' list.txt\n", "```\n", "Display all lines containing a literal period:\n", "```bash\n", "grep '\\.' list.txt\n", "```\n", "\n", "```{admonition} Modifiers: \n", " * - matches zero or more instances of the preceding single character\n", " ? - matches zero or one instance of the preceding regex (implies the 'grep -E' option)\n", " + - matches one or more instances of the preceding regex (implies the 'grep -E' option)\n", " \\{n,m\\} - matches a range of occurrences; \\{n\\} matches exactly n occurrences; \\{n,\\} matches at least n occurrences\n", " | - matches either the regex specified before or after the vertical bar (implies the 'grep -E' option)\n", "```\n", "\n", "Display all lines from `list.txt` containing \"Oop\", \"Oops\", \"Oopss\", etc.:\n", "```bash\n", "grep 'Oops*' list.txt\n", "```\n", "Display all lines containing \"Oops\", \"Oopss\", etc.:\n", "```bash\n", "grep 'Oopss*' list.txt\n", "``` \n", "Display lines containing two or more adjacent digits:\n", "```bash\n", "grep '[0-9][0-9][0-9]*' list.txt\n", "```\n", "Display lines containing the number '3' or the combination '34':\n", "```bash\n", "grep -E '34?' list.txt\n", "```\n", "Display lines containing at least one digit:\n", "```bash\n", "grep -E '[0-9]+' list.txt\n", "```\n", "Display lines containing \"sss\" and \"ssss\":\n", "```bash\n", "grep 's\\{3,4\\}' list.txt\n", "```\n", "Display lines containing any 3, 4, or 5-digit numbers:\n", "```bash\n", "grep '[0-9]\\{3,5\\}' list.txt\n", "```\n", "Display lines containing \"Reboot\", \"reboot\", or \"support\":\n", "```bash\n", "grep -E '[Rr]eboot|support' list.txt\n", "```\n", "Display lines containing any letter (no empty lines):\n", "```bash\n", "grep '[A-Za-z]' list.txt\n", "```\n", "Display lines containing any non-alphanumeric and non-space symbol:\n", "```bash\n", "grep '[^ 0-9A-Za-z]' list.txt\n", "```\n", "Display lines containing an uppercase letter followed by zero or more lowercase letters:\n", "```bash\n", "grep '[A-Z][a-z]*' list.txt\n", "```\n", "Display lines containing a 3-digit telephone number:\n", "```bash\n", "grep 'tel: [0-9]\\{3\\}' list.txt\n", "```\n", "\n", "[Recommended tutorial on Regular expressions and grep](http://linuxcourse.rutgers.edu/2024/html/documents/Bash-Beginners-Guide/chap_04.html)\n", "\n", "
\n", "\n", "## Introduction to `sed`\n", "\n", "The stream editor, `sed`, is used for editing lines in a file or a stream. Its output goes to standard output and can be redirected to a new file.\n", "Syntax:\n", "```bash\n", "sed [options] 'command1' [files]\n", "sed [options] -e 'command1' [-e command2 ...] [files]\n", "sed [options] -f script [files]\n", "```\n", "\n", "To print lines 3 through 5 from `list.txt`:\n", "```bash\n", "sed -n '3,5p' list.txt\n", "```\n", "To delete lines 3 through 5 from `list.txt`:\n", "```bash\n", "sed '3,5d' list.txt\n", "```\n", "\n", "We can use regular expressions enclosed in `/ /` to target specific blocks for commands.\n", "\n", "For example, to delete lines starting with \"O\":\n", "```bash\n", "sed '/^O/d' list.txt\n", "```\n", "To delete empty lines:\n", "```bash\n", "sed '/^$/d' list.txt\n", "```\n", "To translate capital letters `C`, `R`, and `O` into lowercase `c`, `r`, and `o`:\n", "```bash\n", "sed 'y/CRO/cro/' list.txt\n", "```\n", "To replace the first occurrence of \"Oop\" with \"Wee\" on each line:\n", "```bash\n", "sed 's/Oop/Wee/' list.txt\n", "```\n", "To remove the string \"ss\" (replace with an empty entry) for the first occurrence on a line:\n", "```bash\n", "sed 's/ss//' list.txt\n", "```\n", "To remove all occurrences of \"ss\" on a line:\n", "```bash\n", "sed 's/ss//g' list.txt\n", "```\n", "To substitute a single space for any sequence of multiple spaces on a line:\n", "```bash\n", "sed 's/ */ /g' list.txt\n", "```\n", "To substitute an underscore for any sequence of multiple spaces:\n", "```bash\n", "sed 's/ */_/g' list.txt\n", "```\n", "\n", "To make `sed` modify the original file content directly, use the `-i` (in-place) option.\n", "For example, to apply changes to `list2.txt`:\n", "```bash\n", "cp list.txt list2.txt\n", "sed -i 's/ */_/g' list2.txt\n", "```\n", "\n", "[More examples: Useful one-line scripts for sed](http://linuxcourse.rutgers.edu/2024/html/documents/sed1line.txt)\n", "\n", "[Recommended tutorial: The GNU sed stream editor](http://linuxcourse.rutgers.edu/2024/html/documents/Bash-Beginners-Guide/chap_05.html)\n", "\n", "## Printing Lines Containing a Matching Pattern Using `grep`, `awk`, and `sed`\n", "\n", "The following three commands all print out the exact same matching line from `list.txt`:\n", "\n", "```bash\n", "grep Reboot list.txt\n", "\n", "awk '/Reboot/ { print $0 }' list.txt\n", "\n", "sed -n '/Reboot/p' list.txt\n", "```" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.11" } }, "nbformat": 4, "nbformat_minor": 4 }