{ "cells": [ { "cell_type": "code", "execution_count": 1, "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [ "remove_cell", "remove-cell" ] }, "outputs": [], "source": [ "my_notebook_text = \"\"\"\n", "\n", "# Commands, shells, processes, files.\n", "\n", "## Overview\n", "\n", "Today, we'll first talk about the course organization.
\n", "Then we'll discuss the fundamentals of the Linux operating system: kernel, processes, file system, user accounts, and basic commands.\n", "\n", "## Course organization\n", "\n", "- Virtual Desktop (VD) access\n", "- Canvas\n", "- Slack channel\n", "- Course website\n", "- Lectures\n", "- Practical exercises\n", "- Midterm and Final exams\n", "\n", "
\n", "\n", "## Course agenda\n", "\n", "- Lectures and practical recitations on the virtual computers remotely.\n", "- Homeworks on the virtual computers remotely.\n", "- Midterm exam (on October 20 2026).\n", "- Final exam (on December 15 2026).\n", "\n", "
\n", "\n", "## Course topics\n", "\n", "- Command line interface (CLI): Linux system commands, file systems, processes, services.\n", "- Operating system Virtualization with KVM and virtual machine management.\n", "- Package management (apt, dnf, rpm, dpkg).\n", "- Networking (TCP/IP basics, troubleshooting, configuration, services).\n", "- Network File System (NFS).\n", "- Shell scripting, awk, grep, sed, regular expressions.\n", "- Service startup and shutdown, at and cron scheduling.\n", "- Python: conda environment, jupyter notebook, various modules, computing with python.\n", "- Application compilation and libraries.\n", "- Security topics: password, brute force attacks, iptables, port scanning.\n", "- GPU computing: CUDA, neural networks, PyTorch.\n", "
\n", "\n", "\n", "## A typical Server/Desktop/Laptop (Intel Core i7) hardware\n", "\n", "- From the file system, OS and applications are loaded into the RAM.\n", "- The OS (kernel) controls the hardware and processes. \n", "- On the diagram:\n", " - CPU \n", " - RAM\n", " - PCIe - Peripheral Component Interconnect Express, PCIe 3.0 here. Mordern PCIe 4.0 and 5.0 are on newer systems.\n", " - PCH - Platform Controller Hub. Connects PCIe, USB, and SATA.\n", " - DMI - Direct Memory Interface \n", " - LPC - Low Pin Count; a simple interface to slower I/O devices \n", " - Peripheral devices connected to PCIe, SATA, USB interfaces\n", "![title](img/Chipset_CPU.jpg)\n", "- computer architecture\n", "***\n", "\n", "\n", "## Purpose of operating systems\n", "\n", "\n", "[Operating system concepts](https://en.wikipedia.org/wiki/Operating_system)\n", "\n", "Common features\n", "\n", "- [Device drivers](https://en.wikipedia.org/wiki/Device_driver)\n", "- [Interrupts](https://en.wikipedia.org/wiki/Interrupt)\n", "- [Process management (computing)](https://en.wikipedia.org/wiki/Process_management_(computing)) \n", "- [Memory management](https://en.wikipedia.org/wiki/Memory_management) \n", "- [Security](https://en.wikipedia.org/wiki/Computer_security)\n", "- [File system](https://en.wikipedia.org/wiki/File_system)\n", "- [I/O](https://en.wikipedia.org/wiki/Input/output)\n", "- [Networking](https://en.wikipedia.org/wiki/Computer_network) \n", "\n", "

\n", "\n", "![title](img/Operating_system_placement.png)\n", "- Operating system in the computing stack\n", "***\n", "\n", "## Linux kernel and applications\n", "\n", "In general, Linux implies rather just a Kernel than a whole OS.\n", "\n", "Linux OS distributions include:\n", "generic or customized Kernel and\n", "generic or customized GNU software (libraries, shells, and applications).\n", "\n", "Source codes of the Kernel, shells and GNU applications are freely available to anyone.\n", "\n", "Traditional tools for Linux administration and development:\n", " C - programming language\n", " Shell scripting,\n", " Python\n", "\n", "\n", "![Linux kernel](img/linux_kerne_app.jpg)\n", "- Linux kernel, applications, and hardware\n", "***\n", "\n", "## Login to the virtual desktop (Exercise)\n", "\n", "\n", "![Connect to your VD in Ravada cluster](img/Ravada_connection_SSH.png)\n", "- Follow the HOWTO for VD connection in Canvas\n", "\n", "***\n", "\n", "## Login to a Linux terminal\n", "![Login to Linux System](img/user_login.jpg)\n", "- Login procedure\n", "***\n", "\n", "## User accounts\n", "\n", "\n", "In order to login to a system a user has to authenticate with his/her credentials: user name and password.\n", "\n", "Authentication types:\n", "\n", "- local (password/shadow/group) and\n", "- Domain based (NIS, Kerberos, LDAP).\n", "\n", "For local authentication, a user should have an account on the system.\n", "\n", "\n", "\n", "\n", "|Account File | Entry\n", "|:-|:-\n", "| /etc/passwd | mike:x:1001:1001:Michael Whites:/home/mike:/bin/bash\n", "| /etc/shadow | mike:$1$zXCV7fz8ii84grbZhj:14087:0:99999:7:::\n", "| /etc/group | mike:x:1001:mike \n", "| /etc/group | admin:x:112:hostadm,mike\n", "\n", "Accounts can be created only by a superuser (root).\n", "

\n", "Commands to create user accounts:\n", "

adduser (interactive command)\n", "

useradd (can be included in a script).\n", "\n", "***\n", "\n", "### Exercise\n", "Create a new user account by following the instructor.\n", "\n", "Become root:
\n", "\n", "```bash\n", "sudo -s\n", "```\n", "\n", "Create a new user account, mike:
\n", "\n", "```bash\n", "adduser mike\n", "```\n", "\n", "Verify the account existence:
\n", "\n", "```bash\n", "id mike\n", "```\n", "\n", "Change password of the user:
\n", "\n", "```bash\n", "passwd mike\n", "```\n", "\n", "***\n", "\n", "## Linux shells\n", "\n", "When you login to a Linux system, you get a command shell.\n", "\n", "Shells are listed in `/etc/shells`. Default Linux shell: `bash`\n", "\n", "System commands, scripts and applications run in the shell - they become child processes of the shell.\n", "\n", "Besides commands, the shell is using variables.\n", "\n", "You define a shell variable with assignment command, ```=```, for example:\n", "```bash\n", "svar='file_name.txt'\n", "```\n", "The variable is referenced in the shell with ```$``` prefix, for example:\n", "```bash\n", "echo $svar\n", "```\n", "\n", "Shell variables are local to the shell.\n", "\n", "A shell variable becomes an environment variable after executing command export on it:
\n", "\n", "```bash\n", "svar='VAR_1' #initialize shell variable\n", "export svar #becomes environment variable\n", "env | grep svar\n", "```\n", "\n", "Environment variables are inherited by the child shells and processes.\n", "\n", "Commands executed in the shell should be either built-in shell commands, or addressed with the full path, or located in the `PATH` environment variable.
\n", "\n", "```bash\n", "echo $PATH\n", "```\n", "\n", "***\n", "\n", "### Exercise\n", "\n", "Create a new shell variable, svar:
\n", "\n", "```bash\n", "svar='VAR_1'\n", "echo $svar\n", "```\n", "\n", "Start a new child shell and see if svar is defined there:
\n", "\n", "```bash\n", "bash\n", "echo $svar\n", "```\n", "\n", "Exit from the child shell, export the variable, and see if it is defined in a child shell:
\n", "\n", "```bash\n", "exit\n", "export svar\n", "bash\n", "echo $svar\n", "```\n", "\n", "***\n", "\n", "## Data storage hierarchy\n", "\n", "- Data is stored in files.\n", "- Files are created in a file system.\n", "- A file system resides on a volume.\n", "- A volume is created on drives.\n", "\n", "![How data is stored](img/Device_and_file_system.png)\n", "\n", "- Here a file system is shown created on a volume of 3 drives. A file system cal also be created on a whole drive or a slice of the drive (partition).\n", "- To show up as a directory (folder), a file system needs to be mounted.\n", "***\n", "\n", "## HDD and SSD hardware\n", "::::{grid} 2\n", ":::{grid-item-card} Mechanical hard drive\n", "```{image} img/hd_partitions.jpg\n", ":alt: fishy\n", ":class: bg-primary mb-1\n", ":width: 200px\n", ":align: center\n", "```\n", ":::\n", ":::{grid-item-card} Elements of a file system\n", "- The cylinders define partition boundaries. \n", "- The sectors define data blocks.\n", "- A file system is created within a disk partition.\n", "- Files are created within a file system.\n", "- HDD endurance is about 5 years.\n", ":::\n", "::::\n", "\n", "\n", ":::{card}\n", "\n", "NAND cell technology for SSD, SD, NVMe, Flash drive.\n", "^^^\n", "```{image} img/NAND-Cell-Read-Programmed-and-Erased-fd945b79.png\n", ":alt: fishy\n", ":class: bg-primary mb-1\n", ":width: 600px\n", ":align: center\n", "```\n", "\n", "---\n", "NAND cells are combined into strings, pages, blocks, planes, and banks.\n", "```{image} img/Basic-SSD-Diagram-6b8c8849.png\n", ":alt: fishy\n", ":class: bg-primary mb-1\n", ":width: 600px\n", ":align: center\n", "```\n", ":::\n", "\n", "\n", "| SSD Type: | SLC (Single-Level Cell) (1 bit/cell) | MLC (2 bits/cell) | TLC (3 bits) | QLC (4 bits) \n", "|:- | :- | :- | :- | :-\n", "|Endurance | 100k cycles | 10k cycles | 3K cycles | 1K cycles\n", " \n", "\n", "## HDD vs SSD\n", "\n", "### Typical HDD and SSD on a market in 2025\n", "\n", "|Type | Capacity | Price | Throughput | Latency | IOPS |\n", "|:------: | :---: | :---: |:----------: | :----- | :------- |\n", "| SATA HD | 20 TB | $800 | 120 MB/s | 5 ms | 100 |\n", "| SATA SSD | 4 TB | $1400 | 500 MB/s | 0.05 ms | 100,000 |\n", "| NVMe SSD | 4 TB | $500 | 7,000 MB/s | 0.02 ms | 500,000 |\n", "\n", "## Files and Directories\n", "\n", "::::{grid} 2\n", ":::{grid-item-card}\n", "```{image} img/directory_entry.jpg\n", ":alt: fishy\n", ":class: bg-primary mb-1\n", ":width: 200px\n", ":align: center\n", ":::\n", ":::{grid-item-card} INODES contain information about files\n", "\n", "- a pointer to the file data blocks\n", "- link count\n", "- the owner\n", "- the group\n", "- mode\n", "- size\n", "- last access time\n", "- last modification time\n", "- last update time (attributes)\n", ":::\n", "::::\n", "\n", "::::{grid} 2\n", ":::{grid-item-card} Type of files\n", "- Regular or ordinary files\n", "- Directories\n", "- Symbolic links\n", "- Device files\n", "- Directory and Inodes\n", ":::\n", ":::{grid-item-card} Files are stored by blocks\n", "Default block size of ext4 file system is 4K:\n", "```bash\n", "stat -f /home\n", ":::\n", "::::\n", "\n", "## File Permissions and Ownerships\n", "\n", "### Permissions\n", "|Octal\t| Binary\t| Permission |\tMeaning\n", "|:- | :- | :-: | :-\n", "|0\t|000\t|none\t|All turned off\n", "|1\t|001\t|--x\t|Execute\n", "|2\t|010\t|-w-\t|Write\n", "|3\t|011\t|-wx\t|Write, execute\n", "|4\t|100\t|r--\t|Read\n", "|5\t|101\t|r-x\t|Read, execute\n", "|6\t|110\t|rw-\t|Read, write\n", "|7\t|111\t|rwx\t|Read, write, execute\n", "\n", "### File Ownerships: USER (u), GROUP (g), OTHERS (o)\n", "\n", "Changing permissions\n", "Permissions are applied for USER, GROUP and OTHERS (rwx rwx rwx)\n", "\n", "```bash\n", "chmod 660 testf.txt\n", "ls -l testf.txt\n", "```\n", "\n", "```\n", "-rw-rw---- 1 mike staff 0 Jan 18 10:26 testf.txt\n", "```\n", "\n", "umask sets default permission for files and directories.\n", "For example,\n", "\n", "```bash\n", "umask 022\n", "```\n", "\n", "|perm/umask| directory | file\n", "|:- | :- | :-\n", "|permission|777 \t|666 \n", "|umask |-022 \t|-022 \n", "|permission|755 \t|644 \n", "\n", "***\n", "\n", "## Assigning permissions and ownerships on files (Exercise)\n", "\n", "\n", "Create a new directory `EX1` and step into it:\n", "\n", "```bash\n", "mkdir EX1\n", "cd EX1\n", "```\n", "\n", "Check `umask` and create a new file, `mf1.txt`. See the file attributes with command `ls`.\n", "\n", "```bash\n", "umask\n", "touch mf1.txt\n", "ls -l mf1.txt\n", "```\n", "\n", "Change `umask` and create anothe file, `mf2.txt`\n", "\n", "```bash\n", "umask 022\n", "touch mf2.txt\n", "ls -l mf2.txt\n", "```\n", "\n", "Create another file, mf3.exe and make it executable:\n", "\n", "```bash\n", "touch mf3.exe\n", "chmod 755 mf3.exe\n", "ls -l mf3.exe\n", "```\n", "\n", "Try runing files `mf2.txt` and `mf3.exe`, and see which one is runnable:\n", "\n", "```bash\n", "./mf2.txt\n", "./mf3.exe\n", "```\n", "\n", "As you can see only the executable file can run.\n", "
\n", "Changing the user ownership on file `mf1.txt`:\n", "\n", "```bash\n", "sudo chown mike mf1.txt\n", "ls -l mf1.txt\n", "```\n", "\n", "Note, you need to be the root user when changing the file ownership.\n", "
\n", "\n", "Changing the group ownership on file `mf2.txt`:\n", "\n", "```bash\n", "sudo chown :mike mf2.txt\n", "ls -l mf2.txt\n", "```\n", "\n", "Changing both the user and group ownership on file `mf3.exe`:\n", "\n", "```bash\n", "sudo chown mike:mike mf3.exe\n", "ls -l mf*\n", "```\n", "\n", "***\n", "\n", "## Sticky bit (Exercise)\n", "\n", "Sticky bit on a directory protects files in the directory from been modified/removed by non their owners.\n", "\n", "For example `/tmp` directory on a Unix/Linux system has a sticky bit set\n", "\n", "\n", "```bash\n", "ls -ld /tmp\n", "```\n", "\n", "``` \n", "drwxrwxrwt 4 root root 4096 2018-07-25 16:29 /tmp\n", "```\n", "\n", "\n", "\n", "Open a new terminal and become root:\n", "\n", "```bash\n", "sudo -s\n", "```\n", "\n", "Create a new directory, `temp`, and give it a world writable permissions\n", "\n", "```bash\n", "mkdir temp\n", "chmod 777 temp\n", "```\n", "\n", "In the directory, user root creates a new empty file:\n", "\n", "```bash\n", "cd temp\n", "touch ex3\n", "```\n", "\n", "In the other terminal,try removing this file as user hostadm:\n", "\n", "```bash\n", "cd temp\n", "rm ex3\n", "```\n", "\n", "The file has been deleted. \n", "
\n", "\n", "Change the permission on `temp` director in the root user terminal: \n", "\n", "```bash\n", "chmod 1777 temp\n", "```\n", "\n", "Then repeat the steps above with creating and removing file `ex3` in directory `temp`\n", "\n", "***\n", "\n", "\n", "## Processes\n", "\n", "Program running on a system is a process. Linux is a multi-processing (multi-tasking) system.\n", "\n", "Processes are created by forking from a parent process, for example an application is forked from a user shell.\n", "\n", "Process types: \n", "\n", "- User Processes. A user process is one that is initiated by a regular user account and runs in user space.\n", "\n", "- Daemon Processes. A daemon process is an application that is designed to run in the background, usually related to a service.\n", "\n", "- Kernel Threads. Kernel processes execute only in kernel space.\n", "\n", "\n", "Privilege levels. Kernel space and user space:\n", "\n", "\n", "\n", "- Process privilege levels. The kernel threads run in Ring 0, the User and Daemon processes run in Ring 3.\n", "\n", "## Process states\n", "\n", "When created, a process goes through various states:\n", "\n", "\n", "- Evolution of the process state\n", "\n", "***\n", "## Process termination and renice command\n", "\n", "* Program running on a system is a process. Linux is a multi-processing\n", " (multi-tasking) system.\n", " \n", "* Attributes: Lifetime, PID, PPID, UID, GID, env variables, CWD.\n", " \n", "* Process with PID=1 is `init`\n", " \n", "* Monitoring: `ps` (for full listing, use `ps -ef` or `ps -aux`), `top`, `pstree`\n", " \n", "* Signal a running processes with PID=1009:\n", "\n", " ```bash\n", " kill -HUP 1009 #Hang up; re-read config files\n", " kill -1 1009\n", " kill -9 1009 #Kill; stop unconditionally\n", " kill -KILL 1009\n", " kill -15 1009 #Terminate gracefully\n", " kill -TERM 1009\n", " kill -TSTP 1009 #Suspend; can be continued\n", " kill -18 1009\n", " ```\n", " \n", "* Process can be started with lower and higher priority through `nice`\n", "(range: -20 highest, +19 lowest)\n", "\n", " ```bash\n", " nice -10 matlab\n", " nice --10 matlab\n", " ```\n", "\n", "* Process can be re-niced at run time:\n", "\n", " ```bash\n", " renice 5 1009\n", " renice -5 1009\n", " ```\n", " \n", " \n", " \n", "## Exercise with process termination\n", "\n", " \n", "When administrating a Linux system, it is often needed to terminate processes that consume a lot of resources (CPU and/or RAM), and slow down everything else.\n", "
\n", " \n", "* Download a python script, and make it executable:\n", "\n", " ```bash\n", " wget http://capone.rutgers.edu/coursefiles/inf_loop.py\n", " chmod 755 inf_loop.py\n", " ```\n", " \n", "\n", "* Start `inf_loop.py` scrip on a \"background\":\n", "\n", " ```bash\n", " ./inf_loop.py &\n", " ```\n", "\n", "* Find out the PID of the process:\n", "\n", " ```bash\n", " pgrep inf_loop.py\n", " ```\n", "\n", " The output shows the PID of the process, for exmple 4198.\n", "
\n", " Terminate process 4198:\n", "\n", " ```bash\n", " kill -15 4198\n", " ```\n", "\n", " An alternative way to kill an application is by using `pkill` command:\n", "\n", " ```bash\n", " pkill -15 inf_loop.py\n", " ```\n", "\n", " If option `-15` above doesn't work, try `-9`\n", "\n", " \n", "* Reference: Understanding Linux processes\n", "\n", "***\n", "\n", "## Process data streams\n", "\n", "* Most of the Linux system processes have three data streams with file descriptors \"0\", \"1\", and \"3\":\n", " \n", " standard input \"0\" (stdin)\n", "\n", " standard output \"1\" (stdo)\n", "\n", " standard error \"2\" (stderr)\n", "\n", "\n", "* The standard output and standard error are directed to the screen of your monitor; the standard input is read from a keyboard.\n", "\n", " It is possible to redirect the standard output and error into files, for example\n", " ```bash\n", " ps -ef 1 > stdo.out\n", " ```\n", "\n", " or\n", "\n", " ```bash\n", " ps -ef > stdo.out\n", " ```\n", "\n", " To redirect both stdo and stderr to the same file:\n", "\n", " `command > output.txt 2>&1`\n", "\n", " For example, \n", " ```bash\n", " ps -ef > output.txt 2>&1\n", " ```\n", "\n", " If you need to discard the stdo stream, you can re-direct it to `/dev/null`:\n", " ```bash\n", " ps -ef 1>/dev/null\n", " ```\n", "\n", "
\n", " It is also possible to re-direct the standart output of one process into\n", " the standard input of the other using pipes \"|\":\n", " \n", " ```bash\n", " ps -ef | less\n", " ```\n", " \n", " ***\n", "\n", "## Background processes (exercise)\n", "\n", "
\n", "Processes with Input/Output detached from the terminal are called background processes.\n", "
\n", "\n", "\n", "In a terminal window, run script inf_loop.py:\n", "\n", "```bash\n", "inf_loop.py\n", "```\n", "\n", "
\n", "Suspend the interactive process and move it to the background:\n", "\n", "```bash\n", "# Ctrl-Z\n", "jobs\n", "bg %1\n", "```\n", "\n", "Now the shell can be used for running the other commands and applications.\n", "Move the background job (with its ID) back to the foreground and terminate\n", "ing_loop.py process:\n", "\n", "```bash\n", "fg %1\n", "# Ctrl-C\n", "```\n", "\n", "***\n", "\n", "## Commands for directory manipulation (exercise)\n", "\n", "When you login into the system, you automatically get into your home directory\n", "\n", "#1. Determine the directory you are in:\n", "```bash \n", "pwd\n", "```\n", "#2. Go to some other directory, for example, /etc: \n", "```bash\n", "cd /etc\n", "```\n", "#3. Go to your home directory:\n", "```bash\n", "cd $HOME\n", "```\n", " or\n", "```bash \n", "cd ~\n", "```\n", "#4. Go back-and-forth between directories and check where you are:\n", "```bash\n", "cd /etc\n", "pwd\n", "cd -\n", "pwd\n", "cd -\n", "pwd\n", "```\n", "#5. Create a new directory above your home directory:\n", "```bash\n", "mkdir Newdir\n", "```\n", "#6. Go to the new directory and check where you are:\n", "```bash\n", "cd Newdir\n", "pwd\n", "```\n", "#7. Go one step back to the parent directory and check where you are:\n", "```bash\n", "cd ..\n", "pwd\n", "```\n", "#8. Remove the new directory:\n", "```bash\n", "rmdir Newdir\n", "```\n", "#9. Create a new directory above your home directory with subdirectories. Try the following:\n", "```bash\n", "mkdir Newdir/one/two/three\n", "```\n", "Does it work? Try the following:\n", "```bash\n", "mkdir -p Newdir/one/two/three\n", "```\n", "#10. Create a few files in Newdir/one:\n", "```bash\n", "cd Newdir/one; touch f1.t; touch f2.t; touch f3.t\n", "```\n", "#11. Go back into the original directory and try to delete Newdir:\n", "```bash\n", "cd -\n", "rmdir Newdir\n", "```\n", "Does it work ? Try the following:\n", "```bash\n", "rm -R Newdir\n", "```\n", "#12. Repeat step #9;\n", "Copy the directory with its tree into another directory:\n", "```bash\n", "cp -dpR Newdir Newdir-1\n", "```\n", "same as\n", "```bash\n", "cp -a Newdir Newdir-1\n", "```\n", "\n", "## Commands to read text file content (exercise)\n", "\n", "#13. Show content of file /etc/hosts\n", "```bash\n", "cat /etc/hosts\n", "tac /etc/hosts\n", "less /etc/hosts\n", "```\n", "#14. Show the first and last 10 lines of a file\n", "```bash\n", "head -n 10 /etc/nsswitch.conf\n", "tail -n 10 /etc/nsswitch.conf\n", "```\n", "\n", "\n", "## Various important commands (exercise)\n", "\n", "#15. Show date on the system clock\n", "```bash\n", "date\n", "```\n", "#16. File system usage\n", "```bash\n", "df -h\n", "```\n", "#17. Disk usage\n", "```bash\n", "du -h /home\n", "du -s /home\n", "du --max-depth=1 /var\n", "du -x -h /\n", "```\n", "#18. Display data\n", "```bash\n", "echo $[10*3+2]\n", "echo '$[10*3+2]'\n", "echo \"$[10*3+2]\"\n", "```\n", "#19. Determine file type\n", "```bash\n", "file /bin/ls\n", "file /etc/hosts\n", "```\n", "#20. Who is logined to the system\n", "```bash\n", "who\n", "```\n", "#21. The owner of the current shell\n", "```bash\n", "whoami\n", "```\n", "#22. Getting information about a file or command, for example netstat\n", "```bash\n", "whatis whoami\n", "apropos whoami\n", "man whoami\n", "info whoami\n", "```\n", "\n", "\n", "#23. Compressing/uncompressing files with gzip, and bzip2:\n", "```bash\n", "cd ~\n", "cp /etc/hosts hosts.txt\n", "gzip hosts.txt\n", "ls -l\n", "less hosts.txt.gz\n", "more hosts.txt.gz\n", "zcat hosts.txt.gz\n", "gunzip hosts.txt.gz\n", "bzip2 hosts.txt\n", "bzip2 -d hosts.txt.bz2\n", "```\n", "#24. Searching for files by using command find\n", "```bash\n", "cd /tmp\n", "mkdir newfiles; cd newfiles\n", "touch fff.txt\n", "cd ~\n", "find / -name fff.txt\n", "cd /tmp\n", "find . -name fff.txt\n", "find . -name '*ff*'\n", "find /tmp -name fff.txt -exec rm '{}' ';'\n", "find /var -size +1000k\n", "find /var -size +1000000c\n", "find /home -user hostadm\n", "find /var -mtime -5\n", "find /var -maxdepth 2 -mtime -20\n", "```\n", "#25. Searching for system files\n", "```bash\n", "which dpkg\n", "whereis dpkg\n", "locate dpkg\n", "```\n", "\n", "#26. Using tar-gzip to archive a directory.\n", "Create a new directory with subdirectories above your home directory, then tar and gzip it. For example,\n", "```bash\n", "mkdir -p Newdir/one/two/three\n", "cd Newdir/one/two\n", "cp /etc/hosts .\n", "cd three\n", "cp /etc/passwd .\n", "cd ~\n", "tar -cvf Newdir.tar Newdir\n", "gzip Newdir.tar\n", "ls -l Newdir.tar.gz\n", "rm -R Newdir\n", "cp Newdir.tar.gz /tmp\n", "cd /tmp\n", "gunzip Newdir.tar.gz\n", "tar -xvf Newdir.tar\n", "```\n", "#27. To tar and gzip file with one command, try the following:\n", "```bash\n", "tar -zcvf Newdir.tgz Newdir\n", "rm -R Newdir\n", "ls -l Newdir.tgz\n", "tar -zxvf Newdir.tgz\n", "```\n", "#28. Archive/restore from input file list by cpio command:\n", "```bash\n", "cd /etc\n", "find . | cpio -ov > /tmp/etc.cpio\n", "mkdir /tmp/RESTORE\n", "cd /tmp/RESTORE\n", "cpio -iv < ../etc.cpio\n", "```\n", "\n", "#29. Download files from a website:\n", "```bash\n", "wget https://people.sc.fsu.edu/~jburkardt/c_src/laplace_mpi/laplace_mpi.c\n", "```\n", " Similarly\n", "```bash\n", "curl -O https://people.sc.fsu.edu/~jburkardt/c_src/laplace_mpi/laplace_mpi.c\n", "```\n", "\n", "#30. Command history in bash shell:\n", "```bash\n", "history\n", "```\n", "Find a command in the history by the regular expression in its name:\n", "```bash\n", "ctrl + R\n", "```\n", "then type the string contained in the command.\n", "\n", "\n", "#31. Command history file .bash_history is updated when a user exits the shell.\n", "```bash\n", "tail .bash_history\n", "```\n", "The size or file .bash_history is defined by env variable HISTSIZE\n", "```bash\n", "echo $HISTSIZE\n", "```\n", "\n", "\n", "#32. Commands to read hardware information.\n", "CPU info:\n", "```bash\n", "lscpu\n", "```\n", "PCI device info:\n", "```bash\n", "lspci\n", "```\n", "\n", "Install package `lshw`:\n", "```bash\n", "sudo apt install lshw\n", "```\n", "\n", "Memory info:\n", "```bash\n", "lshw -class memory\n", "```\n", "Storage controller:\n", "```bash\n", "lshw -class storage\n", "```\n", "Disks:\n", "```bash\n", "lshw -class disk\n", "```\n", "Network interfaces:\n", "```bash\n", "lshw -class network \n", "```\n", "Video card and display info:\n", "```bash\n", "lshw -class video \n", "```\n", "\n", "\"\"\"" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [ "remove_cell", "remove-cell" ] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Both GOOGLE_API_KEY and GEMINI_API_KEY are set. Using GOOGLE_API_KEY.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Servers are still busy. Waiting 5 seconds before a quick automatic retry...\n" ] }, { "ename": "ServerError", "evalue": "503 UNAVAILABLE. {'error': {'code': 503, 'message': 'This model is currently experiencing high demand. Spikes in demand are usually temporary. Please try again later.', 'status': 'UNAVAILABLE'}}", "output_type": "error", "traceback": [ "\u001b[31m---------------------------------------------------------------------------\u001b[39m", "\u001b[31mServerError\u001b[39m Traceback (most recent call last)", "\u001b[36mCell\u001b[39m\u001b[36m \u001b[39m\u001b[32mIn[2]\u001b[39m\u001b[32m, line 169\u001b[39m\n\u001b[32m 167\u001b[39m time.sleep(\u001b[32m5\u001b[39m)\n\u001b[32m 168\u001b[39m \u001b[38;5;66;03m# Automatic second attempt\u001b[39;00m\n\u001b[32m--> \u001b[39m\u001b[32m169\u001b[39m response = client.models.generate_content(\n\u001b[32m 170\u001b[39m model=\u001b[33m'gemini-3.6-flash'\u001b[39m,\n", "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/lib/python3.13/site-packages/google/genai/models.py:6549\u001b[39m, in \u001b[36mModels.generate_content\u001b[39m\u001b[34m(self, model, contents, config)\u001b[39m\n\u001b[32m 6548\u001b[39m i += \u001b[32m1\u001b[39m\n\u001b[32m-> \u001b[39m\u001b[32m6549\u001b[39m response = \u001b[30;43mself\u001b[39;49m\u001b[30;43m.\u001b[39;49m\u001b[30;43m_generate_content\u001b[39;49m\u001b[30;43m(\u001b[39;49m\n\u001b[32m 6550\u001b[39m \u001b[30;43m \u001b[39;49m\u001b[30;43mmodel\u001b[39;49m\u001b[30;43m=\u001b[39;49m\u001b[30;43mmodel\u001b[39;49m\u001b[30;43m,\u001b[39;49m\u001b[30;43m \u001b[39;49m\u001b[30;43mcontents\u001b[39;49m\u001b[30;43m=\u001b[39;49m\u001b[30;43mcontents\u001b[39;49m\u001b[30;43m,\u001b[39;49m\u001b[30;43m \u001b[39;49m\u001b[30;43mconfig\u001b[39;49m\u001b[30;43m=\u001b[39;49m\u001b[30;43mparsed_config_to_call\u001b[39;49m\n\u001b[32m 6551\u001b[39m \u001b[30;43m\u001b[39;49m\u001b[30;43m)\u001b[39;49m\n\u001b[32m 6553\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m function_map:\n", "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/lib/python3.13/site-packages/google/genai/models.py:4977\u001b[39m, in \u001b[36mModels._generate_content\u001b[39m\u001b[34m(self, model, contents, config)\u001b[39m\n\u001b[32m 4975\u001b[39m request_dict = _common.encode_unserializable_types(request_dict)\n\u001b[32m-> \u001b[39m\u001b[32m4977\u001b[39m response = \u001b[30;43mself\u001b[39;49m\u001b[30;43m.\u001b[39;49m\u001b[30;43m_api_client\u001b[39;49m\u001b[30;43m.\u001b[39;49m\u001b[30;43mrequest\u001b[39;49m\u001b[30;43m(\u001b[39;49m\n\u001b[32m 4978\u001b[39m \u001b[30;43m \u001b[39;49m\u001b[30;43m'\u001b[39;49m\u001b[30;43mpost\u001b[39;49m\u001b[30;43m'\u001b[39;49m\u001b[30;43m,\u001b[39;49m\u001b[30;43m \u001b[39;49m\u001b[30;43mpath\u001b[39;49m\u001b[30;43m,\u001b[39;49m\u001b[30;43m \u001b[39;49m\u001b[30;43mrequest_dict\u001b[39;49m\u001b[30;43m,\u001b[39;49m\u001b[30;43m \u001b[39;49m\u001b[30;43mhttp_options\u001b[39;49m\n\u001b[32m 4979\u001b[39m \u001b[30;43m\u001b[39;49m\u001b[30;43m)\u001b[39;49m\n\u001b[32m 4981\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m config \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;28mgetattr\u001b[39m(\n\u001b[32m 4982\u001b[39m config, \u001b[33m'\u001b[39m\u001b[33mshould_return_http_response\u001b[39m\u001b[33m'\u001b[39m, \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[32m 4983\u001b[39m ):\n", "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/lib/python3.13/site-packages/google/genai/_api_client.py:1650\u001b[39m, in \u001b[36mBaseApiClient.request\u001b[39m\u001b[34m(self, http_method, path, request_dict, http_options)\u001b[39m\n\u001b[32m 1647\u001b[39m http_request = \u001b[38;5;28mself\u001b[39m._build_request(\n\u001b[32m 1648\u001b[39m http_method, path, request_dict, http_options\n\u001b[32m 1649\u001b[39m )\n\u001b[32m-> \u001b[39m\u001b[32m1650\u001b[39m response = \u001b[30;43mself\u001b[39;49m\u001b[30;43m.\u001b[39;49m\u001b[30;43m_request\u001b[39;49m\u001b[30;43m(\u001b[39;49m\u001b[30;43mhttp_request\u001b[39;49m\u001b[30;43m,\u001b[39;49m\u001b[30;43m \u001b[39;49m\u001b[30;43mhttp_options\u001b[39;49m\u001b[30;43m,\u001b[39;49m\u001b[30;43m \u001b[39;49m\u001b[30;43mstream\u001b[39;49m\u001b[30;43m=\u001b[39;49m\u001b[30;43;01mFalse\u001b[39;49;00m\u001b[30;43m)\u001b[39;49m\n\u001b[32m 1651\u001b[39m response_body = (\n\u001b[32m 1652\u001b[39m response.response_stream[\u001b[32m0\u001b[39m] \u001b[38;5;28;01mif\u001b[39;00m response.response_stream \u001b[38;5;28;01melse\u001b[39;00m \u001b[33m'\u001b[39m\u001b[33m'\u001b[39m\n\u001b[32m 1653\u001b[39m )\n", "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/lib/python3.13/site-packages/google/genai/_api_client.py:1443\u001b[39m, in \u001b[36mBaseApiClient._request\u001b[39m\u001b[34m(self, http_request, http_options, stream)\u001b[39m\n\u001b[32m 1441\u001b[39m \u001b[38;5;28;01mreturn\u001b[39;00m retry(\u001b[38;5;28mself\u001b[39m._request_once, http_request, stream) \u001b[38;5;66;03m# type: ignore[no-any-return]\u001b[39;00m\n\u001b[32m-> \u001b[39m\u001b[32m1443\u001b[39m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[30;43mself\u001b[39;49m\u001b[30;43m.\u001b[39;49m\u001b[30;43m_retry\u001b[39;49m\u001b[30;43m(\u001b[39;49m\u001b[30;43mself\u001b[39;49m\u001b[30;43m.\u001b[39;49m\u001b[30;43m_request_once\u001b[39;49m\u001b[30;43m,\u001b[39;49m\u001b[30;43m \u001b[39;49m\u001b[30;43mhttp_request\u001b[39;49m\u001b[30;43m,\u001b[39;49m\u001b[30;43m \u001b[39;49m\u001b[30;43mstream\u001b[39;49m\u001b[30;43m)\u001b[39;49m\n", "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/lib/python3.13/site-packages/tenacity/__init__.py:470\u001b[39m, in \u001b[36mRetrying.__call__\u001b[39m\u001b[34m(self, fn, *args, **kwargs)\u001b[39m\n\u001b[32m 469\u001b[39m \u001b[38;5;28;01mwhile\u001b[39;00m \u001b[38;5;28;01mTrue\u001b[39;00m:\n\u001b[32m--> \u001b[39m\u001b[32m470\u001b[39m do = \u001b[30;43mself\u001b[39;49m\u001b[30;43m.\u001b[39;49m\u001b[30;43miter\u001b[39;49m\u001b[30;43m(\u001b[39;49m\u001b[30;43mretry_state\u001b[39;49m\u001b[30;43m=\u001b[39;49m\u001b[30;43mretry_state\u001b[39;49m\u001b[30;43m)\u001b[39;49m\n\u001b[32m 471\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(do, DoAttempt):\n", "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/lib/python3.13/site-packages/tenacity/__init__.py:371\u001b[39m, in \u001b[36mBaseRetrying.iter\u001b[39m\u001b[34m(self, retry_state)\u001b[39m\n\u001b[32m 370\u001b[39m \u001b[38;5;28;01mfor\u001b[39;00m action \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mself\u001b[39m.iter_state.actions:\n\u001b[32m--> \u001b[39m\u001b[32m371\u001b[39m result = \u001b[30;43maction\u001b[39;49m\u001b[30;43m(\u001b[39;49m\u001b[30;43mretry_state\u001b[39;49m\u001b[30;43m)\u001b[39;49m\n\u001b[32m 372\u001b[39m \u001b[38;5;28;01mreturn\u001b[39;00m result\n", "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/lib/python3.13/site-packages/tenacity/__init__.py:413\u001b[39m, in \u001b[36mBaseRetrying._post_stop_check_actions..exc_check\u001b[39m\u001b[34m(rs)\u001b[39m\n\u001b[32m 412\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m.reraise:\n\u001b[32m--> \u001b[39m\u001b[32m413\u001b[39m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[30;43mretry_exc\u001b[39;49m\u001b[30;43m.\u001b[39;49m\u001b[30;43mreraise\u001b[39;49m\u001b[30;43m(\u001b[39;49m\u001b[30;43m)\u001b[39;49m\n\u001b[32m 414\u001b[39m \u001b[38;5;28;01mraise\u001b[39;00m retry_exc \u001b[38;5;28;01mfrom\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34;01mfut\u001b[39;00m\u001b[34;01m.\u001b[39;00m\u001b[34;01mexception\u001b[39;00m()\n", "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/lib/python3.13/site-packages/tenacity/__init__.py:184\u001b[39m, in \u001b[36mRetryError.reraise\u001b[39m\u001b[34m(self)\u001b[39m\n\u001b[32m 183\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m.last_attempt.failed:\n\u001b[32m--> \u001b[39m\u001b[32m184\u001b[39m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[30;43mself\u001b[39;49m\u001b[30;43m.\u001b[39;49m\u001b[30;43mlast_attempt\u001b[39;49m\u001b[30;43m.\u001b[39;49m\u001b[30;43mresult\u001b[39;49m\u001b[30;43m(\u001b[39;49m\u001b[30;43m)\u001b[39;49m\n\u001b[32m 185\u001b[39m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;28mself\u001b[39m\n", "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/lib/python3.13/concurrent/futures/_base.py:449\u001b[39m, in \u001b[36mFuture.result\u001b[39m\u001b[34m(self, timeout)\u001b[39m\n\u001b[32m 448\u001b[39m \u001b[38;5;28;01melif\u001b[39;00m \u001b[38;5;28mself\u001b[39m._state == FINISHED:\n\u001b[32m--> \u001b[39m\u001b[32m449\u001b[39m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[30;43mself\u001b[39;49m\u001b[30;43m.\u001b[39;49m\u001b[30;43m__get_result\u001b[39;49m\u001b[30;43m(\u001b[39;49m\u001b[30;43m)\u001b[39;49m\n\u001b[32m 451\u001b[39m \u001b[38;5;28mself\u001b[39m._condition.wait(timeout)\n", "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/lib/python3.13/concurrent/futures/_base.py:401\u001b[39m, in \u001b[36mFuture.__get_result\u001b[39m\u001b[34m(self)\u001b[39m\n\u001b[32m 400\u001b[39m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[32m--> \u001b[39m\u001b[32m401\u001b[39m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;28mself\u001b[39m._exception\n\u001b[32m 402\u001b[39m \u001b[38;5;28;01mfinally\u001b[39;00m:\n\u001b[32m 403\u001b[39m \u001b[38;5;66;03m# Break a reference cycle with the exception in self._exception\u001b[39;00m\n", "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/lib/python3.13/site-packages/tenacity/__init__.py:473\u001b[39m, in \u001b[36mRetrying.__call__\u001b[39m\u001b[34m(self, fn, *args, **kwargs)\u001b[39m\n\u001b[32m 472\u001b[39m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[32m--> \u001b[39m\u001b[32m473\u001b[39m result = \u001b[30;43mfn\u001b[39;49m\u001b[30;43m(\u001b[39;49m\u001b[30;43m*\u001b[39;49m\u001b[30;43margs\u001b[39;49m\u001b[30;43m,\u001b[39;49m\u001b[30;43m \u001b[39;49m\u001b[30;43m*\u001b[39;49m\u001b[30;43m*\u001b[39;49m\u001b[30;43mkwargs\u001b[39;49m\u001b[30;43m)\u001b[39;49m\n\u001b[32m 474\u001b[39m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mBaseException\u001b[39;00m: \u001b[38;5;66;03m# noqa: B902\u001b[39;00m\n", "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/lib/python3.13/site-packages/google/genai/_api_client.py:1420\u001b[39m, in \u001b[36mBaseApiClient._request_once\u001b[39m\u001b[34m(self, http_request, stream)\u001b[39m\n\u001b[32m 1419\u001b[39m response = \u001b[38;5;28mself\u001b[39m._httpx_client.send(httpx_request, stream=stream) \u001b[38;5;66;03m# type: ignore[union-attr]\u001b[39;00m\n\u001b[32m-> \u001b[39m\u001b[32m1420\u001b[39m \u001b[30;43merrors\u001b[39;49m\u001b[30;43m.\u001b[39;49m\u001b[30;43mAPIError\u001b[39;49m\u001b[30;43m.\u001b[39;49m\u001b[30;43mraise_for_response\u001b[39;49m\u001b[30;43m(\u001b[39;49m\u001b[30;43mresponse\u001b[39;49m\u001b[30;43m)\u001b[39;49m\n\u001b[32m 1421\u001b[39m \u001b[38;5;28;01mreturn\u001b[39;00m HttpResponse(\n\u001b[32m 1422\u001b[39m response.headers, response \u001b[38;5;28;01mif\u001b[39;00m stream \u001b[38;5;28;01melse\u001b[39;00m [response.text]\n\u001b[32m 1423\u001b[39m )\n", "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/lib/python3.13/site-packages/google/genai/errors.py:155\u001b[39m, in \u001b[36mAPIError.raise_for_response\u001b[39m\u001b[34m(cls, response)\u001b[39m\n\u001b[32m 153\u001b[39m response_json = response.body_segments[\u001b[32m0\u001b[39m].get(\u001b[33m'\u001b[39m\u001b[33merror\u001b[39m\u001b[33m'\u001b[39m, {})\n\u001b[32m--> \u001b[39m\u001b[32m155\u001b[39m \u001b[30;43mcls\u001b[39;49m\u001b[30;43m.\u001b[39;49m\u001b[30;43mraise_error\u001b[39;49m\u001b[30;43m(\u001b[39;49m\u001b[30;43mresponse\u001b[39;49m\u001b[30;43m.\u001b[39;49m\u001b[30;43mstatus_code\u001b[39;49m\u001b[30;43m,\u001b[39;49m\u001b[30;43m \u001b[39;49m\u001b[30;43mresponse_json\u001b[39;49m\u001b[30;43m,\u001b[39;49m\u001b[30;43m \u001b[39;49m\u001b[30;43mresponse\u001b[39;49m\u001b[30;43m)\u001b[39;49m\n", "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/lib/python3.13/site-packages/google/genai/errors.py:186\u001b[39m, in \u001b[36mAPIError.raise_error\u001b[39m\u001b[34m(cls, status_code, response_json, response)\u001b[39m\n\u001b[32m 185\u001b[39m \u001b[38;5;28;01melif\u001b[39;00m \u001b[32m500\u001b[39m <= status_code < \u001b[32m600\u001b[39m:\n\u001b[32m--> \u001b[39m\u001b[32m186\u001b[39m \u001b[38;5;28;01mraise\u001b[39;00m ServerError(status_code, response_json, response)\n\u001b[32m 187\u001b[39m \u001b[38;5;28;01melse\u001b[39;00m:\n", "\u001b[31mServerError\u001b[39m: 503 UNAVAILABLE. {'error': {'code': 503, 'message': 'This model is currently experiencing high demand. Spikes in demand are usually temporary. Please try again later.', 'status': 'UNAVAILABLE'}}", "\nDuring handling of the above exception, another exception occurred:\n", "\u001b[31mServerError\u001b[39m Traceback (most recent call last)", "\u001b[36mCell\u001b[39m\u001b[36m \u001b[39m\u001b[32mIn[2]\u001b[39m\u001b[32m, line 169\u001b[39m\n\u001b[32m 165\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m e.code == \u001b[32m503\u001b[39m:\n\u001b[32m 166\u001b[39m print(\u001b[33m\"Servers are still busy. Waiting 5 seconds before a quick automatic retry...\"\u001b[39m)\n\u001b[32m 167\u001b[39m time.sleep(\u001b[32m5\u001b[39m)\n\u001b[32m 168\u001b[39m \u001b[38;5;66;03m# Automatic second attempt\u001b[39;00m\n\u001b[32m--> \u001b[39m\u001b[32m169\u001b[39m response = client.models.generate_content(\n\u001b[32m 170\u001b[39m model=\u001b[33m'gemini-3.6-flash'\u001b[39m,\n\u001b[32m 171\u001b[39m contents=f\"Rewrite this text for a university course. Make it clear and write WCAG 2.1 AA compliant alt text descriptions for any charts mentioned: {my_notebook_text}\",)\n\u001b[32m 172\u001b[39m print(response.text)\n", "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/lib/python3.13/site-packages/google/genai/models.py:6549\u001b[39m, in \u001b[36mModels.generate_content\u001b[39m\u001b[34m(self, model, contents, config)\u001b[39m\n\u001b[32m 6545\u001b[39m parsed_config_to_call = _extra_utils.get_usage_header(\n\u001b[32m 6546\u001b[39m parsed_config_to_call\n\u001b[32m 6547\u001b[39m )\n\u001b[32m 6548\u001b[39m i += \u001b[32m1\u001b[39m\n\u001b[32m-> \u001b[39m\u001b[32m6549\u001b[39m response = \u001b[30;43mself\u001b[39;49m\u001b[30;43m.\u001b[39;49m\u001b[30;43m_generate_content\u001b[39;49m\u001b[30;43m(\u001b[39;49m\n\u001b[32m 6550\u001b[39m \u001b[30;43m \u001b[39;49m\u001b[30;43mmodel\u001b[39;49m\u001b[30;43m=\u001b[39;49m\u001b[30;43mmodel\u001b[39;49m\u001b[30;43m,\u001b[39;49m\u001b[30;43m \u001b[39;49m\u001b[30;43mcontents\u001b[39;49m\u001b[30;43m=\u001b[39;49m\u001b[30;43mcontents\u001b[39;49m\u001b[30;43m,\u001b[39;49m\u001b[30;43m \u001b[39;49m\u001b[30;43mconfig\u001b[39;49m\u001b[30;43m=\u001b[39;49m\u001b[30;43mparsed_config_to_call\u001b[39;49m\n\u001b[32m 6551\u001b[39m \u001b[30;43m\u001b[39;49m\u001b[30;43m)\u001b[39;49m\n\u001b[32m 6553\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m function_map:\n\u001b[32m 6554\u001b[39m \u001b[38;5;28;01mbreak\u001b[39;00m\n", "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/lib/python3.13/site-packages/google/genai/models.py:4977\u001b[39m, in \u001b[36mModels._generate_content\u001b[39m\u001b[34m(self, model, contents, config)\u001b[39m\n\u001b[32m 4974\u001b[39m request_dict = _common.convert_to_dict(request_dict)\n\u001b[32m 4975\u001b[39m request_dict = _common.encode_unserializable_types(request_dict)\n\u001b[32m-> \u001b[39m\u001b[32m4977\u001b[39m response = \u001b[30;43mself\u001b[39;49m\u001b[30;43m.\u001b[39;49m\u001b[30;43m_api_client\u001b[39;49m\u001b[30;43m.\u001b[39;49m\u001b[30;43mrequest\u001b[39;49m\u001b[30;43m(\u001b[39;49m\n\u001b[32m 4978\u001b[39m \u001b[30;43m \u001b[39;49m\u001b[30;43m'\u001b[39;49m\u001b[30;43mpost\u001b[39;49m\u001b[30;43m'\u001b[39;49m\u001b[30;43m,\u001b[39;49m\u001b[30;43m \u001b[39;49m\u001b[30;43mpath\u001b[39;49m\u001b[30;43m,\u001b[39;49m\u001b[30;43m \u001b[39;49m\u001b[30;43mrequest_dict\u001b[39;49m\u001b[30;43m,\u001b[39;49m\u001b[30;43m \u001b[39;49m\u001b[30;43mhttp_options\u001b[39;49m\n\u001b[32m 4979\u001b[39m \u001b[30;43m\u001b[39;49m\u001b[30;43m)\u001b[39;49m\n\u001b[32m 4981\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m config \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;28mgetattr\u001b[39m(\n\u001b[32m 4982\u001b[39m config, \u001b[33m'\u001b[39m\u001b[33mshould_return_http_response\u001b[39m\u001b[33m'\u001b[39m, \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[32m 4983\u001b[39m ):\n\u001b[32m 4984\u001b[39m return_value = types.GenerateContentResponse(sdk_http_response=response)\n", "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/lib/python3.13/site-packages/google/genai/_api_client.py:1650\u001b[39m, in \u001b[36mBaseApiClient.request\u001b[39m\u001b[34m(self, http_method, path, request_dict, http_options)\u001b[39m\n\u001b[32m 1640\u001b[39m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34mrequest\u001b[39m(\n\u001b[32m 1641\u001b[39m \u001b[38;5;28mself\u001b[39m,\n\u001b[32m 1642\u001b[39m http_method: \u001b[38;5;28mstr\u001b[39m,\n\u001b[32m (...)\u001b[39m\u001b[32m 1645\u001b[39m http_options: Optional[HttpOptionsOrDict] = \u001b[38;5;28;01mNone\u001b[39;00m,\n\u001b[32m 1646\u001b[39m ) -> SdkHttpResponse:\n\u001b[32m 1647\u001b[39m http_request = \u001b[38;5;28mself\u001b[39m._build_request(\n\u001b[32m 1648\u001b[39m http_method, path, request_dict, http_options\n\u001b[32m 1649\u001b[39m )\n\u001b[32m-> \u001b[39m\u001b[32m1650\u001b[39m response = \u001b[30;43mself\u001b[39;49m\u001b[30;43m.\u001b[39;49m\u001b[30;43m_request\u001b[39;49m\u001b[30;43m(\u001b[39;49m\u001b[30;43mhttp_request\u001b[39;49m\u001b[30;43m,\u001b[39;49m\u001b[30;43m \u001b[39;49m\u001b[30;43mhttp_options\u001b[39;49m\u001b[30;43m,\u001b[39;49m\u001b[30;43m \u001b[39;49m\u001b[30;43mstream\u001b[39;49m\u001b[30;43m=\u001b[39;49m\u001b[30;43;01mFalse\u001b[39;49;00m\u001b[30;43m)\u001b[39;49m\n\u001b[32m 1651\u001b[39m response_body = (\n\u001b[32m 1652\u001b[39m response.response_stream[\u001b[32m0\u001b[39m] \u001b[38;5;28;01mif\u001b[39;00m response.response_stream \u001b[38;5;28;01melse\u001b[39;00m \u001b[33m'\u001b[39m\u001b[33m'\u001b[39m\n\u001b[32m 1653\u001b[39m )\n\u001b[32m 1654\u001b[39m \u001b[38;5;28;01mreturn\u001b[39;00m SdkHttpResponse(headers=response.headers, body=response_body)\n", "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/lib/python3.13/site-packages/google/genai/_api_client.py:1443\u001b[39m, in \u001b[36mBaseApiClient._request\u001b[39m\u001b[34m(self, http_request, http_options, stream)\u001b[39m\n\u001b[32m 1440\u001b[39m retry = tenacity.Retrying(**retry_kwargs)\n\u001b[32m 1441\u001b[39m \u001b[38;5;28;01mreturn\u001b[39;00m retry(\u001b[38;5;28mself\u001b[39m._request_once, http_request, stream) \u001b[38;5;66;03m# type: ignore[no-any-return]\u001b[39;00m\n\u001b[32m-> \u001b[39m\u001b[32m1443\u001b[39m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[30;43mself\u001b[39;49m\u001b[30;43m.\u001b[39;49m\u001b[30;43m_retry\u001b[39;49m\u001b[30;43m(\u001b[39;49m\u001b[30;43mself\u001b[39;49m\u001b[30;43m.\u001b[39;49m\u001b[30;43m_request_once\u001b[39;49m\u001b[30;43m,\u001b[39;49m\u001b[30;43m \u001b[39;49m\u001b[30;43mhttp_request\u001b[39;49m\u001b[30;43m,\u001b[39;49m\u001b[30;43m \u001b[39;49m\u001b[30;43mstream\u001b[39;49m\u001b[30;43m)\u001b[39;49m\n", "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/lib/python3.13/site-packages/tenacity/__init__.py:470\u001b[39m, in \u001b[36mRetrying.__call__\u001b[39m\u001b[34m(self, fn, *args, **kwargs)\u001b[39m\n\u001b[32m 468\u001b[39m retry_state = RetryCallState(retry_object=\u001b[38;5;28mself\u001b[39m, fn=fn, args=args, kwargs=kwargs)\n\u001b[32m 469\u001b[39m \u001b[38;5;28;01mwhile\u001b[39;00m \u001b[38;5;28;01mTrue\u001b[39;00m:\n\u001b[32m--> \u001b[39m\u001b[32m470\u001b[39m do = \u001b[30;43mself\u001b[39;49m\u001b[30;43m.\u001b[39;49m\u001b[30;43miter\u001b[39;49m\u001b[30;43m(\u001b[39;49m\u001b[30;43mretry_state\u001b[39;49m\u001b[30;43m=\u001b[39;49m\u001b[30;43mretry_state\u001b[39;49m\u001b[30;43m)\u001b[39;49m\n\u001b[32m 471\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(do, DoAttempt):\n\u001b[32m 472\u001b[39m \u001b[38;5;28;01mtry\u001b[39;00m:\n", "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/lib/python3.13/site-packages/tenacity/__init__.py:371\u001b[39m, in \u001b[36mBaseRetrying.iter\u001b[39m\u001b[34m(self, retry_state)\u001b[39m\n\u001b[32m 369\u001b[39m result = \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[32m 370\u001b[39m \u001b[38;5;28;01mfor\u001b[39;00m action \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mself\u001b[39m.iter_state.actions:\n\u001b[32m--> \u001b[39m\u001b[32m371\u001b[39m result = \u001b[30;43maction\u001b[39;49m\u001b[30;43m(\u001b[39;49m\u001b[30;43mretry_state\u001b[39;49m\u001b[30;43m)\u001b[39;49m\n\u001b[32m 372\u001b[39m \u001b[38;5;28;01mreturn\u001b[39;00m result\n", "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/lib/python3.13/site-packages/tenacity/__init__.py:413\u001b[39m, in \u001b[36mBaseRetrying._post_stop_check_actions..exc_check\u001b[39m\u001b[34m(rs)\u001b[39m\n\u001b[32m 411\u001b[39m retry_exc = \u001b[38;5;28mself\u001b[39m.retry_error_cls(fut)\n\u001b[32m 412\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m.reraise:\n\u001b[32m--> \u001b[39m\u001b[32m413\u001b[39m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[30;43mretry_exc\u001b[39;49m\u001b[30;43m.\u001b[39;49m\u001b[30;43mreraise\u001b[39;49m\u001b[30;43m(\u001b[39;49m\u001b[30;43m)\u001b[39;49m\n\u001b[32m 414\u001b[39m \u001b[38;5;28;01mraise\u001b[39;00m retry_exc \u001b[38;5;28;01mfrom\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34;01mfut\u001b[39;00m\u001b[34;01m.\u001b[39;00m\u001b[34;01mexception\u001b[39;00m()\n", "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/lib/python3.13/site-packages/tenacity/__init__.py:184\u001b[39m, in \u001b[36mRetryError.reraise\u001b[39m\u001b[34m(self)\u001b[39m\n\u001b[32m 182\u001b[39m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34mreraise\u001b[39m(\u001b[38;5;28mself\u001b[39m) -> t.NoReturn:\n\u001b[32m 183\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m.last_attempt.failed:\n\u001b[32m--> \u001b[39m\u001b[32m184\u001b[39m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[30;43mself\u001b[39;49m\u001b[30;43m.\u001b[39;49m\u001b[30;43mlast_attempt\u001b[39;49m\u001b[30;43m.\u001b[39;49m\u001b[30;43mresult\u001b[39;49m\u001b[30;43m(\u001b[39;49m\u001b[30;43m)\u001b[39;49m\n\u001b[32m 185\u001b[39m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;28mself\u001b[39m\n", "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/lib/python3.13/concurrent/futures/_base.py:449\u001b[39m, in \u001b[36mFuture.result\u001b[39m\u001b[34m(self, timeout)\u001b[39m\n\u001b[32m 447\u001b[39m \u001b[38;5;28;01mraise\u001b[39;00m CancelledError()\n\u001b[32m 448\u001b[39m \u001b[38;5;28;01melif\u001b[39;00m \u001b[38;5;28mself\u001b[39m._state == FINISHED:\n\u001b[32m--> \u001b[39m\u001b[32m449\u001b[39m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[30;43mself\u001b[39;49m\u001b[30;43m.\u001b[39;49m\u001b[30;43m__get_result\u001b[39;49m\u001b[30;43m(\u001b[39;49m\u001b[30;43m)\u001b[39;49m\n\u001b[32m 451\u001b[39m \u001b[38;5;28mself\u001b[39m._condition.wait(timeout)\n\u001b[32m 453\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m._state \u001b[38;5;129;01min\u001b[39;00m [CANCELLED, CANCELLED_AND_NOTIFIED]:\n", "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/lib/python3.13/concurrent/futures/_base.py:401\u001b[39m, in \u001b[36mFuture.__get_result\u001b[39m\u001b[34m(self)\u001b[39m\n\u001b[32m 399\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m._exception \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[32m 400\u001b[39m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[32m--> \u001b[39m\u001b[32m401\u001b[39m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;28mself\u001b[39m._exception\n\u001b[32m 402\u001b[39m \u001b[38;5;28;01mfinally\u001b[39;00m:\n\u001b[32m 403\u001b[39m \u001b[38;5;66;03m# Break a reference cycle with the exception in self._exception\u001b[39;00m\n\u001b[32m 404\u001b[39m \u001b[38;5;28mself\u001b[39m = \u001b[38;5;28;01mNone\u001b[39;00m\n", "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/lib/python3.13/site-packages/tenacity/__init__.py:473\u001b[39m, in \u001b[36mRetrying.__call__\u001b[39m\u001b[34m(self, fn, *args, **kwargs)\u001b[39m\n\u001b[32m 471\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(do, DoAttempt):\n\u001b[32m 472\u001b[39m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[32m--> \u001b[39m\u001b[32m473\u001b[39m result = \u001b[30;43mfn\u001b[39;49m\u001b[30;43m(\u001b[39;49m\u001b[30;43m*\u001b[39;49m\u001b[30;43margs\u001b[39;49m\u001b[30;43m,\u001b[39;49m\u001b[30;43m \u001b[39;49m\u001b[30;43m*\u001b[39;49m\u001b[30;43m*\u001b[39;49m\u001b[30;43mkwargs\u001b[39;49m\u001b[30;43m)\u001b[39;49m\n\u001b[32m 474\u001b[39m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mBaseException\u001b[39;00m: \u001b[38;5;66;03m# noqa: B902\u001b[39;00m\n\u001b[32m 475\u001b[39m retry_state.set_exception(sys.exc_info()) \u001b[38;5;66;03m# type: ignore[arg-type]\u001b[39;00m\n", "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/lib/python3.13/site-packages/google/genai/_api_client.py:1420\u001b[39m, in \u001b[36mBaseApiClient._request_once\u001b[39m\u001b[34m(self, http_request, stream)\u001b[39m\n\u001b[32m 1412\u001b[39m httpx_request = \u001b[38;5;28mself\u001b[39m._httpx_client.build_request( \u001b[38;5;66;03m# type: ignore[union-attr]\u001b[39;00m\n\u001b[32m 1413\u001b[39m method=http_request.method,\n\u001b[32m 1414\u001b[39m url=http_request.url,\n\u001b[32m (...)\u001b[39m\u001b[32m 1417\u001b[39m timeout=http_request.timeout,\n\u001b[32m 1418\u001b[39m )\n\u001b[32m 1419\u001b[39m response = \u001b[38;5;28mself\u001b[39m._httpx_client.send(httpx_request, stream=stream) \u001b[38;5;66;03m# type: ignore[union-attr]\u001b[39;00m\n\u001b[32m-> \u001b[39m\u001b[32m1420\u001b[39m \u001b[30;43merrors\u001b[39;49m\u001b[30;43m.\u001b[39;49m\u001b[30;43mAPIError\u001b[39;49m\u001b[30;43m.\u001b[39;49m\u001b[30;43mraise_for_response\u001b[39;49m\u001b[30;43m(\u001b[39;49m\u001b[30;43mresponse\u001b[39;49m\u001b[30;43m)\u001b[39;49m\n\u001b[32m 1421\u001b[39m \u001b[38;5;28;01mreturn\u001b[39;00m HttpResponse(\n\u001b[32m 1422\u001b[39m response.headers, response \u001b[38;5;28;01mif\u001b[39;00m stream \u001b[38;5;28;01melse\u001b[39;00m [response.text]\n\u001b[32m 1423\u001b[39m )\n", "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/lib/python3.13/site-packages/google/genai/errors.py:155\u001b[39m, in \u001b[36mAPIError.raise_for_response\u001b[39m\u001b[34m(cls, response)\u001b[39m\n\u001b[32m 152\u001b[39m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[32m 153\u001b[39m response_json = response.body_segments[\u001b[32m0\u001b[39m].get(\u001b[33m'\u001b[39m\u001b[33merror\u001b[39m\u001b[33m'\u001b[39m, {})\n\u001b[32m--> \u001b[39m\u001b[32m155\u001b[39m \u001b[30;43mcls\u001b[39;49m\u001b[30;43m.\u001b[39;49m\u001b[30;43mraise_error\u001b[39;49m\u001b[30;43m(\u001b[39;49m\u001b[30;43mresponse\u001b[39;49m\u001b[30;43m.\u001b[39;49m\u001b[30;43mstatus_code\u001b[39;49m\u001b[30;43m,\u001b[39;49m\u001b[30;43m \u001b[39;49m\u001b[30;43mresponse_json\u001b[39;49m\u001b[30;43m,\u001b[39;49m\u001b[30;43m \u001b[39;49m\u001b[30;43mresponse\u001b[39;49m\u001b[30;43m)\u001b[39;49m\n", "\u001b[36mFile \u001b[39m\u001b[32m~/miniconda3/lib/python3.13/site-packages/google/genai/errors.py:186\u001b[39m, in \u001b[36mAPIError.raise_error\u001b[39m\u001b[34m(cls, status_code, response_json, response)\u001b[39m\n\u001b[32m 184\u001b[39m \u001b[38;5;28;01mraise\u001b[39;00m ClientError(status_code, response_json, response)\n\u001b[32m 185\u001b[39m \u001b[38;5;28;01melif\u001b[39;00m \u001b[32m500\u001b[39m <= status_code < \u001b[32m600\u001b[39m:\n\u001b[32m--> \u001b[39m\u001b[32m186\u001b[39m \u001b[38;5;28;01mraise\u001b[39;00m ServerError(status_code, response_json, response)\n\u001b[32m 187\u001b[39m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[32m 188\u001b[39m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;28mcls\u001b[39m(status_code, response_json, response)\n", "\u001b[31mServerError\u001b[39m: 503 UNAVAILABLE. {'error': {'code': 503, 'message': 'This model is currently experiencing high demand. Spikes in demand are usually temporary. Please try again later.', 'status': 'UNAVAILABLE'}}" ] } ], "source": [ "import os\n", "import time\n", "import base64 \n", "from google import genai\n", "from google.genai.errors import APIError\n", "\n", "os.environ[\"GEMINI_API_KEY\"] = \"AQ.Ab8RN6K9G5S-h6IWgpUgfmHbIk7b8iZlWTuYptMFqOT4Ph6FnQ\"\n", "client = genai.Client()\n", "\n", "# Use triple quotes \"\" \" so you can paste long paragraphs easily!\n", "\n", "prompt_instructions_simple = f\"Rewrite this text for a university course. Make it clear and write WCAG 2.1 AA compliant alt text descriptions for any charts mentioned: {my_notebook_text}\"\n", " \n", "\n", "prompt_instructions_detailed1 = (\n", " \"Rewrite the following text for a university course to make it clear and engaging. \"\n", " \"For any charts, graphs, or data visuals mentioned, do NOT write short alt text. Instead, \"\n", " \"provide a comprehensive Detailed Description. \"\n", " \"Format each Detailed Description using this structure:\\n\"\n", " \"1. Overview: One sentence describing what the chart is showing.\\n\"\n", " \"2. Setup: Explain the axes, labels, units, and layout.\\n\"\n", " \"3. Key Takeaways: Break down the main data trends, peaks, or critical data points.\\n\\n\"\n", " f\"Course Text:\\n{my_notebook_text}\"\n", ")\n", "# Force the model to include structured Detailed Descriptions AND Text-Based Diagrams\n", "\n", "prompt_instructions_detailed2 = (\n", " \"Rewrite the following text for a university course to make it clear and engaging. \"\n", " \"For EVERY chart, graph, diagram, or image mentioned in the text, you MUST generate \"\n", " \"two components:\\n\\n\"\n", " \n", " \"A) A Comprehensive Detailed Description:\\n\"\n", " \" 1. Overview: One sentence describing what the visual is showing.\\n\"\n", " \" 2. Setup: Explain the axes, labels, units, and layout.\\n\"\n", " \" 3. Key Takeaways: Break down the main data trends, peaks, or critical data points.\\n\\n\"\n", " \n", " \"B) A Text-Based Diagram:\\n\"\n", " \" - Create a clean, text-only layout of the visual asset.\\n\"\n", " \" - Use structured Markdown tables for grid-based data, charts, and plots.\\n\"\n", " \" - Use clean ASCII art flowcharts (e.g., [Step 1] -> [Step 2]) for process diagrams.\\n\"\n", " \" - Never skip this component. Every single visual asset must have a text-based representation.\\n\\n\"\n", " \n", " f\"Course Text:\\n{my_notebook_text}\"\n", ")\n", "\n", "# Prompt tailored strictly for Jupyter Notebook Markdown injection\n", "prompt_instructions_detailed3 = (\n", " \"You are processing raw course text meant for Jupyter Notebook markdown cells.\\n\\n\"\n", " \n", " \"CRITICAL RULE: Do NOT delete, alter, or remove any original image links, image paths, \"\n", " \"or image tags (e.g., lines starting with `![...](...)` or containing ``). \"\n", " \"Leave them exactly where they are in the original text sequence.\\n\\n\"\n", " \n", " \"INSTRUCTION:\\n\"\n", " \"Rewrite the text to make it clear and university-appropriate. Immediately BELOW the \"\n", " \"original untouched image link, add a clean Markdown block containing these two elements:\\n\\n\"\n", " \n", " \"A) A Comprehensive Detailed Description:\\n\"\n", " \" 1. Overview: One sentence describing what the visual shows at a high level.\\n\"\n", " \" 2. Setup: Explain the primary components, labels, and basic layout without hyper-granular technical specs.\\n\"\n", " \" 3. Key Takeaways: Core conceptual trends or interactions shown.\\n\\n\"\n", " \n", " \"B) A Text-Based Diagram:\\n\"\n", " \" - Provide a clean, text-only ASCII or Markdown table layout of the visual asset.\\n\"\n", " \" - Keep it high-level, clear, and perfectly formatted for a standard markdown code block.\\n\\n\"\n", " \n", " f\"Course Text:\\n{my_notebook_text}\"\n", ")\n", "\n", "prompt_instructions_detailed4 = (\n", " \"You are processing raw course text meant for Jupyter Notebook markdown cells.\\n\\n\"\n", " \n", " \"CRITICAL RULE: Do NOT delete, alter, or remove any original image links, image paths, \"\n", " \"or image tags (e.g., lines starting with `![...](...)` or containing ``). \"\n", " \"Leave them exactly where they are in the original text sequence.\\n\\n\"\n", " \n", " \"INSTRUCTION:\\n\"\n", " \"Rewrite the text to make it clear and university-appropriate. Use the following strict sequence \"\n", " \"when dealing with image placements:\\n\\n\"\n", " \n", " \"1. First, insert the NEW Text-Based Structured Alternative:\\n\"\n", " \" - For numerical data charts or grid plots: Provide a clean Markdown Data Table summarizing the information.\\n\"\n", " \" - For architecture, networks, or process workflows: Provide a text flowchart inside a preformatted code block using simple structural symbols like `[Box A] ===> [Box B]` and `|` or `v` paths.\\n\"\n", " \" - Ensure the structure is perfectly neat, readable, and uses standard characters that export flawlessly to static HTML.\\n\\n\"\n", " \n", " \"2. Second, place the ORIGINAL untouched image link exactly as it was.\\n\\n\"\n", " \n", " \"3. Third, immediately BELOW the original image link, add the Comprehensive Detailed Description:\\n\"\n", " \" - Overview: One sentence describing what the visual shows at a high level.\\n\"\n", " \" - Setup: Explain the primary components, labels, axes, or layouts without hyper-granular technical specs.\\n\"\n", " \" - Key Takeaways: Core conceptual trends or interactions shown.\\n\\n\"\n", " \n", " f\"Course Text:\\n{my_notebook_text}\"\n", ")\n", "\n", "prompt_instructions_detailed5 = (\n", " \"You are processing raw course text meant for Jupyter Notebook markdown cells.\\n\\n\"\n", " \n", " \"CRITICAL RULE: Do NOT delete, alter, or remove any original image links, image paths, \"\n", " \"or image tags (e.g., lines starting with `![...](...)` or containing ``). \"\n", " \"Leave them exactly where they are in the original text sequence.\\n\\n\"\n", " \n", " \"INSTRUCTION:\\n\"\n", " \"Rewrite the text to make it clear and university-appropriate.Make it clear and write WCAG 2.1 AA compliant. Use the following strict sequence \"\n", " \"when dealing with image placements:\\n\\n\"\n", " \n", " \"1. First, insert the NEW Text-Based Structured Alternative ABOVE the image:\\n\"\n", " \" - For numerical data charts or grid plots: Provide a clean Markdown Data Table.\\n\"\n", " \" - For hardware architecture, network layouts, or OS Process lifecycles (like fork(), process states, execution threads): \"\n", " \"Provide a high-level text flowchart inside a preformatted ```text code block.\\n\"\n", " \" - For Terminal Commands or Pipelines (like stdin/stdout piping, shell interactions): \"\n", " \"Provide a vertical box-and-arrow sequence map illustrating data flow from one utility to the next.\\n\"\n", " \" - Keep diagram characters perfectly aligned and compatible with static HTML page exports.\\n\\n\"\n", " \" - CRITICAL RULE FOR VIRTUAL DESKTOP LOGIN CONTEXT:\\n\"\n", " \" If the text or diagram is about logging into the Virtual Desktop or using Ravada, \"\n", " \"the text diagram sequence MUST start exactly with this workflow path:\\n\"\n", " \" [Web access to Ravada portal with authentication] ===> [Start VM] ===> [Access VM via SSH with provided port]\\n\\n\"\n", " \n", " \"2. Second, place the ORIGINAL untouched image link exactly as it was.\\n\\n\"\n", " \n", " \"3. Third, immediately BELOW the original image link, add the Comprehensive Detailed Description:\\n\"\n", " \"Inside this caption block, you MUST start with the exact text '**Accessible Image Description:**'. Place it always under the image.\"\n", " \" - Overview: One sentence describing what the visual shows at a high level.\\n\"\n", " \" - Setup: Explain the primary components, labels, axes, or code pathways without hyper-granular specs.\\n\"\n", " \" - Key Takeaways: Core conceptual trends or execution results shown.\\n\\n\"\n", "\n", " f\"Course Text:\\n{my_notebook_text}\"\n", ")\n", "prompt_instructions = prompt_instructions_detailed5\n", "\n", "# Trying the 2.5-flash model which often has lower server congestion\n", "try:\n", " response = client.models.generate_content(\n", " model='models/gemini-3.6-flash',\n", " contents= prompt_instructions,\n", " )\n", "\n", " #print(response.text)\n", "\n", " output_text = response.text\n", "\n", " # Post-process the text to extract the Mermaid syntax and convert it to a valid web image\n", " if \"[MERMAID_START]\" in output_text:\n", " start_idx = output_text.find(\"[MERMAID_START]\") + len(\"[MERMAID_START]\\n\")\n", " end_idx = output_text.find(\"[MERMAID_END]\")\n", " mermaid_code = output_text[start_idx:end_idx].strip()\n", " \n", " # Base64 encode the string for the Mermaid.ink service\n", " encoded_bytes = base64.b64encode(mermaid_code.encode('utf-8'))\n", " encoded_string = encoded_bytes.decode('utf-8')\n", " image_url = f\"https://mermaid.ink{encoded_string}\"\n", " \n", " # Replace the raw markers with a real markdown image tag\n", " compiled_tag = f\"#### Text-Based Flowchart Diagram\\n![Alternative Text Diagram]({image_url})\"\n", " final_output = output_text[:output_text.find(\"[MERMAID_START]\")] + compiled_tag + output_text[end_idx + len(\"[MERMAID_END]\"):]\n", " print(final_output)\n", " else:\n", " print(output_text)\n", " \n", " # --- THE MAGIC TRICK ---\n", " # This forces Jupyter to create a new cell below and paste the AI response inside it!\n", " #get_ipython().set_next_input(response.text)\n", " \n", "except APIError as e:\n", " if e.code == 503:\n", " print(\"Servers are still busy. Waiting 5 seconds before a quick automatic retry...\")\n", " time.sleep(5)\n", " # Automatic second attempt\n", " response = client.models.generate_content(\n", " model='gemini-3.6-flash',\n", " contents=f\"Rewrite this text for a university course. Make it clear and write WCAG 2.1 AA compliant alt text descriptions for any charts mentioned: {my_notebook_text}\",)\n", " print(response.text)\n", " # --- THE MAGIC TRICK ---\n", " # This forces Jupyter to create a new cell below and paste the AI response inside it!\n", " #get_ipython().set_next_input(response.text)\n", " else:\n", " raise e" ] }, { "cell_type": "markdown", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "# Commands, Shells, Processes, and Files\n", "\n", "## Overview\n", "\n", "This module covers the overall course organization and introduces fundamental Linux operating system concepts:\n", "* The Linux kernel\n", "* Process lifecycles and management\n", "* File system structure and permissions\n", "* User accounts and authentication\n", "* Basic terminal commands and shell execution\n", "\n", "---\n", "\n", "## Course Organization\n", "\n", "* Remote Virtual Desktop (VD) Access\n", "* Canvas Learning Management System\n", "* Class Slack Channel\n", "* Official Course Website\n", "* Interactive Lectures\n", "* Hands-on Practical Exercises (recitations and homeworks)\n", "* Midterm and Final Examinations\n", "\n", "---\n", "\n", "## Course Agenda\n", "\n", "* **Lectures & Practical Recitations:** Conducted remotely using dedicated virtual machines.\n", "* **Homework Assignments:** Completed on the virtual desktops. Checked in place by the instructor.\n", "* **Midterm Examination:** October 20, 2026.\n", "* **Final Examination:** December 15, 2026.\n", "\n", "---\n", "\n", "## Course Topics\n", "\n", "* **Command Line Interface (CLI):** Core system commands, directory traversal, process control, user accounts.\n", "* **Operating System Virtualization:** KVM virtualization and VM management workflows.\n", "* **Package Management:** Software installation using `apt`, `dnf`, `rpm`, and `dpkg`.\n", "* **Networking:** TCP/IP concepts, diagnostic utilities, interface configuration, network services.\n", "* **Network File System (NFS):** Remote storage mounting and sharing between Linux systems.\n", "* **Shell Scripting:** `bash` scripting, `awk`, `grep`, `sed`, regular expressions.\n", "* **Task Control and Scheduling:** Systemd, systemctl, and unit files. Deferred execution via `at` and automated recurring jobs via `cron`.\n", "* **Python for Linux administration:** venv and Conda environments, Jupyter Notebooks, various modules, and scripting.\n", "* **Software Compilation:** Source code compilation, Makefiles, shared and static libraries.\n", "* **System Security:** Password security policies, brute-force mitigation, `iptables`, port scanning analysis.\n", "* **GPU Computing:** CUDA architecture, PyTorch framework, neural networks.\n", "\n", "---\n", "\n", "## Hardware Architecture Overview (Intel Core i7 Example)\n", "\n", "* **Execution Flow:** The operating system and applications load from disk storage into system RAM.\n", "* **Kernel Control:** The operating system kernel manages hardware devices and process scheduling.\n", "* **Key Components:**\n", " * **CPU:** Central Processing Unit (executes program instructions).\n", " * **RAM:** Random Access Memory (main system storage for active processes).\n", " * **PCIe:** Peripheral Component Interconnect Express (high-speed expansion bus, such as PCIe 3.0/4.0/5.0).\n", " * **PCH:** Platform Controller Hub (coordinates communications between PCIe, USB, and SATA controllers).\n", " * **DMI:** Direct Memory Interface (interconnects CPU and PCH).\n", " * **LPC:** Low Pin Count bus (connects lower-bandwidth I/O devices).\n", "\n", "```text\n", "+-------------------------------------------------------+\n", "| CPU |\n", "| +---------------------+ +---------------------+ |\n", "| | CPU Cores | | Memory Controller | |\n", "| +---------------------+ +----------+----------+ |\n", "+-----------------------------------------|-------------+\n", " | System Bus / DMI\n", "+-----------------------------------------v-------------+\n", "| PCH (Platform Controller Hub) |\n", "| +------------------+ +---------------+ +--------+ |\n", "| | PCIe Controllers | | SATA / Storage| | USB / | |\n", "| | (High-Speed Bus) | | (HDDs / SSDs) | | LPC | |\n", "| +------------------+ +---------------+ +--------+ |\n", "+-------------------------------------------------------+\n", "```\n", "![title](img/Chipset_CPU.jpg)\n", "\n", "**Accessible Image Description:**\n", "- Overview: Block diagram of a standard computer hardware architecture showing relationships between the CPU, RAM, and Platform Controller Hub (PCH).\n", "- Setup: The CPU contains execution cores and a memory controller linked directly to system RAM, while communicating with peripherals and storage via the DMI link to the PCH.\n", "- Key Takeaways: High-speed subsystems like RAM connect directly to the CPU, whereas lower-speed input/output buses route through the PCH chipset.\n", "\n", "---\n", "\n", "## Purpose of Operating Systems\n", "\n", "Refer to standard [Operating System Concepts](https://en.wikipedia.org/wiki/Operating_system) for background reading.\n", "\n", "### Core Functions\n", "* [Device Drivers](https://en.wikipedia.org/wiki/Device_driver)\n", "* [Interrupt Processing](https://en.wikipedia.org/wiki/Interrupt)\n", "* [Process Management](https://en.wikipedia.org/wiki/Process_management_(computing))\n", "* [Memory Management](https://en.wikipedia.org/wiki/Memory_management)\n", "* [System Security](https://en.wikipedia.org/wiki/Computer_security)\n", "* [File System Control](https://en.wikipedia.org/wiki/File_system)\n", "* [Input/Output (I/O) Handling](https://en.wikipedia.org/wiki/Input/output)\n", "* [Networking](https://en.wikipedia.org/wiki/Computer_network)\n", "\n", "```text\n", "+-------------------------------------------------------+\n", "| User Applications |\n", "| (Web Browsers, Editors, Python Scripts) |\n", "+-------------------------------------------------------+\n", " | System Calls\n", "+-------------------------------------------------------+\n", "| Operating System |\n", "| (Kernel, System Libraries, Device Drivers) |\n", "+-------------------------------------------------------+\n", " | Driver Interfaces\n", "+-------------------------------------------------------+\n", "| Computer Hardware |\n", "| (CPU, RAM, Disk Storage, Network Cards) |\n", "+-------------------------------------------------------+\n", "```\n", "![title](img/Operating_system_placement.png)\n", "\n", "**Accessible Image Description:**\n", "- Overview: Layered computing stack diagram highlighting the operational placement of the operating system.\n", "- Setup: User applications occupy the top layer, interacting with the middle operating system layer via system calls, which manages the physical hardware at the base layer.\n", "- Key Takeaways: The operating system provides a necessary hardware abstraction layer between user-level software applications and raw physical components.\n", "\n", "---\n", "\n", "## Linux Kernel and Applications\n", "\n", "Strictly speaking, \"Linux\" refers specifically to the OS kernel rather than a full distribution.\n", "\n", "* **Linux Distributions:** Combine a plain or modified kernel with standard GNU software tools, libraries, system shells, and user applications.\n", "* **Open Source:** Source code for the Linux kernel, shells, and standard utilities is freely inspectable and configurable.\n", "* **Core Administration Languages:** C, Shell Scripting (`bash`), and Python.\n", "\n", "```text\n", "+-------------------------------------------------------+\n", "| User Space: Applications, Shells, Compilers |\n", "+-------------------------------------------------------+\n", " | System Calls Interface (POSIX)\n", "+-------------------------------------------------------+\n", "| Kernel Space: Process & Memory Management, |\n", "| File Systems, Networking & Device Drivers |\n", "+-------------------------------------------------------+\n", " | Hardware Interfaces\n", "+-------------------------------------------------------+\n", "| Hardware: CPU, Memory, Disk Drives, Network |\n", "+-------------------------------------------------------+\n", "```\n", "![Linux kernel](img/linux_kerne_app.jpg)\n", "\n", "**Accessible Image Description:**\n", "- Overview: Architecture schematic illustrating how the Linux kernel mediates between application software and underlying hardware.\n", "- Setup: Applications run in user space, passing requests through standardized POSIX system call interfaces to kernel subsystems, which govern hardware execution.\n", "- Key Takeaways: Linux enforces operational isolation between user-space programs and kernel-space core routines to guarantee system security and fault tolerance.\n", "\n", "---\n", "\n", "## Virtual Desktop Access Workflow\n", "\n", "To connect to your virtual environment, execute the standardized workflow:\n", "\n", "```text\n", "[Web access to Ravada portal with authentication] ===> [Start VM] ===> [Access VM via SSH with provided port]\n", "```\n", "![Connect to your VD in Ravada cluster](img/Ravada_connection_SSH.png)\n", "\n", "**Accessible Image Description:**\n", "- Overview: User interface connection guide showing how to access a Virtual Desktop instance within the Ravada management portal.\n", "- Setup: Shows web authentication into the Ravada cluster, launching the assigned virtual host, and noting the dynamic port required for an SSH client connection.\n", "- Key Takeaways: Remote lab access relies on web-based VM orchestration followed by targeted port redirection via SSH.\n", "\n", "---\n", "\n", "## Logging into a Linux Terminal\n", "\n", "```text\n", "+---------------+ Prompt User +---------------------+\n", "| User Terminal | ------------------> | Username / Password |\n", "+---------------+ +----------+----------+\n", " |\n", " v\n", " [Authentication System]\n", " (/etc/passwd & /etc/shadow)\n", " |\n", " +-----------------+-----------------+\n", " | |\n", " (Success) (Failure)\n", " | |\n", " v v\n", " [Interactive Shell Spawning] [Access Denied Warning]\n", "```\n", "![Login to Linux System](img/user_login.jpg)\n", "\n", "**Accessible Image Description:**\n", "- Overview: Authentication flow map displaying terminal initial login steps.\n", "- Setup: Illustrates credentials validation through user authentication files, culminating in an established user shell context upon success.\n", "- Key Takeaways: Validating user identity initializes an interactive command shell linked directly to user account privileges.\n", "\n", "---\n", "\n", "## User Accounts and Authentication\n", "\n", "Logging in requires authenticating user credentials (username and password).\n", "\n", "### Authentication Methods\n", "* **Local Authentication:** Uses internal files (`/etc/passwd`, `/etc/shadow`, `/etc/group`).\n", "* **Domain Authentication:** Uses centralized services (NIS, Kerberos, LDAP, Active Directory).\n", "\n", "### Standard Account Files\n", "\n", "| Account Configuration File | Example Entry Syntax | Description |\n", "| :--- | :--- | :--- |\n", "| `/etc/passwd` | `mike:x:1001:1001:Michael Whites:/home/mike:/bin/bash` | User profile, UID, GID, home directory, default shell |\n", "| `/etc/shadow` | `mike:$1$zXCV7fz8ii84grbZhj:14087:0:99999:7:::` | Encrypted user password hash and aging policies |\n", "| `/etc/group` | `mike:x:1001:mike` | Primary user group definition |\n", "| `/etc/group` | `admin:x:112:hostadm,mike` | Secondary group memberships |\n", "\n", "> **Note:** Only the superuser (`root`) can create or modify system accounts.\n", "\n", "### Account Creation Commands\n", "* `adduser`: Interactive high-level script for user creation.\n", "* `useradd`: Low-level system utility suitable for non-interactive automation scripts.\n", "\n", "---\n", "\n", "### Exercise: Account Creation\n", "\n", "1. Elevate session privilege to `root`:\n", " ```bash\n", " sudo -s\n", " ```\n", "\n", "2. Create a new account named `mike`:\n", " ```bash\n", " adduser mike\n", " ```\n", "\n", "3. Confirm account creation and review identity identifiers (UID/GID):\n", " ```bash\n", " id mike\n", " ```\n", "\n", "4. Update the account password:\n", " ```bash\n", " passwd mike\n", " ```\n", "\n", "---\n", "\n", "## Linux Shell Environment\n", "\n", "Upon successful login, an interactive shell processes user input. Default Linux shell: `bash`. Available shells are enumerated in `/etc/shells`.\n", "\n", "Commands, scripts, and applications executed in the shell run as child processes of that shell.\n", "\n", "### Shell Variables vs. Environment Variables\n", "\n", "Define a standard shell variable:\n", "```bash\n", "svar='file_name.txt'\n", "echo $svar\n", "```\n", "\n", "Shell variables remain local to the active shell process. Exporting converts a local variable into an environment variable, allowing child processes to inherit it:\n", "\n", "```bash\n", "svar='VAR_1' # Initialize local variable\n", "export svar # Promote to environment variable\n", "env | grep svar # Confirm export status\n", "```\n", "\n", "Commands entered in the terminal must be shell built-ins, specified via explicit file path, or located within directories declared in the `PATH` environment variable:\n", "\n", "```bash\n", "echo $PATH\n", "```\n", "\n", "---\n", "\n", "### Exercise: Variable Scope and Inheritance\n", "\n", "1. Define a local shell variable:\n", " ```bash\n", " svar='VAR_1'\n", " echo $svar\n", " ```\n", "\n", "2. Spawn a child shell process and test variable access (returns empty):\n", " ```bash\n", " bash\n", " echo $svar\n", " ```\n", "\n", "3. Terminate child shell, export variable, and verify inheritance in a new child shell:\n", " ```bash\n", " exit\n", " export svar\n", " bash\n", " echo $svar\n", " exit\n", " ```\n", "\n", "---\n", "\n", "## Data Storage Architecture Hierarchy\n", "\n", "* **Data Objects:** Stored within structured files.\n", "* **File Systems:** Built on logical storage volumes.\n", "* **Volumes:** Spanning single or multiple physical drive devices.\n", "* **Mounting:** Attaching a file system to a directory node in the system directory tree.\n", "\n", "```text\n", "+-------------------------------------------------------+\n", "| Physical Drives |\n", "| [ Disk 1 ] [ Disk 2 ] [ Disk 3 ] |\n", "+-------------------------------------------------------+\n", " | Aggregated / Partitioned\n", " v\n", "+-------------------------------------------------------+\n", "| Logical Volume |\n", "+-------------------------------------------------------+\n", " | Formatted with\n", " v\n", "+-------------------------------------------------------+\n", "| File System (ext4) |\n", "+-------------------------------------------------------+\n", " | Attached via\n", " v\n", "+-------------------------------------------------------+\n", "| Mount Point Directory |\n", "| (e.g., /home) |\n", "+-------------------------------------------------------+\n", "```\n", "![How data is stored](img/Device_and_file_system.png)\n", "\n", "**Accessible Image Description:**\n", "- Overview: Layered block diagram representing storage abstractions from physical drives to mounted directories.\n", "- Setup: Illustrates physical disks combined into logical storage volumes, formatted with a file system, and linked to a mount directory.\n", "- Key Takeaways: Physical disk blocks are logically abstracted by file systems to present seamless file paths to operating system users.\n", "\n", "---\n", "\n", "## HDD and SSD Hardware Characteristics\n", "\n", "### Hard Disk Drives (HDD)\n", "\n", "```text\n", "+-------------------------------------------------------+\n", "| Hard Disk Drive Structure |\n", "| +-------------------------------------------------+ |\n", "| | Platter / Cylinders (Partition Boundaries) | |\n", "| | +-------------------------------------------+ | |\n", "| | | Sectors / Tracks (Individual Data Blocks) | | |\n", "| | +-------------------------------------------+ | |\n", "| +-------------------------------------------------+ |\n", "+-------------------------------------------------------+\n", "```\n", "```{image} img/hd_partitions.jpg\n", ":alt: Mechanical hard drive structure and partitioning\n", ":class: bg-primary mb-1\n", ":width: 200px\n", ":align: center\n", "```\n", "\n", "**Accessible Image Description:**\n", "- Overview: Diagram illustrating mechanical hard drive physical geometry including platters, tracks, and partitions.\n", "- Setup: Displays rotating platters divided into concentric tracks and sectors, grouped into logical partitions.\n", "- Key Takeaways: Disk partitions follow mechanical cylinder boundaries, defining layout areas for file systems.\n", "\n", "* **Geometry:** Cylinders establish partition boundaries; sectors define individual data blocks.\n", "* **Operational Lifespan:** Typical mechanical HDD endurance is approximately 5 years under continuous operation.\n", "\n", "---\n", "\n", "### Solid State Drives (SSD) and NAND Technology\n", "\n", "```text\n", "+-------------------------------------------------------+\n", "| NAND Flash Cell States |\n", "| [ Programmed State ] <-----> [ Erased State ] |\n", "| (Trapped Electrons/0) (High Voltage/1) |\n", "+-------------------------------------------------------+\n", "```\n", "```{image} img/NAND-Cell-Read-Programmed-and-Erased-fd945b79.png\n", ":alt: NAND flash memory cell state transitions\n", ":class: bg-primary mb-1\n", ":width: 600px\n", ":align: center\n", "```\n", "\n", "**Accessible Image Description:**\n", "- Overview: Schematic of NAND cell charging states during read, program, and erase operations.\n", "- Setup: Shows floating-gate electron trapping mechanisms that delineate binary 1 and 0 logical states.\n", "- Key Takeaways: Solid-state storage retains data non-volatilely by adjusting electrical charge levels in micro-scale semiconductor cells.\n", "\n", "```text\n", "+-------------------------------------------------------+\n", "| SSD Architecture |\n", "| +-------------------------------------------------+ |\n", "| | Memory Controller & Host Interface (SATA/NVMe) | |\n", "| +-------------------------------------------------+ |\n", "| | |\n", "| +-------------------------------------------------+ |\n", "| | NAND Flash Arrays (Banks -> Blocks -> Pages) | |\n", "| +-------------------------------------------------+ |\n", "+-------------------------------------------------------+\n", "```\n", "```{image} img/Basic-SSD-Diagram-6b8c8849.png\n", ":alt: Internal functional block diagram of a Solid State Drive\n", ":class: bg-primary mb-1\n", ":width: 600px\n", ":align: center\n", "```\n", "\n", "**Accessible Image Description:**\n", "- Overview: Architecture block layout showing how NAND flash cells form pages, blocks, planes, and controller channels.\n", "- Setup: Details how memory components connect to an integrated host controller managing parallel data channels.\n", "- Key Takeaways: SSD throughput is maximized by organizing individual NAND cells into parallel access arrays.\n", "\n", "---\n", "\n", "### NAND Cell Endurance Comparison\n", "\n", "| Metric / Cell Type | SLC (Single-Level Cell) | MLC (Multi-Level Cell) | TLC (Triple-Level Cell) | QLC (Quad-Level Cell) |\n", "| :--- | :--- | :--- | :--- | :--- |\n", "| **Bits per Cell** | 1 bit | 2 bits | 3 bits | 4 bits |\n", "| **Write Endurance** | ~100,000 P/E cycles | ~10,000 P/E cycles | ~3,000 P/E cycles | ~1,000 P/E cycles |\n", "\n", "---\n", "\n", "### Hardware Metrics Comparison\n", "\n", "| Device Type | Capacity | Average Cost | Read/Write Throughput | Latency | Random IOPS |\n", "| :--- | :--- | :--- | :--- | :--- | :--- |\n", "| **SATA Mechanical HDD** | 20 TB | ~$800 | ~120 MB/s | ~5.0 ms | ~100 |\n", "| **SATA SSD** | 4 TB | ~$1,400 | ~500 MB/s | ~0.05 ms | ~100,000 |\n", "| **NVMe PCIe SSD** | 4 TB | ~$500 | ~7,000 MB/s | ~0.02 ms | ~500,000 |\n", "\n", "---\n", "\n", "## Inodes, Files, and Directories\n", "\n", "```text\n", "+-------------------+ +------------------------------------+\n", "| Directory Entry | | Inode Table Entry |\n", "| Filename: file1 | ----> | Inode Number: 104523 |\n", "| Inode ID: 104523 | | Permissions: rw-r--r-- |\n", "+-------------------+ | Owner/Group: mike/staff |\n", " | Size: 4096 bytes |\n", " | Data Block Pointers: [B1, B2, B3] |\n", " +------------------------------------+\n", "```\n", "```{image} img/directory_entry.jpg\n", ":alt: Mapping between directory entries and system inodes\n", ":class: bg-primary mb-1\n", ":width: 200px\n", ":align: center\n", "```\n", "\n", "**Accessible Image Description:**\n", "- Overview: Schematic displaying directory record references linking filenames to internal inode records.\n", "- Setup: A directory entry contains filename strings and pointer references to an inode entry storing attributes and data block locations.\n", "- Key Takeaways: Directory records store human-readable names, while file system metadata and physical layout are decoupled inside inodes.\n", "\n", "### Inode Metadata Attributes\n", "* Data block location pointers\n", "* Hard link count\n", "* User owner (UID) and group owner (GID)\n", "* Access permission bits (mode)\n", "* Byte size\n", "* Timestamps: Access (`atime`), Modification (`mtime`), Metadata Change (`ctime`)\n", "\n", "### File Types and Allocation\n", "* **File Types:** Regular files, Directories, Symbolic links, Device node files (block/character).\n", "* **Block Allocation:** Storage space is allocated in discrete block sizes (e.g., standard 4 KB blocks on `ext4`).\n", "\n", "```bash\n", "# Check block size of file system\n", "stat -f /home\n", "```\n", "\n", "---\n", "\n", "## File Permissions and Ownership\n", "\n", "### Permission Bit Modes\n", "\n", "| Octal Value | Binary Representation | String Mode | Access Capabilities |\n", "| :---: | :---: | :---: | :--- |\n", "| **0** | `000` | `---` | No permissions granted |\n", "| **1** | `001` | `--x` | Execute only |\n", "| **2** | `010` | `-w-` | Write only |\n", "| **3** | `011` | `-wx` | Write and Execute |\n", "| **4** | `100` | `r--` | Read only |\n", "| **5** | `101` | `r-x` | Read and Execute |\n", "| **6** | `110` | `rw-` | Read and Write |\n", "| **7** | `111` | `rwx` | Read, Write, and Execute |\n", "\n", "### Permission Classes\n", "Permissions apply across three scopes: **User Owner** (`u`), **Group Owner** (`g`), and **Others** (`o`).\n", "\n", "Example permission modification:\n", "```bash\n", "chmod 660 testf.txt\n", "ls -l testf.txt\n", "# Output: -rw-rw---- 1 mike staff 0 Jan 18 10:26 testf.txt\n", "```\n", "\n", "### Unmasking Default File Creation Modes (`umask`)\n", "The system `umask` subtracts permission bits from base defaults (Directories base: `777`, Files base: `666`).\n", "\n", "| Context | Base Default Permissions | `umask` Setting | Effective Access Mode | Mode Notation |\n", "| :--- | :---: | :---: | :---: | :---: |\n", "| **Directory** | `777` | `022` | `755` | `rwxr-xr-x` |\n", "| **Regular File** | `666` | `022` | `644` | `rw-r--r--` |\n", "\n", "---\n", "\n", "### Exercise: File Permissions and Ownership Setup\n", "\n", "1. Create a practice workspace directory:\n", " ```bash\n", " mkdir EX1\n", " cd EX1\n", " ```\n", "\n", "2. Inspect `umask` and test default permission application:\n", " ```bash\n", " umask\n", " touch mf1.txt\n", " ls -l mf1.txt\n", " ```\n", "\n", "3. Modify `umask` and create a second file:\n", " ```bash\n", " umask 022\n", " touch mf2.txt\n", " ls -l mf2.txt\n", " ```\n", "\n", "4. Create and grant execution rights to an executable file:\n", " ```bash\n", " touch mf3.exe\n", " chmod 755 mf3.exe\n", " ls -l mf3.exe\n", " ```\n", "\n", "5. Test execution behavior:\n", " ```bash\n", " ./mf2.txt # Fails: Permission denied\n", " ./mf3.exe # Executes successfully\n", " ```\n", "\n", "6. Alter user and group ownership (requires superuser access):\n", " ```bash\n", " sudo chown mike mf1.txt\n", " sudo chown :mike mf2.txt\n", " sudo chown mike:mike mf3.exe\n", " ls -l mf*\n", " ```\n", "\n", "---\n", "\n", "### Exercise: Directory Sticky Bit Flags\n", "\n", "The sticky bit permission flag (`t`) on shared directories (e.g., `/tmp`) prevents non-owner users from removing or renaming files created by others.\n", "\n", "```bash\n", "ls -ld /tmp\n", "# Output format: drwxrwxrwt 4 root root 4096 Jul 25 16:29 /tmp\n", "```\n", "\n", "1. **Terminal 1 (Superuser Context):**\n", " ```bash\n", " sudo -s\n", " mkdir /tmp/temp\n", " chmod 777 /tmp/temp\n", " cd /tmp/temp\n", " touch ex3\n", " ```\n", "\n", "2. **Terminal 2 (Regular User Context):**\n", " ```bash\n", " cd /tmp/temp\n", " rm ex3 # File successfully deleted (Sticky bit not yet set)\n", " ```\n", "\n", "3. **Terminal 1 (Enable Sticky Bit):**\n", " ```bash\n", " touch /tmp/temp/ex3\n", " chmod 1777 /tmp/temp\n", " ```\n", "\n", "4. **Terminal 2 (Re-test Deletion):**\n", " ```bash\n", " cd /tmp/temp\n", " rm ex3 # Operation rejected: Permission denied\n", " ```\n", "\n", "---\n", "\n", "## Operating System Processes\n", "\n", "A running program instance in memory constitutes a **process**. Linux enforces multi-processing via hierarchical process spawning (forking from parent processes).\n", "\n", "### Process Categories\n", "* **User Processes:** Initiated by standard accounts operating entirely within user space.\n", "* **Daemon Processes:** Background services running continuously to handle system requests.\n", "* **Kernel Threads:** Operating system execution tasks executing strictly within kernel space.\n", "\n", "### Privilege Protection Rings\n", "\n", "```text\n", "+-------------------------------------------------------------+\n", "| Ring 3: User Space (User Processes, Programs, Daemons) |\n", "+------------------------------+------------------------------+\n", " | System Call Interface\n", " v\n", "+-------------------------------------------------------------+\n", "| Ring 0: Kernel Space (Kernel Threads, Drivers, System Core) |\n", "+-------------------------------------------------------------+\n", "```\n", "\n", "\n", "**Accessible Image Description:**\n", "- Overview: Privilege ring diagram detailing CPU hardware isolation levels between kernel and user execution modes.\n", "- Setup: Shows Ring 0 at the core (kernel space execution) outward to Ring 3 (unprivileged user space execution).\n", "- Key Takeaways: Rings provide security control preventing user processes in Ring 3 from overriding kernel memory or hardware instructions in Ring 0.\n", "\n", "---\n", "\n", "## Process Life Cycle States\n", "\n", "```text\n", "+----------+ Fork/Create +----------+ Scheduler +----------+\n", "| Created | --------------------> | Ready | ------------------> | Running |\n", "+----------+ +----------+ +----+-----+\n", " ^ |\n", " | Yield/Preempt |\n", " +--------------------------------+\n", " | |\n", " I/O Event Wait for I/O\n", " | |\n", " +----+-----+ |\n", " | Blocked | <------------------------+\n", " +----------+\n", " |\n", " | Exit / Terminate\n", " v\n", " +----------+\n", " | Zombie / |\n", " | Terminated|\n", " +----------+\n", "```\n", "\n", "\n", "**Accessible Image Description:**\n", "- Overview: Finite-state machine diagram illustrating process state transitions.\n", "- Setup: Maps process progress through Created, Ready, Running, Blocked (Waiting), and Terminated/Zombie states.\n", "- Key Takeaways: Process movement between states is managed dynamically by system resource calls and CPU scheduling interrupts.\n", "\n", "---\n", "\n", "## Process Monitoring, Signals, and Execution Priority\n", "\n", "* **Primary Attributes:** Process ID (`PID`), Parent Process ID (`PPID`), User ID (`UID`), Group ID (`GID`), Environment block, Current Working Directory (`CWD`).\n", "* **Root Init Process:** `PID 1` (traditionally `init` or `systemd`).\n", "* **Monitoring Tools:** `ps -ef`, `ps aux`, `top`, `htop`, `pstree`.\n", "\n", "### Common Signals (`kill`)\n", "\n", "```bash\n", "kill -HUP 1009 # Hangup (Signal 1): Request configuration reload\n", "kill -INT 1009 # Interrupt (Signal 2): Keyboard interrupt (Ctrl+C)\n", "kill -KILL 1009 # Kill (Signal 9): Immediate uncatchable termination\n", "kill -TERM 1009 # Terminate (Signal 15): Graceful shutdown request\n", "kill -TSTP 1009 # Terminal Stop (Signal 20): Suspend execution (Ctrl+Z)\n", "```\n", "\n", "### Scheduling Priority Adjustment (`nice` and `renice`)\n", "Niceness values range from **-20** (highest scheduling priority) to **+19** (lowest scheduling priority).\n", "\n", "* **Launch with Modified Priority:**\n", " ```bash\n", " nice -n 10 matlab # Lower priority (+10)\n", " nice -n -10 matlab # Higher priority (-10, superuser required)\n", " ```\n", "\n", "* **Modify Priority of Active PID 1009:**\n", " ```bash\n", " renice -p 5 -p 1009 # Adjust PID 1009 priority to +5\n", " ```\n", "\n", "---\n", "\n", "### Exercise: Process Control and Signal Termination\n", "\n", "1. Download and set up a CPU test script:\n", " ```bash\n", " wget http://capone.rutgers.edu/coursefiles/inf_loop.py\n", " chmod 755 inf_loop.py\n", " ```\n", "\n", "2. Execute script in the background:\n", " ```bash\n", " ./inf_loop.py &\n", " ```\n", "\n", "3. Query active PID using `pgrep`:\n", " ```bash\n", " pgrep -l inf_loop.py\n", " ```\n", "\n", "4. Terminate process gracefully using Signal 15 (`SIGTERM`):\n", " ```bash\n", " kill -15 \n", " ```\n", "\n", "5. Alternatively, terminate using target pattern matches:\n", " ```bash\n", " pkill -15 -f inf_loop.py\n", " ```\n", "\n", "---\n", "\n", "## Process Standard Data Streams\n", "\n", "Every standard Linux process initializes with three basic I/O descriptors:\n", "* **Standard Input (`stdin`):** File Descriptor `0` (reads keyboard input by default).\n", "* **Standard Output (`stdout`):** File Descriptor `1` (prints output to display monitor by default).\n", "* **Standard Error (`stderr`):** File Descriptor `2` (prints error logs to display monitor by default).\n", "\n", "```text\n", "[ Input Source (Keyboard/File) ] ---> stdin (FD 0)\n", " |\n", " v\n", " [ Operating Process ]\n", " / \\\n", " / \\\n", " stdout (FD 1) / \\ stderr (FD 2)\n", " v v\n", " [ Terminal / Output File ] [ Log File / Error Screen ]\n", "```\n", "\n", "### I/O Stream Redirection Examples\n", "\n", "```bash\n", "# Redirect standard output to a file (overwrites existing content)\n", "ps -ef > stdout.out\n", "\n", "# Append standard output to a file\n", "ps -ef >> stdout.out\n", "\n", "# Redirect both stdout and stderr to a single output destination\n", "ps -ef > combined_output.txt 2>&1\n", "\n", "# Discard standard output by streaming into null device\n", "ps -ef 1> /dev/null\n", "\n", "# Pipe stdout of process into stdin of reader process\n", "ps -ef | less\n", "```\n", "\n", "---\n", "\n", "### Exercise: Managing Background and Foreground Jobs\n", "\n", "1. Launch execution script interactively:\n", " ```bash\n", " ./inf_loop.py\n", " ```\n", "\n", "2. Suspend running task and move to background execution:\n", " ```text\n", " Press: Ctrl + Z\n", " ```\n", " ```bash\n", " jobs\n", " bg %1\n", " ```\n", "\n", "3. Return background job to active foreground session and cancel job:\n", " ```bash\n", " fg %1\n", " ```\n", " ```text\n", " Press: Ctrl + C\n", " ```\n", "\n", "---\n", "\n", "## Essential Practical Commands Reference\n", "\n", "### Directory Management\n", "\n", "```bash\n", "# 1. Display present working directory path\n", "pwd\n", "\n", "# 2. Change active directory to /etc\n", "cd /etc\n", "\n", "# 3. Return directly to user home directory\n", "cd $HOME # Or simply: cd ~\n", "\n", "# 4. Toggle back to previous directory node\n", "cd -\n", "\n", "# 5. Make a single directory node\n", "mkdir Newdir\n", "\n", "# 6. Traverse down and back up tree paths\n", "cd Newdir\n", "cd ..\n", "\n", "# 7. Remove empty directory\n", "rmdir Newdir\n", "\n", "# 8. Recursively create multi-tiered subdirectories\n", "mkdir -p Newdir/one/two/three\n", "\n", "# 9. Create files using touch compound commands\n", "cd Newdir/one; touch f1.t f2.t f3.t\n", "\n", "# 10. Recursively delete directory trees and contained contents\n", "rm -r Newdir\n", "\n", "# 11. Copy directory trees preserving permissions and attributes\n", "cp -a Newdir Newdir-1\n", "```\n", "\n", "---\n", "\n", "### Viewing Text File Content\n", "\n", "```bash\n", "# Display entire file contents\n", "cat /etc/hosts\n", "\n", "# Read file contents in reverse line order\n", "tac /etc/hosts\n", "\n", "# Page interactively through content\n", "less /etc/hosts\n", "\n", "# Print top or bottom 10 lines of file target\n", "head -n 10 /etc/nsswitch.conf\n", "tail -n 10 /etc/nsswitch.conf\n", "```\n", "\n", "---\n", "\n", "### System Information and Utilities\n", "\n", "```bash\n", "# Display current date, time, and system clock settings\n", "date\n", "\n", "# Display disk space allocation by file system in human-readable units\n", "df -h\n", "\n", "# Calculate directory tree storage usage\n", "du -h /home\n", "du -sh /home\n", "du --max-depth=1 /var\n", "\n", "# Test evaluation vs. literal shell variable string printing\n", "echo $[10*3+2] # Prints evaluated result: 32\n", "echo '$[10*3+2]' # Prints explicit string: $[10*3+2]\n", "echo \"$[10*3+2]\" # Prints evaluated result: 32\n", "\n", "# Identify target file encoding and file formats\n", "file /bin/ls\n", "file /etc/hosts\n", "\n", "# Show user account sessions currently logged in\n", "who\n", "\n", "# Display identity of active shell session user\n", "whoami\n", "\n", "# Query command documentation and manual pages\n", "whatis whoami\n", "apropos whoami\n", "man whoami\n", "```\n", "\n", "---\n", "\n", "### Archiving and File Compression\n", "\n", "```bash\n", "# Compress and decompress files via Gzip\n", "cp /etc/hosts ~/hosts.txt\n", "gzip ~/hosts.txt # Produces hosts.txt.gz\n", "zcat ~/hosts.txt.gz # Read gzip file stream without decompressing\n", "gunzip ~/hosts.txt.gz # Decompress back to raw file\n", "\n", "# Compress and decompress files via Bzip2\n", "bzip2 ~/hosts.txt # Produces hosts.txt.bz2\n", "bzip2 -d ~/hosts.txt.bz2 # Decompress back to raw file\n", "\n", "# Create a Gzip-compressed Tape Archive (.tar.gz / .tgz)\n", "tar -zcvf Newdir.tgz Newdir\n", "\n", "# Inspect and extract tar.gz archives\n", "tar -ztvf Newdir.tgz # List contents without extraction\n", "tar -zxvf Newdir.tgz # Extract contents to current directory\n", "```\n", "\n", "---\n", "\n", "### File Search Operations\n", "\n", "```bash\n", "# Search system file trees by explicit filename match\n", "find / -name fff.txt\n", "\n", "# Search within target path for wildcards and execute action upon match\n", "find /tmp -name \"*ff*\" -exec rm -f '{}' ';'\n", "\n", "# Search file paths based on size metrics (>100MB)\n", "find /var -size +100M\n", "\n", "# Search paths by owner UID\n", "find /home -user hostadm\n", "\n", "# Locate commands and binary locations in system PATH\n", "which dpkg\n", "whereis dpkg\n", "```\n", "\n", "---\n", "\n", "### Web Downloads and Command History\n", "\n", "```bash\n", "# Download web resources directly via terminal commands\n", "wget https://people.sc.fsu.edu/~jburkardt/c_src/laplace_mpi/laplace_mpi.c\n", "curl -O https://people.sc.fsu.edu/~jburkardt/c_src/laplace_mpi/laplace_mpi.c\n", "\n", "# Inspect interactive command-line history\n", "history\n", "\n", "# Reverse-i-search command history shortcut:\n", "# Press Ctrl + R, then type string matches to locate past commands\n", "\n", "# View dynamic history persistence configuration settings\n", "echo $HISTSIZE\n", "tail ~/.bash_history\n", "```\n", "\n", "---\n", "\n", "### Inspecting Hardware Specifications\n", "\n", "```bash\n", "# Query CPU architecture and core layout parameters\n", "lscpu\n", "\n", "# List PCI bus controllers and attached peripheral devices\n", "lspci\n", "\n", "# Install lshw utility for comprehensive hardware reporting\n", "sudo apt install lshw\n", "\n", "# Probe detailed subsystem hardware profiles\n", "lshw -class memory # Memory topology\n", "lshw -class storage # Storage controllers\n", "lshw -class disk # Physical disk drives\n", "lshw -class network # Network interface cards\n", "lshw -class video # Graphics controllers and displays\n", "```\n" ] } ], "metadata": { "celltoolbar": "Slideshow", "kernelspec": { "display_name": "Python (Jupyter AI)", "language": "python", "name": "base" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.11" } }, "nbformat": 4, "nbformat_minor": 4 }