{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Python scripting in Linux\n",
"\n",
"\n",
"- Python libraries for interaction with Linux OS\n",
"\n",
"- Multiple account creation example\n",
"\n",
"- Python scripts in command line\n",
"\n",
"\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Interaction with operating system \n",
"\n",
"There are various python standard libraries utilized for interaction with the Linux environment, file systems, and processes. \n",
"\n",
"We review only a few most commonly used:\n",
"- ```sys``` — System-specific parameters and functions\n",
"- ```os``` — Miscellaneous operating system interfaces \n",
"- ```shutil``` - File operations\n",
"- ```subprocess``` — Process management\n",
"- ```paramiko``` - Remote commands via SSH and file transfer via SFTP\n",
"- ```re``` - Regular expressions\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Run Jupyter notebook in server mode (Exercise)\n",
"\n",
"Check if jupyter is already running:\n",
"```bash\n",
"ps -ef | grep jupyter\n",
"```\n",
"\n",
"If it shows up running without using port 8888, terminate it:\n",
"```bash\n",
"killall -9 jupyter-notebook\n",
"```\n",
"\n",
"Start jupyter notebook in the server mode with listening on port tcp/8888:\n",
"```bash\n",
"cd Python\n",
"jupyter-notebook --no-browser --port=8888 --ip=* &\n",
"```\n",
"\n",
"Find out the IP address and the port number on the Ravada server for your jupyter notebook by login to \n",
" the vdi portal and starting the desktop.\n",
" \n",
"Navigate the browser on your laptop or desktop to the IP address and port, for example \n",
"```c\n",
"http://172.16.26.112:60000\n",
"```\n",
"Login with the password you assigned to the jupyter notebook in the previous exercises.\n",
"\n",
"Start a new python3 project in the jupyter notebook.\n",
"\n",
"This is a Jupyter Notebook.\n",
"\n",
"This computational tool allows for Python programming in a web browser without having to use a multi-window environment. You can write and run your code all in one sitting. \n",
"\n",
" ***"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## sys — System-specific parameters and functions - path, argv, … (Exercise)\n",
"\n",
"Identify the OS platform within Python:\n",
"```c\n",
"import sys\n",
"sys.platform\n",
"```\n",
"Press `-` keys to execute the script in the jupyter cell.\n",
" \n",
"Example how it can be used:\n",
"```c\n",
"\n",
"if sys.platform.startswith('linux'):\n",
" # Linux-specific code here...\n",
" print('we are working in Linux environment')\n",
"else:\n",
" print('different OS')\n",
"```\n",
"\n",
"Get the current PYTHONPATH:\n",
"```c\n",
"print(sys.path)\n",
"```\n",
"Press `-` keys to execute the script in the jupyter cell.\n",
"\n",
"To see what Python we are using:\n",
"```c\n",
"sys.executable\n",
"```\n",
"Press `-` keys to execute the script in the jupyter cell.\n",
"\n",
"\n",
"Second input parameter for the python script:\n",
"```c\n",
"sys.argv[2]\n",
"```\n",
"Press `-` keys to execute the script in the jupyter cell.\n",
"\n",
"Check what methods are in `sys`:\n",
"```c\n",
"dir(sys)\n",
"```\n",
"Press `-` keys to execute the script in the jupyter cell.\n",
"\n",
"***"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## os — Miscellaneous operating system interfaces - chdir, rmdir, system, … (Exercise)\n",
" \n",
"In your Jupyter notebook, copy and paste the python code from the cell below:\n",
"\n",
"``` c\n",
"#Current working directory\n",
"import os\n",
"os.getcwd()\n",
"```\n",
"\n",
"Press `-` keys to execute the script in the jupyter cell.\n",
"\n",
"Try the following in the new cell:\n",
"``` c\n",
"#Change the directory\n",
"os.chdir(\"/etc\")\n",
"os.getcwd()\n",
"```\n",
"\n",
"Press `-` keys to execute the script in the jupyter cell.\n",
"\n",
"\n",
"Try listing the files and directories in /etc:\n",
"```c\n",
"# see the list of files and directories\n",
"os.listdir()\n",
"```\n",
"\n",
"Press `-` keys to execute the script in the jupyter cell.\n",
"\n",
"Check if file `hosts` exists:\n",
"```c\n",
"os.path.isfile(\"hosts\")\n",
"```\n",
"Press `-` keys to execute the script in the jupyter cell.\n",
"Check if file `no_hosts` exists:\n",
"```c\n",
"os.path.isfile('no_hosts')\n",
"```\n",
"Press `-` keys to execute the script in the jupyter cell.\n",
"\n",
"Get the file system statistics on a file\n",
"```c\n",
"os.stat('/etc/hosts')\n",
"```\n",
"Press `-` keys to execute the script in the jupyter cell.\n",
"\n",
"Get the size of a file in bytes:\n",
"```c\n",
"os.path.getsize(\"/etc/hosts\")\n",
"```\n",
"Press `-` keys to execute the script in the jupyter cell.\n",
"\n",
"Get the directory path separator:\n",
"```c\n",
"os.sep\n",
"```\n",
"Press `-` keys to execute the script in the jupyter cell.\n",
"\n",
"Generate the file names in a directory tree by walking the tree either top-down or bottom-up.\n",
"```c\n",
"top='/etc'\n",
"for dirpath, dirnames, filenames in os.walk(top):\n",
" path = dirpath.split(os.sep)\n",
" print(path)\n",
"```\n",
"Press `-` keys to execute the script in the jupyter cell.\n",
"\n",
"Derive the directory path from a full path:\n",
"```c\n",
"full_path=\"/etc/libvirt/qemu/kvm1.xml\"\n",
"os.path.dirname(full_path)\n",
"```\n",
"Press `-` keys to execute the script in the jupyter cell.\n",
"\n",
"Create, rename, and remove directories:\n",
"```c\n",
"os.chdir('/home/hostadm')\n",
"os.mkdir('DIR1')\n",
"os.rename('DIR1','DIR2')\n",
"os.rmdir('DIR2')\n",
"```\n",
"\n",
"Execute a system command without controlling the stdout and stderr:\n",
"```c\n",
"os.system('cd /etc; pwd')\n",
"```\n",
"Press `-` keys to execute the script in the jupyter cell.\n",
"\n",
"The output number is the last command error code.\n",
" \n",
"Check what methods are included in `os`\n",
"```c\n",
"dir(os)\n",
"```\n",
"Press `-` keys to execute the script in the jupyter cell.\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## shutil - file operations - copy, chown, … (Exercise) \n",
"\n",
"```c\n",
"import shutil\n",
"```\n",
"Press `-` keys to execute the script in the jupyter cell.\n",
"\n",
"Copy file /etc/hosts into list2.txt:\n",
"```c\n",
"shutil.copy('/etc/hosts','list2.txt')\n",
"```\n",
"Press `-` keys to execute the script in the jupyter cell.\n",
"\n",
"\n",
"See what files we have in the current working directory:\n",
"```c\n",
"os.listdir()\n",
"```\n",
"Press `-` keys to execute the script in the jupyter cell.\n",
"\n",
"Copy directory tree /etc/network into directory NEW_DIR1\n",
"```c\n",
"shutil.copytree('/etc/network','NEW_DIR1')\n",
"```\n",
"Press `-` keys to execute the script in the jupyter cell.\n",
"\n",
"***"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Calling Linux commands from Python (Exercise)\n",
"\n",
"The `subprocess` module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes. Example of class call\n",
"``` c\n",
"# Create directory DIR1 and check the disk usage in the directory\n",
"import subprocess\n",
"s=subprocess.run(\"mkdir DIR1; du -sh DIR1\", shell=True)\n",
"\n",
"```\n",
"Press `-` keys to execute the script in the jupyter cell.\n",
"\n",
"If you need to get the output of the system command, use method ```getoutput```\"\n",
"```c\n",
" user = \"user1\"\n",
" stdoutdata = subprocess.getoutput(f'id {user}')\n",
" if 'no such user' in stdoutdata:\n",
" print(stdoutdata)\n",
"```\n",
"Press `-` keys to execute the script in the jupyter cell.\n",
"\n",
"For full control of the standard input and the standart output to the command, we use class Popen:\n",
"``` c\n",
"import subprocess\n",
"\n",
"s=subprocess.Popen(\"du -sh /etc\", shell=True, \n",
" stdout=subprocess.PIPE,\n",
" stderr=subprocess.PIPE,)\n",
" \n",
"stdout_value,stderr_value = s.communicate()\n",
"\n",
"print(\"std output=\", stdout_value) \n",
"print(\"std error=\", stderr_value) \n",
"```\n",
"Press `-` keys to execute the script in the jupyter cell.\n",
"\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Scripts runnable from command line (Exercise)\n",
"\n",
"Choose what version of Python you want the script to use:\n",
"\n",
"either the system python, ```/usr/bin/python3```, \n",
"\n",
"or Anaconda, ```/home/hostadm/miniconda3/bin/python3```,\n",
"\n",
"or select the one that is the default in the user environment:\n",
"\n",
"```#!/usr/bin/env python3```\n",
"\n",
"Say, we want to use Anaconda python. Create, for example, file `call.py` with the first line containing:\n",
"```c\n",
"#!/home/hostadm/miniconda3/bin/python3\n",
"```\n",
"\n",
"The snippets from the jupyter cells can be placed in file `call.py`, for example:\n",
"\n",
"```c\n",
"#!/home/hostadm/miniconda3/bin/python3\n",
"\n",
"# see the list of files and directories\n",
"import os\n",
"listf = os.listdir()\n",
"print(listf)\n",
"```\n",
"\n",
"Make the file executable:\n",
"\n",
"```bash\n",
"chmod a+x call.py\n",
"```\n",
"\n",
"Run the script:\n",
"\n",
"```bash\n",
"./call.py\n",
"```\n",
"\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Creating multiple user accounts with python (Exercise)\n",
"\n",
"In directory Python, create a new file, `users.txt`, with editor `nano`:\n",
"\n",
"```{admonition} users.txt:\n",
" mike 2000\n",
" jerry 2001\n",
" sam 2003\n",
" mary 2004\n",
"```\n",
"\n",
"In jupyter notebook, run the following script:\n",
"\n",
"```c\n",
"import os\n",
"\n",
"def user_cmd(user, uid, group, hdir):\n",
" s = f'useradd -m -d {hdir}/{user} -s /bin/bash -u {uid} -g {group} {user}'\n",
" return s\n",
" \n",
"os.chdir('/home/hostadm/Python')\n",
"HDIR = '/NFS/home'\n",
"\n",
"f = open('users.txt','r')\n",
"\n",
"account = {}\n",
"for line in f:\n",
" key, value = line.rstrip().split()\n",
" account[key] = value\n",
"\n",
"f.close() \n",
"\n",
"for keys in account:\n",
" grp_cmd = f'groupadd -g {account[keys]} {keys}'\n",
" print(grp_cmd)\n",
" comm = user_cmd(keys, account[keys], account[keys], HDIR)\n",
" print(comm)\n",
"```\n",
"\n",
"Press `-` keys to execute the script in the jupyter cell.\n",
"\n",
"Function `print()` above is a placeholder for function `os.system()` that would create accounts.\n",
"\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Transferring files via SFTP with paramiko (Exercise)\n",
"\n",
"On your desktop install paramiko from conda-forge python module collection:\n",
"```bash\n",
"conda install -c conda-forge paramiko\n",
"```\n",
"Start `run01` VM and find out its IP address. If you no longer have `run01`, pickup any recent VM, for example, `master`:\n",
"```bash\n",
"virsh start run01\n",
"virsh domifaddr run01\n",
"```\n",
"For example, IP is 192.168.122.134\n",
"\n",
"In Jupyter notebook, import module ```paramiko```:\n",
"```python\n",
"import paramiko\n",
"```\n",
"Initialize SSH client constructor `ssh`:\n",
"```python\n",
"ssh = paramiko.SSHClient()\n",
"```\n",
"Automatically trust unknown hosts (not recommended for production). Otherwise, you can SSH to the client from the command line.\n",
"```python\n",
"ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())\n",
"```\n",
"Establish SSH connection to the client:\n",
"```python\n",
"ssh.connect(hostname='192.168.122.134', username='...', password='...')\n",
"```\n",
"Initialize sftp constructor:\n",
"```python\n",
"sftp = ssh.open_sftp()\n",
"```\n",
"Transfer file ```users.txt``` from local directory Python into the user home directory on the SSH server.\n",
"```bash\n",
"sftp.put('users.txt', 'users.txt' )\n",
"```\n",
"Verify that the file is there via running ```ls``` command via SSH to the server:\n",
"```python\n",
"stdin, stdout, stderr = ssh.exec_command('ls')\n",
"print(stdout.read().decode())\n",
"```\n",
"\n",
"You can also fetch a file via SFTP, for example:\n",
"```python\n",
"sftp.get('vm_id_reset.sh','/tmp/vm_id_reset.sh')\n",
"```\n",
"Verify that file 'vm_id_reset.sh' is already located in /tmp directory:\n",
"```python\n",
"os.path.isfile('/tmp/vm_id_reset.sh')\n",
"```\n",
"\n",
"Close connections:\n",
"```python\n",
"sftp.close()\n",
"ssh.close()\n",
"```\n",
"\n",
"Press `-` keys to execute the script in the jupyter cell.\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"***\n",
"## Regular expressions (RE) for pattern search/replacement (Exercise)\n",
"\n",
"Files can be read line by line into python lists. The lists can be searched for patterns.\n",
"\n",
"Here we use ```re``` library with its methods.\n",
"\n",
"Copy file ```list.txt``` from the shell scripting exercises into directory Python:\n",
"```bash\n",
"cd\n",
"cp bash_scripts/list.txt Python\n",
"```\n",
"Run the python script below to extract the lines containing “Reboot”, “reboot” or “support” strings\n",
" (aka `grep -E '[Rr]eboot|support' list.txt` command):\n",
"```c\n",
"import re\n",
"\n",
"f=open('list.txt','r')\n",
" \n",
"for line in f:\n",
" match = re.search(r'[Rr]eboot|support', line)\n",
" if match:\n",
" print(line)\n",
" \n",
"f.close()\n",
"```\n",
"\n",
"Press `-` keys to execute the script in the jupyter cell.\n",
"\n",
"The combination of two commands for opening and closing a file, ```f=open(...)``` and ```f.close()``` can be replaced with one line: \n",
" ```with open('...','r') as f:```. To eliminate extra blank lines in the printed output, one can add method ```rstrip('\\n')``` to variable ```line```. Below is a concise form of the above script:\n",
"```c\n",
"import re\n",
"\n",
"with open('list.txt','r') as f:\n",
" for line in f:\n",
" match = re.search(r'[Rr]eboot|support', line)\n",
" if match:\n",
" print(line.rstrip('\\n'))\n",
"```\n",
"\n",
"\n",
"Substitute underscore for any number of spaces wherever they occur on the line (aka `sed 's/ */_/g' list.txt`):\n",
"\n",
"```c\n",
"import re\n",
"\n",
"f=open('list.txt','r')\n",
" \n",
"for line in f:\n",
" newline = re.sub(r' *', r'_', line)\n",
" print(newline)\n",
" \n",
"f.close()\n",
"```\n",
"\n",
"Press `-` keys to execute the script in the jupyter cell.\n",
"\n",
"A concise form of the script:\n",
"```c\n",
"import re\n",
"\n",
"with open('list.txt','r') as f: \n",
" for line in f:\n",
" newline = re.sub(r' *', r'_', line)\n",
" print(newline.rstrip('\\n'))\n",
"```\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"***\n",
"## Splitting lines into fileds (Exercise)\n",
"\n",
"Display user names, home directories, and login shell (fields 1, 6 and 7) in `/etc/passwd` file.\n",
"\n",
"\n",
"Aka `awk -F: '{ print $1, $6, $7 }' /etc/passwd` command:\n",
"\n",
"```c\n",
"f=open('/etc/passwd','r')\n",
"\n",
"for line in f:\n",
" f1,f2,f3,f4,f5,f6,f7 = line.split(':')\n",
" print(f1, f6, f7)\n",
" \n",
"f.close()\n",
"```\n",
"\n",
"Press `-` keys to execute the script in the jupyter cell.\n",
"\n",
"A concise form of the script:\n",
"```c\n",
"with open('/etc/passwd','r') as f:\n",
" for line in f:\n",
" f1,f2,f3,f4,f5,f6,f7 = line.rstrip('\\n').split(':')\n",
" print(f1, f6, f7)\n",
"```\n",
"***"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## References\n",
"\n",
"[Python standard library](https://docs.python.org/3/library/)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.13.11"
}
},
"nbformat": 4,
"nbformat_minor": 4
}