Python scripting in Linux
Contents
9. Python scripting in Linux¶
Jupyter notebook installation
Python libraries for interaction with Linux
Multiple account creation example
Sftp file transfer with python
9.1. Python programming environment: Jupyter notebook (Exercise)¶
Install python3-pip on your desktop:
sudo apt install python3-pip
Install Jupyter notebook on your desktop as user hostadm:
pip3 install notebook
Add the PATH environment variable in the bottom of .bashrc file in your home directory:
.bashrc
export PATH=$PATH:/home/hostadm/.local/bin
Source the .bashrc by running command:
source .bashrc
Assign the password to jupyter notes by running command:
jupyter notebook password
Create directory for python projects, and step into the directory:
mkdir Python
cd Python
Start jupyter notes by command below:
jupyter notebook &
Access jupyter notebook through the browser on your virtual desktop by navigating to URL below:
jupyter
http://localhost:8888
Start a new python3 project in the jupyter notebook.
9.2. Interaction with operating system¶
There are various python standard libraries utilized for interaction with the Linux environment, file systems, and processes.
We review only a few most commonly used:
os — Miscellaneous operating system interfaces (chdir, rmdir, system, …)
shutil - File operations (copy, chown, …)
sys — System-specific parameters and functions (path, argv, …)
subprocess — Subprocess management (run commands with process interaction)
re - regular expressions
9.3. os — Miscellaneous operating system interfaces (Exercise)¶
In your Jupyter notebook, copy and paste the python code from the cell below:
#Current working directory
import os
os.getcwd()
Press <shift>-<Enter> keys to execute the script in the jupyter cell.
Try the following in the new cell:
#Change the directory
os.chdir("/etc")
os.getcwd()
Press <shift>-<Enter> keys to execute the script in the jupyter cell.
Try listing the files and directories in /etc:
# see the list of files and directories
os.listdir()
Press <shift>-<Enter> keys to execute the script in the jupyter cell.
Check if file hosts exists:
os.path.isfile("hosts")
Press <shift>-<Enter> keys to execute the script in the jupyter cell.
Check if file no_hosts exists:
os.path.isfile('no_hosts')
Press <shift>-<Enter> keys to execute the script in the jupyter cell.
Get the file system statistics on a file
os.stat('/etc/hosts')
Press <shift>-<Enter> keys to execute the script in the jupyter cell.
Get the directory path separator:
os.sep
Press <shift>-<Enter> keys to execute the script in the jupyter cell.
Generate the file names in a directory tree by walking the tree either top-down or bottom-up.
top='/etc'
for dirpath, dirnames, filenames in os.walk(top):
path = dirpath.split(os.sep)
print(path)
Press <shift>-<Enter> keys to execute the script in the jupyter cell.
Derive the directory path from a full path:
full_path="/etc/libvirt/qemu/kvm1.xml"
os.path.dirname(full_path)
Press <shift>-<Enter> keys to execute the script in the jupyter cell.
Create, rename, and remove directories:
os.chdir('/home/hostadm')
os.mkdir('DIR1')
os.rename('DIR1','DIR2')
os.rmdir('DIR2')
Execute a system command without controlling the stdout and stderr:
os.system('cd /etc; pwd')
Press <shift>-<Enter> keys to execute the script in the jupyter cell.
The output number is the last command error code.
9.4. shutil - copy file and directory tree (Exercise)¶
In your Jupyter notebook, copy and paste the python code from the cell below:
import shutil
shutil.copy('/etc/hosts','hosts_local_copy')
shutil.copytree('/etc','ETC_LOCAL')
Press <shift>-<Enter> keys to execute the script in the jupyter cell.
Check for the new file, hosts_local_copy, and directory ETC_LOCAL in directory Python:
ls -l
9.5. Calling Linux commands from Python (Exercise)¶
The subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes. Example of class call
# Create directory DIR1 and check the disk usage in the directory
import subprocess
s=subprocess.run("mkdir DIR1; du -sh DIR1", shell=True)
Press <shift>-<Enter> keys to execute the script in the jupyter cell.
To get the output from the command:
s=subprocess.run("mkdir DIR1; du -sh DIR1", capture_output=True, text=True, shell=True)
print(s.stdout)
Press <shift>-<Enter> keys to execute the script in the jupyter cell.
To assign the output to variables:
size, dir = str(s.stdout).strip('\n').split()
print(size)
print(dir)
Press <shift>-<Enter> keys to execute the script in the jupyter cell.
For full control of the standard input and the standart output to the command, we use class Popen:
import subprocess
s=subprocess.Popen("du -sh /etc", shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,)
stdout_value,stderr_value = s.communicate()
print("std output=", stdout_value)
print("std error=", stderr_value)
Press <shift>-<Enter> keys to execute the script in the jupyter cell.
9.6. sys — System-specific parameters and functions (Exercise)¶
Identify the OS platform within Python:
import sys
sys.platform
Press <shift>-<Enter> keys to execute the script in the jupyter cell.
Example how it can be used:
if sys.platform.startswith('linux'):
# Linux-specific code here...
print('we are working in Linux environment')
else:
print('different OS')
Get the current PYTHONPATH:
import sys
print(sys.path)
Press <shift>-<Enter> keys to execute the script in the jupyter cell.
9.7. Scripts runnable from command line (Exercise)¶
The above scripts can be placed in a file, for example call.py:
#!/usr/bin/python3
# see the list of files and directories
import os
listf = os.listdir()
print(listf)
Make the file executable:
chmod a+x call.py
Run the script:
./call.py
9.8. Module to import and script to run (Exercise)¶
Clause if __name__ == '__main__': prevents the code below the line from running is the file
is imported as a module in the other script.
Create new file, uidread.py with the following content:
#!/usr/bin/python3
import sys, subprocess
def uidread(user):
cmd = f'getent passwd {user}'
outputraw = subprocess.run(cmd, capture_output=True, shell=True, text=True)
output = str(outputraw.stdout).strip('\n')
uuser, p, uid, gid, who, hdir, shell = output.split(':')
return(uid, gid, hdir)
if __name__ == '__main__':
user = sys.argv[1] # Take user name as an argument in command line
uid, gid, hdir = uidread(user)
print( user, uid, hdir )
Make the script executable, then run it:
chmod a+x uidread.py
./uidread.py
This file works as a runnable script. It also can be imported as a module. For example:
from uidread import uidread
user = 'hostadm'
uid, gid, hdir = uidread(user)
print( uid, hdir )
9.9. Creating multiple user accounts with python (Exercise)¶
In directory Python, create a new file, users.txt, with editor nano:
users.txt:
mike 2000
jerry 2001
sam 2003
mary 2004
In jupyter notebook, run the following script:
import os
def user_cmd(user, uid, group, hdir):
s = f'useradd -m -d {hdir}/{user} -s /bin/bash -u {uid} -g {group} {user}'
return s
os.chdir('/home/hostadm/Python')
HDIR = '/NFS/home'
f = open('users.txt','r')
account = {}
for line in f:
key, value = line.rstrip().split()
account[key] = value
f.close()
for keys in account:
grp_cmd = f'groupadd -g {account[keys]} {keys}'
print(grp_cmd)
comm = user_cmd(keys, account[keys], account[keys], HDIR)
print(comm)
Press <shift>-<Enter> keys to execute the script in the jupyter cell.
Function print() above is a placeholder for function os.system() that would create accounts.
9.10. Transferring files with pysftp (Exercise)¶
On your desktop install pysftp:
pip3 install pysftp
Start run01 VM and find out its IP address:
virsh start run01
virsh domifaddr run01
For example, IP is 192.168.122.134
ssh to the VM to accept the public host key, and disconnect.
ssh 192.168.122.134
exit
In jupyter notebook, run the following steps to upload users.txt onto scripts VM.
Put the correct credentials into username and password:
import pysftp
with pysftp.Connection('192.168.122.134', username='...', password='...') as sftp:
sftp.put('/home/hostadm/Python/users.txt')
Press <shift>-<Enter> keys to execute the script in the jupyter cell.
9.11. Regular expressions re for pattern search/replacement (Exercise)¶
Files can be read line by line into python lists. The lists can be searched for patterns.
Here we use re library with its methods.
Copy file list.txt from the shell scripting exercises into directory Python:
cd
cp bash_scripts/list.txt Python
Run the python script below to extract the lines containing “Reboot”, “reboot” or “support” strings
(aka grep -E '[Rr]eboot|support' list.txt command):
import re
f=open('list.txt','r')
for line in f:
match = re.search(r'[Rr]eboot|support', line)
if match:
print(line)
f.close()
Press <shift>-<Enter> keys to execute the script in the jupyter cell.
Substitute underscore for any number of spaces wherever they occur on the line (aka sed 's/ */_/g' list.txt):
import re
f=open('list.txt','r')
for line in f:
newline = re.sub(r' *', r'_', line)
print(newline)
f.close()
Press <shift>-<Enter> keys to execute the script in the jupyter cell.
9.12. Splitting lines into fileds (Exercise)¶
Display user names, home directories, and login shell (fields 1, 6 and 7) in /etc/passwd file.
Aka awk -F: '{ print $1, $6, $7 }' /etc/passwd command:
f=open('/etc/passwd','r')
for line in f:
f1,f2,f3,f4,f5,f6,f7 = line.split(':')
print(f1, f6, f7)
f.close()
Press <shift>-<Enter> keys to execute the script in the jupyter cell.