10. Python scripting in Linux¶

Python in Linux
Conda environment
Python programming environment: Jupyter notebook
Python libraries for interaction with Linux
Multiple account creation example
Python scripts in command line

10.1. Interaction with operating system¶

There are various python standard libraries utilized for interaction with the Linux environment, file systems, and processes.

We review only a few most commonly used:

sys — System-specific parameters and functions
os — Miscellaneous operating system interfaces
shutil - File operations
subprocess — Process management
pysftp - File transfer via SFTP
re - Regular expressions

10.2. Run Jupyter notebook in server mode (Exercise)¶

Check if jupyter is already running:

ps -ef | grep jupyter

If it shows up running without using port 8888, terminate it:

killall -9 jupyter-notebook

Start jupyter notebook in the server mode with listening on port tcp/8888:

cd Python
jupyter-notebook --no-browser --port=8888 --ip=* &

Navigate the browser on your laptop to the IP address of the container and the port, for example

http://192.168.5.101:8888

Login with the password you assigned to the jupyter notebook in the previous exercises.

10.3. sys — System-specific parameters and functions - path, argv, … (Exercise)¶

Identify the OS platform within Python:

import sys
sys.platform

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

Example how it can be used:

if sys.platform.startswith('linux'):
    # Linux-specific code here...
    print('we are working in Linux environment')
else:
    print('different OS')

Get the current PYTHONPATH:

print(sys.path)

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

Tp see what Pythom we are using:

sys.executable

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

Get size of a file:

size_hosts = sys.getsizeof(/etc/hosts)
print(size_hosts)

Second input parameter for the python script:

sys.argv[2]

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

Check what methods are in sys:

dir(sys)

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

10.4. os — Miscellaneous operating system interfaces - chdir, rmdir, system, … (Exercise)¶

In your Jupyter notebook, copy and paste the python code from the cell below:

#Current working directory
import os
os.getcwd()

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

Try the following in the new cell:

#Change the directory
os.chdir("/etc")
os.getcwd()

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

Try listing the files and directories in /etc:

# see the list of files and directories
os.listdir()

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

Check if file hosts exists:

os.path.isfile("hosts")

Press <shift>-<Enter> keys to execute the script in the jupyter cell. Check if file no_hosts exists:

os.path.isfile('no_hosts')

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

Get the file system statistics on a file

os.stat('/etc/hosts')

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

Get the directory path separator:

os.sep

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

Generate the file names in a directory tree by walking the tree either top-down or bottom-up.

top='/etc'
for dirpath, dirnames, filenames in os.walk(top):
    path = dirpath.split(os.sep)
    print(path)

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

Derive the directory path from a full path:

full_path="/etc/libvirt/qemu/kvm1.xml"
os.path.dirname(full_path)

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

Create, rename, and remove directories:

os.chdir('/home/hostadm')
os.mkdir('DIR1')
os.rename('DIR1','DIR2')
os.rmdir('DIR2')

Execute a system command without controlling the stdout and stderr:

os.system('cd /etc; pwd')

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

The output number is the last command error code.

Check what methods are included in os

dir(os)

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

10.5. shutil - file operations - copy, chown, … (Exercise)¶

import shutil

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

Copy file /etc/hosts into list2.txt:

shutil.copy('/etc/hosts','list2.txt')

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

See what files we have in the current working directory:

os.listdir()

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

Copy directory tree /etc/network into directory NEW_DIR1

shutil.copytree('/etc/network','NEW_DIR1')

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

10.6. Calling Linux commands from Python (Exercise)¶

The subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes. Example of class call

# Create directory DIR1 and check the disk usage in the directory
import subprocess
s=subprocess.run("mkdir DIR1; du -sh DIR1", shell=True)

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

For full control of the standard input and the standart output to the command, we use class Popen:

import subprocess

s=subprocess.Popen("du -sh /etc", shell=True, 
                        stdout=subprocess.PIPE,
                        stderr=subprocess.PIPE,)
    
stdout_value,stderr_value = s.communicate()

print("std output=", stdout_value) 
print("std error=", stderr_value) 

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

10.7. Scripts runnable from command line (Exercise)¶

Choose what version of Python you want the script to use: either the system python, /usr/bin/python3, or Anaconda, /home/hostadm/miniconda3/bin/python3

Say, we want to use Anaconda python. Create, for example, file call.py with the first line containing:

#!/home/hostadm/miniconda3/bin/python3

The snippets from the jupyter cells can be placed in file call.py, for example:

#!/home/hostadm/miniconda3/bin/python3

# see the list of files and directories
import os
listf = os.listdir()
print(listf)

Make the file executable:

chmod a+x call.py

Run the script:

./call.py

10.8. Creating multiple user accounts with python (Exercise)¶

In directory Python, create a new file, users.txt, with editor nano:

users.txt:

mike 2000
jerry 2001
sam 2003
mary 2004

In jupyter notebook, run the following script:

import os

def user_cmd(user, uid, group, hdir):
    s = f'useradd -m -d {hdir}/{user} -s /bin/bash -u {uid} -g {group} {user}'
    return s
    
os.chdir('/home/hostadm/Python')
HDIR = '/NFS/home'

f = open('users.txt','r')

account = {}
for line in f:
    key, value = line.rstrip().split()
    account[key] = value

f.close() 

for keys in account:
    grp_cmd = f'groupadd -g {account[keys]} {keys}'
    print(grp_cmd)
    comm = user_cmd(keys, account[keys], account[keys], HDIR)
    print(comm)

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

Function print() above is a placeholder for function os.system() that would create accounts.

10.9. Transferring files with pysftp (Exercise)¶

On your container, install pysftp:

conda install pysftp

In jupyter notebook, run the following steps to upload users.txt onto your node, for example: Put the correct credentials into username and password:

import pysftp
with pysftp.Connection(node03, username='...', password='...') as sftp:
        sftp.put('/home/hostadm/Python/users.txt')  	

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

10.10. Regular expressions (RE) for pattern search/replacement (Exercise)¶

Files can be read line by line into python lists. The lists can be searched for patterns.

Here we use re library with its methods.

Copy file list.txt from the shell scripting exercises into directory Python:

cd
cp bash_scripts/list.txt Python

Run the python script below to extract the lines containing “Reboot”, “reboot” or “support” strings (aka grep -E '[Rr]eboot|support' list.txt command):

import re

f=open('list.txt','r')
    
for line in f:
    match = re.search(r'[Rr]eboot|support', line)
    if match:
        print(line)
            
f.close()

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

Substitute underscore for any number of spaces wherever they occur on the line (aka sed 's/ */_/g' list.txt):

import re

f=open('list.txt','r')
    
for line in f:
    newline = re.sub(r' *', r'_', line)
    print(newline)
        
f.close()

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

10.11. Splitting lines into fileds (Exercise)¶

Display user names, home directories, and login shell (fields 1, 6 and 7) in /etc/passwd file.

Aka awk -F: '{ print $1, $6, $7 }' /etc/passwd command:

f=open('/etc/passwd','r')

for line in f:
    f1,f2,f3,f4,f5,f6,f7 = line.split(':')
    print(f1, f6, f7)
    
f.close()

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

10.12. References¶

Python standard library

Winlab Summer 2024 Internship project

Python scripting in Linux

Contents

10. Python scripting in Linux¶

10.1. Interaction with operating system¶

10.2. Run Jupyter notebook in server mode (Exercise)¶

10.3. sys — System-specific parameters and functions - path, argv, … (Exercise)¶

10.4. os — Miscellaneous operating system interfaces - chdir, rmdir, system, … (Exercise)¶

10.5. shutil - file operations - copy, chown, … (Exercise)¶

10.6. Calling Linux commands from Python (Exercise)¶

10.7. Scripts runnable from command line (Exercise)¶

10.8. Creating multiple user accounts with python (Exercise)¶

10.9. Transferring files with pysftp (Exercise)¶

10.10. Regular expressions (RE) for pattern search/replacement (Exercise)¶

10.11. Splitting lines into fileds (Exercise)¶

10.12. References¶