Python scripting in Linux

8. Python scripting in Linux#

Python in Linux
Python virtual environment
Conda environment
Python programming environment: Jupyter notebook
Python libraries for interaction with Linux OS
Multiple account creation example
Python scripts in command line

8.1. Python in Linux#

Python comes with Linux distro, however, the version maybe too old.

which python3

/usr/bin/python3

Python modules can be installed with apt systemwide. This would require sudo privilege for package installation.

However in many cases, users should be able to install python modules themselves.

User Virtual Environment: modules can be installed with pip command in the user’s python virtual environment directory.
User Conda Environment: python installation can be done in the user’s home directory with conda.

8.2. Python virtual environment and pip (Exercise)#

On the desktop, install a package for python virtual environment support:

sudo apt install python3.11-venv

Create virtual environment venv_1 in the home directory with the command below:

python3 -m venv venv_1

You can notice a new directory appeared in the home directory:

ls -l  venv_1

Activate the virtual environment with the command below:

source venv_1/bin/activate

Notice how the command prompt is prefixed with venv_1 now. See what command python3 you are currently using:

which python3

It should point at

    /home/hostadm/venv_1/bin/python3

Upgrade module pip:

python3 -m pip install --upgrade pip

See the python packages currently installed in venv_1:

python3 -m pip list

Install package numpy:

python3 -m pip install numpy

Install package jupyter:

python3 -m pip install jupyter

Check the updated list of installed python packages:

python3 -m pip list

Uninstall jupyter:

python3 -m pip uninstall jupyter

Uninstall another package, webcolors:

python3 -m pip uninstall webcolors

Save the environment package list in a file:

python3 -m pip freeze > requirements.txt

Deactivate the virtual environment:

deactivate

See what command python3 you are currently using:

which python3

It should point at

   /usr/bin/python3

Create a new virtual environment and activate it:

python3 -m venv venv_2
source venv_2/bin/activate

Restore the python packages from file requirements.txt in venv_2:

python3 -m pip install -r requirements.txt

See the package list:

python3 -m pip list

Deactivate the virtual environment:

deactivate

Remove the directories with the virtual environments:

rm -rf venv_1
rm -rf venv_2

8.3. Conda environment (Exercise)#

To get the latest python in the user home directory with all needed modules, use Conda.
Miniconda installation instructions

mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm -rf ~/miniconda3/miniconda.sh

After installing, initialize your newly-installed Miniconda:

~/miniconda3/bin/conda init bash

Activate the Conda environment:

source .bashrc

See what Python is used now:

which python3

It should be in th Conda environment now.

To deactivate the Conda environment, run:

conda deactivate

To activate it again:

conda activate

Search Python modules:

conda search jupyter

8.4. Multiple Conda environments (Exercise)#

You may happen to need to install and test a module in the different Conda installation not to break your production system.

Creating a new conda environment, test_env:

conda create -n test_env

List the available conda environents:

conda info --envs

List installed modules in the two environments:

conda list -n base
conda list -n test_env

Switch from base to test_env:

conda deactivate
conda activate test_env

Install package numpy in test_env

conda install numpy

The environment can be saved in a file. The file can be moved to a different system and restored.

conda env export > test_env.yml

Deactivate conda and remove the environment:

conda deactivate
conda remove -n test_env --all

To restore the environment:

conda env create -f test_env.yml

Delete the environment and the backup file:

conda remove -n test_env --all
rm test_env.yml

8.5. Python programming environment: Jupyter notebook (Exercise)#

Install Jupyter notebook on your desktop as user hostadm:

conda install jupyter

Assign the password to jupyter notebook by running command:

jupyter notebook password

Create directory for python projects, and step into the directory:

mkdir Python
cd Python

8.6. Run Jupyter notebook in server mode (Exercise)#

Check if jupyter is already running:

ps -ef | grep jupyter

If it shows up running without using port 8888, terminate it:

killall -9 jupyter-notebook

Start jupyter notebook in the server mode with listening on port tcp/8888:

cd Python
jupyter-notebook --no-browser --port=8888 --ip=* &

Find out the IP address and the port number on the Ravada server for your jupyter notebook by login to the vdi portal and starting the desktop.

Navigate the browser on your laptop or desktop to the IP address and port, for example

http://172.16.26.102:60000

Login with the password you assigned to the jupyter notebook in the previous exercises.

Start a new python3 project in the jupyter notebook.

This is a Jupyter Notebook.

This computational tool allows for Python programming in a web browser without having to use a multi-window environment. You can write and run your code all in one sitting.

8.7. Interaction with operating system#

There are various python standard libraries utilized for interaction with the Linux environment, file systems, and processes.

We review only a few most commonly used:

sys — System-specific parameters and functions
os — Miscellaneous operating system interfaces
shutil - File operations
subprocess — Process management
paramiko - Remote commands via SSH and file transfer via SFTP
re - Regular expressions

8.8. sys — System-specific parameters and functions - path, argv, … (Exercise)#

Identify the OS platform within Python:

import sys
sys.platform

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

Example how it can be used:

if sys.platform.startswith('linux'):
    # Linux-specific code here...
    print('we are working in Linux environment')
else:
    print('different OS')

Get the current PYTHONPATH:

print(sys.path)

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

To see what Python we are using:

sys.executable

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

Second input parameter for the python script:

sys.argv[2]

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

Check what methods are in sys:

dir(sys)

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

8.9. os — Miscellaneous operating system interfaces - chdir, rmdir, system, … (Exercise)#

In your Jupyter notebook, copy and paste the python code from the cell below:

#Current working directory
import os
os.getcwd()

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

Try the following in the new cell:

#Change the directory
os.chdir("/etc")
os.getcwd()

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

Try listing the files and directories in /etc:

# see the list of files and directories
os.listdir()

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

Check if file hosts exists:

os.path.isfile("hosts")

Press <shift>-<Enter> keys to execute the script in the jupyter cell. Check if file no_hosts exists:

os.path.isfile('no_hosts')

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

Get the file system statistics on a file

os.stat('/etc/hosts')

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

Get the size of a file in bytes:

os.path.getsize("/etc/hosts")

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

Get the directory path separator:

os.sep

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

Generate the file names in a directory tree by walking the tree either top-down or bottom-up.

top='/etc'
for dirpath, dirnames, filenames in os.walk(top):
    path = dirpath.split(os.sep)
    print(path)

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

Derive the directory path from a full path:

full_path="/etc/libvirt/qemu/kvm1.xml"
os.path.dirname(full_path)

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

Create, rename, and remove directories:

os.chdir('/home/hostadm')
os.mkdir('DIR1')
os.rename('DIR1','DIR2')
os.rmdir('DIR2')

Execute a system command without controlling the stdout and stderr:

os.system('cd /etc; pwd')

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

The output number is the last command error code.

Check what methods are included in os

dir(os)

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

8.10. shutil - file operations - copy, chown, … (Exercise)#

import shutil

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

Copy file /etc/hosts into list2.txt:

shutil.copy('/etc/hosts','list2.txt')

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

See what files we have in the current working directory:

os.listdir()

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

Copy directory tree /etc/network into directory NEW_DIR1

shutil.copytree('/etc/network','NEW_DIR1')

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

8.11. Calling Linux commands from Python (Exercise)#

The subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes. Example of class call

# Create directory DIR1 and check the disk usage in the directory
import subprocess
s=subprocess.run("mkdir DIR1; du -sh DIR1", shell=True)

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

If you need to get the output of the system command, use method getoutput”

  user = "user1"
  stdoutdata = subprocess.getoutput(f'id {user}')
  if 'no such user' in stdoutdata:
     print(stdoutdata)

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

For full control of the standard input and the standart output to the command, we use class Popen:

import subprocess

s=subprocess.Popen("du -sh /etc", shell=True, 
                        stdout=subprocess.PIPE,
                        stderr=subprocess.PIPE,)
    
stdout_value,stderr_value = s.communicate()

print("std output=", stdout_value) 
print("std error=", stderr_value) 

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

8.12. Scripts runnable from command line (Exercise)#

Choose what version of Python you want the script to use:

either the system python, /usr/bin/python3,

or Anaconda, /home/hostadm/miniconda3/bin/python3,

or select the one that is the default in the user environment:

#!/usr/bin/env python3

Say, we want to use Anaconda python. Create, for example, file call.py with the first line containing:

#!/home/hostadm/miniconda3/bin/python3

The snippets from the jupyter cells can be placed in file call.py, for example:

#!/home/hostadm/miniconda3/bin/python3

# see the list of files and directories
import os
listf = os.listdir()
print(listf)

Make the file executable:

chmod a+x call.py

Run the script:

./call.py

8.13. Creating multiple user accounts with python (Exercise)#

In directory Python, create a new file, users.txt, with editor nano:

users.txt:

mike 2000
jerry 2001
sam 2003
mary 2004

In jupyter notebook, run the following script:

import os

def user_cmd(user, uid, group, hdir):
    s = f'useradd -m -d {hdir}/{user} -s /bin/bash -u {uid} -g {group} {user}'
    return s
    
os.chdir('/home/hostadm/Python')
HDIR = '/NFS/home'

f = open('users.txt','r')

account = {}
for line in f:
    key, value = line.rstrip().split()
    account[key] = value

f.close() 

for keys in account:
    grp_cmd = f'groupadd -g {account[keys]} {keys}'
    print(grp_cmd)
    comm = user_cmd(keys, account[keys], account[keys], HDIR)
    print(comm)

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

Function print() above is a placeholder for function os.system() that would create accounts.

8.14. Transferring files via SFTP with paramiko (Exercise)#

On your desktop install paramiko from conda-forge python module collection:

conda install -c conda-forge paramiko

Start run01 VM and find out its IP address. If you no longer have run01, pickup any recent VM, for example, master:

virsh start run01
virsh domifaddr run01

For example, IP is 192.168.122.134

In Jupyter notebook, import module paramiko:

import paramiko

Initialize SSH client constructor ssh:

ssh = paramiko.SSHClient()

Automatically trust unknown hosts (not recommended for production). Otherwise, you can SSH to the client from the command line.

ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())

Establish SSH connection to the client:

ssh.connect(hostname='192.168.122.134', username='...', password='...')

Initialize sftp constructor:

sftp = ssh.open_sftp()

Transfer file users.txt from local directory Python into the user home directory on the SSH server.

sftp.put('users.txt', 'users.txt' )

Verify that the file is there via running ls command via SSH to the server:

stdin, stdout, stderr = ssh.exec_command('ls')
print(stdout.read().decode())

You can also fetch a file via SFTP, for example:

sftp.get('vm_id_reset.sh','/tmp/vm_id_reset.sh')

Verify that file ‘vm_id_reset.sh’ is already located in /tmp directory:

os.path.isfile('/tmp/vm_id_reset.sh')

Close connections:

sftp.close()
ssh.close()

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

8.15. Regular expressions (RE) for pattern search/replacement (Exercise)#

Files can be read line by line into python lists. The lists can be searched for patterns.

Here we use re library with its methods.

Copy file list.txt from the shell scripting exercises into directory Python:

cd
cp bash_scripts/list.txt Python

Run the python script below to extract the lines containing “Reboot”, “reboot” or “support” strings (aka grep -E '[Rr]eboot|support' list.txt command):

import re

f=open('list.txt','r')
    
for line in f:
    match = re.search(r'[Rr]eboot|support', line)
    if match:
        print(line)
            
f.close()

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

The combination of two commands for opening and closing a file, f=open(...) and f.close() can be replaced with one line: with open('...','r') as f:. To eliminate extra blank lines in the printed output, one can add method rstrip('\n') to variable line. Below is a concise form of the above script:

import re

with open('list.txt','r') as f:
  for line in f:
    match = re.search(r'[Rr]eboot|support', line)
    if match:
        print(line.rstrip('\n'))

Substitute underscore for any number of spaces wherever they occur on the line (aka sed 's/ */_/g' list.txt):

import re

f=open('list.txt','r')
    
for line in f:
    newline = re.sub(r' *', r'_', line)
    print(newline)
        
f.close()

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

A concise form of the script:

import re

with open('list.txt','r') as f:    
  for line in f:
    newline = re.sub(r' *', r'_', line)
    print(newline.rstrip('\n'))

8.16. Splitting lines into fileds (Exercise)#

Display user names, home directories, and login shell (fields 1, 6 and 7) in /etc/passwd file.

Aka awk -F: '{ print $1, $6, $7 }' /etc/passwd command:

f=open('/etc/passwd','r')

for line in f:
    f1,f2,f3,f4,f5,f6,f7 = line.split(':')
    print(f1, f6, f7)
    
f.close()

Press <shift>-<Enter> keys to execute the script in the jupyter cell.

A concise form of the script:

with open('/etc/passwd','r') as f:
  for line in f:
    f1,f2,f3,f4,f5,f6,f7 = line.rstrip('\n').split(':')
    print(f1, f6, f7)

8.17. References#

Python standard library

Python scripting in Linux

Contents

8. Python scripting in Linux#

8.1. Python in Linux#

8.2. Python virtual environment and pip (Exercise)#

8.3. Conda environment (Exercise)#

8.4. Multiple Conda environments (Exercise)#

8.5. Python programming environment: Jupyter notebook (Exercise)#

8.6. Run Jupyter notebook in server mode (Exercise)#

8.7. Interaction with operating system#

8.8. sys — System-specific parameters and functions - path, argv, … (Exercise)#

8.9. os — Miscellaneous operating system interfaces - chdir, rmdir, system, … (Exercise)#

8.10. shutil - file operations - copy, chown, … (Exercise)#

8.11. Calling Linux commands from Python (Exercise)#

8.12. Scripts runnable from command line (Exercise)#

8.13. Creating multiple user accounts with python (Exercise)#

8.14. Transferring files via SFTP with paramiko (Exercise)#

8.15. Regular expressions (RE) for pattern search/replacement (Exercise)#

8.16. Splitting lines into fileds (Exercise)#

8.17. References#