More Python modules

10. More Python modules#

  • gzip - compresses/decompresses binary data, reads/writes data into *.gzip file.

  • zipfile - creates, adds, reads, writes to zip archives.

  • getpass - reads password from keyboard.

  • glob - finds files with matched regexp patterns.

  • pathlib - object-oriented approach to handling filesystem paths.

  • json - encodes/decodes python object into json, reads/writes jason files.

  • argparse - passes parameters to a python script.

  • Exception handling.

10.1. Module gzip (Exercise)#

The module is utilized to

  • compress/decompress binary data

  • write data into compressed *.gz file

  • read data from compressed *.gz file

## See all the methods in the module

import gzip

data = '''
 Create some data. \nIn this particular case, \nthe data is just two sentences stored in a string\n 
 Line 4 \n
 Line 5 \n
 Line 6 \n 
 Line 7
 '''

print(data)

 Create some data. 
In this particular case, 
the data is just two sentences stored in a string
 
 Line 4 

 Line 5 

 Line 6 
 
 Line 7
 

## Store data into gzip file, data.txt.gz.  The procedure is very similar to writing data into a regular file. 
## Notice the write option 'wt' is for writing text:

with gzip.open("data.txt.gz","wt") as f:
    f.write(data)

## Check if file data.txt.gz shows up in the directory:

import os
os.listdir()

['final_exam_fall25.ipynb',
 'python_extra_modules.ipynb',
 'final_exam_fall24.ipynb',
 'Lesson_1.ipynb',
 'final_exam24.ipynb',
 'notebooks.ipynb',
 'midterm2024_qa.ipynb',
 'shell_scripting.ipynb',
 'new_test.zip',
 'final_exam25qa.ipynb',
 'placeholder_midterm2023.ipynb',
 'midterm_exercises.ipynb',
 'NFS.ipynb',
 'index.md',
 '_build',
 'img',
 'logo.png',
 'homework4.ipynb',
 'homework5.ipynb',
 'Final_exam_preparation.ipynb',
 'references.bib',
 'final_exam_fall25_qa.ipynb',
 'midterm2023_qa.ipynb',
 'systemd.ipynb',
 'final_exam22_Q-and-A.ipynb',
 '_config.yml',
 'EXTRACT_DIR',
 'smb.ipynb',
 'midterm_summer2025.ipynb',
 '.ipynb_checkpoints',
 'passwd.txt',
 'services.json',
 'final_exam25.ipynb',
 'compilation.ipynb',
 'python_scripting2.ipynb',
 'midterm2023.ipynb',
 'homework1.ipynb',
 'homework3.ipynb',
 'midterm_fall_2025.ipynb',
 'requirements.txt',
 'midterm_fall_2025_qa.ipynb',
 'conf.py',
 'intro.md',
 'final_exam',
 'additional_midterm_prep_exercises.ipynb',
 'final_exam_fall24_qa.ipynb',
 'Untitled.ipynb',
 'markdown.md',
 'final_exam23.ipynb',
 'data.txt.gz',
 'midterm_qa_summer2025.ipynb',
 'final_exam23qa.ipynb',
 'security.ipynb',
 'final_exam24qa.ipynb',
 'security_short_verssion.ipynb',
 'linux_packaging.ipynb',
 'midterm_fall_2024.ipynb',
 'midterm_summer_2022.ipynb',
 'markdown-notebooks.md',
 'test.zip',
 'midterm2024.ipynb',
 '_toc.yml',
 'homework6.ipynb',
 'python_scripting.ipynb',
 'homework7.ipynb',
 'virtualization.ipynb',
 'networking.ipynb']

%ls data.txt.gz

data.txt.gz

## Let's check if we can read data back from the file. Notice the read option 'rt' is for reading text:

with gzip.open("data.txt.gz","rt") as f:
    data_read = f.read()
    
print(data_read)

 Create some data. 
In this particular case, 
the data is just two sentences stored in a string
 
 Line 4 

 Line 5 

 Line 6 
 
 Line 7
 

## We cal also convert the data into a binary form, then write it into gzip file with 'wb' option

#data_byte = bytes(data, 'utf-8')
data_byte = data.encode('utf-8')

type(data_byte)

bytes

print(data_byte)

b'\n Create some data. \nIn this particular case, \nthe data is just two sentences stored in a string\n \n Line 4 \n\n Line 5 \n\n Line 6 \n \n Line 7\n '

## Store the binary data into gzip file

with gzip.open("data.txt.gz","wb") as f:
    f.write(data_byte)

## This also can be done with GzipFile method:
with gzip.GzipFile("data.txt.gz","wb") as f:
    f.write(data_byte)

## We can also read the gzip file into binary data type by using "rb" option:

with gzip.open("data.txt.gz","rb") as f:
    out_byte = f.read()

print(out_byte)

b'\n Create some data. \nIn this particular case, \nthe data is just two sentences stored in a string\n \n Line 4 \n\n Line 5 \n\n Line 6 \n \n Line 7\n '

## Compressing/decompressing data. It can be applied to binary data only.
## Check the sizes of text data, byte data, and compressed byte data:

import sys

size_txt = sys.getsizeof(data)

out_byte = data.encode('utf-8')
size_bin = sys.getsizeof(out_byte)

out_compressed = gzip.compress(out_byte)
size_compressed = sys.getsizeof(out_compressed)

print(size_txt, '>', size_bin, '>', size_compressed)

180 > 172 > 151

10.2. Module zipfile (Exercise)#

Works with zip archives:

  • Reads list of files and directories from zip archive

  • Extracts files from an archive

  • Reads specific file from an archive

  • Creates a new archive

  • Adds/appends file to the archive

10.2.1. Download a zip file:#

import os
os.system('wget --no-check-certificate https://linuxcourse.rutgers.edu/2024/html/files/test.zip')

--2025-12-20 19:46:10--  https://linuxcourse.rutgers.edu/2024/html/files/test.zip
Resolving linuxcourse.rutgers.edu (linuxcourse.rutgers.edu)... 128.6.238.9
Connecting to linuxcourse.rutgers.edu (linuxcourse.rutgers.edu)|128.6.238.9|:443... connected.
WARNING: The certificate of ‘linuxcourse.rutgers.edu’ is not trusted.
WARNING: The certificate of ‘linuxcourse.rutgers.edu’ doesn't have a known issuer.
HTTP request sent, awaiting response... 200 OK
Length: 844 [application/zip]
Saving to: ‘test.zip.1’

     0K                                                       100% 23.7M=0s

2025-12-20 19:46:10 (23.7 MB/s) - ‘test.zip.1’ saved [844/844]

0

import zipfile

with zipfile.ZipFile('test.zip') as file:   
    # printing all the information of archive file contents using 'printdir' method
    print(file.printdir())

File Name                                             Modified             Size
NEW_DIR1/                                      2021-05-25 14:34:44            0
py_script.py                                   2024-01-10 17:14:22          198
list.txt                                       2024-01-10 17:14:00          105
ip.txt                                         2024-01-11 11:55:14           12
None

with zipfile.ZipFile('test.zip') as file:   
    # ZipFile.namelist() returns a list containing all the members with names of an archive file
    print(file.namelist())

['NEW_DIR1/', 'py_script.py', 'list.txt', 'ip.txt']

with zipfile.ZipFile('test.zip') as file:
    # Open and read the content of a file from the list:
    with file.open(name = file.namelist()[1], mode = 'r') as text_file:
        print(text_file.read().decode('utf-8'))  

#!/home/akotelnikov/miniconda3/bin/python3

import sys, os
script_name = sys.argv[0]
par1 = sys.argv[1]
par2 = sys.argv[2]

print('name=',script_name, 'par1=', par1, 'par2=',par2)


os.system(par1)

 with zipfile.ZipFile('test.zip') as file:

        # ZipFile.infolist() returns a list containing all the members of an archive file
        print(file.infolist())

[<ZipInfo filename='NEW_DIR1/' filemode='drwxr-xr-x' external_attr=0x10>, <ZipInfo filename='py_script.py' compress_type=deflate filemode='-rwxr-xr-x' file_size=198 compress_size=134>, <ZipInfo filename='list.txt' compress_type=deflate filemode='-rw-rw-r--' file_size=105 compress_size=94>, <ZipInfo filename='ip.txt' filemode='-rw-rw-r--' file_size=12>]

## Extract the content of zip file into a new directory, EXTRACT_DIR


with zipfile.ZipFile('test.zip') as f:
        f.extractall('EXTRACT_DIR')

import os
os.listdir('EXTRACT_DIR')

['list.txt', 'ip.txt', 'NEW_DIR1', 'py_script.py']

## Create an empty zipfile:

archive_name = 'new_test.zip'
f = zipfile.ZipFile(archive_name, 'w')
f.close()

## Add a new file, /etc/hosts, to the existing archive. Ovewrite existing files there.
with zipfile.ZipFile('new_test.zip', 'w') as f:
        f.write('/etc/hosts')

## Append a new file, /etc/passwd. This doesn't overwrite the existing files.
with zipfile.ZipFile('new_test.zip', 'a') as f:
        f.write('/etc/passwd')

## Check the content of archive new_test.zip
with zipfile.ZipFile('new_test.zip','r') as f:
    print(f.printdir())

File Name                                             Modified             Size
etc/hosts                                      2025-12-03 15:42:58          179
etc/passwd                                     2024-07-27 16:12:20         1188
None

10.3. Module getpass (Exercise)#

  • Promps for password and reads it from stdin

  • Figures out the user name from the environment

import getpass

## Prompt for password and read user password:

p = getpass.getpass(prompt='Enter the password: ')

---------------------------------------------------------------------------
StdinNotImplementedError                  Traceback (most recent call last)
Cell In[32], line 1
----> 1 p = getpass.getpass(prompt='Enter the password: ')

File ~/miniconda3/lib/python3.12/site-packages/ipykernel/kernelbase.py:1256, in Kernel.getpass(self, prompt, stream)
   1254 if not self._allow_stdin:
   1255     msg = "getpass was called, but this frontend does not support input requests."
-> 1256     raise StdinNotImplementedError(msg)
   1257 if stream is not None:
   1258     import warnings

StdinNotImplementedError: getpass was called, but this frontend does not support input requests.

if p == 'this':
    print('correct password')
else:
    print('password incorrect')

password incorrect

## identify the user and prompt in the loop until success. Maximum 10 attempts

user = getpass.getuser()

count = 0
max_count = 10

#while True:  
while count <= max_count:
    
    pwd = getpass.getpass("User Name : %s" % user)
    
    count += 1

    if pwd == 'password1':
        print("login successful")
        break
    else:
        print("The password you entered is incorrect.")

The password you entered is incorrect.

The password you entered is incorrect.

The password you entered is incorrect.

login successful

10.4. Module glob (Exercise)#

  • Finds files matching a specified pattern and returns them in a list [ ]

  • * matches zero or more characters

  • ? matches one character

  • [0-9] matches any number

  • [a-z] matches any low case character

  • [A-Z] matches any upper case character

import glob

glob.glob('*.zip')

['new_test.zip', 'test.zip']

glob.glob('*[0-9].zip')

[]

glob.glob('*[a-z].zip')

['new_test.zip', 'test.zip']

glob.glob('[A-Z]*[0-9].zip')

[]

glob.glob('[n,I]e*')

['new_test.zip', 'networking.ipynb']

## Search for files specifically in /etc
glob.glob('/etc/*.conf')

['/etc/nftables.conf',
 '/etc/deluser.conf',
 '/etc/ld.so.conf',
 '/etc/reportbug.conf',
 '/etc/sysctl.conf',
 '/etc/xattr.conf',
 '/etc/libaudit.conf',
 '/etc/nsswitch.conf',
 '/etc/sudo_logsrvd.conf',
 '/etc/sensors3.conf',
 '/etc/mke2fs.conf',
 '/etc/ca-certificates.conf',
 '/etc/e2scrub.conf',
 '/etc/logrotate.conf',
 '/etc/gai.conf',
 '/etc/ucf.conf',
 '/etc/resolv.conf',
 '/etc/host.conf',
 '/etc/sudo.conf',
 '/etc/debconf.conf',
 '/etc/pam.conf',
 '/etc/adduser.conf']

## Search for files in directory and subdirectories /etc
glob.glob('/etc/*/*.conf')

['/etc/ld.so.conf.d/x86_64-linux-gnu.conf',
 '/etc/ld.so.conf.d/libc.conf',
 '/etc/fonts/fonts.conf',
 '/etc/apt/listchanges.conf',
 '/etc/gtk-3.0/im-multipress.conf',
 '/etc/security/limits.conf',
 '/etc/security/access.conf',
 '/etc/security/pam_env.conf',
 '/etc/security/namespace.conf',
 '/etc/security/time.conf',
 '/etc/security/faillock.conf',
 '/etc/security/group.conf',
 '/etc/security/sepermit.conf',
 '/etc/selinux/semanage.conf',
 '/etc/environment.d/90qt-a11y.conf',
 '/etc/udev/udev.conf',
 '/etc/dhcp/dhclient.conf',
 '/etc/xdg/user-dirs.conf',
 '/etc/modules-load.d/modules.conf',
 '/etc/rsyslog.d/postfix.conf',
 '/etc/sysctl.d/99-sysctl.conf',
 '/etc/gtk-2.0/im-multipress.conf',
 '/etc/systemd/system.conf',
 '/etc/systemd/networkd.conf',
 '/etc/systemd/pstore.conf',
 '/etc/systemd/timesyncd.conf',
 '/etc/systemd/user.conf',
 '/etc/systemd/logind.conf',
 '/etc/systemd/sleep.conf',
 '/etc/systemd/journald.conf']

## Recursive search in a directory tree
glob.glob('/etc/**/*.conf', recursive=True)

['/etc/nftables.conf',
 '/etc/deluser.conf',
 '/etc/ld.so.conf',
 '/etc/reportbug.conf',
 '/etc/sysctl.conf',
 '/etc/xattr.conf',
 '/etc/libaudit.conf',
 '/etc/nsswitch.conf',
 '/etc/sudo_logsrvd.conf',
 '/etc/sensors3.conf',
 '/etc/mke2fs.conf',
 '/etc/ca-certificates.conf',
 '/etc/e2scrub.conf',
 '/etc/logrotate.conf',
 '/etc/gai.conf',
 '/etc/ucf.conf',
 '/etc/resolv.conf',
 '/etc/host.conf',
 '/etc/sudo.conf',
 '/etc/debconf.conf',
 '/etc/pam.conf',
 '/etc/adduser.conf',
 '/etc/ld.so.conf.d/x86_64-linux-gnu.conf',
 '/etc/ld.so.conf.d/libc.conf',
 '/etc/fonts/fonts.conf',
 '/etc/fonts/conf.avail/20-unhint-small-dejavu-sans.conf',
 '/etc/fonts/conf.avail/64-arphic-uming.conf',
 '/etc/fonts/conf.avail/65-fonts-lmodern.conf',
 '/etc/fonts/conf.avail/20-unhint-small-dejavu-lgc-sans.conf',
 '/etc/fonts/conf.avail/30-droid-noto-mono.conf',
 '/etc/fonts/conf.avail/30-droid-noto.conf',
 '/etc/fonts/conf.avail/58-dejavu-lgc-serif.conf',
 '/etc/fonts/conf.avail/20-unhint-small-dejavu-lgc-serif.conf',
 '/etc/fonts/conf.avail/65-fonts-texgyre-math.conf',
 '/etc/fonts/conf.avail/57-dejavu-sans-mono.conf',
 '/etc/fonts/conf.avail/20-unhint-small-dejavu-lgc-sans-mono.conf',
 '/etc/fonts/conf.avail/65-fonts-texgyre.conf',
 '/etc/fonts/conf.avail/58-dejavu-lgc-sans-mono.conf',
 '/etc/fonts/conf.avail/25-arphic-uming-render.conf',
 '/etc/fonts/conf.avail/65-droid-sans-fallback.conf',
 '/etc/fonts/conf.avail/25-arphic-uming-bitmaps.conf',
 '/etc/fonts/conf.avail/57-dejavu-sans.conf',
 '/etc/fonts/conf.avail/58-dejavu-lgc-sans.conf',
 '/etc/fonts/conf.avail/35-arphic-uming-aliases.conf',
 '/etc/fonts/conf.avail/20-unhint-small-dejavu-serif.conf',
 '/etc/fonts/conf.avail/41-arphic-uming.conf',
 '/etc/fonts/conf.avail/20-unhint-small-dejavu-sans-mono.conf',
 '/etc/fonts/conf.avail/57-dejavu-serif.conf',
 '/etc/fonts/conf.avail/90-arphic-uming-embolden.conf',
 '/etc/fonts/conf.d/60-latin.conf',
 '/etc/fonts/conf.d/20-unhint-small-dejavu-sans.conf',
 '/etc/fonts/conf.d/61-urw-nimbus-sans.conf',
 '/etc/fonts/conf.d/61-urw-fallback-backwards.conf',
 '/etc/fonts/conf.d/30-metric-aliases.conf',
 '/etc/fonts/conf.d/64-arphic-uming.conf',
 '/etc/fonts/conf.d/61-urw-z003.conf',
 '/etc/fonts/conf.d/61-urw-nimbus-mono-ps.conf',
 '/etc/fonts/conf.d/65-fonts-lmodern.conf',
 '/etc/fonts/conf.d/61-urw-standard-symbols-ps.conf',
 '/etc/fonts/conf.d/90-fonts-unfonts-extra.conf',
 '/etc/fonts/conf.d/64-11-tlwg-waree.conf',
 '/etc/fonts/conf.d/20-unhint-small-dejavu-lgc-sans.conf',
 '/etc/fonts/conf.d/61-urw-nimbus-roman.conf',
 '/etc/fonts/conf.d/10-yes-antialias.conf',
 '/etc/fonts/conf.d/61-urw-p052.conf',
 '/etc/fonts/conf.d/61-urw-d050000l.conf',
 '/etc/fonts/conf.d/40-nonlatin.conf',
 '/etc/fonts/conf.d/89-tlwg-umpush-synthetic.conf',
 '/etc/fonts/conf.d/80-delicious.conf',
 '/etc/fonts/conf.d/69-unifont.conf',
 '/etc/fonts/conf.d/48-spacing.conf',
 '/etc/fonts/conf.d/64-21-tlwg-typo.conf',
 '/etc/fonts/conf.d/50-user.conf',
 '/etc/fonts/conf.d/51-local.conf',
 '/etc/fonts/conf.d/45-generic.conf',
 '/etc/fonts/conf.d/90-fonts-linux-libertine.conf',
 '/etc/fonts/conf.d/58-dejavu-lgc-serif.conf',
 '/etc/fonts/conf.d/64-22-tlwg-typist.conf',
 '/etc/fonts/conf.d/89-tlwg-garuda-synthetic.conf',
 '/etc/fonts/conf.d/90-synthetic.conf',
 '/etc/fonts/conf.d/20-unhint-small-dejavu-lgc-serif.conf',
 '/etc/fonts/conf.d/65-nonlatin.conf',
 '/etc/fonts/conf.d/65-fonts-texgyre-math.conf',
 '/etc/fonts/conf.d/64-15-laksaman.conf',
 '/etc/fonts/conf.d/57-dejavu-sans-mono.conf',
 '/etc/fonts/conf.d/65-fonts-persian.conf',
 '/etc/fonts/conf.d/20-unhint-small-vera.conf',
 '/etc/fonts/conf.d/10-hinting-slight.conf',
 '/etc/fonts/conf.d/61-urw-fallback-generics.conf',
 '/etc/fonts/conf.d/64-02-tlwg-norasi.conf',
 '/etc/fonts/conf.d/20-unhint-small-dejavu-lgc-sans-mono.conf',
 '/etc/fonts/conf.d/61-urw-c059.conf',
 '/etc/fonts/conf.d/64-23-tlwg-mono.conf',
 '/etc/fonts/conf.d/65-fonts-texgyre.conf',
 '/etc/fonts/conf.d/58-dejavu-lgc-sans-mono.conf',
 '/etc/fonts/conf.d/70-no-bitmaps.conf',
 '/etc/fonts/conf.d/64-13-tlwg-garuda.conf',
 '/etc/fonts/conf.d/25-arphic-uming-render.conf',
 '/etc/fonts/conf.d/89-tlwg-laksaman-synthetic.conf',
 '/etc/fonts/conf.d/30-intel-clear.conf',
 '/etc/fonts/conf.d/57-dejavu-sans.conf',
 '/etc/fonts/conf.d/10-scale-bitmap-fonts.conf',
 '/etc/fonts/conf.d/61-urw-bookman.conf',
 '/etc/fonts/conf.d/58-dejavu-lgc-sans.conf',
 '/etc/fonts/conf.d/35-arphic-uming-aliases.conf',
 '/etc/fonts/conf.d/90-fonts-baekmuk.conf',
 '/etc/fonts/conf.d/20-unhint-small-dejavu-serif.conf',
 '/etc/fonts/conf.d/30-comic-sans.conf',
 '/etc/fonts/conf.d/41-arphic-uming.conf',
 '/etc/fonts/conf.d/60-generic.conf',
 '/etc/fonts/conf.d/11-lcdfilter-default.conf',
 '/etc/fonts/conf.d/64-01-tlwg-kinnari.conf',
 '/etc/fonts/conf.d/20-unhint-small-dejavu-sans-mono.conf',
 '/etc/fonts/conf.d/45-latin.conf',
 '/etc/fonts/conf.d/64-14-tlwg-umpush.conf',
 '/etc/fonts/conf.d/49-sansserif.conf',
 '/etc/fonts/conf.d/89-tlwg-kinnari-synthetic.conf',
 '/etc/fonts/conf.d/61-urw-gothic.conf',
 '/etc/fonts/conf.d/57-dejavu-serif.conf',
 '/etc/fonts/conf.d/90-arphic-uming-embolden.conf',
 '/etc/fonts/conf.d/90-fonts-unfonts-core.conf',
 '/etc/fonts/conf.d/64-10-tlwg-loma.conf',
 '/etc/ghostscript/cidfmap.d/90gs-cjk-resource-korea1.conf',
 '/etc/ghostscript/cidfmap.d/90gs-cjk-resource-gb1.conf',
 '/etc/ghostscript/cidfmap.d/90gs-cjk-resource-japan2.conf',
 '/etc/ghostscript/cidfmap.d/90gs-cjk-resource-cns1.conf',
 '/etc/ghostscript/cidfmap.d/90gs-cjk-resource-japan1.conf',
 '/etc/ghostscript/fontmap.d/10fonts-urw-base35.conf',
 '/etc/apt/listchanges.conf',
 '/etc/gtk-3.0/im-multipress.conf',
 '/etc/security/limits.conf',
 '/etc/security/access.conf',
 '/etc/security/pam_env.conf',
 '/etc/security/namespace.conf',
 '/etc/security/time.conf',
 '/etc/security/faillock.conf',
 '/etc/security/group.conf',
 '/etc/security/sepermit.conf',
 '/etc/selinux/semanage.conf',
 '/etc/lighttpd/conf-available/90-javascript-alias.conf',
 '/etc/lighttpd/conf-enabled/90-javascript-alias.conf',
 '/etc/environment.d/90qt-a11y.conf',
 '/etc/udev/udev.conf',
 '/etc/dhcp/dhclient.conf',
 '/etc/xdg/user-dirs.conf',
 '/etc/modules-load.d/modules.conf',
 '/etc/rsyslog.d/postfix.conf',
 '/etc/sysctl.d/99-sysctl.conf',
 '/etc/apache2/conf-available/javascript-common.conf',
 '/etc/gtk-2.0/im-multipress.conf',
 '/etc/systemd/system.conf',
 '/etc/systemd/networkd.conf',
 '/etc/systemd/pstore.conf',
 '/etc/systemd/timesyncd.conf',
 '/etc/systemd/user.conf',
 '/etc/systemd/logind.conf',
 '/etc/systemd/sleep.conf',
 '/etc/systemd/journald.conf']

## For recursive search, it is better using iterative method `iglob` to save space in RAM. 
## The iglob runs iteratively so doesn't buffer the output unlike glob.

for filename in glob.iglob('/etc/**/*.conf', recursive=True):
    print(filename)

/etc/nftables.conf
/etc/deluser.conf
/etc/ld.so.conf
/etc/reportbug.conf
/etc/sysctl.conf
/etc/xattr.conf
/etc/libaudit.conf
/etc/nsswitch.conf
/etc/sudo_logsrvd.conf
/etc/sensors3.conf
/etc/mke2fs.conf
/etc/ca-certificates.conf
/etc/e2scrub.conf
/etc/logrotate.conf
/etc/gai.conf
/etc/ucf.conf
/etc/resolv.conf
/etc/host.conf
/etc/sudo.conf
/etc/debconf.conf
/etc/pam.conf
/etc/adduser.conf
/etc/ld.so.conf.d/x86_64-linux-gnu.conf
/etc/ld.so.conf.d/libc.conf
/etc/fonts/fonts.conf
/etc/fonts/conf.avail/20-unhint-small-dejavu-sans.conf
/etc/fonts/conf.avail/64-arphic-uming.conf
/etc/fonts/conf.avail/65-fonts-lmodern.conf
/etc/fonts/conf.avail/20-unhint-small-dejavu-lgc-sans.conf
/etc/fonts/conf.avail/30-droid-noto-mono.conf
/etc/fonts/conf.avail/30-droid-noto.conf
/etc/fonts/conf.avail/58-dejavu-lgc-serif.conf
/etc/fonts/conf.avail/20-unhint-small-dejavu-lgc-serif.conf
/etc/fonts/conf.avail/65-fonts-texgyre-math.conf
/etc/fonts/conf.avail/57-dejavu-sans-mono.conf
/etc/fonts/conf.avail/20-unhint-small-dejavu-lgc-sans-mono.conf
/etc/fonts/conf.avail/65-fonts-texgyre.conf
/etc/fonts/conf.avail/58-dejavu-lgc-sans-mono.conf
/etc/fonts/conf.avail/25-arphic-uming-render.conf
/etc/fonts/conf.avail/65-droid-sans-fallback.conf
/etc/fonts/conf.avail/25-arphic-uming-bitmaps.conf
/etc/fonts/conf.avail/57-dejavu-sans.conf
/etc/fonts/conf.avail/58-dejavu-lgc-sans.conf
/etc/fonts/conf.avail/35-arphic-uming-aliases.conf
/etc/fonts/conf.avail/20-unhint-small-dejavu-serif.conf
/etc/fonts/conf.avail/41-arphic-uming.conf
/etc/fonts/conf.avail/20-unhint-small-dejavu-sans-mono.conf
/etc/fonts/conf.avail/57-dejavu-serif.conf
/etc/fonts/conf.avail/90-arphic-uming-embolden.conf
/etc/fonts/conf.d/60-latin.conf
/etc/fonts/conf.d/20-unhint-small-dejavu-sans.conf
/etc/fonts/conf.d/61-urw-nimbus-sans.conf
/etc/fonts/conf.d/61-urw-fallback-backwards.conf
/etc/fonts/conf.d/30-metric-aliases.conf
/etc/fonts/conf.d/64-arphic-uming.conf
/etc/fonts/conf.d/61-urw-z003.conf
/etc/fonts/conf.d/61-urw-nimbus-mono-ps.conf
/etc/fonts/conf.d/65-fonts-lmodern.conf
/etc/fonts/conf.d/61-urw-standard-symbols-ps.conf
/etc/fonts/conf.d/90-fonts-unfonts-extra.conf
/etc/fonts/conf.d/64-11-tlwg-waree.conf
/etc/fonts/conf.d/20-unhint-small-dejavu-lgc-sans.conf
/etc/fonts/conf.d/61-urw-nimbus-roman.conf
/etc/fonts/conf.d/10-yes-antialias.conf
/etc/fonts/conf.d/61-urw-p052.conf
/etc/fonts/conf.d/61-urw-d050000l.conf
/etc/fonts/conf.d/40-nonlatin.conf
/etc/fonts/conf.d/89-tlwg-umpush-synthetic.conf
/etc/fonts/conf.d/80-delicious.conf
/etc/fonts/conf.d/69-unifont.conf
/etc/fonts/conf.d/48-spacing.conf
/etc/fonts/conf.d/64-21-tlwg-typo.conf
/etc/fonts/conf.d/50-user.conf
/etc/fonts/conf.d/51-local.conf
/etc/fonts/conf.d/45-generic.conf
/etc/fonts/conf.d/90-fonts-linux-libertine.conf
/etc/fonts/conf.d/58-dejavu-lgc-serif.conf
/etc/fonts/conf.d/64-22-tlwg-typist.conf
/etc/fonts/conf.d/89-tlwg-garuda-synthetic.conf
/etc/fonts/conf.d/90-synthetic.conf
/etc/fonts/conf.d/20-unhint-small-dejavu-lgc-serif.conf
/etc/fonts/conf.d/65-nonlatin.conf
/etc/fonts/conf.d/65-fonts-texgyre-math.conf
/etc/fonts/conf.d/64-15-laksaman.conf
/etc/fonts/conf.d/57-dejavu-sans-mono.conf
/etc/fonts/conf.d/65-fonts-persian.conf
/etc/fonts/conf.d/20-unhint-small-vera.conf
/etc/fonts/conf.d/10-hinting-slight.conf
/etc/fonts/conf.d/61-urw-fallback-generics.conf
/etc/fonts/conf.d/64-02-tlwg-norasi.conf
/etc/fonts/conf.d/20-unhint-small-dejavu-lgc-sans-mono.conf
/etc/fonts/conf.d/61-urw-c059.conf
/etc/fonts/conf.d/64-23-tlwg-mono.conf
/etc/fonts/conf.d/65-fonts-texgyre.conf
/etc/fonts/conf.d/58-dejavu-lgc-sans-mono.conf
/etc/fonts/conf.d/70-no-bitmaps.conf
/etc/fonts/conf.d/64-13-tlwg-garuda.conf
/etc/fonts/conf.d/25-arphic-uming-render.conf
/etc/fonts/conf.d/89-tlwg-laksaman-synthetic.conf
/etc/fonts/conf.d/30-intel-clear.conf
/etc/fonts/conf.d/57-dejavu-sans.conf
/etc/fonts/conf.d/10-scale-bitmap-fonts.conf
/etc/fonts/conf.d/61-urw-bookman.conf
/etc/fonts/conf.d/58-dejavu-lgc-sans.conf
/etc/fonts/conf.d/35-arphic-uming-aliases.conf
/etc/fonts/conf.d/90-fonts-baekmuk.conf
/etc/fonts/conf.d/20-unhint-small-dejavu-serif.conf
/etc/fonts/conf.d/30-comic-sans.conf
/etc/fonts/conf.d/41-arphic-uming.conf
/etc/fonts/conf.d/60-generic.conf
/etc/fonts/conf.d/11-lcdfilter-default.conf
/etc/fonts/conf.d/64-01-tlwg-kinnari.conf
/etc/fonts/conf.d/20-unhint-small-dejavu-sans-mono.conf
/etc/fonts/conf.d/45-latin.conf
/etc/fonts/conf.d/64-14-tlwg-umpush.conf
/etc/fonts/conf.d/49-sansserif.conf
/etc/fonts/conf.d/89-tlwg-kinnari-synthetic.conf
/etc/fonts/conf.d/61-urw-gothic.conf
/etc/fonts/conf.d/57-dejavu-serif.conf
/etc/fonts/conf.d/90-arphic-uming-embolden.conf
/etc/fonts/conf.d/90-fonts-unfonts-core.conf
/etc/fonts/conf.d/64-10-tlwg-loma.conf
/etc/ghostscript/cidfmap.d/90gs-cjk-resource-korea1.conf
/etc/ghostscript/cidfmap.d/90gs-cjk-resource-gb1.conf
/etc/ghostscript/cidfmap.d/90gs-cjk-resource-japan2.conf
/etc/ghostscript/cidfmap.d/90gs-cjk-resource-cns1.conf
/etc/ghostscript/cidfmap.d/90gs-cjk-resource-japan1.conf
/etc/ghostscript/fontmap.d/10fonts-urw-base35.conf
/etc/apt/listchanges.conf
/etc/gtk-3.0/im-multipress.conf
/etc/security/limits.conf
/etc/security/access.conf
/etc/security/pam_env.conf
/etc/security/namespace.conf
/etc/security/time.conf
/etc/security/faillock.conf
/etc/security/group.conf
/etc/security/sepermit.conf
/etc/selinux/semanage.conf
/etc/lighttpd/conf-available/90-javascript-alias.conf
/etc/lighttpd/conf-enabled/90-javascript-alias.conf
/etc/environment.d/90qt-a11y.conf
/etc/udev/udev.conf
/etc/dhcp/dhclient.conf
/etc/xdg/user-dirs.conf
/etc/modules-load.d/modules.conf
/etc/rsyslog.d/postfix.conf
/etc/sysctl.d/99-sysctl.conf
/etc/apache2/conf-available/javascript-common.conf
/etc/gtk-2.0/im-multipress.conf
/etc/systemd/system.conf
/etc/systemd/networkd.conf
/etc/systemd/pstore.conf
/etc/systemd/timesyncd.conf
/etc/systemd/user.conf
/etc/systemd/logind.conf
/etc/systemd/sleep.conf
/etc/systemd/journald.conf

10.5. Module pathlib (Exercise)#

The module provides an object-oriented approach to handling filesystem paths.

from pathlib import Path

Path.home()

PosixPath('/home/hostadm')

Create a new directory, /home/hostadm/Python2

p = Path.home() / 'Python2'
p.mkdir()

Create a new file in the directory:

nf = p / 'new_file.txt'
nf
nf.touch()

Check if the directory and the file exist:

nf.is_file()

True

p.is_dir()

True

Rename the file:

nfn = p / 'new_file2.txt'

nf.rename(nfn)

PosixPath('/home/hostadm/Python2/new_file2.txt')

Write data into file:

nf.write_text(data)

139

Read text from data:

s = Path('/home/hostadm/Python2/new_file.txt').read_text()
print(s)

 Create some data. 
In this particular case, 
the data is just two sentences stored in a string
 
 Line 4 

 Line 5 

 Line 6 
 
 Line 7

List files in the directory:

s = [ s.name for s in list(p.glob('*'))]
print(s)

['new_file2.txt', 'new_file.txt']

10.6. Exception handling in Python (Exercise)#

## Example: find all conf files that contain keyword `network`.
## This will throw an error since some files in /etc are not readable by the user.

for filename in glob.iglob('/etc/**/*.conf', recursive=True):
    with open(filename) as f:
        line = f.read()
        if 'network' in line:
            print(filename)

/etc/sysctl.conf
/etc/nsswitch.conf
/etc/sudo.conf
/etc/security/access.conf
/etc/sysctl.d/99-sysctl.conf
/etc/systemd/networkd.conf

To allow the script to run and prompt us about errors, we run it within block

try:
...
except ...

## Lets handle the exception above
for filename in glob.iglob('/etc/**/*.conf', recursive=True):
    try:
        with open(filename) as f:
            line = f.read()
            if 'network' in line:
                print(filename)
                
    except PermissionError as e:
            print(f"can't read {filename}: {e}")

/etc/sysctl.conf
/etc/nsswitch.conf
/etc/sudo.conf
/etc/security/access.conf
/etc/sysctl.d/99-sysctl.conf
/etc/systemd/networkd.conf

There is another, FileNotFoundError, error due to a symbolic link to a missing file.

We can handle multipple exceptions:

for filename in glob.iglob('/etc/**/*.conf', recursive=True):
    try:
        with open(filename) as f:
            line = f.read()
            if 'network' in line:
                print(filename)
                
    except (PermissionError, FileNotFoundError) as e:
            print(f"can't read {filename}: {e}")

/etc/sysctl.conf
/etc/nsswitch.conf
/etc/sudo.conf
/etc/security/access.conf
/etc/sysctl.d/99-sysctl.conf
/etc/systemd/networkd.conf

The last two exceptions we can catch with a generic OSError:

except OSError as e:

In numerical computations, you may need to deal with the division by zero. For example:

denominator = [1.2, 4.0, 0.001, 45.0, 0.0]

numerator = 0.1

for i in denominator:
    s = numerator / i
    print(s)

0.08333333333333334
0.025
100.0
0.0022222222222222222

---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
Cell In[135], line 6
      3 numerator = 0.1
      5 for i in denominator:
----> 6     s = numerator / i
      7     print(s)

ZeroDivisionError: float division by zero

We need to put safe guards with

except ZeroDivisionError as e:

try:
    for i in denominator:
      s = numerator / i
      print(s)

except ZeroDivisionError  as e:
    print('Division by zero, i = ', i)

0.08333333333333334
0.025
100.0
0.0022222222222222222
Division by zero, i =  0.0

10.7. JSON (Exercise)#

JSON is JavaScript Object Notation. It is used to store and transfer data.

  • jason.dump() writes data in json file

  • json.load() reads data from json file

  • json.dumps() encodes a python object (usually dictionary) into json

  • json.loads() decodes json into python object

10.7.1. Dump data to json file#

import json

# Define dictionary with services and their server IP addresses

services = {
  # for all servers in NJ (building : re)
're':
{'dns': ['192.168.3.250', '192.168.11.104'],
'dhcp': '192.168.3.250',
'ldapmaster': '192.168.3.100' ,
'ldap': ['192.168.3.100', '192.168.11.104'],
'ravada': '192.168.3.40'
},
  # For all the servers in FL (building : fl)
'fl':
{'dns': ['192.168.11.104', '192.168.3.250'],
'dhcp': '192.168.11.104',
'ldapmaster': '192.168.3.100',
'ldap': '[192.168.11.104, 192.168.3.100]',
'ravada': '192.168.11.100'
}
}

# the json file where the output must be stored 
with  open("services.json", "w") as out_file:
    json.dump(services, out_file, indent = 4) 
  

%cat services.json

{
    "re": {
        "dns": [
            "192.168.3.250",
            "192.168.11.104"
        ],
        "dhcp": "192.168.3.250",
        "ldapmaster": "192.168.3.100",
        "ldap": [
            "192.168.3.100",
            "192.168.11.104"
        ],
        "ravada": "192.168.3.40"
    },
    "fl": {
        "dns": [
            "192.168.11.104",
            "192.168.3.250"
        ],
        "dhcp": "192.168.11.104",
        "ldapmaster": "192.168.3.100",
        "ldap": "[192.168.11.104, 192.168.3.100]",
        "ravada": "192.168.11.100"
    }
}

10.7.2. Read data from json file:#

with open('services.json','r') as f:
    data= json.load(f)

%cat services.json

{
    "re": {
        "dns": [
            "192.168.3.250",
            "192.168.11.104"
        ],
        "dhcp": "192.168.3.250",
        "ldapmaster": "192.168.3.100",
        "ldap": [
            "192.168.3.100",
            "192.168.11.104"
        ],
        "ravada": "192.168.3.40"
    },
    "fl": {
        "dns": [
            "192.168.11.104",
            "192.168.3.250"
        ],
        "dhcp": "192.168.11.104",
        "ldapmaster": "192.168.3.100",
        "ldap": "[192.168.11.104, 192.168.3.100]",
        "ravada": "192.168.11.100"
    }
}

data

{'re': {'dns': ['192.168.3.250', '192.168.11.104'],
  'dhcp': '192.168.3.250',
  'ldapmaster': '192.168.3.100',
  'ldap': ['192.168.3.100', '192.168.11.104'],
  'ravada': '192.168.3.40'},
 'fl': {'dns': ['192.168.11.104', '192.168.3.250'],
  'dhcp': '192.168.11.104',
  'ldapmaster': '192.168.3.100',
  'ldap': '[192.168.11.104, 192.168.3.100]',
  'ravada': '192.168.11.100'}}

type(data)

dict

for keys in data:
    print(keys,':',data[keys],'\n')

re : {'dns': ['192.168.3.250', '192.168.11.104'], 'dhcp': '192.168.3.250', 'ldapmaster': '192.168.3.100', 'ldap': ['192.168.3.100', '192.168.11.104'], 'ravada': '192.168.3.40'} 

fl : {'dns': ['192.168.11.104', '192.168.3.250'], 'dhcp': '192.168.11.104', 'ldapmaster': '192.168.3.100', 'ldap': '[192.168.11.104, 192.168.3.100]', 'ravada': '192.168.11.100'}

10.7.3. Encode a dictionary into json string:#

json_string = json.dumps(services)

print(json_string)

{"re": {"dns": ["192.168.3.250", "192.168.11.104"], "dhcp": "192.168.3.250", "ldapmaster": "192.168.3.100", "ldap": ["192.168.3.100", "192.168.11.104"], "ravada": "192.168.3.40"}, "fl": {"dns": ["192.168.11.104", "192.168.3.250"], "dhcp": "192.168.11.104", "ldapmaster": "192.168.3.100", "ldap": "[192.168.11.104, 192.168.3.100]", "ravada": "192.168.11.100"}}

print(json.dumps(services, indent=4))

{
    "re": {
        "dns": [
            "192.168.3.250",
            "192.168.11.104"
        ],
        "dhcp": "192.168.3.250",
        "ldapmaster": "192.168.3.100",
        "ldap": [
            "192.168.3.100",
            "192.168.11.104"
        ],
        "ravada": "192.168.3.40"
    },
    "fl": {
        "dns": [
            "192.168.11.104",
            "192.168.3.250"
        ],
        "dhcp": "192.168.11.104",
        "ldapmaster": "192.168.3.100",
        "ldap": "[192.168.11.104, 192.168.3.100]",
        "ravada": "192.168.11.100"
    }
}

10.7.4. Decode json string into a dictionary:#

dict = json.loads(json_string)

print(dict)

{'re': {'dns': ['192.168.3.250', '192.168.11.104'], 'dhcp': '192.168.3.250', 'ldapmaster': '192.168.3.100', 'ldap': ['192.168.3.100', '192.168.11.104'], 'ravada': '192.168.3.40'}, 'fl': {'dns': ['192.168.11.104', '192.168.3.250'], 'dhcp': '192.168.11.104', 'ldapmaster': '192.168.3.100', 'ldap': '[192.168.11.104, 192.168.3.100]', 'ravada': '192.168.11.100'}}

10.7.5. Get the values by referencing the keys in the dictionaries:#

for keys in dict:
    dict2 = dict[keys]
    print(dict2['dns'])

['192.168.3.250', '192.168.11.104']
['192.168.11.104', '192.168.3.250']

10.8. Parsing arguments with argparse (Exercise)#

It allows to pass parameters to a python script.

All the exercises below should run in the Linux command prompt. There are 3 python scripts in the directory: parser_1.py, parser_2.py, and parser_3.py

Exersizes:

# Import the library
import argparse
# Create the parser
parser = argparse.ArgumentParser()
# Add an argument
parser.add_argument('--alpha', type=int, required=True)
# Parse the argument
args = parser.parse_args()

# use and print the argument
value = args.alpha * 10
print('alpha = ', args.alpha, f', value = {args.alpha} * 10 = ', value)

10.8.1. Run the script#

Save the above cell in a script, say parser_1.py, specify that it is a python script in the header:

#!/home/hostadm/miniconda3/bin/python3

Make it executable:

chmod a+x parser_1.py

Run the script with the parameter:

./parser_1.py --alpha 15

10.8.2. Multiple arguments with help options#

import argparse
parser = argparse.ArgumentParser()

parser.add_argument('--a1', type = str, required=True, help = 'First coefficient')
parser.add_argument('--a2', type = str, required=True, help = 'Second coefficient')
parser.add_argument('--a3', type = str, help = 'Third coefficient')

args = parser.parse_args()

s = f'{args.a1}*x^2 + {args.a2}*x + {args.a2}'

print(s)

10.8.3. Invoking help#

Save the content of the cell above in file, say, parser_2.py, then run as follows:

./parser_2.py -h
./parser_2.py  --a1 34 --a2 55 --a3 56

10.8.4. Multiple arguments of the same type#

import argparse
parser = argparse.ArgumentParser()

parser.add_argument('--values', type=int, nargs=3)

args = parser.parse_args()
sum = sum(args.values)
print('Sum:', sum)

Run it with values:

./parser_3.py --values 4 8 9

For any number of values, set nargs = '+'

10.9. Reference#

Good tutorial on subparsers

Contents