Commands, shells, processes, files.

1. Commands, shells, processes, files.#

1.1. Overview#

Today, we’ll first talk about the course organization.
Then we’ll discuss the fundamentals of the Linux operating system: kernel, processes, file system, user accounts, and basic commands.

1.2. Course organization#

Virtual Desktop (VD) access
Canvas
Slack channel
Course website
Lectures
Practical exercises
Midterm and Final exams

1.3. Course agenda#

Lectures and practical recitations on the virtual computers remotely.
Midterm exam (on November 5 2024).
Final exam (on December 17 2024).

1.4. Course topics#

Commands, shells, processes.
Server virtualization with KVM.
Linux package management.
Networking.
Network File System (NFS).
Systemd. Process scheduling (at, cron).
Linux and Windows interoperability.
Application compilation.
Shell scripting, awk, grep, sed.
Python for Linux administration.
Security topics.

1.5. Unix history milestones and foundation of Linux#

1969: a UNIX like OS on PDP 7 was written by Ken Thompson in Bell Labs It was single user, non-multitasking, written in assembler;
1973: C programming language by Dennis Ritchie in Bell Labs;
1975: First widely available version of UNIX (written in C);
1978: “The C Programming Language” book by Kernighan and Ritchie;
Late seventies: two main stream UNIX distributions - System V from Bell Labs and Berkeley release.
Eighties: commercial UNIX operating systems: DEC, Sun, IBM, etc.
1985: GNU, the Free Software Foundation, is founded (Richard Stallman)
1987: The first GNU C compiler, gcc, becomes freely available via ftp.
1991: Starting of the Free Family Berkely like OS, BSD, (NetBSD, FreeBSD, OpenBSD).
1991: Finnish graduate student Linus Torvalds announces release of Linux based on GNU tools.
Now days: numerous distributions of Linux (Links to an external site.); the fastest developing OS; accepted by some commercial hardware and software vendors (IBM, Oracle, HP, Amazon, Google).
Has become the leading OS in High Performance Computing, Cloud Computing, mobile OS and apps, microcontroller devices.

1.6. What makes Linux different from traditional Unix#

GNU Public License (GPL) for the Kernel and OS applications: free to use; get the source; develop and disclose your source.
The same Kernel and GNU C libraries (glibc) for different distributions: if you can run an application on one kind of Linux, its a matter of setting libraries to run it on the other distribution.
Runs on various computer architectures: x86, x86_64, ARM, POWERPC, SPARC, ALPHA, MIPS, HPPA, etc.
The fastest developing OS.
Various vendor support.
Strong community support.

1.7. Why would you run Linux?#

Available free kernel, modifiable for specific tasks. Vendors can easily port their software into Linux, for example, for microcontrollers and file systems.

Various developers can work on the same software without intellectual property issues. Open source software allows upgrading and changing software by the people using it.

No license fee for every version of the software on any computer - no charge for scalability.

It is your choice what commercial software you need to buy.

1.8. Big Red 200 HPC cluster in Indiana University#

Big Red in IU

Big Red 200 features 640 compute nodes, each equipped with 256 GB of memory and two 64-core, 2.25 GHz, 225-watt AMD EPYC 7742 processors. Big Red 200 also includes 64 GPU-accelerated nodes, each with 256 GB of memory, a single 64-core, 2.0 GHz, 225-watt AMD EPYC 7713 processor, and four NVIDIA A100 GPUs. Big Red 200 has a theoretical peak performance (Rpeak) of nearly 7 petaFLOPS.

Big Red 200 is managed with HPE’s Performance Cluster Manager (HPCM) and currently runs SUSE Enterprise Linux Server (SLES) version 15 on the compute, GPU, and login nodes.

1.9. Purpose of operating systems#

Operating system concepts

Common features

title

1.10. Linux on a typical Server/Desktop/Laptop (Intel Core i7) hardware#

From the file system, Linux kernel is loaded into the RAM.
The kernel controls the hardware and processes.
On the diagram:
- PCH - Platform Controller Hub
- DMI - Direct Memory Interface
- LPC - Low Pin Count; a simple interface to slower I/O devices

title

1.11. Linux kernel and applications#

In general, Linux implies rather just a Kernel than a whole OS.

Linux OS distributions include: generic or customized Kernel and generic or customized GNU software (libraries, shells, and applications).

Source codes of the Kernel, shells and GNU applications are freely available to anyone.

Traditional tools for Linux administration and development: C - programming language Shell scripting, Perl, Python

Linux kernel

1.14. User accounts#

In order to login to a system a user has to authenticate with his/her credentials: user name and password.

Authentication types:

local (password/shadow/group) and
Domain based (NIS, Kerberos, LDAP).

For local authentication, a user should have an account on the system.

Account File	Entry
/etc/passwd	mike:x:1001:1001:Michael Whites:/home/mike:/bin/bash
/etc/shadow	mike:$1$zXCV7fz8ii84grbZhj:14087:0:99999:7:::
/etc/group	mike:x:1001:mike
/etc/group	admin:x:112:hostadm,mike

Accounts can be created only by a superuser (root).

Commands to create user accounts:

adduser (interactive command)

useradd (can be included in a script).

1.14.1. Exercise#

Create a new user account by following the instructor.

Become root:

sudo -s

Create a new user account, mike:

adduser mike

Verify the account existence:

id mike

Change password of the user:

passwd mike

1.15. Linux shells#

When you login to a Linux system, you get a command shell.

Shells are listed in /etc/shells. Default Linux shell: bash

System commands, scripts and applications run in the shell - they become child processes of the shell.

Besides commands, the shell is using variables.

You define a shell variable with assignment command, =, for example:

svar='file_name.txt'

The variable is referenced in the shell with $ prefix, for example:

echo $svar

Shell variables are local to the shell.

A shell variable becomes an environment variable after executing command export on it:

svar='VAR_1'  #initialize shell variable
export svar #becomes environment variable
env | grep svar

Environment variables are inherited by the child shells and processes.

Commands executed in the shell should be either built-in shell commands, or addressed with the full path, or located in the PATH environment variable.

echo $PATH

1.15.1. Exercise#

Create a new shell variable, svar:

svar='VAR_1'
echo $svar

Start a new child shell and see if svar is defined there:

bash
echo $svar

Exit from the child shell, export the variable, and see if it is defined in a child shell:

exit
export svar
bash
echo $svar

1.16. Data storage hierarchy#

Data is stored in files.
Files are created in a file system.
A file system rezides on a volume.
A volume is created on drives.

How data is stored

A file system can also be created on the whole drive or a slice of the drive (partition).
To show up as a directory (folder), a file system needs to be mounted.

1.17. HDD and SSD#

Mechanical hard drive

Elements of a file system

The cylinders define partition boundaries.
The sectors define data blocks.
A file system is created within a disk partition.
Files are created within a file system.

NAND cell technology for SSD, SD, NVMe, Flash drive.

NAND cells are combined into strings, pages, blocks, planes, and banks.

1.18. Files and Directories#

INODES contain information about files

a pointer to the file data blocks
link count
the owner
the group
mode
size
last access time
last modification time
last update time (attributes)

Type of files

Regular or ordinary files
Directories
Symbolic links
Device files
Directory and Inodes

Files are stored by blocks

Default block size of ext4 file system is 4K:

stat -f /home

1.19. File Permissions and Ownerships#

1.19.1. Permissions#

Octal	Binary	Permission	Meaning
0	000	none	All turned off
1	001	–x	Execute
2	010	-w-	Write
3	011	-wx	Write, execute
4	100	r–	Read
5	101	r-x	Read, execute
6	110	rw-	Read, write
7	111	rwx	Read, write, execute

1.19.2. File Ownerships: USER (u), GROUP (g), OTHERS (o)#

Changing permissions Permissions are applied for USER, GROUP and OTHERS (rwx rwx rwx)

chmod 660 testf.txt
ls -l  testf.txt

-rw-rw----    1 mike   staff          0 Jan 18 10:26 testf.txt

umask sets default permission for files and directories. For example,

umask 022

perm/umask	directory	file
permission	777	666
umask	-022	-022
permission	755	644

1.20. Assigning permissions and ownerships on files (Exercise)#

Create a new directory EX1 and step into it:

mkdir EX1
cd EX1

Check umask and create a new file, mf1.txt. See the file attributes with command ls.

umask
touch mf1.txt
ls -l mf1.txt

Change umask and create anothe file, mf2.txt

umask 022
touch mf2.txt
ls -l mf2.txt

Create another file, mf3.exe and make it executable:

touch mf3.exe
chmod 755 mf3.exe
ls -l mf3.exe

Try runing files mf2.txt and mf3.exe, and see which one is runnable:

./mf2.txt
./mf3.exe

As you can see only the executable file can run.
Changing the user ownership on file mf1.txt:

sudo chown mike mf1.txt
ls -l mf1.txt

Note, you need to be the root user when changing the file ownership.

Changing the group ownership on file mf2.txt:

sudo chown :mike mf2.txt
ls -l mf2.txt

Changing both the user and group ownership on file mf3.exe:

sudo chown mike:mike mf3.exe
ls -l mf*

1.21. Sticky bit (Exercise)#

Sticky bit on a directory protects files in the directory from been modified/removed by non their owners.

For example /tmp directory on a Unix/Linux system has a sticky bit set

ls -ld /tmp

drwxrwxrwt 4 root root 4096 2018-07-25 16:29 /tmp

Open a new terminal and become root:

sudo -s

Create a new directory, temp, and give it a world writable permissions

mkdir temp
chmod 777 temp

In the directory, user root creates a new empty file:

cd temp
touch ex3

In the other terminal,try removing this file as user hostadm:

cd temp
rm ex3

The file has been deleted.

Change the permission on temp director in the root user terminal:

chmod 1777 temp

Then repeat the steps above with creating and removing file ex3 in directory temp

1.22. Processes#

Program running on a system is a process. Linux is a multi-processing (multi-tasking) system.

Process types:

User Processes. A user process is one that is initiated by a regular user account and runs in user space.
Daemon Processes. A daemon process is an application that is designed to run in the background, usually related to a service.
Kernel Threads. Kernel processes execute only in kernel space.

Privilege levels. Kernel space and user space:

1.23. Process states#

When created, a process goes through various states:

1.24. Process termination and renice command#

Program running on a system is a process. Linux is a multi-processing (multi-tasking) system.
Attributes: Lifetime, PID, PPID, UID, GID, env variables, CWD.
Process with PID=1 is init
Monitoring: ps (for full listing, use ps -ef or ps -aux), top, pstree

Signal a running processes with PID=1009:

kill -HUP 1009    #Hang up; re-read config files
kill -1   1009
kill -9  1009     #Kill; stop unconditionally
kill -KILL 1009
kill -15   1009    #Terminate gracefully
kill -TERM 1009
kill -TSTP 1009   #Suspend; can be continued
kill -18   1009

Process can be started with lower and higher priority through nice (range: -20 highest, +19 lowest)
```
nice -10 matlab
nice --10 matlab
```
Process can be re-niced at run time:
```
renice 5 1009
renice -5 1009
```

1.25. Exercise with process termination#

When administrating a Linux system, it is often needed to terminate processes that consume a lot of resources (CPU and/or RAM), and slow down everything else.

Install package gimp

install the pkg
```
apt install gimp
```
Start gimp application on a “background”:
```
gimp &
```
Find out the PID of the process:
```
pgrep gimp
```
The output shows the PID of the process, for exmple 4198.
Terminate process 4198:
```
kill -15 4198
```
An alternative way to kill an application is by using pkill command:
```
pkill -15 gimp
```
If option -15 above doesn’t work, try -9
Reference: Understanding Linux processes

1.26. Process data streams#

Most of the Linux system processes have three data streams with file descriptors “0”, “1”, and “3”:

standard input “0” (stdin)

standard output “1” (stdo)

standard error “2” (stderr)
The standard output and standard error are directed to the screen of your monitor; the standard input is read from a keyboard.

It is possible to redirect the standard output and error into files, for example
```
ps -ef 1 > stdo.out
```
or
```
ps -ef > stdo.out
```
To redirect both stdo and stderr to the same file:

command > output.txt 2>&1

For example,
```
ps -ef > output.txt 2>&1
```
If you need to discard the stdo stream, you can re-direct it to /dev/null:
```
ps -ef 1>/dev/null
```
It is also possible to re-direct the standart output of one process into the standard input of the other using pipes "|":
```
ps -ef | less
```

1.27. Background processes (exercise)#

Processes with Input/Output detached from the terminal are called background processes.

In a terminal window, run gimp graphics software:

gimp

Suspend the interactive process, gimp, and move it to the background:

# Ctrl-Z
jobs
bg %1

Now the shell can be used for running the other commands and applications. Move the background job (with its ID) back to the foreground and terminate gimp process:

fg %1
# Ctrl-C

1.28. Commands for directory manipulation (exercise)#

When you login into the system, you automatically get into your home directory

#1. Determine the directory you are in:

pwd

#2. Go to some other directory, for example, /etc:

cd /etc

#3. Go to your home directory:

cd $HOME

or

cd ~

#4. Go back-and-forth between directories and check where you are:

cd /etc
pwd
cd -
pwd
cd -
pwd

#5. Create a new directory above your home directory:

mkdir Newdir

#6. Go to the new directory and check where you are:

cd Newdir
pwd

#7. Go one step back to the parent directory and check where you are:

cd ..
pwd

#8. Remove the new directory:

rmdir Newdir

#9. Create a new directory above your home directory with subdirectories. Try the following:

mkdir Newdir/one/two/three

Does it work? Try the following:

mkdir -p Newdir/one/two/three

#10. Create a few files in Newdir/one:

cd Newdir/one; touch f1.t; touch f2.t; touch f3.t

#11. Go back into the original directory and try to delete Newdir:

cd -
rmdir Newdir

Does it work ? Try the following:

rm -R Newdir

#12. Repeat step #9; Copy the directory with its tree into another directory:

cp -dpR Newdir Newdir-1

same as

cp -a Newdir Newdir-1

1.29. Commands to read text file content (exercise)#

#13. Show content of file /etc/hosts

cat /etc/hosts
tac /etc/hosts
less /etc/hosts

#14. Show the first and last 10 lines of a file

head -n 10 /etc/nsswitch.conf
tail -n 10 /etc/nsswitch.conf

1.30. Various important commands (exercise)#

#15. Show date on the system clock

date

#16. File system usage

df -h

#17. Disk usage

du -h /home
du -s /home
du --max-depth=1 /var
du -x -h /

#18. Display data

echo $[10*3+2]
echo '$[10*3+2]'
echo "$[10*3+2]"

#19. Determine file type

file /bin/ls
file /etc/hosts

#20. Who is logined to the system

who

#21. The owner of the current shell

whoami

#22. Getting information about a file or command, for example netstat

whatis netstat
apropos netstat
man netstat
info netstat

#23. Compressing/uncompressing files with gzip, and bzip2:

cd ~
cp /etc/hosts  hosts.txt
gzip hosts.txt
ls -l
less hosts.txt.gz
more hosts.txt.gz
zcat hosts.txt.gz
gunzip hosts.txt.gz
bzip2 hosts.txt
bzip2 -d hosts.txt.bz2

#24. Searching for files by using command find

cd /tmp
mkdir newfiles; cd newfiles
touch fff.txt
cd ~
find / -name fff.txt
cd /tmp
find . -name fff.txt
find . -name '*ff*'
find /tmp -name fff.txt -exec rm '{}' ';'
find /var -size +1000k
find /var -size +1000000c
find /home -user hostadm
find /var -mtime -5
find /var -maxdepth 2 -mtime -20

#25. Searching for system files

which dpkg
whereis dpkg
locate dpkg

#26. Using tar-gzip to archive a directory. Create a new directory with subdirectories above your home directory, then tar and gzip it. For example,

mkdir -p Newdir/one/two/three
cd Newdir/one/two
cp /etc/hosts .
cd three
cp /etc/passwd .
cd ~
tar -cvf Newdir.tar Newdir
gzip Newdir.tar
ls -l Newdir.tar.gz
rm -R Newdir
cp Newdir.tar.gz /tmp
cd /tmp
gunzip Newdir.tar.gz
tar -xvf Newdir.tar

#27. To tar and gzip file with one command, try the following:

tar -zcvf Newdir.tgz Newdir
rm -R Newdir
ls -l Newdir.tgz
tar -zxvf Newdir.tgz

#28. Archive/restore from input file list by cpio command:

cd /etc
find . | cpio -ov  > /tmp/etc.cpio
mkdir /tmp/RESTORE
cd /tmp/RESTORE
cpio -iv < ../etc.cpio

#29. Download files from a website:

wget https://people.sc.fsu.edu/~jburkardt/c_src/laplace_mpi/laplace_mpi.c

Similarly

curl -O https://people.sc.fsu.edu/~jburkardt/c_src/laplace_mpi/laplace_mpi.c

#30. Command history in bash shell:

history

Find a command in the history by the regular expression in its name:

ctrl + R

then type the string contained in the command.

#31. Command history file .bash_history is updated when a user exits the shell.

tail .bash_history

The size or file .bash_history is defined by env variable HISTSIZE

echo $HISTSIZE

#32. Commands to read hardware information. CPU info:

lscpu

PCI device info:

lspci

Memory info:

lshw -class memory

Storage controller:

lshw -class storage

Disks:

lshw -class disk

Network interfaces:

lshw -class network 

Video card and display info:

lshw -class video 