1. Commands, shells, processes, files.¶
1.1. Overview¶
Today, we’ll first talk about the course organization.
Then we’ll discuss the fundamentals of the Linux operating system: kernel, processes, file system, user accounts, and basic commands.
1.2. Course topics¶
Commands, shells, processes
Installation and upgrade
Server virtualization with KVM
Linux package management
Networking
Network File System (NFS)
System Run levels and process scheduling (at, cron)
Linux and Windows interoperability
Building Linux applications and packages
Shell and Python scripting
Password security
Network security
Application security and intrusion detection
File system and communication encryption
1.3. Course agenda¶
Lectures and practical recitations on the lab computers remotely.
Midterm exam (mid June 2021);
Final exam (July 7 2021).
1.4. Unix history milestones and foundation of Linux¶
1969: a UNIX like OS on PDP 7 was written by Ken Thompson in Bell Labs It was single user, non-multitasking, written in assembler;
1973: C programming language by Dennis Ritchie in Bell Labs;
1975: First widely available version of UNIX (written in C);
Late seventies: two main stream UNIX distributions - System V from Bell Labs and Berkeley release.
Eighties: commercial UNIX operating systems: DEC, Sun, IBM, etc.
1985: GNU, the Free Software Foundation, is founded (Richard Stallman)
1987: The first GNU C compiler, gcc, becomes freely available via ftp.
1991: Starting of the Free Family Berkely like OS, BSD, (NetBSD, FreeBSD, OpenBSD).
1991: Finnish graduate student Linus Torvalds announces release of Linux based on GNU tools.
Now days: numerous distributions of Linux (Links to an external site.); the fastest developing OS; accepted by some commercial hardware and software vendors (IBM, Oracle, HP, Amazon, Google).
Has become the leading OS in High Performance Computing, Cloud Computing, mobile OS and apps, microcontroller devices.
1.5. What makes Linux different from traditional Unix¶
GNU Public License (GPL) for the Kernel and OS applications: free to use; get the source; develop and disclose your source.
The same Kernel and GNU C libraries (glibc) for different distributions: if you can run an application on one kind of Linux, its a matter of setting libraries to run it on the other distribution.
Runs on various computer architectures: x86, x86_64, ARM, POWERPC, SPARC, ALPHA, MIPS, HPPA, etc.
The fastest developing OS.
Various vendor support.
Strong community support.
1.6. Why would you run Linux?¶
Available free kernel, modifiable for specific tasks. Vendors can easily port their software into Linux, for example, for microcontrollers and file systems.
Various developers can work on the same software without intellectual property issues. Open source software allows upgrading and changing software by the people using it.
No license fee for every version of the software on any computer - no charge for scalability.
It is your choice what commercial software you need to buy.
1.8. Linux on a typical Server/Desktop/Laptop (Intel Core i7) hardware¶
From the file system, Linux kernel is loaded into the RAM.
The kernel controls the hardware and processes.
On the diagram:
PCH - Platform Controller Hub
DMI - Direct Memory Interface
LPC - Low Pin Count; a simple interface to slower I/O devices
1.9. Linux kernel and applications¶
In general, Linux implies rather just a Kernel than a whole OS.
Linux OS distributions include: generic or customized Kernel and generic or customized GNU software (libraries, shells, and applications).
Source codes of the Kernel, shells and GNU applications are freely available to anyone.
Traditional tools for Linux administration and development: C - programming language Shell scripting, Perl, Python
1.10. Login to a Linux terminal¶
1.11. Login to the virtual desktop (Exercise)¶
1.12. User accounts¶
In order to login to a system a user has to authenticate with his/her credentials: user name and password.
Authentication types:
local (password/shadow/group) and
Domain based (NIS, Kerberos, LDAP).
For local authentication, a user should have an account on the system.
Account File |
Entry |
---|---|
/etc/passwd |
mike:x:1001:1001:Michael Whites:/home/mike:/bin/bash |
/etc/shadow |
mike:\(1\)zXCV7fz8ii84grbZhj:14087:0:99999:7::: |
/etc/group |
mike:x:1001:mike |
/etc/group |
admin:x:112:hostadm,mike |
Accounts can be created only by a superuser (root).
Commands to create user accounts:
adduser (interactive command)
useradd (can be included in a script).
1.12.1. Exercise¶
Create a new user account by following the instructor.
Become root:
sudo -s
Create a new user account, mike:
adduser mike
Verify the account existence:
id mike
Change password of the user:
passwd mike
1.13. Linux shells¶
When you login to a Linux system, you get a command shell.
Shells are listed in /etc/shells
. Default Linux shell: bash
System commands, scripts and applications run in the shell - they become child processes of the shell.
Shell variables are local to the shell.
A shell variable becomes an environment variable after executing command export on it:
svar=VAR_1 #initialize shell variable
export svar #becomes environment variable
env | grep svar
Environment variables are inherited by the child shells and processes.
Commands executed in the shell should be either built-in shell commands, or addressed with the full path, or located in the PATH
environment variable.
echo $PATH
1.13.1. Exercise¶
Create a new shell variable, svar:
svar=VAR_1
echo $svar
Start a new child shell and see if svar is defined there:
bash
echo $svar
Exit from the child shell, export the variable, and see if it is defined in a child shell:
exit
export svar
bash
echo $svar
1.15. File Permissions and Ownerships¶
1.15.1. Permissions¶
Octal |
Binary |
Permission |
Meaning |
---|---|---|---|
0 |
000 |
none |
All turned off |
1 |
001 |
–x |
Execute |
2 |
010 |
-w- |
Write |
3 |
011 |
-wx |
Write, execute |
4 |
100 |
r– |
Read |
5 |
101 |
r-x |
Read, execute |
6 |
110 |
rw- |
Read, write |
7 |
111 |
rwx |
Read, write, execute |
1.15.2. File Ownerships: USER (u), GROUP (g), OTHERS (o)¶
Changing permissions Permissions are applied for USER, GROUP and OTHERS (rwx rwx rwx)
chmod 660 testf.txt
ls -l testf.txt
-rw-rw---- 1 mike staff 0 Jan 18 10:26 testf.txt
umask sets default permission for files and directories. For example,
umask 022
perm/umask |
directory |
file |
---|---|---|
permission |
777 |
666 |
umask |
-022 |
-022 |
permission |
755 |
644 |
1.16. Assigning permissions and ownerships on files (Exercise)¶
Create a new directory EX1
and step into it:
mkdir EX1
cd EX1
Check umask
and create a new file, mf1.txt
. See the file attributes with command ls
.
umask
touch mf1.txt
ls -l mf1.txt
Change umask
and create anothe file, mf2.txt
umask 022
touch mf2.txt
ls -l mf2.txt
Create another file, mf3.exe and make it executable:
touch mf3.exe
chmod 755 mf3.exe
ls -l mf3.exe
Try runing files mf2.txt
and mf3.exe
, and see which one is runnable:
./mf2.txt
./mf3.exe
As you can see only the executable file can run.
Changing the user ownership on file mf1.txt
:
sudo chown mike mf1.txt
ls -l mf1.txt
Note, you need to be the root user when changing the file ownership.
Changing the group ownership on file mf2.txt
:
sudo chown :mike mf2.txt
ls -l mf2.txt
Changing both the user and group ownership on file mf3.exe
:
sudo chown mike:mike mf3.exe
ls -l mf*
1.17. Sticky bit (Exercise)¶
Sticky bit on a directory protects files in the directory from been modified/removed by non their owners.
For example /tmp
directory on a Unix/Linux system has a sticky bit set
ls -ld /tmp
drwxrwxrwt 4 root root 4096 2018-07-25 16:29 /tmp
Open a new terminal and become root:
sudo -s
Create a new directory, temp
, and give it a world writable permissions
mkdir temp
chmod 777 temp
In the directory, user root creates a new empty file:
cd temp
touch ex3
In the other terminal,try removing this file as user hostadm:
cd temp
rm ex3
The file has been deleted.
Change the permission on temp
director in the root user terminal:
chmod 1777 temp
Then repeat the steps above with creating and removing file ex3
in directory temp
1.18. Processes¶
Program running on a system is a process. Linux is a multi-processing (multi-tasking) system.
Process types:
User Processes. A user process is one that is initiated by a regular user account and runs in user space.
Daemon Processes. A daemon process is an application that is designed to run in the background, usually related to a service.
Kernel Threads. Kernel processes execute only in kernel space.
Privilege levels. Kernel space and user space:
1.20. Process termination and renice command¶
Program running on a system is a process. Linux is a multi-processing (multi-tasking) system.
Attributes: Lifetime, PID, PPID, UID, GID, env variables, CWD.
Process with PID=1 is
init
Monitoring:
ps
(for full listing, useps -ef
orps -aux
),top
,pstree
Signal a running processes with PID=1009:
kill -HUP 1009 #Hang up; re-read config files kill -1 1009 kill -9 1009 #Kill; stop unconditionally kill -KILL 1009 kill -15 1009 #Terminate gracefully kill -TERM 1009 kill -TSTP 1009 #Suspend; can be continued kill -18 1009
Process can be started with lower and higher priority through
nice
(range: -20 highest, +19 lowest)nice -10 matlab nice --10 matlab
Process can be re-niced at run time:
renice 5 1009 renice -5 1009
Exercise
When administrating a Linux system, it is often needed to terminate processes that consume a lot of resources (CPU and/or RAM), and slow down everything else.
Install package
gimp
install the pkg
apt-get install gimp
Start
gimp
application on a “background”:gimp &
Find out the PID of the process:
pgrep gimp
The output shows the PID of the process, for exmple 4198.
Terminate process 4198:kill -15 4198
An alternative way to kill an application is by using
pkill
command:pkill -15 gimp
If option
-15
above doesn’t work, try-9
1.21. Process data streams¶
Most of the Linux system processes have three data streams:
standard input “0” (stdin)
standard output “1” (stdo)
standard error “2” (stderr)
The standard output and standard error are directed to the screen of your monitor; the standard input is read from a keyboard.
It is possible to redirect the standard output and error into files, for example
ps -ef 1 > stdo.out
or
ps -ef > stdo.out
To redirect both stdo and stderr to the same file:
command > output.txt 2>&1
For example,
ps -ef > output.txt 2>&1
If you need to discard the stdo stream, you can re-direct it to
/dev/null
:ps -ef 1>/dev/null
It is also possible to re-direct the standart output of one process into the standard input of the other using pipes "|":ps -ef | less
1.22. Background processes (exercise)¶
Processes with Input/Output detached from the terminal are called background processes.
In a terminal window, run gimp graphics software:
gimp
Suspend the interactive process, gimp, and move it to the background:
# Ctrl-Z
jobs
bg %1
Now the shell can be used for running the other commands and applications. Move the background job (with its ID) back to the foreground and terminate gimp process:
fg %1
# Ctrl-C
1.23. Commands for directory manipulation (exercise)¶
When you login into the system, you automatically get into your home directory
Determine the directory you are in:
pwd
Go to some other directory, for example, /etc:
cd /etc
Go to your home directory:
cd $HOME
or
cd ~
Go back-and-forth between directories and check where you are:
cd /etc
pwd
cd -
pwd
cd -
pwd
Create a new directory above your home directory:
mkdir Newdir
Go to the new directory and check where you are:
cd Newdir
pwd
Go one step back to the parent directory and check where you are:
cd ..
pwd
Remove the new directory:
rmdir Newdir
Create a new directory above your home directory with subdirectories. Try the following:
mkdir Newdir/one/two/three
Does it work? Try the following:
mkdir -p Newdir/one/two/three
Create a few files in Newdir/one:
cd Newdir/one; touch f1.t; touch f2.t; touch f3.t
Go back into the original directory and try to delete Newdir:
cd -
rmdir Newdir
Does it work ? Try the following:
rm -R Newdir
Repeat step #9; Copy the directory with its tree into another directory:
cp -dpR Newdir Newdir-1
same as
cp -a Newdir Newdir-1
1.24. Commands to read text file content (exercise)¶
Show content of file /etc/hosts
cat /etc/hosts
tac /etc/hosts
less /etc/hosts
Show the first and last 10 lines of a file
head -n 10 /etc/nsswitch.conf
tail -n 10 /etc/nsswitch.conf
1.25. Various important commands (exercise)¶
Show date on the system clock
date
File system usage
df -h
Disk usage
du -h /home
du -s /home
du --max-depth=1 /var
du -x /
du -x /home
du -x /usr
Display data
echo $[10*3+2]
echo '$[10*3+2]'
echo "$[10*3+2]"
Determine file type
file /bin/ls
file /etc/hosts
Who is logined to the system
who
The owner of the current shell
whoami
Getting information about a file or command, for example netstat
whatis netstat
apropos netstat
man netstat
info netstat
Compressing/uncompressing files with gzip, and bzip2:
cd ~
cp /etc/hosts hosts.txt
gzip hosts.txt
ls -l
less hosts.txt.gz
more hosts.txt.gz
zcat hosts.txt.gz
gzip -d hosts.txt.gz
bzip2 hosts.txt
bzip2 -d hosts.txt.bz2
Searching for files by using command find
cd /tmp
mkdir newfiles; cd newfiles
touch fff.txt
cd ~
find / -name fff.txt
cd /tmp
find . -name fff.txt
find . -name '*ff*'
find /tmp -name fff.txt -exec rm '{}' ';'
find /var -size +1000k
find /var -size +1000000c
find /home -user hostadm
find /var -mtime -5
find /var -maxdepth 2 -mtime -20
Searching for system files
which dpkg
whereis dpkg
locate dpkg
Using tar-gzip to archive a directory. Create a new directory with subdirectories above your home directory, then tar and gzip it. For example,
mkdir -p Newdir/one/two/three
cd Newdir/one/two
cp /etc/hosts .
cd three
cp /etc/passwd .
cd ~
tar -cvf Newdir.tar Newdir
gzip Newdir.tar
ls -l Newdir.tar.gz
rm -R Newdir
cp Newdir.tar.gz /tmp
cd /tmp
gzip -d Newdir.tar.gz
tar -xvf Newdir.tar
To tar and gzip file with one command, try the following:
tar -zcvf Newdir.tgz Newdir
rm -R Newdir
ls -l Newdir.tgz
tar -zxvf Newdir.tgz
Archive/restore from input file list by cpio command:
cd /etc
find . | cpio -ov > /tmp/etc.cpio
mkdir /tmp/RESTORE
cd /tmp/RESTORE
cpio -iv < ../etc.cpio
Command history in bash shell:
history
Find a command in the history by the regular expression in its name:
ctrl + R
then type the string contained in the command. 30. Command history file .bash_history is updated when a user exits the shell.
tail .bash_history
The size or file .bash_history is defined by env variable HISTSIZE
echo $HISTSIZE
Commands to read hardware information. CPU info:
lscpu
PCI device info:
lspci
Memory info:
lshw -class memory
Storage controller:
lshw -class storage
Disks:
lshw -class disk
Network interfaces:
lshw -class network
Video card and display info:
lshw -class video