10. Application compilation¶

Compilation stages.
Static and shared libraries.
GNU make utility.
Environment variable settings.

10.1. Compilation stages¶

Suppose one needs to compile a single source C program, code.c, into an executable binary, code.x:

gcc -o code.x code.c

This implies the following implicit compilation steps that also can be done in 4 stages:

stage	name	command	product
1	Preprocessing	gcc -E code.c -o code.i	#define, #ifdef, #include are processed
2	Compiling	gcc -S code.i -o code.s	Assembly file, code.s
3	Asembling	gcc -c code.s -o code.o	Creates object file in the machine language
4	Linking	gcc code.o -o code.x	Code is linked with the other .o object files and libraries into an executable binary

10.2. Compilation in stages (Exercise)¶

Download code.c

wget http://linuxcourse.rutgers.edu/summer/html/lessons/application_compilation/downloads/code.c

Look at the content of the source file:

less code.c

Compile the source file in the four stages - Preprocessing, Compiling, Assembling, and Linking:

    gcc -E code.c -o  code.i
    gcc -S code.i -o  code.s
    gcc -c code.s -o  code.o
    gcc    code.o -o  code.x -lm

Check the file type of the compilation products:

    file code.i
    file code.s
    file code.o
    file code.x

Browse the content of the two ASCII files, code.i and code.s:

    less code.i
    less code.s

Run the executable, code.x:

./code.x

10.3. Source code with multiple files¶

If a source code consists of several files and needs libraries to be linked with, the compilation procedure can be done either in one step:

gcc code.c extra_1.c extra_2.c -o code.x -lextra

or multiple steps:

    gcc -c code.c
    gcc -c extra_1.c
    gcc -c extra_2.c
    gcc -o code.x code.o extra_1.o extra_2.o -lextra

If the header files *.h and the libraries *.a are located in the directories different from the source *.c files, for example ../include and ../lib, their location needs to be specified as follows:

gcc  -o code.x code.o extra_1.o extra_2.o -I../include -L../lib -lextra

10.4. Static libraries¶

A part of the code that can be compiled separately and linked with the other codes should be built as a library.

Building a static library

    gcc -c lib1.c
    gcc -c lib2.c
    gcc -c lib3.c
    ar -cr libextra.a lib1.o lib2.o lib3.o
    ranlib libextra.a

to see the object files included in the library:

ar -t libextra.a

or

nm libextra.a

10.5. Static library content and compilation (Exerecise)¶

Get the list of modules included in the Math library, libm.a:

ar -t /usr/lib/x86_64-linux-gnu/libm-2.31.a

Btw, file /usr/lib/x86_64-linux-gnu/libm.a is a part of package libc6-dev. If the file is missing on your computer, install package libc6-dev by using command apt-get install libc6-dev.

Compile code.c with the Math library statically:

gcc code.c -o code_stat.x -lm -static

10.6. Shared libraries¶

Shared libraries, unlike static ones, are not compiled into the application binaris. They are loaded into the memory and linked with the application at a run time. Other running codes can use the loaded shared libraries.

Building a shared library

gcc -c -fPIC lib1.c
gcc -c -fPIC lib2.c
gcc -c -fPIC lib3.c
gcc -shared lib1.o lib2.o lib3.o -o libextra.so

To compile the application with the library we would follow the step below:

gcc -o code.x code.c -L. -lextra

To see what the shared libraries the code needs, use ldd command

ldd code.x

If the library is not found, include it into LD_LIBRARY_PATH environment variable, for example:

export LD_LIBRARY_PATH=.

Alternatively the shared library path can be included into /etc/ld.so.conf. After /etc/ld.so.conf has been updated, we need to run command

/sbin/ldconfig -p

10.7. Compilation with a shared library (Exercise)¶

Get the list of shared libraries required by command, ls:

ldd /bin/ls

Compile code.c with the Math shared library (default compilation):

gcc code.c -o code_shared.x -lm

Get the list of shared libraries required by code_stat.x, and code_shared.x:

ldd code_stat.x
ldd code_shared.x

Compare the sizes of the executable files:

ls -lh  code_*.x

10.8. Single code compilation and libraries (Exercise)¶

By following the instructions below, download and compile a C code, singe.c, that computes the scalar product of two vectors:

mkdir single_code
cd single_code
wget http://linuxcourse.rutgers.edu/summer/html/lessons/application_compilation/downloads/single.c
gcc -o app.x single.c
./app.x

When prompted for the input parameters, type in the dimension of the vectors and the value of their components, for example:

ouput/input:

Vector dimension n=2
Input 2 coordinate x: 3 5
Input 2 coordinate y: 1 6

Compile the code with a static library. This time, the original single.c file has been devided into 3 separate modules: main.c, Scalar_Product.h, and Scalar_Product.c. Download the archive and untar it into a new directory:

cd
wget http://linuxcourse.rutgers.edu/summer/html/lessons/application_compilation/downloads/source_code.tgz
mkdir code_static
cp source_code.tgz code_static
cd code_static
tar -zxvf source_code.tgz

Build the static library:

gcc -c Scalar_Product.c
ar -cr libScalar_Product.a Scalar_Product.o
ranlib libScalar_Product.a

Verify the content of library libScalar_Product.a:

ar -t libScalar_Product.a

Compile the main code and link with the library:

gcc -c main.c
gcc -o app.x main.o -L. -lScalar_Product

Make sure the compiled code runs fine:

./app.x

Compile the code with a shared library. Create a new directory and untar the source code in it:

cd ..
mkdir code_so
cp source_code.tgz code_so
cd code_so
tar -zxvf source_code.tgz

Build the shared library:

gcc -fPIC -c Scalar_Product.c
gcc -shared Scalar_Product.o -o libScalar_Product.so

Compile the main code and link with the shared library:

gcc -c main.c
gcc -o apps.x main.o -L. -lScalar_Product

Verify that apps.x executable requires libScalar_Product.so library:

ldd apps.x

It should show that the shared library is not found, libScalar_Product.so => not found. Try running the code

./apps.x

It should fail to run. Set LD_LIBRARY_PATH environment variable to include the current working directory:

export LD_LIBRARY_PATH=./

Check if the loader finds the library and try running the code again:

ldd apps.x
./apps.x

10.9. Software builders¶

To automate software builds, tests, and installation there are specialized developer tools including:

Make
CMake
Bazel
Gradle
Ninja
Meson
npm
MS Visual Studio for linux

10.10. make and Makefile¶

Utility make is used for compilation and installation of a software with many source files.
make keeps track with the dependencies and file access times tamps. Compilation is done in the correct order to meet prerequisites and satisfy the dependencies. If one or several source codes have been updated, only these and the files that depend on them would be recompiled by command make.
Makefile syntax:

Makefile example:

Makefile

app.x: main.o libScalar_Product.a
        gcc -o app.x main.o -L. -lScalar_Product

main.o: main.c Scalar_Product.h
        gcc -c main.c

Scalar_Product.o: Scalar_Product.c
        gcc -c Scalar_Product.c

libScalar_Product.a: Scalar_Product.o
        ar -cr libScalar_Product.a Scalar_Product.o
        ranlib libScalar_Product.a

Makefile or makefile would be processed by running command make:

make

If the Makefile has different name, for example Makefile1, it can be processed as follows:

make -f Makefile1

10.11. Makefile with macros¶

Parameters that appear in the Makefile multiple times or that need to be modified, depending on the environment, can be expressed through macronames.
Makefile with macros example:

Makefile:

#First, we define the MACRONAMES:
SLIB = libScalar_Product.a
APP=app.x
CC = gcc
CFLAGS = -O3
LIBPATH = .

#Then we refer to them by $(MACRONAMES):
$(APP): main.o $(SLIB)
        $(CC) $(CFLAGS) -o $(APP) main.o -L$(LIBPATH) -lScalar_Product

main.o: main.c Scalar_Product.h
        $(CC) $(CFLAGS) -c main.c

Scalar_Product.o: Scalar_Product.c
        $(CC) $(CFLAGS) -c Scalar_Product.c

$(SLIB): Scalar_Product.o
        ar -cru $(SLIB) Scalar_Product.o
        ranlib $(SLIB)

10.12. Makefile with phony targets¶

When a target in a Makefile rather means just an action than a file, it is called a phony target. Makefile with phony targets (clean, install, and uninstall):

Makefile:

SLIB = libScalar_Product.a
APP=app.x
CC = gcc
CFLAGS = -O3
LIBPATH = .
INSTPATH = /usr/local

$(APP): main.o $(SLIB)
        $(CC) $(CFLAGS) -o $(APP) main.o -L$(LIBPATH) -lScalar_Product

main.o: main.c Scalar_Product.h
        $(CC) $(CFLAGS) -c main.c

Scalar_Product.o: Scalar_Product.c
        $(CC) $(CFLAGS) -c Scalar_Product.c

$(SLIB): Scalar_Product.o
        ar -cr $(SLIB) Scalar_Product.o
        ranlib $(SLIB)

clean:
        -rm -f *.o *.a $(APP)

install:
        @cp -p $(APP) $(INSTPATH)/bin ;\
        chown root:root $(INSTPATH)/bin/$(APP) ;\
        cp -p $(SLIB) $(INSTPATH)/lib ;\
        chown root:root $(INSTPATH)/lib/$(SLIB) ;\
        echo "Install $(APP) and $(SLIB) into $(INSTPATH)"

uninstall:
        -@rm -f $(INSTPATH)/bin/$(APP)
        -@rm -f $(INSTPATH)/lib/$(SLIB)
        @echo "Removed $(APP) and $(SLIB) from $(INSTPATH)"

Special symbols in the command prefixes mean:
- - don’t quit on error. For example, -rm won’t quit if there is no files to delete.
@ - don’t print the command itself.

10.13. Makefile with target `all`¶

Target all in a Makefile envokes a sequence of other targets.
Makefile with target all:

Makefile:

SLIB = libScalar_Product.a
APP=app.x
CC = gcc
CFLAGS = -O3
LIBPATH = .
INSTPATH = /usr/local

all: $(APP) install clean

$(APP): main.o $(SLIB)
        $(CC) $(CFLAGS) -o $(APP) main.o -L$(LIBPATH) -lScalar_Product

main.o: main.c Scalar_Product.h
        $(CC) $(CFLAGS) -c main.c

Scalar_Product.o: Scalar_Product.c
        $(CC) $(CFLAGS) -c Scalar_Product.c

$(SLIB): Scalar_Product.o
        ar -cr $(SLIB) Scalar_Product.o
        ranlib $(SLIB)

clean:
        -rm -f *.o *.a $(APP)

install:
        @cp -p $(APP) $(INSTPATH)/bin ;\
        chown root:root $(INSTPATH)/bin/$(APP) ;\
        cp -p $(SLIB) $(INSTPATH)/lib ;\
        chown root:root $(INSTPATH)/lib/$(SLIB) ;\
        echo "Install $(APP) and $(SLIB) into $(INSTPATH)"

uninstall:
        -@rm -f $(INSTPATH)/bin/$(APP)
        -@rm -f $(INSTPATH)/lib/$(SLIB)
        @echo "Removed $(APP) and $(SLIB) from $(INSTPATH)"

10.14. Makefile with suffix rules¶

When there is need to compile many *.c codes into *.o binaries, the suffix rule can be implemented to avoid many target lines in the Makefile.
Makefile with the suffix rule:

Makefile

SLIB = libScalar_Product.a
APP=app.x
CC = gcc
CFLAGS = -O3
LIBPATH = .
INSTPATH = /usr/local


$(APP): main.o $(SLIB)
        $(CC) $(CFLAGS) -o $@ main.o -L$(LIBPATH) -lScalar_Product

.c.o:
        @echo "Compiling" $<
        $(CC) -c $<

$(SLIB): Scalar_Product.o
        ar -cr $@ $<
        ranlib $@

clean:
        -rm -f *.o *.a $(APP)

install:
        @cp -p $(APP) $(INSTPATH)/bin ;\
        chown root:root $(INSTPATH)/bin/$(APP) ;\
        cp -p $(SLIB) $(INSTPATH)/lib ;\
        chown root:root $(INSTPATH)/lib/$(SLIB) ;\
        echo "Install $(APP) and $(SLIB) into $(INSTPATH)"

uninstall:
        -@rm -f $(INSTPATH)/bin/$(APP)
        -@rm -f $(INSTPATH)/lib/$(SLIB)
        @echo "Removed $(APP) and $(SLIB) from $(INSTPATH)"

Special symbol	Meaning
`$@`	name of current target.
`$<`	name of current dependency.
`$*`	name of current dependency without extension.
`$?`	list of dependencies changed more recently than current target.

10.15. Makefile with suffix rules and object file dependencies¶

The suffix rule discussed in the previous slide doesn’t contain dependencies for the targets. Below, we include the dependencies for the object files.
Makefile with the suffix rule and object file dependencies

Makefile:

SLIB = libScalar_Product.a
APP=app.x
CC = gcc
CFLAGS = -O3
LIBPATH = .
INSTPATH = /usr/local
OBJS = main.o

$(APP): main.o $(SLIB)
        $(CC) $(CFLAGS) -o $@ main.o -L$(LIBPATH) -lScalar_Product

.c.o:
        @echo "Compiling" $<
        $(CC) -c $<

$(SLIB): Scalar_Product.o
        ar -cr $@ $<
        ranlib $@

$(OBJS): Scalar_Product.h

clean:
    -rm -f *.o *.a $(APP)

install:
        @cp -p $(APP) $(INSTPATH)/bin ;\
        chown root:root $(INSTPATH)/bin/$(APP) ;\
        cp -p $(SLIB) $(INSTPATH)/lib ;\
        chown root:root $(INSTPATH)/lib/$(SLIB) ;\
        echo "Install $(APP) and $(SLIB) into $(INSTPATH)"

uninstall:
        -@rm -f $(INSTPATH)/bin/$(APP)
        -@rm -f $(INSTPATH)/lib/$(SLIB)
        @echo "Removed $(APP) and $(SLIB) from $(INSTPATH)"

10.16. Makefiles (Exercises)¶

Simple Makefile: Download Makefile1 into directory code_static, created during the exercises with the static library compilation.

cd code_static
rm *.o app.x *.a
wget http://linuxcourse.rutgers.edu/summer/html/lessons/application_compilation/downloads/Makefile1

Check the content of the Makefile:

less Makefile1

Run make for compilation:

make -f Makefile1

Notice the sequence of the compilation steps. Run the command make again. Modify the time stamps of main.c then re-run make and notice the compilation steps made:

touch main.c
make -f Makefile1

Modify the time stamps of Scalar_Product.h then re-run make and notice the compilation steps made:

touch Scalar_Product.h
make -f Makefile1

Modify the time stamps of Scalar_Product.c then re-run make and notice the compilation steps made:

touch Scalar_Product.c
make -f Makefile1

Makefile containing macros, suffix rules, and phony targets. Download Makefile6 into directory code_static:

wget http://linuxcourse.rutgers.edu/summer/html/lessons/application_compilation/downloads/Makefile6

See the content of Makefile6:

less Makefile6

Run the following command to remove the compiled files:

make -f Makefile6 clean

Verify there is no longer files app.x, *.o, and libScalar_Product.a in directory code_static. Run compilation:

make -f Makefile6

Install the compiled binary, app.x and the library, libScalar_Product.a into /usr/local:

sudo make -f Makefile6 install

Check if files /usr/local/bin/app.x and /usr/local/lib/libScalar_Product.a have been created.

ls -l /usr/local/bin
ls -l /usr/local/lib

Remove the installed files:

sudo make -f Makefile6 uninstall

Verify taht the files have been removed:

ls -l /usr/local/bin
ls -l /usr/local/lib

10.17. Multiple Makefiles. Software installation. (Exercises)¶

Compilation and installation of software from its source code usually involves 4 steps: download and untar the archive, configuration, compilation, and installation. Multiple Makefiles may be involved if the software source is complex.

Download the tar archive and extract the files:

wget http://linuxcourse.rutgers.edu/summer/html/lessons/application_compilation/downloads/Project.tgz
tar -zxvf Project.tgz

Check out the directory tree of Project and the files contained there:

apt-get install tree
tree Project

You should notice two Makefiles: one in the root of directory Project and the other in subdirectory src1.

Configuration:

cd Project
./configure

This sets the environment for the Makefiles.

Compilation:

make

Installation:

sudo make install

10.18. Makefiles in the project¶

Command make above processes Makefile in directrory Project:

Makefile:

SRC = src1

all:
        (cd $(SRC); $(MAKE) )

clean:
        (cd $(SRC); $(MAKE) clean )

install:
        (cd $(SRC); $(MAKE) install )

uninstall:
        (cd $(SRC); $(MAKE) uninstall )

This Makefile contains phony targets, envoking make in subdirectory src1, which compiles the source codes, *.c, builds the library, the application binary and places them into subdirectories lib and bin.

Makefile in directory Project/src1:

Makefile:

SLIB = libScalar_Product.a
APP=app.x
CC = gcc
CFLAGS = -O3
LIBPATH = ../lib
INCPATH = ../include
BIN = ../bin
INSTPATH = /usr/local
OBJS = main.o

$(APP): main.o $(SLIB)
    $(CC) $(CFLAGS) -o $@ main.o -L$(LIBPATH) -lScalar_Product
    -cp $@ $(BIN)

.c.o:
    @echo "Compiling" $<
    $(CC) -c $< -I$(INCPATH)

$(SLIB): Scalar_Product.o
    ar -cru $@ $<
    ranlib $@
    -cp $@ $(LIBPATH)

$(OBJS): $(INCPATH)/Scalar_Product.h

clean:
    -rm -f *.o $(LIBPATH)/*.a *.a $(BIN)/$(APP) $(APP)

install:
    @cp -p $(BIN)/$(APP) $(INSTPATH)/bin ;\
    chown root:root $(INSTPATH)/bin/$(APP) ;\
    cp -p $(LIBPATH)/$(SLIB) $(INSTPATH)/lib ;\
    chown root:root $(INSTPATH)/lib/$(SLIB) ;\
    echo "Install $(APP) and $(SLIB) into $(INSTPATH)"

uninstall:
    -@rm -f $(INSTPATH)/bin/$(APP)
    -@rm -f $(INSTPATH)/lib/$(SLIB)
    @echo "Removed $(APP) and $(SLIB) from $(INSTPATH)"

After the compilation procedure completes, there will be binary, object files, sources, in the library in the subdirectoies of Project:

tree:

Project
├── bin
│   └── app.x
├── configure
├── include
│   └── Scalar_Product.h
├── lib
│   └── libScalar_Product.a
├── Makefile
├── src1
│   ├── app.x
│   ├── libScalar_Product.a
│   ├── main.c
│   ├── main.o
│   ├── Makefile
│   ├── Scalar_Product.c
│   └── Scalar_Product.o
├── src2
└── src3

10.19. Environment variables for executables and libraries¶

Executable app.x was installed in the convenbtional directory for binaries, /usr/local/bin. If it was installed, for example, in unconventional directory, say, /usr/local/apps/bin, then the directory needs to be added to the PATH environment variable:

export PATH=$PATH:/usr/local/apps/bin

The installation also copies the library into /usr/local/lib. If shared object libraries are needed by an executable, they need to be either located in the conventional directories, or one should add the path to LD_LIBRARY_PATH environment variable:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib

Alternatively, the library path can be added in a new config file in directory /etc/ld.so.conf.d.

If compiled software contains brings MAN pages, the path to the pages should be added to environment variable MANPATH:

export MANPATH=$MANPATH:/usr/local/man

Alternatively, the new man pages can be added in file /etc/manpath.config.

10.20. References on libraries and Makefiles¶

The textbook: Static libraries and Makefiles

The textbook: Shared libraries

Makefiles in Linux: An Overview

Linux System Administration

Application compilation

Contents

10. Application compilation¶

10.1. Compilation stages¶

10.2. Compilation in stages (Exercise)¶

10.3. Source code with multiple files¶

10.4. Static libraries¶

10.5. Static library content and compilation (Exerecise)¶

10.6. Shared libraries¶

10.7. Compilation with a shared library (Exercise)¶

10.8. Single code compilation and libraries (Exercise)¶

10.9. Software builders¶

10.10. make and Makefile¶

10.11. Makefile with macros¶

10.12. Makefile with phony targets¶

10.13. Makefile with target `all`¶

10.14. Makefile with suffix rules¶

10.15. Makefile with suffix rules and object file dependencies¶

10.16. Makefiles (Exercises)¶

10.17. Multiple Makefiles. Software installation. (Exercises)¶

10.18. Makefiles in the project¶

10.19. Environment variables for executables and libraries¶

10.20. References on libraries and Makefiles¶

Linux System Administration

Application compilation

Contents

10. Application compilation¶

10.1. Compilation stages¶

10.2. Compilation in stages (Exercise)¶

10.3. Source code with multiple files¶

10.4. Static libraries¶

10.5. Static library content and compilation (Exerecise)¶

10.6. Shared libraries¶

10.7. Compilation with a shared library (Exercise)¶

10.8. Single code compilation and libraries (Exercise)¶

10.9. Software builders¶

10.10. make and Makefile¶

10.11. Makefile with macros¶

10.12. Makefile with phony targets¶

10.13. Makefile with target all¶

10.14. Makefile with suffix rules¶

10.15. Makefile with suffix rules and object file dependencies¶

10.16. Makefiles (Exercises)¶

10.17. Multiple Makefiles. Software installation. (Exercises)¶

10.18. Makefiles in the project¶

10.19. Environment variables for executables and libraries¶

10.20. References on libraries and Makefiles¶

10.13. Makefile with target `all`¶