Lmod NDSU lab
Contents
Lmod NDSU lab¶
Outline of the presentation¶
User environments in the Unix/Linux shell.
Tcl/C module examples and motivation for the Lua based Lmod modules.
Setup Lmod for the lab exercises.
MODULEPATH and explanation how LMOD works
Practice with basic module commands and examples.
Module lua file structure.
Useful features of LMOD
LMOD usage tracking through syslog
Why to use environment modules¶
| Reason: | Environment variables affected: |
| Different Compilers: gcc 10.2.1, gcc 12.0.0, Intel 2020, etc | PATH, CPATH, LIBRARY_PATH |
| Different apps: shared object libs (*.so), MPI, Java, Licenses, etc | LD_LIBRARY_PATH, MPI_HOME, JAVA_HOME, LM_LICENSE_FILE |
Environment modules allow switching seamlessly between the user environments without need for modifying /etc/profile or .bashrc files.
Classical TCL/C Environment Modules¶
To overcome the difficulty of setting and changing the Environment variables, the TCL/C Modules were introduced over 2 decades ago.
They were written in C and used TCL language for environment configuration.
Very robust and easy to use, for example, to load Intel 2018 compiler:
module load intel/2018
Check what the modules are loaded:
module list
See what the modules are available to load:
module avail
Unload the module:
module unload intel/2018
TCL/C modules drawback and motivation for migration to Lmod¶
The classical TCL/C modules allow conflicting environment modules to be loaded at the same time. For example:
module switch intel/2018 intel/2013
module list
shows:
intel/2013 openmpi/2.1.2-intel2018
Users should watch what modules are loaded before performing compilation and running their codes.
The rule of thumb is, first, to unload all the modules with command
module purge
Then load specifically what is needed, for example:
module load intel/2018
module load openmpi/2.1.2-intel2018
Better approach is switching to Lmod.
The Lmod configuration can protect users from loading incompatible environment settings.
Lmod (“L” stands for Lua) vs TCL/C modules¶
Lmod is an environment modules implementation that provides all of the functionality of TCL/C Environment Modules plus more features.
Lmod can read modules written in TCL and in lua.
Support for hierarchical module file structure.
MODULEPATH is dynamically updated when modules are loaded.
Makes loaded modules inactive and active to provide sane environment.
User module collections.
Hidden modules.
Optional usage tracking.
Many useful functions in Lua module configuration files, such as family(“compiler”), property(“gpu”), etc
LMOD installation steps¶
Install Lua
Install LMOD
Setup the environment configuration in <module_name>.lua files.
Initialize MODULEPATH and MODULEPATH_ROOT entries in .bashrc
Details are here: Installing Lua and Lmod
The Lab cluster¶
LMOD setup on your lab cluster¶
SSH to to the Orbit firewall.
SSH to one of the nodes, lxc01, lxc02, etc.
Install ansible:
sudo apt install ansible
Download a tar ball with Ansible playbook for LMOD
wget https://linuxcourse.rutgers.edu/LCI_2023/Ansible-lmod-ndsu.tgz
tar -zxvf Ansible-lmod-ndsu.tgz
cd Ansible-lmod*
ansible-playbook install_lmod.yml
In .bashrc file of user ccast, remove the settings for openmpi, and add the following block at the bottom:
if [ -z "$BASHRC_READ" ]
then
export BASHRC_READ=1
. /usr/share/lmod/lmod/init/profile
export MODULEPATH_ROOT=/usr/share/lmod/lmod/modulefiles
export MODULEPATH=$MODULEPATH_ROOT/Core
export LMOD_PACKAGE_PATH=/usr/share/lmod/lmod/libexec
fi
Alternatively you can overwrite his .bashrc file as follows:
cp files/bashrc-lmod ~ccast/.bashrc
module avail
LMOD demo with loading and unloading the modules.¶
Check available modules:
module available
Shortcut:
ml av
See all modules via spider:
module spider
Shortcut:
ml spider
Load module gcc/10.2.1
module load gcc/10.2.1
Shortcut
ml gcc/10.2.1
List loaded modules:
module list
Shortcut:
ml
See what modules need to be loaded for application lammps:
module spider lammps
Load the prereq modules:
module load openmpi/4.1.5
See what modules are available now:
ml av
See if there are hidden modules:
ml --show-hidden av
Load
lammps:
ml lammps
Unloading the base module, gcc/10.2.1, will deactivate the others, openmpi/4.1.4 and lammps:
module unload gcc/10.2.1
The dependent modules will also get deactivated if the different compiler is loaded, for example:
module load intel/2018
To unload all the modules:
module purge
How the things work¶
File .bashrc references the Lua modules and initialize MODULEPATH_ROOT and MODULEPATH environment variables:
if [ -z "$BASHRC_READ" ]
then
export BASHRC_READ=1
. /usr/share/lmod/lmod/init/profile
export MODULEPATH_ROOT=/usr/share/lmod/lmod/modulefiles
export MODULEPATH=$MODULEPATH_ROOT/Core
fi
The initial MODULEPATH=modulefiles/Core
After command
module load gcc/10.2.1, the MODULEPATH gets updated:MODULEPATH=modulefiles/Compiler/gcc/10.2.1:modulefiles/CoreAfter adding another module,
module load openmpi/4.1.5, the MODULEPATH contains the directory with filelammps.lua
modulefiles/
|-- Compiler
| |-- gcc
| | \-- 10.2.1
| | \-- openmpi
| | \-- 4.1.5.lua <-- 2 updates:
| \-- intel "MODULEPATH=modulefiles/MPI/gcc/10.2.1/4.1.5:$MODULEPATH"
| |-- 2013
| | \-- openmpi
| | |-- 1.6.5.lua
| | \-- 1.8.8.lua
| \-- 2018
| -- openmpi
| |-- 1.8.8.lua
| |-- 2.1.1.lua
| \-- 2.1.2.lua
|-- Core
| |-- gcc
| | \-- 10.2.1.lua <--- 1 updates:
| \-- intel "MODULEPATH=modulefiles/Compiler/gcc/10.2.1:modulefiles/Core"
| |-- 2013.lua
| \-- 2018.lua
\-- MPI
|-- gcc
| \-- 10.2.1
| \-- 4.1.5
| \-- lammps.lua <-- 3 is in the updated MODULEPATH
\-- intel
\-- 2018
\-- 2.1.2
\-- vasp.lua
Module file example: 10.2.1.lua¶
-- -*- lua -*-
help([[ This is a gcc 10.2.1 compiler, came with Red Hat 8.6 packages]])
-- Local variables
local version = "10.2.1"
-- Whatis description
whatis("Description: GCC compiler")
-- Setup Modulepath for packages built by this compiler
local mroot = os.getenv("MODULEPATH_ROOT")
local mdir = pathJoin(mroot,"Compiler/gcc", version)
prepend_path("MODULEPATH", mdir)
-- Set family for this module
family("compiler")
Helpful LMOD functionalities.¶
Module tags, for example:
add_property("state","testing")
add_property("arch","gpu")
Hide modules when software is updated or decommissioned. In the recent versions or lmod, version 8.x, make the module file hodden:
mv lammps.lua .lammps.lua
In the older versions of lmod, 6.x and 7.x,
place file .modulerc.lua in the same directory with the module, for example, in directory modulefiles/MPI/gcc/10.2.1/4.1.5:
hide_version("lammps")
See what modules need to be loaded for application lammps:
module --show-hidden spider
User module collections can be saved and restored by a user, for example:
module load gcc/10.2.1
module load openmpi/4.1.5
module save gcc-openmpi
It saves the environment configuration into ~/.lmod.d/gcc-openmpi Lua file.
To restore the environment:
module restore gcc-openmpi
One can restore the module collections through .bashrc:
module -q restore gcc-openmpi
Lmod can track environment module usage by users.
Lmod configuration for module usage.¶
Add new path,
/usr/share/lmod/lmod/libexec_log, to LMOD_PACKAGE_PATH into file .bashrc for ccast:
export LMOD_PACKAGE_PATH=/usr/share/lmod/lmod/libexec:/usr/share/lmod/lmod/libexec_log
Create directory
/usr/share/lmod/lmod/libexec_log:
sudo mkdir /usr/share/lmod/lmod/libexec_log
Copy file
SitePackage_LOGS.luaintoSitePackage.luain the new directory:
sudo cp /usr/share/lmod/lmod/libexec/SitePackage_LOGS.lua /usr/share/lmod/lmod/libexec_log/SitePackage.lua
Login to
ccast.
sudo su - ccast
Make sure the updated LMOD_PACKAGE_PATH environment variable is in the user environment.
echo $LMOD_PACKAGE_PATH
Start loading modules as
ccast, for example:
module load gcc/10.2.1 openmpi/4.1.5
Check the log updates with ModuleUsageTracking tag in /var/log/messages
grep ModuleUsageTracking /var/log/messages