LUNAROPS · OPERATIONAL UPLINK 100% UPTIME 1,247d POSTS 893 JEFF.MOON@LUNAROPS.DEV UTC --:--:--

Environment Modules with Lmod: How `module load` Actually Works

13 min read The Physics of Computing
hpclmodenvironment-moduleslualinuxsoftware-managementshell
Contents

Every HPC user has typed module load gcc/13.2.0 openmpi/5.0.1 and watched their compiler, MPI library, and a dozen environment variables magically rearrange themselves. It looks like black magic the first time. It is not. It is a reasonably thin Lua script manipulating shell environment variables — and once you understand the mechanism, you can build a software tree that serves hundreds of users with conflicting toolchain requirements and no filesystem collisions.

This post is about Lmod, the Lua-based environment modules system that has largely replaced the older TCL Environment Modules on HPC clusters. We will cover how module loading actually manipulates your shell, how hierarchical modules let you express “OpenMPI built against this GCC version,” how to build a maintainable module tree, how collections and restoring sessions works, and the pitfalls that every site hits.

Why environment modules exist

Software on an HPC cluster has a hard problem to solve. You have:

  • Multiple compilers (GCC 9, 11, 13, Intel Classic, Intel OneAPI, NVIDIA HPC, AMD AOCC).
  • Multiple MPI implementations (OpenMPI 4.x, 5.x, MPICH, Intel MPI, MVAPICH2) each needing to be built against each compiler.
  • Multiple scientific libraries (FFTW, HDF5, NetCDF, PETSc, Trilinos) each needing to be built against each compiler + MPI combination.
  • Multiple versions of each, because one research group wants GCC 9 for reproducibility and another wants GCC 13 for C++23 features.

If you install all of these into /usr/local, paths collide, LD_LIBRARY_PATH becomes a monster, and users cannot pick a coherent toolchain. The solution is to install everything in isolated prefixes and provide a mechanism to set PATH, LD_LIBRARY_PATH, MANPATH, PKG_CONFIG_PATH, and CPATH for a chosen combination on demand. That mechanism is environment modules.

The mechanism: how module load actually works

module is a shell function (or alias) — not a binary. When you run module load gcc/13.2.0, here is the sequence:

  1. The module function calls the Lmod executable (lmod in Lua, typically /usr/share/lmod/lmod/libexec/lmod), passing it the action (load), the arguments, and the current shell type.
  2. Lmod finds the module file — typically .lua or TCL — in one of the directories in MODULEPATH.
  3. Lmod executes the module file in a sandboxed Lua environment, where functions like prepend_path, setenv, and load are defined. The module file does not itself modify the shell; it tells Lmod what it wants to do.
  4. Lmod collects all the requested changes (push /opt/gcc/13.2.0/bin onto PATH, set CC=gcc, etc.) and emits shell-specific code (e.g., export PATH=/opt/gcc/13.2.0/bin:$PATH; export CC=gcc;) to stdout.
  5. The module shell function evaluates that output. That is the critical step — Lmod cannot modify your shell directly; it can only print export statements that your shell evaluates.

You can see this yourself:

1
2
3
4
5

module --redirect load gcc/13.2.0
# Prints the shell code that would be evaluated

# Or tell Lmod exactly what shell to emit:
lmod bash load gcc/13.2.0

Unload reverses the process. Lmod tracks which paths were prepended by which modules (in _ModuleTable_ inside LOADEDMODULES bookkeeping) and removes exactly those entries. This is why you should never manually edit PATH to add directories a module owns — the unload will not find them to remove.

A minimal module file

Here is a real, minimal Lua module file for GCC 13.2.0:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

-- /opt/modulefiles/Core/gcc/13.2.0.lua

help([[
GCC 13.2.0 — GNU Compiler Collection
]])

whatis("Name:        gcc")
whatis("Version:     13.2.0")
whatis("Description: GNU Compiler Collection (C, C++, Fortran)")

local root = "/opt/software/gcc/13.2.0"

prepend_path("PATH",            pathJoin(root, "bin"))
prepend_path("LD_LIBRARY_PATH", pathJoin(root, "lib64"))
prepend_path("LD_LIBRARY_PATH", pathJoin(root, "lib"))
prepend_path("MANPATH",         pathJoin(root, "share/man"))
prepend_path("INFOPATH",        pathJoin(root, "share/info"))
prepend_path("PKG_CONFIG_PATH", pathJoin(root, "lib64/pkgconfig"))

setenv("CC",  "gcc")
setenv("CXX", "g++")
setenv("FC",  "gfortran")
setenv("F77", "gfortran")
setenv("F90", "gfortran")

family("compiler")

Functions worth knowing:

  • prepend_path("VAR", "value") — prepend to a colon-separated path, avoiding duplicates.
  • append_path — append instead.
  • remove_path — opposite of prepend.
  • setenv("VAR", "value") — set a variable outright.
  • unsetenv("VAR") — remove.
  • pushenv("VAR", "value") — set a variable but remember the previous value; popenv restores it.
  • load("otherModule") — load another module as a dependency.
  • depends_on("otherModule") — load if not loaded; track as a dependency. If the module is also depends_on by several other modules, it does not unload until the last of them unloads. This is the dependency-counting feature that makes clean module trees possible.
  • conflict("openmpi") — prevent loading if another module conflicts.
  • family("compiler") — any two modules with the same family are mutually exclusive; loading a second one auto-unloads the first. Standard families: compiler, mpi, blas.
  • prereq("gcc"), prereq_any("gcc", "intel") — require that another module is loaded first.
  • help(...) — shown by module help <name>.
  • whatis(...) — indexed for module whatis and module spider.

The MODULEPATH

Lmod finds modules by searching directories in MODULEPATH. A minimal site looks like:

/opt/modulefiles/Core

Inside Core, each module is a directory containing version files:

/opt/modulefiles/Core/
├── gcc/
│   ├── 11.4.0.lua
│   ├── 12.3.0.lua
│   ├── 13.2.0.lua
│   └── .modulerc.lua       -- optional: aliases, defaults
├── python/
│   ├── 3.11.6.lua
│   └── 3.12.1.lua
└── cmake/
    ├── 3.27.8.lua
    └── 3.28.3.lua

.modulerc.lua in a module directory sets defaults and aliases:

1
2
3

-- /opt/modulefiles/Core/gcc/.modulerc.lua
module_version("gcc/13.2.0", "default")
module_version("gcc/13.2.0", "latest")

Now module load gcc resolves to gcc/13.2.0.

Hierarchical modules: the killer feature

Here is the problem flat modules do not solve well: users expect to load gcc, then openmpi, and get an OpenMPI built against that GCC. If OpenMPI is flat, you have to name modules openmpi/5.0.1-gcc13.2.0, openmpi/5.0.1-gcc12.3.0, openmpi/5.0.1-intel2024.0 — and users have to remember to match versions.

Hierarchical modules solve this by stacking MODULEPATH entries when you load a compiler. Structure:

/opt/modulefiles/
├── Core/                         # Compilers, compiler-independent tools
│   ├── gcc/13.2.0.lua
│   ├── intel/2024.0.0.lua
│   └── cmake/3.28.3.lua
├── Compiler/                     # Built against a specific compiler
│   ├── gcc/13.2.0/
│   │   ├── openmpi/5.0.1.lua
│   │   ├── openblas/0.3.26.lua
│   │   └── fftw/3.3.10.lua
│   └── intel/2024.0.0/
│       └── openmpi/5.0.1.lua
└── MPI/                          # Built against a specific compiler + MPI
    ├── gcc/13.2.0/openmpi/5.0.1/
    │   ├── hdf5/1.14.3.lua
    │   ├── netcdf-c/4.9.2.lua
    │   └── petsc/3.20.5.lua
    └── intel/2024.0.0/openmpi/5.0.1/
        └── hdf5/1.14.3.lua

The key: when gcc/13.2.0 is loaded, it calls prepend_path("MODULEPATH", "/opt/modulefiles/Compiler/gcc/13.2.0") — which makes the compiler-specific modules visible. When openmpi/5.0.1 is loaded (from that directory), it prepends /opt/modulefiles/MPI/gcc/13.2.0/openmpi/5.0.1.

The compiler module:

1
2
3
4
5
6
7
8
9

-- /opt/modulefiles/Core/gcc/13.2.0.lua
local root = "/opt/software/gcc/13.2.0"
prepend_path("PATH",            pathJoin(root, "bin"))
-- ... other paths ...

-- Make compiler-built modules visible
prepend_path("MODULEPATH", "/opt/modulefiles/Compiler/gcc/13.2.0")

family("compiler")

The MPI module:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

-- /opt/modulefiles/Compiler/gcc/13.2.0/openmpi/5.0.1.lua
local root = "/opt/software/gcc/13.2.0/openmpi/5.0.1"
prepend_path("PATH",            pathJoin(root, "bin"))
prepend_path("LD_LIBRARY_PATH", pathJoin(root, "lib"))

prepend_path("MODULEPATH", "/opt/modulefiles/MPI/gcc/13.2.0/openmpi/5.0.1")

setenv("MPI_HOME", root)
setenv("OMPI_MCA_btl", "^openib,tcp")  -- UCX instead; example tuning

family("mpi")

Now users do:

1
2
3

module load gcc/13.2.0
module load openmpi/5.0.1
module load hdf5

And automatically get GCC 13.2.0-built OpenMPI and an HDF5 built for that GCC+OpenMPI combination. Switch compilers:

1

module swap gcc intel/2024.0.0

Lmod notices that the compiler family changed, unloads everything that was loaded from below the old gcc/13.2.0 MODULEPATH, then tries to reload everything with the new compiler’s modules. If a replacement exists, it loads it; if not, it warns you. This is one of Lmod’s strongest features — the TCL environment modules do not do this.

module spider: the documentation users actually use

module avail shows what is loadable given the current MODULEPATH. In a hierarchical setup this is insufficient — HDF5 is not “available” until you load a compiler and MPI. module spider walks the whole tree:

1
2
3

module spider                       # show every module
module spider hdf5                  # show all hdf5 versions
module spider hdf5/1.14.3           # show how to load this exact version

The last one is gold. It prints:

You will need to load all module(s) on any one of the lines below
before the "hdf5/1.14.3" module is available to load.

      gcc/13.2.0  openmpi/5.0.1
      intel/2024.0.0  openmpi/5.0.1
      intel/2024.0.0  intel-mpi/2021.11

This is the “what do I need to load?” answer users want, and it comes for free from the hierarchy.

Collections: save your working environment

Users often build up a specific combination of modules they use daily. Collections let them save it:

1
2
3
4
5

module load gcc/13.2.0 openmpi/5.0.1 cmake hdf5 python/3.12.1
module save mydev

# Later, clean shell:
module restore mydev

module save (no name) saves as default. Restoring the default happens automatically at login if LMOD_SYSTEM_DEFAULT_MODULES does not override it.

Collections live in ~/.lmod.d/ as plain text files. Committing them to a dotfile repo works well.

User-level modules

Users can add their own modules:

1
2
3
4
5
6
7
8
9

mkdir -p ~/privatemodules/myapp
cat > ~/privatemodules/myapp/1.0.lua <<'EOF'
local root = os.getenv("HOME") .. "/builds/myapp/1.0"
prepend_path("PATH",            pathJoin(root, "bin"))
prepend_path("LD_LIBRARY_PATH", pathJoin(root, "lib"))
EOF

module use ~/privatemodules
module load myapp/1.0

module use PATH prepends to MODULEPATH. module use -a PATH appends. module unuse PATH removes.

A common pattern: team shared modules live at /shared/<group>/modulefiles, and a group login script does module use /shared/<group>/modulefiles automatically.

Properties and colored display

Properties tag modules with attributes. Users filter by them. Common:

1
2
3
4
5

-- In a GPU-enabled module:
add_property("arch", "gpu")

-- In an experimental build:
add_property("state", "testing")

Colors are configured in /etc/lmod/lmodrc.lua:

1
2
3
4
5
6
7
8

propT = {
    arch = {
        validT = { gpu = 1, cpu = 1 },
        displayT = {
            gpu = { short = "G", color = "green", doc = "Built for GPU" },
        },
    },
}

Now module avail shows a (G) next to GPU modules in green. Small touch, but users notice.

Using Lmod with Spack or EasyBuild

Nobody writes module files by hand at scale. Spack (spack install) and EasyBuild (eb) both generate Lmod-compatible module files automatically:

1
2
3
4
5
6

# Spack
spack install hdf5@1.14.3 %gcc@13.2.0 ^openmpi@5.0.1
spack module lmod refresh

# EasyBuild
eb HDF5-1.14.3-GCC-13.2.0-OpenMPI-5.0.1.eb --robot

Both generate hierarchical module trees by default and the output drops straight into MODULEPATH. If you are building a software tree from scratch today, use one of them — do not hand-write module files for everything. This post covers the mechanism so you can read, debug, and customize what they generate.

Startup performance: the spider cache

With 500 modules in a hierarchy, module avail can get slow — Lmod has to walk directories and parse files. The fix is the spider cache:

1
2

# Rebuild the cache (do this in a cron after installing modules)
$LMOD_DIR/update_lmod_system_cache_files -d /opt/lmod/cache /opt/modulefiles

Lmod will use the cache instead of walking the tree. Users see avail in ~100 ms instead of a few seconds. Run this hourly via cron or after every module tree change.

# /etc/cron.d/lmod-cache
*/15 * * * * root /opt/lmod/update_lmod_system_cache_files -d /opt/lmod/cache /opt/modulefiles

Module naming conventions: choose early, hold forever

Once users have scripts with module load foo/1.2.3 in them, renaming foo breaks everything. Some conventions that have aged well:

  • Lowercase module names. gcc, not GCC. openmpi, not OpenMPI.
  • Version strings match upstream. GCC’s version is 13.2.0, not 13.2. FFTW is 3.3.10. PETSc is 3.20.5. Never abbreviate — 3.2 is ambiguous.
  • Category prefixes are optional and dangerous. bio/samtools/1.19 looks nice until you decide samtools is not really “bio.” Prefer flat naming.
  • Variants in the version, not the name. If you have a debug build: openmpi/5.0.1-debug, not openmpi-debug/5.0.1.
  • Keep old versions around. Papers cite specific software versions. Nuking gcc/9.3.0 because “it is old” will break irreproducible research six months later.

Debugging Lmod

module --debug is your friend. So is LMOD_TRACING=yes:

1

LMOD_TRACING=yes module load hdf5

Prints every step of the resolution. Reveals why a module is found, why it is not found, and which MODULEPATH provided the match.

If a module is not visible that should be:

  1. module use it explicitly — is the MODULEPATH entry there?
  2. module spider — does Lmod see the module at all?
  3. Check the Lua file for syntax errors — lua modulefile.lua will not quite work (Lmod injects its functions), but lmod bash load modulefile.lua with verbosity will flag parse errors.

Common errors:

  • “Lmod has detected the following error” with a Lua traceback — usually a typo in the module file; the line number is accurate.
  • “A conflicting module” — two modules from the same family both want to load. Unload one.
  • Module loads but binaries are not found — double-check the path in prepend_path. Use module show <name> to see exactly what the module is doing.
  • Path not unloaded on module unload — someone manually added an entry to PATH that the module did not own. Only modules should own their paths.

Other things that surprise newcomers

  • module is not available in all shells by default. For login shells, /etc/profile.d/lmod.sh sets up the module function. For non-login non-interactive shells (like a SLURM batch script with #!/bin/bash — not -l), you need to source the init script yourself, or use #!/bin/bash -l, or module use $LMOD_SYSTEM_DEFAULT_MODULES.
  • module load in a script does not survive into child shells unless the child shell re-sources lmod.sh. This is why SLURM job scripts typically start with source /etc/profile.d/lmod.sh (or -l on the shebang) before loading modules.
  • Lmod runs as your user — it has no special privileges. Module files are owned by root typically, but the Lua runs as you, so do not trust unsafe code in user-provided module files on shared systems.
  • module purge unloads everything, then re-loads the sticky modules — those listed in LMOD_SYSTEM_DEFAULT_MODULES. Useful for reset-to-clean workflows. module purge --force skips the sticky list and really unloads everything.
  • Pinning a version: module load gcc/13.2.0 is pinned; module load gcc picks the default. Advise users to pin in reproducible scripts.

A site architecture sketch

If you are building a module system from scratch for a moderately sized cluster, here is a durable starting point:

/opt/software/            # installed software (Spack-managed)
  gcc/13.2.0/...
  intel/2024.0.0/...
  gcc/13.2.0/openmpi/5.0.1/...
  gcc/13.2.0/openmpi/5.0.1/hdf5/1.14.3/...

/opt/modulefiles/         # Lmod MODULEPATH root
  Core/
    gcc/
    intel/
    cmake/
    git/
  Compiler/gcc/13.2.0/
    openmpi/
    openblas/
    fftw/
  Compiler/intel/2024.0.0/
    openmpi/
    intel-mpi/
  MPI/gcc/13.2.0/openmpi/5.0.1/
    hdf5/
    petsc/
  MPI/intel/2024.0.0/openmpi/5.0.1/
    hdf5/

/opt/lmod/cache/          # spider cache

/etc/lmod/lmodrc.lua:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

scDescriptT = {
    {
        ["dir"]       = "/opt/lmod/cache",
        ["timestamp"] = "/opt/lmod/cache/timestamp",
    },
}

propT = {
    arch = {
        validT   = { gpu = 1 },
        displayT = { gpu = { short = "G", color = "green", doc = "GPU build" } },
    },
}

/etc/profile.d/z00_lmod.sh (ordered z00_ so it runs last):

1
2
3
4

export MODULEPATH=/opt/modulefiles/Core:/opt/modulefiles/site
export LMOD_SYSTEM_DEFAULT_MODULES="StdEnv"
source /usr/share/lmod/lmod/init/profile
module refresh

StdEnv is a module users rely on as the baseline — it can do nothing, or it can set sensible defaults (prepend_path("PATH", "/opt/tools/bin")).

Wrapping up

Lmod is small, legible, and surprisingly deep. The basic mechanism — Lua-defined changes emitted as shell code — takes a few minutes to understand. The hierarchical layout, with compiler and MPI families stacking MODULEPATH entries, is the piece that makes a serious cluster maintainable: users think in toolchains, not in ad-hoc version suffixes.

Do not write module files by hand at scale. Let Spack or EasyBuild generate them. But do understand the mechanism, because the day you need to debug why module swap gcc intel does not reload HDF5 the way users expect, or why the spider cache is stale, or why a user’s $PATH has a dangling entry from something they unloaded three hours ago — you will need to know what is happening underneath, and almost none of that is in the module man page.

Comments