Spack and EasyBuild: Reproducible HPC Software Stacks Without Losing Your Sanity

There are two ways to manage scientific software on a cluster. The first is to install each package by hand: download, configure, make, make install, pray that ./configure --prefix=/opt/fftw/3.3.10-gcc13 finds the right MPI, curse when it did not, rebuild, write a module file, forget to document it, and six months later have no idea why netcdf-fortran is linked against a different HDF5 than netcdf-c. The second is to use a tool that treats every build as a reproducible recipe and automatically generates coherent toolchain combinations. The two dominant tools for the second approach are Spack and EasyBuild.

Both solve the same problem, but with different philosophies, different community cultures, and different strengths. This post covers what each one does well, the mental models you need to use them effectively, and — by the end — enough to decide which one fits your site (or whether you need both).

The problem in one paragraph

On an HPC cluster, you need twelve versions of FFTW because one user wants it built with GCC 9 + OpenMPI 4 + AVX2, another wants GCC 13 + OpenMPI 5 + AVX-512, a third wants Intel OneAPI + Intel MPI + MKL-backed, and a fourth wants all of the above but compiled with -march=znver4 for the new AMD nodes. Each of these produces a different binary, a different installation prefix, and a different module file, and — crucially — any downstream package (HDF5, NetCDF, PETSc) has to be built against the specific combination of dependencies it will actually use at runtime. Doing this by hand, for hundreds of packages and dozens of toolchains, is madness. Spack and EasyBuild are the tools that make it not madness.

Spack

Spack (“Supercomputer PACKage manager”) came out of Lawrence Livermore around 2013 and has become the most widely adopted HPC package manager. It is a Python program that describes packages as Python classes (called “recipes”) with explicit dependencies, variants, and build logic.

The mental model: specs

The core abstraction in Spack is a spec — a constraint that describes the software you want. Specs are compositional:

1
2
3
4
5

spack install hdf5
spack install hdf5@1.14.3
spack install hdf5@1.14.3 %gcc@13.2.0
spack install hdf5@1.14.3 %gcc@13.2.0 +mpi ^openmpi@5.0.1
spack install hdf5@1.14.3 %gcc@13.2.0 +mpi ^openmpi@5.0.1 +fortran +cxx target=zen4

Read the sigils:

• @ — version (hdf5@1.14.3).
• % — compiler (%gcc@13.2.0).
• +/~ — variant on/off (+mpi, ~shared).
• ^ — dependency constraint (^openmpi@5.0.1 means “the openmpi dependency must be this version”).
• target= — target architecture.
• os= — operating system.
• cflags="...", cppflags=..., ldflags=... — pass compiler flags.

The whole thing is called a “constraint” — Spack’s solver (called clingo-based, and genuinely a SAT solver) figures out the concrete versions of every transitive dependency that satisfy the constraint. You can ask for what you want; Spack figures out how to build it.

Concretization

Before Spack installs anything, it concretizes your spec — picks concrete versions for everything:

1

spack spec -I hdf5 +mpi ^openmpi

This prints a tree showing every dependency, its version, its variants, and whether it is already installed ([+]) or would need to be built. Read this every time before hitting install. If the solver chose something you did not expect, fix it now — either by adding more constraints to your spec or by editing packages.yaml to bias preferences.

Installation and unique hashes

Every installed package lives in a prefix named after its hash, which encodes the entire build configuration:

/opt/spack/opt/spack/linux-ubuntu22.04-zen4/gcc-13.2.0/hdf5-1.14.3-q7f8hjvzb5t3...

This is intentional: two different builds of the same HDF5 get two different prefixes, do not collide, and each downstream dependency links against the specific build it was built for. Your /opt/software directory becomes a content-addressable store. This is what makes Spack reproducible.

Environments

Environments are Spack’s answer to “how do I manage a coherent software stack?” An environment is a named, isolated set of specs:

1
2
3
4
5
6
7
8
9

spack env create mystack
spack env activate mystack

spack add gcc@13.2.0
spack add openmpi@5.0.1 %gcc@13.2.0
spack add hdf5@1.14.3 %gcc@13.2.0 ^openmpi@5.0.1
spack add petsc@3.20.5 %gcc@13.2.0 ^openmpi@5.0.1

spack install

You can also define environments declaratively in spack.yaml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

spack:
  specs:
    - gcc@13.2.0
    - openmpi@5.0.1 %gcc@13.2.0
    - hdf5@1.14.3 %gcc@13.2.0 ^openmpi@5.0.1 +fortran +cxx
    - petsc@3.20.5 %gcc@13.2.0 ^openmpi@5.0.1

  compilers:
    - compiler:
        spec: gcc@13.2.0
        paths:
          cc:  /opt/gcc/13.2.0/bin/gcc
          cxx: /opt/gcc/13.2.0/bin/g++
          f77: /opt/gcc/13.2.0/bin/gfortran
          fc:  /opt/gcc/13.2.0/bin/gfortran
        operating_system: ubuntu22.04
        target: x86_64

  concretizer:
    unify: true   # force a single coherent resolution across all specs

  view: /opt/view/mystack

  modules:
    default:
      enable: [lmod]
      lmod:
        hierarchy: [mpi]
        hash_length: 0

This file is the entire software stack. Commit it to git. Anyone — you next year, a new admin, a colleague at another site — can spack env activate . in the directory and spack install to reproduce the same stack.

The unify: true flag is important: it tells the concretizer that all specs in the environment must share a single coherent dependency graph. Without it, you can end up with two versions of the same library in the environment, which defeats the point.

Views: a flat directory for users

By default, every Spack package is in its own hash prefix. This is great for correctness, terrible for PATH. A view is a merged, symlinked tree:

1
2
3
4
5

view:
  default:
    root: /opt/view/mystack
    link_type: symlink
    select: ['^mpi']   # only include MPI-linked packages in this view

Users add /opt/view/mystack/bin to PATH and see everything. For multiple views per environment (one for MPI builds, one for serial builds, one for debug), use named views.

Chaining and upstream instances

Spack instances can chain to each other — a user or group Spack can use packages already installed in a central Spack without rebuilding them:

1
2
3
4

# ~/.spack/upstreams.yaml
upstreams:
  system:
    install_tree: /opt/spack/opt/spack

Now when the user runs spack install hdf5, if HDF5 with a matching spec exists in the system instance, they get it. Only what is missing gets built locally. This is how a site provides a base stack while letting users extend it with their own variants without duplicating terabytes of build output.

Build caches

Rebuilding GCC or PETSc takes an hour or more. Spack build caches store pre-built binary packages that any instance can pull:

1
2
3
4
5

spack mirror add mycache /shared/spack-cache
spack buildcache push mycache /opt/spack/opt/spack/linux-.../hdf5-...
# On another machine:
spack buildcache update-index mycache
spack install hdf5 ...           # pulls from cache instead of building

The E4S project and the Spack buildcache service now provide large upstream caches for common HPC stacks — check before you build from source.

Recipes

A Spack recipe is a Python class:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

# var/spack/repos/builtin/packages/myapp/package.py
from spack.package import *

class Myapp(CMakePackage):
    homepage = "https://example.org/myapp"
    git      = "https://github.com/example/myapp.git"
    url      = "https://example.org/myapp/releases/1.0.tar.gz"

    version("1.0", sha256="...")
    version("main", branch="main")

    variant("mpi",    default=True,  description="Build with MPI support")
    variant("openmp", default=False, description="Build with OpenMP support")

    depends_on("cmake@3.20:", type="build")
    depends_on("mpi",  when="+mpi")
    depends_on("hdf5 +mpi", when="+mpi")
    depends_on("hdf5 ~mpi", when="~mpi")

    def cmake_args(self):
        return [
            self.define_from_variant("MYAPP_ENABLE_MPI",    "mpi"),
            self.define_from_variant("MYAPP_ENABLE_OPENMP", "openmp"),
        ]

Recipes are small, readable Python. You can add site-specific recipes in a local repo and keep them out of the upstream tree:

1
2
3

spack repo create ~/spack-repo
spack repo add ~/spack-repo
# now ~/spack-repo/packages/myapp/package.py is discoverable

When Spack hurts

• Concretization can be slow for large environments (minutes, sometimes). Improving, but still real.
• Dependency conflicts sometimes require you to understand why the solver chose a path you did not want. spack spec -I ... and spack config get concretizer are your friends.
• Rolling the tree forward (upgrading Spack itself, then reconcretizing environments) occasionally breaks recipes. Pin Spack versions for production stacks. spack clone a specific commit rather than tracking develop.
• Long compile times — GCC from source is ~45 minutes, LLVM is hours, PETSc pulls a dozen dependencies. Use build caches aggressively.

EasyBuild

EasyBuild came out of Ghent University around 2011 and is especially popular in European HPC centers. It is a Python framework that drives builds via easyconfig files — small, declarative descriptions of a build — and easyblock classes that implement the actual build steps.

The mental model: easyconfigs + toolchains

EasyBuild’s key abstraction is the toolchain. A toolchain is a named, versioned combination of compiler + MPI + math libraries:

• foss-2023b: gcc 13.2.0 + openmpi 4.1.6 + openblas + fftw + scalapack.
• intel-2023b: Intel OneAPI + Intel MPI + MKL.
• gompi-2023b: gcc + openmpi without the math stack.

An easyconfig is a Python dict:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

# HDF5-1.14.3-gompi-2023b.eb
easyblock = 'ConfigureMake'

name     = 'HDF5'
version  = '1.14.3'

homepage = 'https://portal.hdfgroup.org/display/support'
description = "HDF5 is a data model, library, and file format."

toolchain = {'name': 'gompi', 'version': '2023b'}
toolchainopts = {'pic': True, 'usempi': True}

source_urls = ['https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-%(version_major_minor)s/hdf5-%(version)s/src/']
sources = [SOURCE_TAR_GZ]
checksums = ['abc123...']

dependencies = [
    ('zlib',   '1.2.13'),
    ('Szip',   '2.1.1'),
]

configopts = '--with-zlib=$EBROOTZLIB --with-szlib=$EBROOTSZIP '
configopts += '--enable-fortran --enable-cxx --enable-parallel'

sanity_check_paths = {
    'files': ['bin/h5cc', 'bin/h5fc', 'lib/libhdf5.a'],
    'dirs':  ['include'],
}

moduleclass = 'data'

Build it:

1

eb HDF5-1.14.3-gompi-2023b.eb --robot

--robot tells EasyBuild to follow dependencies — it walks the dependency graph and builds everything the target needs, in the right order.

Where EasyBuild wins

• Enormous catalog of easyconfigs. The upstream easyconfigs repo has thousands of tested, versioned recipes — particularly strong in bioinformatics, molecular dynamics, and the life sciences. For many European HPC sites, “has an easyconfig” is a necessary condition for supporting a package.
• Toolchains are explicit and named. When a user reports a problem with “HDF5 in the foss-2023b stack,” both of you know exactly what that means. Spack’s naming is more implicit.
• Minimal editing to reproduce. You copy HDF5-1.14.3-gompi-2023b.eb, change a version, run eb --robot. For small tweaks this is faster than editing a Spack recipe.
• Built-in reproducibility through toolchain versions. Every six months a new toolchain vintage (-2024a, -2024b) is released; all easyconfigs for that vintage share the same base. You know exactly what foss-2024a resolves to because it is documented.

Environments

EasyBuild does not have environments in the Spack sense; it relies on Lmod modules (which it generates automatically) for runtime selection. That is philosophically cleaner — the module system does what it is designed to do, and EasyBuild just produces module files.

You install a set of packages for a toolchain:

1
2

eb --robot foss-2023b.eb
eb --robot HDF5-1.14.3-gompi-2023b.eb Boost-1.83.0-GCC-13.2.0.eb PETSc-3.20.5-foss-2023b.eb

And users load them:

1
2
3

module load foss/2023b
module load HDF5/1.14.3
module load PETSc/3.20.5

The hierarchical module structure (see the Lmod post) matches cleanly.

Extensibility

EasyBuild’s build logic lives in easyblocks — Python classes that handle common build systems (ConfigureMake, CMakeMake, PythonPackage, PythonBundle, Binary, etc.) and package-specific ones where standard logic does not work. You write an easyblock only when an easyconfig with an existing easyblock is insufficient; most packages use stock easyblocks.

When EasyBuild hurts

• Less flexible than Spack’s constraint language. Mixing toolchain pieces (GCC 13 with an OpenBLAS from a different toolchain) is harder to express.
• Toolchain vintages lock you in. If you need a GCC version between releases, you either wait, override, or start managing custom toolchains.
• Slower adoption of bleeding-edge packages outside the mainstream set. Spack’s recipe count is larger and moves faster for niche tools.
• No built-in concept of environments / views — you are fully dependent on Lmod for composition.

Head-to-head: pick one (or both)

It is genuinely normal to use both: Spack for the bleeding-edge and custom pieces, EasyBuild for the stable scientific applications users actually run. The two can coexist on the same Lmod tree, writing modules to different subdirectories of MODULEPATH.

A practical build example

Let me walk through building a small stack — GCC 13 + OpenMPI 5 + HDF5 + PETSc — in each, end to end.

Spack version

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

# Install Spack
git clone -c feature.manyFiles=true https://github.com/spack/spack.git /opt/spack
. /opt/spack/share/spack/setup-env.sh

# Find existing compilers (picks up system GCC, icx, etc.)
spack compiler find

# Bootstrap: build a new GCC with the system compiler
spack install gcc@13.2.0
spack load gcc@13.2.0
spack compiler find         # register the newly built GCC

# Set up an environment
mkdir -p /opt/stacks/sci
cd /opt/stacks/sci
cat > spack.yaml <<'EOF'
spack:
  specs:
    - openmpi@5.0.1 %gcc@13.2.0 fabrics=ucx schedulers=slurm
    - hdf5@1.14.3   %gcc@13.2.0 +fortran +cxx +mpi ^openmpi@5.0.1
    - petsc@3.20.5  %gcc@13.2.0 +hdf5 +mumps +scalapack ^openmpi@5.0.1

  concretizer:
    unify: true

  view: /opt/view/sci

  modules:
    default:
      enable: [lmod]
      lmod:
        hierarchy: [mpi]
        hash_length: 0
        core_compilers: [gcc@13.2.0]

  config:
    install_tree:
      root: /opt/software/spack
    build_jobs: 32
EOF

spack env activate .
spack concretize -f
spack install
spack module lmod refresh --delete-tree -y

# Add generated modules to MODULEPATH
echo 'module use /opt/software/spack/modules/lmod/Core' >> /etc/profile.d/spack.sh

Three hours later, users have GCC 13, OpenMPI 5, HDF5, and PETSc with clean Lmod modules.

EasyBuild version

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# Install EasyBuild via pip (or via easybuild's bootstrap script)
python3 -m pip install --user easybuild

# Configure EasyBuild
export EASYBUILD_PREFIX=/opt/easybuild
export EASYBUILD_MODULES_TOOL=Lmod
export EASYBUILD_MODULE_NAMING_SCHEME=HierarchicalMNS

# Look at what toolchain provides what we want
eb --search HDF5-1.14.3
eb --search PETSc-3.20.5

# Build the whole dependency chain (--robot walks it)
eb HDF5-1.14.3-gompi-2023b.eb PETSc-3.20.5-foss-2023b.eb --robot

# Add to MODULEPATH
echo 'module use /opt/easybuild/modules/all' >> /etc/profile.d/easybuild.sh

Three hours later, same outcome, different module names (HDF5/1.14.3-gompi-2023b instead of hdf5/1.14.3-...). Users do module load foss/2023b HDF5 PETSc.

Reproducibility

If somebody can reproduce your stack six months or six years from now, you have won. Both tools provide real reproducibility, but you have to use them deliberately:

Spack

• Pin Spack itself. In your environment, clone a specific Spack commit into a subdirectory (or reference it via spack clone).
• Commit spack.yaml and the generated spack.lock to git. spack.lock records exact concrete specs including dependency hashes.
• Pin compilers by adding them explicitly to compilers.yaml with exact paths; do not rely on spack compiler find discovering whatever is on PATH.
• Mirror tarballs to an internal location using spack mirror create; do not rely on upstream URLs being alive in 2032.
• Freeze build-cache binaries and keep them alongside the env. If you can pull the binaries, you do not need to rebuild.

EasyBuild

• Pin the EasyBuild version: pip install easybuild==4.9.0. Newer easybuild occasionally reinterprets easyconfigs slightly.
• Pin the easyconfigs repo to a specific commit. git clone --branch v4.9.0 or use tags.
• Copy the exact easyconfigs you used into your site repo — do not rely on the upstream project still having HDF5-1.14.3-gompi-2023b.eb available unchanged.
• Use --fetch separately to pre-download all sources, archive them, and feed them from a local mirror.

Common failure modes and fixes

Compilers Spack found are not the ones you want. Edit ~/.spack/packages.yaml:

1
2
3
4
5

packages:
  all:
    compiler: [gcc@13.2.0, gcc@12.3.0]
  openmpi:
    variants: +legacylaunchers fabrics=ucx schedulers=slurm

This sets site-wide defaults and avoids having to specify them in every spec.

Concretization picks the “wrong” dependency version. Tighten the constraint. hdf5 ^zlib@1.3 forces zlib 1.3. Or preference: in packages.yaml,

1
2
3

packages:
  zlib:
    version: [1.3]

EasyBuild cannot find a CUDA installation. CUDA is not typically rebuilt by EasyBuild — instead, point EasyBuild at an existing install:

1
2
3
4

# CUDA.eb: use externals
builddependencies = [('CUDA', '12.3.2')]
# and configure:
# EB_CUDA_ROOT=/opt/cuda/12.3.2

EasyBuild build fails in sanity_check. Sanity checks verify that expected files and modules exist after install. If a variant changes the installed file set, update the sanity_check_paths dict in your easyconfig. This catches silent install failures — do not disable it.

Spack installs the same package twice. Probably because unify: false (the default used to be when_possible; now prefer true). Or you have two specs with conflicting constraints. Re-concretize with unify: true.

Build caches are not being used. spack install uses a cache only if the spec hash matches exactly — any spec difference (different GCC, different microarch) invalidates the cache entry. Narrow your target:

1

spack config add "packages:all:target:[zen4]"

so that local builds match the cached target.

A few more patterns worth stealing

Per-group environments. Each research group has its own spack.yaml chained to the site instance via upstream. They get their funky dependencies without polluting the shared tree and without rebuilding everything.

Container + Spack. Use Spack inside Docker/Apptainer for strictly-reproducible builds. spack containerize will emit a Dockerfile or Apptainer definition from an environment.

EasyBuild in CI. Run eb --robot --dry-run-short in CI for every PR that touches easyconfigs. Actually building is slower than most CI allows, but the dry-run catches typos, missing dependencies, and toolchain misalignments quickly.

Module naming schemes. Pick one early:

• Spack: hash_length: 0 (drop hashes from module names) for cleaner user-facing names.
• EasyBuild: HierarchicalMNS gives you the compiler + MPI hierarchy from the Lmod post; otherwise EasyBuildMNS gives you flat names with toolchain suffixes.

Keep old installs. Disk is cheap; reproducible research is not. Do not delete gcc@9.3.0 just because it is “old.” Mark it with Lmod properties (e.g., arch=legacy) and leave it alone.

Wrapping up

Spack and EasyBuild both solve the same problem: stop building HPC software by hand. They have different personalities — Spack is flexible, expressive, and moves fast; EasyBuild is declarative, toolchain-centric, and plays well with stable institutional release schedules. Most sites end up happy with one or the other; a few sites thoughtfully use both.

Whichever you pick, the value comes from committing to it — writing recipes, defining environments, mirroring sources, pinning versions, publishing build caches. The tool does not give you reproducible software; the discipline does. The tool just makes the discipline practical.