Deterministic and Reproducible Builds: Why Your Build Should Be a Pure Function

A build is a function. It takes source code as input and produces artifacts as output. Like any function, the question is: given the same inputs, does it always produce the same output?

For most software projects, the answer is no. Run a build twice on different machines, or on the same machine a week apart, and you’ll likely get different bytes in your output — different embedded timestamps, different tool versions, different ordering of symbols resolved by the linker. Most of the time this doesn’t matter. Sometimes it matters enormously.

This post covers why reproducible builds matter, the specific sources of non-determinism you need to eliminate, and the tools that make hermeticity achievable at scale.

Why Reproducibility Matters

Supply Chain Security

The most compelling reason is security. The SolarWinds attack compromised the build pipeline itself — malicious code was injected during the build process, not into the source repository. The resulting binaries couldn’t be traced back to the legitimate source.

If your build is reproducible, an independent party can take the same source code, run the same build, and verify that the resulting binary is identical to what you shipped. A compromised build pipeline would produce different bytes. This is exactly what the Reproducible Builds project enables for Linux distributions.

Debian, Fedora, NixOS, and Arch Linux all have ongoing efforts to make every package in their repositories reproducible. NixOS leads — over 95% of packages are currently reproducible.

Debugging Across Time

Non-deterministic builds make debugging harder. When a bug appears in production but not in your local build, is it a real difference in behavior, or just a different ordering of symbols in the binary? Reproducible builds remove this variable.

When you can reproduce the exact binary that’s running in production, your debug symbols actually match. Stack traces resolve correctly. Profiling data aligns with source.

Caching

Deterministic builds are a prerequisite for effective build caching. If the same source produces different outputs each time, no cache can help you. Remote build caches (Bazel’s remote cache, Gradle’s build cache) only work if builds are hermetic — every input is declared, and identical inputs always produce identical outputs.

The economic argument: Google’s internal Bazel deployment reportedly reduces engineer-facing build times by over 90% through remote caching. The entire model depends on reproducibility.

Sources of Non-Determinism

Before you can fix non-determinism, you need to know where it comes from.

Timestamps

The most common culprit. Many build tools embed the current time into output artifacts:

• __DATE__ and __TIME__ macros in C/C++
• Java .class files contain timestamps
• Go binaries embed build time
• Archive formats (tar, zip, jar) include file modification times
• Debian packages embed the build date

Fix: SOURCE_DATE_EPOCH is a specification for how build tools should handle this. Set it to a fixed value, and compliant tools will use that timestamp instead of the current time:

1
2
3
4
5
6
7

export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)

# Or pin to a specific date
export SOURCE_DATE_EPOCH=1700000000

# Now build — most tools respect this
make

Major languages and tools that respect SOURCE_DATE_EPOCH:

• GCC, Clang (for __DATE__/__TIME__)
• Go toolchain
• Python (.pyc files)
• Rust (via --remap-path-prefix)
• GNU tar, zip, ar
• Debian packaging tools

File System Ordering

Many build tools iterate over directory entries. File system ordering is not guaranteed to be consistent across different systems or even across runs on the same system.

1
2
3
4

# Non-deterministic: glob ordering is filesystem-dependent
import glob
sources = glob.glob('src/**/*.c', recursive=True)
compile(sorted(sources))  # Fix: always sort

The fix is simple: always sort file lists before processing. This is a common source of non-determinism that’s easy to fix and hard to notice.

Parallel Builds

Parallel builds introduce race conditions in how outputs are assembled. The order in which compilation units finish can affect the final binary if your build system doesn’t handle this carefully.

Bazel handles this correctly by design — its action graph ensures dependencies are respected and outputs are merged deterministically. Make’s parallel mode (make -j) can be non-deterministic if Makefiles have incorrect dependency declarations.

Tool Versions

If your build uses whatever version of gcc, python, or node happens to be on the machine, builds on different machines or at different times will use different tool versions, potentially producing different outputs.

This is the hardest problem to solve without a hermetic build system. A new version of a compiler may generate slightly different code for the same source.

Fix: Pin all tool versions explicitly. This means:

• Docker images with pinned base images (FROM debian:12.5 not FROM debian:latest)
• Lockfiles for language package managers
• Version-pinned toolchains in your build system

Randomness in Build Tools

Some tools generate UUIDs, random padding, or non-deterministic identifiers during builds. Common offenders:

• Debug info (some compilers use build IDs derived from random data)
• Protocol buffer generated code (historically had non-deterministic field ordering)
• Some linkers include random sections

For debug build IDs, use a content-based ID instead of a random one:

1
2
3
4
5

# Clang: use content-based build ID
-Wl,--build-id=sha1

# GCC
-Wl,--build-id=sha1

Path Embedding

Build tools often embed absolute paths into binaries — for debug info (DWARF), coverage profiles, or error messages. The same source built in /home/alice/project and /home/bob/project will produce different binaries.

Fix: Use path remapping:

1
2
3
4
5
6
7
8

# Go: normalize module paths
go build -trimpath ./...

# GCC/Clang: remap paths in debug info
-fdebug-prefix-map=/home/alice/project=.

# Rust: normalize paths
RUSTFLAGS="--remap-path-prefix $HOME=~"

Bazel: Hermetic Builds at Scale

Bazel is Google’s open-source build system, descended from the internal Blaze system. Its central design principle is hermeticity: every build action runs in a sandbox with only explicitly declared inputs available.

The Core Model

In Bazel, you define build targets in BUILD files:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

# BUILD
cc_library(
    name = "mylib",
    srcs = ["mylib.cc"],
    hdrs = ["mylib.h"],
    deps = ["@abseil//absl/strings"],
)

cc_binary(
    name = "myapp",
    srcs = ["main.cc"],
    deps = [":mylib"],
)

py_test(
    name = "mylib_test",
    srcs = ["mylib_test.py"],
    deps = [":mylib_py"],
)

Every target declares its inputs explicitly. Bazel’s sandbox ensures that if a file isn’t declared as an input, the build action can’t see it. This makes “it works on my machine” impossible by construction — if the build succeeds, it’s because all inputs are correctly declared.

Remote Caching

Bazel can cache build outputs in a remote store keyed by a cryptographic hash of all inputs:

# .bazelrc
build --remote_cache=grpcs://cache.example.com
build --remote_executor=grpcs://remote.example.com  # optional: remote execution

# With Google Cloud Storage
build --remote_cache=https://storage.googleapis.com/my-build-cache
build --google_default_credentials

When a build action runs, Bazel computes a cache key from:

• The action’s command line
• The content hash of all input files
• Environment variables explicitly listed as inputs
• The Bazel version and toolchain configuration

If a cache hit exists for that key, the outputs are fetched rather than computed. This works across machines — CI builds populate the cache, developer builds hit it.

Toolchain Configuration

Bazel manages toolchains explicitly. Instead of relying on the system GCC, you declare which compiler to use:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# WORKSPACE or MODULE.bazel
http_archive(
    name = "llvm_toolchain",
    sha256 = "...",
    urls = ["https://github.com/bazel-contrib/toolchains_llvm/releases/download/..."],
)

load("@llvm_toolchain//:toolchain.bzl", "llvm_toolchain")

llvm_toolchain(
    name = "llvm18",
    llvm_version = "18.1.8",
)

With this, all developers and CI runs use exactly the same LLVM 18.1.8, regardless of what’s installed on their system. Bazel downloads it automatically.

Bzlmod: Dependency Management

Modern Bazel (6.0+) uses Bzlmod for external dependency management, with a lockfile (MODULE.bazel.lock) that pins exact versions and content hashes:

1
2
3
4
5
6

# MODULE.bazel
module(name = "myapp", version = "1.0")

bazel_dep(name = "rules_go", version = "0.46.0")
bazel_dep(name = "gazelle", version = "0.35.0")
bazel_dep(name = "com_google_absl", version = "20240116.2")

Run bazel mod deps to generate the lockfile. The lockfile pins content hashes, so builds are reproducible even if upstream repositories change.

Build without Bytes

One of Bazel’s most powerful features: --build_without_the_bytes. Bazel can execute the full build remotely without downloading intermediate artifacts to the local machine — only the final outputs you request are transferred. This makes building large projects on a laptop with a fast remote executor dramatically more efficient.

Nix: Reproducibility Through Purity

Nix takes a different approach. Rather than sandboxing individual build actions, Nix defines each package as a pure function of its inputs. Every package in the Nix store is identified by a cryptographic hash of everything that influenced its build: source code, dependencies, build scripts, and compiler flags.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

# A simple Nix derivation
{ stdenv, fetchurl, zlib }:

stdenv.mkDerivation rec {
  pname = "myapp";
  version = "1.2.3";

  src = fetchurl {
    url = "https://example.com/myapp-${version}.tar.gz";
    sha256 = "sha256-abc123...";  # content-addressed
  };

  buildInputs = [ zlib ];

  buildPhase = ''
    make -j$NIX_BUILD_CORES
  '';

  installPhase = ''
    make install PREFIX=$out
  '';
}

The output is stored at a path like /nix/store/4k8abl9z...-myapp-1.2.3/, where the hash encodes all inputs. Two builds with identical inputs always produce the same hash. If the outputs at that path already exist, Nix doesn’t rebuild — it’s essentially a global build cache.

Nix for Developer Environments

This extends to developer environments. A flake.nix specifies exactly which tools are available in a dev shell:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

# flake.nix
{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.11";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = { nixpkgs, flake-utils, ... }:
    flake-utils.lib.eachDefaultSystem (system:
      let pkgs = nixpkgs.legacyPackages.${system};
      in {
        devShells.default = pkgs.mkShell {
          packages = with pkgs; [
            go_1_23
            golangci-lint
            delve
            gotools
          ];

          # Set SOURCE_DATE_EPOCH for reproducible builds
          shellHook = ''
            export SOURCE_DATE_EPOCH=1700000000
          '';
        };
      });
}

Running nix develop gives every developer the same Go 1.23, same linter, same debugger — regardless of what’s installed on their system. The flake.lock file pins exact revisions of nixpkgs, so the environment is identical across time.

Comparing Nix and Bazel

For most teams: use Nix for developer environments and Bazel (or a language-native tool like Cargo, Gradle) for build incremental compilation. They complement each other.

SLSA: Supply Chain Levels for Software Artifacts

SLSA (Supply-chain Levels for Software Artifacts, pronounced “salsa”) is a framework from Google for assessing and improving build pipeline security. It defines four levels:

Build Provenance

The core artifact is a provenance attestation: a signed document that states “this artifact was built from these sources, by this build system, at this time, producing this hash.”

GitHub Actions can generate SLSA provenance automatically for releases:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

# .github/workflows/release.yml
jobs:
  build:
    outputs:
      hashes: ${{ steps.hash.outputs.hashes }}
    steps:
      - uses: actions/checkout@v4
      - name: Build
        run: |
          go build -trimpath -o myapp ./...
      - name: Generate hashes
        id: hash
        run: |
          echo "hashes=$(sha256sum myapp | base64 -w0)" >> $GITHUB_OUTPUT

  provenance:
    needs: [build]
    permissions:
      actions: read
      id-token: write
      contents: write
    uses: slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v2.0.0
    with:
      base64-subjects: "${{ needs.build.outputs.hashes }}"
      upload-assets: true

This generates a multiple.intoto.jsonl attestation file alongside your release artifacts. Users can verify it:

1
2
3
4
5
6
7
8

# Install slsa-verifier
go install github.com/slsa-framework/slsa-verifier/v2/cli/slsa-verifier@latest

# Verify
slsa-verifier verify-artifact myapp \
  --provenance-path multiple.intoto.jsonl \
  --source-uri github.com/your-org/your-repo \
  --source-tag v1.2.3

Cosign for Container Images

For container images, cosign (part of the Sigstore project) provides signing and attestation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Sign an image (keyless, using Sigstore's transparency log)
cosign sign ghcr.io/your-org/your-image:v1.2.3

# Attach an SBOM
syft ghcr.io/your-org/your-image:v1.2.3 -o cyclonedx-json > sbom.json
cosign attest --predicate sbom.json --type cyclonedx ghcr.io/your-org/your-image:v1.2.3

# Verify
cosign verify ghcr.io/your-org/your-image:v1.2.3 \
  --certificate-identity-regexp "https://github.com/your-org/your-repo" \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

Keyless signing uses short-lived certificates from Sigstore’s Fulcio CA, with entries logged in Rekor’s transparency log. No private key to manage, no key rotation — just OIDC tokens from your CI provider.

diffoscope: Comparing Binaries

When a build is supposed to be reproducible but isn’t, you need to find out why. diffoscope is the tool for this. It recursively compares files and archives, unpacking them layer by layer and producing a human-readable diff.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

# Install
apt install diffoscope
pip install diffoscope

# Compare two builds of the same binary
diffoscope build1/myapp build2/myapp

# HTML output (more readable for large diffs)
diffoscope build1/myapp build2/myapp --html diff.html

# Compare Docker images
diffoscope image1.tar image2.tar --html diff.html

# Compare .deb packages
diffoscope package_v1.deb package_v2.deb

# JSON output for processing
diffoscope build1/ build2/ --json diff.json

diffoscope handles: ELF binaries, ar/tar/zip archives, Java JARs, Debian packages, RPMs, container images, PDF files, and more. It uses objdump, readelf, and similar tools to extract structured information from binary formats before diffing.

Example output when timestamps are embedded:

├── readelf --wide --decompress --hex-dump=.rodata {}
│   @@ -1,4 +1,4 @@
│   -  [00000000] 42 75 69 6c 74 20 61 74 20 4a 61 6e 20 31 35 20  Built at Jan 15
│   +  [00000000] 42 75 69 6c 74 20 61 74 20 46 65 62 20 32 38 20  Built at Feb 28

Practical Starting Points

For a Go Project

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Set SOURCE_DATE_EPOCH from git
export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)

# Build with path normalization
go build -trimpath -ldflags="-s -w" ./...

# Verify: build twice, compare hashes
sha256sum myapp
go build -trimpath -ldflags="-s -w" -o myapp2 ./...
sha256sum myapp2
# Hashes should match

For a Docker Image

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

FROM debian:12.5-slim

# Pin package versions
RUN apt-get update && apt-get install -y \
    ca-certificates=20230311 \
    && rm -rf /var/lib/apt/lists/*

COPY --chown=nobody:nobody app /app/app

USER nobody
ENTRYPOINT ["/app/app"]

1
2
3
4
5
6
7
8
9

# Build with SOURCE_DATE_EPOCH
docker build \
  --build-arg SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) \
  -t myimage:$(git describe --tags) .

# Verify with diffoscope
docker save myimage:v1 > image1.tar
docker save myimage:v1 > image2.tar
diffoscope image1.tar image2.tar

For a Python Package

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# Build a wheel reproducibly
SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) pip wheel .

# Or with flit
SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) flit build

# Verify
pip wheel . -o dist1/
pip wheel . -o dist2/
diffoscope dist1/*.whl dist2/*.whl

For a Rust Project

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# Rust is almost reproducible out of the box
# Add path normalization
RUSTFLAGS="--remap-path-prefix $HOME=~" cargo build --release

# Set SOURCE_DATE_EPOCH
SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) cargo build --release

# Verify
cargo build --release
cp target/release/myapp myapp1
cargo build --release
diff <(sha256sum myapp1) <(sha256sum target/release/myapp)

The Reproducible Builds Checklist

Work through this list when making a build reproducible:

• Set SOURCE_DATE_EPOCH from the most recent git commit
• Remove __DATE__/__TIME__ usage from source code
• Use -trimpath (Go) or --remap-path-prefix (Rust) or -fdebug-prefix-map (GCC/Clang)
• Sort all file lists before processing
• Pin all dependency versions with lockfiles
• Pin base Docker images by digest, not just tag
• Pin tool versions (compiler, linker, build system)
• Use content-based build IDs (-Wl,--build-id=sha1)
• Test reproducibility: build twice, compare with diffoscope
• Add a CI job that builds twice and asserts byte-for-byte equality
• For releases: generate SLSA provenance and sign with cosign

Conclusion

Reproducible builds are not an academic concern. They’re the technical foundation for verifiable software supply chain security, effective build caching, and trustworthy debugging. The tools to achieve them — SOURCE_DATE_EPOCH, Bazel, Nix, SLSA generators, cosign, diffoscope — are mature and well-documented.

The path forward doesn’t require adopting everything at once. Start with SOURCE_DATE_EPOCH and path normalization for quick wins. Add build caching (Bazel remote cache, Gradle build cache) for the performance win. Add SLSA provenance to your release workflow for the security win. Each step independently valuable; together, they make your software supply chain auditable and trustworthy.

The question isn’t whether your build should be reproducible. The question is how much of your supply chain attack surface you’re comfortable leaving open.