Container Image Hardening: Distroless, Multi-Stage Builds, and Vulnerability Scanning

The average developer pulls node:latest, writes a Dockerfile, ships it, and moves on. Inside that image lives a full Debian environment: a shell, a package manager, curl, wget, compilers, and hundreds of packages that your application never touches at runtime. Every one of those packages is a potential CVE. Every installed binary is a pivot point for an attacker who achieves code execution.

Container image hardening is the practice of aggressively narrowing what ends up in your production images. The goal is deceptively simple: ship only what the application needs to run — nothing more. This post covers the full toolkit: multi-stage builds, distroless base images, Trivy scanning, Cosign signing, and the Kubernetes runtime controls that complete the picture.

Why Container Image Hardening Matters

The supply chain attack surface

A container image isn’t just your code. It’s a stack of layers, each one potentially introducing:

• Base image packages — the OS userland, system libraries, and utilities bundled into ubuntu:22.04 or node:20
• Build tools — compilers, linkers, test frameworks, and dev headers that are useful at build time but dangerous in production
• Application dependencies — npm, pip, Maven, and Cargo packages, each with their own transitive dependency trees
• Secrets accidentally baked in — .env files, SSH keys, API tokens copied into layers during development

The supply chain attack surface compounds at every layer. When SolarWinds was compromised in 2020, the attack vector was injecting malicious code into a build artifact before signing. The same class of attack applies to container images: a compromised base image, a typosquatted package, or a developer accidentally pushing an image with credentials baked in can all cascade silently into production.

Real-world impact: Log4Shell and container sprawl

When Log4Shell (CVE-2021-44228) dropped in December 2021, organizations running containers scrambled to answer one question: which of our images contains log4j? For teams with fat images — Java applications deployed on openjdk:11 with the full JDK, hundreds of Maven dependencies, and no SBOM — the answer required forensic work across every running image. Teams with lean, hardened images with software bill of materials (SBOMs) could answer the question in minutes.

Fat images also slow down your scanner. A 1.2GB openjdk:11 image with every Maven dependency baked in takes far longer to scan than a 25MB image containing only the application JAR and the JRE. The bloat isn’t just wasted disk space — it’s wasted scanning time, wasted bandwidth, and more CVEs to triage.

The cost of a fat image

The principle maps directly to classical security: minimize attack surface by removing everything not required for the application to function.

Understanding Image Layers and Attack Surface

How Docker layers work

Every instruction in a Dockerfile creates a new layer. Layers are cached, content-addressed, and stacked. When you write:

1
2
3
4
5

FROM ubuntu:22.04
RUN apt-get update && apt-get install -y curl python3 python3-pip
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY app/ /app/

You get five layers. The second RUN instruction alone can pull in dozens of transitive packages. Critically, even if you later run RUN apt-get purge curl in a subsequent layer, the curl binary still exists in the earlier layer and is extractable from the image.

This is the immutability problem: you can’t retroactively remove something from a lower layer. The only correct approach is to never add it in the first place.

The apt-get install trap

When you install a package in a Dockerfile, you’re not installing one thing. You’re installing a package and all of its transitive dependencies. python3-pip on Ubuntu 22.04 pulls in python3, python3.10, python3-distutils, libpython3.10, and more. A seemingly innocent RUN apt-get install -y git can add 30+ packages.

In production, you need none of that. You needed git during the build to clone a dependency. Once the build is done, git is attack surface.

Base image comparison

Choosing your base image is the highest-leverage decision you make in a Dockerfile. The difference between choices is dramatic:

alpine is a popular middle ground: small, has a shell for debugging, uses musl libc instead of glibc. Be aware that musl and glibc are not fully compatible — some C programs compiled against glibc will not run on Alpine. For statically compiled languages like Go, this is a non-issue.

scratch is for fully statically compiled binaries with zero runtime dependencies. A Go binary compiled with CGO_DISABLED=1 is the canonical example.

The “bloat creep” problem

Bloat often creeps in gradually. A developer adds RUN apt-get install -y vim to the Dockerfile to help debug a production issue. They fix the issue and forget the line. Six months later, you have a production image containing a full text editor with its own CVE history. Automated scanning in CI is the only reliable defense against this pattern.

Multi-Stage Builds: The Foundational Pattern

Multi-stage builds are the single most impactful change you can make to your Dockerfiles. The concept is straightforward: use one or more intermediate containers to build your application, then copy only the artifacts into a minimal final image. Build dependencies — compilers, test runners, dev headers, build scripts — never reach the production image.

Go application

Go is the ideal language for minimal containers. With CGO_DISABLED=1, Go produces a fully static binary with zero runtime dependencies. The result: a ~2MB distroless image from a ~500MB build environment.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

# syntax=docker/dockerfile:1

# ---- Build Stage ----
FROM golang:1.22-alpine AS builder

WORKDIR /build

# Download dependencies first (layer caching — deps change less often than code)
COPY go.mod go.sum ./
RUN go mod download

# Copy source and build
COPY . .

# Run tests before building
RUN go test ./... -v

# Produce a static binary — no CGO, no runtime dependencies
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build \
    -ldflags="-w -s -extldflags '-static'" \
    -o /build/app \
    ./cmd/server

# ---- Final Stage ----
FROM gcr.io/distroless/static:nonroot

# Copy only the binary from the build stage
COPY --from=builder /build/app /app

# distroless/static:nonroot runs as uid 65532 by default
USER nonroot:nonroot

EXPOSE 8080

ENTRYPOINT ["/app"]

The final image contains: your binary, SSL certificates (for HTTPS client calls), timezone data, and nothing else. No shell. No package manager. No curl. An attacker with code execution has almost nothing to work with.

The -ldflags="-w -s" strips debug symbols and the symbol table, reducing binary size. -extldflags '-static' ensures full static linking.

Node.js application

Node.js can’t produce a single static binary, so the strategy is different: separate the build dependencies (TypeScript compiler, webpack, eslint) from the runtime dependencies.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

# syntax=docker/dockerfile:1

# ---- Build Stage ----
FROM node:20-alpine AS builder

WORKDIR /app

# Install ALL dependencies (including devDependencies for build tools)
COPY package.json package-lock.json ./
RUN npm ci

# Copy source and build
COPY tsconfig.json ./
COPY src/ ./src/
RUN npm run build

# Run tests
RUN npm test

# ---- Production Stage ----
FROM node:20-alpine AS production

# Create non-root user
RUN addgroup -S appgroup && adduser -S appuser -G appgroup

WORKDIR /app

# Install ONLY production dependencies
COPY package.json package-lock.json ./
RUN npm ci --omit=dev && npm cache clean --force

# Copy built artifacts from builder stage
COPY --from=builder /app/dist ./dist

# Set ownership
RUN chown -R appuser:appgroup /app

USER appuser

EXPOSE 3000

HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
  CMD node -e "require('http').get('http://localhost:3000/health', r => process.exit(r.statusCode === 200 ? 0 : 1))"

CMD ["node", "dist/index.js"]

The --omit=dev flag in the production stage ensures TypeScript, webpack, jest, and all other build/test tooling are excluded. Only packages listed under dependencies (not devDependencies) are installed.

Python application

Python requires its standard library and site-packages at runtime, but not pip or setuptools.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

# syntax=docker/dockerfile:1

# ---- Build Stage ----
FROM python:3.12-slim AS builder

WORKDIR /build

# Install build dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
    gcc \
    && rm -rf /var/lib/apt/lists/*

# Install dependencies to a custom location
COPY requirements.txt .
RUN pip install --no-cache-dir --prefix=/install -r requirements.txt

# ---- Final Stage ----
FROM gcr.io/distroless/python3-debian12

# Copy installed packages from builder
COPY --from=builder /install /usr/local

# Copy application source
COPY app/ /app/

WORKDIR /app

USER nonroot:nonroot

EXPOSE 8000

CMD ["main.py"]

The key trick is pip install --prefix=/install which installs packages to a separate directory that we then COPY into the final image. The gcc compiler used to build any C extensions stays in the builder stage.

gcr.io/distroless/python3-debian12 includes the Python interpreter but no pip, no setuptools, no shell, and no package manager.

Java/JVM application

Java needs the JVM at runtime but not the full JDK (compiler, javac, jlink, jshell, etc.). The JRE is roughly half the size of the JDK.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

# syntax=docker/dockerfile:1

# ---- Build Stage ----
FROM maven:3.9-eclipse-temurin-21 AS builder

WORKDIR /build

# Copy POM first for dependency caching
COPY pom.xml .
RUN mvn dependency:go-offline -B

# Copy source and build
COPY src/ ./src/
RUN mvn package -B -DskipTests=false

# ---- Final Stage ----
FROM eclipse-temurin:21-jre-alpine

# Create non-root user
RUN addgroup -S javagroup && adduser -S javauser -G javagroup

WORKDIR /app

# Copy only the built JAR
COPY --from=builder /build/target/app-*.jar app.jar

RUN chown javauser:javagroup app.jar

USER javauser

EXPOSE 8080

HEALTHCHECK --interval=30s --timeout=10s --start-period=30s --retries=3 \
  CMD wget -q -O- http://localhost:8080/actuator/health || exit 1

ENTRYPOINT ["java", "-XX:+UseContainerSupport", "-XX:MaxRAMPercentage=75.0", "-jar", "app.jar"]

eclipse-temurin:21-jre-alpine gives you a JRE on Alpine, producing an image that’s typically 80-100MB — significantly smaller than the ~400MB build image. For even smaller Java images, consider using jlink to produce a custom JRE containing only the modules your application actually uses.

Common multi-stage mistakes

Wrong COPY –from reference: COPY --from=0 refers to the first stage by index. This is fragile — if you add a stage, the indices shift. Always use named stages: AS builder, AS tester, AS production.

Forgetting .dockerignore: Without a .dockerignore, COPY . . sends your entire project directory as build context, including .git, node_modules, .env files, and test fixtures. This bloats the build context, can bake in secrets, and defeats layer caching.

A solid .dockerignore:

.git
.gitignore
.env
.env.*
node_modules
__pycache__
*.pyc
*.pyo
.pytest_cache
.coverage
coverage/
dist/
build/
*.log
.DS_Store
Thumbs.db
*.md
tests/
docs/
.github/

Distroless Images

What distroless means

“Distroless” means the image contains no Linux distribution — no shell, no package manager, no coreutils, no apt, no yum, no apk. It contains only your application and its runtime dependencies. The term was coined by Google, which open-sourced the gcr.io/distroless image collection.

Without a shell, an attacker who achieves remote code execution in your container faces a much harder lateral movement problem. They can’t run bash, can’t use apt install to bring in new tools, can’t trivially enumerate the filesystem. They’re constrained to what your application binary can do.

Google’s distroless images

The gcr.io/distroless collection provides images for common runtimes:

Each image has a :debug variant that adds a busybox shell, enabling interactive debugging:

1
2
3
4
5
6

# Run your container with the debug variant for troubleshooting
docker run --rm -it --entrypoint=/busybox/sh gcr.io/distroless/static:debug

# In production Kubernetes, use kubectl debug to attach a debug container
# without modifying the running pod
kubectl debug -it mypod --image=gcr.io/distroless/static:debug --target=mycontainer

Never deploy the :debug variant to production — it reintroduces the shell you removed.

Chainguard Images

Chainguard offers a commercially-backed alternative to distroless based on the Wolfi Linux distribution (a purpose-built, glibc-based OS designed for containers). Key advantages:

• Rebuilt daily — Chainguard images are rebuilt every day against the latest package versions, meaning CVEs are patched faster
• Minimal by default — every image starts from a minimal base and adds only what’s necessary
• SBOM included — every image ships with a software bill of materials
• Signed with Cosign — every image is cryptographically signed

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# Chainguard static image — functionally equivalent to gcr.io/distroless/static
FROM cgr.dev/chainguard/static AS final

# Chainguard Go builder
FROM cgr.dev/chainguard/go:latest AS builder
WORKDIR /build
COPY . .
RUN go build -o app ./cmd/server

FROM cgr.dev/chainguard/static
COPY --from=builder /build/app /app
ENTRYPOINT ["/app"]

Chainguard’s free tier covers the most popular images. Their paid tier adds older version tags (the free tier only offers :latest).

Debugging distroless containers

The lack of a shell requires a different debugging workflow. In Kubernetes, use ephemeral debug containers:

1
2
3
4
5
6
7
8

# Attach a debug container to a running pod without restarting it
kubectl debug -it <pod-name> \
  --image=busybox:latest \
  --target=<container-name> \
  --copy-to=debug-pod

# Or start a copy of the pod with a debug shell
kubectl debug <pod-name> -it --image=ubuntu --share-processes --copy-to=debug-pod

For Docker:

1
2
3
4
5
6
7
8
9

# Start your normal container
docker run -d --name myapp myapp:latest

# Attach a debug container that shares the process namespace
docker run -it --rm \
  --pid=container:myapp \
  --net=container:myapp \
  --volumes-from=myapp \
  busybox:latest sh

When distroless isn’t practical

Distroless is a poor fit for:

• Applications with shell-based entrypoints — if your CMD is a shell script, you need a shell. Refactor to exec form (CMD ["node", "server.js"]) or keep a shell but use Alpine instead.
• Applications requiring dynamic linking against system libraries not included in distroless — some native extensions, database drivers, or FFI-based libraries pull in system dependencies. Use distroless/cc or stay on Alpine.
• Heavy debugging environments — if you’re in early development and need fast iteration, use Alpine and switch to distroless for production.

Dockerfile Best Practices for Security

Run as non-root

Containers run as root by default. If an attacker exploits your application, they get root in the container. Depending on the container runtime configuration and whether the host shares the user namespace, this can translate to significant host privileges.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# Alpine-based image
RUN addgroup -S appgroup && adduser -S appuser -G appgroup
USER appuser

# Debian/Ubuntu-based image
RUN groupadd --gid 1001 appgroup && \
    useradd --uid 1001 --gid appgroup --shell /bin/false --create-home appuser
USER 1001

# distroless/static:nonroot runs as uid 65532 automatically
FROM gcr.io/distroless/static:nonroot
# nonroot user is already set — no USER instruction needed

Use numeric UIDs in USER instructions rather than names — distroless images don’t have /etc/passwd, so name-based lookups will fail.

Pin base images to digests

Tags are mutable. golang:1.22-alpine today and golang:1.22-alpine in six weeks can point to different images if the upstream maintainers push a new build. Use digest pinning to guarantee you’re always building on the exact same base:

1
2
3
4
5
6
7
8
9

# Bad — tag can change
FROM golang:1.22-alpine

# Good — pinned to exact digest
FROM golang:1.22-alpine@sha256:cdc86d9f363e8786845bea2040312b4efa321b828acdeb26f393faa864d533d1

# Get the current digest:
# docker pull golang:1.22-alpine
# docker inspect golang:1.22-alpine --format='{{index .RepoDigests 0}}'

Automate digest updates with Renovate Bot or Dependabot, which can open PRs when new digests are available for your pinned images.

No secrets in layers

Secrets baked into image layers are permanently readable from image history, even if you delete them in a subsequent layer:

1
2

# This is permanently readable even if you RUN rm /run/secrets in the next layer
docker history myapp:latest --no-trunc

Use BuildKit secret mounts instead:

1
2
3
4
5
6
7
8
9

# syntax=docker/dockerfile:1

FROM python:3.12-slim AS builder

# Mount a secret at build time — it never appears in any layer
RUN --mount=type=secret,id=pip_token \
    pip install --no-cache-dir \
    --index-url https://$(cat /run/secrets/pip_token)@private-pypi.example.com/simple/ \
    -r requirements.txt

1
2
3
4

# Pass the secret at build time — it's never written to disk or image layers
docker buildx build \
  --secret id=pip_token,env=PIP_TOKEN \
  -t myapp:latest .

For SSH keys needed during build (e.g., private git repos):

1

RUN --mount=type=ssh git clone git@github.com:org/private-repo.git

1

docker buildx build --ssh default=$SSH_AUTH_SOCK -t myapp:latest .

Minimize RUN instructions

Every RUN instruction creates a layer. Chain related commands to keep intermediate state out of the final image:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# Bad — apt cache lives in a layer even after rm -rf
RUN apt-get update
RUN apt-get install -y curl
RUN rm -rf /var/lib/apt/lists/*

# Good — single layer, cache cleaned in same operation
RUN apt-get update && apt-get install -y --no-install-recommends \
    curl \
    ca-certificates \
    && rm -rf /var/lib/apt/lists/*

Drop capabilities

Linux capabilities divide root’s privileges into distinct units. Most applications need zero capabilities. Drop them all and add back only what’s required:

1
2
3
4
5

# Docker run
docker run --cap-drop ALL --cap-add NET_BIND_SERVICE myapp:latest

# NET_BIND_SERVICE: allow binding to ports below 1024
# Alternatively: run on a high port (8080) and use a reverse proxy

1
2
3
4
5
6
7

# Kubernetes securityContext
securityContext:
  capabilities:
    drop:
      - ALL
    add:
      - NET_BIND_SERVICE  # Only if truly needed

No-new-privileges

Prevent privilege escalation via setuid binaries:

1

docker run --security-opt no-new-privileges:true myapp:latest

1
2

securityContext:
  allowPrivilegeEscalation: false

HEALTHCHECK instruction

Define a health check in your Dockerfile so Docker and orchestration systems can detect unhealthy containers:

1
2
3
4
5
6
7

# HTTP health check
HEALTHCHECK --interval=30s --timeout=5s --start-period=15s --retries=3 \
  CMD wget -qO- http://localhost:8080/health || exit 1

# TCP check for non-HTTP services
HEALTHCHECK --interval=10s --timeout=3s \
  CMD nc -z localhost 5432 || exit 1

Vulnerability Scanning with Trivy

Trivy is an open-source vulnerability and misconfiguration scanner from Aqua Security. It’s comprehensive, fast, and integrates cleanly into CI/CD pipelines.

What Trivy scans

• OS packages — APK, DEB, RPM packages and their CVEs from the National Vulnerability Database
• Language packages — package-lock.json, requirements.txt, go.sum, Cargo.lock, Gemfile.lock, pom.xml
• Dockerfiles — misconfigurations like running as root, missing HEALTHCHECK, unpinned tags
• Kubernetes manifests — missing resource limits, privileged containers, host namespaces
• Secrets — API keys, passwords, tokens in image layers or source files
• SBOM — can generate and scan CycloneDX and SPDX software bills of materials

Installation

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

# macOS
brew install aquasecurity/trivy/trivy

# Debian/Ubuntu
sudo apt-get install wget apt-transport-https gnupg
wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | gpg --dearmor | sudo tee /usr/share/keyrings/trivy.gpg > /dev/null
echo "deb [signed-by=/usr/share/keyrings/trivy.gpg] https://aquasecurity.github.io/trivy-repo/deb generic main" | sudo tee /etc/apt/sources.list.d/trivy.list
sudo apt-get update && sudo apt-get install trivy

# Docker (no installation required)
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
  aquasec/trivy:latest image myapp:latest

# Check version
trivy --version

Scanning images

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# Basic scan — shows all vulnerabilities
trivy image nginx:latest

# Only show HIGH and CRITICAL
trivy image --severity HIGH,CRITICAL nginx:latest

# Only show vulnerabilities with available fixes
trivy image --ignore-unfixed nginx:latest

# JSON output for processing
trivy image --format json --output trivy-report.json myapp:latest

# SARIF output for GitHub Security tab
trivy image --format sarif --output trivy.sarif myapp:latest

# Fail the command if any CRITICAL vulnerabilities are found (for CI)
trivy image --exit-code 1 --severity CRITICAL myapp:latest

# Scan a local tar archive (useful when Docker socket isn't available)
docker save myapp:latest | trivy image --input -

# Include secret scanning
trivy image --scanners vuln,secret myapp:latest

Scanning Dockerfiles for misconfigurations

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Scan a single Dockerfile
trivy config ./Dockerfile

# Scan an entire directory (Dockerfiles, Kubernetes manifests, Terraform)
trivy config ./

# Example findings:
# AVD-DS-0002: Specify a tag in the FROM statement for image 'ubuntu'
# AVD-DS-0006: Add a HEALTHCHECK instruction
# AVD-DS-0001: Do not use the root USER
# AVD-DS-0026: Use COPY instead of ADD for files

Scanning filesystems and repositories

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Scan the current directory (finds vulnerabilities in lock files)
trivy fs .

# Scan a specific path
trivy fs /path/to/project

# Scan a remote git repository
trivy repo https://github.com/your-org/your-repo

# Include secret scanning in filesystem scan
trivy fs --scanners vuln,secret .

Generating an SBOM

1
2
3
4
5
6
7
8

# Generate CycloneDX SBOM
trivy image --format cyclonedx --output sbom-cyclonedx.json myapp:latest

# Generate SPDX SBOM
trivy image --format spdx-json --output sbom-spdx.json myapp:latest

# Scan an existing SBOM for vulnerabilities
trivy sbom sbom-cyclonedx.json

SBOMs are how you answer “do any of our images contain log4j?” in minutes rather than days. Storing SBOMs as artifacts alongside your container images is increasingly a compliance requirement (NIST SP 800-218, Executive Order 14028).

The .trivyignore file

Some findings are accepted risks, false positives, or simply not applicable. Document suppressions with justifications:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# .trivyignore

# CVE-2023-4911 (glibc): Not exploitable — no SUID binaries in image and running nonroot
CVE-2023-4911

# CVE-2022-1271 (xz): Only affects systems with specific locale configurations we don't use
# Accepted risk — awaiting upstream fix, revisit 2026-06-01
CVE-2022-1271

# False positive — this is test data, not a real credential
generic-api-key:path/to/testdata/fixtures/sample.txt

Signing and Verifying Images with Cosign

Cosign is part of the Sigstore project. It provides container image signing and verification, allowing you to cryptographically verify that the image you’re deploying is the exact image your CI/CD pipeline built and scanned.

Why signing matters

An image tag (myapp:v1.2.3) is a mutable pointer. It can be changed. Someone with registry write access can push a malicious image to that tag and your deployment pipeline will happily pull and run it. Signing ties the cryptographic identity of the image (its digest) to a trusted key or identity.

Key-pair signing

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# Install cosign
brew install cosign  # macOS
# Or: go install github.com/sigstore/cosign/v2/cmd/cosign@latest

# Generate a key pair
cosign generate-key-pair
# Creates cosign.key (private, protect this!) and cosign.pub

# Sign an image — pushes signature to the registry as an OCI artifact
cosign sign --key cosign.key myregistry.io/myapp:v1.2.3

# Verify before deployment
cosign verify --key cosign.pub myregistry.io/myapp:v1.2.3

Keyless signing with OIDC (GitHub Actions)

Keyless signing uses the OIDC token from your CI provider as proof of identity. The private key is ephemeral — it exists only for the duration of the signing operation and is attested to by Sigstore’s Fulcio certificate authority.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

- name: Sign image with Cosign (keyless)
  uses: sigstore/cosign-installer@v3

- name: Sign
  run: |
    cosign sign --yes \
      --rekor-url https://rekor.sigstore.dev \
      ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${{ steps.build.outputs.digest }}
  env:
    COSIGN_EXPERIMENTAL: 1

Policy enforcement

Require signed images in Kubernetes using Kyverno:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-signed-images
spec:
  validationFailureAction: Enforce
  rules:
  - name: verify-image-signature
    match:
      resources:
        kinds:
        - Pod
    verifyImages:
    - image: "myregistry.io/myapp:*"
      key: |-
        -----BEGIN PUBLIC KEY-----
        MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE...
        -----END PUBLIC KEY-----

Runtime Security

A hardened image is necessary but not sufficient. The container runtime and Kubernetes configuration determine what the running container can actually do.

Kubernetes securityContext

Apply these settings to every production workload:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  template:
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        runAsGroup: 1000
        fsGroup: 1000
        seccompProfile:
          type: RuntimeDefault  # Apply the container runtime's default seccomp profile
      containers:
      - name: myapp
        image: myregistry.io/myapp:v1.2.3
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          capabilities:
            drop:
              - ALL
        volumeMounts:
        - name: tmp
          mountPath: /tmp
        - name: cache
          mountPath: /app/cache
      volumes:
      - name: tmp
        emptyDir: {}
      - name: cache
        emptyDir: {}

The readOnlyRootFilesystem: true setting is particularly powerful — it means an attacker who achieves code execution can’t write new files to the filesystem (no installing tools, no writing web shells). Any application writes must go to explicitly mounted emptyDir volumes.

Kubernetes Pod Security Standards

Kubernetes 1.25+ includes built-in Pod Security Standards (replacing the deprecated PodSecurityPolicy). Apply them at the namespace level:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    # Enforce the restricted standard — most locked down
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    # Warn and audit for the restricted standard
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/audit: restricted

The restricted standard requires: non-root user, no privilege escalation, read-only root filesystem, dropped capabilities, and a seccomp profile.

Falco: runtime threat detection

Falco monitors running containers for suspicious behavior at the syscall level. Default rules detect:

• A shell spawned inside a container
• Unexpected outbound network connections
• Sensitive file reads (/etc/shadow, /etc/passwd)
• Container drift (new executables written and then executed)
• Privilege escalation attempts

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# Example Falco rule: alert on shell execution in containers
- rule: Terminal shell in container
  desc: A shell was used as the entrypoint or exec'd into a container
  condition: >
    spawned_process and container
    and shell_procs
    and proc.tty != 0
    and container_entrypoint
  output: >
    A shell was spawned in a container with an attached terminal
    (user=%user.name user_loginuid=%user.loginuid %container.info
    shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)
  priority: NOTICE

CI/CD Integration: Shifting Left

The best time to catch a vulnerability is before it reaches production — ideally before it’s even pushed to the registry. Here’s a complete GitHub Actions workflow that builds, scans, signs, and conditionally pushes:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122

name: Build, Scan, Sign, and Push

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  build-scan-push:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
      security-events: write  # For SARIF upload
      id-token: write         # For keyless Cosign signing

    steps:
    - name: Checkout
      uses: actions/checkout@v4

    - name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v3

    - name: Log in to Container Registry
      uses: docker/login-action@v3
      with:
        registry: ${{ env.REGISTRY }}
        username: ${{ github.actor }}
        password: ${{ secrets.GITHUB_TOKEN }}

    - name: Extract metadata
      id: meta
      uses: docker/metadata-action@v5
      with:
        images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
        tags: |
          type=sha,prefix=,format=short
          type=ref,event=branch
          type=semver,pattern={{version}}

    - name: Build image (without push)
      id: build
      uses: docker/build-push-action@v5
      with:
        context: .
        push: false
        load: true  # Load into local Docker daemon for scanning
        tags: ${{ steps.meta.outputs.tags }}
        labels: ${{ steps.meta.outputs.labels }}
        cache-from: type=gha
        cache-to: type=gha,mode=max

    - name: Cache Trivy DB
      uses: actions/cache@v4
      with:
        path: ~/.cache/trivy
        key: trivy-db-${{ github.run_id }}
        restore-keys: trivy-db-

    - name: Run Trivy vulnerability scan
      uses: aquasecurity/trivy-action@master
      with:
        image-ref: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ steps.meta.outputs.version }}
        format: sarif
        output: trivy-results.sarif
        severity: HIGH,CRITICAL
        exit-code: 1        # Fail the build on HIGH or CRITICAL findings
        ignore-unfixed: true  # Only fail on fixable vulnerabilities

    - name: Upload Trivy SARIF to GitHub Security
      uses: github/codeql-action/upload-sarif@v3
      if: always()  # Upload even if the scan step failed
      with:
        sarif_file: trivy-results.sarif

    - name: Scan Dockerfile for misconfigurations
      uses: aquasecurity/trivy-action@master
      with:
        scan-type: config
        scan-ref: .
        exit-code: 1

    - name: Push image (only if scan passed)
      uses: docker/build-push-action@v5
      with:
        context: .
        push: ${{ github.ref == 'refs/heads/main' }}
        tags: ${{ steps.meta.outputs.tags }}
        labels: ${{ steps.meta.outputs.labels }}
        cache-from: type=gha

    - name: Install Cosign
      if: github.ref == 'refs/heads/main'
      uses: sigstore/cosign-installer@v3

    - name: Sign image (keyless)
      if: github.ref == 'refs/heads/main'
      run: |
        cosign sign --yes \
          ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${{ steps.build.outputs.digest }}
      env:
        COSIGN_EXPERIMENTAL: 1

    - name: Generate SBOM
      if: github.ref == 'refs/heads/main'
      uses: aquasecurity/trivy-action@master
      with:
        image-ref: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ steps.meta.outputs.version }}
        format: cyclonedx
        output: sbom.cyclonedx.json

    - name: Upload SBOM artifact
      if: github.ref == 'refs/heads/main'
      uses: actions/upload-artifact@v4
      with:
        name: sbom
        path: sbom.cyclonedx.json

This workflow:

1. Builds the image using BuildKit with layer caching
2. Scans for HIGH and CRITICAL vulnerabilities — fails the build if any are found
3. Uploads results to GitHub’s Security tab (visible under Security > Code scanning alerts)
4. Scans the Dockerfile for misconfigurations
5. Only pushes to the registry if all scans pass and the branch is main
6. Signs the pushed image with keyless Cosign
7. Generates and stores a CycloneDX SBOM

The ignore-unfixed: true setting is important for pipeline hygiene: it only fails on vulnerabilities where a patched version of the package is available. Unfixed CVEs are reported but don’t block the build — otherwise you’ll find yourself blocked on vulnerabilities that have no available fix and no reasonable mitigation.

Practical Hardening Checklist

Before and after: the impact of hardening

The checklist

• Use a multi-stage build — build dependencies never reach the final image
• Use distroless or Alpine for the final stage — no shell, no package manager, minimal CVE surface
• Pin base images to digests — FROM golang:1.22-alpine@sha256:... — never use latest
• Run as a non-root user — USER 1001 or USER nonroot:nonroot
• Create and maintain a .dockerignore — exclude .git, .env, node_modules, test files
• No secrets in image layers — use BuildKit secret mounts (--mount=type=secret)
• Scan with Trivy in CI — --exit-code 1 to fail on CRITICAL, upload SARIF to GitHub
• Scan Dockerfile for misconfigurations — trivy config ./Dockerfile
• Drop all Linux capabilities — --cap-drop ALL in Docker, capabilities: drop: [ALL] in Kubernetes
• Set readOnlyRootFilesystem: true in Kubernetes securityContext
• Set allowPrivilegeEscalation: false in Kubernetes securityContext
• Apply a seccomp profile — seccompProfile: type: RuntimeDefault
• Sign the image with Cosign — keyless signing in CI, verify on deploy
• Generate and store an SBOM — CycloneDX or SPDX, stored as a CI artifact or pushed to the registry
• Define a HEALTHCHECK — in the Dockerfile, verified by your orchestrator
• Apply Kubernetes Pod Security Standards — restricted profile for production namespaces

Putting It All Together

Container image hardening isn’t a single action — it’s a set of layered practices that compound. A Go service that starts as an 800MB build environment ends up as a 2MB distroless image with near-zero CVEs, no shell, no package manager, non-root execution, read-only filesystem, and a signed, SBOM-attested artifact in your registry.

The practices here aren’t exotic or difficult to implement. Multi-stage builds are a one-time refactor. Adding Trivy to CI takes 15 minutes. Switching FROM ubuntu:22.04 to FROM gcr.io/distroless/static:nonroot for a Go service is a one-line change. The security benefit is immediate and permanent.

The attackers targeting containers aren’t exploiting configuration drift in your Kubernetes seccomp profiles — they’re exploiting outdated packages in fat images pulled six months ago and never updated. Fix the basics first, scan consistently, and layer in the runtime controls as your program matures.

Start with multi-stage builds and Trivy in CI. Everything else follows naturally.