HashiCorp Nomad: A Simpler Kubernetes Alternative

April 19, 2026 16 min read DevOps

nomadorchestrationhashicorpkubernetesconsuldevopsscheduling

Contents

Kubernetes is the default answer to “how do I run containers at scale,” and most of the time it’s a fine one. But “most of the time” hides a real distribution of teams who either don’t need the complexity, don’t need just-containers, or are burning a disproportionate fraction of their engineering effort on the platform itself rather than what it runs. HashiCorp Nomad is the answer when you want most of what Kubernetes gives you without most of what Kubernetes asks from you.

Nomad has been around since 2015, ships as a single Go binary (around 300 MB on disk, including both server and client), and schedules not just containers but also Docker/Podman, Java apps, Windows services, raw executables, QEMU VMs, and basically anything else you write a driver for. It runs Cloudflare’s edge, Roblox’s fleet, and a quiet slice of enterprise infrastructure where Kubernetes was evaluated and rejected for being too much. This post covers what Nomad actually is, how its scheduling model differs from Kubernetes, the core primitives (jobs, task groups, allocations), how Consul integration fills in what Nomad doesn’t do itself, and where Nomad genuinely wins versus where Kubernetes is still the right choice.

The core idea

A Nomad cluster is two things:

Servers (3 or 5 of them, for Raft consensus) — store cluster state, run the scheduler, accept API requests.
Clients — run on every host where you want workloads to land. Receive allocation assignments, fork tasks via drivers, stream logs and status back.

That’s the entire control plane. No etcd operator, no kubelet separate from kube-proxy separate from a CRI-O separate from a CNI separate from a CSI. One binary, started in server or client mode, talks to its peers over gossip (Serf) and RPC. The cluster is up in about five minutes, and once it’s up, it stays up in my experience — Raft is solid, the gossip layer handles partitions gracefully, and Nomad’s failure modes tend to be obvious rather than subtle.

The philosophy comes through clearly: Nomad does scheduling and allocation lifecycle. Service discovery is Consul’s job. Secrets are Vault’s job. Networking (beyond basic bridge/host/CNI) is your problem. Observability is your problem. Ingress is your problem. That separation is both Nomad’s biggest critique (“we have to stitch things together!”) and its biggest strength (“the scheduler isn’t also trying to be a network policy engine and a storage orchestrator”).

Jobs, task groups, allocations — the model

Three nested primitives you’ll learn in ten minutes:

Job — the deployable unit. Roughly equivalent to a Kubernetes Deployment. Written in HCL.
Task group — a set of tasks co-located on the same host. Like a Kubernetes Pod: the tasks in one group share a network namespace and a local volume if you ask for one, and they live and die together.
Task — a single process managed by a driver (docker, exec, java, raw_exec, qemu, etc.).

A minimal job:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42


job "web" {
  datacenters = ["dc1"]
  type        = "service"

  group "frontend" {
    count = 3

    network {
      port "http" { to = 8080 }
    }

    service {
      name     = "web-frontend"
      port     = "http"
      provider = "consul"
      check {
        type     = "http"
        path     = "/healthz"
        interval = "10s"
        timeout  = "2s"
      }
    }

    task "app" {
      driver = "docker"

      config {
        image = "registry.internal/web:1.4.2"
        ports = ["http"]
      }

      resources {
        cpu    = 500     # MHz
        memory = 256     # MB
      }

      env {
        DB_HOST = "postgres.service.consul"
      }
    }
  }
}

Run it:

nomad job run web.nomad.hcl

Nomad evaluates: does this job exist? It’s new. Create three allocations of the frontend group. Place each on a client that has 500 MHz CPU and 256 MB memory to spare. Reserve a port on each client. Start the docker container. Register the service in Consul. Start health-checking it.

That’s the whole flow. Reading the evaluation:

$ nomad job status web
ID            = web
Type          = service
Priority      = 50
Datacenters   = dc1
Status        = running
Periodic      = false

Allocations
ID        Node ID   Task Group  Version  Desired  Status    Created   Modified
ab123     nodeA     frontend    0        run      running   2m ago    2m ago
cd456     nodeB     frontend    0        run      running   2m ago    2m ago
ef789     nodeC     frontend    0        run      running   2m ago    2m ago

Each allocation has an ID, a node assignment, a target state, and a current state. Logs are nomad alloc logs ab123 app. Exec into a running task with nomad alloc exec ab123 app /bin/sh. Kill one and the scheduler replaces it within seconds.

Job types

Nomad knows several job types:

service — long-running. Replacements when one dies. Rolling deploys. What you’d use Deployments for in K8s.
batch — run to completion. Similar to K8s Jobs. Nomad tracks exit status and does not replace them when they finish.
system — one allocation per client node. DaemonSet equivalent. Log shippers, node exporters, security agents.
sysbatch — system + batch. Run a one-shot task on every node.
periodic — like service or batch, but re-run on a cron schedule.
parameterized — accept runtime parameters. Good for ad-hoc job execution — “run this image with these args” as a job submission.

The permutations let you handle almost any workload. The one thing Nomad does not natively do well is StatefulSet-style ordered rollouts with persistent per-instance identity — you can approximate it, but StatefulSets in K8s are more fully-featured.

The scheduler: bin-packing with feasibility and scoring

Nomad’s scheduler is a two-phase algorithm: feasibility (which nodes can run this?) then scoring (of those, which is best?).

Feasibility filters by:

Driver availability (the node has docker, or exec, etc.)
Resource availability (enough CPU, memory, disk)
Constraints (e.g., constraint { attribute = "$\{attr.kernel.name\}" value = "linux" })
Node class and datacenter matching
Affinity rules
Existing allocations (spread and anti-affinity)

Scoring ranks the feasible set by:

Bin-packing preference (prefer nodes closer to full — packs workloads tighter)
Spread rules (spread across rack, zone, whatever you declare)
Affinity bonuses (prefer nodes where your allocation’s siblings already run, for locality)

Knobs on the job:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


constraint {
  attribute = "${node.class}"
  value     = "compute"
}

affinity {
  attribute = "${meta.rack}"
  value     = "rack-a"
  weight    = 50
}

spread {
  attribute = "${node.datacenter}"
}

The scheduler is pluggable internally — there’s a service scheduler, batch, system, and spread. They share the feasibility-and-scoring framework but differ in how they ingest changes and how they handle preemption.

Preemption — high-priority jobs can preempt lower-priority jobs when resources are tight. This is a clean primitive Kubernetes gets via PriorityClasses but at the cost of extra machinery; in Nomad it’s priority = 80 on the job and a preemption policy on the server config.

Drivers: not just containers

Nomad doesn’t assume containers. Drivers are the abstraction for “how do I start a thing”:

docker — the obvious one
podman — for rootless containers
exec — runs a binary on the host with cgroup isolation, no container image
raw_exec — no isolation, use for system tools
java — JVM with classpath / JAR support
qemu — full VMs (with image files)
containerd — direct CRI
singularity / apptainer — HPC container runtime via community plugin

This flexibility is genuinely useful. Not everything needs to be a container: a static binary compiled to exec runs with less overhead and no image-build cycle. A legacy Java app that expects to be a plain JVM process can be a java task without wrapping. Workloads that want VM-level isolation (multitenancy, GPU passthrough) can use qemu in the same cluster as the docker jobs.

Rolling deploys

Updates default to rolling:

1
2
3
4
5
6
7
8
9


update {
  max_parallel     = 2
  min_healthy_time = "30s"
  healthy_deadline = "5m"
  progress_deadline = "10m"
  auto_revert      = true
  canary           = 1
  stagger          = "30s"
}

canary = 1 deploys one canary, pauses until you promote it (nomad job promote web), then rolls the rest. auto_revert kicks in if the deployment fails to reach healthy state within progress_deadline. All of this is plainer than a Kubernetes Deployment with its revisionHistoryLimit, maxSurge, maxUnavailable, progressDeadlineSeconds, spread across the strategy and the rollout status. Fewer knobs, and the knobs are named what they do.

Consul integration: the service discovery story

Nomad doesn’t do service discovery. Consul does. They’re designed to cooperate.

When a job has a service block, the Nomad client registers the running allocation with the local Consul agent, including health checks. Other services can then resolve the name:

postgres.service.consul
web-frontend.service.consul

Consul DNS returns healthy instances. Consul’s HTTP API gives richer catalog queries. Consul handles the service mesh piece (Connect) when you want mTLS between services.

From the inside of the allocation, ${NOMAD_ADDR_http} and friends give you the host:port the task was bound to. You rarely need to hardcode anything.

Consul Connect (the service mesh) runs sidecar proxies (Envoy) alongside your tasks. Enabling it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


service {
  name = "web-frontend"
  port = "http"
  connect {
    sidecar_service {
      proxy {
        upstreams {
          destination_name = "postgres"
          local_bind_port  = 5432
        }
      }
    }
  }
}

Nomad spawns an Envoy sidecar in the task group, registers it with Consul, and configures the proxy to route traffic to postgres via mTLS. From the app’s perspective, it connects to localhost:5432. The plumbing is invisible.

Vault integration: secrets without leaking

Vault integration is similarly first-class. Declare the secrets a task needs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


task "app" {
  driver = "docker"

  vault {
    policies = ["web-app-read"]
  }

  template {
    data = <<EOT
      DB_PASS="{{ with secret "database/creds/web-app" }}{{ .Data.password }}{{ end }}"
    EOT
    destination = "secrets/db.env"
    env         = true
  }

  config {
    image = "web:latest"
  }
}

Nomad’s client acquires a Vault token scoped to the allocation, renders the template from Vault secrets, writes it to secrets/db.env, and injects as env vars. The token is revoked when the allocation ends. Dynamic DB credentials (Vault’s database secrets engine) are one template block away — the app gets a just-in-time DB user that expires automatically.

Networking

Nomad networking is either “host” (share the host’s netns — fast, no isolation, port conflicts are a real thing), “bridge” (a Linux bridge per host, allocations get private IPs), or a CNI plugin (Calico, Cilium, anything CNI-compatible).

1
2
3
4
5


network {
  mode = "bridge"
  port "http" { to = 8080 }
  port "metrics" { to = 9100 }
}

to = 8080 says “the container listens on 8080 inside; assign any port on the host outside.” Nomad picks one and sets NOMAD_HOST_PORT_http. Combined with Consul’s DNS, clients find the allocation via the service name; they don’t hardcode ports.

If you want a proper overlay with pod-like networking, install Cilium or Calico and reference the CNI network. Nomad doesn’t ship its own overlay — it defers to CNI.

Ingress: Nomad has no built-in ingress. Run Traefik, HAProxy, nginx, or Consul’s own gateway as a Nomad job, configure it via Consul catalog, and you have dynamic routing. In practice this is a few-hundred-line HCL job and you’re done.

Storage

Nomad supports:

Ephemeral disk — scratch space tied to the allocation’s lifetime.
Host volumes — bind-mount a path from the host. Simple; node affinity is your problem.
CSI — Container Storage Interface plugins. Works with any CSI driver (Ceph, AWS EBS, NFS, Portworx, Longhorn). Supports ReadWriteOnce and ReadWriteMany modes.

Stateful workloads are possible but require thought. The volume block in a group:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


volume "data" {
  type      = "csi"
  source    = "postgres-data"
  read_only = false
  attachment_mode = "file-system"
  access_mode     = "single-node-writer"
}

task "postgres" {
  volume_mount {
    volume      = "data"
    destination = "/var/lib/postgresql/data"
  }
}

Feels similar to Kubernetes PVCs, with less ceremony.

Autoscaling

Nomad has a separate binary — nomad-autoscaler — that handles horizontal autoscaling of job allocations and cluster-level autoscaling of client nodes. Policies are HCL, scrape Prometheus or other sources, and drive count or node-pool size.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


scaling {
  enabled = true
  min     = 2
  max     = 20

  policy {
    cooldown = "1m"
    evaluation_interval = "10s"

    check "cpu_usage" {
      source = "prometheus"
      query  = "avg(nomad_client_allocs_cpu_total_percent{job=\"web\"})"
      strategy "target-value" {
        target = 70
      }
    }
  }
}

Not as integrated as Kubernetes HPA/VPA/CA, but the moving parts are visible and debuggable. When autoscaling misbehaves, you can read a plain Prometheus query and a plain HCL policy to diagnose.

ACLs and multi-tenancy

Nomad’s ACL system is simpler than RBAC. Tokens have policies; policies are HCL blocks granting scoped permissions:

1
2
3
4
5
6
7
8


namespace "production" {
  policy = "read"
  capabilities = ["submit-job", "read-job"]
}

node {
  policy = "read"
}

Namespaces isolate jobs — similar to K8s namespaces. Node pools (added in Nomad 1.6) segment client nodes so you can say “this job only runs on gpu pool” without custom constraints.

Multi-tenancy in Nomad is reasonable but not its primary use case. Nomad is more commonly deployed per-team or per-business-unit rather than as a shared multi-tenant platform.

When Nomad wins

Concrete scenarios where I’d pick Nomad over Kubernetes:

Mixed workloads beyond containers. A fleet running Java apps, static binaries, Windows services, and the occasional container. Nomad’s drivers cover it; Kubernetes essentially requires everything be containerized.
Small-to-medium teams. A team of 10 engineers where platform engineering is a sideline, not a dedicated function. Nomad + Consul + Vault is runnable by one engineer part-time; Kubernetes typically wants a dedicated platform team.
Windows workloads. Nomad handles Windows clients natively. Kubernetes’ Windows support is present but noticeably less polished.
Edge / IoT. Nomad’s small footprint (one ~300 MB binary, minimal dependencies) runs well on edge nodes. K3s narrows the gap, but plain Nomad is still simpler.
Batch + service in one scheduler. Running scientific batch workloads and service workloads in the same cluster. Nomad handles both with the same primitives; Kubernetes asks you to reach for Volcano, Kueue, or Argo Workflows for batch.
HashiCorp-heavy shops. If you already run Consul and Vault, Nomad is a lower-effort addition than a full Kubernetes + external-secrets + Istio + cert-manager stack.
Bare-metal control. Kubernetes is easier on a cloud provider with managed control plane; Nomad’s self-hosted experience on bare metal is straightforward.

When Kubernetes wins

Equally honestly:

Ecosystem. Helm charts, operators, cert-manager, Prometheus Operator, Istio, Flux/Argo, every piece of modern open-source ops. Kubernetes has it; Nomad’s equivalent is often “write it yourself or find the community module.”
Hosted offerings. EKS/GKE/AKS mean you never build the control plane. Nomad has HCP Consul/Nomad but the managed offerings are less mature.
Sophisticated networking and policies. NetworkPolicy, Gateway API, service mesh features in the K8s ecosystem are deeper and more standardized.
Operator pattern. For complex stateful services (Postgres, Kafka, Elastic, etc.), the CRD-and-controller model is where most production-grade automation lives.
Hiring. Kubernetes knowledge is common on the market; Nomad knowledge is rarer.
Compliance / regulated industries. Kubernetes has more third-party compliance tooling, audit frameworks, and vendor-supported distributions targeting FedRAMP, HIPAA, etc.

Be honest with your team: if you’re going to run on Kubernetes anyway because of hosted offerings, ecosystem, or hiring, that’s a legitimate reason. Nomad wins by doing less, and “doing less” is only a win if you don’t need the “more.”

Operational realities

Running Nomad in production for a few years, the things that actually matter:

Gossip tuning. Serf’s default timers are fine for small clusters (<100 nodes). At larger scale, bump reconcile_interval and the probe intervals so the gossip layer doesn’t beat up the network.
Raft log growth. Server disks fill up if you don’t compact. Set raft_boltdb.NoFreelistSync and monitor /var/lib/nomad/server/raft.
GC policies. Finished allocations linger by default. Tune job_gc_threshold, eval_gc_threshold, deployment_gc_threshold — otherwise nomad job status gets cluttered and API responses slow.
Logging. Nomad captures task stdout/stderr and rotates it. Configure log_collection_enabled and ship it off to Loki/Elastic/whatever — the client disk is not the right long-term home.
Monitoring. Nomad exposes Prometheus metrics natively (nomad_* for server metrics, nomad_client_* for client). Combined with the Consul exporter you get cluster health easily.
Upgrades. Nomad’s upgrade story is good — rolling upgrades of servers one at a time, clients upgrade independently. Follow the version-skew docs: clients can be one major version behind servers, but not ahead.
Job versioning. nomad job inspect shows the HCL submitted. Keep your job files in git; Nomad won’t do that for you.

A pragmatic production stack

A typical Nomad-based platform, for contrast with a K8s-based one:

Nomad + Consul + Vault trio on 3-5 shared servers.
Traefik as a Nomad job, registered with Consul, providing ingress for HTTP services.
Prometheus + Grafana + Loki as Nomad jobs, scraping Nomad/Consul/Vault natively.
MinIO as Nomad jobs for object storage.
Nomad CSI + Ceph RBD for stateful workloads’ persistent volumes.
nomad-pack for templating job files (analogous to Helm).
Jobs deployed via CI on git push — nomad job run in a pipeline, validated against staging first.

That setup is approximately 1,500 lines of HCL in a platform repo and can run hundreds of services on 20-50 nodes with one engineer keeping an eye on it. The Kubernetes equivalent is 10x more config, more CRDs, more operators, more moving parts.

Closing

Nomad isn’t a Kubernetes killer and never pretended to be. It’s a different point on the orchestration spectrum — more scheduling, less “platform in a box.” For a team that already has good answers for service discovery and secrets (or is willing to adopt Consul and Vault), Nomad replaces an enormous chunk of Kubernetes complexity with something you can actually hold in your head.

The question to ask yourself is not “Nomad or Kubernetes?” but “what do I need from an orchestrator that my team can actually operate?” If you need a scheduler for containerized workloads, a basic rollout story, service health checking, secrets injection, and nothing else — Nomad will do it with less overhead. If you need the richness of the K8s ecosystem, the operator pattern, or you have a specific requirement (service mesh, network policies, stateful workload controllers), Kubernetes is probably still right.

What I’ve come to believe after running both is that the choice is not between “simple” and “powerful.” It’s between two different philosophies of what a platform should do: one composes a few focused tools (Nomad, Consul, Vault), the other bundles everything into one control plane. Either can work. The mistake is picking the heavier option when you don’t actually need what it costs you.