Linux Namespaces and cgroups: The Kernel Primitives That Make Containers Possible

When you run docker run nginx, something remarkable happens in about 300 milliseconds: a new isolated execution environment appears, with its own process tree, its own network stack, its own filesystem view, and its own resource limits — all without a hypervisor, without booting a kernel, without hardware virtualization.

This is made possible by two Linux kernel features: namespaces and cgroups. Container runtimes like Docker, containerd, and podman are, at their core, orchestrators that configure these two kernel primitives and then execute your process inside them.

Understanding namespaces and cgroups doesn’t just satisfy intellectual curiosity. It explains container behavior that otherwise seems magical or arbitrary, makes debugging easier, informs better container security decisions, and gives you the vocabulary to understand what Kubernetes is actually managing.

Namespaces: Isolation via Virtualized Views

A namespace wraps a global system resource and makes a process believe it has its own private instance of that resource. Processes in different namespaces can’t see or affect each other’s namespaced resources.

Linux currently has eight namespace types:

Each Docker container gets its own instance of most of these. Let’s walk through them.

PID Namespace

In a PID namespace, the first process gets PID 1. It cannot see processes in the parent namespace. From the host, you can see the container’s processes with their real PIDs.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Start a container
docker run -d --name demo nginx

# From the HOST — nginx has a real PID (e.g. 4821)
ps aux | grep nginx

# From INSIDE the container — nginx thinks it's PID 1
docker exec demo ps aux
# PID   USER     COMMAND
#   1   root     nginx: master process
#  32   nginx    nginx: worker process

This has an important implication: PID 1 in a container is special. In Linux, PID 1 is responsible for reaping zombie processes (adopting orphans and waiting on them). If your container’s PID 1 doesn’t handle SIGTERM and reap children, you’ll get zombie processes and containers that don’t stop cleanly.

This is why minimal base images like scratch can cause issues when your app spawns subprocesses — there’s no init process to handle reaping. Solutions:

1
2
3
4
5
6
7
8

# Use tini as a minimal init for your container
FROM ubuntu:24.04
RUN apt-get install -y tini
ENTRYPOINT ["/usr/bin/tini", "--"]
CMD ["/your/app"]

# Or with Docker's built-in init
# docker run --init your-image

Experimenting with PID Namespaces

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# Create a new PID namespace and run bash inside it
# unshare(1) creates new namespaces for a process
sudo unshare --pid --fork --mount-proc bash

# Inside the new namespace:
ps aux
# PID   USER     COMMAND
#   1   root     bash        ← this shell is PID 1
#   2   root     ps aux

# From another terminal on the host:
ps aux | grep bash
# The bash process has a real host PID (not 1)

Network Namespace

A network namespace gives a process its own complete network stack: interfaces, IP addresses, routing tables, iptables rules, and socket table. Processes in different network namespaces cannot communicate directly — they need a virtual ethernet pair (veth pair) bridged between namespaces.

This is exactly what Docker does for every container:

Host network namespace          Container network namespace
─────────────────────           ──────────────────────────
eth0 (192.168.1.100)            eth0 (172.17.0.2)
docker0 bridge (172.17.0.1) ←──veth pair──→ eth0@if5
iptables NAT rules              lo (127.0.0.1)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

# List network namespaces
ip netns list

# Docker creates network namespaces but doesn't always register them
# with iproute2. Find them via container PID:
CONTAINER_PID=$(docker inspect --format '{{.State.Pid}}' demo)

# Enter the container's network namespace from the host
sudo nsenter --target $CONTAINER_PID --net ip addr
# Shows the container's network interfaces from the host

# Create a network namespace manually
sudo ip netns add myns

# Run a command in it
sudo ip netns exec myns ip addr
# 1: lo: <LOOPBACK> mtu 65536 — only loopback, no other interfaces

# Create a veth pair and move one end into the namespace
sudo ip link add veth0 type veth peer name veth1
sudo ip link set veth1 netns myns

# Configure both ends
sudo ip addr add 10.0.0.1/24 dev veth0
sudo ip link set veth0 up
sudo ip netns exec myns ip addr add 10.0.0.2/24 dev veth1
sudo ip netns exec myns ip link set veth1 up
sudo ip netns exec myns ip link set lo up

# Now the host can ping the namespace
ping -c 1 10.0.0.2

This manual process is essentially what container runtimes automate.

Mount Namespace

A mount namespace gives a process its own view of the filesystem. Changes to mount points in one namespace don’t affect others. This is how containers get their own root filesystem.

1
2
3
4
5
6
7
8
9

# Create a new mount namespace and bind-mount a directory
sudo unshare --mount bash

# Inside the new mount namespace:
mount --bind /tmp/myroot /mnt   # Only visible in this namespace
mount | grep mnt

# From the host: /mnt is NOT mounted
mount | grep mnt   # empty

Docker uses this to:

1. Create an overlay filesystem combining the image layers as the container root
2. Bind-mount volume paths from the host into the container
3. Create /proc, /sys, /dev mounts specific to the container

Overlay Filesystems

Container images are made of layers. The overlay filesystem merges them:

Image layer (read-only):    /usr, /lib, /etc  (from ubuntu:24.04)
Image layer (read-only):    /app              (from your COPY instruction)
Container layer (writable): any writes go here
                            ↓
Merged view (what the container sees): everything above unified

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# Inspect Docker's overlay mounts for a container
docker inspect demo --format '{{json .GraphDriver}}'
# Shows LowerDir (image layers), UpperDir (writable layer), MergedDir (union view)

# Look at the actual overlay mount on the host
sudo cat /proc/mounts | grep overlay
# overlay /var/lib/docker/overlay2/abc123/merged overlay
#   lowerdir=/var/lib/docker/overlay2/abc123/l/...,
#   upperdir=/var/lib/docker/overlay2/abc123/diff,
#   workdir=/var/lib/docker/overlay2/abc123/work

UTS Namespace

The simplest namespace — isolates hostname and domain name:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Container has its own hostname
docker run --rm ubuntu hostname
# 3f4a9b2c1d8e   (container ID)

# Set a custom hostname
docker run --rm --hostname mycontainer ubuntu hostname
# mycontainer

# From the host: hostname unchanged
hostname
# your-actual-host

User Namespace

User namespaces are the most powerful and the most nuanced. They map user and group IDs between namespaces: UID 0 (root) inside the container can map to an unprivileged UID on the host.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# Without user namespaces: container root IS host root
docker run --rm ubuntu id
# uid=0(root) gid=0(root) groups=0(root)
# → If this process escapes the container, it's root on the host

# With user namespaces (rootless containers):
podman run --rm ubuntu id
# uid=0(root) gid=0(root)   ← looks like root inside
# But on the host, the process runs as your unprivileged user
# e.g. uid=1000(jmoon)

This is the core of rootless containers — container root maps to an unprivileged host user, so a container escape doesn’t grant host root.

1
2
3
4

# See the UID mapping for a process
cat /proc/$(docker inspect --format '{{.State.Pid}}' demo)/uid_map
#        0       1000          1   ← container UID 0 = host UID 1000
#        1     100000      65536   ← container UIDs 1-65536 = host UIDs 100000+

IPC Namespace

Isolates System V IPC (shared memory, semaphores, message queues) and POSIX message queues. Containers in different IPC namespaces can’t communicate via these mechanisms.

1
2
3

# Share IPC namespace between containers (for legacy apps that communicate via shared memory)
docker run -d --name shmem-writer --ipc=shareable myapp-writer
docker run -d --name shmem-reader --ipc=container:shmem-writer myapp-reader

nsenter: Entering Namespaces

nsenter is the tool for entering a running process’s namespaces from the host — invaluable for debugging:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

CONTAINER_PID=$(docker inspect --format '{{.State.Pid}}' demo)

# Enter all namespaces of the container (like docker exec but lower-level)
sudo nsenter --target $CONTAINER_PID \
  --mount --uts --ipc --net --pid \
  -- /bin/bash

# Enter only the network namespace (keep host filesystem and PID view)
sudo nsenter --target $CONTAINER_PID --net -- ip route

# Useful: run a tool that's on the HOST inside the container's network namespace
# (e.g. tcpdump isn't in the container image but you need to capture its traffic)
sudo nsenter --target $CONTAINER_PID --net -- \
  tcpdump -i eth0 -w /tmp/container-traffic.pcap

cgroups: Resource Control

While namespaces control what a process can see, control groups (cgroups) control what it can use — CPU, memory, I/O, network bandwidth, and more.

Linux has two versions:

• cgroups v1: Original implementation, controllers are mounted separately, complex hierarchy
• cgroups v2: Unified hierarchy, all controllers under /sys/fs/cgroup, the modern standard (default in recent kernels and distros)

1
2
3
4
5
6

# Check which version your system uses
stat -fc %T /sys/fs/cgroup
# tmpfs → v1
# cgroup2fs → v2

# Most modern systems (Ubuntu 22.04+, RHEL 9+) use v2

The cgroup Hierarchy

cgroups are organized as a tree. A child group inherits limits from its parent and can set stricter limits but not looser ones.

/sys/fs/cgroup/
├── system.slice/           ← systemd unit slice
│   ├── docker.service/
│   └── sshd.service/
├── user.slice/
│   └── user-1000.slice/
└── docker/                 ← Docker's container cgroups
    ├── <container-id-1>/
    └── <container-id-2>/

1
2
3
4

# See a container's cgroup
CONTAINER_ID=$(docker inspect --format '{{.Id}}' demo)
cat /proc/$(docker inspect --format '{{.State.Pid}}' demo)/cgroup
# 0::/docker/<container-id>

Memory Control

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

# Run a container with a 256MB memory limit
docker run -d --memory=256m --memory-swap=256m nginx

# Inspect limits from the host (cgroups v2)
CGROUP_PATH="/sys/fs/cgroup/docker/$CONTAINER_ID"
cat "$CGROUP_PATH/memory.max"          # hard limit
cat "$CGROUP_PATH/memory.current"      # current usage
cat "$CGROUP_PATH/memory.swap.current" # swap usage

# Memory events (OOM kills, throttling)
cat "$CGROUP_PATH/memory.events"
# low 0
# high 0
# max 0
# oom 3          ← this container has OOMKilled 3 times
# oom_kill 3

When a container exceeds its memory limit, the kernel’s OOM killer sends SIGKILL to a process in the cgroup. Kubernetes exposes this as OOMKilled in kubectl describe pod.

CPU Control

cgroups v2 uses a CPU weight system and CPU bandwidth control:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

# Run with CPU limits
docker run -d --cpus=1.5 nginx
# Translates to: cpu.max = "150000 100000" (150ms per 100ms period = 1.5 CPUs)

CGROUP_PATH="/sys/fs/cgroup/docker/$CONTAINER_ID"

# CPU bandwidth (hard limit)
cat "$CGROUP_PATH/cpu.max"
# 150000 100000   ← 150ms every 100ms = 1.5 CPUs hard cap

# CPU weight (soft limit / scheduling priority)
cat "$CGROUP_PATH/cpu.weight"
# 100   ← default weight; higher = more CPU when contention exists

# CPU statistics
cat "$CGROUP_PATH/cpu.stat"
# usage_usec 12345678     ← total CPU microseconds used
# user_usec 9876543
# system_usec 2469135
# throttled_usec 456789   ← time throttled due to cpu.max limit
# throttle_periods 123

High throttled_usec means the container is CPU-throttled — it wants more CPU than its limit allows. This manifests as increased latency even when the host has spare CPU capacity.

1
2
3
4
5
6

# Check throttling for all Docker containers
for dir in /sys/fs/cgroup/docker/*/; do
  id=$(basename "$dir")
  throttled=$(grep throttled_usec "$dir/cpu.stat" | awk '{print $2}')
  echo "$id: ${throttled}µs throttled"
done

I/O Control

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

# Run with I/O limits
docker run -d \
  --device-write-bps /dev/sda:10mb \
  --device-read-bps /dev/sda:50mb \
  nginx

CGROUP_PATH="/sys/fs/cgroup/docker/$CONTAINER_ID"

# I/O limits (cgroups v2)
cat "$CGROUP_PATH/io.max"
# 8:0 rbps=52428800 wbps=10485760   ← device 8:0, read 50MB/s, write 10MB/s

# I/O statistics
cat "$CGROUP_PATH/io.stat"
# 8:0 rbytes=12345678 wbytes=9876543 rios=1234 wios=567 dbytes=0 dios=0

Creating cgroups Manually

You don’t need Docker to use cgroups:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

# Create a new cgroup (cgroups v2)
sudo mkdir /sys/fs/cgroup/myapp

# Set a 256MB memory limit
echo "268435456" | sudo tee /sys/fs/cgroup/myapp/memory.max

# Set a 0.5 CPU limit (50ms per 100ms period)
echo "50000 100000" | sudo tee /sys/fs/cgroup/myapp/cpu.max

# Move the current shell into this cgroup
echo $$ | sudo tee /sys/fs/cgroup/myapp/cgroup.procs

# Everything this shell spawns is now subject to the limits
# Try to exceed memory: python3 -c "x = 'a' * 300_000_000"
# → Killed (OOM killer fires)

# Remove the cgroup (must be empty first)
rmdir /sys/fs/cgroup/myapp

How Docker Puts It Together

When you run docker run --memory=256m --cpus=1 -v /data:/data nginx, here’s what actually happens:

1. Pull/verify image layers

2. Create overlay filesystem
   → Mount namespace: new overlay mount from image layers
   → Writable layer added on top

3. Create network namespace
   → New network namespace
   → Create veth pair: one end in host (connected to docker0 bridge), one in container
   → Assign IP from Docker's subnet (172.17.x.x)
   → Configure iptables NAT rules on host for port publishing

4. Create PID namespace
   → New PID namespace
   → Container process will be PID 1 within it

5. Create UTS namespace
   → Set container hostname (from --hostname or container ID)

6. Create IPC namespace
   → Isolated IPC objects

7. Set up cgroups
   → Create /sys/fs/cgroup/docker/<container-id>/
   → Write memory.max = 268435456 (256MB)
   → Write cpu.max = "100000 100000" (1 CPU)

8. Clone the process into all namespaces
   → syscall: clone(flags=CLONE_NEWPID|CLONE_NEWNET|CLONE_NEWNS|CLONE_NEWUTS|CLONE_NEWIPC)

9. Add process to cgroup
   → Write PID to /sys/fs/cgroup/docker/<container-id>/cgroup.procs

10. exec() the entrypoint
    → execve("/docker-entrypoint.sh", ...)

All of this is orchestrated by the container runtime (containerd, runc) based on an OCI runtime spec. The runtime spec is a JSON document describing exactly which namespaces, cgroup limits, mounts, and process to create.

1
2

# Look at the OCI runtime spec Docker generated for a container
sudo cat /run/containerd/io.containerd.runtime.v2.task/moby/$CONTAINER_ID/config.json | jq .

Seccomp and Capabilities: The Other Security Layers

Namespaces and cgroups are isolation and resource control. Two more kernel features complete the container security picture:

Capabilities break root’s privileges into ~40 granular permissions. A container process might run as root (UID 0) inside the namespace but have only the capabilities it actually needs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# Default Docker capabilities (dropped from full root)
docker run --rm ubuntu capsh --print
# Current: cap_chown, cap_dac_override, cap_fowner, cap_fsetid,
#          cap_kill, cap_setgid, cap_setuid, cap_setpcap, cap_net_bind_service,
#          cap_net_raw, cap_sys_chroot, cap_mknod, cap_audit_write, cap_setfcap

# Drop all capabilities
docker run --rm --cap-drop=ALL ubuntu capsh --print
# Current: =   (empty — no capabilities)

# Drop all, add only what's needed
docker run --rm --cap-drop=ALL --cap-add=NET_BIND_SERVICE nginx

Seccomp filters system calls using Berkeley Packet Filter (BPF) rules. Docker’s default seccomp profile blocks ~44 syscalls that containers rarely need but that are historically exploitable:

1
2
3
4
5

# Run with a custom seccomp profile
docker run --security-opt seccomp=./my-seccomp.json nginx

# Run without any seccomp filtering (not recommended)
docker run --security-opt seccomp=unconfined nginx

Practical Implications for Container Operations

Why “OOMKilled” happens: The kernel’s OOM killer fires when a process in the cgroup tries to allocate memory beyond memory.max. Set limits thoughtfully — too tight and you’ll see OOMKills under load; too loose and a misbehaving container can starve other workloads.

Why CPU throttling hurts latency: Even when the host has spare CPU, a container at its cpu.max limit gets throttled. P99 latency spikes while CPU usage looks healthy. The fix is raising the CPU limit, not adding more containers.

Why container networking has overhead: Each container packet traverses virtual ethernet, the Docker bridge, iptables rules, and back. For high-throughput services, host networking (--net=host) removes this overhead at the cost of namespace isolation.

Why rootless containers matter: User namespace mapping means a container escape doesn’t grant host root. All modern container runtimes (Podman by default, Docker with rootless mode) support this.

Why pid=host is dangerous: --pid=host removes PID namespace isolation — the container can see and send signals to all host processes. Rarely needed, often requested, generally a bad idea.

Why privileged containers are effectively uncontained: --privileged disables seccomp filtering, grants all capabilities, and allows access to all host devices. It’s essentially running a process on the host with a different filesystem. Use it only when you explicitly need host-level access (like building container images or running a container runtime inside a container).

Tools for Exploration

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

# lsns — list all namespaces on the system
sudo lsns

# pstree with namespace indicators
ps -eo pid,pidns,netns,mntns,cmd --sort pid | head -30

# cinf — container runtime introspection tool
# https://github.com/mhausenblas/cinf
cinf --namespace $CONTAINER_PID

# bpftrace — trace cgroup events
sudo bpftrace -e 'tracepoint:cgroup:cgroup_mkdir { printf("new cgroup: %s\n", str(args->path)); }'

# systemd-cgls — view cgroup tree
systemd-cgls

# systemd-cgtop — cgroup resource usage, like top for cgroups
systemd-cgtop

Understanding namespaces and cgroups transforms container operations from “I run docker commands and things happen” to genuine comprehension of the isolation model. The next time a container misbehaves, OOMKills, throttles, or leaks into the host, you’ll know exactly where in the kernel to look.