Kubernetes Debugging in Production

Kubernetes debugging is not hard once you have a repeatable mental model. The problem is that most engineers learn it reactively — something breaks in production, they grep through logs for an hour, stumble on the answer, and move on without internalizing the pattern. The next incident feels just as novel.

This post is an attempt to systematize the investigation. Every Kubernetes failure mode falls into one of six categories: the pod crashes immediately on startup, the pod gets OOMKilled, the pod never schedules, DNS resolution fails, network policies block traffic, or TLS certificates are expired or misconfigured. Each category has a canonical set of commands and a specific thing to look for. Once you know the flowchart, most incidents resolve in under ten minutes.

The commands below assume kubectl with appropriate cluster access. Everything that works in staging works in production — the difference is how much pressure you feel while running it.

The Five-Step Mental Model

Before going through each failure category, the general sequence that applies to every broken pod:

1. Status — kubectl get pod <name> -o wide
What state is the pod in? The status column — Pending, CrashLoopBackOff, OOMKilled, Running — tells you which category you are in before you look at anything else. The -o wide flag adds the node name and pod IP, which matter more than you would expect.

2. Events — kubectl describe pod <name>, Events section
The events timeline is the most information-dense output in Kubernetes. It records every scheduling decision, image pull attempt, container start, kill, and back-off. Read it chronologically. It tells you what Kubernetes did, in order, and usually explains why things went wrong.

3. Logs — kubectl logs <name> --previous
What did the application output before it died? The --previous flag is the one you need in a crash loop; it shows the last completed container run rather than the current (empty or truncated) one.

4. Describe — the rest of kubectl describe pod
Verify the configuration: image, resource requests and limits, probes, volume mounts, environment variables, node affinity, tolerations. Configuration errors are invisible until you look.

5. Node — kubectl describe node <name>
If steps 1–4 do not explain the failure, the problem is beneath the pod layer. Node conditions (memory pressure, disk pressure), allocated resources, and node-level events are where to look.

The split between infrastructure problems and application problems runs through step 5. Signs it is infrastructure: multiple pods failing simultaneously, the pod never leaves Pending, or node conditions are not Ready. Signs it is application: the pod schedules and pulls the image, then immediately crashes or produces error output.

CrashLoopBackOff

CrashLoopBackOff means the container is exiting repeatedly and Kubernetes is applying an exponential back-off before restarting it. The back-off starts at 10 seconds and doubles — 10s, 20s, 40s, 80s, 160s — capping at 5 minutes. By the time you are looking at a pod with 15 restarts, the gap between crashes and restarts is 5 minutes. The application may only be alive for a few seconds in each cycle.

Reading the exit code

The exit code is the first thing to check:

1
2

kubectl get pod <pod-name> \
  -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}'

Or read it from kubectl describe pod in the Last State block:

Last State:     Terminated
  Reason:       Error
  Exit Code:    137
  Started:      Fri, 23 May 2026 10:32:01 +0000
  Finished:     Fri, 23 May 2026 10:32:15 +0000

Exit code meanings:

Exit codes 137 and 143 are the most common in production. Exit 137 is almost always either OOMKilled (the kernel killed the process) or a runaway container killed by a liveness probe. Exit 143 means the container received SIGTERM during a graceful shutdown window and did not exit in time—or the liveness probe declared the container unhealthy and triggered a SIGTERM.

Logs from the last crash

1

kubectl logs <pod-name> --previous

If the container exits before writing anything useful, the logs will be empty. In that case, check whether the problem is at entrypoint evaluation time rather than application startup—a missing environment variable, a missing secret mount, or a wrong binary path. The describe pod output will show the exact command being run under Containers → Command.

Health probe timing

A common pattern for spurious CrashLoopBackOff on pods that otherwise work: the liveness probe fires before the application has finished initializing and repeatedly kills the container. The application starts normally, handles no traffic, then gets killed 30 seconds in because the probe decided it was unhealthy. Logs show a clean startup. The application never panicked. The pod just dies.

Kubernetes 1.18 added the startupProbe for exactly this scenario. It runs exclusively during startup, and once it succeeds, the liveness probe takes over. Until the startup probe succeeds, the liveness probe does not fire:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

startupProbe:
  httpGet:
    path: /healthz
    port: 8080
  failureThreshold: 30      # Try up to 30 times
  periodSeconds: 10         # 10 seconds between attempts
  # Up to 300 seconds for the app to start
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  periodSeconds: 15
  failureThreshold: 3

The startupProbe prevents the liveness probe from interfering during slow starts. The initialDelaySeconds field on livenessProbe is the older alternative; startupProbe is the correct tool for this problem.

Ephemeral debugging

When the container crashes too quickly to exec into, use an ephemeral debug container that shares the pod’s namespaces but runs a separate image:

1

kubectl debug -it <pod-name> --image=busybox:latest --target=<container-name>

From inside the ephemeral container, you can inspect the filesystem at /proc/<pid>/root, check environment variables, and test whether expected files and commands are present. The ephemeral container shares the pod’s network namespace, so you can also test connectivity from the pod’s IP.

OOMKilled

OOMKilled means the Linux kernel’s OOM killer terminated the process because the container’s cgroup memory limit was exceeded. Exit code 137 and Reason: OOMKilled in kubectl describe pod confirm it:

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137
  Started:      Fri, 23 May 2026 14:02:00 +0000
  Finished:     Fri, 23 May 2026 14:02:31 +0000

Note the distinction: Reason: OOMKilled versus Reason: Error with exit code 137. A SIGKILL sent for a reason other than OOM (a failing liveness probe, an external kill -9) also produces exit code 137 but shows Reason: Error. Look at the Reason field, not just the exit code.

Resource visibility

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

# Current usage for a specific pod
kubectl top pod <pod-name> -n <namespace>

# All pods in a namespace, sorted by memory
kubectl top pod -n <namespace> --sort-by=memory

# Node utilization
kubectl top node

# Allocated resources breakdown on a node
kubectl describe node <node-name>
# Look for the "Allocated resources" section:
# Resource           Requests   Limits
# --------           --------   ------
# cpu                3750m      4000m
# memory             5Gi        6Gi

From inside a running container, the actual cgroup memory usage (not the inflated value top may show):

1
2
3
4
5

# cgroup v2 (Kubernetes 1.25+ default on most distros)
kubectl exec -it <pod-name> -- cat /sys/fs/cgroup/memory.current

# cgroup v1
kubectl exec -it <pod-name> -- cat /sys/fs/cgroup/memory/memory.usage_in_bytes

The top command inside a container can lie — it reads from /proc/meminfo which on some kernel configurations reflects the node’s total memory, not the container’s limit. Trust the cgroup values.

QoS classes and eviction order

Kubernetes assigns one of three Quality of Service classes to every pod based on its resource spec. When a node is under memory pressure and must evict pods, it evicts in BestEffort → Burstable → Guaranteed order:

Guaranteed: requests equal limits for all containers. These pods are evicted last and are never OOMKilled due to node-level pressure (only if they exceed their own limits).

1
2
3
4
5
6
7

resources:
  requests:
    memory: "512Mi"
    cpu: "250m"
  limits:
    memory: "512Mi"
    cpu: "250m"

Burstable: at least one container has a request or limit, but requests do not equal limits. Evicted second.

1
2
3
4
5
6
7

resources:
  requests:
    memory: "256Mi"
    cpu: "100m"
  limits:
    memory: "512Mi"
    cpu: "500m"

BestEffort: no requests or limits set on any container. Evicted first. Never use this for anything stateful or latency-sensitive.

The common mistake is setting limits without requests. This looks like Guaranteed to the scheduler (it sees a resource specification) but Kubernetes actually creates a Burstable pod if requests are absent. Set both, and set them based on observed usage.

JVM containers

Java applications are the most frequent OOMKilled offenders because the JVM manages its own heap separately from the OS, and without container-awareness, it will allocate heap based on the total node memory rather than the container limit. A pod with a 512 MiB limit on a 64 GiB node will have the JVM try to allocate a multi-gigabyte heap and immediately get killed.

The fix is two flags:

-XX:+UseContainerSupport       # Read cgroup limits instead of node memory
-XX:MaxRAMPercentage=75.0      # Allocate up to 75% of cgroup limit as heap

UseContainerSupport is available from Java 8u191, Java 10, and all Java 11+ builds. It is not enabled by default in older Java 8 releases. Without it, -Xmx is calculated relative to node memory.

MaxRAMPercentage=75.0 leaves 25% of the container’s memory limit for the JVM’s off-heap needs: metaspace, code cache, direct buffers, and native libraries. Setting -Xmx too close to the limit (e.g., -Xmx490m on a 512 MiB container) leaves no room for these and results in OOMKilled despite having set a limit explicitly.

Vertical Pod Autoscaler for right-sizing

VPA in recommendation-only mode observes actual usage and suggests correct requests and limits without making changes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: my-app-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  updatePolicy:
    updateMode: "Off"

After a few days of traffic, inspect the recommendations:

1

kubectl describe vpa my-app-vpa

The output includes lowerBound, target, and upperBound for each container. Use target for requests and upperBound (or 1.2 × target) for limits. This eliminates the guesswork from initial sizing and catches gradual memory growth before it causes incidents.

Pending Pods

A pod in Pending state has not been scheduled to any node. The scheduler has evaluated all nodes and rejected all of them, or it is waiting for a dependency (a PVC, an init container) before it can proceed.

Reading the scheduler’s rejection reason

1

kubectl describe pod <pod-name>

The Events section will contain a FailedScheduling event with a message that looks like:

0/5 nodes are available: 2 Insufficient cpu, 3 node(s) had taint {dedicated: worker}, that the pod didn't tolerate.

Read this as: “the scheduler evaluated 5 nodes. 2 failed because they do not have enough CPU. 3 have a taint this pod does not tolerate.” The numbers add up to more than 5 because multiple failure reasons can apply to the same node.

Common rejection messages and what they mean:

Node resource exhaustion

1
2
3
4
5

kubectl describe node <node-name>
# Under "Allocated resources":
# Resource           Requests     Limits
# cpu                3900m (97%)  4000m (100%)
# memory             7400Mi (92%) 8192Mi (100%)

When CPU or memory requests are at 95%+ of node capacity, new pods cannot schedule even if the node appears to have headroom in kubectl top node (which shows actual usage, not allocated requests). The scheduler respects requests, not actual utilization.

Taints and tolerations

1
2
3
4
5
6

# Check node taints
kubectl describe node <node-name> | grep -A5 Taints

# Example output:
# Taints: dedicated=worker:NoSchedule
#         spot-instance=true:NoExecute

A NoSchedule taint prevents pods without the matching toleration from scheduling. A NoExecute taint additionally evicts pods already running on the node. To run on a tainted node, the pod needs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

tolerations:
- key: "dedicated"
  operator: "Equal"
  value: "worker"
  effect: "NoSchedule"
- key: "spot-instance"
  operator: "Equal"
  value: "true"
  effect: "NoExecute"
  tolerationSeconds: 300    # Evict after 300s if still tainted

PVC binding failures

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

kubectl describe pvc <pvc-name> -n <namespace>
# Events section shows why it is not binding:

# StorageClass not found:
# "storageclass.storage.k8s.io/fast-ssd not found"

# No suitable PV:
# "no persistent volumes available for this claim and no storage class is set"

# Provisioner not running:
# "waiting for a volume to be created, either by external provisioner 'ebs.csi.aws.com'..."

For dynamic provisioning, check that the CSI driver pods are running:

1

kubectl get pod -n kube-system | grep csi

DNS Failures

Every pod’s /etc/resolv.conf is injected by kubelet at startup:

nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5 edns0

The nameserver is the ClusterIP of the kube-dns service. The search domains are appended to unqualified names before resolution. The ndots:5 option means any name with fewer than 5 dots is treated as relative and tried against each search domain before being tried as an absolute name.

This has an unintuitive performance consequence: a query for api.github.com (2 dots, fewer than 5) is tried as api.github.com.default.svc.cluster.local, then api.github.com.svc.cluster.local, then api.github.com.cluster.local, then api.github.com (the actual target). Each failed search domain query hits CoreDNS and possibly an upstream resolver. Under load, this creates measurable DNS latency. The fix is to always use FQDNs for external names (trailing dot: api.github.com.) or to lower ndots in the pod’s dnsConfig.

Testing DNS from inside a pod

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Run a temporary debug pod with nslookup
kubectl run -it --rm dns-debug --image=busybox --restart=Never \
  -- nslookup kubernetes.default

# Test a specific service in another namespace
kubectl run -it --rm dns-debug --image=busybox --restart=Never \
  -- nslookup my-service.production.svc.cluster.local

# Explicit DNS server test
kubectl run -it --rm dns-debug --image=busybox --restart=Never \
  -- nslookup my-service.default.svc.cluster.local 10.96.0.10

The FQDN format for Kubernetes services is <service-name>.<namespace>.svc.cluster.local. Cross-namespace service calls require the full qualified name; the short form only works within the same namespace.

CoreDNS not running

1
2
3
4
5
6
7
8

kubectl get pod -n kube-system -l k8s-app=kube-dns
# Should show 2+ Running pods for a production cluster

kubectl logs -n kube-system -l k8s-app=kube-dns --tail=100
# Look for: connection refused, plugin errors, SERVFAIL patterns

kubectl describe service -n kube-system kube-dns
# Verify the ClusterIP matches the nameserver in /etc/resolv.conf

Network policy blocking port 53

This is the most common DNS failure after default-deny policies are applied. The pods can’t resolve names because the egress rule to kube-dns on port 53 (both UDP and TCP) was not included:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: production
spec:
  podSelector: {}       # Apply to all pods in namespace
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53

Both UDP and TCP on port 53 are required. DNS uses UDP for normal queries and falls back to TCP for responses larger than 512 bytes (common with DNSSEC or large TXT records).

CoreDNS configuration

The CoreDNS ConfigMap in kube-system controls forwarding and caching:

1

kubectl get configmap -n kube-system coredns -o yaml

The Corefile inside typically looks like:

.:53 {
    errors
    health {
       lameduck 5s
    }
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
       pods insecure
       fallthrough in-addr.arpa ip6.arpa
       ttl 30
    }
    prometheus :9153
    forward . /etc/resolv.conf {
       max_concurrent 1000
    }
    cache 30
    loop
    reload
    loadbalance
}

The forward directive controls where non-cluster DNS queries go. The cache 30 line caches responses for 30 seconds. If you are seeing DNS failures under high load, increasing the cache TTL or adding additional CoreDNS replicas is more effective than tuning application retry logic.

Network Policy Blocks

Network policies are enforced by the CNI plugin, not by Kubernetes itself. If your CNI does not support NetworkPolicy (Flannel without a policy controller, for example), kubectl get networkpolicy will show your policies but they will have no effect. Confirm policy enforcement is active before assuming network policies are responsible for a connectivity failure.

Checking what policies apply

1
2
3
4
5

# All network policies across all namespaces
kubectl get networkpolicy -A

# Full spec for a specific policy
kubectl describe networkpolicy <name> -n <namespace>

The critical fields in the describe output:

Pod Selector:     app=my-app         <- Which pods does this policy govern?
Allowing ingress traffic from:
  Pods Selectors: app=frontend       <- Only these pods can send traffic in
  Ports: 8080/TCP
Allowing egress traffic to:
  Ports: 5432/TCP                    <- Only outbound to port 5432 allowed

An empty Pod Selector ({}) means the policy applies to all pods in the namespace.

The default-deny pattern

Many security-conscious setups apply a default-deny policy to a namespace before adding allow rules:

1
2
3
4
5
6
7
8
9

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

This policy allows no traffic at all — neither inbound nor outbound — from any pod in the namespace. Once it is applied, every connection must be explicitly permitted. If pods suddenly cannot reach anything after a namespace migration, this policy is usually the reason.

Testing connectivity directly

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# From a source pod, attempt to reach a destination
kubectl exec -it <source-pod> -- curl -v http://<destination-service>:8080

# Test port reachability with netcat
kubectl exec -it <source-pod> -- nc -zv <destination-pod-ip> 8080
# -z: scan only (don't send data), -v: verbose

# If the source pod lacks curl or nc, use an ephemeral debug container
kubectl debug -it <source-pod> --image=nicolaka/netshoot --target=<container>
# nicolaka/netshoot includes curl, wget, nc, dig, nslookup, tcpdump, iperf3

The nicolaka/netshoot image is the standard Kubernetes network debugging image. It runs in the target pod’s network namespace, so all connectivity tests reflect what the application container can actually reach.

From inside netshoot:

1
2
3
4
5
6
7
8

# Test that a service resolves and connects
curl -v http://my-service.default.svc.cluster.local:8080/health

# Capture raw traffic (requires NET_ADMIN or running as privileged)
tcpdump -i eth0 -n port 8080

# Trace routing path
traceroute my-service.default.svc.cluster.local

CNI-specific tooling

Cilium: The hubble CLI provides flow-level visibility into what traffic was allowed and dropped:

1
2
3
4
5

# Observe flows for a specific pod (requires Hubble to be enabled)
cilium hubble observe --pod default/my-pod-abc123 --last 100

# Look for "dropped" flows and the policy that dropped them
# Output includes: source, destination, verdict (FORWARDED/DROPPED), drop reason

Calico: calicoctl shows Calico-native policies that may not appear in kubectl get networkpolicy (GlobalNetworkPolicy resources):

1
2

calicoctl get networkpolicies -A
calicoctl get globalnetworkpolicies

The forgotten egress rules

The most common network policy mistake in practice is applying ingress rules without the corresponding egress rules. A pod that is correctly allowed to receive traffic from the frontend cannot make outbound calls to the database if there is no egress rule permitting it. The failure looks like a timeout on the database connection rather than a refused connection, which sends engineers looking at database logs instead of network policies.

Egress rules needed for a typical backend pod:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

egress:
# DNS — always required
- to:
  - namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: kube-system
  ports:
  - protocol: UDP
    port: 53
  - protocol: TCP
    port: 53

# Database
- to:
  - podSelector:
      matchLabels:
        app: postgres
  ports:
  - protocol: TCP
    port: 5432

# External HTTPS
- to:
  - ipBlock:
      cidr: 0.0.0.0/0
      except:
      - 169.254.169.254/32   # Block IMDS if not needed
  ports:
  - protocol: TCP
    port: 443

Certificate Errors

TLS failures in Kubernetes fall into two categories: application pods failing to establish connections to services (expired or untrusted certificates), and the Kubernetes API server failing to call admission webhooks (webhook certificate issues).

Application TLS debugging

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# Verbose TLS handshake from inside a pod
kubectl exec -it <pod-name> -- curl -v https://my-service:443

# Look for output like:
# * SSL certificate verify result: certificate has expired (10), continuing anyway
# * OpenSSL SSL_connect: SSL_ERROR_SYSCALL
# * TLSv1.3 (IN), TLS alert, certificate expired (509)

# Show the certificate details for a service
kubectl exec -it <pod-name> -- \
  openssl s_client -connect my-service:443 -showcerts 2>/dev/null | \
  openssl x509 -noout -subject -dates

To check an existing TLS secret directly:

1
2
3
4
5
6
7
8
9

kubectl get secret <tls-secret-name> -n <namespace> \
  -o jsonpath='{.data.tls\.crt}' | base64 -d | \
  openssl x509 -noout -subject -issuer -dates

# Output:
# subject= /CN=my-service.default.svc
# issuer= /CN=my-cluster-ca
# notBefore=Nov 23 10:15:00 2025 GMT
# notAfter=May 23 10:15:00 2026 GMT

cert-manager certificate issuance chain

cert-manager manages certificates through a chain of resources: Certificate → CertificateRequest → Order (for ACME issuers). Failures can occur at any stage.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

# Top-level status
kubectl get certificate -A
# READY=True means the certificate is current and valid
# READY=False means it is failing to issue or renew

# Detail on a specific certificate
kubectl describe certificate <name> -n <namespace>
# Look at Conditions:
# Type: Ready, Status: False, Reason: Pending, Message: "Waiting on CertificateRequest..."

# Follow the chain down
kubectl describe certificaterequest <name> -n <namespace>
# If ACME, follow to Order:
kubectl describe order <name> -n <namespace>
# Shows challenge status — HTTP-01 challenges require port 80 to be reachable
# DNS-01 challenges require a working DNS provider credential

The most common cert-manager failure in production: the ACME challenge endpoint is not reachable. For HTTP-01 validation, Let’s Encrypt must be able to reach http://<your-domain>/.well-known/acme-challenge/<token>. If your Ingress controller does not route that path, or a firewall blocks inbound HTTP, the challenge fails and the certificate never issues.

Webhook certificate failures

When an admission webhook’s certificate expires, kubectl apply starts failing with:

Error from server (InternalError): error when creating "resource.yaml": 
Internal error occurred: failed calling webhook "validation.my-operator.io": 
Post "https://my-operator-webhook.kube-system.svc:443/validate": 
x509: certificate has expired or is not yet valid

Diagnose:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

# Find the webhook configuration
kubectl get validatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations

# Check the caBundle field — this is what the API server uses to verify the webhook
kubectl get validatingwebhookconfiguration <name> \
  -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | base64 -d | \
  openssl x509 -noout -dates

# Check the TLS secret the webhook pod uses
kubectl get secret <webhook-tls-secret> -n <webhook-namespace> \
  -o jsonpath='{.data.tls\.crt}' | base64 -d | \
  openssl x509 -noout -dates

# The caBundle in the webhook configuration and the cert in the secret
# must match. If cert-manager is managing the webhook cert via cert-manager.io/inject-ca-from
# annotation, the caBundle is injected automatically. If it is absent or mismatched,
# the controller-manager CA injector may not be running.
kubectl get pod -n cert-manager | grep cainjector

Emergency bypass if a broken webhook is blocking all applies (use with caution):

1
2
3
4

# Delete the webhook configuration temporarily
kubectl delete validatingwebhookconfiguration <name>
# Apply your fix
# Re-apply the webhook configuration

Control plane certificate expiry (kubeadm clusters)

For clusters managed with kubeadm, all control plane certificates expire after one year by default:

1

kubeadm certs check-expiration

Output:

CERTIFICATE                       EXPIRES                  RESIDUAL TIME   CA
admin.conf                         May 23, 2027 10:15 UTC   364d            no
apiserver                          May 23, 2027 10:15 UTC   364d            no
apiserver-etcd-client              May 23, 2027 10:15 UTC   364d            no
apiserver-kubelet-client           May 23, 2027 10:15 UTC   364d            no
controller-manager.conf            May 23, 2027 10:15 UTC   364d            no
etcd-healthcheck-client            May 23, 2027 10:15 UTC   364d            no
etcd-peer                          May 23, 2027 10:15 UTC   364d            no
etcd-server                        May 23, 2027 10:15 UTC   364d            no
front-proxy-client                 May 23, 2027 10:15 UTC   364d            no
scheduler.conf                     May 23, 2027 10:15 UTC   364d            no

Renew before expiry:

1
2

kubeadm certs renew all
systemctl restart kubelet

kubeadm automatically renews certificates when a control plane upgrade is performed, which is why clusters that are regularly upgraded rarely hit this problem. Clusters that are never upgraded and are more than a year old are the risk.

The Debugging Toolkit

Events sorted by time

1

kubectl get events --sort-by=.lastTimestamp -n <namespace>

This is the most underused command in daily Kubernetes work. The events log is a chronological record of everything that happened in a namespace — scheduling decisions, image pulls, probe failures, back-offs, volume mounts. It is far more readable than searching through kubectl describe output for individual pods, especially during incidents involving multiple pods or a rolling deployment that went sideways.

1
2
3
4
5

# Watch events in real-time
kubectl get events -n <namespace> --watch

# Filter to events for a specific object
kubectl get events --field-selector involvedObject.name=<pod-name> -n <namespace>

Rollout inspection and rollback

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# Watch a rolling deployment in progress
kubectl rollout status deployment/<name> -n <namespace>
# Blocks until complete or failed, prints replica counts

# See revision history
kubectl rollout history deployment/<name>

# Roll back to the previous revision
kubectl rollout undo deployment/<name>

# Roll back to a specific revision
kubectl rollout undo deployment/<name> --to-revision=3

kubectl rollout undo is the fastest response to a bad deploy. It immediately starts rolling instances back to the previous ReplicaSet without any manifests or CI/CD involvement.

Node-level debugging

When pods on a specific node are failing and the problem is not visible from the pod layer, debug the node directly:

1

kubectl debug node/<node-name> -it --image=ubuntu

This creates a pod running in the node’s host namespaces with the node’s root filesystem mounted at /host. From there:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# Switch to the node's filesystem
chroot /host

# Check kubelet status
systemctl status kubelet
journalctl -u kubelet -n 100

# Check container runtime
crictl ps -a        # containerd
# or
docker ps -a        # Docker (older clusters)

# Check disk pressure
df -h

Multi-pod log streaming

1
2
3
4
5
6

# Logs from all pods with a label selector
kubectl logs -l app=my-app --all-containers=true --tail=100 -f

# If stern is installed (https://github.com/stern/stern)
stern my-app -n production
# Shows logs from all my-app* pods, color-coded by pod name, with timestamps

Port-forward for service testing

1
2
3
4
5
6
7
8

# Forward to a pod
kubectl port-forward pod/<pod-name> 8080:8080

# Forward to a service (routes to one backend pod)
kubectl port-forward service/<service-name> 8080:8080

# Access in another terminal
curl -v http://localhost:8080/health

Port-forwarding bypasses Ingress, load balancers, and network policies on the ingress path. If the application responds over port-forward but not through the normal route, the problem is in the routing layer (Ingress configuration, Service selector, or network policy on the Ingress controller) rather than in the application.

Quick Reference

Failure mode         First command                        Key output to find
─────────────────────────────────────────────────────────────────────────────────
CrashLoopBackOff     kubectl logs <pod> --previous        Application error message
                     kubectl describe pod <pod>           Exit Code, Last State Reason
OOMKilled            kubectl describe pod <pod>           Reason: OOMKilled
                     kubectl top pod <pod>                Memory usage vs limit
Pending              kubectl describe pod <pod>           Events: 0/N nodes available
                     kubectl describe node <node>         Allocated resources, Taints
DNS failure          kubectl run -it debug --image=       nslookup output
                       busybox -- nslookup <svc>
                     kubectl get pod -n kube-system       CoreDNS pods running?
Network policy       kubectl describe networkpolicy       Ingress/Egress rules
                     kubectl debug -it <pod>              curl/nc from pod's network
                       --image=nicolaka/netshoot
Cert error           openssl x509 -noout -dates           notAfter date
                     kubectl describe certificate         Conditions: Reason, Message
                     kubectl describe order               Challenge status (ACME)
Any pod              kubectl get events                   Chronological incident log
                       --sort-by=.lastTimestamp

The five-step sequence — status, events, logs, describe, node — applies to every entry in that table. The specific commands per failure mode narrow down which step you spend time on. Most failures resolve at step 2 (events) or step 3 (logs). When they do not, the problem is almost always configuration (step 4) or a degraded node (step 5).