Karpenter: Kubernetes Node Autoscaling Done Right

May 23, 2026 19 min read DevOps

kubernetesawseksclouddevopsinfrastructure-as-codecost-optimization

The Kubernetes cluster-autoscaler works by watching for unschedulable pods, finding a node group whose launch template would accommodate them, and incrementing that node group’s desired count. AWS then starts an EC2 instance from the group’s predefined launch template. When nodes are underutilized, the autoscaler drains and terminates them. This works, but it has a fundamental constraint: you must define your node groups in advance, each with a fixed instance type and configuration. Scaling up means picking which predefined box to add more of.

Karpenter removes this constraint. It watches directly for unschedulable pods and, rather than picking a node group, picks the cheapest EC2 instance type that satisfies the pod’s resource requests and scheduling constraints — from any family, any size, any generation — and provisions it directly via the EC2 Fleet API. The decision happens in 45–60 seconds. There is no node group to manage. When nodes become underutilized, Karpenter consolidates workloads and terminates the underused nodes, potentially replacing multiple smaller nodes with fewer larger ones.

For clusters with variable or unpredictable workloads, Karpenter routinely reduces EC2 spend by 25–40% through bin-packing alone, and by 50–90% on batch or CI/CD workloads where Spot is appropriate. The configuration is simpler once you understand the two core CRDs. The operational model is different enough from cluster-autoscaler that it warrants careful understanding before deploying to production.

How Karpenter Works

Karpenter is a Kubernetes controller that runs inside the cluster and watches for pods in the Pending state with the condition Unschedulable. When it finds them, it:

Reads the pod’s resource requests (cpu, memory, extended resources like nvidia.com/gpu)
Reads the pod’s scheduling constraints (nodeSelector, nodeAffinity, topologySpreadConstraints, taints/tolerations)
Evaluates which NodePools the pod is eligible for
Selects the cheapest EC2 instance type from the eligible set that fits the pod and any other pending pods that could co-locate on the same node
Calls the EC2 Fleet API to launch the instance
Creates a Node and NodeClaim object in Kubernetes
The node joins the cluster; kubelet registers it; the pod is scheduled

┌──────────────────────────────────────────────────────────────┐
│                      Karpenter Controller                    │
│                                                              │
│  Watch Pods ──► Filter Unschedulable ──► Evaluate NodePools  │
│                                                │             │
│                                                ▼             │
│                                    Select cheapest fitting   │
│                                    instance type             │
│                                                │             │
│                                                ▼             │
│                                    EC2 Fleet API             │
│                                    (launch instance)         │
│                                                │             │
│                                                ▼             │
│                                    NodeClaim → Node joins    │
│                                    cluster → Pod scheduled   │
└──────────────────────────────────────────────────────────────┘

The reverse path — consolidation — also runs continuously. Karpenter evaluates whether any node is underutilized enough that its pods could be scheduled on other existing nodes. If so, it cordons the node, evicts the pods (respecting PodDisruptionBudgets), and terminates the EC2 instance. The pods reschedule onto remaining nodes, which Karpenter may then further consolidate. This ongoing bin-packing is the source of most of the cost savings.

Installation on EKS

Karpenter requires specific IAM permissions and a running EKS cluster. The recommended installation path uses Helm after configuring the prerequisites.

IAM setup

Karpenter needs two IAM roles: one for the controller itself (to call EC2 APIs and manage instances), and one for the nodes it provisions.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63


# Set environment variables
export CLUSTER_NAME="my-cluster"
export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
export AWS_REGION="us-east-1"
export KARPENTER_VERSION="1.5.0"
export KARPENTER_NAMESPACE="kube-system"

# Create node IAM role (instances need this to join the cluster)
cat <<EOF > node-trust-policy.json
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"Service": "ec2.amazonaws.com"},
    "Action": "sts:AssumeRole"
  }]
}
EOF

aws iam create-role --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \
  --assume-role-policy-document file://node-trust-policy.json

# Attach required policies to node role
for policy in AmazonEKSWorkerNodePolicy AmazonEKS_CNI_Policy \
              AmazonEC2ContainerRegistryReadOnly AmazonSSMManagedInstanceCore; do
  aws iam attach-role-policy \
    --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \
    --policy-arn "arn:aws:iam::aws:policy/${policy}"
done

# Create instance profile
aws iam create-instance-profile \
  --instance-profile-name "KarpenterNodeInstanceProfile-${CLUSTER_NAME}"
aws iam add-role-to-instance-profile \
  --instance-profile-name "KarpenterNodeInstanceProfile-${CLUSTER_NAME}" \
  --role-name "KarpenterNodeRole-${CLUSTER_NAME}"

# Create controller IAM role using IRSA (IAM Roles for Service Accounts)
# Karpenter controller needs permissions to manage EC2, describe subnets/SGs, etc.
aws iam create-role --role-name "KarpenterControllerRole-${CLUSTER_NAME}" \
  --assume-role-policy-document "$(cat <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/$(
        aws eks describe-cluster --name ${CLUSTER_NAME} \
          --query "cluster.identity.oidc.issuer" --output text | sed 's|https://||'
      )"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "$(aws eks describe-cluster --name ${CLUSTER_NAME} \
          --query "cluster.identity.oidc.issuer" --output text | sed 's|https://||'):sub":
          "system:serviceaccount:${KARPENTER_NAMESPACE}:karpenter"
      }
    }
  }]
}
EOF
)"

The full controller policy is documented at karpenter.sh and covers EC2 Fleet, EC2 instance management, pricing API, SQS (for interruption events), and IAM PassRole to the node role. Use the policy from the official docs rather than writing it by hand — it is updated with each Karpenter version.

SQS queue for interruption events

Karpenter handles Spot interruption notices, instance health events, and scheduled maintenance by subscribing to an SQS queue fed by EventBridge:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


resource "aws_sqs_queue" "karpenter_interruption" {
  name                      = "karpenter-interruption-${var.cluster_name}"
  message_retention_seconds = 300
  sqs_managed_sse_enabled   = true
}

# EventBridge rules to forward interruption events
resource "aws_cloudwatch_event_rule" "spot_interruption" {
  name        = "karpenter-spot-interruption"
  event_pattern = jsonencode({
    source      = ["aws.ec2"]
    detail-type = ["EC2 Spot Instance Interruption Warning"]
  })
}

resource "aws_cloudwatch_event_target" "spot_interruption" {
  rule      = aws_cloudwatch_event_rule.spot_interruption.name
  target_id = "KarpenterInterruptionQueue"
  arn       = aws_sqs_queue.karpenter_interruption.arn
}

# Also forward: EC2 Instance State-change Notification,
# EC2 Instance Rebalance Recommendation,
# AWS Health Event (for scheduled maintenance)

Helm installation

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


helm registry login public.ecr.aws --username AWS \
  --password $(aws ecr-public get-login-password --region us-east-1)

helm install karpenter oci://public.ecr.aws/karpenter/karpenter \
  --version "${KARPENTER_VERSION}" \
  --namespace "${KARPENTER_NAMESPACE}" \
  --create-namespace \
  --set "settings.clusterName=${CLUSTER_NAME}" \
  --set "settings.interruptionQueue=karpenter-interruption-${CLUSTER_NAME}" \
  --set controller.resources.requests.cpu=1 \
  --set controller.resources.requests.memory=1Gi \
  --set controller.resources.limits.cpu=1 \
  --set controller.resources.limits.memory=1Gi \
  --wait

# IMPORTANT: Run Karpenter on a node group NOT managed by Karpenter
# Use a node selector or affinity to pin Karpenter pods to a managed node group

The Two Core CRDs

EC2NodeClass

EC2NodeClass defines the AWS-specific infrastructure configuration for nodes Karpenter provisions. It specifies the AMI family, subnets, security groups, instance profile, and any additional node configuration.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43


apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  # AMI family — AL2023 is the current recommended Amazon Linux
  amiSelectorTerms:
    - alias: al2023@latest    # always latest; pin to a version for production

  # Subnets by tag — Karpenter discovers subnets matching these tags
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: my-cluster
        kubernetes.io/role/internal-elb: "1"

  # Security groups by tag
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: my-cluster

  # IAM instance profile for nodes
  instanceProfile: "KarpenterNodeInstanceProfile-my-cluster"

  # Additional tags applied to provisioned EC2 instances
  tags:
    Environment: production
    ManagedBy: karpenter

  # Custom user data appended to the bootstrap script
  userData: |
    #!/bin/bash
    echo "vm.max_map_count=262144" >> /etc/sysctl.d/99-custom.conf
    sysctl -p /etc/sysctl.d/99-custom.conf

  # Block device mappings — increase root volume, add data volume
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 50Gi
        volumeType: gp3
        encrypted: true
        iops: 3000
        throughput: 125

Tag your subnets and security groups with karpenter.sh/discovery: <cluster-name> so Karpenter can discover them:

1
2
3
4
5
6
7
8
9


# Tag subnets
aws ec2 create-tags \
  --resources subnet-0abc subnet-0def subnet-0ghi \
  --tags Key=karpenter.sh/discovery,Value=my-cluster

# Tag security groups
aws ec2 create-tags \
  --resources sg-0abc \
  --tags Key=karpenter.sh/discovery,Value=my-cluster

Pinning AMI versions for production: alias: al2023@latest always uses the newest AMI, which is appropriate for dev/staging but risks unexpected behavior changes in production. Pin to a specific AMI ID or use @v20250501 style versioning once you have validated an AMI:

1
2
3
4


amiSelectorTerms:
  - alias: al2023@v20250501   # specific AMI version
  # or by ID:
  - id: ami-0abc123def456

NodePool

NodePool defines the scheduling constraints, instance types, capacity types, and disruption behavior for a class of nodes. Multiple NodePools can coexist — for example, one for general workloads and one for GPU workloads.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67


apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    metadata:
      labels:
        node-type: general

    spec:
      # Reference the EC2NodeClass
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default

      # Capacity types: on-demand, spot, or both
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]

        # Instance families — diverse selection improves Spot availability
        - key: karpenter.k8s.aws/instance-family
          operator: In
          values: ["m5", "m5a", "m5n", "m6i", "m6a", "m7i", "m7a",
                   "c5", "c5a", "c5n", "c6i", "c6a", "c7i",
                   "r5", "r5a", "r6i", "r6a", "r7i"]

        # CPU range — avoid tiny instances (overhead) and huge ones (waste on failure)
        - key: karpenter.k8s.aws/instance-cpu
          operator: In
          values: ["4", "8", "16", "32"]

        # Hypervisor — nitro only (better performance, required for some features)
        - key: karpenter.k8s.aws/instance-hypervisor
          operator: In
          values: ["nitro"]

        # Architecture
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]

        # Zones — spread across AZs for resilience
        - key: topology.kubernetes.io/zone
          operator: In
          values: ["us-east-1a", "us-east-1b", "us-east-1c"]

  # Limits — cap total resources this NodePool can provision
  limits:
    cpu: "1000"
    memory: 4000Gi

  # Disruption policy
  disruption:
    # Replace nodes when their spec drifts from the NodePool/NodeClass definition
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 30s   # wait 30s after node becomes underutilized before acting

    # Disruption budgets — limit concurrent node replacements
    budgets:
      - nodes: "10%"          # default: max 10% of nodes disrupted at once
      - schedule: "0 8 * * *" # additional: during business hours, max 5% disrupted
        duration: 8h
        nodes: "5%"

Spot Instances and Capacity Fallback

Listing spot before on-demand in the capacity-type requirement does not force Spot — it includes both and lets Karpenter choose. Karpenter’s scheduler prefers the cheapest option, which is usually Spot when available.

When Spot capacity is unavailable for all selected instance types, Karpenter falls back to on-demand automatically. No manual intervention required.

For workloads that must be on-demand (production databases, stateful services), use a separate NodePool that excludes Spot:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


# on-demand-only NodePool
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]   # Spot excluded
      taints:
        - key: workload-type
          value: stateful
          effect: NoSchedule

Pods that need on-demand nodes add the corresponding toleration:

1
2
3
4
5
6
7


spec:
  tolerations:
    - key: workload-type
      value: stateful
      effect: NoSchedule
  nodeSelector:
    karpenter.sh/capacity-type: on-demand

Spot interruption handling

When AWS sends a Spot interruption notice (2-minute warning), Karpenter:

Detects the notice via the SQS queue (EventBridge → SQS → Karpenter)
Cordons the node immediately
Begins draining pods (evicting them respecting PodDisruptionBudgets)
Simultaneously provisions a replacement node
Terminated node’s pods reschedule onto the new node

The entire sequence — from interruption notice to pod running on replacement — completes within the 2-minute window for most workloads. This is faster than the cluster-autoscaler + aws-node-termination-handler combination, which is a separate component with additional latency.

For batch workloads that tolerate interruption, configure pods to checkpoint state before eviction using a preStop hook or SIGTERM handler:

1
2
3
4
5


lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "checkpoint-state.sh && sleep 5"]
terminationGracePeriodSeconds: 120

Consolidation and Disruption

Consolidation is Karpenter’s ongoing optimization loop. After provisioning nodes, Karpenter continuously evaluates whether the current node set is optimal. If it finds nodes whose workloads could be packed onto fewer nodes, it evicts and terminates the underused ones.

WhenEmptyOrUnderutilized is the most aggressive setting:

Empty nodes (no pods except DaemonSets): terminated immediately after consolidateAfter delay
Underutilized nodes: Karpenter simulates moving the node’s pods to other nodes. If the simulation succeeds (all pods can be scheduled elsewhere), it evicts and terminates

WhenEmpty only removes completely empty nodes — safer for production but leaves underutilized capacity running.

Disruption budgets

Disruption budgets limit how many nodes Karpenter can simultaneously disrupt (for consolidation, drift, or emptiness). Without budgets, aggressive consolidation can evict many pods at once, overwhelming remaining nodes or violating application availability requirements.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


disruption:
  consolidationPolicy: WhenEmptyOrUnderutilized
  consolidateAfter: 1m
  budgets:
    # Never disrupt more than 20% of this NodePool's nodes at once
    - nodes: "20%"

    # During working hours, be more conservative
    - schedule: "0 9 * * 1-5"   # 9am weekdays
      duration: 9h               # until 6pm
      nodes: "5%"

    # Allow no disruption during a deployment window
    - schedule: "0 2 * * *"     # 2am daily
      duration: 30m
      nodes: "0"                # freeze during deployment
      reasons:
        - Underutilized
        - Drifted

Budgets interact with PodDisruptionBudgets (PDBs). Karpenter respects PDBs during eviction — if evicting a pod would violate a PDB (minAvailable or maxUnavailable), Karpenter waits. Configure PDBs for all production workloads:

1
2
3
4
5
6
7
8
9


apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
spec:
  selector:
    matchLabels:
      app: api
  maxUnavailable: 1    # at most 1 pod unavailable during eviction

Weighted NodePools

When multiple NodePools match a pod’s requirements, Karpenter uses weights to prefer one over another. Higher weight = preferred. This is useful for graduated cost tiers: prefer Spot, fall back to cheaper on-demand families, fall back to more expensive families only as a last resort.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


# NodePool 1: Spot-first, broad instance diversity
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: spot-preferred
spec:
  weight: 100    # highest preference
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]
        - key: karpenter.k8s.aws/instance-family
          operator: In
          values: ["m5", "m6i", "m7i", "c5", "c6i", "c7i", "r5", "r6i"]
  limits:
    cpu: "500"
---
# NodePool 2: On-demand fallback, cost-optimized families
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: ondemand-fallback
spec:
  weight: 50     # used when spot-preferred cannot schedule
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
        - key: karpenter.k8s.aws/instance-family
          operator: In
          values: ["m5", "m6i", "c5", "c6i"]
  limits:
    cpu: "500"

Karpenter tries the highest-weight NodePool first. If no instance type in that pool can fit the pending pods (Spot unavailable, limits reached, instance family doesn’t fit), it falls through to the next pool by weight.

GPU Node Provisioning

GPU workloads require a dedicated NodePool using GPU instance families. Karpenter provisions GPU nodes on demand and terminates them when workloads finish — ideal for batch ML training and inference jobs that run periodically rather than continuously.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57


apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: gpu
spec:
  amiSelectorTerms:
    - alias: al2023@latest
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: my-cluster
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: my-cluster
  instanceProfile: "KarpenterNodeInstanceProfile-my-cluster"
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 200Gi   # GPU instances need more root volume for model storage
        volumeType: gp3
        encrypted: true
---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu
spec:
  weight: 100
  template:
    metadata:
      labels:
        node-type: gpu
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: gpu
      requirements:
        - key: karpenter.k8s.aws/instance-family
          operator: In
          values: ["p3", "p4d", "p4de", "p5", "g4dn", "g5", "g6"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
      # Taint GPU nodes — only GPU-tolerating pods land here
      taints:
        - key: nvidia.com/gpu
          effect: NoSchedule
  limits:
    cpu: "256"
    memory: 2000Gi
    nvidia.com/gpu: "32"   # cap at 32 GPUs across all GPU nodes
  disruption:
    consolidationPolicy: WhenEmpty   # don't consolidate GPU nodes mid-job
    consolidateAfter: 5m

GPU-requesting pods must tolerate the taint and request the GPU resource:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


spec:
  tolerations:
    - key: nvidia.com/gpu
      effect: NoSchedule
  containers:
    - name: training
      image: pytorch/pytorch:2.3.0-cuda12.1-cudnn8-runtime
      resources:
        requests:
          nvidia.com/gpu: "1"
        limits:
          nvidia.com/gpu: "1"

Install the NVIDIA device plugin for Kubernetes to expose GPU resources to the scheduler:

1
2
3
4


helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
helm install nvdp nvdp/nvidia-device-plugin \
  --namespace kube-system \
  --set failOnInitError=false

Drift Detection

Drift occurs when the configuration of a running node diverges from what the NodePool or EC2NodeClass specifies — the AMI has been updated, a new instance type requirement was added, the security groups changed. Karpenter detects this automatically and marks drifted nodes for replacement.

Drift replacement follows the same disruption budget rules as consolidation. Karpenter provisions a new node matching the current spec before evicting pods from the old node. For AMI updates, this is how you perform rolling node upgrades without manual intervention:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Pin AMI in EC2NodeClass, update this value to trigger rolling replacement
amiSelectorTerms:
  - alias: al2023@v20250601   # change to v20250701 → all nodes replaced rolling

# Control the rollout pace via disruption budget
disruption:
  budgets:
    - nodes: "25%"    # replace 25% of nodes at a time
      reasons:
        - Drifted

Updating the AMI alias in the EC2NodeClass triggers drift on all nodes in NodePools referencing it. Karpenter replaces them rolling, respecting the budget. New nodes get the new AMI; existing pods move to new nodes; old nodes are terminated. A fully automated, controlled node upgrade.

Topology Spread and Node Affinity

Use topologySpreadConstraints to distribute pods across Karpenter-provisioned nodes and AZs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


spec:
  topologySpreadConstraints:
    # Spread evenly across AZs — Karpenter provisions in multiple AZs to satisfy this
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          app: api

    # Spread across nodes within each AZ — avoid too many pods per node
    - maxSkew: 2
      topologyKey: kubernetes.io/hostname
      whenUnsatisfiable: ScheduleAnyway
      labelSelector:
        matchLabels:
          app: api

When Karpenter receives multiple pending pods with zone spread constraints, it provisions nodes across AZs to satisfy the constraints rather than stacking all pods on nodes in a single AZ.

Combine with nodeAffinity to prefer certain node characteristics without hard requirements:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


affinity:
  nodeAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 80
        preference:
          matchExpressions:
            - key: karpenter.sh/capacity-type
              operator: In
              values: ["spot"]
      - weight: 20
        preference:
          matchExpressions:
            - key: karpenter.sh/capacity-type
              operator: In
              values: ["on-demand"]

Observability

Key metrics

Karpenter exposes Prometheus metrics. The most operationally important:

Metric	What it tells you
`karpenter_nodes_total`	Total nodes managed by Karpenter
`karpenter_nodeclaims_total`	NodeClaims by lifecycle state
`karpenter_provisioner_scheduling_duration_seconds`	Time from pod pending to node launch decision
`karpenter_nodeclaims_disrupted_total`	Nodes disrupted by reason (Consolidation, Drift, Emptiness)
`karpenter_nodeclaims_launched_total`	Nodes launched by NodePool and capacity type
`karpenter_interruption_received_messages_total`	Spot interruption events received
`karpenter_pods_state`	Pods by state (pending, bound, etc.)

A dashboard with these metrics surfaces the provisioning latency, consolidation activity, and Spot interruption frequency in one view. The EKS Best Practices guide includes a sample Grafana dashboard.

Useful kubectl commands

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


# View all NodeClaims (Karpenter's representation of provisioned nodes)
kubectl get nodeclaims

# Describe a specific NodeClaim (shows instance type, zone, capacity type)
kubectl describe nodeclaim <name>

# View NodePools and their current usage vs limits
kubectl get nodepools -o wide

# Check Karpenter controller logs
kubectl logs -n kube-system -l app.kubernetes.io/name=karpenter --tail=100 -f

# See why a pod is unschedulable
kubectl describe pod <pending-pod-name>

# View nodes Karpenter provisioned (labeled with nodepool)
kubectl get nodes -l karpenter.sh/nodepool

# See instance type and capacity type per node
kubectl get nodes -L karpenter.sh/capacity-type,karpenter.k8s.aws/instance-type

Migrating from Cluster Autoscaler

Running both simultaneously during migration is supported. Cluster-autoscaler manages its node groups; Karpenter manages its own nodes. Pods scheduled to cluster-autoscaler node groups continue as before; new workloads or workloads that drain off cluster-autoscaler nodes migrate to Karpenter-provisioned nodes.

Migration sequence:

Install Karpenter with NodePools matching your current node group configuration
Set cluster-autoscaler node groups to min=current, max=current (freeze scale-out)
Gradually cordon and drain cluster-autoscaler nodes — Karpenter provisions replacements
Remove cluster-autoscaler node groups once drained
Uninstall cluster-autoscaler

Do not set cluster-autoscaler node group min to 0 before Karpenter is handling the workload — you risk all pods going unschedulable simultaneously.

Honest Trade-offs

What Karpenter gets right. The bin-packing and consolidation are genuinely better than cluster-autoscaler. The Spot handling is native rather than bolted on. The flexibility of picking from any instance type rather than predefined node groups makes cost optimization automatic rather than manual. For teams spending meaningful money on EC2 in EKS clusters, the payback period on the operational investment is typically measured in weeks.

Consolidation disrupts workloads. Karpenter’s consolidation model means running pods get evicted and rescheduled more frequently than with cluster-autoscaler. For stateful applications, database connections, or workloads with warm caches, this matters. Configure PDBs, appropriate consolidationPolicy settings, and disruption budgets to control this — but understand that aggressive consolidation and zero disruption are in tension.

NodePool limits require attention. Forgetting to set limits in a NodePool means Karpenter can provision an unbounded number of nodes in response to a workload spike (or a misconfigured deployment with thousands of replicas). Set limits on every NodePool and configure billing alarms.

AMI pinning vs drift. Using @latest AMI aliases means Karpenter automatically replaces nodes when AWS publishes new AMIs. This is good for security but can surprise you with unexpected node churn. Pin AMIs in production and update them deliberately during maintenance windows.

EKS-only in practice. The karpenter.k8s.aws provider is AWS-specific. There are community providers for Azure (AKS) and other platforms, but the maturity and feature parity lag the AWS provider. If you run multi-cloud Kubernetes, cluster-autoscaler remains the portable option.

Karpenter on Karpenter is not supported. The Karpenter controller must run on nodes not managed by Karpenter — typically a small managed node group (2 nodes, m5.large or similar). If the Karpenter controller pod lands on a node that Karpenter is trying to consolidate, it terminates itself. Use node affinity or a dedicated node group for the controller.