Karpenter: Intelligent Node Autoscaling for Kubernetes

March 27, 2026 18 min read Virtualization & Containers

kuberneteskarpenterautoscalingawscost-optimizationdevopscloud

Contents

The Kubernetes Cluster Autoscaler has served clusters well for years, but it was designed around a fundamental constraint: it scales pre-defined Auto Scaling Groups (ASGs). When pods are pending, it picks an ASG, scales it up, waits for the node to join, and eventually the pod gets scheduled. This works, but it’s slow (2–10 minutes to get a node), inflexible (you’re locked into the instance types in each ASG), and wasteful (nodes are often over-provisioned to catch the next burst).

Karpenter is a CNCF graduated project (donated by AWS, now broadly adopted) that takes a completely different approach: it watches for unschedulable pods directly, computes the optimal instance type for those pods right now, and provisions that exact instance in under 60 seconds — often under 30. It doesn’t manage ASGs. It talks to the cloud provider API directly and consolidates nodes continuously to eliminate waste.

This guide covers Karpenter’s architecture, installation, NodePool configuration, spot instance handling, consolidation, and the patterns that make it genuinely transformative for cluster costs.

How Karpenter Differs from Cluster Autoscaler

The conceptual difference matters before diving into YAML.

Cluster Autoscaler:

Detects pending pods
Simulates each configured ASG to find one that would fit the pods
Increments that ASG’s desired count
Waits for the ASG to launch an instance and register with Kubernetes
Kubelet starts, pods schedule

Karpenter:

Detects pending pods
Evaluates all instance types available in the region against pod requirements
Picks the cheapest option that fits (right-sizing, not over-provisioning)
Calls the EC2 API (or equivalent) directly to launch that instance with a pre-configured launch template
Node registers via a fast-path bootstrap; pod schedules in ~30–60 seconds

The key differences:

Feature	Cluster Autoscaler	Karpenter
Instance selection	Fixed ASG instance types	All instance types in region
Provisioning time	2–10 minutes	30–90 seconds
Spot fallback	Manual ASG configuration	Automatic, multi-family
Bin-packing	No	Yes — consolidates underused nodes
Node deprovisioning	Scale-down after 10+ min idle	Aggressive consolidation
Configuration	Per-ASG node groups	Declarative NodePools
Overhead	High (many node groups)	Low (one controller)

Installing Karpenter on AWS EKS

Prerequisites

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


# Set environment variables
export CLUSTER_NAME="my-cluster"
export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
export AWS_REGION="us-east-1"
export KARPENTER_VERSION="1.1.0"
export KARPENTER_NAMESPACE="kube-system"

# Get your cluster endpoint
export CLUSTER_ENDPOINT=$(aws eks describe-cluster \
  --name "${CLUSTER_NAME}" \
  --query "cluster.endpoint" \
  --output text)

IAM Setup

Karpenter needs permissions to create and terminate EC2 instances on your behalf:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


# Create the Karpenter node IAM role
aws cloudformation deploy \
  --stack-name "Karpenter-${CLUSTER_NAME}" \
  --template-file "https://raw.githubusercontent.com/aws/karpenter-provider-aws/v${KARPENTER_VERSION}/website/content/en/preview/getting-started/getting-started-with-karpenter/cloudformation.yaml" \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameter-overrides "ClusterName=${CLUSTER_NAME}"

# Grant the Karpenter node role access to join the cluster
eksctl create iamidentitymapping \
  --username system:node:{{EC2PrivateDNSName}} \
  --cluster "${CLUSTER_NAME}" \
  --arn "arn:aws:iam::${AWS_ACCOUNT_ID}:role/KarpenterNodeRole-${CLUSTER_NAME}" \
  --group system:bootstrappers \
  --group system:nodes

# Create the Karpenter controller IAM role (IRSA)
eksctl create iamserviceaccount \
  --cluster "${CLUSTER_NAME}" \
  --name karpenter \
  --namespace "${KARPENTER_NAMESPACE}" \
  --role-name "KarpenterControllerRole-${CLUSTER_NAME}" \
  --attach-policy-arn "arn:aws:iam::${AWS_ACCOUNT_ID}:policy/KarpenterControllerPolicy-${CLUSTER_NAME}" \
  --approve

# Tag the subnets and security groups so Karpenter can discover them
for SUBNET in $(aws ec2 describe-subnets \
  --filters "Name=tag:kubernetes.io/cluster/${CLUSTER_NAME},Values=owned,shared" \
  --query 'Subnets[].SubnetId' --output text); do
  aws ec2 create-tags --resources "${SUBNET}" \
    --tags "Key=karpenter.sh/discovery,Value=${CLUSTER_NAME}"
done

SECURITY_GROUP=$(aws eks describe-cluster \
  --name "${CLUSTER_NAME}" \
  --query "cluster.resourcesVpcConfig.clusterSecurityGroupId" \
  --output text)
aws ec2 create-tags --resources "${SECURITY_GROUP}" \
  --tags "Key=karpenter.sh/discovery,Value=${CLUSTER_NAME}"

Install with Helm

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


# Install Karpenter
helm registry login public.ecr.aws --username AWS \
  --password $(aws ecr-public get-login-password --region us-east-1)

helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter \
  --version "${KARPENTER_VERSION}" \
  --namespace "${KARPENTER_NAMESPACE}" \
  --create-namespace \
  --set "settings.clusterName=${CLUSTER_NAME}" \
  --set "settings.interruptionQueue=${CLUSTER_NAME}" \
  --set controller.resources.requests.cpu=1 \
  --set controller.resources.requests.memory=1Gi \
  --set controller.resources.limits.cpu=1 \
  --set controller.resources.limits.memory=1Gi \
  --wait

# Verify
kubectl get pods -n kube-system -l app.kubernetes.io/name=karpenter

NodePool: Defining What Karpenter Can Provision

The NodePool resource is Karpenter’s core configuration. It defines constraints — which instance types, architectures, capacity types, and zones are allowed — and policies for node expiry and consolidation.

A Production-Ready NodePool

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70


# nodepools/general-purpose.yaml
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general-purpose
spec:
  # Template for nodes Karpenter provisions
  template:
    metadata:
      labels:
        karpenter.sh/nodepool: general-purpose
      # Taints can be added here if this pool is for specific workloads
      # taints:
      #   - key: dedicated
      #     value: general
      #     effect: NoSchedule

    spec:
      # Reference to EC2NodeClass (AWS-specific configuration)
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default

      requirements:
        # Allow both on-demand and spot
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]

        # Allow multiple architectures (Graviton is cheaper)
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]

        # Instance family restrictions — exclude GPU, bare metal, nano sizes
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]   # compute, general, memory optimized

        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["2"]             # Only gen 3+ instances

        # Minimum 2 vCPU, maximum 64 vCPU
        - key: karpenter.k8s.aws/instance-cpu
          operator: Gt
          values: ["1"]
        - key: karpenter.k8s.aws/instance-cpu
          operator: Lt
          values: ["65"]

        # Stay in the cluster's availability zones
        - key: topology.kubernetes.io/zone
          operator: In
          values: ["us-east-1a", "us-east-1b", "us-east-1c"]

  # Limits: Karpenter won't provision beyond these totals
  limits:
    cpu: "1000"          # Total CPU across all nodes in this pool
    memory: 4000Gi       # Total memory limit

  # Disruption policy: how Karpenter removes nodes
  disruption:
    # Replace nodes after 720 hours (30 days) — forces OS/kernel updates
    expireAfter: 720h

    # Consolidation: aggressively bin-pack and remove underused nodes
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m   # How long to wait before consolidating

EC2NodeClass: AWS-Specific Configuration

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56


# nodeclasses/default.yaml
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  # AMI selection — use the latest EKS-optimized AMI for your cluster version
  amiSelectorTerms:
    - alias: al2023@latest   # Amazon Linux 2023, latest patch

  # Subnet selection via tags
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: "my-cluster"

  # Security group selection
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: "my-cluster"

  # IAM role for the nodes
  role: "KarpenterNodeRole-my-cluster"

  # Instance store: use NVMe ephemeral disks if available
  instanceStorePolicy: RAID0

  # User data runs on node boot (before kubelet starts)
  userData: |
    #!/bin/bash
    # Custom node bootstrap configuration
    cat <<EOF >> /etc/kubernetes/bootstrap.sh
    # Additional kubelet flags
    --max-pods=110
    EOF

  # EBS volume configuration
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 100Gi
        volumeType: gp3
        iops: 3000
        throughput: 125
        encrypted: true
        deleteOnTermination: true

  # Metadata options — disable IMDSv1 for security
  metadataOptions:
    httpEndpoint: enabled
    httpProtocolIPv6: disabled
    httpPutResponseHopLimit: 1    # Prevents SSRF from containers
    httpTokens: required          # Force IMDSv2

  tags:
    Team: platform
    ManagedBy: karpenter

Spot Instance Handling

Spot instances can be 60–90% cheaper than on-demand. Karpenter handles spot interruptions gracefully, but the configuration needs to be right.

Multi-Family Spot for Maximum Availability

The key to reliable spot is requesting many instance families simultaneously. If one family is interrupted or unavailable, Karpenter falls back to another:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: spot-workers
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      requirements:
        # Spot only for this pool
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]

        # Wide instance family selection — more diversity = more availability
        - key: karpenter.k8s.aws/instance-family
          operator: In
          values:
            - m5, m5a, m5n, m5d
            - m6i, m6a, m6in
            - m7i, m7a, m7g       # m7g is Graviton (arm64)
            - c5, c5a, c5n
            - c6i, c6a, c6in
            - c7i, c7a, c7g
            - r5, r5a, r5n
            - r6i, r6a

        # 4-32 vCPU range gives Karpenter lots of flexibility
        - key: karpenter.k8s.aws/instance-cpu
          operator: In
          values: ["4", "8", "16", "32"]

  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 30s    # Aggressive for spot — reclaim quickly

Spot Interruption Handling

When AWS reclaims a spot instance, it sends a 2-minute warning. Karpenter watches for these via an SQS queue and proactively drains the node before the instance terminates:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


# Create the interruption queue (referenced in helm install above)
aws sqs create-queue \
  --queue-name "${CLUSTER_NAME}" \
  --attributes '{
    "MessageRetentionPeriod": "300"
  }'

# Subscribe to EC2 spot interruption events
aws events put-rule \
  --name "KarpenterInterruptionRule" \
  --event-pattern '{
    "source": ["aws.ec2"],
    "detail-type": ["EC2 Spot Instance Interruption Warning",
                    "EC2 Instance Rebalance Recommendation",
                    "EC2 Instance State-change Notification"]
  }'

Karpenter then:

Receives the 2-minute warning via SQS
Cordons the node immediately
Launches a replacement node (on-demand if spot is unavailable)
Drains workloads to the new node
All within the 2-minute window in most cases

Workloads That Should Use Spot

Not every workload is spot-appropriate. Use node selectors and tolerations to control placement:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


# Stateless workers — perfect for spot
apiVersion: apps/v1
kind: Deployment
metadata:
  name: image-processor
spec:
  template:
    spec:
      # Request spot nodes
      nodeSelector:
        karpenter.sh/capacity-type: spot
      # Tolerate spot interruptions gracefully
      terminationGracePeriodSeconds: 90
      containers:
        - name: processor
          image: my-image-processor:latest
          # Implement checkpoint/resume logic

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


# Stateful or latency-sensitive — force on-demand
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-api
spec:
  template:
    spec:
      nodeSelector:
        karpenter.sh/capacity-type: on-demand
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: karpenter.sh/capacity-type
                    operator: In
                    values: ["on-demand"]

Multiple NodePools for Different Workload Types

Real clusters have diverse workload requirements. Use multiple NodePools with weights and requirements to segment them:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


# nodepools/compute-optimized.yaml
# For CPU-heavy workloads: data processing, compilation, ML inference
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: compute-optimized
spec:
  template:
    metadata:
      labels:
        workload-type: compute
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      taints:
        - key: workload-type
          value: compute
          effect: NoSchedule
      requirements:
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c"]        # Compute-optimized only
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 2m
  weight: 50    # Higher weight = preferred when multiple pools match

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


# nodepools/memory-optimized.yaml
# For Redis, Elasticsearch, in-memory databases
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: memory-optimized
spec:
  template:
    metadata:
      labels:
        workload-type: memory
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      taints:
        - key: workload-type
          value: memory
          effect: NoSchedule
      requirements:
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["r"]        # Memory-optimized: r5, r6i, r7g...
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]   # Databases need stability
  disruption:
    # Don't consolidate memory nodes as aggressively
    consolidationPolicy: WhenEmpty
    expireAfter: 720h

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


# nodepools/gpu.yaml
# For ML training and inference
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu
spec:
  template:
    metadata:
      labels:
        workload-type: gpu
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: gpu
      taints:
        - key: nvidia.com/gpu
          effect: NoSchedule
      requirements:
        - key: karpenter.k8s.aws/instance-family
          operator: In
          values: ["p3", "p4d", "g4dn", "g5"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
  limits:
    cpu: "200"
    memory: 800Gi
    nvidia.com/gpu: "20"    # Limit total GPUs in pool

Workloads opt into a specific pool with tolerations:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


# ML training job that uses GPU nodes
apiVersion: batch/v1
kind: Job
metadata:
  name: model-training
spec:
  template:
    spec:
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
        - key: workload-type
          value: gpu
          effect: NoSchedule
      nodeSelector:
        workload-type: gpu
      containers:
        - name: trainer
          image: my-training:latest
          resources:
            limits:
              nvidia.com/gpu: "1"

Consolidation: Eliminating Waste

Karpenter’s consolidation is one of its most powerful features. After the initial burst scales up nodes, Karpenter continuously evaluates whether nodes can be packed more efficiently and terminates the underused ones.

How Consolidation Works

Karpenter’s consolidation algorithm runs on a loop:

Find nodes that are candidates (not hosting PDBs preventing disruption, not with do-not-disrupt annotation)
Simulate evicting all pods from the candidate node
Check if those pods could schedule on remaining nodes
If yes, cordon and drain the candidate, terminate the instance

With WhenEmptyOrUnderutilized, Karpenter will replace two smaller underused nodes with one larger node if it’s cheaper — actively bin-packing your workloads.

Protecting Workloads from Disruption

Use PodDisruptionBudgets to ensure consolidation doesn’t violate availability requirements:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Karpenter respects PDBs during consolidation
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
spec:
  minAvailable: 2      # Always keep 2 replicas up
  selector:
    matchLabels:
      app: api

For workloads that must never be disrupted (long-running batch jobs):

1
2
3
4
5
6


# Annotate the pod to prevent Karpenter from disrupting it
apiVersion: v1
kind: Pod
metadata:
  annotations:
    karpenter.sh/do-not-disrupt: "true"   # Karpenter will not evict this pod

Consolidation Tuning

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


spec:
  disruption:
    # WhenEmpty: only consolidate completely empty nodes (safest)
    # WhenEmptyOrUnderutilized: actively re-pack pods (most savings)
    consolidationPolicy: WhenEmptyOrUnderutilized

    # How long a node must be consolidation-eligible before action is taken
    # Shorter = more aggressive savings, more pod disruption
    consolidateAfter: 1m

    # Node max age — forces replacement even if busy (keeps AMIs fresh)
    expireAfter: 720h   # 30 days

Scheduling Best Practices for Karpenter

Topology Spread Constraints

Tell Karpenter how to spread pods across zones and nodes for HA:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 6
  template:
    spec:
      topologySpreadConstraints:
        # Spread evenly across availability zones
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: api
        # No more than 2 replicas per node
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: api

Karpenter uses these constraints when deciding how many nodes to provision and where — it will provision nodes in the required zones to satisfy the spread.

Resource Requests: The Foundation of Good Scheduling

Karpenter’s bin-packing is only as good as your resource requests. Pods with no requests will be packed onto nodes with no regard for actual usage:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


containers:
  - name: api
    resources:
      requests:
        # Set requests at the p95 of actual usage
        # Use kubectl top pods or VPA recommendations
        cpu: "500m"
        memory: "512Mi"
      limits:
        # Set limits higher — allow bursting
        cpu: "2"
        memory: "1Gi"

Use the Vertical Pod Autoscaler in recommendation mode to get request suggestions based on actual usage:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: api-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  updatePolicy:
    updateMode: "Off"   # Recommendation only — don't auto-update

1
2
3


# Check VPA recommendations
kubectl describe vpa api-vpa
# Look for: Recommendation > Container Recommendations > Target

Drift Detection and Node Replacement

Karpenter detects when nodes have drifted from their NodePool or EC2NodeClass specification and replaces them automatically. This is invaluable for:

AMI updates: when you update the amiSelectorTerms to a new AMI, Karpenter automatically rolls nodes to the new AMI using a rolling replacement strategy
NodePool changes: update instance requirements, and Karpenter replaces nodes that no longer match
EC2NodeClass changes: update security groups or user data, nodes are rolled

1
2
3
4
5
6
7


# Trigger a manual drift-based replacement by annotating nodes
kubectl annotate node <node-name> \
  karpenter.sh/disruption=drifted

# Or force immediate node replacement (use carefully in production)
kubectl delete node <node-name>
# Karpenter provisions a replacement before the workloads are disrupted

Observability and Debugging

Key Metrics

Karpenter exposes Prometheus metrics on port 8000:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# prometheus scrape config
- job_name: karpenter
  kubernetes_sd_configs:
    - role: endpoints
      namespaces:
        names: [kube-system]
  relabel_configs:
    - source_labels: [__meta_kubernetes_service_name]
      action: keep
      regex: karpenter

Key metrics to dashboard:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# Nodes provisioned over time
sum(karpenter_nodes_created_total) by (nodepool)

# Current node count by pool and capacity type
sum(karpenter_nodes_total) by (nodepool, capacity_type)

# Pod scheduling latency (time from pending to scheduled)
histogram_quantile(0.99, rate(karpenter_pods_startup_duration_seconds_bucket[5m]))

# Interruptions handled
sum(rate(karpenter_interruption_actions_performed_total[5m])) by (action)

# Consolidation actions
sum(rate(karpenter_disruption_actions_performed_total[5m])) by (disruption_action)

Debugging Unschedulable Pods

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


# Check why Karpenter isn't provisioning a node
kubectl logs -n kube-system -l app.kubernetes.io/name=karpenter \
  --since=5m | grep -i "failed\|error\|unable\|cannot"

# Check the NodePool's current status
kubectl get nodepool general-purpose -o yaml | grep -A 20 status

# See what Karpenter computed for a pending pod
kubectl describe pod <pending-pod>
# Look for: Events, specifically "FailedScheduling" messages

# List all nodes Karpenter manages
kubectl get nodes -l karpenter.sh/nodepool

# See disruption decisions
kubectl logs -n kube-system -l app.kubernetes.io/name=karpenter \
  | grep -i "disruption\|consolidat\|drift"

Common Issues

Pods stay pending despite Karpenter running:

Check that pod requirements (node selectors, tolerations) match the NodePool requirements
Verify subnet/security group tags are correct
Check IAM permissions with aws iam simulate-principal-policy
Look for limits in the NodePool that are already hit

Nodes launch but pods don’t schedule:

Instance profile / node IAM role not mapped to aws-auth ConfigMap
VPC CNI can’t assign enough IPs (check kubectl describe ds aws-node)

Consolidation too aggressive (too many pod restarts):

Add PodDisruptionBudgets to critical workloads
Increase consolidateAfter to give pods more time
Use karpenter.sh/do-not-disrupt: "true" on long-running jobs

Cost Analysis: Karpenter vs Cluster Autoscaler

Real-world data from teams that switched consistently shows 20–60% cost reduction. Here’s why:

Right-sizing at provision time: Cluster Autoscaler picks a pre-defined node type from an ASG. If you have a 4-CPU ASG and a pod requests 3.5 CPU, you waste 0.5 CPU per pod. Karpenter selects the exact instance size for the workload.

Spot diversity: A typical Cluster Autoscaler spot setup has 3–5 instance types per ASG. Karpenter considers all available instance types and selects based on current spot pricing — typically finding 20–40% cheaper spot options.

Aggressive consolidation: Cluster Autoscaler removes nodes after 10+ minutes of underutilization (configurable, but slow by default). Karpenter consolidates within minutes, continuously rebinning workloads.

No over-provisioned node groups: Teams running Cluster Autoscaler typically create many ASGs to cover different workload types. Each ASG needs buffer capacity. Karpenter’s flexible provisioning eliminates this overhead.

A rough estimation for a team spending $10,000/month on EC2:

Strategy	Monthly Cost
Cluster Autoscaler (on-demand only)	$10,000
Cluster Autoscaler (with spot, 3 families)	$6,000
Karpenter (on-demand, right-sized)	$7,500
Karpenter (spot + on-demand, many families)	$2,500–$3,500

The combination of right-sizing and broad spot family selection typically yields the biggest savings.

Production Checklist

Before running Karpenter in production:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# checklist.md
- [ ] IMDSv2 required in EC2NodeClass (httpTokens: required)
- [ ] EBS volumes encrypted in EC2NodeClass
- [ ] NodePool limits set (prevent runaway scaling)
- [ ] PodDisruptionBudgets on all critical workloads
- [ ] Spot interruption queue configured and monitored
- [ ] Karpenter pods running on dedicated on-demand nodes
       (use nodeAffinity to keep Karpenter off spot)
- [ ] expireAfter set (forces AMI updates)
- [ ] VPA recommendations reviewed for major workloads
- [ ] Karpenter metrics dashboard created
- [ ] Alert on karpenter_nodes_total approaching limit
- [ ] do-not-disrupt annotation on batch jobs
- [ ] Tested spot interruption behavior in staging

Keep Karpenter itself on stable on-demand nodes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


# Keep Karpenter off spot to avoid self-disruption
apiVersion: apps/v1
kind: Deployment
metadata:
  name: karpenter
  namespace: kube-system
spec:
  template:
    spec:
      nodeSelector:
        karpenter.sh/capacity-type: on-demand
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: karpenter.sh/nodepool
                    operator: DoesNotExist   # Run on a managed node group

Conclusion

Karpenter changes the economics of running Kubernetes in the cloud. By talking directly to the EC2 API with full knowledge of instance pricing and availability, it consistently makes better provisioning decisions than a human operator managing ASGs — faster, cheaper, and with less operational overhead.

The configuration surface is small: a NodePool describing what you want, an EC2NodeClass describing the AWS-specific details, and PodDisruptionBudgets protecting your critical workloads. Within those constraints, Karpenter handles everything else: instance selection, spot interruptions, node consolidation, drift detection, and AMI rolling.

If you’re running EKS (or AKS, GKE via the cloud-provider ports) and still using Cluster Autoscaler, migrating to Karpenter is one of the highest-ROI changes you can make. The 30–60% cost reduction pays for the migration effort within weeks.