AWS EKS Deep Dive: Managed Kubernetes on AWS

Amazon EKS is AWS’s managed Kubernetes offering. AWS runs the control plane — multi-AZ etcd, API server, controller manager, scheduler — on infrastructure you never touch. You supply the nodes (or don’t, with Fargate). Simple premise; genuinely complex implementation once you hit production. This post covers everything that matters, including the decisions that come back to bite you at 2 a.m.

Current state as of May 2026: EKS supports Kubernetes 1.33, 1.34, and 1.35 on standard support. Versions 1.30–1.32 are on extended support (additional cost). The default version for new clusters is 1.35. Karpenter v1.x (GA since August 2024, currently at ~v1.12) is stable. EKS Pod Identity reached GA in late 2023. EKS access entries reached GA in early 2024. AL2 AMIs were end-of-life November 2025 — if you’re still on AL2 nodes, that’s your first action item.

Table of Contents

1. Control Plane Architecture
2. Worker Node Models: The Big Decision
3. Networking Deep Dive — VPC CNI and Everything That Flows From It
4. IRSA: IAM Roles for Service Accounts
5. EKS Pod Identity (the newer way)
6. EKS Add-ons Lifecycle
7. Managed Node Groups in Depth
8. Karpenter on EKS
9. Fargate Profiles
10. Cluster Upgrades with Zero Downtime
11. EKS Anywhere
12. Cluster Access Management: aws-auth vs Access Entries
13. Cost Optimization Patterns
14. Observability

1. Control Plane Architecture

AWS manages the control plane entirely. When you create a cluster, AWS provisions at minimum two API server instances and three etcd nodes spread across three Availability Zones in the region. You pay a flat $0.10/hour per cluster regardless of size. The cluster endpoint is a Network Load Balancer that AWS manages; you can make it public, private, or both.

The control plane runs in an AWS-owned VPC. It communicates with your worker nodes via cross-VPC ENIs that AWS injects into your VPC — these are the “cluster security group” ENIs you’ll see if you go looking. The cross-account ENI model is why your nodes need the cluster security group attached (or at least port 443 and 10250 open from the control plane CIDR).

Control plane logging — disabled by default, must be explicitly enabled. Five log types: api, audit, authenticator, controllerManager, scheduler. They land in CloudWatch Logs under /aws/eks/<cluster-name>/cluster. The audit log is what you need for security investigations. The authenticator log shows every IAM auth attempt. Enable them all in production; the cost is negligible, the regret for not having them is not.

1
2
3
4

aws eks update-cluster-config \
  --name my-cluster \
  --region us-east-1 \
  --logging '{"clusterLogging":[{"types":["api","audit","authenticator","controllerManager","scheduler"],"enabled":true}]}'

Terraform:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

resource "aws_eks_cluster" "main" {
  name     = "my-cluster"
  role_arn = aws_iam_role.cluster.arn
  version  = "1.35"

  vpc_config {
    subnet_ids              = concat(var.private_subnet_ids, var.public_subnet_ids)
    endpoint_private_access = true
    endpoint_public_access  = true
    public_access_cidrs     = ["YOUR_OFFICE_CIDR/32"]
    security_group_ids      = [aws_security_group.cluster_additional.id]
  }

  enabled_cluster_log_types = ["api", "audit", "authenticator", "controllerManager", "scheduler"]

  encryption_config {
    provider {
      key_arn = aws_kms_key.eks.arn
    }
    resources = ["secrets"]
  }

  access_config {
    authentication_mode                         = "API_AND_CONFIG_MAP"
    bootstrap_cluster_creator_admin_permissions = true
  }

  depends_on = [
    aws_iam_role_policy_attachment.cluster_AmazonEKSClusterPolicy,
  ]
}

gotcha: The subnet_ids in vpc_config are used by EKS to place cross-account ENIs, not to place your nodes. Node placement is controlled by the subnets in your node groups. Cluster subnets must have at least 5 available IPs at upgrade time — a constraint that has surprised many people running tight subnets.

2. Worker Node Models: The Big Decision

Managed node groups are the right default for most teams. They handle the rolling update lifecycle (cordon, drain, replace) and use Launch Templates under the hood. The main constraint is that all nodes in a group use the same instance type family and size (though you can use a list).

Self-managed nodes are rarely the answer in 2026 unless you need something genuinely custom: specific kernel patches, custom CNI, non-EKS-optimized AMI, or GPU setups with unusual requirements. The operational burden is real.

Fargate eliminates node management entirely but comes with hard constraints (see section 9). Use it for workloads where the constraints don’t apply.

Karpenter is the right answer when you need heterogeneous fleets, aggressive cost optimization with Spot, or sub-minute node provisioning. It replaces the Cluster Autoscaler and does bin-packing at decision time rather than relying on static node groups.

The practical production pattern for most clusters: Managed node groups for your baseline system workloads (coredns, kube-proxy, monitoring agents) + Karpenter for application workloads.

3. Networking Deep Dive

VPC CNI: Secondary IP Mode vs Prefix Delegation

The default AWS VPC CNI (aws-vpc-cni) assigns real VPC IP addresses to every pod. Each pod gets an IP from your VPC subnet — no overlay network, no NAT for inter-pod traffic. This is elegant for observability and security group enforcement, but it burns IPs fast.

Secondary IP mode (default): Each node pre-warms a pool of secondary IPs from ENI slots. The number of IPs per node is bounded by (max ENIs per instance type) × (IPs per ENI - 1). On an m5.xlarge (3 ENIs × 15 IPs = 45 IPs minus 3 for primary = 42 max pods). This is the “ran out of IPs at 3 a.m.” failure mode.

Prefix delegation mode: Assigns /28 IPv4 prefixes (16 IPs) to ENI slots instead of individual IPs. An m5.xlarge with prefix delegation: 3 ENIs × 15 prefix slots × 16 IPs = 720 IPs available. This is a 17x improvement over secondary IP mode.

Enable prefix delegation:

1
2

kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true
kubectl set env daemonset aws-node -n kube-system WARM_PREFIX_TARGET=1

Or via the EKS add-on:

1
2
3
4
5

aws eks update-addon \
  --cluster-name my-cluster \
  --addon-name vpc-cni \
  --configuration-values '{"env":{"ENABLE_PREFIX_DELEGATION":"true","WARM_PREFIX_TARGET":"1"}}' \
  --resolve-conflicts PRESERVE

Prefix delegation gotcha: The /28 blocks must be contiguous in your subnet. If your subnet is fragmented (common in long-running clusters), prefix allocation fails and pods stay Pending. Solution: Use dedicated subnets for node ENIs, or pre-warm with WARM_PREFIX_TARGET. IPv6 clusters require prefix delegation — it’s the only mode.

Subnet math: Before enabling prefix delegation, check for fragmentation:

1
2
3

# Check how many /28 blocks are available in a subnet
aws ec2 describe-subnets --subnet-ids subnet-xxxxx \
  --query 'Subnets[*].AvailableIpAddressCount'

Custom Networking (Different Subnets for Pods and Nodes)

If your nodes are in subnets with few IPs (common in older VPC designs) but you have large RFC 1918 secondary CIDR blocks, use custom networking to put pod IPs in the secondary CIDR subnets.

1. Associate a secondary CIDR (100.64.0.0/16 is a popular choice) with your VPC
2. Create subnets from it per AZ
3. Create ENIConfig objects pointing to those subnets
4. Set AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true on the aws-node DaemonSet
5. Label nodes with k8s.amazonaws.com/eniConfig=<az-name> or set ENI_CONFIG_LABEL_DEF=topology.kubernetes.io/zone

gotcha: Custom networking and security groups for pods together use the security group from the security groups for pods config, not from ENIConfig. Only one wins.

Security Groups for Pods

Requires VPC CNI ENABLE_POD_ENI=true. Workable only on Nitro-based instances that have IsTrunkingCompatible: true (most m5, c5, r5, m6g, c6g, r6g — but NOT the t family). The VPC resource controller attaches a trunk ENI to the node and branches pod-specific ENIs off it.

The SecurityGroupPolicy CRD selects pods and applies a security group:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

apiVersion: vpcresources.k8s.aws/v1beta1
kind: SecurityGroupPolicy
metadata:
  name: my-sg-policy
  namespace: my-app
spec:
  podSelector:
    matchLabels:
      role: database-client
  securityGroups:
    groupIds:
      - sg-0123456789abcdef0

gotcha: POD_SECURITY_GROUP_ENFORCING_MODE defaults to strict, which breaks NodePort and LoadBalancer services with externalTrafficPolicy: Local. Switch to standard mode in production if you use those. Also: pods with security groups on NodeLocal DNSCache clusters require VPC CNI >= 1.11.0 and standard enforcing mode.

IPv6 EKS Clusters

IPv6 mode assigns each pod a unique IPv6 address from the VPC. No more IP exhaustion problems. Requires prefix delegation (it’s the only mode for IPv6). Each pod gets a /128 from a /80 prefix.

IPv6 constraints: AWS load balancers must support IPv6 (most do now). External clients need IPv6 connectivity or a dual-stack setup. Some third-party tools still don’t handle IPv6 well.

CoreDNS Tuning for High-Throughput Clusters

Default CoreDNS scales poorly beyond ~1000 pods because every DNS query goes to the CoreDNS ClusterIP, which is iptables NAT. At scale this produces conntrack table exhaustion and the infamous “5-second DNS timeout” (DNAT race condition + conntrack timeout).

NodeLocal DNSCache: A DaemonSet that runs a DNS cache on each node at 169.254.20.10. Pods query the local cache instead of the ClusterIP, bypassing iptables DNAT entirely.

1
2

# Deploy NodeLocal DNSCache (use the official manifest from k8s docs)
kubectl apply -f https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml

CoreDNS Corefile tuning:

.:53 {
    errors
    health {
        lameduck 30s
    }
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
        pods insecure
        fallthrough in-addr.arpa ip6.arpa
        ttl 30
    }
    prometheus :9153
    forward . /etc/resolv.conf {
        prefer_udp
        max_concurrent 1000
    }
    cache 30
    loop
    reload
    loadbalance
}

Scale CoreDNS replicas with the Cluster Proportional Autoscaler: one replica per 256 cores or 16 nodes, whichever comes first. If you’re throttling at the VPC DNS resolver (1024 packets/second per ENI limit), add replicas faster.

4. IRSA: IAM Roles for Service Accounts

IRSA lets pods assume IAM roles without storing credentials anywhere. Under the hood: the kube-apiserver projects a short-lived OIDC JWT token into the pod’s filesystem. The AWS SDK calls sts:AssumeRoleWithWebIdentity with that token. STS validates the token signature against the OIDC provider public key.

Step 1: Create the OIDC Provider

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

# Get the OIDC issuer URL
OIDC_URL=$(aws eks describe-cluster --name my-cluster \
  --query "cluster.identity.oidc.issuer" --output text)

# Get the thumbprint
THUMBPRINT=$(openssl s_client -connect \
  $(echo $OIDC_URL | cut -d'/' -f3):443 \
  -showcerts </dev/null 2>/dev/null | \
  openssl x509 -fingerprint -noout | \
  sed 's/SHA1 Fingerprint=//' | tr -d ':' | tr '[:upper:]' '[:lower:]')

# Create the provider
aws iam create-open-id-connect-provider \
  --url $OIDC_URL \
  --client-id-list sts.amazonaws.com \
  --thumbprint-list $THUMBPRINT

Or with eksctl (recommended):

1
2
3

eksctl utils associate-iam-oidc-provider \
  --cluster my-cluster \
  --approve

Step 2: Trust Policy

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:my-namespace:my-service-account",
          "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:aud": "sts.amazonaws.com"
        }
      }
    }
  ]
}

StringEquals vs StringLike: Use StringEquals to lock down to a specific service account. Use StringLike with a wildcard (system:serviceaccount:my-namespace:*) to allow any service account in a namespace. Be careful with wildcards — a compromised namespace means a compromised role.

Step 3: Create the Role and Annotate the ServiceAccount

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# Create role
aws iam create-role \
  --role-name my-pod-role \
  --assume-role-policy-document file://trust-policy.json

# Attach permissions
aws iam attach-role-policy \
  --role-name my-pod-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess

# Annotate the ServiceAccount
kubectl annotate serviceaccount my-service-account \
  -n my-namespace \
  eks.amazonaws.com/role-arn=arn:aws:iam::123456789012:role/my-pod-role

Or all at once with eksctl:

1
2
3
4
5
6
7

eksctl create iamserviceaccount \
  --name my-service-account \
  --namespace my-namespace \
  --cluster my-cluster \
  --role-name my-pod-role \
  --attach-policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess \
  --approve

How the Token Projection Works

The mutating webhook (pod-identity-webhook) sees any pod using an annotated service account and injects two things:

• A projected volume with the OIDC token at /var/run/secrets/eks.amazonaws.com/serviceaccount/token
• Environment variables: AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN

The AWS SDK picks these up automatically via the default credential chain. Token lifetime is 24 hours by default; the kubelet refreshes it before expiry.

Cross-account IRSA: The trust policy stays in account A (where the cluster is). Account B creates a role with a trust policy that allows Account A’s pod role to assume it via sts:AssumeRole. The pod assumes the Account A role first, then calls sts:AssumeRole to switch to Account B. Alternatively, modify the trust policy in Account B to directly federate the OIDC provider from Account A — both patterns work.

gotcha: The OIDC provider URL is cluster-specific. If you have 50 clusters, you have 50 OIDC providers, and every IAM role needs its trust policy updated per cluster. This is one of the main motivations for Pod Identity.

5. EKS Pod Identity

GA since late 2023. Pod Identity is a cleaner approach that doesn’t use per-cluster OIDC trust policies. The trust relationship uses a fixed service principal:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "pods.eks.amazonaws.com"
      },
      "Action": [
        "sts:AssumeRole",
        "sts:TagSession"
      ]
    }
  ]
}

Same role, any EKS cluster. No OIDC provider needed. No per-cluster trust policy edits.

Setup

1. Install the Pod Identity Agent add-on (DaemonSet on each node):

1
2
3

aws eks create-addon \
  --cluster-name my-cluster \
  --addon-name eks-pod-identity-agent

2. Create a Pod Identity Association:

1
2
3
4
5

aws eks create-pod-identity-association \
  --cluster-name my-cluster \
  --namespace my-namespace \
  --service-account my-service-account \
  --role-arn arn:aws:iam::123456789012:role/my-pod-role

The Pod Identity Agent runs on each node at 169.254.170.23 (link-local). When a pod starts, the agent issues credentials to it directly. The load is per-node, not per-pod-per-SDK-call — a meaningful scalability improvement over IRSA at large cluster sizes.

IRSA vs Pod Identity: When to Use Which

Practical guidance: For new workloads on standard EKS clusters with EC2 nodes, use Pod Identity. For Fargate workloads, EKS Anywhere, cross-account with direct federation, or legacy compatibility, use IRSA. Both can coexist in the same cluster.

gotcha: Pod Identity associations are eventually consistent. Don’t create/update them in tight deployment automation loops — add a brief wait or check for the association to propagate.

6. EKS Add-ons Lifecycle

EKS manages a curated set of add-ons that it can install, update, and patch. Core add-ons installed automatically: vpc-cni, coredns, kube-proxy. Others you install on demand.

AWS-managed add-ons (key ones):

• vpc-cni — pod networking
• coredns — cluster DNS
• kube-proxy — service networking (iptables/ipvs rules on each node)
• aws-ebs-csi-driver — EBS storage (requires IRSA or Pod Identity)
• aws-efs-csi-driver — EFS storage
• snapshot-controller — VolumeSnapshot CRDs
• eks-pod-identity-agent — Pod Identity
• amazon-cloudwatch-observability — Container Insights
• adot — AWS Distro for OpenTelemetry

Version Management

Add-ons are NOT automatically upgraded when you upgrade the cluster. You must initiate add-on upgrades separately, and you can only upgrade one minor version at a time.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

# See available versions
aws eks describe-addon-versions \
  --addon-name vpc-cni \
  --kubernetes-version 1.35 \
  --query 'addons[].addonVersions[].addonVersion'

# Update an add-on
aws eks update-addon \
  --cluster-name my-cluster \
  --addon-name vpc-cni \
  --addon-version v1.19.0-eksbuild.1 \
  --resolve-conflicts PRESERVE \
  --configuration-values '{"env":{"ENABLE_PREFIX_DELEGATION":"true"}}'

# Check add-on status
aws eks describe-addon \
  --cluster-name my-cluster \
  --addon-name vpc-cni \
  --query 'addon.status'

Conflict Resolution Modes

• OVERWRITE: EKS overwrites any in-cluster customizations with the add-on defaults. Use for initial creation or when you want to reset to defaults.
• PRESERVE: EKS updates the add-on version but preserves any fields you’ve modified in-cluster via Kubernetes API. Use for updates. This is now the recommended default for updates.
• NONE: The update fails if there are conflicts. Use if you want explicit control and don’t want silent overwrites.

Terraform pattern:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

resource "aws_eks_addon" "vpc_cni" {
  cluster_name             = aws_eks_cluster.main.name
  addon_name               = "vpc-cni"
  addon_version            = "v1.19.0-eksbuild.1"
  service_account_role_arn = aws_iam_role.vpc_cni.arn

  resolve_conflicts_on_create = "OVERWRITE"
  resolve_conflicts_on_update = "PRESERVE"

  configuration_values = jsonencode({
    env = {
      ENABLE_PREFIX_DELEGATION = "true"
      WARM_PREFIX_TARGET        = "1"
    }
  })
}

resource "aws_eks_addon" "ebs_csi" {
  cluster_name             = aws_eks_cluster.main.name
  addon_name               = "aws-ebs-csi-driver"
  addon_version            = "v1.38.1-eksbuild.1"
  service_account_role_arn = aws_iam_role.ebs_csi.arn

  resolve_conflicts_on_create = "OVERWRITE"
  resolve_conflicts_on_update = "PRESERVE"
}

resource "aws_eks_addon" "coredns" {
  cluster_name  = aws_eks_cluster.main.name
  addon_name    = "coredns"
  addon_version = "v1.11.4-eksbuild.2"

  resolve_conflicts_on_create = "OVERWRITE"
  resolve_conflicts_on_update = "PRESERVE"
}

gotcha: If you created your cluster with eksctl without a config file, or with tools other than the AWS Console, you got self-managed add-ons (Helm charts or bare manifests), not EKS-managed add-ons. To get EKS management, you must adopt the add-on — set --resolve-conflicts OVERWRITE on the first create-addon call. After that, use PRESERVE.

gotcha: The eks:addon-cluster-admin ClusterRoleBinding must exist for add-ons to install. If someone deleted it for security hardening reasons (it does bind cluster-admin), add-ons stop working. Check before blaming the add-on itself.

7. Managed Node Groups in Depth

Managed node groups create an Auto Scaling Group in your account. EKS owns the lifecycle — provisioning, updating, terminating. You own the configuration.

AMI Families (as of 2026)

AL2 is gone (EOL November 2025). Current options:

• AL2023 (AL2023_x86_64_STANDARD, AL2023_ARM_64_STANDARD, AL2023_x86_64_NEURON, AL2023_x86_64_NVIDIA) — recommended default
• Bottlerocket (BOTTLEROCKET_x86_64, BOTTLEROCKET_ARM_64, BOTTLEROCKET_x86_64_NVIDIA) — minimal OS, atomic updates, best security posture
• Windows (WINDOWS_CORE_2019_x86_64, WINDOWS_FULL_2022_x86_64, etc.)

AL2023 uses nodeadm for initialization — a YAML-based config system. If you use custom user data with a launch template, you must use MIME multi-part format for AL2023, TOML for Bottlerocket.

Terraform: Managed Node Group

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

resource "aws_eks_node_group" "general" {
  cluster_name    = aws_eks_cluster.main.name
  node_group_name = "general"
  node_role_arn   = aws_iam_role.node.arn
  subnet_ids      = var.private_subnet_ids

  ami_type       = "AL2023_x86_64_STANDARD"
  instance_types = ["m7g.xlarge", "m6g.xlarge"]  # Note: mixing ARM types
  capacity_type  = "ON_DEMAND"
  disk_size      = 50

  scaling_config {
    desired_size = 3
    min_size     = 2
    max_size     = 10
  }

  update_config {
    max_unavailable_percentage = 25  # or max_unavailable = 1
  }

  labels = {
    role = "general"
  }

  taint {
    key    = "dedicated"
    value  = "general"
    effect = "NO_SCHEDULE"
  }

  launch_template {
    id      = aws_launch_template.node.id
    version = aws_launch_template.node.latest_version
  }

  lifecycle {
    ignore_changes = [scaling_config[0].desired_size]
  }
}

Update Strategies

• FORCE_UPDATE: Ignore PodDisruptionBudgets. Nodes are replaced regardless. Use only when you know what you’re doing.
• RESPECT_POD_DISRUPTION_BUDGET: Default. Honors PDBs. If a PDB blocks eviction, the update stalls. EKS will wait and retry. The update can appear stuck if your PDBs are too aggressive.

gotcha: If a PDB has minAvailable: 1 on a single-replica deployment, a node drain will block forever. Check your PDBs before upgrades. The standard debugging move is kubectl get pdb -A and kubectl describe pdb <name> to see which pods are blocking.

Custom User Data (AL2023 with nodeadm)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

# In the launch template, UserData field, Base64-encoded MIME multi-part:
Content-Type: multipart/mixed; boundary="==BOUNDARY=="
MIME-Version: 1.0

--==BOUNDARY==
Content-Type: application/node.eks.aws

---
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  kubelet:
    config:
      maxPods: 110
    flags:
      - --node-labels=custom-label=my-value

--==BOUNDARY==
Content-Type: text/x-shellscript

#!/bin/bash
# Custom initialization here
/usr/bin/setup-monitoring.sh

--==BOUNDARY==--

8. Karpenter on EKS

Karpenter v1 (GA August 2024, currently ~v1.12) replaces the Kubernetes Cluster Autoscaler for most EKS use cases. The Cluster Autoscaler scales existing node groups. Karpenter provisions new nodes from scratch based on pending pod requirements — it bins packs at decision time.

Key CRDs: NodePool and EC2NodeClass

EC2NodeClass — defines the AWS-specific node configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  # Either 'role' (Karpenter manages instance profile) or 'instanceProfile' (you manage it)
  role: "KarpenterNodeRole-my-cluster"

  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: "my-cluster"

  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: "my-cluster"

  # AL2023 is the recommended AMI family
  amiFamily: AL2023

  # Pin to a specific AMI release (recommended for production)
  amiSelectorTerms:
    - alias: al2023@latest

  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 50Gi
        volumeType: gp3
        encrypted: true
        iops: 3000
        throughput: 125

  tags:
    Environment: production
    ManagedBy: karpenter

NodePool — defines scheduling requirements and disruption behavior:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    metadata:
      labels:
        node-type: general
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default

      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]  # Multi-arch = more instance diversity
        - key: kubernetes.io/os
          operator: In
          values: ["linux"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gte
          values: ["4"]
        - key: karpenter.k8s.aws/instance-size
          operator: NotIn
          values: ["nano", "micro", "small", "medium"]

      # Replace nodes after 720 hours (30 days) — forces AMI refresh
      expireAfter: 720h

      # Max time to drain before forceful delete
      terminationGracePeriod: 48h

  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m

    # Rate-limit disruption: max 20% of nodes at once, never during daily maintenance
    budgets:
      - nodes: "20%"
      - nodes: "0"
        schedule: "0 2 * * *"   # No disruption at 2 AM UTC
        duration: 30m
        reasons: ["Underutilized"]

  limits:
    cpu: "1000"
    memory: 2000Gi

  # Higher weight = higher priority when multiple NodePools match
  weight: 50

Spot Interruption Handling

Karpenter watches an SQS queue (populated by EventBridge rules) for EC2 Spot interruption warnings. On receiving a warning, it immediately cordons and drains the node, giving the 2-minute window maximum use. The EventBridge rules you need:

1
2
3
4
5
6

# Karpenter creates these automatically during install via Helm,
# but here's what they watch:
# - EC2 Spot Instance Interruption Warning
# - EC2 Instance Rebalance Recommendation
# - EC2 Instance State-change Notification
# - EC2 Scheduled Change (Health events)

For Spot-to-Spot consolidation (replacing a larger spot instance with a smaller cheaper one), Karpenter requires a minimum of 15 diverse instance types in the NodePool requirements. Spot-to-Spot consolidation is off by default — the risk of consolidation triggering a cascade if spot capacity is tight is real.

Bin-Packing and Instance Selection

Karpenter selects the cheapest instance type that fits the pending pods’ combined resource requests. It evaluates all instance types matching the NodePool requirements, simulates scheduling all pending pods onto each candidate instance type, and picks the lowest-cost option. This is fundamentally different from Cluster Autoscaler which scales up the cheapest node group — Karpenter picks the cheapest individual instance type at decision time.

For cost optimization, the key levers are:

1. Broad instance type diversity: The more instance types in the requirements, the more likely Karpenter finds cheap capacity.
2. Multi-arch: Including arm64 (Graviton) opens ~40% cheaper instances for equivalent compute.
3. Spot first: Set karpenter.sh/capacity-type to include spot with on-demand as fallback.
4. Consolidation: WhenEmptyOrUnderutilized continuously optimizes the fleet.

Drift Detection

Karpenter hashes the EC2NodeClass spec and NodePool requirements. When the hash changes (e.g., you update the AMI alias, change security groups, or upgrade the cluster and a new AMI becomes the latest for the family), existing nodes are marked as “drifted” and replaced rolling. This is how cluster upgrades propagate to Karpenter nodes automatically — upgrade the control plane, Karpenter detects AMI drift, replaces nodes.

gotcha: expireAfter: 720h (30 days) ensures nodes are replaced at least monthly even if no configuration changes. Without this, you can end up with very old nodes that are technically not drifted but running ancient AMIs.

gotcha: The karpenter.sh/do-not-disrupt: "true" annotation on a pod prevents voluntary disruption (consolidation, expiry, drift). It does NOT prevent interruption events or forced termination. Use it for genuinely long-running stateful work, but audit your pods periodically — it’s common for this annotation to get orphaned.

9. Fargate Profiles

Fargate runs pods on AWS-managed microVMs — one pod per VM, kernel isolation, no node to manage.

Profile Configuration

1
2
3
4
5
6

aws eks create-fargate-profile \
  --cluster-name my-cluster \
  --fargate-profile-name my-profile \
  --pod-execution-role-arn arn:aws:iam::123456789012:role/AmazonEKSFargatePodExecutionRole \
  --subnets subnet-abc123 subnet-def456 \
  --selectors '[{"namespace":"production"},{"namespace":"batch","labels":{"tier":"fargate"}}]'

Pods matching ANY selector (namespace + optional label set) run on Fargate. Selectors are OR’d, not AND’d. A pod matches if it satisfies any one selector completely.

eksctl YAML:

1
2
3
4
5
6
7

fargateProfiles:
  - name: default
    selectors:
      - namespace: kube-system
        labels:
          k8s-app: kube-dns
      - namespace: production

What Does Not Work on Fargate

Hard limitations (as of 2026, confirmed from AWS docs):

The most commonly hit limitation in production: DaemonSets. If you’re using Datadog, Fluent Bit, or any other agent as a DaemonSet, it won’t run on Fargate nodes. Your options are sidecars in each pod (adds complexity, resource overhead) or run those workloads on EC2 nodes.

Cost

Fargate pricing is based on vCPU-seconds and GB-seconds of memory. Fargate rounds up to the next available configuration (minimum 0.25 vCPU, 0.5 GB). For a typical web service pod requesting 500m CPU / 512Mi, Fargate charges for 0.5 vCPU / 1 GB (rounds up). At US-East-1 rates, that’s roughly $0.020 vCPU-hour + $0.002 GB-hour ≈ $0.012/hour per pod. Compare to a t3.medium (2 vCPU, 4 GB) at ~$0.042/hour that can run 4 such pods ($0.0105/pod/hour). At small scale, Fargate is comparable. At large scale with proper bin-packing (Karpenter on Graviton Spot), EC2 is much cheaper.

When Fargate makes sense: Batch jobs with infrequent, unpredictable scheduling needs. Security-sensitive workloads requiring VM-level isolation. Teams that want zero node operations burden and accept the cost premium. Overflow burst capacity alongside a smaller EC2 baseline.

10. Cluster Upgrades with Zero Downtime

The rule that matters most: You cannot skip minor versions. 1.33 → 1.35 requires two upgrades: 1.33 → 1.34, then 1.34 → 1.35. Plan your upgrade cadence accordingly — falling multiple versions behind is an operational emergency.

Skew Policy

From Kubernetes 1.28+: the supported skew between the API server and kubelet extended from n-2 to n-3. Your nodes can be up to 3 minor versions behind the control plane. In practice, keep nodes within 1 version — the compatibility window exists for operational grace, not as a long-term config.

The Upgrade Sequence

1. Pre-upgrade checks

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

# Check for deprecated API usage
aws eks list-insights \
  --cluster-name my-cluster \
  --region us-east-1

# Run kubent to find deprecated API usage
kubent  # github.com/doitintl/kube-no-trouble

# Verify cluster subnets have ≥ 5 free IPs
aws ec2 describe-subnets \
  --subnet-ids $(aws eks describe-cluster --name my-cluster \
    --query 'cluster.resourcesVpcConfig.subnetIds' --output text) \
  --query 'Subnets[*].[SubnetId,AvailableIpAddressCount]' \
  --output table

# Verify the EKS IAM role still exists and has correct permissions
ROLE_ARN=$(aws eks describe-cluster --name my-cluster \
  --query 'cluster.roleArn' --output text)
aws iam get-role --role-name ${ROLE_ARN##*/}

2. Upgrade the control plane

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# Using AWS CLI
aws eks update-cluster-version \
  --name my-cluster \
  --kubernetes-version 1.35

# Monitor the update
aws eks describe-cluster \
  --name my-cluster \
  --query 'cluster.status'

# Watch for completion
aws eks wait cluster-active --name my-cluster

The API server briefly cycles during control plane upgrade. AWS does a rolling update across its managed instances, so there is a short period of reduced API server capacity. kubectl calls may get transient errors. This is expected and typically lasts 10–30 minutes.

3. Upgrade add-ons (immediately after control plane)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

# Find compatible versions for target k8s version
aws eks describe-addon-versions \
  --kubernetes-version 1.35 \
  --addon-name coredns \
  --query 'addons[].addonVersions[?compatibilities[?defaultVersion==`true`]].addonVersion'

# Upgrade each add-on
for addon in vpc-cni coredns kube-proxy aws-ebs-csi-driver; do
  echo "Upgrading $addon..."
  # Get the default version for the new k8s version
  VERSION=$(aws eks describe-addon-versions \
    --kubernetes-version 1.35 \
    --addon-name $addon \
    --query 'addons[].addonVersions[?compatibilities[?defaultVersion==`true`]].addonVersion | [0][0]' \
    --output text)
  
  aws eks update-addon \
    --cluster-name my-cluster \
    --addon-name $addon \
    --addon-version $VERSION \
    --resolve-conflicts PRESERVE
done

4. Upgrade node groups

1
2
3
4
5
6
7
8
9

# Managed node group upgrade
aws eks update-nodegroup-version \
  --cluster-name my-cluster \
  --nodegroup-name general \
  --kubernetes-version 1.35 \
  --update-config '{"maxUnavailablePercentage": 25}'

# Watch node rollout
kubectl get nodes --watch

5. Restart Fargate workloads

1
2
3

# Fargate nodes don't auto-upgrade — you must redeploy pods
kubectl get pods -A -o wide | grep fargate | awk '{print $1, $2}' | \
  while read ns pod; do kubectl rollout restart deployment -n $ns; done

PodDisruptionBudgets During Upgrades

The node drain during upgrade respects PDBs. A well-configured PDB for a critical service:

1
2
3
4
5
6
7
8
9

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-api
spec:
  minAvailable: "80%"
  selector:
    matchLabels:
      app: my-api

topologySpreadConstraints to ensure pods land on multiple nodes/AZs during drain:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: my-api
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: my-api

Blue/Green Cluster Upgrades

Alternative to in-place when: you need to skip multiple versions at once, you want a rollback option, or you’re migrating to a new cluster architecture simultaneously.

Downsides: Costs 2x during migration, API endpoint and OIDC issuer change (update kubeconfig everywhere, update IRSA trust policies for new cluster), stateful workloads require data migration. With GitOps (ArgoCD/Flux) this is much easier — just point the source at the new cluster endpoint.

eksctl Upgrade Workflow

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# Upgrade control plane
eksctl upgrade cluster --name my-cluster --version 1.35 --approve

# Upgrade add-ons
eksctl utils update-coredns --cluster my-cluster --approve
eksctl utils update-kube-proxy --cluster my-cluster --approve  
eksctl utils update-aws-node --cluster my-cluster --approve

# Upgrade managed node groups
eksctl upgrade nodegroup \
  --name general \
  --cluster my-cluster \
  --kubernetes-version 1.35

11. EKS Anywhere

EKS Anywhere lets you run EKS-conformant clusters on your own infrastructure using the same APIs and tooling as cloud EKS. It is not the same as EKS — the control plane runs on your infrastructure (not AWS-managed), and many cloud-EKS features don’t exist.

Infrastructure Providers

• Bare metal (Tinkerbell-based provisioner)
• VMware vSphere
• Nutanix
• Apache CloudStack
• AWS Snow (for edge/disconnected environments)
• Docker (local development only)

Air-Gapped Operation

EKS Anywhere is designed for air-gapped environments. Pull all container images to a local registry mirror first:

1
2
3
4
5
6
7

# Download artifacts
eksctl anywhere download artifacts --bundles-override bundles.yaml

# Copy images to local registry
eksctl anywhere import images \
  --input eks-anywhere-downloads.tar.gz \
  --bundles bundles.yaml

For bare metal air-gapped clusters, you must set osImageURL and hookImagesURLPath in the cluster spec to point to your local image repository.

Curated Packages

EKS Anywhere Curated Packages are AWS-tested software packages available for enterprise subscribers: Prometheus, Grafana, Harbor (container registry), Emissary Ingress, MetalLB, Cert-Manager, ADOT, and others. Packages require an AWS Enterprise subscription and connectivity to AWS Package Controller (or mirror to local registry for air-gapped).

Key Differences from Cloud EKS

Use cases: On-premises environments with data residency requirements, factories/edge locations with intermittent connectivity, compliance environments where data cannot leave a physical location, air force-style disconnected classified deployments.

12. Cluster Access Management

The Old Way: aws-auth ConfigMap

The kube-system/aws-auth ConfigMap maps IAM principals to Kubernetes RBAC subjects. It was the only way to grant access for years. It is now deprecated.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: arn:aws:iam::123456789012:role/MyNodeRole
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes
    - rolearn: arn:aws:iam::123456789012:role/MyDevRole
      username: dev-user
      groups:
        - dev-team
  mapUsers: |
    - userarn: arn:aws:iam::123456789012:user/alice
      username: alice
      groups:
        - system:masters

The problem with aws-auth: It’s a ConfigMap. A typo breaks cluster access for everyone. The cluster creator has implicit cluster-admin — if they leave the company and their IAM user/role is deleted, you can potentially lose admin access to the cluster. No CloudTrail audit trail for changes. Messy to manage with IaC.

The New Way: EKS Access Entries (GA 2024)

Access entries are API objects — created via EKS API, tracked in CloudTrail, manageable with IaC. They work for clusters with platform version >= required (effectively all current clusters).

Step 1: Enable API authentication mode

1
2
3
4
5
6
7
8

# Check current mode
aws eks describe-cluster --name my-cluster \
  --query 'cluster.accessConfig.authenticationMode'

# Enable access entries (use API_AND_CONFIG_MAP during migration)
aws eks update-cluster-config \
  --name my-cluster \
  --access-config authenticationMode=API_AND_CONFIG_MAP

Step 2: Create access entries

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

# Grant cluster admin to a role
aws eks create-access-entry \
  --cluster-name my-cluster \
  --principal-arn arn:aws:iam::123456789012:role/MyAdminRole

aws eks associate-access-policy \
  --cluster-name my-cluster \
  --principal-arn arn:aws:iam::123456789012:role/MyAdminRole \
  --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy \
  --access-scope '{"type":"cluster"}'

# Grant namespace-scoped access
aws eks create-access-entry \
  --cluster-name my-cluster \
  --principal-arn arn:aws:iam::123456789012:role/MyDevRole \
  --kubernetes-groups '["dev-team"]'

aws eks associate-access-policy \
  --cluster-name my-cluster \
  --principal-arn arn:aws:iam::123456789012:role/MyDevRole \
  --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSEditPolicy \
  --access-scope '{"type":"namespace","namespaces":["production","staging"]}'

AWS managed access policies:

• AmazonEKSClusterAdminPolicy — cluster-admin
• AmazonEKSAdminPolicy — admin (no cluster-level resources)
• AmazonEKSEditPolicy — edit (create/update/delete namespaced resources, no RBAC)
• AmazonEKSViewPolicy — view (read-only)
• AmazonEKSAdminViewPolicy — view everything including secrets

Terraform:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

resource "aws_eks_access_entry" "admin" {
  cluster_name  = aws_eks_cluster.main.name
  principal_arn = aws_iam_role.cluster_admin.arn
  type          = "STANDARD"
}

resource "aws_eks_access_policy_association" "admin" {
  cluster_name  = aws_eks_cluster.main.name
  principal_arn = aws_iam_role.cluster_admin.arn
  policy_arn    = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy"

  access_scope {
    type = "cluster"
  }
}

Migration from aws-auth

The safest migration path:

1. Set authenticationMode=API_AND_CONFIG_MAP — both work simultaneously
2. Create access entries for all principals currently in aws-auth
3. Test access with each principal
4. Set authenticationMode=API — aws-auth is fully ignored
5. (Optional) Delete aws-auth to avoid confusion

gotcha: When switching to API mode, only the original cluster creator’s access entry is automatically created. All other entries from aws-auth must be manually recreated. If you delete aws-auth entries for managed node groups before creating the corresponding access entries, nodes stop functioning. Check that access entries for node roles exist before removing them from aws-auth.

13. Cost Optimization Patterns

Spot Instances with Karpenter

The most impactful cost lever. Spot can be 70–90% cheaper than on-demand for the same instance. The key to stable Spot usage:

1. Diversify instance types broadly — 15+ instance types means AWS reclaiming one type doesn’t take you down
2. Use capacity-weighted selection — Karpenter does this automatically
3. Handle interruptions — Karpenter with SQS interrupt queue gives 2-minute warning drain
4. Don’t run stateful workloads on Spot without checkpointing or replication

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

# NodePool for mixed on-demand baseline + spot burst
---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: on-demand-baseline
spec:
  template:
    spec:
      nodeClassRef:
        name: default
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["m", "r"]
  weight: 100  # Prefer on-demand for system workloads

---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: spot-burst
spec:
  template:
    spec:
      nodeClassRef:
        name: default
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r", "t"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gte
          values: ["3"]
  weight: 50  # Lower priority, used for burst

Use pod annotations to control which pool a workload lands on:

1
2
3

spec:
  nodeSelector:
    karpenter.sh/capacity-type: spot  # Force spot

Graviton (arm64) Nodes

AWS Graviton instances (m7g, c7g, r7g, m6g, c6g, r6g) are typically 20–40% cheaper than equivalent x86 instances with equal or better performance for most workloads. Multi-arch images are necessary.

Build multi-arch images:

1
2
3
4

docker buildx build \
  --platform linux/amd64,linux/arm64 \
  -t my-registry/my-app:latest \
  --push .

Enable Graviton in Karpenter NodePool:

1
2
3
4

requirements:
  - key: kubernetes.io/arch
    operator: In
    values: ["amd64", "arm64"]  # Karpenter picks cheapest

gotcha: Some language runtimes and native extensions have subtle arm64 issues. Test thoroughly. Java, Python, Node.js, and Go are well-supported. C extensions and some Rust crates need recompilation.

Compute Savings Plans

Savings Plans commit to a consistent dollar amount of compute usage per hour (not specific instance types), giving 20–60% discount. For EKS:

1. Compute Savings Plans (most flexible) — apply to any EC2 usage including EKS nodes, Lambda, Fargate
2. EC2 Instance Savings Plans — locked to instance family and region, highest discount

Strategy: Use Compute Savings Plans for your on-demand baseline (what you’d run even at minimum load). Let Karpenter handle bursts with Spot. Layer multiple Savings Plans as your baseline grows.

KEDA for Event-Driven Scale-to-Zero

KEDA (Kubernetes Event-Driven Autoscaler) extends HPA with external event sources and enables scale-to-zero between events.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: my-worker
spec:
  scaleTargetRef:
    name: my-worker-deployment
  minReplicaCount: 0   # Scale to zero!
  maxReplicaCount: 50
  triggers:
    - type: aws-sqs-queue
      authenticationRef:
        name: keda-sqs-auth
      metadata:
        queueURL: https://sqs.us-east-1.amazonaws.com/123456789/my-queue
        queueLength: "5"     # Scale up when queue depth > 5
        awsRegion: us-east-1
    - type: kafka
      metadata:
        bootstrapServers: kafka:9092
        consumerGroup: my-group
        topic: my-topic
        lagThreshold: "100"

When combined with Karpenter, scale-to-zero workloads cause nodes to consolidate (WhenEmpty), so you pay nothing between event bursts.

VPA for Rightsizing

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: my-app
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  updatePolicy:
    updateMode: "Off"  # Recommendation only — don't auto-update in production until tested
  resourcePolicy:
    containerPolicies:
      - containerName: "*"
        minAllowed:
          cpu: 50m
          memory: 64Mi
        maxAllowed:
          cpu: 4
          memory: 8Gi

Run VPA in Off mode for two weeks, then check kubectl describe vpa my-app for the recommended values. Apply them to your deployment. This alone can cut compute costs 20–40% on over-provisioned services.

14. Observability

Container Insights

CloudWatch Container Insights collects cluster, node, pod, and container metrics. Two deployment models:

1. DaemonSet (traditional): CloudWatch agent DaemonSet + Fluent Bit DaemonSet
2. Managed add-on: amazon-cloudwatch-observability add-on (installs and manages both)

1
2
3
4

aws eks create-addon \
  --cluster-name my-cluster \
  --addon-name amazon-cloudwatch-observability \
  --service-account-role-arn arn:aws:iam::123456789012:role/CloudWatchAgentRole

The add-on installs the CloudWatch agent and Fluent Bit. Requires an IAM role with CloudWatchAgentServerPolicy for the agent’s service account (use IRSA or Pod Identity).

EKS Control Plane Logging

Five log streams under /aws/eks/<cluster>/cluster in CloudWatch Logs:

• kube-apiserver — API server requests
• kube-apiserver-audit — every API call with who made it and what they did
• authenticator — IAM authentication attempts (success and failure)
• kube-controller-manager — reconciliation loops, garbage collection
• kube-scheduler — scheduling decisions

For security investigations, the kube-apiserver-audit log is essential. Query it with CloudWatch Logs Insights:

# Find all secret reads in the last hour
fields @timestamp, @message
| filter @logStream like /kube-apiserver-audit/
| filter objectRef.resource == "secrets"
| filter verb == "get"
| sort @timestamp desc
| limit 100

Fluent Bit for Log Shipping

Fluent Bit is the standard log shipper for EKS. Installed automatically with the amazon-cloudwatch-observability add-on, or deployed independently.

Custom Fluent Bit configuration for shipping to multiple destinations:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

[INPUT]
    Name              tail
    Tag               kube.*
    Path              /var/log/containers/*.log
    Parser            docker
    DB                /var/log/flb_kube.db
    Mem_Buf_Limit     5MB
    Skip_Long_Lines   On
    Refresh_Interval  10

[FILTER]
    Name                kubernetes
    Match               kube.*
    Kube_URL            https://kubernetes.default.svc:443
    Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
    Merge_Log           On
    Keep_Log            Off
    K8S-Logging.Parser  On
    K8S-Logging.Exclude On

[OUTPUT]
    Name                cloudwatch_logs
    Match               kube.*
    region              us-east-1
    log_group_name      /eks/my-cluster/containers
    log_stream_prefix   ${HOST}-
    auto_create_group   On

[OUTPUT]
    Name                opensearch
    Match               kube.*
    Host                my-opensearch.us-east-1.es.amazonaws.com
    Port                443
    TLS                 On
    Index               eks-logs
    Type                _doc

ADOT (AWS Distro for OpenTelemetry)

ADOT is AWS’s distribution of OpenTelemetry Collector, available as an EKS add-on. Use it for:

• Shipping metrics to CloudWatch (Container Insights) or Amazon Managed Prometheus
• Shipping traces to AWS X-Ray
• Forwarding to third-party observability platforms (Datadog, Dynatrace) via OTLP

1
2
3
4

aws eks create-addon \
  --cluster-name my-cluster \
  --addon-name adot \
  --service-account-role-arn arn:aws:iam::123456789012:role/AdotRole

Configure via the AmazonCloudWatchAgent CRD (deployed by the add-on) or directly with OpenTelemetryCollector CRD.

gotcha: ADOT and Container Insights both use CloudWatch. Running both without coordination creates duplicate metrics and double cost. The amazon-cloudwatch-observability add-on includes ADOT-based collection internally. Don’t deploy the ADOT add-on separately if you’re using the CloudWatch Observability add-on unless you need traces or custom pipelines.

Common Production Gotchas Summary

1. IP exhaustion: Enable prefix delegation early. Retrofitting it onto running clusters with fragmented subnets is painful.
2. aws-auth typo: One malformed YAML in aws-auth breaks ALL cluster access. Migrate to access entries.
3. Add-ons not auto-upgraded: After a cluster upgrade, add-ons stay at the old version. Explicitly upgrade them — especially VPC CNI and CoreDNS which have version compatibility requirements.
4. Skipping minor versions: You can’t. Going from 1.30 to 1.35 requires 5 separate upgrades.
5. PDB + single-replica deployments: minAvailable: 1 on a 1-replica deployment makes node drain impossible. Use maxUnavailable: 1 instead, or ensure >= 2 replicas.
6. Fargate + DaemonSets: Doesn’t work. Every monitoring, security scanning, or network tool that runs as a DaemonSet will not see Fargate pods.
7. IRSA + OIDC per cluster: Every cluster has its own OIDC issuer URL in the trust policy. Use Pod Identity for new clusters to avoid this multiplication.
8. Security groups for pods + t-family instances: t3, t3a, t4g instances don’t support trunk ENIs. Security groups for pods silently doesn’t work on them.
9. Karpenter drift without expireAfter: Nodes won’t get replaced unless configuration changes. Set expireAfter: 720h to force monthly rotation.
10. VPC subnet selection in cluster config: The subnets in the EKS cluster VPC config are for cross-account control-plane ENIs, not for your nodes. Don’t confuse them.
11. Fargate Spot: It doesn’t exist for EKS. Don’t plan for it.
12. Control plane logging disabled by default: Enable it before you need to debug. You cannot query logs that weren’t captured.
13. Extended support cost: Clusters on Kubernetes versions past standard support incur additional hourly charges. Budget for this if you defer upgrades.

Sources:

• EKS Kubernetes Version Lifecycle
• Amazon EKS and EKS Distro now supports Kubernetes version 1.35
• Announcing Karpenter 1.0
• Karpenter Compatibility
• Karpenter Disruption
• Karpenter NodePools
• Karpenter EC2NodeClasses
• EKS Pod Identities
• Amazon EKS Pod Identity blog
• IRSA - Assign IAM roles to service accounts
• EKS Access Entries
• Migrating aws-auth to access entries
• Best Practices for Cluster Upgrades
• Security Groups for Pods
• Amazon VPC CNI Best Practices
• Prefix Mode for Linux
• EKS Add-ons
• Fargate on EKS
• EKS Anywhere Overview
• EKS Anywhere Air-gapped
• EKS Cluster Services Scaling
• Mixing Graviton with x86 on EKS
• EKS endoflife.date
• AL2023 upgrade for EKS