Multi-Cluster Kubernetes: Fleet Management, Cross-Cluster Service Discovery, and Traffic Routing

March 26, 2026 18 min read Virtualization & Containers

kubernetesmulti-clustercluster-apikarmadafederationplatform-engineering

Contents

Single-cluster Kubernetes gets you surprisingly far. Many organizations run production workloads on one cluster for years without hitting fundamental limits. But there are real reasons to run multiple clusters — regulatory data residency, fault isolation, team autonomy, blast radius reduction, and geographic distribution. When those reasons apply, multi-cluster architecture introduces a new set of hard problems.

This guide covers the core patterns: why multi-cluster, how to provision and manage a fleet with Cluster API, how to distribute workloads with Karmada, and how to handle the genuinely difficult problems of cross-cluster service discovery and traffic routing.

When Multi-Cluster Makes Sense

Before diving into tools, be honest about the costs. Each additional cluster multiplies your operational surface: more control planes to upgrade, more certificate rotations, more etcd backups, more network policies to audit. The management overhead is real.

Go multi-cluster when:

Regulatory isolation: GDPR/HIPAA data that cannot leave a jurisdiction, PCI workloads requiring hard boundaries
Blast radius reduction: a bad deployment or control plane failure shouldn’t take down everything — production and staging in separate clusters means staging incidents don’t cascade
Team autonomy: large organizations where teams need full cluster-admin without sharing blast radius with other teams
Geographic distribution: users in different regions need low-latency access; putting workloads closer to users requires clusters in multiple regions
Different security postures: internet-facing workloads in one cluster, internal workloads in another — different firewall rules, different compliance requirements
Scaling limits: very large clusters (1000+ nodes) hit etcd performance limits; horizontal scaling via multiple clusters avoids this

Don’t go multi-cluster when:

You’re doing it for environment separation (dev/staging/prod) — namespaces handle this fine with proper RBAC
Your team doesn’t have the operational maturity to manage one cluster well
You haven’t hit a real constraint that multi-cluster solves

Cluster API: Declarative Cluster Lifecycle Management

Cluster API (CAPI) is the Kubernetes-native way to provision and manage clusters. It brings the reconciler pattern to cluster lifecycle: you declare the desired state of a cluster as Kubernetes objects, and CAPI controllers make it real.

Architecture

CAPI has two layers:

Management cluster: a Kubernetes cluster where CAPI controllers run and cluster objects are stored. This is your “meta-cluster” — you apply Cluster, MachineDeployment, and related objects here to provision workload clusters.

Workload clusters: the clusters CAPI provisions and manages. They’re target clusters for your actual applications.

CAPI separates concerns via providers:

Bootstrap providers: configure new machines (kubeadm, k3s, RKE2)
Control plane providers: manage the control plane (KubeadmControlPlane, Talos)
Infrastructure providers: provision the underlying compute (AWS, Azure, GCP, vSphere, MAAS, Docker for local dev)

Install CAPI with clusterctl

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


# Install clusterctl
curl -L https://github.com/kubernetes-sigs/cluster-api/releases/latest/download/clusterctl-linux-amd64 \
  -o clusterctl
chmod +x clusterctl && sudo mv clusterctl /usr/local/bin/

# Initialize CAPI on a management cluster (AWS example)
export AWS_REGION=us-east-1
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_B64ENCODED_CREDENTIALS=$(clusterawsadm bootstrap credentials encode-as-profile)

clusterctl init --infrastructure aws

# For a multi-provider setup
clusterctl init \
  --infrastructure aws:v2.5.0 \
  --bootstrap kubeadm:v1.7.0 \
  --control-plane kubeadm:v1.7.0

# Verify controllers are running
kubectl -n capi-system get pods
kubectl -n capa-system get pods      # AWS provider

Provisioning a Cluster

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105


# cluster.yaml — a complete cluster definition for AWS
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: production-us-east
  namespace: clusters
  labels:
    env: production
    region: us-east-1
spec:
  clusterNetwork:
    pods:
      cidrBlocks: ["10.244.0.0/16"]
    services:
      cidrBlocks: ["10.96.0.0/12"]
  infrastructureRef:
    kind: AWSCluster
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
    name: production-us-east
  controlPlaneRef:
    kind: KubeadmControlPlane
    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    name: production-us-east-cp
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: AWSCluster
metadata:
  name: production-us-east
  namespace: clusters
spec:
  region: us-east-1
  sshKeyName: my-ssh-key
  network:
    vpc:
      availabilityZoneUsageLimit: 3
      cidrBlock: "10.0.0.0/16"
---
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
  name: production-us-east-cp
  namespace: clusters
spec:
  replicas: 3           # HA control plane
  version: v1.29.0
  machineTemplate:
    infrastructureRef:
      kind: AWSMachineTemplate
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
      name: production-us-east-cp-machine
  kubeadmConfigSpec:
    initConfiguration:
      nodeRegistration:
        name: "{{ ds.meta_data.local_hostname }}"
        kubeletExtraArgs:
          cloud-provider: external
    clusterConfiguration:
      apiServer:
        extraArgs:
          cloud-provider: external
      controllerManager:
        extraArgs:
          cloud-provider: external
    joinConfiguration:
      nodeRegistration:
        kubeletExtraArgs:
          cloud-provider: external
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: AWSMachineTemplate
metadata:
  name: production-us-east-cp-machine
  namespace: clusters
spec:
  template:
    spec:
      instanceType: m5.xlarge
      iamInstanceProfile: control-plane.cluster-api-provider-aws.sigs.k8s.io
      ami:
        id: ami-0123456789abcdef0   # Ubuntu 22.04 with kubeadm pre-installed
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: production-us-east-workers
  namespace: clusters
spec:
  clusterName: production-us-east
  replicas: 5
  selector:
    matchLabels:
      cluster.x-k8s.io/cluster-name: production-us-east
  template:
    spec:
      clusterName: production-us-east
      version: v1.29.0
      bootstrap:
        configRef:
          kind: KubeadmConfigTemplate
          apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
          name: production-us-east-workers
      infrastructureRef:
        kind: AWSMachineTemplate
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
        name: production-us-east-workers-machine

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# Apply the cluster definition
kubectl apply -f cluster.yaml

# Watch provisioning progress
clusterctl describe cluster production-us-east -n clusters

# Get the kubeconfig for the new cluster
clusterctl get kubeconfig production-us-east -n clusters > prod-us-east.kubeconfig

# Verify
kubectl --kubeconfig=prod-us-east.kubeconfig get nodes

ClusterClass: Reusable Cluster Templates

ClusterClass lets you define a cluster template once and instantiate it many times with different parameters:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49


apiVersion: cluster.x-k8s.io/v1beta1
kind: ClusterClass
metadata:
  name: production-aws-class
  namespace: clusters
spec:
  controlPlane:
    ref:
      apiVersion: controlplane.cluster.x-k8s.io/v1beta1
      kind: KubeadmControlPlaneTemplate
      name: production-cp-template
    machineInfrastructure:
      ref:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
        kind: AWSMachineTemplate
        name: production-cp-machine-template
  infrastructure:
    ref:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
      kind: AWSClusterTemplate
      name: production-aws-template
  workers:
    machineDeployments:
      - class: default-worker
        template:
          bootstrap:
            ref:
              apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
              kind: KubeadmConfigTemplate
              name: production-worker-bootstrap
          infrastructure:
            ref:
              apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
              kind: AWSMachineTemplate
              name: production-worker-machine-template
  variables:
    - name: workerCount
      required: true
      schema:
        openAPIV3Schema:
          type: integer
          minimum: 1
          maximum: 100
    - name: instanceType
      required: false
      schema:
        openAPIV3Schema:
          type: string
          default: m5.large

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


# Instantiate clusters from the class
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: team-payments-cluster
  namespace: clusters
spec:
  topology:
    class: production-aws-class
    version: v1.29.0
    controlPlane:
      replicas: 3
    workers:
      machineDeployments:
        - name: workers
          class: default-worker
          replicas: 10
    variables:
      - name: workerCount
        value: 10
      - name: instanceType
        value: m5.2xlarge

Workload Distribution with Karmada

Karmada (Kubernetes Armada) is a multi-cluster workload management platform. You deploy your applications to Karmada’s control plane, and Karmada distributes them to member clusters according to policies you define.

Architecture

┌──────────────────────────────────────────────────────────────┐
│                    Karmada Control Plane                      │
│                                                              │
│   Karmada API Server (superset of Kubernetes API)            │
│   Karmada Controller Manager                                 │
│   Karmada Scheduler                                          │
│   etcd                                                       │
└───────────────────────┬──────────────────────────────────────┘
                        │
         ┌──────────────┼──────────────┐
         ▼              ▼              ▼
   Cluster: us-east  Cluster: eu-west  Cluster: ap-southeast
   (member)          (member)          (member)

You submit standard Kubernetes objects to Karmada. Karmada propagates them to member clusters and synchronizes status back.

Install Karmada

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


# Install karmadactl
curl -s https://raw.githubusercontent.com/karmada-io/karmada/master/hack/install-cli.sh | sudo bash

# Init Karmada control plane (requires a host cluster)
karmadactl init \
  --kubeconfig ~/.kube/config \
  --karmada-data /etc/karmada \
  --karmada-pki /etc/karmada/pki \
  --cert-external-ip <KARMADA_API_SERVER_IP>

# This creates karmada-system namespace with the control plane components
# Get the Karmada kubeconfig
export KUBECONFIG=/etc/karmada/karmada-apiserver.config

# Join member clusters
karmadactl join us-east \
  --kubeconfig /etc/karmada/karmada-apiserver.config \
  --cluster-kubeconfig ~/.kube/us-east.kubeconfig

karmadactl join eu-west \
  --kubeconfig /etc/karmada/karmada-apiserver.config \
  --cluster-kubeconfig ~/.kube/eu-west.kubeconfig

karmadactl join ap-southeast \
  --kubeconfig /etc/karmada/karmada-apiserver.config \
  --cluster-kubeconfig ~/.kube/ap-southeast.kubeconfig

# View member clusters
kubectl get clusters
# NAME           VERSION   MODE   READY   AGE
# us-east        v1.29.0   Push   True    5m
# eu-west        v1.29.0   Push   True    3m
# ap-southeast   v1.29.0   Push   True    2m

PropagationPolicy: Controlling Distribution

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53


# Deploy an application to all clusters
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-api
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web-api
  template:
    metadata:
      labels:
        app: web-api
    spec:
      containers:
        - name: web-api
          image: mycompany/web-api:v2.1.0
          ports:
            - containerPort: 8080
---
# PropagationPolicy tells Karmada where to send this Deployment
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: web-api-propagation
  namespace: production
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: web-api
  placement:
    clusterAffinity:
      clusterNames:
        - us-east
        - eu-west
        - ap-southeast
    replicaScheduling:
      replicaSchedulingType: Divided     # Split replicas across clusters
      replicaDivisionPreference: Weighted
      weightPreference:
        staticClusterWeight:
          - targetCluster:
              clusterNames: [us-east]
            weight: 5      # 50% of replicas
          - targetCluster:
              clusterNames: [eu-west]
            weight: 3      # 30% of replicas
          - targetCluster:
              clusterNames: [ap-southeast]
            weight: 2      # 20% of replicas

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


# Deploy only to clusters with specific labels (region-aware)
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: eu-only-policy
  namespace: production
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: eu-data-processor
  placement:
    clusterAffinity:
      labelSelector:
        matchLabels:
          region: europe     # Clusters labeled region=europe
    spreadConstraints:
      - spreadByField: cluster
        maxGroups: 2          # At most 2 clusters
        minGroups: 1          # At least 1 cluster

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


# Failover: primary cluster with fallback
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: ha-failover-policy
  namespace: production
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: critical-service
  placement:
    clusterAffinity:
      clusterNames: [us-east]
  failover:
    application:
      decisionConditions:
        tolerationSeconds: 120   # Wait 2 min before failover
      gracePeriodSeconds: 600
      purgeMode: Immediately
    cluster:
      decisionConditions:
        tolerationSeconds: 300

OverridePolicy: Per-Cluster Customization

Often you need slightly different configs per cluster — different image registries, different resource limits, different environment variables:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


apiVersion: policy.karmada.io/v1alpha1
kind: OverridePolicy
metadata:
  name: eu-registry-override
  namespace: production
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
  targetCluster:
    clusterAffinity:
      labelSelector:
        matchLabels:
          region: europe
  overrideRules:
    - targetCluster:
        clusterNames: [eu-west]
      overriders:
        # Rewrite image registry for EU cluster (data residency)
        imageOverrider:
          - component: Registry
            operator: replace
            value: eu.gcr.io
        # Add EU-specific environment variables
        plaintext:
          - path: /spec/template/spec/containers/0/env/-
            operator: add
            value:
              name: DATA_REGION
              value: eu-west-1

Viewing Propagation Status

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# Check where a resource was propagated and its status per cluster
kubectl get resourcebinding web-api-deployment -n production -o yaml

# See status across all clusters
kubectl get rb -n production

# Karmada provides aggregated status
kubectl get deployment web-api -n production
# NAME      READY   UP-TO-DATE   AVAILABLE
# web-api   27/27   27           27
# (sum across all three clusters: 5+3+2 × 3 replicas each)

Cross-Cluster Service Discovery

This is where multi-cluster gets genuinely hard. A pod in us-east sending a request to a Service in eu-west — how does it resolve the name and route the traffic?

Several approaches exist, each with different trade-offs.

Approach 1: Submariner

Submariner creates encrypted tunnels between clusters and extends Service DNS across cluster boundaries.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


# Install subctl
curl -Ls https://get.submariner.io | bash

# Prepare a broker cluster (stores cross-cluster state)
subctl deploy-broker --kubeconfig broker.kubeconfig

# Join clusters to the broker
subctl join broker-info.subm \
  --kubeconfig us-east.kubeconfig \
  --clusterid us-east \
  --natt=false \
  --cable-driver libreswan

subctl join broker-info.subm \
  --kubeconfig eu-west.kubeconfig \
  --clusterid eu-west \
  --natt=false \
  --cable-driver libreswan

# Export a Service from us-east so eu-west can reach it
kubectl --kubeconfig us-east.kubeconfig apply -f - <<EOF
apiVersion: multicluster.x-k8s.io/v1alpha1
kind: ServiceExport
metadata:
  name: orders-service
  namespace: production
EOF

# Now from eu-west, resolve:
# orders-service.production.svc.clusterset.local
# → resolves to the ClusterIP in us-east (via Submariner's Lighthouse DNS plugin)

Submariner creates a full IPsec/WireGuard mesh between clusters — pod IPs are routable across clusters, not just Services.

Approach 2: Istio Multi-Primary

Istio’s multi-cluster mode connects clusters via the service mesh control plane. Each cluster has its own Istio control plane (multi-primary), and they share service discovery via a shared trust root.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42


# Create a shared root CA for both clusters
# (Istio's cert-manager integration can handle this)

# Install Istio on cluster 1 with multi-cluster config
istioctl install --kubeconfig=us-east.kubeconfig \
  -f - <<EOF
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: us-east
      network: us-east-network
EOF

# Install Istio on cluster 2
istioctl install --kubeconfig=eu-west.kubeconfig \
  -f - <<EOF
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: eu-west
      network: eu-west-network
EOF

# Enable endpoint discovery between clusters
# (create remote secrets so each cluster's Istiod can watch the other's API)
istioctl create-remote-secret \
  --kubeconfig=us-east.kubeconfig \
  --name=us-east | \
  kubectl apply --kubeconfig=eu-west.kubeconfig -f -

istioctl create-remote-secret \
  --kubeconfig=eu-west.kubeconfig \
  --name=eu-west | \
  kubectl apply --kubeconfig=us-east.kubeconfig -f -

With multi-primary Istio, a Service in us-east can route requests to pods in eu-west using locality-aware load balancing:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


# DestinationRule: prefer local pods, fall back to remote
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: orders-service
  namespace: production
spec:
  host: orders-service.production.svc.cluster.local
  trafficPolicy:
    loadBalancer:
      localityLbSetting:
        enabled: true
        distribute:
          - from: us-east/us-east-1a/*
            to:
              "us-east/us-east-1a/*": 80   # Prefer same AZ
              "us-east/*/":           15   # Same region fallback
              "eu-west/*/":            5   # Remote region last resort
        failover:
          - from: us-east
            to: eu-west
    outlierDetection:
      consecutive5xxErrors: 3
      interval: 30s
      baseEjectionTime: 30s

Approach 3: KubeSlice / Liqo

Liqo takes a unique approach — it makes remote clusters look like virtual nodes in the local cluster. Pods scheduled on a “virtual node” actually run on the remote cluster.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


# Install liqoctl
curl --fail -LS "https://github.com/liqotech/liqo/releases/latest/download/liqoctl-linux-amd64.tar.gz" | \
  tar -xz --no-same-owner -C /tmp liqoctl
sudo install -o root -g root -m 0755 /tmp/liqoctl /usr/local/bin/liqoctl

# Install Liqo on both clusters
liqoctl install kubeadm --kubeconfig us-east.kubeconfig
liqoctl install kubeadm --kubeconfig eu-west.kubeconfig

# Peer the clusters
liqoctl peer out-of-band eu-west \
  --kubeconfig us-east.kubeconfig \
  --remote-kubeconfig eu-west.kubeconfig

# From us-east's perspective, eu-west appears as a virtual node
kubectl --kubeconfig us-east.kubeconfig get nodes
# NAME              STATUS   ROLES
# us-east-node-1    Ready    worker
# us-east-node-2    Ready    worker
# liqo-eu-west      Ready    agent    ← virtual node representing eu-west

# Schedule pods onto the remote cluster using nodeSelector
spec:
  nodeSelector:
    liqo.io/remote-cluster-id: eu-west

Traffic Routing Across Clusters

Once service discovery works, you need to route traffic intelligently. Common patterns:

Global Load Balancing (DNS-based)

Use a global DNS layer to steer users to the nearest cluster:

User in Europe → DNS query for api.company.com
  → Route53 / Cloudflare / NS1 geolocation routing
  → Returns IP of eu-west Load Balancer
  → Hits eu-west cluster

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


# AWS Route53 latency-based routing
aws route53 change-resource-record-sets --hosted-zone-id Z123 \
  --change-batch '{
    "Changes": [{
      "Action": "CREATE",
      "ResourceRecordSet": {
        "Name": "api.company.com",
        "Type": "A",
        "Region": "us-east-1",
        "SetIdentifier": "us-east",
        "AliasTarget": {
          "HostedZoneId": "Z123",
          "DNSName": "us-east-alb.us-east-1.elb.amazonaws.com",
          "EvaluateTargetHealth": true
        }
      }
    }]
  }'

Active-Active with Global Anycast

Cloudflare, Fastly, and similar services offer global anycast where a single IP routes to the nearest POP, which then proxies to the nearest healthy cluster backend. Zero DNS TTL concerns, instant failover.

Active-Passive Failover

For disaster recovery: primary cluster handles all traffic, secondary is warm standby. DNS health checks trigger failover.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


# ExternalDNS with health checks (Kubernetes-native)
apiVersion: externaldns.k8s.io/v1alpha1
kind: DNSEndpoint
metadata:
  name: api-endpoint
spec:
  endpoints:
    - dnsName: api.company.com
      recordTTL: 30
      recordType: A
      targets:
        - 203.0.113.1    # us-east LB
      providerSpecific:
        - name: aws/health-check-id
          value: "hc-primary"
        - name: aws/failover
          value: PRIMARY
    - dnsName: api.company.com
      recordTTL: 30
      recordType: A
      targets:
        - 203.0.113.2    # eu-west LB
      providerSpecific:
        - name: aws/failover
          value: SECONDARY

Fleet Management with GitOps

Managing dozens of clusters requires GitOps. ArgoCD ApplicationSets are the standard pattern:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


# Deploy the same application to every cluster managed by ArgoCD
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: web-api-fleet
  namespace: argocd
spec:
  generators:
    # Generate one Application per cluster registered in ArgoCD
    - clusters:
        selector:
          matchLabels:
            env: production    # Only production clusters
  template:
    metadata:
      name: "web-api-{{name}}"    # {{name}} is the cluster name
    spec:
      project: production
      source:
        repoURL: https://github.com/company/k8s-manifests
        targetRevision: main
        path: "apps/web-api/overlays/{{metadata.labels.region}}"
      destination:
        server: "{{server}}"     # cluster's API server URL
        namespace: production
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

Cluster-Level Bootstrapping with ArgoCD

When Cluster API provisions a new cluster, trigger ArgoCD to bootstrap it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


# ArgoCD App-of-Apps for cluster bootstrapping
# Applied automatically when a new cluster is registered
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cluster-bootstrap
  namespace: argocd
spec:
  source:
    repoURL: https://github.com/company/cluster-config
    targetRevision: main
    path: bootstrap/
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

bootstrap/
├── cilium.yaml              # CNI
├── cert-manager.yaml        # TLS
├── external-dns.yaml        # DNS management
├── argocd.yaml              # GitOps agent
├── metrics-server.yaml      # Resource metrics
└── cluster-autoscaler.yaml  # Auto-scaling

Observability Across Clusters

Distributed tracing and metrics need to span clusters.

Federated Prometheus with Thanos

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


# Thanos Sidecar on each cluster's Prometheus
# Uploads metrics to shared object storage (S3)
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
  namespace: monitoring
spec:
  replicas: 2
  thanos:
    image: quay.io/thanos/thanos:v0.35.0
    objectStorageConfig:
      secret:
        name: thanos-s3-config
  externalLabels:
    cluster: us-east     # Tag all metrics with cluster name
    region: us-east-1

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


# Thanos Query — aggregates metrics from all clusters
apiVersion: apps/v1
kind: Deployment
metadata:
  name: thanos-query
  namespace: monitoring-global
spec:
  template:
    spec:
      containers:
        - name: thanos-query
          image: quay.io/thanos/thanos:v0.35.0
          args:
            - query
            - --store=thanos-store-us-east:10901
            - --store=thanos-store-eu-west:10901
            - --store=thanos-store-ap-southeast:10901
            - --query.replica-label=prometheus_replica

Unified Logging with Loki

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


# Loki with multi-tenancy — each cluster is a tenant
# Promtail on each cluster sends with X-Scope-OrgID header
apiVersion: v1
kind: ConfigMap
metadata:
  name: promtail-config
data:
  promtail.yaml: |
    clients:
      - url: https://loki.monitoring-global.svc/loki/api/v1/push
        tenant_id: us-east    # Cluster identifier
    scrape_configs:
      - job_name: kubernetes-pods
        kubernetes_sd_configs:
          - role: pod
        pipeline_stages:
          - docker: {}
        relabel_configs:
          - source_labels: [__meta_kubernetes_namespace]
            target_label: namespace
          - source_labels: [__meta_kubernetes_pod_name]
            target_label: pod

Common Pitfalls

Certificate management: each cluster needs its own PKI, and cross-cluster mTLS (e.g., Istio multi-cluster) requires a shared root CA. Use cert-manager with a shared Vault PKI backend, or manage with the same CA for all clusters from day one.

Clock skew: JWT tokens and distributed tracing assume synchronized clocks. Ensure NTP is configured consistently across all clusters. Clock skew > 5 minutes breaks Kubernetes API authentication.

Resource version conflicts: Karmada propagates objects to member clusters. If someone edits an object directly on a member cluster, Karmada will reconcile it back. Enforce via OPA/Kyverno: no direct edits to propagated resources on member clusters.

Namespace collision: if you deploy to multiple clusters without Karmada-style federation, ensure namespace names and naming conventions are consistent. An application that assumes production namespace exists on all clusters will fail if one cluster uses prod.

Upgrade coordination: multi-cluster often means multi-version. Plan upgrades cluster by cluster, and test that your applications tolerate running on different Kubernetes versions simultaneously.

Quick Reference

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


# Cluster API
clusterctl init --infrastructure aws         # Bootstrap management cluster
clusterctl get kubeconfig <name>             # Get workload cluster kubeconfig
kubectl get clusters -A                      # List all managed clusters
kubectl get machinedeployments -A            # List machine deployments
clusterctl describe cluster <name>           # Detailed cluster status
clusterctl move --to-kubeconfig dest.config  # Move cluster ownership

# Karmada
karmadactl init                              # Bootstrap Karmada control plane
karmadactl join <name> --cluster-kubeconfig  # Join a member cluster
kubectl get clusters                          # List member clusters
kubectl get propagationpolicies -A            # List propagation policies
kubectl get resourcebindings -A               # See where resources are bound

# Submariner
subctl deploy-broker                         # Deploy broker
subctl join broker-info.subm                 # Join cluster to mesh
subctl verify                                # Test connectivity
subctl show connections                      # Show active tunnels
subctl diagnose all                          # Diagnose issues

# Multi-cluster debugging
kubectl config get-contexts                   # List all cluster contexts
kubectl config use-context us-east            # Switch context
kubectl --context eu-west get pods -A         # Query specific cluster
kubectx                                       # kubectx tool for fast switching

Multi-cluster Kubernetes is not a complexity you should add speculatively. But when you genuinely need it — for compliance, resilience, or geographic distribution — these tools give you a path that’s far more manageable than stitching it together manually. Cluster API removes the snowflake problem from cluster provisioning. Karmada removes the copy-paste problem from workload deployment. Submariner and Istio remove the hard boundary problem from service communication. Used together, they turn “we have 20 clusters” from a nightmare into a manageable fleet.

When Multi-Cluster Makes Sense

Cluster API: Declarative Cluster Lifecycle Management

Architecture

Install CAPI with clusterctl

Provisioning a Cluster

ClusterClass: Reusable Cluster Templates

Workload Distribution with Karmada

Architecture

Install Karmada

PropagationPolicy: Controlling Distribution

OverridePolicy: Per-Cluster Customization

Viewing Propagation Status

Cross-Cluster Service Discovery

Approach 1: Submariner

Approach 2: Istio Multi-Primary

Approach 3: KubeSlice / Liqo

Traffic Routing Across Clusters

Global Load Balancing (DNS-based)

Active-Active with Global Anycast

Active-Passive Failover

Fleet Management with GitOps

Cluster-Level Bootstrapping with ArgoCD

Observability Across Clusters

Federated Prometheus with Thanos

Unified Logging with Loki

Common Pitfalls

Quick Reference

Comments