LUNAROPS · OPERATIONAL UPLINK 100% UPTIME 1,247d POSTS 893 JEFF.MOON@LUNAROPS.DEV UTC --:--:--

ArgoCD ApplicationSets Patterns

argocdgitopskubernetesapplicationsetscicdhelmkustomize

When you first adopt ArgoCD, you write Application manifests by hand. One YAML file per (app, cluster) pair, maybe a bit of Kustomize or Helm to vary the values, but each Application is a deliberate, named object. This works beautifully until you look up and realize you have 40 services × 4 environments × 3 clusters = 480 Application manifests, all nearly identical, all drifting out of sync because nobody remembered to copy the change into dev-staging-use1 when they did it for prod-eu2.

ApplicationSets are the answer. An ApplicationSet is a template plus a generator; the generator produces a list of parameter sets, and for each parameter set the template is materialized into a concrete Application. One CRD, N Applications. Change the template once, all the Applications regenerate. Add a cluster to the generator, new Applications appear. Remove a cluster, the Applications are deleted (and with them, the workloads).

This post walks the generators that matter, the patterns that actually scale (pull-request preview environments, matrix-of-clusters, progressive rollout), the pitfalls (where ApplicationSets bite you), and how to wire it all up in a way that doesn’t become its own unreadable YAML engine.

The model

An ApplicationSet has three parts:

  1. Generators — produce a list of parameter objects. A generator might be “list these clusters,” “list each directory under this git path,” “list each GitHub PR on this repo,” “list each branch,” or a matrix/merge of other generators.
  2. Template — an Application manifest with Go-template placeholders that reference parameters from the generator.
  3. Sync policy / go-template settings — how and when to reconcile.

A minimal example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: platform-services
  namespace: argocd
spec:
  generators:
    - list:
        elements:
          - cluster: prod-us-east-1
            url: https://1.2.3.4
          - cluster: prod-eu-west-1
            url: https://5.6.7.8
  template:
    metadata:
      name: '{{cluster}}-metrics'
    spec:
      project: default
      source:
        repoURL: https://github.com/org/platform
        path: metrics/base
        targetRevision: main
      destination:
        server: '{{url}}'
        namespace: monitoring
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Two clusters in the generator → two Applications, one per cluster, both pointing at metrics/base in git and deploying to the monitoring namespace.

That’s the pattern. Everything else is about what kind of generator to use and how to template against it.

Generators

list

The simplest. A static list of parameter objects. Good for small, stable sets. Terrible when the list gets long or changes often.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
generators:
  - list:
      elements:
        - name: dev
          namespace: dev
          replicas: "1"
        - name: staging
          namespace: staging
          replicas: "2"
        - name: prod
          namespace: prod
          replicas: "5"

cluster

Enumerates clusters registered in ArgoCD. Filter by labels:

1
2
3
4
5
6
generators:
  - clusters:
      selector:
        matchLabels:
          environment: production
          cloud: aws

Register a cluster with argocd cluster add, attach labels, and it shows up in ApplicationSet generators automatically. This is the cleanest way to express “deploy to all production clusters” — add a new cluster, label it environment: production, and Applications appear.

Cluster secrets are the underlying implementation — each registered cluster is a Secret with label argocd.argoproj.io/secret-type: cluster. You can label them by running kubectl label secret -n argocd ... or at registration time.

git (directories)

Walks a git repo, emits one element per matching subdirectory:

1
2
3
4
5
6
generators:
  - git:
      repoURL: https://github.com/org/apps
      revision: main
      directories:
        - path: applications/*

Adding a new directory under applications/ automatically materializes a new Application. This is the classic “app-of-apps-without-the-manual-app-of-apps” pattern.

git (files)

Same idea but emits one element per JSON/YAML file, with its content parsed and available as parameters:

1
2
3
4
5
6
generators:
  - git:
      repoURL: https://github.com/org/apps
      revision: main
      files:
        - path: clusters/*/config.yaml

If clusters/prod-us-east/config.yaml contains:

1
2
3
name: prod-us-east
region: us-east-1
replicas: 5

Then all three fields are available as {{name}}, {{region}}, {{replicas}} in the template.

pullRequest

Emits an element per open PR on a repo. This is the foundation of preview environments:

1
2
3
4
5
6
7
8
9
generators:
  - pullRequest:
      github:
        owner: org
        repo: web
        tokenRef:
          secretName: github-token
          key: token
      requeueAfterSeconds: 120

Each generated element includes {{number}}, {{branch}}, {{head_sha}}, labels, author. Open a PR → preview environment deploys. Close or merge → environment destroys. Dev experience: magical. Operational experience: requires thinking about blast radius (see below).

scmProvider

Enumerates all repos in an organization (or matching a pattern) and emits one element per repo. Useful for cluster-wide policies or observability apps that should land in every repo’s deployment.

matrix

Combines two generators into the cross-product:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
generators:
  - matrix:
      generators:
        - clusters:
            selector:
              matchLabels:
                env: production
        - git:
            repoURL: https://github.com/org/apps
            revision: main
            directories:
              - path: applications/*

For each (cluster, app) pair, one Application. If you have 3 production clusters and 20 apps, that’s 60 Applications. Deploy all apps to all production clusters; remove a cluster, 20 Applications disappear; remove an app, 3 Applications disappear.

Matrix is where ApplicationSets go from “nice” to “genuinely load-bearing.” Most medium-to-large deployments are matrix-based.

merge

Combines two generators, joining on a key, so you can add extra data from one generator to another:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
generators:
  - merge:
      mergeKeys:
        - server
      generators:
        - clusters: { }
        - list:
            elements:
              - server: https://1.2.3.4
                replicas: "5"
              - server: https://5.6.7.8
                replicas: "2"

The clusters generator produces cluster entries; the list generator produces replicas keyed by server. Merge joins them on server, so each element has both cluster fields and the corresponding replicas.

Pattern 1: one repo, many clusters

The simplest real-world pattern:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: monitoring-stack
  namespace: argocd
spec:
  goTemplate: true
  generators:
    - clusters:
        selector:
          matchLabels:
            tier: production
  template:
    metadata:
      name: '{{.name}}-monitoring'
    spec:
      project: platform
      source:
        repoURL: https://github.com/org/platform
        path: monitoring
        targetRevision: main
        helm:
          valueFiles:
            - 'values-{{.metadata.labels.region}}.yaml'
          values: |
            global:
              clusterName: {{.name}}
      destination:
        server: '{{.server}}'
        namespace: monitoring
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

goTemplate: true enables Go template syntax (.name, .metadata.labels.region) instead of the legacy {{name}} style. The Go template style is more robust for nested structures.

For each production cluster, the monitoring Helm chart deploys using a values file named for the cluster’s region label. Add a cluster → kube-prometheus-stack deploys there automatically.

Pattern 2: matrix of environments and apps

A platform with many apps, many environments:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: apps-matrix
  namespace: argocd
spec:
  goTemplate: true
  generators:
    - matrix:
        generators:
          - git:
              repoURL: https://github.com/org/apps
              revision: main
              files:
                - path: apps/*/config.yaml
          - list:
              elements:
                - env: dev
                  cluster: dev-cluster
                  valuesPath: dev
                - env: staging
                  cluster: staging-cluster
                  valuesPath: staging
                - env: prod
                  cluster: prod-cluster
                  valuesPath: prod
  template:
    metadata:
      name: '{{.appName}}-{{.env}}'
    spec:
      project: '{{.project}}'
      source:
        repoURL: '{{.repo}}'
        path: '{{.path}}'
        targetRevision: '{{.targetRev}}'
        helm:
          valueFiles:
            - values.yaml
            - 'values-{{.valuesPath}}.yaml'
      destination:
        name: '{{.cluster}}'
        namespace: '{{.namespace}}'
      syncPolicy:
        automated:
          prune: '{{ne .env "prod"}}'   # don't auto-prune prod
          selfHeal: true

Each app has a config.yaml in its directory describing appName, repo, path, targetRev, etc. The matrix combines with the environment list to produce (app, env) pairs. A new app → add apps/<name>/config.yaml, get 3 Applications. A new environment → add to the list, all apps deploy there.

Notice the cute bit: prune: '{{ne .env "prod"}}' disables auto-prune in prod. You can conditionalize sync behavior per environment.

Pattern 3: pull-request preview environments

The feature that sells ArgoCD to developers. For every open PR, spin up a dedicated preview environment; tear it down when the PR is merged or closed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: web-pr-previews
  namespace: argocd
spec:
  goTemplate: true
  generators:
    - pullRequest:
        github:
          owner: org
          repo: web
          labels:
            - preview
          tokenRef:
            secretName: github-token
            key: token
        requeueAfterSeconds: 60
  template:
    metadata:
      name: 'web-pr-{{.number}}'
      labels:
        type: preview
    spec:
      project: previews
      source:
        repoURL: https://github.com/org/web
        path: deploy/overlays/preview
        targetRevision: '{{.head_sha}}'
        kustomize:
          images:
            - 'web=ghcr.io/org/web:{{.head_sha}}'
          commonAnnotations:
            preview.org/pr: '{{.number}}'
            preview.org/author: '{{.author}}'
      destination:
        server: https://preview-cluster
        namespace: 'web-pr-{{.number}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

Workflow:

  1. Developer opens PR on the web repo, applies the preview label.
  2. ArgoCD’s ApplicationSet controller polls the GitHub API every 60 seconds.
  3. Sees the PR, generates an Application named web-pr-123.
  4. Application syncs Kustomize overlay to a namespace web-pr-123 on the preview cluster.
  5. The Kustomize images: override points at a container image tagged with the PR’s commit SHA — your CI has to publish that image on PR push.
  6. PR is merged or closed: the next poll cycle sees it’s gone, the Application is deleted, and with it the namespace (if you enabled prune).

The subtle parts:

  • Image tag coordination. The preview environment expects ghcr.io/org/web:<sha> — your CI must publish that tag. A simple GitHub Action on pull_request events does this.
  • Ingress / DNS. Each preview needs a URL. A wildcard DNS record (*.previews.internal) plus a cert-manager wildcard cert plus a Kustomize overlay that templates the ingress hostname per-PR (web-pr-123.previews.internal). The commonAnnotations can seed this.
  • Credentials. Preview environments often need DB access, API keys, etc. Use external-secrets pulling from Vault scoped to preview environments, or a shared “preview” secret set that’s safe enough.
  • Cleanup. When prune happens, namespace deletion takes PVCs with it if you don’t carefully manage reclaim policy. Usually what you want, occasionally not.
  • Cost. If every PR spins up a namespace with 3 pods for 48 hours, the numbers add up. Add a ttl to the ApplicationSet template (a plain label) and a CronJob that deletes ApplicationSets older than a week if still no activity.

Pattern 4: progressive rollout via labeling

You want the same app to roll out to clusters in waves: canary first, then early-prod, then all-prod. One ApplicationSet using cluster labels:

1
2
3
4
5
6
7
generators:
  - clusters:
      selector:
        matchExpressions:
          - key: rollout-wave
            operator: In
            values: ["canary"]

Deploy only to canary clusters. Verify. Update the ApplicationSet to include early-prod. Verify. Then all-prod.

The ApplicationSet controller doesn’t do progressive rollout itself — you advance the generator selectors manually (or via a meta-CI pipeline) to widen the scope. For fully-automated progressive rollouts, combine ApplicationSets with Argo Rollouts (which does canary weight management inside a single cluster) and ApplicationSet’s strategy field (which does progressive rollout across Applications).

strategy.rollingSync in ApplicationSet is newer:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
strategy:
  type: RollingSync
  rollingSync:
    steps:
      - matchExpressions:
          - key: wave
            operator: In
            values: ["0"]
      - matchExpressions:
          - key: wave
            operator: In
            values: ["1"]
        maxUpdate: 50%
      - matchExpressions:
          - key: wave
            operator: In
            values: ["2"]

Applications labeled wave=0 sync first, then wave=1 (up to 50% at a time), then wave=2. The ApplicationSet waits for each step’s Applications to report Healthy before advancing.

Patterns that bite

ApplicationSet as a footgun multiplier

An ApplicationSet can generate 500 Applications. A bad template renders 500 broken Applications simultaneously. A bad destination.namespace value can point 500 Applications at a namespace they shouldn’t touch.

Defend with:

  • spec.syncPolicy dry-run first. --dry-run on the ApplicationSet before applying.
  • preserveResourcesOnDeletion: true on the ApplicationSet — deleting the ApplicationSet does not delete the Applications it generated, letting you recover.
  • goTemplateOptions: ["missingkey=error"] — fail the template rather than render <no value> into critical fields.
  • Diffing before apply. Render the ApplicationSet locally with argocd appset commands and inspect the output before committing.

The polling vs webhook gap

ApplicationSet generators poll. git generator defaults to a 3-minute poll interval. pullRequest can go down to 30 seconds. scmProvider similar. For a preview-environment workflow, 60-second lag feels instant; for a production deploy, 3 minutes is an eternity.

Webhooks narrow the gap: configure your git provider to POST to ArgoCD’s webhook endpoint on push, and the git generator responds immediately. But webhook delivery isn’t perfect — add polling as a fallback.

Go template quirks

The Go template engine in ApplicationSets is strict. Attempting to reference an undefined field silently renders as empty string (unless missingkey=error). Some functions that work in Helm templates (like default) need explicit imports or may not be available. Stick to the built-in Go template vocabulary plus what ApplicationSet’s extensions provide (sprig-style functions when goTemplateOptions includes them).

Cluster label drift

If you rely on cluster generators with label selectors, the label state of your cluster secrets is load-bearing config. Label everywhere consistently. A typo like envirnoment: production silently excludes the cluster from selectors looking for environment.

Orphaned resources when templates change

Rename {{cluster}}-monitoring to {{cluster}}-monitor-stack in the template, and every existing Application of the old name gets deleted and recreated with the new name. If you haven’t set preserveResourcesOnDeletion, that can mean a brief window where the underlying workloads are garbage-collected. Always ship template name changes with preserveResourcesOnDeletion: true as a safety measure.

Org hygiene

For a platform serving many teams, some rules I’ve converged on:

  1. One git repo per team for their Applications and their ApplicationSets. The platform team owns infrastructure ApplicationSets (monitoring, ingress, secrets operator); app teams own their own.
  2. AppProject for each team — RBAC-scoped, namespace-whitelisted, destination-cluster-whitelisted. ApplicationSets reference their AppProject; Apps generated by the ApplicationSet inherit its scope.
  3. matchExpressions over matchLabels — more expressive, easier to narrow over time.
  4. ApplicationSets are themselves deployed via GitOps — the bootstrap ArgoCD app manages them. Meta-turtle all the way down.
  5. Per-environment values files live next to the Kustomize overlay, not in a separate config repo. Reduces repo-to-repo coordination.
  6. Name prefixes / suffixes in templates{{.appName}}-{{.env}} style. Consistent naming helps discoverability and RBAC scoping.
  7. Labels on generated Applicationsenvironment, team, tier — make it easy to argocd app list -l team=payments etc.

Observability

ApplicationSets emit events and Prometheus metrics. Alerts worth writing:

  • argocd_appset_info{health_status="Degraded"} — the set itself is unhealthy.
  • argocd_app_info{sync_status!="Synced"} grouped by ApplicationSet label — some generated app is out of sync.
  • argocd_app_sync_total{phase="Failed"} — recent sync failures.
  • kube_customresource_argocd_appset via kube-state-metrics if you want set-level metadata.

A Grafana dashboard with a row per ApplicationSet showing “expected count vs actual count vs healthy count” catches template errors fast.

Where ApplicationSets fall short

Not everything fits:

  1. Complex per-cluster values. ApplicationSets templates are flat-ish. For clusters with substantially different configurations beyond a few label-derived values, you end up with a tangle. At that point, prefer separate Applications per cluster or switch to a templating layer (Helm, Jsonnet) that’s more expressive than Go templates.

  2. Lifecycle hooks per Application. The ApplicationSet controller doesn’t do custom hooks — “only sync this app if that app is healthy” is expressible via strategy.rollingSync, but “run a migration job before deploying” is best handled at the Application level (sync waves) or via Argo Workflows.

  3. Cross-cluster dependencies. “Deploy DB in cluster A before API in cluster B” — ApplicationSet doesn’t help. You need orchestration above it.

  4. Per-cluster toggle of individual apps. If cluster B should have apps {a, b, c} but cluster C should have {a, c, d}, a matrix generator gets awkward. Consider one ApplicationSet per app with a clusters generator and a label selector; gives per-app fine-grained control.

Closing

ApplicationSets are the feature that makes ArgoCD viable past a few clusters. Used well, they turn what would be thousands of hand-maintained Application YAMLs into a few dozen generator-plus-template specs that stay in sync automatically. The pieces that take a while to internalize — generators, templates, matrix composition, preview-environment plumbing — are worth the learning because they pay back every time you add a cluster or a service.

The anti-pattern to resist is shoving too much logic into the template. ApplicationSet templates are Go text templates evaluating against a few flat parameter objects; they’re not Helm charts. When you find yourself writing conditionals and loops inside the template, push that logic back into a real templating engine (Helm, Kustomize) and have ApplicationSet just parameterize it. Used like that, the pattern stays clean and the whole platform becomes a few dozen YAML files that generate hundreds of Applications predictably.

Comments