ArgoCD ApplicationSets Patterns
When you first adopt ArgoCD, you write Application manifests by hand. One YAML file per (app, cluster) pair, maybe a bit of Kustomize or Helm to vary the values, but each Application is a deliberate, named object. This works beautifully until you look up and realize you have 40 services × 4 environments × 3 clusters = 480 Application manifests, all nearly identical, all drifting out of sync because nobody remembered to copy the change into dev-staging-use1 when they did it for prod-eu2.
ApplicationSets are the answer. An ApplicationSet is a template plus a generator; the generator produces a list of parameter sets, and for each parameter set the template is materialized into a concrete Application. One CRD, N Applications. Change the template once, all the Applications regenerate. Add a cluster to the generator, new Applications appear. Remove a cluster, the Applications are deleted (and with them, the workloads).
This post walks the generators that matter, the patterns that actually scale (pull-request preview environments, matrix-of-clusters, progressive rollout), the pitfalls (where ApplicationSets bite you), and how to wire it all up in a way that doesn’t become its own unreadable YAML engine.
The model
An ApplicationSet has three parts:
- Generators — produce a list of parameter objects. A generator might be “list these clusters,” “list each directory under this git path,” “list each GitHub PR on this repo,” “list each branch,” or a matrix/merge of other generators.
- Template — an Application manifest with Go-template placeholders that reference parameters from the generator.
- Sync policy / go-template settings — how and when to reconcile.
A minimal example:
|
|
Two clusters in the generator → two Applications, one per cluster, both pointing at metrics/base in git and deploying to the monitoring namespace.
That’s the pattern. Everything else is about what kind of generator to use and how to template against it.
Generators
list
The simplest. A static list of parameter objects. Good for small, stable sets. Terrible when the list gets long or changes often.
|
|
cluster
Enumerates clusters registered in ArgoCD. Filter by labels:
|
|
Register a cluster with argocd cluster add, attach labels, and it shows up in ApplicationSet generators automatically. This is the cleanest way to express “deploy to all production clusters” — add a new cluster, label it environment: production, and Applications appear.
Cluster secrets are the underlying implementation — each registered cluster is a Secret with label argocd.argoproj.io/secret-type: cluster. You can label them by running kubectl label secret -n argocd ... or at registration time.
git (directories)
Walks a git repo, emits one element per matching subdirectory:
|
|
Adding a new directory under applications/ automatically materializes a new Application. This is the classic “app-of-apps-without-the-manual-app-of-apps” pattern.
git (files)
Same idea but emits one element per JSON/YAML file, with its content parsed and available as parameters:
|
|
If clusters/prod-us-east/config.yaml contains:
|
|
Then all three fields are available as {{name}}, {{region}}, {{replicas}} in the template.
pullRequest
Emits an element per open PR on a repo. This is the foundation of preview environments:
|
|
Each generated element includes {{number}}, {{branch}}, {{head_sha}}, labels, author. Open a PR → preview environment deploys. Close or merge → environment destroys. Dev experience: magical. Operational experience: requires thinking about blast radius (see below).
scmProvider
Enumerates all repos in an organization (or matching a pattern) and emits one element per repo. Useful for cluster-wide policies or observability apps that should land in every repo’s deployment.
matrix
Combines two generators into the cross-product:
|
|
For each (cluster, app) pair, one Application. If you have 3 production clusters and 20 apps, that’s 60 Applications. Deploy all apps to all production clusters; remove a cluster, 20 Applications disappear; remove an app, 3 Applications disappear.
Matrix is where ApplicationSets go from “nice” to “genuinely load-bearing.” Most medium-to-large deployments are matrix-based.
merge
Combines two generators, joining on a key, so you can add extra data from one generator to another:
|
|
The clusters generator produces cluster entries; the list generator produces replicas keyed by server. Merge joins them on server, so each element has both cluster fields and the corresponding replicas.
Pattern 1: one repo, many clusters
The simplest real-world pattern:
|
|
goTemplate: true enables Go template syntax (.name, .metadata.labels.region) instead of the legacy {{name}} style. The Go template style is more robust for nested structures.
For each production cluster, the monitoring Helm chart deploys using a values file named for the cluster’s region label. Add a cluster → kube-prometheus-stack deploys there automatically.
Pattern 2: matrix of environments and apps
A platform with many apps, many environments:
|
|
Each app has a config.yaml in its directory describing appName, repo, path, targetRev, etc. The matrix combines with the environment list to produce (app, env) pairs. A new app → add apps/<name>/config.yaml, get 3 Applications. A new environment → add to the list, all apps deploy there.
Notice the cute bit: prune: '{{ne .env "prod"}}' disables auto-prune in prod. You can conditionalize sync behavior per environment.
Pattern 3: pull-request preview environments
The feature that sells ArgoCD to developers. For every open PR, spin up a dedicated preview environment; tear it down when the PR is merged or closed.
|
|
Workflow:
- Developer opens PR on the
webrepo, applies thepreviewlabel. - ArgoCD’s ApplicationSet controller polls the GitHub API every 60 seconds.
- Sees the PR, generates an Application named
web-pr-123. - Application syncs Kustomize overlay to a namespace
web-pr-123on the preview cluster. - The Kustomize
images:override points at a container image tagged with the PR’s commit SHA — your CI has to publish that image on PR push. - PR is merged or closed: the next poll cycle sees it’s gone, the Application is deleted, and with it the namespace (if you enabled prune).
The subtle parts:
- Image tag coordination. The preview environment expects
ghcr.io/org/web:<sha>— your CI must publish that tag. A simple GitHub Action onpull_requestevents does this. - Ingress / DNS. Each preview needs a URL. A wildcard DNS record (
*.previews.internal) plus a cert-manager wildcard cert plus a Kustomize overlay that templates the ingress hostname per-PR (web-pr-123.previews.internal). ThecommonAnnotationscan seed this. - Credentials. Preview environments often need DB access, API keys, etc. Use
external-secretspulling from Vault scoped to preview environments, or a shared “preview” secret set that’s safe enough. - Cleanup. When prune happens, namespace deletion takes PVCs with it if you don’t carefully manage reclaim policy. Usually what you want, occasionally not.
- Cost. If every PR spins up a namespace with 3 pods for 48 hours, the numbers add up. Add a
ttlto the ApplicationSet template (a plain label) and a CronJob that deletes ApplicationSets older than a week if still no activity.
Pattern 4: progressive rollout via labeling
You want the same app to roll out to clusters in waves: canary first, then early-prod, then all-prod. One ApplicationSet using cluster labels:
|
|
Deploy only to canary clusters. Verify. Update the ApplicationSet to include early-prod. Verify. Then all-prod.
The ApplicationSet controller doesn’t do progressive rollout itself — you advance the generator selectors manually (or via a meta-CI pipeline) to widen the scope. For fully-automated progressive rollouts, combine ApplicationSets with Argo Rollouts (which does canary weight management inside a single cluster) and ApplicationSet’s strategy field (which does progressive rollout across Applications).
strategy.rollingSync in ApplicationSet is newer:
|
|
Applications labeled wave=0 sync first, then wave=1 (up to 50% at a time), then wave=2. The ApplicationSet waits for each step’s Applications to report Healthy before advancing.
Patterns that bite
ApplicationSet as a footgun multiplier
An ApplicationSet can generate 500 Applications. A bad template renders 500 broken Applications simultaneously. A bad destination.namespace value can point 500 Applications at a namespace they shouldn’t touch.
Defend with:
spec.syncPolicydry-run first.--dry-runon the ApplicationSet before applying.preserveResourcesOnDeletion: trueon the ApplicationSet — deleting the ApplicationSet does not delete the Applications it generated, letting you recover.goTemplateOptions: ["missingkey=error"]— fail the template rather than render<no value>into critical fields.- Diffing before apply. Render the ApplicationSet locally with
argocd appsetcommands and inspect the output before committing.
The polling vs webhook gap
ApplicationSet generators poll. git generator defaults to a 3-minute poll interval. pullRequest can go down to 30 seconds. scmProvider similar. For a preview-environment workflow, 60-second lag feels instant; for a production deploy, 3 minutes is an eternity.
Webhooks narrow the gap: configure your git provider to POST to ArgoCD’s webhook endpoint on push, and the git generator responds immediately. But webhook delivery isn’t perfect — add polling as a fallback.
Go template quirks
The Go template engine in ApplicationSets is strict. Attempting to reference an undefined field silently renders as empty string (unless missingkey=error). Some functions that work in Helm templates (like default) need explicit imports or may not be available. Stick to the built-in Go template vocabulary plus what ApplicationSet’s extensions provide (sprig-style functions when goTemplateOptions includes them).
Cluster label drift
If you rely on cluster generators with label selectors, the label state of your cluster secrets is load-bearing config. Label everywhere consistently. A typo like envirnoment: production silently excludes the cluster from selectors looking for environment.
Orphaned resources when templates change
Rename {{cluster}}-monitoring to {{cluster}}-monitor-stack in the template, and every existing Application of the old name gets deleted and recreated with the new name. If you haven’t set preserveResourcesOnDeletion, that can mean a brief window where the underlying workloads are garbage-collected. Always ship template name changes with preserveResourcesOnDeletion: true as a safety measure.
Org hygiene
For a platform serving many teams, some rules I’ve converged on:
- One git repo per team for their Applications and their ApplicationSets. The platform team owns infrastructure ApplicationSets (monitoring, ingress, secrets operator); app teams own their own.
- AppProject for each team — RBAC-scoped, namespace-whitelisted, destination-cluster-whitelisted. ApplicationSets reference their AppProject; Apps generated by the ApplicationSet inherit its scope.
matchExpressionsovermatchLabels— more expressive, easier to narrow over time.- ApplicationSets are themselves deployed via GitOps — the bootstrap ArgoCD app manages them. Meta-turtle all the way down.
- Per-environment values files live next to the Kustomize overlay, not in a separate config repo. Reduces repo-to-repo coordination.
- Name prefixes / suffixes in templates —
{{.appName}}-{{.env}}style. Consistent naming helps discoverability and RBAC scoping. - Labels on generated Applications —
environment,team,tier— make it easy toargocd app list -l team=paymentsetc.
Observability
ApplicationSets emit events and Prometheus metrics. Alerts worth writing:
argocd_appset_info{health_status="Degraded"}— the set itself is unhealthy.argocd_app_info{sync_status!="Synced"}grouped by ApplicationSet label — some generated app is out of sync.argocd_app_sync_total{phase="Failed"}— recent sync failures.kube_customresource_argocd_appsetvia kube-state-metrics if you want set-level metadata.
A Grafana dashboard with a row per ApplicationSet showing “expected count vs actual count vs healthy count” catches template errors fast.
Where ApplicationSets fall short
Not everything fits:
-
Complex per-cluster values. ApplicationSets templates are flat-ish. For clusters with substantially different configurations beyond a few label-derived values, you end up with a tangle. At that point, prefer separate Applications per cluster or switch to a templating layer (Helm, Jsonnet) that’s more expressive than Go templates.
-
Lifecycle hooks per Application. The ApplicationSet controller doesn’t do custom hooks — “only sync this app if that app is healthy” is expressible via
strategy.rollingSync, but “run a migration job before deploying” is best handled at the Application level (sync waves) or via Argo Workflows. -
Cross-cluster dependencies. “Deploy DB in cluster A before API in cluster B” — ApplicationSet doesn’t help. You need orchestration above it.
-
Per-cluster toggle of individual apps. If cluster B should have apps {a, b, c} but cluster C should have {a, c, d}, a matrix generator gets awkward. Consider one ApplicationSet per app with a
clustersgenerator and a label selector; gives per-app fine-grained control.
Closing
ApplicationSets are the feature that makes ArgoCD viable past a few clusters. Used well, they turn what would be thousands of hand-maintained Application YAMLs into a few dozen generator-plus-template specs that stay in sync automatically. The pieces that take a while to internalize — generators, templates, matrix composition, preview-environment plumbing — are worth the learning because they pay back every time you add a cluster or a service.
The anti-pattern to resist is shoving too much logic into the template. ApplicationSet templates are Go text templates evaluating against a few flat parameter objects; they’re not Helm charts. When you find yourself writing conditionals and loops inside the template, push that logic back into a real templating engine (Helm, Kustomize) and have ApplicationSet just parameterize it. Used like that, the pattern stays clean and the whole platform becomes a few dozen YAML files that generate hundreds of Applications predictably.
Comments