GitOps is a simple idea with deep implications: the desired state of your infrastructure lives in Git, and an automated system continuously reconciles the actual state toward it. You never kubectl apply in production. You never SSH into a server to patch a config. Every change is a pull request, every deployment is a merge, and the entire history of what changed and why lives in your version control system.
Argo CD is the most widely adopted GitOps engine for Kubernetes. It watches Git repositories, detects drift between what’s declared and what’s running, and syncs them — automatically or on demand. This guide goes beyond the basics into the patterns that matter at production scale: the App of Apps pattern for bootstrapping entire clusters, progressive delivery with Argo Rollouts, RBAC for multi-team organizations, multi-cluster management, and building the notification pipelines that keep teams informed.
How Argo CD Works
Argo CD runs as a set of controllers inside your Kubernetes cluster. Its core loop:
- Fetch the desired state from a Git repository (at a specific commit/branch/tag)
- Render the manifests (raw YAML, Helm chart, Kustomize overlay, Jsonnet, etc.)
- Compare the rendered manifests against what’s actually in the cluster
- Report any differences (drift detection)
- Sync — apply the desired state to the cluster, either automatically or when triggered
The key insight is that Argo CD renders manifests at sync time from source, not from a pre-rendered artifact. This means helm template runs against the live chart at the specified version every time Argo CD checks for drift, so you always have an accurate picture of what’s declared.
The Application CRD
Argo CD’s primary resource is the Application:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
|
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: my-api
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io # Delete resources when App is deleted
spec:
project: production
source:
repoURL: https://github.com/myorg/gitops-config
targetRevision: main
path: apps/my-api/overlays/production
destination:
server: https://kubernetes.default.svc # In-cluster
namespace: production
syncPolicy:
automated:
prune: true # Delete resources removed from Git
selfHeal: true # Re-sync if cluster state drifts
syncOptions:
- CreateNamespace=true
- PrunePropagationPolicy=foreground
- RespectIgnoreDifferences=true
retry:
limit: 5
backoff:
duration: 5s
factor: 2
maxDuration: 3m
|
automated.prune: true: If a resource is removed from Git, Argo CD deletes it from the cluster. Without this, orphaned resources accumulate silently.
automated.selfHeal: true: If someone manually changes a resource (kubectl edit), Argo CD overwrites it back to what Git says within minutes. This enforces GitOps discipline — Git is the only source of truth.
retry: Transient failures (webhook not ready, admission controller timing) are retried with exponential backoff.
Installation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
|
# Create namespace and install Argo CD
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
# Wait for all pods to be ready
kubectl wait --for=condition=available --timeout=300s \
deployment/argocd-server -n argocd
# Get initial admin password
kubectl get secret argocd-initial-admin-secret \
-n argocd \
-o jsonpath="{.data.password}" | base64 -d
# Port-forward to access the UI
kubectl port-forward svc/argocd-server -n argocd 8080:443
# Log in via CLI
argocd login localhost:8080 \
--username admin \
--password <initial-password> \
--insecure
# Change the admin password
argocd account update-password
|
For production, expose Argo CD via Traefik or an ingress:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
|
# argocd-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: argocd-server
namespace: argocd
annotations:
traefik.ingress.kubernetes.io/router.entrypoints: websecure
traefik.ingress.kubernetes.io/router.tls.certresolver: letsencrypt
# Argo CD uses gRPC — needed for argocd CLI to work through the ingress
traefik.ingress.kubernetes.io/router.middlewares: argocd-argocd-grpc@kubernetescrd
spec:
rules:
- host: argocd.yourdomain.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: argocd-server
port:
number: 80
tls:
- hosts:
- argocd.yourdomain.com
|
Repository Structure: The GitOps Config Repo
A clean repo structure is the foundation of maintainable GitOps. Separate the application source code repos from the GitOps config repo:
gitops-config/ # Separate repo from application code
├── bootstrap/ # Cluster bootstrap (App of Apps)
│ └── root-app.yaml
├── argocd/ # Argo CD config itself
│ ├── projects.yaml
│ └── repositories.yaml
├── apps/ # Application definitions
│ ├── infrastructure/ # Cluster infrastructure
│ │ ├── cert-manager/
│ │ ├── traefik/
│ │ ├── monitoring/
│ │ └── external-secrets/
│ └── services/ # Application workloads
│ ├── api/
│ │ ├── base/ # Kustomize base
│ │ │ ├── deployment.yaml
│ │ │ ├── service.yaml
│ │ │ └── kustomization.yaml
│ │ └── overlays/
│ │ ├── staging/
│ │ │ └── kustomization.yaml
│ │ └── production/
│ │ └── kustomization.yaml
│ └── frontend/
└── clusters/ # Cluster-specific config
├── production/
└── staging/
Application code repos hold the Dockerfile, source code, and application Helm chart/base manifests.
GitOps config repo holds the desired state of every cluster — which chart versions, which image tags, which config values.
The CI pipeline for an application updates the config repo (bumps an image tag or chart version) as its final step. Argo CD detects the change and syncs the cluster. The flow:
Developer pushes code
→ CI builds image, tags it :abc1234
→ CI opens PR against gitops-config, bumping image tag
→ PR is reviewed and merged
→ Argo CD detects the change and deploys
The App of Apps Pattern
Manually creating Argo CD Application resources for every service is tedious and defeats the purpose of GitOps — Argo CD’s own configuration would be outside of Git. The App of Apps pattern solves this: a single “root” Application manages all other Applications.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
|
# bootstrap/root-app.yaml
# This is the only resource you ever apply manually — once
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: root
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/myorg/gitops-config
targetRevision: main
path: clusters/production # Points to the cluster's app-of-apps directory
destination:
server: https://kubernetes.default.svc
namespace: argocd
syncPolicy:
automated:
prune: true
selfHeal: true
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
# clusters/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
# Infrastructure apps
- apps/cert-manager.yaml
- apps/traefik.yaml
- apps/monitoring.yaml
- apps/external-secrets.yaml
# Service apps
- apps/api.yaml
- apps/frontend.yaml
- apps/worker.yaml
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
|
# clusters/production/apps/api.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: api
namespace: argocd
spec:
project: production
source:
repoURL: https://github.com/myorg/gitops-config
targetRevision: main
path: apps/services/api/overlays/production
destination:
server: https://kubernetes.default.svc
namespace: production
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
|
The bootstrap sequence:
kubectl apply -f bootstrap/root-app.yaml — the only manual step- Root app syncs, creating all child
Application resources in Argo CD
- Each child Application syncs, deploying its workload to the cluster
- Everything is now self-managing — add a new service by adding a file to
clusters/production/apps/
ApplicationSets: Generating Applications Programmatically
For larger organizations with many clusters or many services following the same pattern, ApplicationSet generates multiple Application resources from a template:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
|
# Generate one Application per directory in apps/services/
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: services
namespace: argocd
spec:
generators:
- git:
repoURL: https://github.com/myorg/gitops-config
revision: main
directories:
- path: apps/services/*
template:
metadata:
name: '{{path.basename}}'
spec:
project: production
source:
repoURL: https://github.com/myorg/gitops-config
targetRevision: main
path: '{{path}}/overlays/production'
destination:
server: https://kubernetes.default.svc
namespace: '{{path.basename}}'
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
|
Now adding a new service is as simple as creating the directory apps/services/my-new-service/ — the ApplicationSet automatically detects it and creates the corresponding Application.
Multi-cluster with ApplicationSet:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
|
# Deploy the same app to every registered cluster
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: monitoring-stack
namespace: argocd
spec:
generators:
- clusters: {} # All registered clusters
template:
metadata:
name: 'monitoring-{{name}}'
spec:
project: platform
source:
repoURL: https://github.com/myorg/gitops-config
targetRevision: main
path: apps/infrastructure/monitoring
destination:
server: '{{server}}'
namespace: monitoring
syncPolicy:
automated:
prune: true
selfHeal: true
|
Projects: Isolation and Access Control
Argo CD Projects define boundaries around groups of Applications — which repos they can deploy from, which clusters/namespaces they can deploy to, and which resource types are allowed.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
|
# argocd/projects.yaml
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: production
namespace: argocd
spec:
description: Production workloads
# Which repos this project's apps can pull from
sourceRepos:
- https://github.com/myorg/*
- https://charts.helm.sh/stable
- https://charts.bitnami.com/bitnami
# Which clusters/namespaces apps in this project can deploy to
destinations:
- server: https://kubernetes.default.svc
namespace: production
- server: https://kubernetes.default.svc
namespace: production-* # Wildcard namespaces
# Deny cluster-scoped resources (ClusterRole, etc.) — only namespaced
clusterResourceWhitelist: [] # Empty = deny all cluster-scoped
# Allow only these namespaced resource types
namespaceResourceWhitelist:
- group: apps
kind: Deployment
- group: apps
kind: StatefulSet
- group: ""
kind: Service
- group: ""
kind: ConfigMap
- group: ""
kind: Secret
- group: networking.k8s.io
kind: Ingress
- group: argoproj.io
kind: Rollout
# Sync windows: only allow auto-sync during business hours
syncWindows:
- kind: allow
schedule: "0 9 * * 1-5" # Mon-Fri 9am
duration: 8h
applications:
- "*"
- kind: deny
schedule: "0 17 * * 5" # Friday 5pm
duration: 48h # No auto-sync over the weekend
applications:
- "*"
manualSync: true # But allow manual sync
# Roles within this project
roles:
- name: developer
description: Read access + manual sync
policies:
- p, proj:production:developer, applications, get, production/*, allow
- p, proj:production:developer, applications, sync, production/*, allow
groups:
- myorg:developers
- name: readonly
policies:
- p, proj:production:readonly, applications, get, production/*, allow
groups:
- myorg:contractors
|
Sync windows are powerful for reducing blast radius: auto-sync only happens during business hours when your team is available to respond. Weekend deployments require a deliberate manual sync.
RBAC: Multi-Team Access Control
Argo CD’s RBAC uses Casbin policies that look like:
p, <subject>, <resource>, <action>, <object>, <effect>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
|
# argocd-cm ConfigMap patch — RBAC configuration
apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-rbac-cm
namespace: argocd
data:
policy.default: role:readonly # Default role for authenticated users
policy.csv: |
# Platform team: full access
p, role:platform-admin, *, *, *, allow
g, myorg:platform-team, role:platform-admin
# SRE team: manage all apps, read cluster resources
p, role:sre, applications, *, */*, allow
p, role:sre, clusters, get, *, allow
p, role:sre, repositories, get, *, allow
g, myorg:sre, role:sre
# App teams: manage their own project only
p, role:app-developer, applications, get, my-project/*, allow
p, role:app-developer, applications, sync, my-project/*, allow
p, role:app-developer, applications, action/*, my-project/*, allow
g, myorg:app-team, role:app-developer
# Read-only for everyone else (covered by policy.default)
|
SSO Integration
Connect Argo CD to your identity provider (GitHub, Google, Okta, LDAP) so team members log in with their existing credentials:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
|
# argocd-cm ConfigMap
data:
url: https://argocd.yourdomain.com
dex.config: |
connectors:
- type: github
id: github
name: GitHub
config:
clientID: $dex.github.clientID
clientSecret: $dex.github.clientSecret
orgs:
- name: myorg
teams:
- platform-team
- sre
- app-team
|
With SSO configured, myorg:platform-team in the RBAC policies maps to the GitHub team “platform-team” in the “myorg” organization.
Helm and Kustomize Integration
Helm Applications
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
|
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: cert-manager
namespace: argocd
spec:
source:
repoURL: https://charts.jetstack.io
chart: cert-manager
targetRevision: v1.14.0
helm:
releaseName: cert-manager
values: |
installCRDs: true
replicaCount: 2
resources:
requests:
cpu: 10m
memory: 32Mi
# Or reference a values file in your config repo:
valueFiles:
- values.yaml
- values-production.yaml
|
Kustomize Applications with Image Override
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: api
namespace: argocd
spec:
source:
repoURL: https://github.com/myorg/gitops-config
targetRevision: main
path: apps/services/api/overlays/production
kustomize:
images:
# Override the image tag in the overlay
- myregistry/api:latest=myregistry/api:abc1234def
|
Multi-Source Applications (Argo CD v2.6+)
Combine a Helm chart from one repo with values from another:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: grafana
namespace: argocd
spec:
sources:
# Helm chart from upstream
- repoURL: https://grafana.github.io/helm-charts
chart: grafana
targetRevision: 7.3.0
helm:
valueFiles:
- $values/apps/infrastructure/grafana/values.yaml
# Values from your config repo (referenced as $values)
- repoURL: https://github.com/myorg/gitops-config
targetRevision: main
ref: values
|
This pattern keeps chart versions separate from your values — you can upgrade a chart version without touching your values file.
Progressive Delivery with Argo Rollouts
Standard Kubernetes Deployment rollouts are all-or-nothing: you update the image, the rollout controller replaces pods, and if something goes wrong you rollback. Argo Rollouts adds canary and blue-green deployments with automated analysis.
Installation
1
2
3
4
5
6
7
8
|
kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts \
-f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml
# Install the kubectl plugin
curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-linux-amd64
chmod +x kubectl-argo-rollouts-linux-amd64
mv kubectl-argo-rollouts-linux-amd64 /usr/local/bin/kubectl-argo-rollouts
|
Canary Rollout
Replace your Deployment with a Rollout:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
|
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: api
namespace: production
spec:
replicas: 10
selector:
matchLabels:
app: api
template:
metadata:
labels:
app: api
spec:
containers:
- name: api
image: myregistry/api:v2.0.0
ports:
- containerPort: 8080
resources:
requests:
cpu: 100m
memory: 128Mi
strategy:
canary:
# Traffic management via your ingress controller
trafficRouting:
nginx:
stableIngress: api-stable
# Argo Rollouts manages the canary ingress automatically
steps:
# Step 1: send 5% of traffic to canary, wait for analysis
- setWeight: 5
- analysis:
templates:
- templateName: success-rate
args:
- name: service-name
value: api-canary
# Step 2: pause for manual observation (optional — remove for full automation)
- pause:
duration: 10m
# Step 3: ramp up to 20%
- setWeight: 20
- analysis:
templates:
- templateName: success-rate
- templateName: latency-p99
# Step 4: ramp to 50%
- setWeight: 50
- pause:
duration: 5m
# Step 5: full rollout (100% to new version)
- setWeight: 100
# If analysis fails, automatically abort and rollback
analysis:
successfulRunHistoryLimit: 3
unsuccessfulRunHistoryLimit: 3
|
Analysis Templates
Analysis templates define what “success” means for a canary:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
|
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
name: success-rate
namespace: production
spec:
args:
- name: service-name
metrics:
- name: success-rate
interval: 1m
successCondition: result[0] >= 0.95 # 95% success rate
failureLimit: 3 # Fail after 3 consecutive bad checks
provider:
prometheus:
address: http://prometheus.monitoring.svc.cluster.local:9090
query: |
sum(
rate(http_requests_total{
service="{{args.service-name}}",
status_code!~"5.*"
}[5m])
) /
sum(
rate(http_requests_total{
service="{{args.service-name}}"
}[5m])
)
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
name: latency-p99
namespace: production
spec:
args:
- name: service-name
metrics:
- name: p99-latency
interval: 1m
successCondition: result[0] <= 0.5 # P99 under 500ms
failureLimit: 2
provider:
prometheus:
address: http://prometheus.monitoring.svc.cluster.local:9090
query: |
histogram_quantile(0.99,
sum(rate(http_request_duration_seconds_bucket{
service="{{args.service-name}}"
}[5m])) by (le)
)
|
When an analysis step runs, Argo Rollouts queries Prometheus every minute. If success-rate drops below 95% three times in a row, the rollout aborts and traffic is automatically shifted back to the stable version — no human intervention needed.
Blue-Green Rollout
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
|
strategy:
blueGreen:
activeService: api-active # Service pointing to current stable version
previewService: api-preview # Service pointing to new version
autoPromotionEnabled: false # Require manual promotion (or set to true)
autoPromotionSeconds: 300 # Auto-promote after 5 minutes if no analysis failure
prePromotionAnalysis:
templates:
- templateName: smoke-test
args:
- name: service-name
value: api-preview
postPromotionAnalysis:
templates:
- templateName: success-rate
args:
- name: service-name
value: api-active
scaleDownDelaySeconds: 300 # Keep old (blue) version for 5 minutes after promotion
|
Blue-green keeps the old version running until you’re confident in the new one. Traffic switches instantly (service selector update) rather than gradually. The old version is available for instant rollback within the scaleDownDelaySeconds window.
Observing Rollouts
1
2
3
4
5
6
7
8
9
10
11
|
# Watch a rollout in progress
kubectl argo rollouts get rollout api -n production --watch
# Manually promote a paused rollout
kubectl argo rollouts promote api -n production
# Abort a rollout (rollback to stable)
kubectl argo rollouts abort api -n production
# Manually retry a failed rollout
kubectl argo rollouts retry rollout api -n production
|
Multi-Cluster Management
Argo CD’s hub-and-spoke model: one Argo CD instance (the “hub”) manages multiple clusters (“spokes”).
Registering Clusters
1
2
3
4
5
6
7
8
9
10
11
|
# Add a cluster to Argo CD management
argocd cluster add production-eks \
--kubeconfig ~/.kube/production \
--name production-eks
argocd cluster add staging-gke \
--kubeconfig ~/.kube/staging \
--name staging-gke
# List registered clusters
argocd cluster list
|
Behind the scenes, this creates a Secret in the argocd namespace with the cluster credentials.
A common pattern: deploy to staging first, then promote to production after validation.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
|
# clusters/staging/apps/api.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: api-staging
namespace: argocd
spec:
source:
repoURL: https://github.com/myorg/gitops-config
targetRevision: main
path: apps/services/api/overlays/staging
destination:
name: staging-gke # Named cluster reference
namespace: staging
syncPolicy:
automated:
prune: true
selfHeal: true
---
# clusters/production/apps/api.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: api-production
namespace: argocd
spec:
source:
repoURL: https://github.com/myorg/gitops-config
targetRevision: main
path: apps/services/api/overlays/production
destination:
name: production-eks
namespace: production
# No automated sync for production — require manual approval
syncPolicy:
syncOptions:
- CreateNamespace=true
|
The promotion flow:
- CI merges a PR bumping
api to v2.0.0 in overlays/staging
- Argo CD auto-syncs staging
- After validation (automated tests, smoke tests, team sign-off), open a PR bumping
overlays/production to v2.0.0
- PR review is the production deployment approval
- Merge → Argo CD detects the diff → manual sync or auto-sync with a sync window
Drift Detection and Notifications
Drift Detection
Argo CD continuously compares desired state (Git) with actual state (cluster). The OutOfSync condition indicates drift:
1
2
3
4
5
6
7
8
9
10
11
12
|
# Check sync status across all apps
argocd app list
# Get detailed diff for a specific app
argocd app diff my-api
# Example output:
# ===== apps/Deployment production/api ======
# 7c7
# < image: myregistry/api:v1.9.0
# ---
# > image: myregistry/api:v2.0.0-hotfix
|
If someone applies a hotfix directly with kubectl while selfHeal: false, Argo CD surfaces it as drift but doesn’t overwrite it — giving you visibility without automatic rollback in cases where you’ve disabled self-healing.
Notifications
Argo CD Notifications sends messages when application state changes:
1
2
|
kubectl apply -n argocd \
-f https://raw.githubusercontent.com/argoproj/argo-cd/stable/notifications_catalog/install.yaml
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
|
# argocd-notifications-cm ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-notifications-cm
namespace: argocd
data:
# Slack integration
service.slack: |
token: $slack-token
# Triggers — when to send
trigger.on-sync-succeeded: |
- when: app.status.operationState.phase in ['Succeeded']
send: [app-sync-succeeded]
trigger.on-sync-failed: |
- when: app.status.operationState.phase in ['Error', 'Failed']
send: [app-sync-failed]
trigger.on-health-degraded: |
- when: app.status.health.status == 'Degraded'
send: [app-health-degraded]
trigger.on-deployed: |
- when: app.status.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy'
send: [app-deployed]
oncePer: app.status.sync.revision
# Templates — what to send
template.app-sync-succeeded: |
slack:
attachments: |
[{
"title": "✅ {{.app.metadata.name}} synced",
"color": "good",
"fields": [
{"title": "Revision", "value": "{{.app.status.sync.revision | truncate 7}}", "short": true},
{"title": "Environment", "value": "{{.app.spec.destination.namespace}}", "short": true}
]
}]
template.app-sync-failed: |
slack:
attachments: |
[{
"title": "❌ {{.app.metadata.name}} sync failed",
"color": "danger",
"text": "{{.app.status.operationState.message}}",
"fields": [
{"title": "Revision", "value": "{{.app.status.sync.revision | truncate 7}}", "short": true}
]
}]
template.app-deployed: |
slack:
text: "🚀 {{.app.metadata.name}} deployed {{.app.status.sync.revision | truncate 7}} to {{.app.spec.destination.namespace}}"
template.app-health-degraded: |
slack:
attachments: |
[{
"title": "⚠️ {{.app.metadata.name}} health degraded",
"color": "warning",
"text": "{{range .app.status.conditions}}{{.message}}\n{{end}}"
}]
|
1
2
3
4
5
6
|
# Annotation on Application resources to subscribe to notifications
metadata:
annotations:
notifications.argoproj.io/subscribe.on-sync-failed.slack: deployments
notifications.argoproj.io/subscribe.on-deployed.slack: deployments
notifications.argoproj.io/subscribe.on-health-degraded.slack: alerts
|
Secrets Management
Argo CD syncs whatever is in Git. Secrets in Git are a serious security problem. The standard approaches:
External Secrets Operator
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
|
# ExternalSecret pulls from AWS Secrets Manager, Vault, GCP Secret Manager, etc.
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: api-database-credentials
namespace: production
spec:
refreshInterval: 1h
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
target:
name: api-database-credentials # Creates this Kubernetes Secret
creationPolicy: Owner
data:
- secretKey: DATABASE_URL
remoteRef:
key: production/api/database
property: url
- secretKey: DATABASE_PASSWORD
remoteRef:
key: production/api/database
property: password
|
The ExternalSecret manifest is safe to commit to Git — it contains no secrets, only references.
Sealed Secrets
1
2
3
4
5
6
7
8
9
10
11
12
|
# Install kubeseal
kubeseal --fetch-cert \
--controller-name=sealed-secrets \
--controller-namespace=kube-system > pub-cert.pem
# Seal a secret
kubectl create secret generic my-secret \
--from-literal=password=supersecret \
--dry-run=client -o yaml \
| kubeseal --cert pub-cert.pem > my-sealed-secret.yaml
# Commit my-sealed-secret.yaml to Git — it can only be decrypted by your cluster
|
Argo CD Vault Plugin
For Helm-based workflows, the Argo CD Vault Plugin substitutes <secret/path#key> placeholders in manifests at sync time:
1
2
3
4
|
# values.yaml (safe to commit — no actual secrets)
database:
url: <path:secret/data/production/api#database_url>
password: <path:secret/data/production/api#database_password>
|
Practical GitOps Workflow
Tying everything together for a typical feature deployment:
1. Developer opens PR with code changes in application repo
→ CI: lint, test, build, push image (myregistry/api:abc1234)
2. CI opens automated PR against gitops-config repo:
# apps/services/api/overlays/staging/kustomization.yaml
images:
- name: myregistry/api
newTag: abc1234 ← updated by CI
3. Team reviews and merges staging PR
→ Argo CD detects change, syncs to staging cluster within ~3 minutes
4. Automated smoke tests run against staging
→ Pass: Slack notification "✅ api deployed abc1234 to staging"
5. After QA sign-off, another PR updates production overlay:
# apps/services/api/overlays/production/kustomization.yaml
images:
- name: myregistry/api
newTag: abc1234 ← promoted from staging
6. Production PR review = production deployment approval gate
→ Senior engineer reviews diff and merges
7. Argo CD syncs production
→ Argo Rollouts starts canary: 5% traffic to new version
→ Analysis runs: success rate, latency P99
→ Gradually shifts traffic over 30 minutes
→ If analysis passes: 100% to new version
→ Slack: "🚀 api deployed abc1234 to production"
Every step is auditable. The Git history is the deployment history. Rolling back is git revert. Understanding what changed between any two deployments is git diff.
Health Checks and Sync Status
Argo CD evaluates the health of deployed resources using built-in logic for common types and custom health checks for CRDs:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
|
-- Custom health check for a CRD (in argocd-cm)
-- Checks if a custom resource has reached "Ready" condition
hs = {}
if obj.status ~= nil then
if obj.status.conditions ~= nil then
for i, condition in ipairs(obj.status.conditions) do
if condition.type == "Ready" and condition.status == "False" then
hs.status = "Degraded"
hs.message = condition.message
return hs
end
if condition.type == "Ready" and condition.status == "True" then
hs.status = "Healthy"
return hs
end
end
end
end
hs.status = "Progressing"
hs.message = "Waiting for Ready condition"
return hs
|
1
2
3
4
5
|
# argocd-cm ConfigMap
data:
resource.customizations.health.mygroup_MyResource: |
hs = {}
-- lua health check code above
|
Ignoring Expected Differences
Some fields legitimately differ between Git and the live cluster — admission webhooks inject fields, HPA modifies replica counts, operators add annotations:
1
2
3
4
5
6
7
|
# argocd-cm ConfigMap
data:
resource.customizations.ignoreDifferences.apps_Deployment: |
jsonPointers:
- /spec/replicas # Managed by HPA
jqPathExpressions:
- '.spec.template.metadata.annotations["kubectl.kubernetes.io/last-applied-configuration"]'
|
Or per-application:
1
2
3
4
5
6
7
8
9
10
11
|
# Application spec
spec:
ignoreDifferences:
- group: apps
kind: Deployment
jsonPointers:
- /spec/replicas
- group: ""
kind: Secret
jsonPointers:
- /data # Managed by external-secrets, values change frequently
|
The GitOps Mindset Shift
The hardest part of adopting Argo CD isn’t the tooling — it’s the cultural change. “I need to deploy this hotfix right now” becomes “I need to push a commit and wait for Argo CD to sync.” That feels slower until you’ve experienced the alternative: a cluster where nobody knows what’s deployed, changes aren’t documented, and rolling back means hoping someone remembers the previous state.
The constraints GitOps imposes are what make it valuable. Every deployment is reviewed. Every configuration change is attributable. Every rollback is a revert. The audit trail isn’t a compliance checkbox — it’s how you understand what’s running and why, at 2am when something breaks.
Start small: pick one non-critical application, commit its manifests to a Git repo, and let Argo CD manage it. The “aha” moment — when you realize you can understand your entire cluster’s desired state by reading a directory — comes quickly.
Comments