Progressive Delivery: Safe Deployments with Feature Flags, Canaries, and Argo Rollouts
Traditional deployments are binary: the old version is running, then the new version is running. Every deployment is a calculated gamble — you tested in staging, you reviewed the code, but you won’t really know if it works until it’s in production handling real traffic. If it fails, you’re scrambling to roll back while users experience errors.
Progressive delivery is the set of techniques that makes deployments incremental rather than binary. Instead of “old code off, new code on,” you ask: “What’s the smallest amount of traffic we can send to the new version to learn whether it’s safe, before we commit?”
The result isn’t just safer deployments. It’s a fundamentally different relationship with risk. When you can deploy to 1% of traffic, measure, and roll back in 30 seconds if something looks wrong — you deploy more frequently, with less fear, and with higher quality feedback loops.
This guide covers the full progressive delivery spectrum: feature flags for decoupling deploy from release, canary releases for gradual traffic shifting, blue-green for instant rollback, and Argo Rollouts for automating all of it on Kubernetes.
The Progressive Delivery Spectrum
Low risk ◄─────────────────────────────────────────────────► High risk
Feature flags Shadow mode Canary A/B test Blue-green Rolling
(0% live users) (copy traffic) (1-10%) (50/50) (100% swap) (rolling %)
Each technique has different trade-offs in risk, complexity, and what you can learn:
| Technique |
% Users affected at start |
Rollback time |
What you learn |
| Feature flag (off) |
0% |
Instant (config change) |
N/A — not running |
| Shadow / mirror |
0% (read-only copy) |
Instant |
Correctness, performance |
| Canary |
1–5% |
30 seconds |
Real-world error/latency |
| A/B test |
50% each variant |
Minutes |
Business metrics |
| Blue-green |
0% → 100% in one step |
Seconds (LB flip) |
Deployment correctness |
| Rolling update |
Gradual per pod |
Minutes |
Works but no traffic control |
Feature Flags: Decouple Deploy from Release
The most powerful idea in progressive delivery is separating deploy (code in production) from release (feature visible to users). Feature flags let you ship code that’s completely dark — deployed, tested in production, but affecting zero users until you’re ready.
The three types of feature flags
Release flags: Temporary. Used to dark-launch a feature and graduate it to 100%. Deleted after full release.
1
2
3
4
|
if feature_flags.enabled("new-checkout-flow", user=current_user):
return new_checkout_flow(cart)
else:
return legacy_checkout_flow(cart)
|
Experiment flags: A/B tests. Route users to different variants and measure which performs better. Deleted after the experiment concludes.
Operational flags: Long-lived kill switches. Enable degraded-mode features, disable expensive operations under load.
1
2
3
4
5
|
# Operational flag: disable AI recommendations if the model service is overloaded
if feature_flags.enabled("ai-product-recommendations"):
recommendations = await ai_service.recommend(user_id)
else:
recommendations = get_popular_items() # Fallback
|
Building a simple feature flag system
You don’t need a vendor for basic feature flags. A Redis-backed flag store with percentage rollout takes an afternoon to build:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
|
# flags.py
import hashlib
import json
import redis
from dataclasses import dataclass
from typing import Optional
@dataclass
class FlagConfig:
enabled: bool
rollout_percentage: float = 100.0 # 0.0 to 100.0
user_allowlist: list[str] = None # Always-on for these users
user_denylist: list[str] = None # Always-off for these users
attributes: dict = None # Extra metadata
class FeatureFlags:
def __init__(self, redis_client: redis.Redis):
self._redis = redis_client
self._cache = {} # Local cache with TTL
def enabled(self, flag_name: str, user_id: Optional[str] = None) -> bool:
config = self._get_config(flag_name)
if config is None or not config.enabled:
return False
# Allowlist always wins
if user_id and config.user_allowlist and user_id in config.user_allowlist:
return True
# Denylist always loses
if user_id and config.user_denylist and user_id in config.user_denylist:
return False
# Percentage rollout: deterministic hash so the same user always gets
# the same experience (sticky bucketing)
if config.rollout_percentage < 100.0:
if user_id is None:
return False
# Hash the user+flag combination for stable bucketing
bucket_key = f"{flag_name}:{user_id}"
hash_val = int(hashlib.md5(bucket_key.encode()).hexdigest(), 16)
bucket = (hash_val % 10000) / 100.0 # 0.0 to 99.99
return bucket < config.rollout_percentage
return True
def _get_config(self, flag_name: str) -> Optional[FlagConfig]:
# Check local cache first
if flag_name in self._cache:
return self._cache[flag_name]
raw = self._redis.get(f"flags:{flag_name}")
if raw is None:
return None
data = json.loads(raw)
config = FlagConfig(**data)
self._cache[flag_name] = config # Cache for 30s in practice
return config
def set_flag(self, flag_name: str, config: FlagConfig):
self._redis.set(
f"flags:{flag_name}",
json.dumps(config.__dict__),
)
self._cache.pop(flag_name, None)
# Usage
flags = FeatureFlags(redis.Redis(host="redis"))
# Dark launch: 0% rollout
flags.set_flag("new-checkout-flow", FlagConfig(
enabled=True,
rollout_percentage=0.0,
user_allowlist=["eng-test-user-1", "eng-test-user-2"], # Only internal
))
# Gradual rollout: 5% → 25% → 100%
flags.set_flag("new-checkout-flow", FlagConfig(
enabled=True,
rollout_percentage=5.0,
))
|
Using OpenFeature: the vendor-neutral standard
OpenFeature is a CNCF standard for feature flag SDKs. It lets you switch providers (LaunchDarkly → Unleash → your own) without changing application code.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
|
// Go — OpenFeature with the Unleash provider
import (
"github.com/open-feature/go-sdk/pkg/openfeature"
unleash "github.com/unleash/unleash-client-go/v4"
)
// Initialize once at startup
openfeature.SetProvider(NewUnleashProvider(unleash.Initialize(unleash.UnleashConfig{
AppName: "payment-service",
Url: "https://unleash.example.com/api",
ApiKey: os.Getenv("UNLEASH_API_KEY"),
})))
client := openfeature.NewClient("payment-service")
// Evaluate a flag with user context
evalCtx := openfeature.NewEvaluationContext(
userID,
map[string]interface{}{
"email": user.Email,
"plan": user.Plan,
"country": user.Country,
},
)
enabled, err := client.BooleanValue(
ctx, "new-checkout-flow", false, evalCtx,
)
if enabled {
return newCheckout(ctx, cart)
}
return legacyCheckout(ctx, cart)
|
LaunchDarkly integration (managed service)
For teams that want a managed solution with targeting rules, analytics, and audit logs:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
|
# Python — LaunchDarkly SDK
import ldclient
from ldclient.config import Config
ldclient.set_config(Config(os.environ["LAUNCHDARKLY_SDK_KEY"]))
client = ldclient.get()
# Rich targeting context
context = {
"kind": "user",
"key": user.id,
"email": user.email,
"plan": user.subscription_plan,
"country": user.country,
"beta_tester": user.is_beta_tester,
}
show_new_feature = client.variation("new-checkout-flow", context, default=False)
|
Canary Releases: Gradual Traffic Shifting
A canary release sends a small percentage of real traffic to the new version and compares its behavior against the stable version before proceeding.
Canary with Kubernetes and Nginx Ingress
The simplest canary: two Deployments, weighted Ingress annotations:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
|
# stable deployment — v1.2.0
apiVersion: apps/v1
kind: Deployment
metadata:
name: payment-service-stable
namespace: production
spec:
replicas: 9 # 90% of pods
selector:
matchLabels:
app: payment-service
version: stable
template:
metadata:
labels:
app: payment-service
version: stable
spec:
containers:
- name: payment-service
image: myorg/payment-service:v1.2.0
---
# canary deployment — v1.3.0
apiVersion: apps/v1
kind: Deployment
metadata:
name: payment-service-canary
namespace: production
spec:
replicas: 1 # 10% of pods
selector:
matchLabels:
app: payment-service
version: canary
template:
metadata:
labels:
app: payment-service
version: canary
spec:
containers:
- name: payment-service
image: myorg/payment-service:v1.3.0
---
# Canary Ingress — weight-based routing
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: payment-service-canary
namespace: production
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-weight: "10" # 10% to canary
spec:
rules:
- host: api.example.com
http:
paths:
- path: /api/payments
pathType: Prefix
backend:
service:
name: payment-service-canary
port:
number: 8080
|
Limitation: Pod-ratio canaries (9 stable pods + 1 canary = 10%) are coarse. For precise traffic percentages, use a service mesh or Argo Rollouts.
Canary with Istio VirtualService
Istio gives precise percentage control independent of pod count:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
|
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: payment-service
namespace: production
spec:
hosts:
- payment-service
http:
- route:
- destination:
host: payment-service
subset: stable
weight: 95
- destination:
host: payment-service
subset: canary
weight: 5 # Exactly 5%, regardless of pod count
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: payment-service
namespace: production
spec:
host: payment-service
subsets:
- name: stable
labels:
version: stable
- name: canary
labels:
version: canary
|
Argo Rollouts: Automated Progressive Delivery on Kubernetes
Argo Rollouts is the most complete solution for progressive delivery on Kubernetes. It replaces the standard Deployment object with a Rollout resource that natively supports canary and blue-green strategies with automated analysis.
Installation
1
2
3
4
5
6
|
kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f \
https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml
# Install the kubectl plugin
brew install argoproj/tap/kubectl-argo-rollouts
|
Canary Rollout with automated analysis
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
|
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: payment-service
namespace: production
spec:
replicas: 10
revisionHistoryLimit: 3
selector:
matchLabels:
app: payment-service
template:
metadata:
labels:
app: payment-service
spec:
containers:
- name: payment-service
image: myorg/payment-service:v1.3.0
ports:
- containerPort: 8080
strategy:
canary:
# Use Istio for traffic splitting (decoupled from pod count)
trafficRouting:
istio:
virtualService:
name: payment-service-vsvc
routes:
- primary
destinationRule:
name: payment-service-destrule
canarySubsetName: canary
stableSubsetName: stable
# Analysis to run during the rollout
analysis:
startingStep: 2 # Start analysis at step 2
steps:
# Step 1: 5% canary, pause and wait for manual approval or auto-analysis
- setWeight: 5
- pause: {duration: 5m} # Bake time: watch metrics for 5 minutes
# Step 2: Run analysis (defined separately as AnalysisTemplate)
- analysis:
templates:
- templateName: payment-service-success-rate
args:
- name: service-name
value: payment-service
# Step 3: Bump to 20% if analysis passed
- setWeight: 20
- pause: {duration: 5m}
# Step 4: 50% — watch for 10 minutes
- setWeight: 50
- pause: {duration: 10m}
# Step 5: Full rollout
- setWeight: 100
# Automatic rollback: if error rate spikes, abort and roll back
# This is checked continuously throughout the rollout
antiAffinity:
preferredDuringSchedulingIgnoredDuringExecution: {}
|
AnalysisTemplate: automated canary gates
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
|
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
name: payment-service-success-rate
namespace: production
spec:
args:
- name: service-name
metrics:
# Gate 1: Error rate must stay below 1%
- name: success-rate
interval: 1m
count: 5 # Run 5 times (over 5 minutes)
successCondition: result[0] >= 0.99
failureLimit: 1 # Allow 1 failure before aborting rollout
provider:
prometheus:
address: http://prometheus:9090
query: |
sum(rate(http_requests_total{
service="{{args.service-name}}",
version="canary",
status!~"5.."
}[1m]))
/
sum(rate(http_requests_total{
service="{{args.service-name}}",
version="canary"
}[1m]))
# Gate 2: P99 latency must stay under 500ms
- name: latency-p99
interval: 1m
count: 5
successCondition: result[0] <= 0.5
failureLimit: 1
provider:
prometheus:
address: http://prometheus:9090
query: |
histogram_quantile(0.99,
sum(rate(http_request_duration_seconds_bucket{
service="{{args.service-name}}",
version="canary"
}[1m])) by (le)
)
# Gate 3: Canary error rate must not be significantly worse than stable
- name: error-rate-comparison
interval: 2m
count: 3
# Canary error rate must be <= 2x stable error rate
successCondition: result[0] <= result[1] * 2
failureLimit: 1
provider:
prometheus:
address: http://prometheus:9090
query: |
(
sum(rate(http_requests_total{service="{{args.service-name}}",version="canary",status=~"5.."}[2m]))
/
sum(rate(http_requests_total{service="{{args.service-name}}",version="canary"}[2m]))
)
or vector(0)
# Second query for stable comparison
|
Background analysis: continuous monitoring throughout rollout
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
# Run this analysis continuously from step 1, not just at gates
spec:
strategy:
canary:
steps:
- setWeight: 5
- pause: {duration: 5m}
- setWeight: 25
- pause: {duration: 10m}
- setWeight: 100
# Runs in background throughout entire rollout — aborts if it fails
analysis:
templates:
- templateName: payment-service-success-rate
startingStep: 1 # Start at step 1 (immediately)
args:
- name: service-name
value: payment-service
|
Rollout management with kubectl plugin
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
|
# Watch rollout progress in real time
kubectl argo rollouts get rollout payment-service -n production --watch
# Manually promote to next step (when pause has no duration)
kubectl argo rollouts promote payment-service -n production
# Abort and roll back to stable
kubectl argo rollouts abort payment-service -n production
# Retry after fixing an abort
kubectl argo rollouts retry rollout payment-service -n production
# Set specific canary weight manually
kubectl argo rollouts set image payment-service payment-service=myorg/payment-service:v1.3.1
# Check analysis run status
kubectl argo rollouts get rollout payment-service -n production
# Shows: ✔ Step 2/5 | AnalysisRun: running | Weight: 20%
|
Blue-Green Deployments
Blue-green maintains two identical production environments (blue = current, green = new). Traffic is switched atomically — 0% to 100% in a single operation — but the old environment stays live for instant rollback.
Load Balancer
│
┌─────────┴─────────┐
│ │
Blue (v1) Green (v2)
[ACTIVE] [IDLE / STANDBY]
On deploy:
1. Deploy v2 to green
2. Run smoke tests against green (via internal URL)
3. Flip load balancer: green becomes ACTIVE
4. Blue remains running for instant rollback window (15 minutes)
5. After validation window: blue is decommissioned
Argo Rollouts blue-green strategy
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
|
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: frontend
namespace: production
spec:
replicas: 5
selector:
matchLabels:
app: frontend
template:
metadata:
labels:
app: frontend
spec:
containers:
- name: frontend
image: myorg/frontend:v2.0.0
strategy:
blueGreen:
# Active service (takes production traffic)
activeService: frontend-active
# Preview service (routes to green before cutover — for testing)
previewService: frontend-preview
# Don't auto-promote — require manual approval
autoPromotionEnabled: false
# Keep blue running for 30 minutes after promotion for rollback
scaleDownDelaySeconds: 1800
# Run analysis against the preview (green) before promoting
prePromotionAnalysis:
templates:
- templateName: smoke-test
args:
- name: service-url
value: "http://frontend-preview"
# Run analysis after promotion to confirm health
postPromotionAnalysis:
templates:
- templateName: post-deploy-validation
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
|
# Services for blue-green
---
apiVersion: v1
kind: Service
metadata:
name: frontend-active # Always points to current live pods
namespace: production
spec:
selector:
app: frontend
# Argo Rollouts manages the "rollouts-pod-template-hash" selector
---
apiVersion: v1
kind: Service
metadata:
name: frontend-preview # Points to new (green) pods during rollout
namespace: production
spec:
selector:
app: frontend
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
|
# Smoke test AnalysisTemplate for blue-green pre-promotion
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
name: smoke-test
namespace: production
spec:
args:
- name: service-url
metrics:
- name: homepage-smoke-test
count: 1
successCondition: result == "200"
provider:
web:
url: "{{args.service-url}}/healthz"
timeoutSeconds: 10
jsonPath: "{$.status}"
- name: api-smoke-test
count: 1
successCondition: result == "200"
provider:
web:
url: "{{args.service-url}}/api/v1/ping"
timeoutSeconds: 10
jsonPath: "{$.status_code}"
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
# Blue-green workflow
# 1. Update image — Argo creates green pods, routes preview service to them
kubectl argo rollouts set image frontend frontend=myorg/frontend:v2.0.0 -n production
# 2. Watch green come up
kubectl argo rollouts get rollout frontend -n production --watch
# 3. Test against preview URL (green)
curl http://frontend-preview.production.svc.cluster.local/healthz
# 4. Promote: flip active service from blue to green
kubectl argo rollouts promote frontend -n production
# 5. If something goes wrong after promotion (within 30-minute window):
kubectl argo rollouts abort frontend -n production
# This routes traffic back to blue immediately
|
Mirror / Shadow Traffic
Before a canary, consider shadow mode: duplicate all production traffic to the new version, but throw away its responses. The new version processes real traffic with zero user impact.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
|
# Istio VirtualService: mirror 100% of traffic to canary
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: payment-service
spec:
hosts:
- payment-service
http:
- route:
- destination:
host: payment-service
subset: stable
weight: 100
# Shadow: send copy of all requests to canary
# Users see only stable's responses
mirror:
host: payment-service
subset: canary
mirrorPercentage:
value: 100.0 # Shadow 100% of traffic
|
Use shadow mode to:
- Validate new code handles production traffic patterns without errors
- Profile performance under real load before any user exposure
- Test database migration correctness (compare query results between old and new)
A/B Testing with Argo Rollouts
A/B testing routes different user segments to different variants and measures business outcomes (conversion, revenue, engagement) rather than just technical metrics.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
|
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: checkout-ab-test
namespace: production
spec:
strategy:
canary:
trafficRouting:
istio:
virtualService:
name: checkout-vsvc
routes:
- primary
destinationRule:
name: checkout-destrule
canarySubsetName: variant-b
stableSubsetName: variant-a
steps:
# 50/50 split — hold until manual analysis
- setWeight: 50
- pause: {} # Indefinite pause — promote manually after analysis
# Header-based routing: force specific users to a variant for testing
# (handled in the VirtualService, not here)
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
|
# A/B routing: internal QA team always gets variant B
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: checkout-vsvc
spec:
hosts:
- checkout-service
http:
# Route QA users to variant B via header
- match:
- headers:
x-variant:
exact: "b"
route:
- destination:
host: checkout-service
subset: variant-b
# Everyone else: 50/50 split
- route:
- destination:
host: checkout-service
subset: variant-a
weight: 50
- destination:
host: checkout-service
subset: variant-b
weight: 50
|
GitOps Integration: Argo CD + Argo Rollouts
Argo Rollouts integrates naturally with Argo CD for a complete GitOps progressive delivery pipeline.
1
2
3
4
5
6
7
8
9
|
# Application in git → Argo CD applies → Argo Rollouts manages traffic
#
# Workflow:
# 1. Developer bumps image tag in git
# 2. Argo CD detects drift, syncs the Rollout resource
# 3. Argo Rollouts executes the canary strategy
# 4. Analysis gates pass/fail automatically
# 5. If all gates pass: Argo Rollouts promotes to 100%
# 6. If any gate fails: Argo Rollouts aborts, Argo CD shows "Degraded"
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
|
# argocd-application.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: payment-service
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/myorg/payment-service
targetRevision: HEAD
path: deploy/production
destination:
server: https://kubernetes.default.svc
namespace: production
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
# Custom health check: consider Argo Rollout health, not just Deployment health
ignoreDifferences:
- group: argoproj.io
kind: Rollout
jsonPointers:
- /spec/paused
|
Notification on rollout events
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
|
# Argo Rollouts notifications to Slack
apiVersion: v1
kind: ConfigMap
metadata:
name: argo-rollouts-notification-cm
namespace: argo-rollouts
data:
service.slack: |
token: $slack-token
template.rollout-completed: |
message: |
:white_check_mark: Rollout *{{.rollout.metadata.name}}* completed successfully
New version: `{{index .rollout.spec.template.spec.containers 0 "image"}}`
template.rollout-aborted: |
message: |
:rotating_light: Rollout *{{.rollout.metadata.name}}* ABORTED
Analysis failure — rolled back to stable version.
Check: kubectl argo rollouts get rollout {{.rollout.metadata.name}}
trigger.on-rollout-completed: |
- send: [rollout-completed]
when: rollout.status.phase == "Healthy"
trigger.on-rollout-aborted: |
- send: [rollout-aborted]
when: rollout.status.phase == "Degraded"
|
Practical Rollout Strategy Decision Guide
Q: Do you need to test in production before any users see it?
YES → Feature flag (dark launch)
NO → Continue
Q: Do you need instant rollback capability?
YES + can tolerate brief 0→100% switch → Blue-green
YES + want gradual exposure → Canary with Argo Rollouts
Q: Do you want to measure business outcomes (conversion, revenue)?
YES → A/B test with feature flags + analytics
NO → Canary based on technical metrics
Q: Is the change a database schema migration?
YES → Shadow mode first, then canary (never jump straight to blue-green)
Q: Is this a public-facing service with strict SLOs?
YES → Argo Rollouts with automated AnalysisTemplates
NO → Simple canary with manual promotion gates is fine
Observability for Progressive Delivery
Your rollout is only as good as the signals you use to evaluate it.
The four golden signals per variant
Track these for both stable and canary versions simultaneously:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
# Error rate — canary vs stable
sum(rate(http_requests_total{status=~"5..", version="canary"}[2m]))
/
sum(rate(http_requests_total{version="canary"}[2m]))
# Latency P99 — canary
histogram_quantile(0.99,
sum(rate(http_request_duration_seconds_bucket{version="canary"}[2m])) by (le)
)
# Throughput — ensure canary is actually receiving traffic
sum(rate(http_requests_total{version="canary"}[1m]))
# Saturation — canary pods aren't resource-constrained
sum(rate(container_cpu_usage_seconds_total{pod=~"payment-service-canary.*"}[2m]))
/ sum(kube_pod_container_resource_limits{pod=~"payment-service-canary.*", resource="cpu"})
|
Grafana dashboard for rollouts
Build a dashboard with two identical panels side by side — stable vs canary. The visual comparison makes regressions obvious at a glance:
┌────────────────────────┬────────────────────────┐
│ Stable (v1.2.0) │ Canary (v1.3.0) [10%] │
├────────────────────────┼────────────────────────┤
│ Error rate: 0.08% │ Error rate: 0.09% │ ✅
│ P99 latency: 180ms │ P99 latency: 185ms │ ✅
│ Throughput: 9,000 rps │ Throughput: 1,000 rps │ ✅
│ CPU: 42% │ CPU: 44% │ ✅
└────────────────────────┴────────────────────────┘
Common Pitfalls
Skipping the bake time
Analysis templates catch regression in metrics, but some issues take time to manifest (memory leaks, connection pool exhaustion, cache poisoning). Always include at least 5–10 minutes of bake time between steps.
Canary that never gets promoted
If manual promotion requires an engineer to remember to promote, it will sit at 5% indefinitely. Either automate promotion based on analysis, or set a calendar reminder. Stale canaries confuse everyone.
Testing the wrong metrics
If your canary routes to a subset of pods but all pods share the same database, a database regression will affect both stable and canary equally — and your comparison gates will show no difference. Know which metrics actually differ between versions.
Schema migrations before canary
If v2 requires a database schema change that’s incompatible with v1, a canary where v1 and v2 run simultaneously will corrupt data. Use the expand/contract pattern: v2 should be backward-compatible with the current schema; migrate schema after v2 is fully rolled out.
Not testing rollback
Rollback paths get rusty. Add a quarterly drill: deploy a known-bad version, watch the analysis fail, confirm the automatic rollback works end-to-end. Rollback procedures that aren’t tested don’t work when you need them.
Quick Reference
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
|
# Argo Rollouts commands
kubectl argo rollouts list rollouts -n production
kubectl argo rollouts get rollout <name> -n production --watch
kubectl argo rollouts promote <name> -n production # Advance to next step
kubectl argo rollouts promote <name> -n production --full # Skip all steps
kubectl argo rollouts abort <name> -n production # Abort & rollback
kubectl argo rollouts retry rollout <name> -n production # Retry after abort
kubectl argo rollouts pause <name> -n production # Pause at current step
kubectl argo rollouts resume <name> -n production # Resume from pause
kubectl argo rollouts set image <name> <container>=<image> -n production
# Check analysis runs
kubectl get analysisrun -n production
kubectl describe analysisrun <name> -n production
# Dashboard (port-forward Argo Rollouts UI)
kubectl argo rollouts dashboard -n production
# Open http://localhost:3100
|
Summary
Progressive delivery transforms deployments from high-stakes binary switches into low-risk incremental shifts. The key insight is that every deployment is also a measurement opportunity — if you route 1% of traffic to a new version and observe its behavior against real data, you get signal that no amount of staging testing can provide.
The practical stack for most teams:
- Feature flags for decoupling code deploy from feature release (OpenFeature + any backend)
- Argo Rollouts for canary and blue-green with automated analysis gates
- Prometheus + Grafana for the signals that drive canary promotion/abort decisions
- Argo CD for GitOps-driven rollout triggering
Start with the simplest tool that fits your risk level. A manual canary with two Deployments and weighted Ingress rules is dramatically safer than a rolling update with no traffic control, and it takes an hour to set up. Graduate to Argo Rollouts with automated analysis when you have the Prometheus metrics to back it up — that’s when deployments become genuinely fearless.
Comments