Secrets Rotation Without Downtime: Rotating Credentials in Live Systems

March 27, 2026 21 min read Security

securitysecretsvaultkubernetestlsdevopsoperationsdatabase

Contents

Rotating secrets in production is one of those problems that sounds simple until you do it. Stop the app, change the password, restart the app — fine for a hobby project. In production, “restart the app” means downtime, dropped requests, and a deployment pipeline that may take 20 minutes. Multiply that by every database, every API integration, every TLS certificate, on a regular rotation schedule, and you have a maintenance burden that most teams quietly skip.

The patterns in this guide make rotation a non-event. The application keeps serving traffic throughout. No deployment required. No downtime. The secret changes underneath it, and the application adapts.

Why Rotation Matters (and Why Teams Skip It)

A compromised credential that is never rotated is compromised forever. A rotated credential that gets stolen has a bounded exposure window — the attacker’s access ends at the next rotation.

Most regulatory frameworks (SOC 2, PCI DSS, HIPAA) require credential rotation. Most teams meet this requirement in the audit by demonstrating they could rotate, not by actually rotating. The reason is that rotation is operationally painful when it requires coordinated application restarts.

The goal of this guide: make rotation cheap enough that you do it frequently and automatically.

The Core Problem: Connection Pools

The hard part of credential rotation isn’t changing the password in the database — that’s one SQL statement. The hard part is that your application has a pool of open connections authenticated with the old password, and new connections need the new password, and both need to work simultaneously during the transition.

Application (connection pool)          Database
├── conn 1  ←── authenticated with ────→ old password
├── conn 2  ←── authenticated with ────→ old password
├── conn 3  ←── authenticated with ────→ old password
│
│  ← You change the password here →
│
├── conn 4  ←── tries to connect with ─→ new password ✓
└── conn 1  ←── existing conn still ──→ still valid (TCP session active)

Existing open connections stay valid after a password change on most databases — the authentication happened when the connection was opened. New connections authenticate with the new password. The pool naturally drains old connections over time (idle timeout, max lifetime) and establishes new ones.

This means the transition window is: from when you change the password until all old connections have cycled out. Your job is to manage this window safely.

Pattern 1: The Dual-Credential Pattern

The dual-credential pattern is the foundation of zero-downtime rotation for any credential type. The idea: always maintain two valid credentials simultaneously. Rotate one, then the other, with enough time between for the application to fully transition.

Phase 1 (Normal operation):
  Active credential:  password_A  ← application uses this
  Standby credential: password_B  ← also valid, not in use

Phase 2 (Rotation begins):
  Both credentials valid simultaneously
  Application reconfigures to use password_B
  Connections using password_A drain naturally

Phase 3 (Cleanup):
  Active credential:  password_B  ← application uses this
  Standby credential: password_A  ← revoke or regenerate as new standby

This requires database support for multiple valid credentials per user. PostgreSQL roles support this natively with multiple authentication methods. For simpler systems, use two separate users with identical permissions.

PostgreSQL Dual-User Pattern

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


-- Create two users with identical permissions
CREATE ROLE app_user_a LOGIN PASSWORD 'password_a_initial';
CREATE ROLE app_user_b LOGIN PASSWORD 'password_b_initial';

-- Grant both users the same permissions (via a shared role)
CREATE ROLE app_role;
GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO app_role;
GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA public TO app_role;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO app_role;

GRANT app_role TO app_user_a;
GRANT app_role TO app_user_b;

Your application config stores which user is currently active:

1
2
3
4
5
6
7


# config.yaml
database:
  active_user: app_user_a          # which user is currently in use
  password_a: "{{ secret:db/app_user_a }}"
  password_b: "{{ secret:db/app_user_b }}"
  host: postgres:5432
  name: myapp

Rotation Script

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67


#!/usr/bin/env bash
# rotate-db-password.sh
# Rotates the standby user's password and swaps active/standby roles.

set -euo pipefail

VAULT_ADDR="${VAULT_ADDR:-https://vault.internal}"
DB_HOST="${DB_HOST:-postgres}"
APP_NAMESPACE="${APP_NAMESPACE:-production}"
DRAIN_SECONDS="${DRAIN_SECONDS:-120}"  # time for old connections to cycle out

log() { echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] $*"; }

# Determine which user is currently standby
ACTIVE_USER=$(vault kv get -field=active_user "secret/app/database")
if [[ "$ACTIVE_USER" == "app_user_a" ]]; then
    STANDBY_USER="app_user_b"
    STANDBY_SECRET_PATH="secret/app/db_user_b"
else
    STANDBY_USER="app_user_a"
    STANDBY_SECRET_PATH="secret/app/db_user_a"
fi

log "Active user: $ACTIVE_USER, rotating standby: $STANDBY_USER"

# 1. Generate new password for standby user
NEW_PASSWORD=$(openssl rand -base64 32 | tr -d '/+=')

# 2. Change the standby user's password in the database
log "Changing $STANDBY_USER password in database"
PGPASSWORD="$(vault kv get -field=password secret/app/db_admin)" \
    psql -h "$DB_HOST" -U "app_admin" -d "myapp" \
    -c "ALTER ROLE $STANDBY_USER PASSWORD '$NEW_PASSWORD';"

# 3. Update the new password in Vault
log "Updating password in Vault"
vault kv put "$STANDBY_SECRET_PATH" password="$NEW_PASSWORD"

# 4. Flip the application to use the standby user (now freshly rotated)
log "Switching application to use $STANDBY_USER"
vault kv patch "secret/app/database" active_user="$STANDBY_USER"

# 5. Trigger application config reload (without restart)
# This depends on your application — options below

# Option A: Kubernetes — annotate to trigger config reload via Reloader
kubectl annotate deployment myapp -n "$APP_NAMESPACE" \
    "secret.reloader.stakater.com/reload=$(date +%s)" --overwrite

# Option B: Send SIGHUP to processes that watch config
kubectl exec -n "$APP_NAMESPACE" deploy/myapp -- kill -HUP 1

# Option C: Applications using Vault Agent — automatic (no action needed)

# 6. Wait for drain: old connections using the previous active user cycle out
log "Waiting ${DRAIN_SECONDS}s for connection pool drain..."
sleep "$DRAIN_SECONDS"

# 7. Verify the application is healthy
log "Verifying application health"
HEALTH=$(curl -sf "https://myapp.internal/readyz" | jq -r '.status')
if [[ "$HEALTH" != "ok" ]]; then
    log "ERROR: Application health check failed after rotation!"
    exit 1
fi

log "Rotation complete. New active user: $STANDBY_USER"

Pattern 2: Dynamic Secrets with HashiCorp Vault

Dynamic secrets are the highest-security approach: Vault generates a unique, short-lived credential for each application instance. No credential is ever shared, stored in config, or manually managed.

Application Instance 1  →  Vault  →  CREATE USER app_i1_abc123 (expires in 1h)
Application Instance 2  →  Vault  →  CREATE USER app_i1_def456 (expires in 1h)

Each instance has its own credential.
Leaked credential from Instance 1 cannot be used by an attacker to impersonate Instance 2.
When the lease expires, Vault runs: DROP USER app_i1_abc123

Setting Up Vault Database Secrets Engine

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34


# Enable the database secrets engine
vault secrets enable database

# Configure Vault's connection to PostgreSQL
vault write database/config/myapp-postgres \
    plugin_name=postgresql-database-plugin \
    allowed_roles="app-readonly,app-readwrite" \
    connection_url="postgresql://{{username}}:{{password}}@postgres:5432/myapp?sslmode=require" \
    username="vault_admin" \
    password="$(vault read -field=password secret/bootstrap/vault-admin)"

# Define a role — Vault will execute these statements to create credentials
vault write database/roles/app-readwrite \
    db_name=myapp-postgres \
    creation_statements="
        CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';
        GRANT app_role TO \"{{name}}\";
    " \
    revocation_statements="
        REVOKE app_role FROM \"{{name}}\";
        DROP ROLE IF EXISTS \"{{name}}\";
    " \
    default_ttl="1h" \
    max_ttl="24h"

# Test it
vault read database/creds/app-readwrite
# Key                Value
# ---                -----
# lease_id           database/creds/app-readwrite/AbCdEfGhIjKlMnOp
# lease_duration     1h
# lease_renewable    true
# password           A1B2C3D4E5F6G7H8I9J0
# username           v-app-abc123def456

Vault Agent Sidecar (Kubernetes)

Vault Agent runs as a sidecar container, handles authentication, renews leases, and writes credentials to a shared volume or environment variables. Your application just reads from a file — it never talks to Vault directly.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "myapp"
        vault.hashicorp.com/agent-inject-secret-db-creds: "database/creds/app-readwrite"
        vault.hashicorp.com/agent-inject-template-db-creds: |
          {{- with secret "database/creds/app-readwrite" -}}
          DATABASE_URL=postgresql://{{ .Data.username }}:{{ .Data.password }}@postgres:5432/myapp
          {{- end }}
        # Restart the app when credentials are renewed
        vault.hashicorp.com/agent-inject-command-db-creds: "kill -HUP 1"
    spec:
      serviceAccountName: myapp
      containers:
        - name: myapp
          image: myapp:latest
          env:
            - name: SECRETS_FILE
              value: /vault/secrets/db-creds
          volumeMounts:
            - name: vault-secrets
              mountPath: /vault/secrets
              readOnly: true

The Vault Agent writes /vault/secrets/db-creds which contains:

DATABASE_URL=postgresql://v-app-abc123:A1B2C3D4@postgres:5432/myapp

When the credential is about to expire, Vault Agent renews the lease, writes new credentials to the file, and sends SIGHUP to your application. Your application’s SIGHUP handler re-reads the config and refreshes the connection pool.

Application-Side Lease Renewal

For applications that need more control, use the Vault Go/Python SDK directly:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125


// pkg/db/vault_credentials.go
package db

import (
    "context"
    "database/sql"
    "fmt"
    "sync"
    "time"

    vault "github.com/hashicorp/vault/api"
    "go.uber.org/zap"
)

type VaultDBCredentials struct {
    client    *vault.Client
    role      string
    db        **sql.DB // pointer to pointer so we can swap the DB
    mu        sync.RWMutex
    log       *zap.Logger
    leaseID   string
    expiresAt time.Time
}

func NewVaultDBCredentials(client *vault.Client, role, dsn string, log *zap.Logger) (*VaultDBCredentials, error) {
    v := &VaultDBCredentials{
        client: client,
        role:   role,
        log:    log,
    }

    if err := v.rotate(context.Background(), dsn); err != nil {
        return nil, fmt.Errorf("initial credential fetch: %w", err)
    }

    // Background renewal loop
    go v.renewLoop(context.Background(), dsn)

    return v, nil
}

// DB returns the current database connection (thread-safe)
func (v *VaultDBCredentials) DB() *sql.DB {
    v.mu.RLock()
    defer v.mu.RUnlock()
    return *v.db
}

func (v *VaultDBCredentials) rotate(ctx context.Context, dsnTemplate string) error {
    secret, err := v.client.Logical().ReadWithContext(ctx,
        fmt.Sprintf("database/creds/%s", v.role))
    if err != nil {
        return fmt.Errorf("fetching credentials from Vault: %w", err)
    }

    username := secret.Data["username"].(string)
    password := secret.Data["password"].(string)

    dsn := fmt.Sprintf("host=postgres dbname=myapp user=%s password=%s sslmode=require",
        username, password)

    newDB, err := sql.Open("postgres", dsn)
    if err != nil {
        return fmt.Errorf("opening new connection: %w", err)
    }

    // Verify the new connection works before swapping
    if err := newDB.PingContext(ctx); err != nil {
        newDB.Close()
        return fmt.Errorf("new connection failed ping: %w", err)
    }

    newDB.SetMaxOpenConns(25)
    newDB.SetMaxIdleConns(5)
    newDB.SetConnMaxLifetime(55 * time.Minute) // slightly less than the 1h lease

    v.mu.Lock()
    oldDB := v.db
    v.db = &newDB
    v.leaseID = secret.LeaseID
    v.expiresAt = time.Now().Add(time.Duration(secret.LeaseDuration) * time.Second)
    v.mu.Unlock()

    // Close old connections after a drain period
    if oldDB != nil && *oldDB != nil {
        go func() {
            time.Sleep(30 * time.Second)
            (*oldDB).Close()
        }()
    }

    v.log.Info("database credentials rotated",
        zap.String("username", username),
        zap.String("lease_id", secret.LeaseID),
        zap.Time("expires_at", v.expiresAt),
    )
    return nil
}

func (v *VaultDBCredentials) renewLoop(ctx context.Context, dsn string) {
    for {
        v.mu.RLock()
        expiresAt := v.expiresAt
        leaseID := v.leaseID
        v.mu.RUnlock()

        // Renew at 75% of lease lifetime to leave buffer
        renewAt := expiresAt.Add(-time.Until(expiresAt) / 4)
        select {
        case <-time.Until(renewAt) |> time.After:
        case <-ctx.Done():
            return
        }

        // Try to renew the lease
        _, err := v.client.Sys().RenewWithContext(ctx, leaseID, 0)
        if err != nil {
            v.log.Warn("lease renewal failed, rotating credentials",
                zap.Error(err))
            if err := v.rotate(ctx, dsn); err != nil {
                v.log.Error("credential rotation failed", zap.Error(err))
            }
        }
    }
}

Pattern 3: API Key Rotation

API keys for third-party services (Stripe, Twilio, SendGrid, AWS) need rotation without service interruption. The process is almost always:

Generate a new key in the provider’s dashboard
Update your secret store with the new key
Deploy/reload the application to use the new key
Verify everything works
Revoke the old key

The challenge is step 3 and 5 — particularly if multiple services or deployments share the same key.

The Versioned API Key Pattern

Store keys with version tags. Your application can try the current key, and fall back to the previous key during the transition window:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51


// pkg/secrets/api_key.go
package secrets

import (
    "context"
    "fmt"
    "sync"
    "time"
)

type VersionedAPIKey struct {
    mu         sync.RWMutex
    current    string
    previous   string
    rotatedAt  time.Time
    gracePeriod time.Duration // how long to keep previous key valid
}

func (k *VersionedAPIKey) Rotate(newKey string) {
    k.mu.Lock()
    defer k.mu.Unlock()
    k.previous = k.current
    k.current = newKey
    k.rotatedAt = time.Now()
}

// TryWithFallback executes fn with the current key.
// If fn returns an auth error, it retries with the previous key during
// the grace period. This handles the race where one instance is still
// using the old key while another has already rotated.
func (k *VersionedAPIKey) TryWithFallback(
    ctx context.Context,
    fn func(ctx context.Context, key string) error,
) error {
    k.mu.RLock()
    current := k.current
    previous := k.previous
    rotatedAt := k.rotatedAt
    k.mu.RUnlock()

    err := fn(ctx, current)
    if err == nil {
        return nil
    }

    // Only fall back if it looks like an auth error AND we're in the grace period
    if isAuthError(err) && time.Since(rotatedAt) < k.gracePeriod && previous != "" {
        return fn(ctx, previous)
    }
    return err
}

Rotating AWS Credentials

AWS IAM supports two access keys per user, making the dual-credential pattern straightforward:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52


#!/usr/bin/env bash
# rotate-aws-keys.sh
set -euo pipefail

IAM_USER="${IAM_USER:-myapp-service-account}"
SECRET_PATH="${SECRET_PATH:-secret/app/aws}"

log() { echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] $*"; }

# 1. Get current key ID (so we can delete it at the end)
OLD_KEY_ID=$(vault kv get -field=aws_access_key_id "$SECRET_PATH")

# 2. Create new access key
log "Creating new AWS access key for $IAM_USER"
NEW_KEY=$(aws iam create-access-key --user-name "$IAM_USER")
NEW_KEY_ID=$(echo "$NEW_KEY" | jq -r '.AccessKey.AccessKeyId')
NEW_SECRET=$(echo "$NEW_KEY" | jq -r '.AccessKey.SecretAccessKey')

# 3. Store new key in Vault
log "Storing new key in Vault"
vault kv put "$SECRET_PATH" \
    aws_access_key_id="$NEW_KEY_ID" \
    aws_secret_access_key="$NEW_SECRET"

# 4. Trigger application reload
log "Triggering application config reload"
kubectl rollout restart deployment/myapp -n production

# 5. Wait for rollout and verify health
log "Waiting for rollout..."
kubectl rollout status deployment/myapp -n production --timeout=5m

# 6. Verify the new key works (check CloudWatch or application health)
sleep 30
HEALTH=$(curl -sf "https://myapp.internal/readyz" | jq -r '.status')
if [[ "$HEALTH" != "ok" ]]; then
    log "ERROR: Health check failed. Rolling back key rotation."
    # Restore old key in Vault
    vault kv put "$SECRET_PATH" \
        aws_access_key_id="$OLD_KEY_ID" \
        # Can't recover old secret from Vault since we overwrote it
        # This is why you keep a backup or use Vault's versioning
    exit 1
fi

# 7. Delete the old key
log "Deleting old key $OLD_KEY_ID"
aws iam delete-access-key \
    --user-name "$IAM_USER" \
    --access-key-id "$OLD_KEY_ID"

log "AWS key rotation complete. New key: $NEW_KEY_ID"

Better still: use IAM roles instead of long-lived access keys wherever possible. AWS SDK automatically refreshes instance role credentials without any rotation needed.

Pattern 4: TLS Certificate Rotation

TLS certificates expire. Let them expire without a rotation process and you have an outage. Automate rotation and it becomes invisible.

cert-manager in Kubernetes (The Right Way)

cert-manager is the Kubernetes-native solution for certificate lifecycle management. It provisions certificates from Let’s Encrypt, Vault PKI, or any ACME-compatible CA, and renews them automatically before expiry.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41


# certificate.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: myapp-tls
  namespace: production
spec:
  secretName: myapp-tls-cert        # Kubernetes Secret that stores the cert
  duration: 2160h                    # 90 days
  renewBefore: 720h                  # Renew 30 days before expiry

  dnsNames:
    - myapp.example.com
    - api.myapp.example.com

  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer

---
# clusterissuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ops@example.com
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
      - http01:
          ingress:
            class: nginx
      - dns01:  # For wildcard certs or internal services
          cloudflare:
            email: ops@example.com
            apiTokenSecretRef:
              name: cloudflare-api-token
              key: api-token

When cert-manager renews the certificate, it updates the Secret myapp-tls-cert. Your Ingress controller (Nginx, Traefik) watches for Secret changes and reloads the certificate without restarting. Zero downtime rotation happens automatically.

Verify cert-manager is renewing correctly:

1
2
3
4
5
6
7
8
9


# Check certificate status
kubectl describe certificate myapp-tls -n production

# Check renewal events
kubectl get events -n production --field-selector reason=Issued

# Check expiry
kubectl get secret myapp-tls-cert -n production -o jsonpath='{.data.tls\.crt}' \
    | base64 -d | openssl x509 -noout -dates

Internal PKI with Vault

For internal services that need mTLS but don’t need publicly trusted certificates, Vault’s PKI secrets engine is the right tool:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


# Enable PKI engine
vault secrets enable pki
vault secrets tune -max-lease-ttl=87600h pki  # 10 year CA

# Generate internal CA
vault write pki/root/generate/internal \
    common_name="Internal CA" \
    ttl=87600h

# Configure issuing and CRL URLs
vault write pki/config/urls \
    issuing_certificates="https://vault.internal/v1/pki/ca" \
    crl_distribution_points="https://vault.internal/v1/pki/crl"

# Enable intermediate CA (best practice — don't issue from root directly)
vault secrets enable -path=pki_int pki
vault secrets tune -max-lease-ttl=43800h pki_int  # 5 year max

# Issue intermediate CA cert (sign with root, then import)
vault write pki_int/intermediate/generate/internal \
    common_name="Internal Intermediate CA" > int_csr.json
vault write pki/root/sign-intermediate \
    csr="$(jq -r .data.csr int_csr.json)" \
    format=pem_bundle ttl=43800h > int_cert.json
vault write pki_int/intermediate/set-signed \
    certificate="$(jq -r .data.certificate int_cert.json)"

# Create a role for issuing service certs
vault write pki_int/roles/internal-service \
    allowed_domains="internal,svc.cluster.local" \
    allow_subdomains=true \
    allow_glob_domains=true \
    max_ttl="24h"          # Short-lived — rotate every 24h

# Issue a certificate (Vault Agent does this automatically)
vault write pki_int/issue/internal-service \
    common_name="myapp.production.svc.cluster.local" \
    ttl="24h"

With max_ttl=24h, certificates expire daily. Combined with Vault Agent renewing 6 hours before expiry, you get automatic rotation every ~18 hours with zero operator intervention.

Configuring Applications for Hot Certificate Reload

Your application needs to pick up the new certificate without restarting:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101


// pkg/tls/reloading_config.go
package tls

import (
    "crypto/tls"
    "crypto/x509"
    "os"
    "sync"
    "time"

    "go.uber.org/zap"
)

// ReloadingCertificate watches a certificate file and reloads it
// automatically when it changes. The TLS server uses GetCertificate
// to always get the current certificate.
type ReloadingCertificate struct {
    certFile string
    keyFile  string
    log      *zap.Logger

    mu          sync.RWMutex
    certificate *tls.Certificate
    modTime     time.Time
}

func NewReloadingCertificate(certFile, keyFile string, log *zap.Logger) (*ReloadingCertificate, error) {
    rc := &ReloadingCertificate{
        certFile: certFile,
        keyFile:  keyFile,
        log:      log,
    }
    if err := rc.reload(); err != nil {
        return nil, err
    }
    go rc.watchAndReload()
    return rc, nil
}

func (rc *ReloadingCertificate) reload() error {
    cert, err := tls.LoadX509KeyPair(rc.certFile, rc.keyFile)
    if err != nil {
        return err
    }
    info, err := os.Stat(rc.certFile)
    if err != nil {
        return err
    }

    rc.mu.Lock()
    rc.certificate = &cert
    rc.modTime = info.ModTime()
    rc.mu.Unlock()

    rc.log.Info("TLS certificate loaded",
        zap.String("cert_file", rc.certFile),
        zap.Time("mod_time", info.ModTime()),
    )
    return nil
}

func (rc *ReloadingCertificate) watchAndReload() {
    ticker := time.NewTicker(30 * time.Second)
    defer ticker.Stop()
    for range ticker.C {
        info, err := os.Stat(rc.certFile)
        if err != nil {
            rc.log.Error("stat cert file", zap.Error(err))
            continue
        }

        rc.mu.RLock()
        unchanged := info.ModTime().Equal(rc.modTime)
        rc.mu.RUnlock()

        if unchanged {
            continue
        }

        rc.log.Info("certificate file changed, reloading")
        if err := rc.reload(); err != nil {
            rc.log.Error("failed to reload certificate", zap.Error(err))
        }
    }
}

// GetCertificate is used as tls.Config.GetCertificate
// It's called for each new TLS connection, so it always uses the latest cert.
func (rc *ReloadingCertificate) GetCertificate(*tls.ClientHelloInfo) (*tls.Certificate, error) {
    rc.mu.RLock()
    defer rc.mu.RUnlock()
    return rc.certificate, nil
}

// Usage:
// rc, err := tls.NewReloadingCertificate("/vault/secrets/tls.crt", "/vault/secrets/tls.key", log)
// server := &http.Server{
//     TLSConfig: &tls.Config{
//         GetCertificate: rc.GetCertificate,
//     },
// }

Monitoring Certificate Expiry

Never let a certificate expire unnoticed:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


# prometheus-cert-alerts.yaml
groups:
  - name: tls
    rules:
      - alert: CertificateExpiringWarning
        expr: |
          (probe_ssl_earliest_cert_expiry - time()) / 86400 < 30
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "Certificate expiring in {{ $value | humanizeDuration }}"
          description: "{{ $labels.instance }} certificate expires soon"

      - alert: CertificateExpiringSoon
        expr: |
          (probe_ssl_earliest_cert_expiry - time()) / 86400 < 7
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: "Certificate expiring in {{ $value | humanizeDuration }} — URGENT"

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# blackbox-exporter config for certificate monitoring
modules:
  http_2xx_cert:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: [200]
      tls_config:
        insecure_skip_verify: false

Rotation Runbook Template

Document your rotation procedures before you need them. This is what a complete rotation runbook looks like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


# Database Password Rotation Runbook

## When to Run
- Scheduled: every 90 days (automated via cron)
- Unscheduled: immediately if credential compromise suspected

## Prerequisites
- [ ] Vault access with `secret/app/database` write permission
- [ ] kubectl access to production namespace
- [ ] Monitoring dashboard open to watch error rates during rotation

## Automated Rotation (Normal Path)

Run the rotation script:
```bash
./scripts/rotate-db-password.sh \
    --app myapp \
    --namespace production \
    --drain-seconds 120

Monitor: error_rate and db_connection_errors metrics during the drain period.

Manual Rotation (Emergency Path)

Use if the automated script fails or if credential compromise is suspected.

Step 1: Generate new password

1
2


NEW_PASS=$(openssl rand -base64 32 | tr -d '/+=')
echo "New password: $NEW_PASS"  # store this temporarily

Step 2: Create new password in database (both old and new valid)

1

psql -h postgres -U admin -c "ALTER ROLE app_user_b PASSWORD '$NEW_PASS';"

Step 3: Update Vault

1
2


vault kv put secret/app/db_user_b password="$NEW_PASS"
vault kv patch secret/app/database active_user="app_user_b"

Step 4: Reload application config

1
2


kubectl rollout restart deployment/myapp -n production
kubectl rollout status deployment/myapp -n production

Step 5: Verify (wait 2 minutes, then check)

1
2


curl -sf https://myapp.internal/readyz
# Should return: {"status": "ok"}

Step 6: Revoke old user’s password (after verifying app is healthy)

1
2
3


# Generate a random garbage password to invalidate old credentials
OLD_GARBAGE=$(openssl rand -base64 32)
psql -h postgres -U admin -c "ALTER ROLE app_user_a PASSWORD '$OLD_GARBAGE';"

Rollback Procedure

If the application fails health checks after rotation:

Restore old active user in Vault:

1

vault kv patch secret/app/database active_user="app_user_a"

Restart application:

1

kubectl rollout restart deployment/myapp -n production

Investigate failure in application logs before re-attempting rotation

Post-Rotation Verification

Application error rate returns to baseline within 5 minutes
No authentication failed errors in application logs
Database connection count is stable
Old user’s password has been invalidated
Rotation logged in audit trail (Vault audit log)

Contact

Primary on-call: via PagerDuty
Database admin: @dbadmin on Slack
Security team: @security (for suspected compromise cases)


---

## Kubernetes Secrets: Automatic Reload with Stakater Reloader

If you're using Kubernetes Secrets directly (not Vault), the [Stakater Reloader](https://github.com/stakater/Reloader) watches for Secret and ConfigMap changes and triggers rolling restarts automatically:

```bash
helm repo add stakater https://stakater.github.io/stakater-charts
helm install reloader stakater/reloader -n reloader --create-namespace

1
2
3
4
5
6
7


# deployment.yaml
metadata:
  annotations:
    # Restart when this specific secret changes
    secret.reloader.stakater.com/reload: "myapp-db-credentials"
    # Or restart when ANY referenced secret changes
    reloader.stakater.com/auto: "true"

Now your rotation workflow becomes:

1
2
3
4
5
6
7


# Rotate the password
kubectl create secret generic myapp-db-credentials \
    --from-literal=password="$(openssl rand -base64 32)" \
    --dry-run=client -o yaml | kubectl apply -f -

# Reloader detects the change and triggers a rolling restart automatically
# No manual intervention needed

The Hierarchy of Secret Rotation Maturity

Level 0 — Manual, on-incident only Passwords are rotated when someone notices they’re stale or after a breach. High risk, high toil.

Level 1 — Manual on a schedule A calendar reminder to rotate credentials every 90 days. Still toil, but predictable exposure windows.

Level 2 — Scripted rotation Rotation scripts exist and are documented. Still requires a human to run them. Good enough for most teams.

Level 3 — Automated rotation Cron job or CI pipeline triggers rotation automatically. Humans review logs, not run scripts. This is the target for most production systems.

Level 4 — Dynamic secrets Credentials are ephemeral and per-instance. No rotation needed — each lease expiry is effectively a rotation. Highest security, requires Vault or equivalent.

The right target depends on your security posture. Level 3 is achievable by most teams within a sprint using the patterns in this guide. Level 4 is worth investing in for credentials with high blast radius (production databases, payment processors).