The On-Call Handbook: Rotations, Runbooks, and Recovering Without Burning Out

Being on call is one of the most demanding parts of engineering work. A 3 AM page that wakes you from deep sleep, a cascading failure you’ve never seen before, the pressure to fix it fast before users notice — it demands technical skill, clear thinking under stress, and emotional resilience simultaneously.

Done badly, on-call destroys engineers. Teams with poorly designed rotations, alert storms, and no runbooks see burnout, attrition, and a perverse incentive structure where the engineers who understand the system least are the ones who suffer most when it breaks.

Done well, on-call is a forcing function for reliability. Every page is feedback: this system is not self-healing enough, this runbook is incomplete, this alert threshold is wrong. Teams that treat on-call as a discipline — not just a burden shared round-robin — build systems that are genuinely reliable and engineers who genuinely understand them.

This guide covers everything from rotation mechanics to runbook writing to the human side of recovery after a hard incident.

The Fundamental Goal: Sleep Through the Night

Before anything else, internalize this principle from Google’s SRE book: if on-call is so painful that engineers are suffering, you have a reliability problem, not a scheduling problem. Adding more engineers to the rotation, buying fancier alerting tools, and writing compensation policies all treat symptoms. The root cause is always the same: the system pages too much because it fails too often.

Healthy on-call metrics (targets to aim for):

• Fewer than 2 pages per on-call shift that require active response
• Less than 25% of on-call time spent on reactive work vs proactive improvements
• Mean time to acknowledge (MTTA) < 5 minutes during business hours, < 15 minutes overnight
• Zero “noisy” alerts — every page is actionable, every alert has a runbook

If you’re getting 20 pages a week, you don’t need better tooling. You need to fix what’s causing 20 pages a week.

Rotation Design

Primary + secondary model

The most battle-tested structure for teams of 6+:

Primary on-call:  First responder. Acknowledges pages, works the incident.
Secondary on-call: Backup if primary doesn't acknowledge in 5 minutes.
                  Also available as a second pair of eyes on hard problems.
Incident commander: (optional, for larger incidents) Coordinates response,
                    handles communication, keeps primary focused on the fix.

Rotation length trade-offs

Recommendation for most teams: One-week primary/secondary rotation with a soft rule that engineers swap out after two consecutive on-call weeks. Anyone on back-to-back rotations accumulates sleep debt and makes worse decisions.

Minimum viable rotation size

You need enough engineers to make the burden bearable. Rule of thumb: one person on call per rotation = max one week on call per N weeks. With a 6-person team, each engineer is on call 1 week in 6. That’s reasonable.

Under 4 engineers in a rotation, the burden becomes unsustainable. Options:

• Reduce scope (not all services need 24/7 coverage)
• Tiered coverage (business hours only for lower-severity services)
• External escalation for off-hours (managed service, MSP)

Time-zone aware scheduling

Team spread: NYC, London, Singapore

On-call hours (local business hours coverage):
  NYC (UTC-5):      08:00–18:00 ET = 13:00–23:00 UTC
  London (UTC+0):   08:00–18:00 GMT = 08:00–18:00 UTC
  Singapore (UTC+8): 09:00–18:00 SGT = 01:00–10:00 UTC

Overlap gaps:
  23:00–01:00 UTC: No coverage (NYC off, Singapore not yet)
  10:00–13:00 UTC: No coverage (Singapore off, NYC not yet)

Solution options:
  1. NYC extends to midnight (paid, not ideal)
  2. Rotating "stretch" person covers the gaps
  3. Accept 2-hour coverage gaps for non-critical services
  4. Singapore starts at 07:00 SGT to close the morning gap

Fairness rules

Write these down explicitly — ambiguity breeds resentment:

1. Holidays: Rotate who covers major holidays. Track and compensate (comp time or cash).
2. Swaps: Any engineer can request a swap 48+ hours in advance; requestee decides. Management doesn’t mandate swaps.
3. New hire ramp: New engineers shadow on-call for one rotation before going primary.
4. Senior engineers pull weight: Seniority doesn’t exempt anyone. If anything, senior engineers should take proportionally more weekend/holiday shifts.
5. Compensation: Be explicit. Google and many companies pay a per-shift stipend plus per-page compensation. Even a token amount signals that the organization values the work.

Alert Design: The Foundation of Sane On-Call

The single biggest lever for improving on-call quality is alert hygiene. Most teams alert on symptoms of symptoms — internal metrics that may or may not correlate with user impact. The goal is to alert on user-visible impact and route to the person who can fix it.

The four categories of alerts

             User impact?
             YES              NO
Actionable?  YES │ PAGE NOW   │ TICKET (fix soon)
             NO  │ PAGE NOW*  │ DELETE IT
                 │  *investigate missing instrumentation

Every alert in your system should be in one cell of this matrix. If it’s not actionable, it doesn’t belong as a page.

Alert properties checklist

Before any alert goes to production, it must have:

• A clear name describing what broke, not what’s measuring it
  • Bad: node_cpu_usage_high
  • Good: PaymentService: Error rate above 1% SLO threshold
• A runbook link in the alert annotation (not optional)
• Severity label (critical, warning, info) with consistent definitions
• Owner label so routing is unambiguous
• Noise test: Has this alert fired in the past 90 days? If never, is the threshold realistic? If too often, is it actionable each time?

Alert severity definitions (write these down)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# Prometheus alert labels
severity: critical
# Definition: User-visible outage or data loss. Revenue impact. Page immediately
# regardless of time. Requires incident commander for 30+ minutes.
# Examples: Payment processing down, API returning 100% errors, data corruption

severity: warning
# Definition: Degraded but not broken. SLO at risk. Page during business hours;
# queue for on-call acknowledgment overnight (next 4h window).
# Examples: P99 latency trending up, error rate at 0.5% (50% of budget)

severity: info
# Definition: Informational. Never pages. Appears in dashboards and digests.
# Examples: Disk at 70%, cert expiring in 30 days, deployment completed

Prometheus alert examples with proper hygiene

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65

groups:
  - name: payment_service_slos
    rules:
      # CRITICAL: SLO breach — users are experiencing errors right now
      - alert: PaymentServiceHighErrorRate
        expr: |
          (
            sum(rate(http_requests_total{service="payment-service", status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total{service="payment-service"}[5m]))
          ) > 0.01
        for: 2m
        labels:
          severity: critical
          team: payments
          service: payment-service
        annotations:
          summary: "Payment service error rate {{ $value | humanizePercentage }} (SLO: < 1%)"
          description: |
            The payment service is returning errors to {{ $value | humanizePercentage }} of requests.
            This is above the 1% SLO threshold and is causing user-visible failures.
          runbook_url: "https://runbooks.example.com/payment-service/high-error-rate"
          dashboard_url: "https://grafana.example.com/d/payments-overview"

      # WARNING: Error budget burn rate — trending toward breach
      - alert: PaymentServiceErrorBudgetBurn
        expr: |
          (
            sum(rate(http_requests_total{service="payment-service", status=~"5.."}[1h]))
            /
            sum(rate(http_requests_total{service="payment-service"}[1h]))
          ) > 0.005  # 50% of error budget in 1h window
        for: 5m
        labels:
          severity: warning
          team: payments
        annotations:
          summary: "Payment service error budget burning at 2x rate"
          runbook_url: "https://runbooks.example.com/payment-service/error-budget-burn"

      # CRITICAL: Latency SLO breach
      - alert: PaymentServiceHighLatency
        expr: |
          histogram_quantile(0.99,
            sum(rate(http_request_duration_seconds_bucket{service="payment-service"}[5m])) by (le)
          ) > 2.0
        for: 5m
        labels:
          severity: critical
          team: payments
        annotations:
          summary: "Payment service P99 latency {{ $value | humanizeDuration }} (SLO: < 2s)"
          runbook_url: "https://runbooks.example.com/payment-service/high-latency"

      # INFO: Pod count low — not paging, but visible in dashboards
      - alert: PaymentServiceLowPodCount
        expr: |
          kube_deployment_status_replicas_available{deployment="payment-service"} < 2
        for: 10m
        labels:
          severity: warning
          team: payments
        annotations:
          summary: "Payment service running on {{ $value }} pods (minimum: 2)"
          runbook_url: "https://runbooks.example.com/payment-service/low-pod-count"

Reducing alert fatigue: practical techniques

Group related alerts: PagerDuty and Alertmanager can group alerts that fire simultaneously. One incident, not 20 separate pages.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# Alertmanager routing: group payment alerts into one incident
route:
  group_by: [service, severity]
  group_wait: 30s      # Wait 30s for more alerts before sending
  group_interval: 5m   # Don't re-notify more than every 5m
  repeat_interval: 4h  # Don't re-send resolved alerts for 4h

  routes:
    - match:
        team: payments
        severity: critical
      receiver: payments-pagerduty
      group_by: [service]  # All payment alerts → one incident

    - match:
        severity: warning
      receiver: payments-slack

Dead man’s switch alerts: Alert when something stops happening that should happen regularly.

1
2
3
4
5
6
7
8

# Alert if no successful health check in 5 minutes
- alert: PaymentServiceHeartbeatMissing
  expr: absent(up{job="payment-service"} == 1) for 5m
  labels:
    severity: critical
  annotations:
    summary: "Payment service is not being scraped — may be down"
    runbook_url: "https://runbooks.example.com/payment-service/heartbeat-missing"

Inhibition rules: Suppress child alerts when the parent fires.

1
2
3
4
5
6
7

# Don't page for high latency if the service is completely down
inhibit_rules:
  - source_match:
      alertname: PaymentServiceDown
    target_match:
      service: payment-service
    equal: [service]

Writing Runbooks That Actually Work

A runbook is not documentation. It is a decision tree for a stressed engineer who was just woken at 3 AM and needs to resolve an incident in the next 15 minutes.

What makes a runbook fail

• Too long: Engineers skip to the fix and miss critical context
• Too vague: “Check the logs” — which logs? Where? For what?
• Outdated: Commands that no longer work, dashboards that moved, services that were renamed
• Missing rollback: What to do if the fix makes things worse
• No severity guidance: When to escalate vs keep debugging alone

Runbook template

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

# Payment Service: High Error Rate

**Alert**: `PaymentServiceHighErrorRate`
**Severity**: Critical
**Owner**: @payments-team
**Last reviewed**: 2026-03-01

---

## Quick summary
The payment service is returning 5xx errors above the 1% SLO threshold.
Common causes: database connection exhaustion, upstream Stripe API degradation,
bad deployment.

---

## Immediate triage (2 minutes)

1. **Check the dashboard** → [Payment Service Overview](https://grafana.example.com/d/payments)
   - Is error rate rising, falling, or steady?
   - Which endpoint is affected? (All endpoints = infra issue; one endpoint = code issue)

2. **Check if a deploy happened recently**
   ```bash
   kubectl rollout history deployment/payment-service -n payments

→ If yes, jump to Rollback procedure

3. Check pod status

   1	kubectl get pods -n payments -l app=payment-service

   → If CrashLoopBackOff or OOMKilled, jump to Pod issues

Diagnosis

Is the database healthy?

1
2
3
4
5
6
7
8
9

# Check DB connection pool metrics
kubectl exec -n payments deploy/payment-service -- \
  curl -s localhost:9090/metrics | grep db_pool

# Check DB pod status
kubectl get pods -n databases -l app=postgres

# Check DB logs for errors
kubectl logs -n databases postgres-0 --tail=50

→ If DB errors, see Database runbook

Is the upstream Stripe API degraded?

1
2

curl -s https://status.stripe.com/api/v2/status.json | jq '.status.indicator'
# "none" = healthy, "minor/major/critical" = degraded

→ If Stripe is degraded, enable Stripe fallback mode

Is it a specific instance?

1
2
3
4
5

# Check error rate per pod
kubectl top pods -n payments -l app=payment-service

# Check logs from the highest-error pod
kubectl logs -n payments <pod-name> --tail=100 | grep '"level":"error"'

Fixes

Rollback

If a recent deploy is the cause:

1
2
3
4

kubectl rollout undo deployment/payment-service -n payments
# Verify rollback succeeded
kubectl rollout status deployment/payment-service -n payments
# Check error rate drops within 2 minutes

Restart pods (connection pool reset)

If DB connection exhaustion is suspected:

1

kubectl rollout restart deployment/payment-service -n payments

⚠️ Warning: This causes a brief traffic blip. Confirm HPA has ≥ 3 pods before restarting.

Scale up

If pods are OOMKilled or CPU throttled:

1

kubectl scale deployment/payment-service -n payments --replicas=6

Stripe fallback mode

If Stripe is degraded, switch to queued payment processing:

1
2
3

kubectl set env deployment/payment-service PAYMENT_MODE=async -n payments
# This queues payments for processing when Stripe recovers
# Post a message in #customer-support: "Payment processing is in async mode"

Remember to revert: kubectl set env deployment/payment-service PAYMENT_MODE=sync -n payments

Escalation

After the incident

• File incident report in Linear
• Update this runbook if anything was wrong or missing
• Check if error budget policy requires a feature freeze

Related runbooks: Database Connection Exhaustion · Pod CrashLoop · Stripe API Integration

### Runbook hygiene practices

**Review quarterly**: Assign runbook ownership. Every runbook has an owner responsible for keeping it current. Stale runbooks are actively harmful.

**Test your runbooks**: On a slow day, have a junior engineer follow a runbook with no context. Where do they get stuck? That's where the runbook is broken.

**"Runbook debt" backlog**: Every time an on-call engineer improvises something not in the runbook, they add a ticket to document it afterward. This is not optional — it's part of the on-call contract.

**Automated runbook checks**: Validate links in runbooks aren't 404, commands in code blocks are syntactically valid, and dashboard links resolve.

---

## Incident Management: Running the Call

For incidents lasting more than 15 minutes or involving more than one engineer, structure the response.

### Roles

**Incident Commander (IC)**: Does not fix the problem. Coordinates people, manages communication, tracks the timeline, decides when to escalate. Keeps the channel focused.

**Technical Lead**: Owns diagnosis and remediation. Reports status to IC.

**Communications Lead** (for customer-visible incidents): Writes the status page update, communicates with customer success, keeps stakeholders informed so they're not interrupting the tech team.

**Scribe**: Takes notes in the incident doc — what was tried, what the timeline is, what hypotheses were explored. The postmortem is largely written by the scribe in real time.

### The incident Slack channel structure

#incidents-active ├── #inc-2026-03-27-payment-outage (created at incident open) │ Timeline of events, commands run, hypotheses │ IC posts regular status updates every 15 minutes │ └── #inc-2026-03-27-payment-customer (if customer-facing) External communication drafts Customer success updates

### The incident doc (live)

Create an incident doc at incident declaration. Update it in real time.

```markdown
# INC-2026-047: Payment Service High Error Rate

**Start**: 2026-03-27 03:14 UTC
**End**: [open]
**Severity**: P1
**IC**: Jane Smith
**Tech Lead**: Bob Johnson
**Status**: Active investigation

## Timeline
03:14 - PagerDuty alert: PaymentServiceHighErrorRate (5.2% error rate)
03:16 - Jane acknowledged, opened incident channel
03:18 - Bob joins, checks dashboard
03:20 - Identified: error rate started at 03:11, correlates with deployment at 03:10
03:22 - Hypothesis: deployment introduced a regression in payment validation
03:25 - Bob runs rollback: `kubectl rollout undo deployment/payment-service`
03:27 - Error rate dropping: 5.2% → 2.1% → 0.3%
03:30 - Error rate back to baseline (0.08%), incident resolved

## Root cause (preliminary)
v2.4.1 deployment introduced a validation bug that rejected valid card numbers
starting with "37" (Amex). ~12% of payment attempts failed.

## Impact
- Duration: 19 minutes
- Estimated failed payments: ~340
- Revenue impact: ~$8,500 (estimate)

## Action items
- [ ] Fix the validation bug (Bob, 2026-03-28)
- [ ] Add Amex-specific tests to payment validation suite (Bob, 2026-03-28)
- [ ] Add pre-deploy smoke test for payment types (Alice, 2026-04-05)
- [ ] Review rollback runbook — rollback took 3 minutes, should be 1 (IC, 2026-04-01)

Communication cadence during incidents

Severity P1 (customer-facing outage):
  - Status page update within 10 minutes of detection
  - Internal stakeholder update every 15 minutes
  - Customer success briefed within 20 minutes

Severity P2 (degraded, not fully down):
  - Internal update within 20 minutes
  - Status page update if > 30 minutes

Template for status page / stakeholder updates:
---
[TIME] We are investigating an issue affecting payment processing.
Customers may experience payment failures. We will provide an update in 15 minutes.

[TIME] We have identified the cause (a recent deployment) and are rolling back.
Error rate is decreasing. Next update in 10 minutes.

[TIME] Service has been restored as of 03:30 UTC. We are monitoring for
recurrence and will publish a full postmortem within 48 hours.
---

Postmortems: Learning From Failure

A postmortem is not about blame. It’s about systemic learning. The goal is to understand why the failure was possible and to prevent the class of failures — not to punish the individuals involved.

The blameless postmortem principle

People make mistakes. What matters is: why did the system allow that mistake to have this impact? If a misconfigured flag caused an outage, the right question is not “why did the engineer set the wrong flag?” It’s:

• Why did a misconfigured flag reach production without being caught?
• Why didn’t automated tests catch it?
• Why didn’t canary deployment limit the blast radius?
• Why did it take 19 minutes to detect, not 2?

Address systems and processes, not individuals.

Postmortem template

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93

# Postmortem: Payment Validation Regression — 2026-03-27

**Date**: 2026-03-28
**Authors**: Jane Smith, Bob Johnson
**Severity**: P1
**Duration**: 19 minutes (03:11 – 03:30 UTC)
**Detection**: Automated alert (PagerDuty) at 03:14 UTC (3 minutes after start)

---

## Summary
A code change in v2.4.1 introduced a regex validation bug that caused the payment
service to reject valid American Express card numbers. Approximately 340 payment
attempts failed over 19 minutes before a rollback restored service.

---

## Impact
- 340 failed payment attempts
- ~$8,500 estimated lost revenue
- 0 data loss or corruption
- 47 customer support tickets filed within 24 hours

---

## Timeline
| Time (UTC) | Event |
|---|---|
| 03:10 | v2.4.1 deployed to production |
| 03:11 | First payment failures begin (Amex cards only) |
| 03:14 | Alert fires: error rate at 5.2% |
| 03:16 | On-call acknowledges |
| 03:22 | Root cause identified: v2.4.1 deployment |
| 03:25 | Rollback initiated |
| 03:30 | Error rate back to baseline |

---

## Root Cause
The `validateCardNumber()` function was refactored in v2.4.1 to use a new regex
library. The migrated regex `^[0-9]{16}$` correctly validates Visa/Mastercard
(16 digits) but rejects Amex (15 digits, starts with 34 or 37). The original
function handled this case; the refactored version did not.

---

## Contributing Factors

**Why was this not caught in testing?**
The test suite had no test case for Amex card numbers. Card validation tests
used a single hardcoded Visa test number. The test suite passed with 100% coverage
of existing test cases while completely missing the Amex case.

**Why did this reach production?**
The CI pipeline ran unit tests but had no end-to-end payment flow test covering
all card types. The staging environment does not have a configured Amex test account.

**Why did detection take 3 minutes?**
The alert threshold is 1%, and the initial error rate took 3 minutes to cross
that threshold because Amex cards represent ~12% of traffic, not 100%.

**Why did rollback take 5 minutes instead of 1?**
The on-call engineer had to find the rollback command in the runbook and then
wait for the deployment rollout. The runbook has the command but doesn't note
the typical rollout duration (~3 minutes).

---

## Action Items

| Action | Owner | Due | Priority |
|---|---|---|---|
| Add Amex test cases to card validation suite | Bob | 2026-03-28 | High |
| Add all card type test numbers to CI test data | Bob | 2026-04-01 | High |
| Create staging Stripe Amex test account | Alice | 2026-04-05 | Medium |
| Add end-to-end payment tests covering Visa/MC/Amex/Discover | Charlie | 2026-04-15 | Medium |
| Update rollback runbook with timing expectations | Jane | 2026-04-01 | Low |
| Consider canary deployment for payment service changes | Team | 2026-04-30 | Medium |

---

## What Went Well
- Detection was fast (3 minutes) — alerting is well-configured
- Rollback decision was made quickly once cause was identified
- Incident doc was maintained in real time — postmortem was easy to write
- Communication to stakeholders was timely

---

## Lessons Learned
Test data needs to cover all card network types, not just the card number
the team happens to use internally. This is a broader principle: test data
should be systematically comprehensive, not accumulated organically.

Postmortem meeting format

Duration: 45-60 minutes

0:00 - 0:05  Review the timeline (everyone reads it beforehand)
0:05 - 0:20  "5 Whys" on each contributing factor
             Facilitator asks "why?" until you reach a systemic cause
0:20 - 0:35  Review/refine action items: specific, owned, time-bound
0:35 - 0:45  "What went well" — explicitly reinforce good behavior
0:45 - 0:50  Decide: is this a public postmortem? Who needs to receive it?
0:50 - 0:55  Team check-in — how is everyone doing?

The Human Side: Preventing Burnout

On-call stress is cumulative. Engineers who don’t process the psychological weight of repeated middle-of-the-night incidents, near-misses, and the constant low-level anxiety of being on call will eventually burn out — or leave.

Acknowledge the emotional reality

After a hard incident — especially one with customer impact or data loss — engineers often feel:

• Guilt (“I should have caught this in review”)
• Anxiety (“Is the fix actually stable?”)
• Exhaustion (adrenaline crash after the incident)
• Frustration (the underlying issue isn’t new)

Teams that pretend none of this happens don’t make it stop — they make it harder to talk about. A simple “that was a rough night, how is everyone doing?” in the postmortem goes a long way.

Concrete protections

Recovery time: If an engineer is paged between 10 PM and 6 AM and spends more than 30 minutes responding, they should not be expected to work a full day the next day. This should be written policy, not informal understanding.

Back-to-back rotation limit: No engineer should be primary on-call for more than two consecutive weeks. Enforce this in your scheduling tool.

Incident debrief: For P1 incidents, schedule a 30-minute debrief the following morning — not to investigate, but to check in with the engineers involved and ensure they have support.

Oncall load visibility: Track and publish oncall load metrics per engineer. If one person is getting 10x more pages than others, something is wrong (either the system or the rotation).

Time in lieu: For organizations without on-call compensation, be explicit about time-in-lieu for excessive on-call load. “You’ll get time back” needs to actually happen.

Rotational wellbeing check

After each on-call rotation, ask the engineer:

1. How many times were you paged? (Track this)
2. How disruptive was each page? (Scale of 1–5)
3. Did you have the runbooks/tools you needed?
4. What’s one thing we could do to make the next rotation easier?

Aggregate these answers. Trend them. Act on them.

On-Call Tooling Reference

PagerDuty / Opsgenie setup best practices

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

# PagerDuty service configuration
escalation_policy:
  - level: 1
    targets: [primary-on-call]
    escalation_delay: 5m   # 5 minutes before escalating to secondary
  - level: 2
    targets: [secondary-on-call]
    escalation_delay: 10m  # 10 more minutes before manager
  - level: 3
    targets: [engineering-manager]

# Notification rules per engineer
urgent:        # Critical severity
  - push immediately
  - SMS after 2m if not acknowledged
  - phone call after 5m if not acknowledged

non-urgent:    # Warning severity
  - push immediately
  - email after 5m
  # NO phone call, NO SMS — warning alerts should not interrupt sleep

Runbook automation with Rundeck / Ansible AWX

For runbook steps that are consistently executed (rollbacks, restarts, scaling), automate them as one-click jobs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

# Rundeck job: payment-service-rollback
name: "Payment Service Rollback"
description: "Rolls back payment-service to the previous deployment"
schedule: null  # Ad-hoc, not scheduled
sequence:
  - type: command
    exec: "kubectl rollout undo deployment/payment-service -n payments"
  - type: command
    exec: "kubectl rollout status deployment/payment-service -n payments --timeout=120s"
  - type: command
    exec: "kubectl get pods -n payments -l app=payment-service"
notifications:
  onsuccess:
    - type: slack
      webhook: "https://hooks.slack.com/..."
      message: "Payment service rollback succeeded by ${job.username}"
  onfailure:
    - type: pagerduty
      message: "Automated rollback FAILED — manual intervention required"

Status page (Statuspage.io / Cachet)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

# Automate status page updates from Alertmanager
# alertmanager-webhook → status page API

# When PaymentServiceHighErrorRate fires:
POST https://api.statuspage.io/v1/pages/{page_id}/incidents
{
  "incident": {
    "name": "Payment Processing Degraded",
    "status": "investigating",
    "body": "We are investigating an issue affecting payment processing.",
    "impact_override": "major",
    "component_ids": ["payment-service-component-id"],
    "component": {
      "status": "degraded_performance"
    }
  }
}

# When resolved:
PATCH https://api.statuspage.io/v1/pages/{page_id}/incidents/{incident_id}
{
  "incident": {
    "status": "resolved",
    "body": "This issue has been resolved. We will publish a postmortem within 48 hours.",
    "component": {
      "status": "operational"
    }
  }
}

On-Call Maturity Checklist

Use this to assess your team’s on-call program:

Rotation design

• Rotation size ≥ 4 engineers
• Written rotation rules covering holidays, swaps, and new hires
• Compensation or time-in-lieu policy documented
• Time-zone coverage gaps acknowledged and addressed

Alert hygiene

• Every alert has a runbook link
• Every alert has an owner label
• Alert severity definitions are written down and consistent
• < 5 pages per on-call week (average)
• Alert noise review every quarter
• Inhibition rules prevent alert storms

Runbooks

• Every service with a paging alert has a runbook
• Every runbook has a “quick summary” (triage in < 2 minutes)
• Every runbook has explicit escalation triggers
• Runbooks are reviewed quarterly
• Runbooks are tested (junior engineer can follow without help)

Incident management

• IC role defined and practiced
• Incident doc template exists and is used
• Status page updates happen within 10 minutes for P1s
• Communication cadence defined for each severity

Postmortems

• Every P1 gets a postmortem within 48 hours
• P2s get postmortems when they’re recurring or interesting
• Postmortems are blameless (focus on systems, not individuals)
• Action items from postmortems are tracked to completion
• Postmortems are shared with the broader team

Wellbeing

• Recovery time policy exists and is enforced
• Back-to-back rotation limit enforced (≤ 2 weeks)
• Post-incident check-ins happen after hard incidents
• On-call load per engineer is tracked and published

Summary

Sustainable on-call comes down to three things working together:

1. Systems that fail rarely and recover automatically — every page is a sign that something isn’t self-healing enough. Fix the system, not just the symptom.

2. Processes that make response fast and low-stress — clear rotation rules, well-designed alerts, runbooks that work at 3 AM, incident management that distributes the cognitive load.

3. Culture that treats engineers as humans — recovery time after hard incidents, blameless postmortems, visible on-call load, explicit compensation, and leaders who model the behavior they want to see.

The teams that get this right don’t have easier systems — they have systems that have been through more incidents and gotten better each time. Every postmortem action item that closes a gap, every runbook that gets updated after an incident, every alert that gets tuned — it compounds. The best on-call programs aren’t built; they’re grown through a disciplined response to every failure.