Incident Response Playbook

An incident will happen. A database goes down, a deployment introduces a critical bug, an attacker gains access to a production system, a cloud region fails. The question isn’t whether you’ll face incidents — it’s whether you’ll face them with a plan or without one.

Teams that handle incidents well share a common trait: they’ve thought through the process before the incident. They have documented roles, communication channels, escalation paths, and runbooks. When everything is on fire at 2 AM, the last thing you want to be doing is figuring out the process.

This playbook covers the full incident lifecycle — detection through postmortem — with concrete templates, communication scripts, and the practical habits that separate teams that recover quickly from those that spiral.

Incident Response Phases

Every incident follows the same arc, regardless of cause:

Detection → Triage → Containment → Eradication → Recovery → Postmortem

Each phase has a clear goal, a set of actions, and a handoff point to the next phase. Rushing a phase — jumping from detection to recovery without containment — leads to recurring incidents and incomplete fixes.

Phase 1: Detection

Detection is recognising that something is wrong. Detection sources fall into three categories:

Automated alerts — your monitoring stack tells you before users do. Alertmanager, PagerDuty, Grafana on-call, Opsgenie. This is where you want to catch incidents.

User reports — customers notice before your monitoring does. This is a signal that your observability coverage has gaps.

Internal discovery — an engineer notices something anomalous while doing unrelated work.

Building Detection That Works

The goal is high signal, low noise. An alert that fires constantly gets ignored. An alert that never fires misses incidents. The balance:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

# Prometheus alerting rules — examples of well-calibrated alerts
groups:
  - name: availability
    rules:
      # Fire only after 5 minutes of elevated error rate — reduces flaps
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
          / sum(rate(http_requests_total[5m])) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Error rate above 5% for 5 minutes"
          description: "{{ $value | humanizePercentage }} of requests are failing"
          runbook: "https://wiki.internal/runbooks/high-error-rate"

      # Latency SLO breach
      - alert: P99LatencyHigh
        expr: |
          histogram_quantile(0.99,
            sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
          ) > 2.0
        for: 3m
        labels:
          severity: warning
        annotations:
          summary: "P99 latency above 2s for service {{ $labels.service }}"
          runbook: "https://wiki.internal/runbooks/latency"

      # Dead man's switch — alerts if the alerting pipeline itself fails
      - alert: WatchdogSilent
        expr: vector(1)
        labels:
          severity: info
        annotations:
          summary: "This alert always fires — silence means alerting is broken"

Every alert should have a runbook annotation pointing to documented response steps. If you can’t write a runbook for an alert, the alert probably shouldn’t exist.

On-Call Rotation Setup

Rotation structure:
- Primary on-call: first responder, owns the incident until handed off
- Secondary on-call: escalation if primary is unavailable
- Escalation path: team lead → engineering manager → CTO (for major incidents)

On-call expectations:
- Respond to pages within 5 minutes during business hours
- Respond within 15 minutes outside business hours
- Acknowledge or escalate within the SLA — never let a page go unanswered
- Hand off context explicitly when rotating

Incident severity levels establish response expectations:

Phase 2: Triage

Triage answers: What’s broken, how bad is it, and who needs to know right now?

The first 5 minutes of an incident set the tone. Resist the urge to immediately start fixing. Take 2 minutes to understand the situation before touching anything.

The First Five Minutes

□ Acknowledge the alert (prevents duplicate pages)
□ Open the incident channel: #incident-YYYY-MM-DD-brief-description
□ Declare your role: "I'm taking incident command"
□ Post initial assessment:
    - What is broken?
    - What is working?
    - How many users are affected?
    - When did it start?
    - What changed recently? (last deploy, config change, infra change)
□ Set severity
□ Page additional responders if SEV-1 or SEV-2

Incident Commander Role

For anything SEV-2 and above, designate an Incident Commander (IC). The IC does not do hands-on debugging. The IC:

• Maintains situational awareness across all responders
• Assigns tasks explicitly (“Alice, investigate the database connection pool; Bob, check recent deployments”)
• Drives communication — internal updates every 15 minutes, customer-facing updates as needed
• Keeps the timeline
• Decides when to escalate
• Calls the all-clear

Without an IC, incidents devolve into everyone debugging in parallel with no coordination, duplicate efforts, and no one watching the big picture.

Triage Checklist

□ Check the timeline: when did this start?
□ Check recent changes:
    git log --since="2 hours ago"  # recent commits
    kubectl rollout history deployment/myapp  # recent K8s deployments
    terraform show  # recent infra changes
    Check change management log / deploy log

□ Check monitoring dashboards:
    - Error rate trend (when did it spike?)
    - Latency trend
    - Traffic volume (DDoS? traffic spike?)
    - Infrastructure metrics (CPU, memory, disk, network)

□ Check dependencies:
    - Database connectivity and replication lag
    - External API status pages
    - Cloud provider status (status.aws.amazon.com, etc.)

□ Scope the impact:
    - How many users / requests affected?
    - Which specific features / endpoints?
    - Which regions or environments?
    - Any data integrity concerns?

□ Assign severity based on answers above
□ Post initial status update to incident channel
□ Page additional responders if needed

Phase 3: Containment

Containment limits the blast radius. The goal is to stop the damage from spreading, even if you haven’t fixed the root cause yet.

Containment is not fixing. You’re buying time to understand the problem properly.

Containment Strategies

Rollback the last change. This is almost always the first thing to try when a deployment caused the incident.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# Kubernetes — rollback to previous deployment
kubectl rollout undo deployment/myapp
kubectl rollout status deployment/myapp

# Helm rollback
helm rollback myapp 0   # 0 = previous revision

# GitHub Actions — re-run previous successful workflow
# Or revert the commit and deploy:
git revert HEAD --no-edit
git push

# Terraform — if infra change caused it
terraform plan  # verify what a rollback would do
# If safe:
git revert HEAD
terraform apply

Feature flags. Turn off the specific feature causing the problem without a full rollback:

1
2
3
4

# LaunchDarkly / Unleash — disable a flag via API
curl -X PATCH https://app.launchdarkly.com/api/v2/flags/default/risky-feature \
  -H "Authorization: $LD_API_KEY" \
  -d '[{"op": "replace", "path": "/environments/production/on", "value": false}]'

Traffic management. Redirect away from the broken component:

1
2
3
4
5
6
7
8
9

# Nginx / load balancer — remove a broken backend temporarily
# Mark server down in upstream
nginx -s reload

# Kubernetes — scale to zero and let other replicas handle load
kubectl scale deployment/myapp --replicas=0
# Or cordon a broken node
kubectl cordon node-3
kubectl drain node-3 --ignore-daemonsets --delete-emptydir-data

Database containment. If a query or migration is causing table locks:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

-- Find and kill the offending query
SELECT pid, query, query_start, state
FROM pg_stat_activity
WHERE state != 'idle'
ORDER BY query_start;

SELECT pg_cancel_backend(pid);   -- gentle: cancel the query
SELECT pg_terminate_backend(pid);  -- forceful: kill the connection

-- Check for locks
SELECT * FROM pg_locks pl
JOIN pg_stat_activity pa ON pl.pid = pa.pid
WHERE NOT granted;

Security containment. If a breach is suspected:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

# Isolate a compromised instance — revoke cloud credentials immediately
aws iam delete-access-key --user-name compromised-user --access-key-id AKIAIOSFODNN7

# Rotate secrets
vault lease revoke -prefix aws/creds/
kubectl delete secret compromised-secret

# Block network access (if not needed for investigation)
# Cloud security group: remove all inbound rules
# Or quarantine the instance to a restricted VLAN

# Preserve evidence BEFORE making changes
# Capture memory: sudo avml /evidence/memory.lime
# Capture disk: sudo dd if=/dev/sda | gzip > /evidence/disk.img.gz
# Export logs: kubectl logs pod/compromised-pod --all-containers > /evidence/pod-logs.txt
# Snapshot the cloud instance before termination

Phase 4: Eradication

Eradication removes the root cause. This is the investigation phase — understand why the incident happened before declaring victory.

Root Cause Investigation

Use the 5 Whys technique: ask “why” repeatedly until you reach a systemic cause, not just a symptom.

Symptom: API is returning 500 errors
Why? Database connection pool is exhausted
Why? Query latency increased 10x in the last hour
Why? A new index was dropped during a migration
Why? The migration script didn't check if the index existed before dropping it
Why? There was no review process for database migrations affecting production
Root cause: No migration review process → missed dependency
Fix: Restore the index; add migration review checklist to deployment process

The root cause is rarely “someone made a mistake.” It’s usually a gap in a system: missing tests, missing review, missing monitoring, missing documentation.

Investigation Commands

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

# Timeline reconstruction
journalctl --since "2026-03-26 02:00:00" --until "2026-03-26 03:30:00" -u myapp

# Correlate logs across services
kubectl logs deployment/myapp --since=2h | grep -E "ERROR|WARN|panic"

# Check what changed around the incident start time
git log --all --since="3 hours ago" --oneline
kubectl rollout history deployment/myapp
kubectl get events --sort-by='.lastTimestamp' -A | tail -50

# Trace a specific request through distributed systems
# (use trace ID from error logs)
# Query your APM: Jaeger, Tempo, Honeycomb

# Database — slow query log
-- PostgreSQL: queries that ran during the incident window
SELECT query, calls, mean_exec_time, max_exec_time, stddev_exec_time
FROM pg_stat_statements
WHERE mean_exec_time > 1000  -- ms
ORDER BY mean_exec_time DESC;

# Check for resource exhaustion
sar -u -r -d -n DEV 1 -f /var/log/sa/saYYMMDD  # historical resource usage
dmesg | grep -E "OOM|killed|oom"  # OOM kills

Eradication Checklist

□ Root cause identified and documented
□ All affected systems identified (not just the one that alerted)
□ Fix implemented and tested in staging
□ Fix reviewed by a second person
□ Deployment plan for the fix confirmed with IC
□ Rollback plan for the fix documented
□ Any stolen credentials rotated / revoked
□ Any compromised data identified and customers notified if required

Phase 5: Recovery

Recovery restores normal service. This is different from containment — you’re not just stopping the bleeding, you’re healing.

Recovery Checklist

□ Deploy the fix (with rollback plan ready)
□ Monitor key metrics for 30 minutes post-fix:
    - Error rate returning to baseline
    - Latency returning to baseline
    - No new alerts firing
□ Verify end-to-end functionality manually
□ Check data integrity if the incident could have affected stored data
□ Restore any disabled features / traffic routing changes
□ Verify backups are intact and recent
□ Remove any temporary containment measures
□ Send all-clear communication
□ Update status page

All-Clear Communication Template

[RESOLVED] Incident: API 500 Errors
Duration: 02:14 - 03:47 UTC (1 hour 33 minutes)
Impact: ~18% of API requests failed with 500 errors

Root cause: A database index was inadvertently dropped during the v2.14.1
deployment migration, causing query latency to increase 10x and exhausting
the connection pool.

Resolution: Index restored; application redeployed. All metrics have
returned to baseline.

We will be publishing a full postmortem within 48 hours.

— Engineering Team

Phase 6: Postmortem

The postmortem is the most valuable phase that teams most often skip or rush. A well-run postmortem turns a painful incident into permanent improvement. A badly run postmortem (or none at all) means the same incident happens again.

Blameless Culture

The foundation of effective postmortems is psychological safety. If engineers fear being blamed, they’ll be guarded, leave out details, and the postmortem will produce surface-level findings. The goal is to understand the system, not to identify a culpable human.

“Blameless” doesn’t mean “no accountability.” It means:

• We assume everyone acted rationally given the information they had at the time
• We focus on what went wrong in the system, not who did something wrong
• Individuals can make mistakes; postmortems should fix the conditions that allow mistakes to have large impact

Postmortem Template

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82

# Postmortem: [Brief Title]
**Date:** 2026-03-26
**Author:** @alice
**Reviewers:** @bob, @carol
**Status:** Draft / In Review / Final

## Summary

One paragraph: what happened, how long it lasted, and what the impact was.

Example: On 2026-03-26 at 02:14 UTC, a database index was dropped during
the v2.14.1 deployment. This caused API error rates to reach 18% for
1 hour 33 minutes, affecting approximately 4,200 users.

## Impact

- **Duration:** 02:14 UTC – 03:47 UTC (1h 33m)
- **User impact:** ~4,200 users; 18% of API requests failed with HTTP 500
- **Revenue impact:** $X,XXX estimated (if applicable)
- **Data integrity:** No data loss
- **SLO burn:** 0.8% of monthly error budget consumed

## Timeline

All times UTC.

| Time  | Event |
|-------|-------|
| 01:58 | v2.14.1 deployment initiated |
| 02:03 | Migration script ran, dropped index unintentionally |
| 02:14 | Error rate spike detected by Alertmanager |
| 02:16 | PagerDuty paged @alice (primary on-call) |
| 02:19 | @alice acknowledged, opened #incident-2026-03-26-api-errors |
| 02:22 | @alice declared SEV-2, paged @bob for investigation support |
| 02:35 | Determined recent deployment as likely cause |
| 02:41 | Rollback attempted; did not resolve (schema change can't be rolled back) |
| 02:50 | Root cause identified: missing index on orders.user_id |
| 03:12 | Fix prepared and reviewed |
| 03:31 | Fix deployed to production |
| 03:47 | Error rate returned to baseline; incident declared resolved |
| 04:00 | Customer communication sent |

## Root Cause

The v2.14.1 migration script (0042_cleanup_old_indexes.sql) dropped the
`idx_orders_user_id` index under the assumption it was unused. A recent
query was added in v2.13.0 that depended on this index, but no integration
test covered the query's performance under load. The migration had no
review step checking index usage statistics before dropping.

## What Went Well

- Alert fired within 1 minute of the first errors
- On-call response was prompt (3 minutes to acknowledge)
- Incident channel was opened and responders were coordinated quickly
- Customer communication was clear and timely

## What Went Poorly

- Rollback did not resolve the incident because the schema change was not
  reversible by re-deploying the previous application version
- Time from "deployment as likely cause" to "root cause identified" was
  15 minutes — we should have checked `pg_stat_user_indexes` sooner
- No staging environment reproduced the issue because staging has a much
  smaller dataset, so the missing index didn't cause a visible slowdown

## Action Items

| Action | Owner | Due Date | Priority |
|--------|-------|----------|----------|
| Add pg_stat_user_indexes check to migration review checklist | @alice | 2026-04-02 | High |
| Add query performance test to CI (test with realistic data volume) | @bob | 2026-04-09 | High |
| Document database rollback procedures in runbook | @carol | 2026-04-05 | Medium |
| Add index-drop warning to migration linter | @dave | 2026-04-16 | Medium |
| Seed staging database with production-scale anonymised data | @alice | 2026-04-30 | Low |

## Lessons Learned

The system had no guard against dropping indexes that were in active use.
The missing check was not a human failure — the engineer who wrote the
migration followed the existing process correctly. The process itself
had no step to verify index usage before removal.

Running the Postmortem Meeting

Hold the postmortem meeting within 48–72 hours while details are fresh:

Meeting structure (60 minutes):

0:00 – 0:05  Ground rules: blameless, focus on systems, no interrupting
0:05 – 0:20  Walk through the timeline together
             Each person adds context from their perspective
             "What were you seeing at 02:35?"

0:20 – 0:35  Root cause discussion
             Apply 5 Whys to reach systemic causes
             Avoid stopping at "human error"

0:35 – 0:50  Action item brainstorming
             What would have prevented this?
             What would have detected it sooner?
             What would have resolved it faster?
             Prioritise ruthlessly — 3 high-quality actions beat 15 mediocre ones

0:50 – 0:60  Assign owners and due dates
             Owner = single person accountable for completion
             No unassigned action items leave the room

Runbooks: Decision Trees for 2 AM

A runbook is a documented procedure for responding to a specific alert or scenario. Good runbooks make experienced engineers faster and make on-call rotatable to less experienced engineers.

Runbook Template

1
2
3
4
5
6
7
8

# Runbook: High Error Rate (HTTP 5xx)

**Alert:** `HighErrorRate`
**Severity:** SEV-2 default; upgrade to SEV-1 if > 50% of requests fail

## Quick Diagnosis

### Step 1: Check which service is failing

kubectl get pods -A | grep -v Running kubectl top pods -A

**If no pods are crashing** → go to Step 2

### Step 2: Check recent deployments

kubectl rollout history deployment/myapp git log –since=“1 hour ago” –oneline

**If a recent deployment** → consider rollback (Step 3)
**If no recent deployment** → go to Step 4

### Step 3: Rollback

kubectl rollout undo deployment/myapp kubectl rollout status deployment/myapp –timeout=5m

Monitor error rate for 5 minutes. If resolved → open postmortem and close incident.
If not resolved → continue to Step 4.

### Step 4: Check database connectivity

kubectl exec -it deployment/myapp –
psql $DATABASE_URL -c “SELECT 1”

**If connection fails** → check database runbook [link]

### Step 5: Check downstream dependencies
- Stripe: https://status.stripe.com
- SendGrid: https://status.sendgrid.com
- AWS us-east-1: https://status.aws.amazon.com

**If a dependency is down** → post status update, monitor, nothing to fix on our end.

### Step 6: Escalate
If none of the above resolved the issue, page the secondary on-call and
escalate to SEV-1. Post in #incident channel:
"Unable to identify root cause after 30 minutes. Escalating to SEV-1.
Paging @secondary-oncall."

## Useful Commands

```bash
# Recent application logs
kubectl logs deployment/myapp --since=30m | grep -E "ERROR|panic|fatal"

# Current connection pool status
kubectl exec -it deployment/myapp -- curl localhost:8080/debug/db

# Database slow queries
kubectl exec -it deployment/postgres -- psql -U postgres -c \
  "SELECT query, calls, mean_exec_time FROM pg_stat_statements
   ORDER BY mean_exec_time DESC LIMIT 10;"

Contacts

• Database team: @db-team (Slack) / db-oncall@example.com
• Infrastructure: @infra-team (Slack) / infra-oncall@example.com
• Security (if breach suspected): security@example.com / +1-555-0100

---

## Communication Templates

### Internal Status Update (every 15 minutes during active incident)

[UPDATE 03:15 UTC — SEV-2 — API Error Rate]

Status: INVESTIGATING

Current situation: Error rate is at 18%, unchanged since 02:14. We have identified the recent deployment as a likely cause and are investigating the specific change.

Next update: 03:30 UTC or sooner if status changes.

IC: @alice | Investigators: @bob

### External Status Page Update

Investigating — API Errors We are investigating an increase in API error rates. Our team has been engaged and is actively working to resolve this. We will provide an update in 30 minutes.

Posted: 2026-03-26 02:30 UTC

Identified — API Errors We have identified the cause of the API error rate increase and are deploying a fix. We expect resolution within 20 minutes.

Posted: 2026-03-26 03:15 UTC

Resolved — API Errors The issue causing elevated API error rates has been resolved. All systems are operating normally. A postmortem will be published within 48 hours.

Duration: 02:14 UTC – 03:47 UTC Impact: ~18% of API requests

Posted: 2026-03-26 03:55 UTC

---

## Metrics to Track Over Time

A mature incident response program measures itself:

| Metric | Target | How to Measure |
|--------|--------|---------------|
| Mean Time to Detect (MTTD) | < 5 minutes | Alert timestamp – incident start timestamp |
| Mean Time to Acknowledge (MTTA) | < 5 min business hours, < 15 min off-hours | Ack timestamp – page timestamp |
| Mean Time to Resolve (MTTR) | Track trend, improve quarter-over-quarter | Resolution timestamp – detection timestamp |
| Incident frequency by severity | Trend down over time | Count per week/month |
| Action item completion rate | > 80% completed on time | Track in project management tool |
| Postmortem publication rate | 100% of SEV-1/2 | Count published / count incidents |

Review these metrics monthly. Rising MTTD means your alerting has gaps. Rising MTTR means your runbooks or tooling need work. Low action item completion means your follow-through process is broken.

---

## Practical Starting Point

If your team has no incident response process today, start here — not with the full playbook:

1. **Create a dedicated incident Slack channel** (`#incidents`). When something goes wrong, everyone goes there.

2. **Write three runbooks** for your three most common alerts. Even a basic decision tree is infinitely better than nothing.

3. **Run a game day.** Deliberately break something in a staging environment and practise the response. Find the gaps before a real incident does.

4. **Hold a postmortem for your next incident**, even if it's just a 30-minute conversation. Write one page. Assign one action item. That habit compounds.

The difference between teams that handle incidents gracefully and those that scramble isn't experience — it's preparation. A written process, however simple, forces clarity. Clarity under pressure is what makes incidents survivable.