Alerting That Doesn't Burn You Out: Fighting Alert Fatigue and Building Sane On-Call

March 25, 2026 36 min read DevOps

alertingmonitoringon-callsredevopsprometheuspagerdutyobservabilityoperations

Contents

It’s 2:47 AM. Your phone screams. You grab it, squint at the screen: HighMemoryUsage on prod-web-03. You silence it and go back to sleep. An hour later: HighMemoryUsage on prod-web-04. Silence. Then at 4:15 AM: CheckoutServiceDown. You silence that one too — and miss the actual incident that’s been bleeding revenue since 4:11.

That sequence is alert fatigue. And it’s not a hypothetical. It’s happening on your team right now if your pager fires more than a few times per night and engineers are treating it like a snooze button.

This post is about fixing that. We’ll cover what alert fatigue actually costs, how to design alerts that mean something, how to configure Alertmanager to route intelligently, how to write runbooks that survive contact with a half-awake engineer at 3 AM, and how to build an on-call rotation that doesn’t chew through your best people.

The Alert Fatigue Problem

Alert fatigue is the numbing effect of too many alerts. When the pager goes off dozens of times a day — and most of those pages either resolve on their own, require no action, or have no clear owner — engineers stop taking them seriously. The first instinct shifts from “what’s wrong?” to “how do I make this stop?”

The research is clear: teams receiving more than 100 alerts per day show significantly lower response rates and slower mean time to acknowledge (MTTA). More alerts do not produce more vigilance. They produce less.

The Boy Who Cried Wolf

Here’s the feedback loop that kills on-call teams:

Someone adds an alert that fires on minor fluctuations
The alert fires frequently; engineers investigate and find nothing actionable
Engineers learn that this alert is noise; they start silencing it without looking
The underlying metric that triggered the alert eventually indicates a real problem
Nobody notices because the alert’s credibility was exhausted weeks ago

Once your team has internalized that “most alerts are probably nothing,” you’ve lost. The alert that matters will look identical to the 40 alerts before it that weren’t.

What Alert Fatigue Actually Costs

The immediate cost is obvious: a real incident gets missed or delayed because the signal drowned in noise. But the longer-term costs are worse:

Engineer burnout and attrition. Sleep deprivation is cumulative. An on-call rotation that wakes someone up three or four times a night, week after week, is a retention crisis waiting to happen. Your most senior engineers — who are hardest to replace — are often the most embedded in on-call rotations and the first to leave when on-call becomes unsustainable.

Alert blindness. When every alert fires, no alert feels important. Engineers develop a mental model of “the pager is just background noise” that’s very hard to undo even after you fix the underlying alerting.

Silencing without investigation. When silencing becomes a reflex, real problems get silenced too. The engineer isn’t negligent — they’re responding rationally to a broken system.

Symptoms Your Alerting Is Broken

You don’t need to run a formal study to know your alerting is in trouble. Look for these signs:

On-call engineers dread the rotation and visibly exhaust after their shift
The same alerts fire repeatedly, week after week, without anyone fixing the underlying issue
When an alert fires, the first question is “is this real?” rather than “what do I do?”
Silence is the primary response to a significant fraction of pages
Nobody knows what some alerts mean — the original author has long since moved on
You have alerts that have been firing continuously for weeks with no resolution

If you recognize three or more of these, you have a systemic problem that a few tuned thresholds won’t fix.

The Philosophy of Good Alerting

Before touching configuration, align your team on first principles. Bad alerting usually isn’t a technical problem — it’s a failure to agree on what an alert is for.

Alert on Symptoms, Not Causes

The single most impactful shift in alerting philosophy is moving from cause-based to symptom-based alerting.

Cause-based alert: CPU usage > 80% Symptom-based alert: HTTP error rate > 5% on checkout service

The CPU alert might fire during a batch job, a one-time data migration, or a viral traffic spike — none of which require waking someone up. The error rate alert tells you something users are actually experiencing right now. That’s worth a page.

Ask this about every alert: “If this fires, is something bad happening to a user?” If the answer is “not necessarily,” you’re alerting on a cause, and you should either convert it to a symptom-based alert or demote it to a dashboard metric.

Every Alert Must Be Actionable

If there’s no defined action to take when an alert fires, it’s not an alert — it’s a metric. Put it on a dashboard, trend it over time, review it in your weekly meeting. Don’t page anyone about it.

Every alert should have a clear answer to: “What does the on-call engineer do right now?”

Pages Should Be Rare and Urgent

A page wakes someone up. It interrupts dinner. It pulls someone out of a meeting. It’s a significant intrusion on someone’s life. Reserve it for user-impacting incidents that require immediate human action.

If your team is averaging more than two or three pages per on-call shift, most of those pages are probably not page-worthy. They should be tickets, warnings, or nothing at all.

The Four Severity Levels

Establish clear, shared definitions for severity. Here’s a framework that works well:

Severity	When to Use	Response
Page (P1)	Users are impacted right now, or will be within minutes	Wake someone up immediately, any hour
Ticket (P2)	Important issue, but not user-impacting yet; degraded performance, SLO at risk	Create a ticket, fix during business hours
Warning	Trending in a bad direction; worth awareness but no immediate action needed	Dashboard, weekly review
Info/Debug	Not an alert; just data	Dashboard metric only

The key distinction between P1 and P2 isn’t urgency to you — it’s urgency to users. An OOM kill that took down one of five redundant pods is a P2 (degraded but not down). An OOM kill that took down your only checkout pod is a P1.

The Signal-to-Noise Test

Apply this heuristic to every alert: if an engineer’s first instinct upon seeing it fire is to silence it or dismiss it, that alert is noise. It might contain real signal buried under unreliable triggering, but the engineer has already learned to ignore it — which means the signal is lost regardless.

Auditing Your Existing Alerts

You can’t fix what you can’t see. Before designing new alerts, audit what you already have.

Pulling Alert History from Prometheus

Query your Prometheus instance for currently firing alerts:

1

ALERTS{alertstate="firing"}

For a historical view, use the Alertmanager API to export alert history, or query your long-term storage (Thanos, Cortex, Mimir) if you have it:

1
2
3
4
5


# List all currently firing alerts via Alertmanager API
curl -s http://localhost:9093/api/v2/alerts | jq '.[] | {alertname: .labels.alertname, severity: .labels.severity, startsAt: .startsAt}'

# Export alerts from the last 30 days (if using Alertmanager with persistent storage)
curl -s "http://localhost:9093/api/v2/alerts?active=true&silenced=false&inhibited=false" | jq '.'

The Actionability Audit

For each alert in your system, answer these four questions:

What does this alert mean in plain English?
What does the on-call engineer do when it fires?
How often has it fired in the last 30 days?
How many of those firings required actual action?

If you can’t answer questions 1 or 2 without researching, that’s an “unknown alert” — it needs documentation or deletion. If the answer to question 4 is “almost none,” it’s noise and needs to be fixed or removed.

Categories of Broken Alerts

Flapping alerts fire and resolve repeatedly in short cycles. They usually indicate a threshold set too close to normal operating range. The fix is either raising the threshold, adding a longer evaluation window, or using a for: clause in Prometheus to require the condition to persist before firing.

Always-on alerts have been firing continuously for weeks or months. Nobody fixed the underlying issue. These are often the most dangerous — engineers have learned to ignore them, so if the condition worsens, nobody notices.

Unknown alerts were written by someone who has since left, or were copied from an online template without adaptation. Nobody on the current team knows what they mean.

Duplicate alerts represent the same underlying problem three different ways. When the problem occurs, all three fire simultaneously, tripling the noise and confusing the on-call engineer about which one to act on.

The One-Month Silence Experiment

For a controlled audit: for one month, when an alert fires and the on-call engineer determines no action is needed, require them to tag it (in PagerDuty, a comment, a spreadsheet — whatever you have) as “no action required.” At the end of the month, sort by frequency of “no action required” firings. Those are your noisiest, least actionable alerts. Fix or delete them first.

Designing Actionable Alerts

The Three-Question Test

Before adding any alert, run it through this filter:

Is this user-impacting, or will it be imminently? If no, it’s not a page.
Does the engineer know exactly what to do? If no, write the runbook before adding the alert.
Does this need attention right now (vs. during business hours)? If no, it’s a ticket or warning, not a page.

All three must be “yes” for a P1 page. At least the first two must be “yes” for any alert at all.

USE and RED Methods

Two complementary frameworks help you decide what to alert on:

USE (for infrastructure resources):

Utilization — what percentage of the resource is being used?
Saturation — is work queuing up because the resource is at capacity?
Errors — is the resource reporting errors?

RED (for services/APIs):

Rate — how many requests per second?
Errors — what fraction of requests are failing?
Duration — how long are requests taking?

For databases, use both: USE for the database host (CPU, disk, connections), RED for the query layer (query rate, error rate, latency).

SLO-Based Alerting

The most sophisticated and user-centric approach is to alert on your Service Level Objective (SLO) burn rate rather than on raw metrics. This directly ties your alerting to user experience commitments.

The core idea: if you have a 99.9% availability SLO (allowing 43.8 minutes of downtime per month), you can calculate how fast you’re “burning” your error budget. An error rate that would exhaust the entire month’s budget in one hour is an emergency. An error rate that would exhaust it in two weeks is something to fix during business hours.

Multi-window, multi-burn-rate alerting uses two windows to catch both fast burns (acute incidents) and slow burns (gradual degradation):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40


# Fast burn: 5% of monthly error budget consumed in 1 hour
# (14.4x the normal burn rate — this is a page)
#
# Slow burn: 10% of monthly error budget consumed in 6 hours
# (5x the normal burn rate — this is a ticket)

groups:
  - name: slo_alerts
    rules:
      # Fast burn — P1 page
      - alert: ErrorBudgetBurnRateFast
        expr: |
          (
            rate(http_requests_total{status=~"5.."}[1h])
            /
            rate(http_requests_total[1h])
          ) > (14.4 * 0.001)
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Fast error budget burn rate on {{ $labels.service }}"
          description: "Error rate is {{ $value | humanizePercentage }}, burning error budget 14x faster than sustainable. Will exhaust monthly budget in ~1 hour."
          runbook_url: "https://wiki.example.com/runbooks/error-budget-burn"

      # Slow burn — P2 ticket
      - alert: ErrorBudgetBurnRateSlow
        expr: |
          (
            rate(http_requests_total{status=~"5.."}[6h])
            /
            rate(http_requests_total[6h])
          ) > (5 * 0.001)
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Slow error budget burn rate on {{ $labels.service }}"
          description: "Error rate is {{ $value | humanizePercentage }}, burning error budget 5x faster than sustainable over the last 6 hours."
          runbook_url: "https://wiki.example.com/runbooks/error-budget-burn"

Symptom-Based Prometheus Alert Examples

Here are production-ready alert rules that focus on symptoms rather than causes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82


groups:
  - name: service_health
    rules:

      # High error rate — users are getting errors
      - alert: HighErrorRate
        expr: |
          (
            rate(http_requests_total{status=~"5.."}[5m])
            /
            rate(http_requests_total[5m])
          ) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High HTTP error rate on {{ $labels.job }}"
          description: "Error rate is {{ $value | humanizePercentage }} over the last 5 minutes (threshold: 5%)."
          runbook_url: "https://wiki.example.com/runbooks/high-error-rate"

      # High latency — users are waiting too long
      - alert: HighLatencyP99
        expr: |
          histogram_quantile(0.99,
            rate(http_request_duration_seconds_bucket[5m])
          ) > 2
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "P99 latency above 2s on {{ $labels.job }}"
          description: "P99 latency is {{ $value | humanizeDuration }} over the last 5 minutes."
          runbook_url: "https://wiki.example.com/runbooks/high-latency"

      # Service down — Prometheus can't scrape the target
      - alert: ServiceDown
        expr: up == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Service {{ $labels.job }} is down"
          description: "Prometheus has been unable to scrape {{ $labels.instance }} for 2 minutes."
          runbook_url: "https://wiki.example.com/runbooks/service-down"

      # Disk predicted to fill — catch it before it's an emergency
      - alert: DiskWillFillIn24Hours
        expr: |
          predict_linear(node_filesystem_avail_bytes{fstype!="tmpfs"}[6h], 24*3600) < 0
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "Disk on {{ $labels.instance }} predicted full in 24h"
          description: "Filesystem {{ $labels.mountpoint }} on {{ $labels.instance }} is predicted to run out of space within 24 hours based on the last 6 hours of growth."
          runbook_url: "https://wiki.example.com/runbooks/disk-predicted-full"

      # Pod crash-looping — Kubernetes workload is unstable
      - alert: PodCrashLooping
        expr: |
          rate(kube_pod_container_status_restarts_total[15m]) > 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} is crash-looping"
          description: "Container {{ $labels.container }} has restarted {{ $value | humanize }} times in the last 15 minutes."
          runbook_url: "https://wiki.example.com/runbooks/pod-crash-looping"

      # High memory pressure — system may OOM soon
      - alert: MemoryPressureHigh
        expr: |
          (
            node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes
          ) / node_memory_MemTotal_bytes > 0.90
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Memory usage above 90% on {{ $labels.instance }}"
          description: "Memory utilization is {{ $value | humanizePercentage }}. OOM risk is elevated."
          runbook_url: "https://wiki.example.com/runbooks/high-memory"

Notice what these alerts have in common: a for: clause (to filter out spikes), a runbook_url annotation, and a description that tells the engineer both the current value and the threshold. An alert that fires and immediately answers “what’s happening and how bad is it?” costs far less cognitive overhead than one that just says FIRING: HighLatencyP99.

Prometheus Alertmanager Configuration

Alertmanager handles routing, deduplication, grouping, and silencing of alerts. A well-configured Alertmanager dramatically reduces notification noise even when alert rules are imperfect.

Routing Alerts to the Right Destination

The goal: critical alerts page the on-call engineer, warnings go to a Slack channel, and informational alerts go somewhere engineers can review without being interrupted.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84


# alertmanager.yml
global:
  smtp_smarthost: 'smtp.example.com:587'
  smtp_from: 'alertmanager@example.com'
  resolve_timeout: 5m

route:
  # Default receiver for anything not matched below
  receiver: 'slack-warnings'
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s        # Wait 30s before sending first notification (allows grouping)
  group_interval: 5m     # Wait 5m before sending new notifications for same group
  repeat_interval: 4h    # Re-notify after 4h if still firing

  routes:
    # P1/Critical → PagerDuty (wakes people up)
    - match:
        severity: critical
      receiver: 'pagerduty-critical'
      group_wait: 10s
      repeat_interval: 1h

    # Warning → Slack #alerts (visible but not interruptive)
    - match:
        severity: warning
      receiver: 'slack-warnings'
      group_wait: 1m
      repeat_interval: 6h

    # Info → Slack #monitoring-noise (or a dead-letter receiver)
    - match:
        severity: info
      receiver: 'slack-noise'
      group_wait: 5m
      repeat_interval: 24h

receivers:
  - name: 'pagerduty-critical'
    pagerduty_configs:
      - service_key: '<YOUR_PAGERDUTY_INTEGRATION_KEY>'
        description: '{{ template "pagerduty.default.description" . }}'
        details:
          firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
          runbook: '{{ (index .Alerts 0).Annotations.runbook_url }}'

  - name: 'slack-warnings'
    slack_configs:
      - api_url: '<YOUR_SLACK_WEBHOOK_URL>'
        channel: '#alerts'
        send_resolved: true
        title: '{{ template "slack.default.title" . }}'
        text: |
          {{ range .Alerts }}
          *Alert:* {{ .Annotations.summary }}
          *Severity:* {{ .Labels.severity }}
          *Description:* {{ .Annotations.description }}
          *Runbook:* {{ .Annotations.runbook_url }}
          {{ end }}

  - name: 'slack-noise'
    slack_configs:
      - api_url: '<YOUR_SLACK_WEBHOOK_URL>'
        channel: '#monitoring-noise'
        send_resolved: false
        title: '[INFO] {{ .GroupLabels.alertname }}'
        text: '{{ (index .Alerts 0).Annotations.description }}'

inhibit_rules:
  # If the host is down, suppress all alerts from processes on that host.
  # This prevents a cascade of "app is down" pages when the real cause is
  # "the VM is unreachable."
  - source_match:
      alertname: 'ServiceDown'
      severity: 'critical'
    target_match_re:
      severity: 'critical|warning'
    equal: ['instance']

  # If the entire cluster is degraded, suppress individual pod alerts
  - source_match:
      alertname: 'KubernetesNodeNotReady'
    target_match_re:
      alertname: 'PodCrashLooping|PodNotReady'
    equal: ['node']

Grouping: Don’t Send 50 Individual Alerts

The group_by configuration is one of the most impactful settings in Alertmanager. Without grouping, a disk issue on 20 nodes sends 20 pages. With grouping, it sends one.

1
2
3
4
5


route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s      # Collect alerts for 30s before first notification
  group_interval: 5m   # Don't re-notify about the same group for 5m
  repeat_interval: 4h  # Re-notify if still firing after 4h

group_wait is the most important of these. Setting it to 30 seconds allows related alerts that trigger within the same window to be bundled into a single notification. Without it, a rolling restart that brings down 10 pods simultaneously generates 10 separate pages in 10 seconds.

Inhibition: Suppress Downstream Noise

Inhibition rules suppress alerts that are consequences of a more fundamental alert already firing. The canonical example: if the host is unreachable, every process on that host will also appear to be down. You don’t want 15 alerts about individual services when the root cause is one NodeDown alert.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


inhibit_rules:
  # Suppress service-level alerts when the host itself is down
  - source_match:
      alertname: 'NodeDown'
    target_match_re:
      alertname: '.*'
    equal: ['instance']

  # Suppress warning-level when critical already firing for same service
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'service']

Silences: For Maintenance, Not for Covering Noise

Silences mute matching alerts for a defined period. They’re the right tool for planned maintenance windows — not for suppressing noisy alerts you haven’t fixed yet.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


# Install amtool
# On Linux:
wget https://github.com/prometheus/alertmanager/releases/download/v0.27.0/alertmanager-0.27.0.linux-amd64.tar.gz

# Configure amtool
cat > ~/.config/amtool/config.yml <<EOF
alertmanager.url: http://localhost:9093
author: your-name
comment_required: true
EOF

# Add a silence for a maintenance window
amtool silence add \
  --comment "Planned maintenance: upgrading prod-db-01" \
  --duration "2h" \
  instance="prod-db-01"

# List active silences
amtool silence query

# List all currently firing alerts
amtool alert query

# Test routing for a hypothetical alert
amtool config routes test \
  --verify.receivers='pagerduty-critical' \
  severity=critical alertname=ServiceDown

Runbooks: Turning Alerts Into Procedures

An alert without a runbook is noise. Even a good alert, firing correctly for a real problem, fails its job if the on-call engineer has to figure out what to do from scratch at 3 AM.

A runbook is a procedure: the specific steps the on-call engineer follows when a specific alert fires. It doesn’t need to cover every possible cause — it needs to give the engineer a starting point, the most common causes and fixes, and a clear escalation path.

What a Good Runbook Contains

Alert name and description: what metric is triggering, what threshold
Severity and escalation owner: who to page next if this isn’t resolved in X minutes
Plain English explanation: what is actually broken or at risk
Initial investigation steps: specific commands to run, dashboards to check
Common causes: ranked by likelihood, with their specific fixes
Escalation path: who to call when you’re stuck
Links: dashboards, log queries, architecture diagrams, related runbooks

Runbook Template

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


# Runbook: [AlertName]

**Alert:** `AlertName`
**Severity:** Critical / Warning
**Last Reviewed:** YYYY-MM-DD
**Owner:** Platform Team / Service Team

---

## What This Alert Means

[One or two sentences in plain English. What is broken or at risk? What does the user experience?]

---

## Immediate Actions

1. Check the dashboard: [link]
2. Check recent deployments: `kubectl rollout history deployment/<name> -n <namespace>`
3. [Next specific step with actual command]

---

## Investigation

### Step 1: Confirm the alert is real

```bash
# Command to verify the condition

Step 2: Identify the cause

[Specific investigation steps with commands]

Common Causes and Fixes

Cause 1: Recent bad deployment

Symptoms: Error rate increased sharply after a deploy Fix:

1

kubectl rollout undo deployment/<service-name> -n production

Cause 2: [Next cause]

Symptoms: [How to recognize this] Fix: [Steps to resolve]

Escalation

After 15 minutes without mitigation: page the service team lead
After 30 minutes: page the engineering manager and open a customer-facing status update
Escalation contact: [Name / PagerDuty escalation policy]

Dashboard: [link]
Logs (Grafana/Loki): [link]
Architecture diagram: [link]
Related runbooks: [link]
Postmortem for last occurrence: [link]


### Complete Runbook: HighErrorRate

```markdown
# Runbook: HighErrorRate

**Alert:** `HighErrorRate`
**Severity:** Critical
**Last Reviewed:** 2026-03-01
**Owner:** Platform Team

---

## What This Alert Means

More than 5% of HTTP requests to a production service are returning 5xx errors.
Users are actively experiencing failures. This alert warrants immediate investigation.

---

## Immediate Actions

1. Check the error rate dashboard: https://grafana.example.com/d/service-overview
2. Identify *which* service is affected (check `job` label on the alert)
3. Check for recent deployments in the last 30 minutes

---

## Investigation

### Step 1: Determine which endpoint is failing

```bash
# In Grafana, run this query in Explore:
sum by (handler, status) (
  rate(http_requests_total{job="$job", status=~"5.."}[5m])
)

Step 2: Check the application logs

1
2
3
4
5
6
7
8


# Kubernetes
kubectl logs -n production deployment/<service-name> --since=15m | grep -i error | tail -50

# Docker Compose
docker compose logs --since=15m <service-name> | grep -i error | tail -50

# Loki (Grafana Explore)
{app="<service-name>", namespace="production"} |= "error" | logfmt | line_format "{{.msg}}"

Step 3: Check for a recent deployment

1
2
3


kubectl rollout history deployment/<service-name> -n production
# If a recent rollout exists, check what changed:
kubectl describe deployment/<service-name> -n production | grep -A 5 "Image:"

Step 4: Check upstream dependencies

1
2
3
4
5
6


# Check if the database is reachable
kubectl exec -n production deployment/<service-name> -- \
  nc -zv <db-host> 5432

# Check if downstream services are healthy
curl -sf https://api.example.com/health

Common Causes and Fixes

Cause 1: Bad deployment

Symptoms: Error rate spiked immediately after a deployment event Fix:

1
2


kubectl rollout undo deployment/<service-name> -n production
kubectl rollout status deployment/<service-name> -n production

Notify the team in #incidents that a rollback was performed.

Cause 2: Database connection exhaustion

Symptoms: Errors contain “too many connections” or “connection pool exhausted” Fix: Restart the service to reset the connection pool. Identify the root cause (query leak, traffic spike) and address after stabilization.

1

kubectl rollout restart deployment/<service-name> -n production

Cause 3: Downstream dependency down

Symptoms: Errors contain the name of a dependent service, timeout errors Fix: Check the status of the dependency. If it’s a third-party service, check their status page. Consider enabling the circuit breaker or returning a degraded response.

Cause 4: Malformed request from a specific client

Symptoms: 400-level errors (check if it’s actually 4xx being miscounted as 5xx) Fix: Identify the client IP from logs. Block or rate-limit if it’s a bad actor. Fix the parsing if it’s a legitimate client hitting an edge case.

Escalation

After 10 minutes without identifying the cause: page the service team lead via PagerDuty
After 20 minutes without mitigation: post to #status-page and notify #customer-success
After 30 minutes: page the engineering manager

Dashboard: https://grafana.example.com/d/service-overview
Logs: https://grafana.example.com/d/loki-logs
Architecture: https://wiki.example.com/architecture/checkout-service
Postmortem history: https://wiki.example.com/postmortems?tag=high-error-rate


### Complete Runbook: DiskPredictedFull

```markdown
# Runbook: DiskWillFillIn24Hours

**Alert:** `DiskWillFillIn24Hours`
**Severity:** Warning
**Last Reviewed:** 2026-02-15
**Owner:** Infrastructure Team

---

## What This Alert Means

Based on disk usage growth over the last 6 hours, a filesystem is on track to run out
of space within 24 hours. This is a P2 — it doesn't need immediate night-time response,
but it must be investigated before the disk actually fills (which will cause a P1).

---

## Investigation

### Step 1: Identify the filesystem and current usage

```bash
df -hT | grep -v tmpfs

Step 2: Find what’s consuming space

1
2
3
4
5
6
7
8


# Find largest directories from the root
du -sh /* 2>/dev/null | sort -rh | head -20

# Drill into the top directory
du -sh /var/* 2>/dev/null | sort -rh | head -20

# Find recently modified large files
find /var -type f -size +100M -mtime -7 2>/dev/null | sort -k5 -rn

Step 3: Check log files specifically

1
2
3
4
5
6
7
8
9


# Check journal size
journalctl --disk-usage

# Find large log files
find /var/log -type f -name "*.log" -o -name "*.gz" | xargs du -sh 2>/dev/null | sort -rh | head -20

# Check Docker overlay storage
du -sh /var/lib/docker/overlay2/
docker system df

Step 4: Check for core dumps

1
2


find / -name "core" -o -name "core.*" 2>/dev/null | head -20
ls -lh /var/crash/ 2>/dev/null

Common Causes and Fixes

Cause 1: Unrotated application logs

Symptoms: Large files in /var/log or application log directories Fix:

1
2
3
4
5


# Force log rotation
logrotate -f /etc/logrotate.conf

# Verify log rotation is configured for the application
cat /etc/logrotate.d/<appname>

Cause 2: Docker images/containers accumulating

Symptoms: /var/lib/docker is the large consumer Fix:

1
2
3
4
5


# Remove stopped containers, unused images, dangling volumes
docker system prune -f

# More aggressive (removes all unused images, not just dangling)
docker system prune -af --volumes

Warning: --volumes will remove named volumes not attached to a running container. Verify before running in production.

Cause 3: Database write-ahead log (WAL) not being cleaned

Symptoms: Large files in Postgres data directory Fix: Check replication lag — if a replica is too far behind, WAL files accumulate. Fix the replica, or temporarily increase max_wal_size while you investigate.

Cause 4: Unexpected data growth (legitimate traffic increase)

Symptoms: All data appears to be normal application data, just more of it Fix: Add storage, resize the volume, or implement data archival/TTL policies. This is a planning issue, not an emergency.

Escalation

After 4 hours without resolution: escalate to infrastructure team lead
If disk is now >95% full: treat as P1 and page immediately

Dashboard: https://grafana.example.com/d/node-exporter
Capacity planning doc: https://wiki.example.com/capacity-planning


### Complete Runbook: ServiceDown

```markdown
# Runbook: ServiceDown

**Alert:** `ServiceDown`
**Severity:** Critical
**Last Reviewed:** 2026-03-01
**Owner:** Platform Team

---

## What This Alert Means

Prometheus has been unable to scrape a service endpoint for 2 consecutive minutes.
The service is either crashed, unreachable, or its metrics endpoint is broken.

---

## Immediate Actions

1. Verify the alert is real (not a Prometheus scrape issue)
2. Check if users are impacted: https://status.example.com
3. Check the service in Kubernetes or Docker

---

## Investigation

### Step 1: Is this a scrape issue or a real outage?

```bash
# Try to reach the service directly
curl -sf http://<service-host>:<port>/health
curl -sf http://<service-host>:<port>/metrics | head -5

If the service responds normally, the issue is in Prometheus’s network path to the target, not the service itself. Check Prometheus targets at http://localhost:9090/targets.

Step 2: Check service status (Kubernetes)

1
2
3
4
5
6
7
8


# Check pod status
kubectl get pods -n production -l app=<service-name>

# Check recent events
kubectl get events -n production --sort-by=.lastTimestamp | tail -20

# Check logs from the (possibly crashing) pod
kubectl logs -n production -l app=<service-name> --previous --tail=100

Step 3: Check service status (Docker)

1
2
3
4
5
6
7
8


# Check container status
docker ps -a | grep <service-name>

# Check logs
docker logs <container-name> --tail=100

# Check exit code
docker inspect <container-name> | jq '.[0].State'

Step 4: Check recent deployments

1
2


kubectl rollout history deployment/<service-name> -n production
git log --oneline -10  # if you have access to the repo

Step 5: Check resource constraints

1
2
3
4
5
6


# Kubernetes: check if OOM killed
kubectl describe pod <pod-name> -n production | grep -A 5 "OOMKilled\|Limits\|Requests"

# Check node resources
kubectl top nodes
kubectl top pods -n production

Common Causes and Fixes

Cause 1: OOM kill

Symptoms: Container state shows OOMKilled: true Fix: Restart the service (it may already be restarting). Identify the memory growth root cause (memory leak, unexpectedly large request, missing memory limits). Consider temporarily increasing memory limits.

1
2


kubectl set resources deployment/<service-name> -n production \
  --limits=memory=2Gi

Cause 2: Bad deployment

Symptoms: Pod is crashing immediately after a recent rollout Fix:

1

kubectl rollout undo deployment/<service-name> -n production

Cause 3: Dependency unavailable (database, cache, external API)

Symptoms: Application logs show connection errors to a dependency at startup Fix: Investigate the dependency. If it’s a database, check connection limits and whether the DB is healthy. If it’s an external service, check their status page and consider graceful degradation.

Cause 4: Configuration error

Symptoms: Application crashes with a configuration-related error in logs Fix: Identify the bad config value. Check if a ConfigMap or Secret was recently changed.

1
2


kubectl get configmap <name> -n production -o yaml
kubectl get events -n production | grep ConfigMap

Escalation

After 5 minutes without identifying the cause: page the service team lead
After 10 minutes without mitigation: post to #incidents and #status-page
If the service cannot be restored in 15 minutes: page the engineering manager

Dashboard: https://grafana.example.com/d/service-overview
Architecture: https://wiki.example.com/architecture
Dependency map: https://wiki.example.com/dependencies


### Storing and Linking Runbooks

**Where to store them:** The best runbooks live alongside your alert rules in the same git repository. This makes them easy to find, keeps them in version control, and means the person modifying alert thresholds is prompted to also update the runbook.

monitoring/ ├── rules/ │ ├── service_health.yml │ └── infrastructure.yml └── runbooks/ ├── high-error-rate.md ├── disk-predicted-full.md └── service-down.md


**Linking runbooks from alert annotations** is the key integration. Every alert should have a `runbook_url` annotation that the on-call engineer can click immediately from PagerDuty, Alertmanager, or Grafana:

```yaml
annotations:
  runbook_url: "https://wiki.example.com/runbooks/high-error-rate"
  # Or if runbooks are in git:
  runbook_url: "https://github.com/your-org/monitoring/blob/main/runbooks/high-error-rate.md"

Keeping runbooks current: A runbook that hasn’t been touched since the service was first deployed is probably wrong. After every incident, review and update the relevant runbook. Add the new cause you discovered, remove investigation steps that turned out to be irrelevant, update commands for the current environment.

On-Call Rotation Design

Goals of a Good Rotation

An on-call rotation should distribute burden fairly, ensure genuine coverage at all hours, and be sustainable enough that engineers are still effective after their shift. That last point is often missed: an engineer who slept two hours the previous night is not providing the same quality of response as a rested one. Sustainability is a reliability concern, not just a welfare concern.

Rotation Models

Weekly handoff is the most common for small teams. One engineer carries the pager for a week, hands off to the next. Simple to schedule, provides continuity within a week (the on-call engineer accumulates context), but requires a minimum of 4–5 engineers for the rotation to not be constant.

Follow-the-sun uses multiple timezone-separated engineers so that each person is only on-call during their business hours. Requires at least one person per major timezone with genuine coverage. Excellent for user experience and engineer welfare; complex to schedule and requires cross-timezone handoff processes.

Shadow rotation pairs a junior engineer with the primary on-call for training purposes. The shadow receives all the same pages but is not responsible for response. After a rotation, they’re ready to be primary. Critical for onboarding new team members safely.

Team Size Considerations

Fewer than 4 engineers in a rotation means each engineer is on-call more than once per month, which typically becomes unsustainable within a few quarters. At 4 engineers, each person is on-call roughly one week in four — manageable but tight if incidents are frequent.

The target is around 6–8 engineers in a rotation. This provides a reasonable buffer for vacations, sick time, and attrition without anyone feeling like they’re perpetually on-call.

If your team is too small for a sustainable rotation, that’s a hiring argument, not an alerting problem to engineer around.

Compensation

On-call work has real costs to the engineer’s time, sleep, and attention. Compensation should reflect that:

On-call pay: a stipend for being reachable and available during the rotation, regardless of whether pages occur
Incident response pay: additional compensation for actual incidents during off-hours, often calculated per-incident or per-hour
Compensatory time: time off after a particularly disruptive on-call week

The exact structure varies by company and jurisdiction, but the principle should be non-negotiable: on-call work is work, and it should be paid like work.

The Handoff Ritual

A structured handoff between outgoing and incoming on-call engineers transfers context that isn’t captured in ticket systems. Cover:

Active incidents or known degraded systems
Changes deployed in the last 48 hours that might cause issues
Known noisy alerts: “the DiskWarning on prod-db-02 has been firing all week, it’s a known issue, ticket #4821 is tracking it”
Systems currently in maintenance mode or under silences
Any unusual traffic patterns or upcoming events (product launches, marketing campaigns) that might spike load

A 15-minute handoff call (or a written summary in a dedicated Slack channel) prevents the incoming engineer from inheriting hidden context gaps.

Response Time SLAs

Set clear expectations for how quickly on-call engineers should respond:

Severity	Acknowledgment	Initial Response
P1 (Critical)	5 minutes	15 minutes
P2 (Warning)	30 minutes	Next business day

These aren’t just guidelines — your PagerDuty escalation policy should enforce them automatically, escalating to a secondary if the primary hasn’t acknowledged within the P1 window.

On-Call Tooling

PagerDuty

PagerDuty is the enterprise standard for on-call management. Key features for effective on-call:

Schedules: define rotation schedules with override support for vacations
Escalation policies: automatic escalation if the primary doesn’t acknowledge within N minutes
Incident management: incident timeline, response plays, status page integration
Analytics: MTTA, MTTR, alert volume per person — the data you need to make the case for reducing on-call burden

PagerDuty’s free tier is limited; most teams need the paid tier for more than two escalation levels.

OpsGenie

OpsGenie (Atlassian) covers similar ground to PagerDuty with strong Jira and Confluence integration. If your org is already invested in the Atlassian ecosystem, OpsGenie’s integrations are seamless.

Grafana On-Call (Open Source)

Grafana On-Call (formerly Amixr) is self-hostable and open source, making it attractive for teams that want to keep their entire observability stack internal. It integrates directly with Grafana and supports Slack, phone, and SMS notifications.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


# grafana-oncall docker-compose snippet
services:
  oncall:
    image: grafana/oncall:latest
    ports:
      - "8080:8080"
    environment:
      - DATABASE_TYPE=sqlite3
      - BROKER_TYPE=redis
      - REDIS_URI=redis://redis:6379/0
      - SECRET_KEY=<your-secret-key>
      - GRAFANA_API_URL=http://grafana:3000
    depends_on:
      - redis

  redis:
    image: redis:7-alpine

Homelab: Alertmanager + ntfy.sh

For homelabs and self-hosted setups that don’t need full incident management infrastructure, a combination of Alertmanager and ntfy.sh provides mobile push notifications at zero cost.

ntfy.sh is a simple, open-source pub/sub notification service. You can self-host it or use the free public instance.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# docker-compose.yml for self-hosted ntfy
services:
  ntfy:
    image: binwiederhier/ntfy:latest
    command: serve
    ports:
      - "8080:80"
    volumes:
      - ./ntfy-data:/var/lib/ntfy
    environment:
      - NTFY_BASE_URL=http://ntfy.home.example.com
      - NTFY_CACHE_FILE=/var/lib/ntfy/cache.db
      - NTFY_AUTH_FILE=/var/lib/ntfy/auth.db
      - NTFY_AUTH_DEFAULT_ACCESS=deny-all

Wire Alertmanager to ntfy via the webhook receiver:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# In alertmanager.yml receivers section
receivers:
  - name: 'ntfy-critical'
    webhook_configs:
      - url: 'http://ntfy.home.example.com/alerts'
        http_config:
          authorization:
            credentials: '<your-ntfy-token>'
        send_resolved: true
        # ntfy uses custom headers for priority and tags
        # Use a custom template or a small proxy to set these

Install the ntfy mobile app (Android/iOS), subscribe to your alerts topic, and your homelab will push notifications directly to your phone — no third-party cloud service needed.

The Incident Lifecycle

Every page should follow a consistent lifecycle. When the process is ad hoc, incidents take longer to resolve and context gets lost.

Detection → Acknowledgment → Investigation → Mitigation → Resolution → Postmortem

Detection: Alertmanager fires; PagerDuty pages the on-call engineer.

Acknowledgment: The on-call engineer confirms they’ve seen the alert and are investigating. Acknowledging within the SLA (5 minutes for P1) stops the escalation timer. Acknowledging is a promise: “I am looking at this.”

Investigation: The engineer follows the runbook, gathers data, and identifies the cause. The 5-minute rule: if you’ve been investigating for 5 minutes without meaningful progress, ask for help. Don’t sit alone spinning your wheels while the incident grows.

Mitigation: Stopping the bleeding — rolling back, restarting a service, disabling a feature flag — even before the root cause is fully understood. Mitigation restores service; root cause analysis can happen after users are no longer impacted.

Resolution: The service is restored and stable. Close the incident in PagerDuty, update the status page, notify stakeholders.

Postmortem: Write it while the incident is fresh. The postmortem is not a blame session — it’s a structured analysis of what happened and how to prevent it from happening again.

Incident Communication

For P1 incidents, communication should run in parallel with investigation:

Create an incident Slack channel immediately: #inc-2026-03-25-checkout-down. Centralize all discussion there.
Update the status page as soon as the incident is confirmed: even “We are investigating reports of issues with checkout” is better than silence.
Notify stakeholders (customer success, support, management) at regular intervals: every 15–30 minutes.

Log your actions in the incident channel as you go. “Rolled back to v2.4.1 at 03:42 UTC” is invaluable for the postmortem timeline.

The Blameless Postmortem

The blameless postmortem is a commitment: we are analyzing systems and processes, not assigning fault to individuals. When people fear blame, they hide information that would help prevent future incidents.

A good postmortem answers:

What happened, and what was the user impact?
What was the timeline of detection, response, and resolution?
What caused it? (Use five-whys to get to root causes, not just proximate causes)
What went well in our response?
What went poorly?
What action items will prevent recurrence? (Specific, assigned, with due dates)

The action items are the product of the postmortem. A postmortem with no action items is just storytelling.

Building a Healthier On-Call Culture

Fixing your alerting configuration is the technical half of the problem. The other half is the culture and practices around on-call.

Measuring On-Call Health

You cannot improve what you do not measure. Track these metrics:

Metric	What It Tells You
MTTA (Mean Time to Acknowledge)	How quickly engineers respond; high MTTA suggests burnout or unclear ownership
MTTR (Mean Time to Resolve)	Incident response efficiency; high MTTR suggests poor runbooks or tooling
Alert volume per engineer per week	Total burden; track this per person to spot unfair distribution
Noise percentage	Fraction of alerts that required no action; target below 10%
Pages per shift	Total interruptions; more than 2–3 per night is unsustainable
On-call satisfaction (survey)	Subjective experience matters; ask quarterly

Most on-call tooling (PagerDuty, OpsGenie) provides these analytics built-in. If you’re running a homelab setup, export Alertmanager data and analyze in Grafana.

The Weekly Alert Review

Dedicate 30 minutes per week to reviewing the previous week’s alerts as a team. Go through:

Which alerts fired most? Are they legitimate?
Which pages woke someone up? Were they worth waking someone up?
Are there alerts that fired and required no action? Those are candidates for demotion or deletion.
Are there any new “always-on” alerts that have been firing all week?

This meeting prevents alert debt from accumulating silently. It also normalizes the conversation: engineers should feel safe saying “this alert is noise” without it being treated as a complaint.

The Toil Reduction Sprint

Dedicate 20% of every sprint to toil reduction — fixing flapping alerts, improving runbooks, automating common incident responses. This is often called “reliability work” or “operational excellence work.”

Without explicit time allocation, this work never happens. The feature backlog always feels more urgent than fixing a noisy alert. But the compounding cost of that noisy alert — the accumulated sleep interruptions, the boy-who-cried-wolf effect, the engineers who quietly start updating their LinkedIn — outweighs almost any feature.

Track toil reduction work alongside features. Make it visible to leadership. The business case is real: reducing on-call burden improves retention, and retaining experienced engineers is far cheaper than replacing them.

Psychological Safety

Engineers need to feel safe raising on-call concerns without judgment. The team lead’s job is to create an environment where:

Saying “this alert is noisy” is welcomed, not dismissed
Escalating during an incident is encouraged, not seen as weakness
A missed page due to genuine alert fatigue is treated as a system problem, not an individual failure
Postmortems don’t produce blame, even when a human made a clear mistake

Psychological safety isn’t just a cultural nicety — it’s a reliability mechanism. Teams where engineers fear judgment hide problems, delay escalation, and produce worse incidents. Teams where engineers feel safe raising concerns catch problems earlier and resolve them faster.

Putting It All Together

Alert fatigue is not inevitable. It’s the accumulated result of individual decisions that each seemed reasonable at the time: adding one more alert “just in case,” setting thresholds low to be cautious, not cleaning up old alerts because it’s not urgent.

Reversing it requires deliberate effort: auditing what you have, deleting what you don’t need, redesigning what’s left around user impact, writing runbooks that give engineers something to do, and building the team rituals that prevent the debt from accumulating again.

The investments pay for themselves quickly. An engineering team that trusts its alerting responds faster, escalates appropriately, and sleeps better. The engineers who would have left in six months because of unsustainable on-call — they stay.

Start with the audit. Pick your five noisiest alerts this week. Ask the three questions. Write one runbook. Schedule the first weekly alert review. You don’t have to fix everything at once; you just have to start, and keep going.

Useful tools and references:

Alertmanager documentation — routing, inhibition, silences
Google SRE Book: Alerting on SLOs — multi-window burn rate alerting
PagerDuty Incident Response Guide — free, comprehensive incident management framework
ntfy.sh documentation — self-hosted push notifications
Grafana On-Call — open source on-call management

The Alert Fatigue Problem

The Boy Who Cried Wolf

What Alert Fatigue Actually Costs

Symptoms Your Alerting Is Broken

The Philosophy of Good Alerting

Alert on Symptoms, Not Causes

Every Alert Must Be Actionable

Pages Should Be Rare and Urgent

The Four Severity Levels

The Signal-to-Noise Test

Auditing Your Existing Alerts

Pulling Alert History from Prometheus

The Actionability Audit

Categories of Broken Alerts

The One-Month Silence Experiment

Designing Actionable Alerts

The Three-Question Test

USE and RED Methods

SLO-Based Alerting

Symptom-Based Prometheus Alert Examples

Prometheus Alertmanager Configuration

Routing Alerts to the Right Destination

Grouping: Don’t Send 50 Individual Alerts

Inhibition: Suppress Downstream Noise

Silences: For Maintenance, Not for Covering Noise

Runbooks: Turning Alerts Into Procedures

What a Good Runbook Contains

Runbook Template

Step 2: Identify the cause

Common Causes and Fixes

Cause 1: Recent bad deployment

Cause 2: [Next cause]

Escalation

Related Links

Step 2: Check the application logs

Step 3: Check for a recent deployment

Step 4: Check upstream dependencies

Common Causes and Fixes

Cause 1: Bad deployment

Cause 2: Database connection exhaustion

Cause 3: Downstream dependency down

Cause 4: Malformed request from a specific client

Escalation

Related Links

Step 2: Find what’s consuming space

Step 3: Check log files specifically

Step 4: Check for core dumps

Common Causes and Fixes

Cause 1: Unrotated application logs

Cause 2: Docker images/containers accumulating

Cause 3: Database write-ahead log (WAL) not being cleaned

Cause 4: Unexpected data growth (legitimate traffic increase)

Escalation

Related Links

Step 2: Check service status (Kubernetes)

Step 3: Check service status (Docker)

Step 4: Check recent deployments

Step 5: Check resource constraints

Common Causes and Fixes

Cause 1: OOM kill

Cause 2: Bad deployment

Cause 3: Dependency unavailable (database, cache, external API)

Cause 4: Configuration error

Escalation

Related Links

On-Call Rotation Design

Goals of a Good Rotation

Rotation Models

Team Size Considerations

Compensation

The Handoff Ritual

Response Time SLAs

On-Call Tooling

PagerDuty

OpsGenie

Grafana On-Call (Open Source)

Homelab: Alertmanager + ntfy.sh

The Incident Lifecycle

Detection → Acknowledgment → Investigation → Mitigation → Resolution → Postmortem

Incident Communication