SLOs and Error Budgets: The Engineering Discipline Behind Reliable Services

March 26, 2026 23 min read DevOps

sresloslierror-budgetreliabilityalertingprometheusobservability

Contents

Reliability is a feature. But unlike most features, it competes directly with velocity — every hour spent hardening a service is an hour not spent shipping new functionality. Service Level Objectives (SLOs) resolve this tension by making reliability a negotiated contract: you agree on how reliable a service needs to be, you measure it honestly, and the remaining headroom becomes your error budget — the allowance for risk-taking.

This isn’t abstract theory. SLOs are how Google, Netflix, and Shopify manage hundreds of services with small teams. This guide covers the complete lifecycle: defining meaningful SLIs, setting defensible SLOs, computing error budgets, wiring up burn rate alerts that work, and running SLO reviews that actually change behavior.

The Vocabulary (Getting it Straight)

These three terms are often conflated:

SLI (Service Level Indicator)
  A quantitative measurement of some aspect of service behavior.
  Example: "the fraction of HTTP requests returning 2xx or 3xx"

SLO (Service Level Objective)
  A target value or range for an SLI, measured over a time window.
  Example: "99.9% of requests succeed, measured over a rolling 30-day window"

SLA (Service Level Agreement)
  A contract between you and a customer with financial penalties for violations.
  Example: "We guarantee 99.9% uptime or we credit 10% of your monthly bill"

SLOs are internal goals. SLAs are external promises. Your SLO should be tighter than your SLA — if your SLA is 99.9%, your internal SLO might be 99.95%, giving you a buffer before contractual obligations kick in.

What Makes a Good SLI

The SLI is the measurement everything else is built on. Bad SLIs produce bad SLOs.

The Four Golden Signals as SLI Candidates

Signal	SLI Form	Good for
Availability	`good_requests / total_requests`	APIs, web services
Latency	`requests_below_threshold / total_requests`	User-facing services
Throughput	`bytes_processed / time`	Batch pipelines, streaming
Error rate	`failed_requests / total_requests`	Background jobs

Request/Response SLIs (Most Common)

For HTTP services, define “good” requests precisely:

Good request: HTTP response delivered in ≤500ms with status code not in {5xx}
Bad request: timeout, 5xx response, or response time >500ms

SLI = count(good requests over window) / count(total requests over window)

Avoid vague SLIs like “uptime” (binary, doesn’t capture partial degradation) or “error rate” (doesn’t normalize for traffic volume changes).

Latency SLIs

Don’t use average latency — it hides tail latency that affects real users. Use a percentile threshold:

Good: request latency ≤ 200ms
Measured at: p99 (the 99th percentile request in a window)

SLI = count(requests where latency < 200ms) / count(all requests)

This formulation lets you track latency as a ratio, the same way you track availability — which means the same alerting machinery works for both.

What to Exclude from SLI Measurement

Not every request should count toward your SLI:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Exclude from SLI measurement:
# - Health check endpoints (/health, /ready, /ping)
# - Maintenance windows (planned downtime with customer notice)
# - Requests that fail due to client error (4xx) — these aren't your fault
# - Requests during a known infrastructure incident already being handled

# Include:
# - All genuine user-initiated requests
# - Bot traffic if it represents real customer usage (e.g., API clients)
# - Failed requests — 5xx and timeouts are always bad events

Setting the SLO Target

The number you pick matters enormously. Here’s how to think about it.

The Nines Table

SLO        Error Budget (30 days)   Error Budget (28 days)
99%        7h 18m                   6h 43m
99.5%      3h 39m                   3h 22m
99.9%      43m 49s                  40m 19s
99.95%     21m 54s                  20m 10s
99.99%     4m 22s                   4m 1s
99.999%    26s                      24s

A 99.999% SLO for a web app is almost certainly wrong — it means you have 26 seconds a month to deploy, rollback, restart, or respond to any incident. Most web apps don’t need this, and targeting it burns engineering time on diminishing returns.

How to Pick the Right Number

Start with measurement, not aspiration. Look at the last 90 days of actual reliability data:

1
2
3
4
5
6


# Historical availability for your service (last 90 days)
1 - (
  sum(increase(http_requests_total{status=~"5..", job="api"}[90d]))
  /
  sum(increase(http_requests_total{job="api"}[90d]))
)

If you’ve been running at 99.7% for the last 90 days, setting an SLO of 99.9% means you’re already violating it. Set the initial SLO slightly below your actual performance, then tighten it as you improve.

Consider the cost to users. A 99.5% SLO means users experience roughly 22 minutes of errors per month. For a payment service, that’s catastrophic. For a developer dashboard, it might be acceptable.

Consider the cost to implement. Going from 99.9% to 99.99% often requires architectural changes (active-active multi-region, zero-downtime deploys, etc.) that cost 10× more than the previous order of reliability.

The Reliability Stack

Services depend on each other. A composed SLO is the product of its dependencies:

Your service SLO:     99.9%
Database dependency:  99.95%
Cache dependency:     99.99%
Cloud provider:       99.99%

Maximum achievable composite availability:
0.999 × 0.9995 × 0.9999 × 0.9999 ≈ 99.83%

If your SLO is 99.9% but your critical path includes a 99.95% dependency,
you need redundancy or circuit breakers to achieve your target.

Error Budgets

The error budget is what’s left after you subtract your SLO from 100%.

SLO = 99.9%
Error Budget = 100% - 99.9% = 0.1%

Over 30 days (2,592,000 seconds):
  0.1% of 2,592,000 = 2,592 seconds = 43 minutes 12 seconds

Over 30 days at 100 req/s = 259,200,000 requests:
  0.1% of 259,200,000 = 259,200 allowed bad requests

The error budget has a crucial property: it’s shared between reliability work and feature development. When the budget is full, the team can move fast. When it’s depleted, reliability work takes priority. This transforms reliability from a perpetual argument (“we should be more reliable!”) into a mechanical decision (“we’ve spent 80% of our budget, so we slow down deployments this sprint”).

Error Budget Policy

Document the policy before you need it — don’t negotiate in the middle of an incident:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


## Error Budget Policy — Checkout Service

**Budget healthy (>50% remaining)**
- Deploy at normal cadence (multiple times per day)
- Feature work proceeds normally
- Postmortems for incidents are suggested but optional

**Budget at risk (10–50% remaining)**
- Deployments limited to business hours with engineer on standby
- Any new feature requiring risky infrastructure changes needs reliability review
- Weekly SLO review added to team standup

**Budget exhausted (<10% remaining)**
- Feature freeze: no new deployments except reliability fixes and hotfixes
- On-call rotations increased
- Incident postmortem required for all incidents >5 minutes
- Engineering leadership notified
- Budget reviewed weekly until above 50%

**Budget reset**
- Monthly on the 1st of each month
- Historical data retained for trend analysis

Implementing SLOs with Prometheus

Defining the Recording Rules

Recording rules pre-compute expensive queries so dashboards and alerts are fast:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103


# prometheus/rules/slo-checkout-service.yml
groups:
  - name: slo_checkout_service
    interval: 30s
    rules:
      # --- Raw SLI components ---

      # Good requests: 2xx/3xx responses delivered under 500ms
      - record: job:http_requests_good:rate5m
        expr: |
          sum(rate(http_request_duration_seconds_count{
            job="checkout-service",
            status_code!~"5..",
            le="0.5"
          }[5m])) by (job)

      # Total requests
      - record: job:http_requests_total:rate5m
        expr: |
          sum(rate(http_request_duration_seconds_count{
            job="checkout-service"
          }[5m])) by (job)

      # --- SLI ratio (rolling windows) ---
      # 5-minute window (for fast burn alerts)
      - record: job:slo_availability:ratio_rate5m
        expr: |
          job:http_requests_good:rate5m
          /
          job:http_requests_total:rate5m

      # 30-minute window
      - record: job:slo_availability:ratio_rate30m
        expr: |
          sum(rate(http_request_duration_seconds_count{
            job="checkout-service", status_code!~"5..", le="0.5"
          }[30m])) by (job)
          /
          sum(rate(http_request_duration_seconds_count{
            job="checkout-service"
          }[30m])) by (job)

      # 1-hour window
      - record: job:slo_availability:ratio_rate1h
        expr: |
          sum(rate(http_request_duration_seconds_count{
            job="checkout-service", status_code!~"5..", le="0.5"
          }[1h])) by (job)
          /
          sum(rate(http_request_duration_seconds_count{
            job="checkout-service"
          }[1h])) by (job)

      # 6-hour window
      - record: job:slo_availability:ratio_rate6h
        expr: |
          sum(rate(http_request_duration_seconds_count{
            job="checkout-service", status_code!~"5..", le="0.5"
          }[6h])) by (job)
          /
          sum(rate(http_request_duration_seconds_count{
            job="checkout-service"
          }[6h])) by (job)

      # 3-day window (for slow burn alerts)
      - record: job:slo_availability:ratio_rate3d
        expr: |
          sum(rate(http_request_duration_seconds_count{
            job="checkout-service", status_code!~"5..", le="0.5"
          }[3d])) by (job)
          /
          sum(rate(http_request_duration_seconds_count{
            job="checkout-service"
          }[3d])) by (job)

      # --- Error budget consumption ---
      # SLO target = 0.999 (99.9%)
      # Error budget burn rate = actual error rate / target error rate
      # Burn rate of 1 = consuming budget at exactly the right pace
      # Burn rate of 2 = consuming budget twice as fast as allowed

      - record: job:slo_error_budget_burn:rate1h
        expr: |
          (1 - job:slo_availability:ratio_rate1h) / (1 - 0.999)

      - record: job:slo_error_budget_burn:rate6h
        expr: |
          (1 - job:slo_availability:ratio_rate6h) / (1 - 0.999)

      - record: job:slo_error_budget_burn:rate3d
        expr: |
          (1 - job:slo_availability:ratio_rate3d) / (1 - 0.999)

      # --- 30-day error budget remaining ---
      - record: job:slo_error_budget_remaining:ratio
        expr: |
          1 - (
            (1 - job:slo_availability:ratio_rate30m)
            /
            (1 - 0.999)
            *
            (5 / (30 * 24 * 60))  # fraction of 30-day window consumed
          )

Multi-Window Burn Rate Alerts

The key insight from the Google SRE Workbook: simple threshold alerts on error rate are terrible for SLOs. They either alert too slowly (you’re burning budget for hours before anyone notices) or fire too often (transient spikes trigger pages at 3am that resolve themselves).

The solution is multi-window burn rate alerting. The burn rate tells you how fast you’re consuming your error budget relative to the allowed pace. A burn rate of 1 means you’ll exactly exhaust the budget at the end of the window. A burn rate of 14.4 means you’ll exhaust a 30-day budget in 50 hours.

Burn Rate  Error Rate (for 99.9% SLO)   Budget exhausted in
1×         0.1%                          30 days (end of period)
2×         0.2%                          15 days
5×         0.5%                          6 days
10×        1%                            3 days
14.4×      1.44%                         50 hours
36×        3.6%                          20 hours
72×        7.2%                          10 hours

The multi-window approach uses two windows to filter noise:

Short window: detects the problem quickly
Long window: confirms it’s not a transient spike

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84


# prometheus/rules/slo-alerts-checkout-service.yml
groups:
  - name: slo_alerts_checkout_service
    rules:

      # === TIER 1: PAGE IMMEDIATELY ===
      # Burn rate >14.4× over 1h AND >14.4× over 5m
      # = exhausting 30-day budget in <50 hours
      # Detection time: ~5 minutes | False positive rate: very low
      - alert: CheckoutServiceSLOCritical
        expr: |
          (
            job:slo_error_budget_burn:rate1h{job="checkout-service"} > 14.4
            and
            (1 - job:slo_availability:ratio_rate5m{job="checkout-service"}) / (1 - 0.999) > 14.4
          )
        for: 2m
        labels:
          severity: critical
          team: checkout
          slo: "checkout-service-availability"
        annotations:
          summary: "Checkout service burning error budget >14.4× (page now)"
          description: |
            Error budget burn rate is {{ $value | humanize }}× the allowed rate.
            At this pace the 30-day error budget will be exhausted in
            {{ printf "%.1f" (div 720 $value) }} hours.
            Current error rate: ~{{ printf "%.2f" (mul $value 0.001) }}%
          runbook_url: "https://wiki.internal/runbooks/checkout-service-slo"
          dashboard_url: "https://grafana.internal/d/checkout-slo"

      # === TIER 2: PAGE WITHIN 30 MINUTES ===
      # Burn rate >6× over 6h AND >6× over 30m
      # = exhausting 30-day budget in ~5 days
      - alert: CheckoutServiceSLOWarning
        expr: |
          (
            job:slo_error_budget_burn:rate6h{job="checkout-service"} > 6
            and
            job:slo_availability:ratio_rate30m{job="checkout-service"} < (1 - 6 * 0.001)
          )
        for: 15m
        labels:
          severity: warning
          team: checkout
          slo: "checkout-service-availability"
        annotations:
          summary: "Checkout service burning error budget >6× (investigate soon)"
          description: |
            Sustained elevated error rate detected.
            Burn rate: {{ $value | humanize }}× over 6 hours.
            Budget will be exhausted in ~{{ printf "%.1f" (div 120 $value) }} hours at this pace.

      # === TIER 3: TICKET (NOT A PAGE) ===
      # Burn rate >3× over 3 days
      # = slow leak consuming budget faster than expected
      - alert: CheckoutServiceSLOSlowBurn
        expr: |
          job:slo_error_budget_burn:rate3d{job="checkout-service"} > 3
        for: 1h
        labels:
          severity: info
          team: checkout
          slo: "checkout-service-availability"
        annotations:
          summary: "Checkout service slow error budget burn (create ticket)"
          description: |
            Error budget is being consumed at {{ $value | humanize }}× the allowed rate
            over the past 3 days. Budget remaining: ~{{ ... }}%.
            This won't exhaust the budget immediately but warrants investigation.

      # === BUDGET EXHAUSTION WARNING ===
      - alert: CheckoutServiceErrorBudgetLow
        expr: |
          job:slo_error_budget_remaining:ratio{job="checkout-service"} < 0.1
        for: 5m
        labels:
          severity: warning
          team: checkout
        annotations:
          summary: "Checkout service error budget below 10%"
          description: |
            Only {{ printf "%.1f" (mul $value 100) }}% of the monthly error budget remains.
            Per the error budget policy, feature deployments are now restricted.

Why This Alert Structure Works

Tier 1 (Critical page): 14.4× burn rate
  Window: 1h + 5m
  → Catches: complete outages, severe degradation
  → Miss rate: essentially zero for >10 minute outages
  → False positive rate: very low (requires sustained high rate in both windows)

Tier 2 (Warning page): 6× burn rate
  Window: 6h + 30m
  → Catches: elevated error rates that are burning budget in days
  → Allows: 30 minutes to respond (not 3am panic)
  → False positive rate: very low (needs sustained rate over 30m confirmed by 6h)

Tier 3 (Ticket): 3× burn rate
  Window: 3 days
  → Catches: slow leaks, creeping degradation nobody noticed
  → Action: investigate before budget gets critical
  → False positive rate: negligible (3-day window filters all transients)

The Error Budget Dashboard

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73


# Generate Grafana dashboard JSON via Python (or use Grafonnet/Jsonnet)
# Key panels to include:

panels = [
    {
        "title": "SLO Status",
        "type": "stat",
        "expr": "job:slo_availability:ratio_rate30m * 100",
        "thresholds": {
            "steps": [
                {"color": "red",    "value": None},    # < SLO
                {"color": "yellow", "value": 99.5},    # close to SLO
                {"color": "green",  "value": 99.9},    # at or above SLO
            ]
        },
        "unit": "percent",
        "decimals": 3,
    },
    {
        "title": "Error Budget Remaining (30-day rolling)",
        "type": "gauge",
        "expr": "job:slo_error_budget_remaining:ratio * 100",
        "min": 0,
        "max": 100,
        "thresholds": [
            {"color": "red",    "value": 0},
            {"color": "yellow", "value": 10},
            {"color": "green",  "value": 50},
        ],
    },
    {
        "title": "Burn Rate (1h)",
        "type": "timeseries",
        "expr": "job:slo_error_budget_burn:rate1h",
        "thresholds": [
            {"value": 1,    "color": "green"},
            {"value": 6,    "color": "yellow"},
            {"value": 14.4, "color": "red"},
        ],
    },
    {
        "title": "Request Volume and Error Rate",
        "type": "timeseries",
        "exprs": [
            {
                "label": "Requests/s",
                "expr": "sum(rate(http_requests_total{job='checkout-service'}[5m]))",
            },
            {
                "label": "Error rate %",
                "expr": "(1 - job:slo_availability:ratio_rate5m) * 100",
            },
        ],
    },
    {
        "title": "Latency Percentiles",
        "type": "timeseries",
        "exprs": [
            "histogram_quantile(0.50, rate(http_request_duration_seconds_bucket{job='checkout-service'}[5m]))",
            "histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{job='checkout-service'}[5m]))",
            "histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{job='checkout-service'}[5m]))",
        ],
        "labels": ["p50", "p95", "p99"],
    },
    {
        "title": "Error Budget Burndown (30-day)",
        "type": "timeseries",
        "description": "How much budget has been consumed over the current period",
        "expr": "(1 - job:slo_error_budget_remaining:ratio) * 100",
        # Show a reference line at 100% (budget exhausted)
        # Show a diagonal "ideal burn" line from 0 to 100% over 30 days
    },
]

Automated Error Budget Tracking

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125


# scripts/error_budget_report.py
# Run daily to generate Slack/email reports

import requests
from datetime import datetime, timedelta
import json

PROMETHEUS_URL = "http://prometheus:9090"
SLO_TARGET = 0.999
WINDOW_DAYS = 30

def query_prometheus(query: str) -> float:
    response = requests.get(
        f"{PROMETHEUS_URL}/api/v1/query",
        params={"query": query},
    )
    result = response.json()["data"]["result"]
    if not result:
        return None
    return float(result[0]["value"][1])


def get_error_budget_report(service: str) -> dict:
    sli = query_prometheus(
        f'sum(increase(http_request_duration_seconds_count{{'
        f'job="{service}", status_code!~"5..", le="0.5"'
        f'}}[{WINDOW_DAYS}d])) / '
        f'sum(increase(http_request_duration_seconds_count{{'
        f'job="{service}"'
        f'}}[{WINDOW_DAYS}d]))'
    )

    burn_rate_1h = query_prometheus(
        f'job:slo_error_budget_burn:rate1h{{job="{service}"}}'
    )

    budget_remaining = query_prometheus(
        f'job:slo_error_budget_remaining:ratio{{job="{service}"}}'
    )

    total_requests = query_prometheus(
        f'sum(increase(http_request_duration_seconds_count{{'
        f'job="{service}"}}[{WINDOW_DAYS}d]))'
    )

    allowed_bad = total_requests * (1 - SLO_TARGET) if total_requests else 0
    actual_bad = total_requests * (1 - sli) if (total_requests and sli) else 0
    budget_consumed = actual_bad / allowed_bad if allowed_bad else 0

    return {
        "service": service,
        "period": f"Last {WINDOW_DAYS} days",
        "slo_target": f"{SLO_TARGET * 100:.3f}%",
        "actual_availability": f"{(sli or 0) * 100:.4f}%",
        "slo_met": (sli or 0) >= SLO_TARGET,
        "budget_remaining_pct": f"{(budget_remaining or 0) * 100:.1f}%",
        "budget_consumed_pct": f"{budget_consumed * 100:.1f}%",
        "allowed_bad_minutes": f"{allowed_bad / (total_requests / (WINDOW_DAYS * 24 * 60)):.1f}" if total_requests else "N/A",
        "actual_bad_minutes": f"{actual_bad / (total_requests / (WINDOW_DAYS * 24 * 60)):.1f}" if total_requests else "N/A",
        "current_burn_rate": f"{burn_rate_1h:.2f}×" if burn_rate_1h else "N/A",
        "status": _get_status(budget_remaining, burn_rate_1h),
    }


def _get_status(budget_remaining: float, burn_rate: float) -> str:
    if budget_remaining is None:
        return "UNKNOWN"
    if budget_remaining < 0.0:
        return "EXHAUSTED"
    if budget_remaining < 0.10:
        return "CRITICAL"
    if budget_remaining < 0.50 or (burn_rate and burn_rate > 6):
        return "AT_RISK"
    return "HEALTHY"


def format_slack_message(reports: list[dict]) -> dict:
    status_emoji = {
        "HEALTHY": ":white_check_mark:",
        "AT_RISK": ":warning:",
        "CRITICAL": ":rotating_light:",
        "EXHAUSTED": ":fire:",
        "UNKNOWN": ":grey_question:",
    }

    blocks = [
        {
            "type": "header",
            "text": {"type": "plain_text", "text": f"SLO Report — {datetime.now().strftime('%Y-%m-%d')}"}
        }
    ]

    for r in reports:
        emoji = status_emoji[r["status"]]
        slo_icon = ":white_check_mark:" if r["slo_met"] else ":x:"

        blocks.append({
            "type": "section",
            "text": {
                "type": "mrkdwn",
                "text": (
                    f"*{r['service']}* {emoji}\n"
                    f"SLO {slo_icon} | Availability: *{r['actual_availability']}* (target: {r['slo_target']})\n"
                    f"Budget remaining: *{r['budget_remaining_pct']}* | "
                    f"Consumed: {r['budget_consumed_pct']} | "
                    f"Current burn: {r['current_burn_rate']}"
                )
            }
        })

    return {"blocks": blocks}


if __name__ == "__main__":
    services = ["checkout-service", "api-gateway", "payment-service", "user-service"]
    reports = [get_error_budget_report(s) for s in services]

    # Print report
    for r in reports:
        print(json.dumps(r, indent=2))

    # Send to Slack
    slack_webhook = "https://hooks.slack.com/services/..."
    payload = format_slack_message(reports)
    requests.post(slack_webhook, json=payload)

Running an SLO Review

The SLO review is where the rubber meets the road. Without a regular ritual to examine the data and make decisions, SLOs become shelfware.

Who Should Attend

Engineering team (all members, not just on-call)
Product manager (owns feature prioritization)
Engineering manager
Optional: customer success (for customer impact context)

Cadence

Frequency	When to use
Weekly	Budget is at risk (<50%), team is in an active reliability push
Bi-weekly	Normal operations, budget is healthy
Monthly	End-of-period review, trend analysis, SLO target revision

Meeting Agenda Template

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


## SLO Review — Checkout Service — 2026-03-26

### 1. Budget Status (5 min)
- Current budget remaining: ____%
- Trend: improving / stable / degrading
- On track to meet SLO this period? Y/N

### 2. Incidents Since Last Review (10 min)
For each incident that consumed >5% of the error budget:
- What happened?
- How long did it take to detect? (MTTD)
- How long to mitigate? (MTTR)
- What user impact (requests affected, revenue impact if known)?
- Postmortem complete? Action items assigned?

### 3. Alert Quality Review (5 min)
- False positives since last review: ___
- Alerts that fired but were NOT paged: ___
- Incidents where alerts fired AFTER users reported: ___
- Action: tune thresholds? Add coverage?

### 4. Error Budget Policy Decisions (10 min)
Based on current budget status:
- Deployment cadence: normal / restricted / freeze?
- Any risky changes planned this period? (database migrations, dependency upgrades)
- Reliability work to prioritize this sprint?

### 5. SLO Target Health (5 min, monthly only)
- Is our SLO target still appropriate?
- Are we consistently >20% above target? → SLO may be too loose
- Did we exhaust budget? → SLO may be too tight, or reliability work needed
- Any customer feedback suggesting the SLO doesn't match actual expectations?

### 6. Action Items
| Action | Owner | Due |
|--------|-------|-----|
| ... | ... | ... |

Diagnosing Budget Consumption

When you see budget draining faster than expected, work through this checklist:

1. When did it start?
   → Correlate with deploys (check your deploy markers in Grafana)
   → Correlate with external events (cloud provider incident, traffic spike)

2. Which endpoint/operation?
   → Break down error rate by route, method, status code
   → PromQL: rate(http_requests_total{status=~"5.."}[5m]) by (route)

3. Which error type?
   → 500 Internal Server Error → application bug
   → 502 Bad Gateway → upstream dependency
   → 503 Service Unavailable → capacity/overload
   → 504 Gateway Timeout → slow dependency

4. Which population of users?
   → By region? By customer tier? By client version?
   → rate(http_requests_total{status=~"5.."}[5m]) by (region, customer_tier)

5. Is it correlated with traffic volume?
   → If errors spike with traffic: capacity or resource exhaustion issue
   → If errors are constant regardless of traffic: broken dependency or config

6. Is it recoverable without deploy?
   → Circuit breaker trip? Restart the affected pod
   → Cache poisoning? Flush the cache
   → Resource leak? Rolling restart

Advanced Patterns

Multi-Service SLOs (Composite)

For a user journey that spans multiple services (e.g., “checkout” = product API + cart + payment + inventory), define a composite SLO:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Composite SLO: all services must be available for the journey to succeed
# Good request = all four services returned success

min(
  job:slo_availability:ratio_rate5m{job=~"product-api|cart-service|payment-service|inventory-service"}
)

# Or model it explicitly via synthetic transaction monitoring:
# Run a synthetic checkout every 60s, measure end-to-end success rate
probe_success{job="synthetic-checkout"} == 1

Latency SLOs with Multiple Thresholds

User perception of latency isn’t binary. Model it with multiple thresholds:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# Apdex-style SLO: 100% of requests, weighted by user experience
# "Satisfied" (<200ms) = 1.0 weight
# "Tolerable" (200ms–1s) = 0.5 weight
# "Frustrated" (>1s) = 0.0 weight

(
  sum(rate(http_request_duration_seconds_bucket{le="0.2", job="api"}[5m]))
  +
  sum(rate(http_request_duration_seconds_bucket{le="1.0", job="api"}[5m]))
  -
  sum(rate(http_request_duration_seconds_bucket{le="0.2", job="api"}[5m]))
  * 0.5
)
/
sum(rate(http_request_duration_seconds_count{job="api"}[5m]))

Availability SLO for Batch Jobs

Request/response SLIs don’t apply to batch jobs. Use a different model:

1
2
3
4
5
6


# Batch job SLI: fraction of pipeline runs that complete successfully within deadline
# Good run = completed with status="success" AND runtime < 4 hours

batch_job_runs_total{status="success", runtime_bucket="lt_4h"}
/
batch_job_runs_total

1
2
3
4
5
6
7


# Alert: pipeline hasn't succeeded in 6 hours
- alert: BatchPipelineFailure
  expr: |
    time() - batch_job_last_success_timestamp{job="etl-pipeline"} > 6 * 3600
  for: 5m
  annotations:
    summary: "ETL pipeline hasn't completed successfully in 6 hours"

SLO with Planned Maintenance Windows

Exclude maintenance windows from SLI calculations to avoid burning budget on planned downtime:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


# maintenance_windows.py
# Store in a time series (e.g., write 1 during maintenance, 0 otherwise)

import subprocess
from datetime import datetime, timezone

def annotate_maintenance_start(service: str, reason: str):
    """Write a Prometheus metric signaling maintenance start."""
    timestamp = int(datetime.now(timezone.utc).timestamp() * 1000)
    # Use pushgateway or write to a file for node exporter textfile collector
    with open(f"/var/lib/node_exporter/textfile_collector/maintenance.prom", "w") as f:
        f.write(f'# HELP service_maintenance_active 1 if service is in planned maintenance\n')
        f.write(f'# TYPE service_maintenance_active gauge\n')
        f.write(f'service_maintenance_active{{service="{service}",reason="{reason}"}} 1\n')

def annotate_maintenance_end(service: str):
    with open(f"/var/lib/node_exporter/textfile_collector/maintenance.prom", "w") as f:
        f.write(f'service_maintenance_active{{service="{service}"}} 0\n')

1
2
3
4
5
6
7


# SLI that excludes maintenance windows
(
  sum(rate(http_requests_total{job="checkout-service", status!~"5.."}[5m]))
  /
  sum(rate(http_requests_total{job="checkout-service"}[5m]))
)
unless on() service_maintenance_active{service="checkout-service"} == 1

Common SLO Anti-Patterns

Setting SLOs without measuring current performance first You end up with a number that’s either trivially achievable (always green, no signal) or already violated (permanent red, nobody pays attention).

Using SLAs as SLOs Your SLA is a legal floor. Your SLO should be a meaningful engineering target above it — tighter by enough margin that you catch problems before the SLA trips.

Measuring availability from inside the service Internal metrics miss: load balancer failures, DNS issues, network problems between regions, and anything that prevents requests from reaching your service. Always supplement with external synthetic monitoring.

Ignoring tail latency Average latency is useless. 1% of users experiencing 10-second responses is a crisis. Measure at p95 and p99.

Too many SLOs If a team maintains SLOs for 20 metrics, none of them will be taken seriously. Three to five SLOs per service is usually right. Cover: availability, latency, and optionally data freshness or throughput.

SLOs without error budget policies An SLO without a policy attached is just a metric. The policy (what changes when the budget is at risk) is where the value lives.

Tooling Reference

Tool	Role
Prometheus	SLI metric collection and recording rules
Alertmanager	Routing burn rate alerts to right channels
Grafana	Error budget dashboards
Sloth	Generates Prometheus rules from a simple SLO YAML spec
OpenSLO	Vendor-neutral SLO spec format
Pyrra	Kubernetes operator for SLOs with auto-generated dashboards
SLO Exporter	Compute SLIs from application logs

Sloth: SLO-as-Code

Sloth generates all the recording rules and alerts from a simple spec:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


# checkout-slo.yml
version: "prometheus/v1"
service: "checkout-service"
labels:
  team: checkout
  tier: "1"

slos:
  - name: "requests-availability"
    objective: 99.9
    description: "99.9% of checkout requests succeed under 500ms"
    sli:
      events:
        error_query: |
          sum(rate(http_request_duration_seconds_count{
            job="checkout-service",
            status_code=~"(5..|408|429)"
          }[{{.window}}]))
          +
          sum(rate(http_request_duration_seconds_bucket{
            job="checkout-service",
            le="0.5"
          }[{{.window}}]))
          - sum(rate(http_request_duration_seconds_count{
            job="checkout-service"
          }[{{.window}}]))
        total_query: |
          sum(rate(http_request_duration_seconds_count{
            job="checkout-service"
          }[{{.window}}]))
    alerting:
      name: CheckoutServiceSLO
      page_alert:
        labels:
          severity: critical
      ticket_alert:
        labels:
          severity: warning

1
2
3
4
5


# Generate Prometheus rules from the spec
sloth generate -i checkout-slo.yml -o checkout-slo-rules.yml

# Apply to Kubernetes via PrometheusRule CRD
kubectl apply -f checkout-slo-rules.yml

Putting It All Together

A mature SLO practice looks like this:

Week 1: Measure
  - Instrument your service with request metrics (counter + histogram)
  - Define SLI queries: what's "good"?
  - Look at 90 days of historical data
  - Set initial SLO at ~10th percentile of historical performance

Week 2: Alert
  - Create recording rules for SLI windows
  - Implement multi-window burn rate alerts
  - Tune: did any alerts fire? Were they correct?

Week 3: Policy
  - Write the error budget policy document
  - Get sign-off from product and engineering leadership
  - Add budget remaining to team dashboard

Month 2+: Review
  - Run monthly SLO reviews
  - Adjust SLO targets as you improve (or as you discover the target was wrong)
  - Track MTTD and MTTR as operational health metrics
  - Retire or replace SLOs that don't correlate with user experience

SLOs work because they externalize the reliability conversation from “we should be more reliable” (subjective, political) to “we have 4 hours of error budget left this month” (objective, mechanical). That shift — from debate to arithmetic — is what makes engineering teams faster, not slower.

Filed under: Observability Deep Dives