SLOs in Practice: Beyond the Math

The SLO conversation in most organizations goes one of two ways. Either it stays theoretical — engineers understand the concepts, there are some dashboards, but nobody actually changes behavior based on the error budget. Or it gets implemented too rigidly, producing alert fatigue and resentment because the numbers don’t reflect what users actually experience.

Getting SLOs to work requires more than correct math. It requires alert designs that page at the right time, error budget policies that have teeth, tooling that keeps the overhead low, and — hardest of all — organizational buy-in that makes the whole thing worth doing.

This post covers all of it: the alerting math that actually works, the tools that automate the boilerplate, and the organizational patterns that make SLOs stick.

A Brief Grounding in the Concepts

If you’re already comfortable with SLIs, SLOs, and error budgets, skip this section. If not, a quick grounding:

SLI (Service Level Indicator): a quantitative measure of service behavior. The most common ones:

• Availability: fraction of requests that succeeded (good_requests / total_requests)
• Latency: fraction of requests completing under a threshold (requests_under_100ms / total_requests)
• Throughput: requests processed per second (less common for SLOs)
• Error rate: fraction of requests returning errors

SLO (Service Level Objective): a target for the SLI over a time window. “99.9% of requests succeed over a 30-day rolling window.”

Error budget: the amount of unreliability the SLO permits. For a 99.9% SLO over 30 days: (1 - 0.999) × 30 × 24 × 60 = 43.2 minutes of downtime. That’s your budget to spend on deployments, incidents, and maintenance.

SLA (Service Level Agreement): a contract with consequences. SLAs are usually a floor below your SLO — if you commit to 99.9% SLO internally, your SLA might be 99.5%.

Why Simple Threshold Alerts on SLOs Fail

The first instinct when you have an SLO is to alert when you breach it. “Alert if error rate > 0.1% in the last 5 minutes.” This seems reasonable but has two fatal problems:

Too noisy: a 5-minute spike that costs you 5 minutes of error budget out of 43 minutes doesn’t warrant waking anyone up at 2 AM. You’d burn through on-call patience long before the SLO is at risk.

Too slow: if you’re burning error budget slowly (0.2% errors, twice the allowed rate), a 5-minute alert window won’t catch it until you’ve already spent half your budget.

The Google SRE book introduced a better framework: burn rate alerting.

Burn Rate Alerting

The burn rate is how fast you’re consuming error budget relative to the rate at which it replenishes. A burn rate of 1 means you’re consuming budget at exactly the rate it regenerates — you’ll end the SLO window exactly at the objective. A burn rate of 2 means you’re consuming twice as fast as it replenishes — you’ll exhaust the budget halfway through the window.

For a 30-day SLO with a 0.1% error budget (99.9% target):

• Total error budget: 43.2 minutes
• Budget per hour: 43.2 / (30 × 24) = 0.06 minutes = 3.6 seconds/hour

If your error rate is 1% (10× the allowed 0.1%), your burn rate is 10. You’ll exhaust the entire 30-day budget in 3 days.

The alert threshold becomes: at this burn rate, how long until the budget is exhausted?

time_to_exhaustion = budget_remaining / burn_rate

A burn rate of 14.4 means the budget exhausts in:

30 days / 14.4 = 2.08 days ≈ 50 hours

That’s worth paging someone. A burn rate of 1 is not.

Multi-Window Multi-Burn-Rate (MWMB) Alerting

The Google-recommended approach uses two alert windows at each of two burn rates:

Why two windows per alert? The short window detects the problem quickly; the long window provides confidence that it’s real and not a brief spike. You alert only when both windows show the burn rate threshold exceeded. This dramatically reduces false positives.

Why 5 minutes + 1 hour for the critical alert?

• 1-hour window catches sustained elevated error rates
• 5-minute window ensures the condition is still active right now (avoids alerting on a problem that resolved itself)

Prometheus Alert Rules

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74

# slo-alerts.yaml
groups:
  - name: api-availability-slo
    rules:
      # Burn rate calculation: ratio of error rate to budget rate
      # For 99.9% SLO: budget = 0.001, so burn_rate = error_rate / 0.001

      # Critical: 14.4x burn rate — exhausts budget in ~50 hours
      - alert: HighErrorBudgetBurn_Critical
        expr: |
          (
            sum(rate(http_requests_total{job="api",status=~"5.."}[1h]))
            /
            sum(rate(http_requests_total{job="api"}[1h]))
          ) > (14.4 * 0.001)  # 14.4 * (1 - SLO)
          and
          (
            sum(rate(http_requests_total{job="api",status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total{job="api"}[5m]))
          ) > (14.4 * 0.001)
        for: 2m
        labels:
          severity: critical
          slo: api-availability
        annotations:
          summary: "API error budget burning at 14.4x rate"
          description: >
            Error rate is {{ $value | humanizePercentage }} over the last hour.
            At this rate, the 30-day error budget will be exhausted in
            {{ printf "%.1f" (div 720.0 14.4) }} hours.
            Current budget remaining: check SLO dashboard.

      # Serious: 6x burn rate — exhausts budget in ~5 days
      - alert: HighErrorBudgetBurn_Serious
        expr: |
          (
            sum(rate(http_requests_total{job="api",status=~"5.."}[6h]))
            /
            sum(rate(http_requests_total{job="api"}[6h]))
          ) > (6 * 0.001)
          and
          (
            sum(rate(http_requests_total{job="api",status=~"5.."}[30m]))
            /
            sum(rate(http_requests_total{job="api"}[30m]))
          ) > (6 * 0.001)
        for: 15m
        labels:
          severity: warning
          slo: api-availability
        annotations:
          summary: "API error budget burning at 6x rate"

      # Warning: 3x burn rate — creates a ticket, not a page
      - alert: HighErrorBudgetBurn_Warning
        expr: |
          (
            sum(rate(http_requests_total{job="api",status=~"5.."}[1d]))
            /
            sum(rate(http_requests_total{job="api"}[1d]))
          ) > (3 * 0.001)
          and
          (
            sum(rate(http_requests_total{job="api",status=~"5.."}[2h]))
            /
            sum(rate(http_requests_total{job="api"}[2h]))
          ) > (3 * 0.001)
        for: 1h
        labels:
          severity: info
          slo: api-availability
        annotations:
          summary: "API error budget burning at 3x rate — investigate this week"

Latency SLOs

Latency SLOs use histograms. The SLI is “fraction of requests completing under X ms”:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

# Latency SLO: 99% of requests under 500ms
- alert: LatencyBudgetBurn_Critical
  expr: |
    (
      1 - (
        sum(rate(http_request_duration_seconds_bucket{job="api",le="0.5"}[1h]))
        /
        sum(rate(http_request_duration_seconds_count{job="api"}[1h]))
      )
    ) > (14.4 * 0.01)  # 14.4 * (1 - 0.99)
    and
    (
      1 - (
        sum(rate(http_request_duration_seconds_bucket{job="api",le="0.5"}[5m]))
        /
        sum(rate(http_request_duration_seconds_count{job="api"}[5m]))
      )
    ) > (14.4 * 0.01)
  for: 2m
  labels:
    severity: critical
    slo: api-latency

Defining Good SLIs

The math only helps if your SLI actually measures what users care about. Common mistakes:

Measuring the wrong thing: if your load balancer returns 200 OK with an error message in the body, a simple status != 5xx SLI will miss those errors. Measure at the right layer.

Ignoring partial availability: if 20% of users are hitting a bad shard but the overall error rate is low, your SLO won’t fire. Consider per-region, per-shard, or per-user-cohort SLIs for services with sharded data.

Using averages for latency: a p50 latency SLO misses tail latency. Use a histogram-based SLI: “99th percentile of requests complete in under 1 second.” If you can’t use histograms, at minimum use p99.

Not excluding maintenance windows: planned downtime should be excluded from error budget consumption. Keep a configuration that marks maintenance periods and excludes them from SLI calculation.

Including health check traffic: if your monitoring pings your service every 10 seconds and you have very little real traffic, health check traffic dominates and makes your SLI look artificially good or bad depending on whether the checks are included correctly.

The SLI Specification Format

A clean way to document SLIs before implementing them:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

# sli-spec.yaml
slo:
  name: api-availability
  description: "Fraction of API requests that complete successfully"

  sli:
    events:
      error_query: |
        sum(rate(http_requests_total{job="api", status=~"5.."}[{{.window}}]))
      total_query: |
        sum(rate(http_requests_total{job="api"}[{{.window}}]))

  objective:
    target: 0.999   # 99.9%
    window: 30d

  alerting:
    page_alert:
      burn_rate: 14.4
    ticket_alert:
      burn_rate: 3

Tools: Sloth and Pyrra

Writing and maintaining SLO alert rules by hand is error-prone and tedious. Two tools automate the generation of MWMB alert rules from a higher-level spec.

Sloth

Sloth generates Prometheus alert rules and recording rules from an SLO spec. You define the SLO once; Sloth generates the multi-window burn rate alerts automatically.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

# api-slos.yaml
version: "prometheus/v1"

service: "api"
labels:
  team: platform
  env: production

slos:
  - name: "requests-availability"
    objective: 99.9
    description: "99.9% of API requests succeed"

    sli:
      events:
        error_query: sum(rate(http_requests_total{job="api",status=~"5.."}[{{.window}}]))
        total_query: sum(rate(http_requests_total{job="api"}[{{.window}}]))

    alerting:
      name: APIHighErrorRate
      page_alert:
        labels:
          severity: critical
          channel: pagerduty
      ticket_alert:
        labels:
          severity: warning
          channel: slack

  - name: "requests-latency"
    objective: 99
    description: "99% of requests complete in under 500ms"

    sli:
      events:
        error_query: >
          sum(rate(http_request_duration_seconds_bucket{job="api",le="0.5"}[{{.window}}]))
        total_query: >
          sum(rate(http_request_duration_seconds_count{job="api"}[{{.window}}]))
      # Note: invert because this is a "good events" query
      # Sloth handles the inversion automatically for latency SLOs

Generate the Prometheus rules:

1
2
3
4
5
6
7
8

# Install
go install github.com/slok/sloth/cmd/sloth@latest

# Generate rules
sloth generate -i api-slos.yaml -o api-slo-rules.yaml

# Or output to stdout and pipe to kubectl
sloth generate -i api-slos.yaml | kubectl apply -f -

Sloth also generates recording rules that pre-compute error budget metrics — critical for dashboards, since computing burn rates over long windows in real-time is expensive.

Pyrra

Pyrra is similar to Sloth but adds a built-in UI for visualizing SLO status and error budget consumption. It runs as a Kubernetes operator:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

# pyrra-slo.yaml
apiVersion: pyrra.dev/v1alpha1
kind: ServiceLevelObjective
metadata:
  name: api-availability
  namespace: monitoring
spec:
  target: "99.9"
  window: 30d

  serviceMonitorSelector:
    matchLabels:
      app: api

  indicator:
    ratio:
      errors:
        metric: http_requests_total{job="api",status=~"5.."}
      total:
        metric: http_requests_total{job="api"}

1
2
3
4
5
6
7
8
9

# Install via Helm
helm repo add pyrra https://pyrra-dev.github.io/pyrra
helm install pyrra pyrra/pyrra -n monitoring

# Apply SLO definition
kubectl apply -f pyrra-slo.yaml

# Pyrra generates PrometheusRules automatically
kubectl get prometheusrules -n monitoring

Pyrra’s UI shows each SLO’s current status, error budget remaining as a percentage and in time, and historical burn rate trends.

Choosing Between Sloth and Pyrra

The Error Budget Policy

An error budget is only useful if it changes behavior. Without a policy, a team with 0% error budget remaining behaves identically to a team at 50% — there are no consequences or guardrails.

An error budget policy answers three questions:

1. What does the team commit to do while the budget is healthy?
2. What happens when the budget is depleted?
3. What happens when the budget is critically low (< 25%)?

Sample Error Budget Policy

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

# Error Budget Policy: API Team

## Definitions
- Error budget period: 30 rolling days
- SLO: 99.9% availability, 99th percentile latency < 500ms
- Error budget: 43.2 minutes downtime OR 5.18 minutes latency violations

## When budget is healthy (> 50% remaining)
- Normal development and deployment velocity
- Feature work prioritized as usual
- Deploy during business hours without additional approval

## When budget is below 50%
- All production deployments require approval from team lead
- Post-deployment monitoring window increased to 30 minutes
- Weekly review of reliability work in sprint planning

## When budget is below 25%
- No non-critical deployments until budget recovers
- One engineer per sprint dedicated to reliability work
- Daily check-in with engineering manager
- Incident review for all events that consumed > 5% of budget

## When budget is exhausted (0%)
- Deployment freeze until budget recovers to 25%
- Reliability work becomes top priority, overriding roadmap
- Escalation to VP Engineering if not recovered within 5 days
- Post-mortem required covering reliability strategy

## Budget resets and exceptions
- Planned maintenance is excluded from budget consumption
- Novel failure modes from third-party dependencies may be excluded (requires
  review and approval from SRE team)
- Budget resets when 30-day rolling window naturally moves past the incident

The specific thresholds matter less than the policy being written down, agreed upon, and actually enforced. Teams that have a clear policy can make rational decisions about risk. Teams without one fight the same arguments every sprint.

Grafana Dashboards for SLOs

Standard dashboard layout for an SLO:

Row 1: Status panel

• Current SLO status (Met / At Risk / Breached) — large colored indicator
• Error budget remaining (as percentage and minutes/hours)
• Current burn rate

Row 2: Error budget burn over time

• Graph of error budget remaining over the 30-day window
• Annotated with incidents and deployments

Row 3: SLI details

• Error rate over time (with SLO threshold line)
• Latency percentiles (p50, p90, p99) over time
• Request volume over time

Row 4: Alert status

• Table of active SLO alerts with burn rate and severity

Grafana dashboard as code (using the Grafana Terraform provider or a dashboard JSON):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

{
  "title": "API SLO Dashboard",
  "panels": [
    {
      "title": "Error Budget Remaining",
      "type": "stat",
      "targets": [{
        "expr": "1 - (sum(increase(http_requests_total{job='api',status=~'5..'}[30d])) / sum(increase(http_requests_total{job='api'}[30d]))) / (1 - 0.999)",
        "legendFormat": "Budget Remaining"
      }],
      "fieldConfig": {
        "defaults": {
          "unit": "percentunit",
          "thresholds": {
            "steps": [
              {"color": "red", "value": 0},
              {"color": "yellow", "value": 0.25},
              {"color": "green", "value": 0.5}
            ]
          }
        }
      }
    }
  ]
}

The recording rules generated by Sloth or Pyrra make these dashboard queries fast — they pre-compute the rolling window metrics so dashboards don’t need to scan weeks of raw data.

Organizational Challenges

The technical implementation is the easy part. SLOs fail for organizational reasons far more often than technical ones.

“Who owns the SLO?”

SLOs only work when a team owns both the reliability of the service and the error budget policy. Shared services with unclear ownership don’t get reliable — they get argued about.

The pattern that works: SLO ownership mirrors service ownership. The team that develops and operates a service owns its SLOs. Platform/SRE teams can advise on SLI selection and tooling, but the product team owns the numbers and the budget.

“The SLO doesn’t reflect user experience”

This is the most common technical complaint, and it’s often valid. An SLO based on HTTP 5xx rate looks fine while users are experiencing slow page loads, timeouts handled with retries, or degraded functionality that returns 200 OK.

Fixes:

• Add a user-journey SLO that measures end-to-end critical paths, not just the API layer
• Add synthetic monitoring (Grafana k6, Playwright in a cron) that simulates real user actions
• Collect real user monitoring (RUM) data and use it to calibrate SLOs

“The error budget is always green, so nothing changes”

If the error budget never gets threatened, the SLO is too loose. This feels safe but means the SLO provides no signal. The error budget should feel meaningfully constrained — if a team deploys 10 times a week, they should feel some budget pressure from normal operations.

Use actual historical data to set SLOs. Look at your last 90 days of error rate. If you’ve been running at 99.97% availability, setting the SLO at 99.9% means the budget is nearly always full and has no behavioral effect. Set it at 99.95% instead.

“We keep getting paged on things that aren’t real problems”

This is the burn rate calibration problem. Common causes:

• SLI measures something users don’t care about (health check traffic, internal retries)
• Burn rate thresholds are too sensitive for actual traffic patterns
• No for duration on alerts (single-minute spikes trigger pages)

The fix is iteration. Treat alert configuration like code — review it in retrospectives, track false positive rates, and tune aggressively in the first few months.

“Product keeps pushing features and ignoring the error budget”

The error budget policy only works if leadership enforces it. If a VP can override the deployment freeze because “this feature has to ship,” the policy is theater.

This is primarily a management problem, not a technical one. Escalation paths and executive sponsorship of reliability work have to be established before an incident, not during one.

Getting Buy-In: Start Small

The pattern that works for introducing SLOs into skeptical organizations:

1. Pick one service with a motivated team — don’t roll out organization-wide on day one
2. Set obviously achievable SLOs initially — the goal is building the instrumentation and process, not setting an aggressive bar
3. Focus on making dashboards useful — teams need to see value before they’ll accept constraints
4. Let the first error budget burn happen naturally — don’t engineer a crisis, but don’t paper over an incident that consumes budget either
5. Hold the retrospective — after the first budget-consuming incident, walk through what happened with the policy in hand
6. Tighten the SLO after 3–6 months — once the team trusts the instrumentation

The teams that implement SLOs successfully almost always describe it as “the first time we had a shared language for reliability conversations with product management.” That’s the real value — not the math, but the ability to say “we’ve consumed 40% of our error budget this month, so the next deployment needs extra scrutiny” in a meeting and have everyone understand what that means.

Quick Reference

Burn Rate → Time to Exhaustion (30-day SLO)

Standard MWMB Alert Thresholds

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

# For a 30-day SLO at target T (e.g., 0.999)
# Budget rate = 1 - T (e.g., 0.001)

critical_alert:
  burn_rate: 14.4
  long_window: 1h
  short_window: 5m
  paging: yes

serious_alert:
  burn_rate: 6
  long_window: 6h
  short_window: 30m
  paging: yes

warning_alert:
  burn_rate: 3
  long_window: 1d
  short_window: 2h
  paging: no  # ticket

low_alert:
  burn_rate: 1
  long_window: 3d
  short_window: 6h
  paging: no  # ticket

Minimum Traffic for SLO Reliability

SLOs on low-traffic services produce noisy signals. A single error on a service receiving 1 request/minute moves the error rate from 0% to 100% for that minute. Rule of thumb: you need at least 100–1,000 events per measurement window for the SLI to be statistically meaningful.

If traffic is too low, consider:

• Synthetic monitoring as the SLI source
• Longer measurement windows (24-hour windows instead of 5-minute windows)
• Count-based SLOs instead of rate-based (alert if more than N errors in a window)

SLOs done well give engineering teams a precise, shared vocabulary for reliability discussions. They make the relationship between toil, incidents, and user experience explicit. They give product managers a principled way to think about risk. And they give on-call engineers something better than gut feel to guide escalation decisions.

The math is straightforward. The tools handle the boilerplate. The hard part is the culture — and the way to build it is to start with a small team, do it right, and let the results speak for themselves.