SLO-as-Code with Sloth and Pyrra: Multi-Window Burn-Rate Alerts, Error Budget Policy, and Grafana Dashboards

Service Level Objectives are the most useful reliability tool most teams are still doing wrong. Not because the math is hard — it isn’t — but because the organizational and tooling setup around them usually breaks down before the SLOs ever make it into production alerting. This post covers the full stack: the conceptual foundations, the multi-window burn-rate alert math from the Google SRE Workbook, two mature SLO-as-code tools (Sloth v0.16 and Pyrra v0.10), complete worked YAML examples, Grafana integration, and the harder-to-automate conversation about what to actually do when the budget runs out.

Part 1: SLO Fundamentals

The Three-Letter Soup: SLI, SLO, SLA

These terms are often conflated, and conflating them causes real problems.

Service Level Indicator (SLI) is the raw measurement — a carefully defined quantitative measure of service behavior. The SLI is a ratio: the number of “good” events divided by the total number of events over a time window. For an HTTP service, a common availability SLI is:

SLI = (requests that returned non-5xx) / (total requests)

A latency SLI might be:

SLI = (requests completing in under 200ms) / (total requests)

Service Level Objective (SLO) is the target you set for the SLI. “99.9% of requests succeed” or “95% of requests complete in under 200ms” are SLOs. The SLO is an internal commitment — it is the threshold below which reliability work must be prioritized.

Service Level Agreement (SLA) is the external, contractual promise made to customers, typically with financial consequences for breach. SLAs should always be set looser than SLOs. If your SLO is 99.9%, your SLA might be 99.5% — the gap gives you room to detect and correct issues before a breach becomes a legal or financial problem.

The key distinction: SLIs are measurements, SLOs are goals, SLAs are contracts. Only SLIs belong in Prometheus.

Error Budgets

The error budget is derived directly from the SLO:

error_budget = 1 - SLO_target

For a 99.9% availability SLO over 30 days:

• Error budget = 0.1% = 43.2 minutes of downtime-equivalent
• Expressed as events: if you receive 10 million requests/month, you have 10,000 “allowed” failures

The error budget reframes reliability from a question of “did we achieve uptime?” to “how much risk capital do we have left to spend?” This reframing is the entire point. With an error budget, you can tell product teams: “We have 60% of this month’s budget remaining. We can afford two more deployments with moderate risk, or one large risky change, but not three large changes in the next two weeks.”

Why SLOs Beat Raw Uptime Metrics

“Five nines” uptime sounds impressive until you realize 99.999% allows 5.25 minutes of downtime per year — but nothing about when it happens. Five minutes of downtime at 2 AM on a Sunday is a very different event than five minutes at peak trading hours.

SLOs, measured with error budgets over rolling windows, capture:

1. User impact — you are measuring actual request outcomes, not whether a health check is green
2. Rate of consumption — burn rate tells you how fast you are spending the budget
3. Remaining runway — teams can make deployment decisions based on remaining budget, not gut feel
4. Cross-team alignment — a single number that means the same thing to product, infra, and on-call

The Four Golden Signals

The Google SRE Book defines four signals that, together, give a complete picture of service health:

Of these, latency and errors are the most directly user-impactful and should be the first SLOs you define. Traffic and saturation are better used as leading indicators and capacity signals than as SLO targets.

Part 2: Multi-Window Multi-Burn-Rate Alerting

Why Single-Threshold Alerts Fail

The naive approach to SLO alerting is: “Alert when error rate exceeds 0.1% over the last hour.” This fails in two directions simultaneously.

A slow leak — say, 0.09% error rate sustained for a week — will never fire the alert but will consume the entire monthly budget by day 10. A brief spike — 5% errors for three minutes — fires the alert, wakes someone up at 2 AM, and turns out to be a transient blip that never recurs and consumed maybe 0.01% of the budget.

The Google SRE Workbook’s solution is multi-window, multi-burn-rate alerting. The key insight is the burn rate concept.

Burn Rate Math

Burn rate measures how fast you are consuming the error budget relative to the steady-state rate that would exactly exhaust it at the end of the window.

A burn rate of 1.0 means you will use up the entire error budget by exactly the end of the 30-day window. A burn rate of 14.4 means you will exhaust the budget in 2.08 days (30 days / 14.4).

For any combination of window length, budget consumption target, and SLO:

burn_rate = (budget_fraction_consumed × total_window_duration) / alert_window_duration

Example: detecting that 2% of the monthly budget is consumed in 1 hour on a 30-day window:

burn_rate = (0.02 × 30 days) / (1 hour)
           = (0.02 × 720 hours) / 1 hour
           = 14.4

In terms of error rate, for a 99.9% SLO:

error_rate_threshold = burn_rate × (1 - SLO)
                     = 14.4 × 0.001
                     = 0.0144  (1.44%)

The Four Alert Tiers (Google SRE Workbook Table 5-8)

The Workbook recommends the following configuration for a 99.9% SLO on a 30-day window:

Each tier uses two windows — a long window to detect the sustained trend and a short window (roughly 1/12 the length) to confirm the issue is current and not a historical artifact that already resolved.

The alert fires only when both windows simultaneously exceed the threshold:

(error_rate[short_window] > burn_rate * (1 - SLO))
AND
(error_rate[long_window] > burn_rate * (1 - SLO))

Alert Logic as PromQL

For a 99.9% SLO service (error budget = 0.001), the page-level tier 1 alert in raw PromQL looks like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# Tier 1: Page — 14.4x burn rate, 2% budget in 1h
(
  sum(rate(http_requests_total{job="myservice", code=~"5.."}[5m]))
  /
  sum(rate(http_requests_total{job="myservice"}[5m]))
) > (14.4 * 0.001)
AND
(
  sum(rate(http_requests_total{job="myservice", code=~"5.."}[1h]))
  /
  sum(rate(http_requests_total{job="myservice"}[1h]))
) > (14.4 * 0.001)

In practice, you never write these by hand. That is what Sloth and Pyrra are for.

Sensitivity vs. False Positive Trade-off

• Higher burn rate threshold (faster alert): catches problems faster, more false positives from transient spikes
• Lower burn rate threshold (slower alert): less noise, but you may burn significant budget before paging anyone
• Shorter short window: faster reset after the issue clears, more susceptible to spikes
• Longer short window: more stable signal, slower to clear

The Workbook’s tier 1 configuration (14.4x, 5m/1h) will wake someone up if you are burning at a rate that would exhaust your budget in two days. It has a 5-minute reset time after the issue clears. This is a reasonable starting point for most services. Start with the Workbook defaults and adjust based on your team’s actual alert-to-incident ratio.

Part 3: Sloth — SLO-as-Code Generator

What Sloth Is

Sloth (v0.16.0, released April 2026) is a CLI tool and Kubernetes controller that takes a human-readable SLO specification and generates the complete set of Prometheus recording rules and multi-window burn-rate alerting rules. You write the SLO intent; Sloth writes the PromQL.

Sloth supports three spec formats:

• Default (version: "prometheus/v1") — YAML files processed by the CLI
• Kubernetes CRD (apiVersion: sloth.slok.dev/v1) — watched by the in-cluster controller
• OpenSLO — vendor-neutral format

And three output backends:

• Prometheus (standard PrometheusRule or raw YAML)
• Thanos / Cortex / Mimir (recording rule groups with appropriate labels)

What Sloth Generates

For each SLO in your spec, Sloth generates three rule groups:

1. SLI rules — recording rules computing error ratios at multiple time windows: 5m, 30m, 1h, 2h, 6h, 1d, 3d, 30d. Named with the pattern slo:sli_error:ratio_rate<window>.

2. Metadata rules — informative metrics for dashboards: slo:objective:ratio, slo:error_budget:ratio, slo:time_period:days, slo:current_burn_rate:ratio, slo:period_burn_rate:ratio.

3. Alert rules — the multi-window, multi-burn-rate alerts for both page and ticket severity.

CLI Usage

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

# Install
curl -fsSL https://github.com/slok/sloth/releases/latest/download/sloth-linux-amd64 -o sloth
chmod +x sloth

# Generate rules from a spec file
sloth generate -i ./slos/myservice.yml

# Output to a specific file
sloth generate -i ./slos/myservice.yml -o ./rules/myservice-slo-rules.yml

# Validate without generating
sloth validate -i ./slos/myservice.yml

# Kubernetes mode: apply a CRD and let the controller handle it
kubectl apply -f ./slos/myservice-crd.yml

# Launch the built-in UI (v0.16+, experimental)
sloth server --prometheus-url=http://prometheus:9090

The validate command is critical for CI pipelines — run it as a pre-commit hook or in your GitOps pipeline to catch malformed SLOs before they reach the cluster.

Default (CLI) Spec Format

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80

version: "prometheus/v1"
service: "myservice"
labels:
  owner: "platform-team"
  repo: "myorg/myservice"
  tier: "1"

slos:
  # --- Availability SLO ---
  - name: "requests-availability"
    objective: 99.9
    description: >
      99.9% of all HTTP requests to myservice return a non-5xx response.
      Excludes /healthz and /readyz endpoints.
    labels:
      category: "availability"
    sli:
      events:
        error_query: >
          sum(rate(http_requests_total{
            job="myservice",
            code=~"5.."
          }[{{.window}}]))
        total_query: >
          sum(rate(http_requests_total{
            job="myservice"
          }[{{.window}}]))
    alerting:
      name: MyServiceHighErrorRate
      labels:
        category: "availability"
      annotations:
        summary: "High error rate on myservice"
        runbook_url: "https://wiki.example.com/runbooks/myservice-error-rate"
      page_alert:
        labels:
          severity: "critical"
          routing_key: "platform-oncall"
      ticket_alert:
        labels:
          severity: "warning"
          slack_channel: "#platform-reliability"

  # --- Latency SLO (using raw SLI) ---
  - name: "requests-latency-p95-200ms"
    objective: 95
    description: >
      95% of successful HTTP requests to myservice complete in under 200ms.
    labels:
      category: "latency"
    sli:
      raw:
        error_ratio_query: |
          1 - (
            sum(rate(http_request_duration_seconds_bucket{
              job="myservice",
              le="0.2",
              code!~"5.."
            }[{{.window}}]))
            /
            sum(rate(http_request_duration_seconds_count{
              job="myservice",
              code!~"5.."
            }[{{.window}}]))
          )
    alerting:
      name: MyServiceHighLatency
      labels:
        category: "latency"
      annotations:
        summary: "High p95 latency on myservice"
        runbook_url: "https://wiki.example.com/runbooks/myservice-latency"
      page_alert:
        labels:
          severity: "critical"
          routing_key: "platform-oncall"
      ticket_alert:
        labels:
          severity: "warning"
          slack_channel: "#platform-reliability"

The {{.window}} placeholder is substituted by Sloth with each of the configured time windows when generating recording rules. For the events SLI type, Sloth divides error_query by total_query. For the raw SLI type, you provide the complete error ratio expression directly — useful for latency SLOs where you are computing a histogram quantile fraction, or when your metrics already express a satisfaction score.

Kubernetes CRD Spec Format

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81

apiVersion: sloth.slok.dev/v1
kind: PrometheusServiceLevel
metadata:
  name: myservice-slos
  namespace: monitoring
  labels:
    app.kubernetes.io/managed-by: sloth
spec:
  service: "myservice"
  labels:
    owner: "platform-team"
    repo: "myorg/myservice"
    tier: "1"

  slos:
    - name: "requests-availability"
      objective: 99.9
      description: "99.9% of HTTP requests return non-5xx."
      labels:
        category: "availability"
      sli:
        events:
          errorQuery: >
            sum(rate(http_requests_total{
              job="myservice",
              code=~"5.."
            }[{{.window}}]))
          totalQuery: >
            sum(rate(http_requests_total{
              job="myservice"
            }[{{.window}}]))
      alerting:
        name: MyServiceHighErrorRate
        labels:
          category: "availability"
        annotations:
          summary: "High error rate on myservice"
          runbook_url: "https://wiki.example.com/runbooks/myservice-error-rate"
        pageAlert:
          labels:
            severity: "critical"
            routing_key: "platform-oncall"
        ticketAlert:
          labels:
            severity: "warning"
            slack_channel: "#platform-reliability"

    - name: "requests-latency-p95-200ms"
      objective: 95
      description: "95% of successful requests complete in under 200ms."
      labels:
        category: "latency"
      sli:
        raw:
          errorRatioQuery: |
            1 - (
              sum(rate(http_request_duration_seconds_bucket{
                job="myservice",
                le="0.2",
                code!~"5.."
              }[{{.window}}]))
              /
              sum(rate(http_request_duration_seconds_count{
                job="myservice",
                code!~"5.."
              }[{{.window}}]))
            )
      alerting:
        name: MyServiceHighLatency
        labels:
          category: "latency"
        annotations:
          summary: "p95 latency exceeding 200ms on myservice"
          runbook_url: "https://wiki.example.com/runbooks/myservice-latency"
        pageAlert:
          labels:
            severity: "critical"
        ticketAlert:
          labels:
            severity: "warning"
            slack_channel: "#platform-reliability"

Note the camelCase field names (errorQuery, totalQuery, pageAlert, ticketAlert, errorRatioQuery) in the Kubernetes CRD format vs. snake_case (error_query, total_query, page_alert, ticket_alert) in the default CLI format.

Generated PrometheusRules Output

Running sloth generate -i myservice.yml for the availability SLO produces output like this (abbreviated to show the structure):

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214

# Code generated by Sloth (v0.16.0): https://sloth.dev
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: sloth-slo-myservice
  namespace: monitoring
  labels:
    app.kubernetes.io/managed-by: sloth
    app.kubernetes.io/component: SLO
spec:
  groups:
    # ------------------------------------------------------------------
    # Group 1: SLI recording rules (one per time window)
    # ------------------------------------------------------------------
    - name: sloth-slo-sli-recordings-myservice-requests-availability
      rules:
        - record: slo:sli_error:ratio_rate5m
          expr: |
            (
              sum(rate(http_requests_total{job="myservice",code=~"5.."}[5m]))
              /
              sum(rate(http_requests_total{job="myservice"}[5m]))
            )
          labels:
            sloth_id: "myservice-requests-availability"
            sloth_service: "myservice"
            sloth_slo: "requests-availability"
            sloth_window: "5m"
            owner: "platform-team"
            category: "availability"

        - record: slo:sli_error:ratio_rate30m
          expr: |
            (
              sum(rate(http_requests_total{job="myservice",code=~"5.."}[30m]))
              /
              sum(rate(http_requests_total{job="myservice"}[30m]))
            )
          labels:
            sloth_id: "myservice-requests-availability"
            sloth_service: "myservice"
            sloth_slo: "requests-availability"
            sloth_window: "30m"
            owner: "platform-team"
            category: "availability"

        - record: slo:sli_error:ratio_rate1h
          expr: |
            (
              sum(rate(http_requests_total{job="myservice",code=~"5.."}[1h]))
              /
              sum(rate(http_requests_total{job="myservice"}[1h]))
            )
          labels:
            sloth_id: "myservice-requests-availability"
            sloth_service: "myservice"
            sloth_slo: "requests-availability"
            sloth_window: "1h"

        - record: slo:sli_error:ratio_rate2h
          expr: |
            (
              sum(rate(http_requests_total{job="myservice",code=~"5.."}[2h]))
              /
              sum(rate(http_requests_total{job="myservice"}[2h]))
            )
          labels:
            sloth_id: "myservice-requests-availability"
            sloth_window: "2h"

        - record: slo:sli_error:ratio_rate6h
          expr: |
            (
              sum(rate(http_requests_total{job="myservice",code=~"5.."}[6h]))
              /
              sum(rate(http_requests_total{job="myservice"}[6h]))
            )
          labels:
            sloth_id: "myservice-requests-availability"
            sloth_window: "6h"

        - record: slo:sli_error:ratio_rate1d
          expr: |
            (
              sum(rate(http_requests_total{job="myservice",code=~"5.."}[1d]))
              /
              sum(rate(http_requests_total{job="myservice"}[1d]))
            )
          labels:
            sloth_id: "myservice-requests-availability"
            sloth_window: "1d"

        - record: slo:sli_error:ratio_rate3d
          expr: |
            (
              sum(rate(http_requests_total{job="myservice",code=~"5.."}[3d]))
              /
              sum(rate(http_requests_total{job="myservice"}[3d]))
            )
          labels:
            sloth_id: "myservice-requests-availability"
            sloth_window: "3d"

    # ------------------------------------------------------------------
    # Group 2: Metadata recording rules (for dashboards)
    # ------------------------------------------------------------------
    - name: sloth-slo-meta-recordings-myservice-requests-availability
      rules:
        - record: slo:objective:ratio
          expr: "vector(0.9990000000000001)"
          labels:
            sloth_id: "myservice-requests-availability"
            sloth_service: "myservice"
            sloth_slo: "requests-availability"

        - record: slo:error_budget:ratio
          expr: "vector(0.0009999999999999)"
          labels:
            sloth_id: "myservice-requests-availability"
            sloth_service: "myservice"
            sloth_slo: "requests-availability"

        - record: slo:time_period:days
          expr: "vector(30)"
          labels:
            sloth_id: "myservice-requests-availability"
            sloth_service: "myservice"
            sloth_slo: "requests-availability"

        - record: slo:current_burn_rate:ratio
          expr: |
            slo:sli_error:ratio_rate5m{sloth_id="myservice-requests-availability"}
            / on(sloth_id) group_left
            slo:error_budget:ratio{sloth_id="myservice-requests-availability"}
          labels:
            sloth_id: "myservice-requests-availability"
            sloth_service: "myservice"
            sloth_slo: "requests-availability"

        - record: slo:period_burn_rate:ratio
          expr: |
            slo:sli_error:ratio_rate30d{sloth_id="myservice-requests-availability"}
            / on(sloth_id) group_left
            slo:error_budget:ratio{sloth_id="myservice-requests-availability"}
          labels:
            sloth_id: "myservice-requests-availability"
            sloth_service: "myservice"
            sloth_slo: "requests-availability"

        - record: slo:period_error_budget_remaining:ratio
          expr: |
            1 - slo:period_burn_rate:ratio{sloth_id="myservice-requests-availability"}
          labels:
            sloth_id: "myservice-requests-availability"
            sloth_service: "myservice"
            sloth_slo: "requests-availability"

    # ------------------------------------------------------------------
    # Group 3: Alert rules (multi-window, multi-burn-rate)
    # ------------------------------------------------------------------
    - name: sloth-slo-alerts-myservice-requests-availability
      rules:
        # Page Tier 1: 14.4x burn rate — 2% budget consumed in 1h
        - alert: MyServiceHighErrorRate
          expr: |
            (
              max(slo:sli_error:ratio_rate5m{sloth_id="myservice-requests-availability"} > (14.4 * 0.0009999999999999))
              and
              max(slo:sli_error:ratio_rate1h{sloth_id="myservice-requests-availability"} > (14.4 * 0.0009999999999999))
            )
            or
            (
              max(slo:sli_error:ratio_rate30m{sloth_id="myservice-requests-availability"} > (6 * 0.0009999999999999))
              and
              max(slo:sli_error:ratio_rate6h{sloth_id="myservice-requests-availability"} > (6 * 0.0009999999999999))
            )
          labels:
            sloth_id: "myservice-requests-availability"
            sloth_service: "myservice"
            sloth_slo: "requests-availability"
            sloth_severity: "page"
            severity: "critical"
            routing_key: "platform-oncall"
            category: "availability"
            owner: "platform-team"
          annotations:
            summary: "High error rate on myservice"
            runbook_url: "https://wiki.example.com/runbooks/myservice-error-rate"
            title: "({{"{{$labels.sloth_service}}"}}) {{"{{"}}$labels.sloth_slo{{"}}"}} SLO error budget burn rate is too fast."

        # Ticket Tier: 3x and 1x burn rates — slower consumption
        - alert: MyServiceHighErrorRate
          expr: |
            (
              max(slo:sli_error:ratio_rate2h{sloth_id="myservice-requests-availability"} > (3 * 0.0009999999999999))
              and
              max(slo:sli_error:ratio_rate1d{sloth_id="myservice-requests-availability"} > (3 * 0.0009999999999999))
            )
            or
            (
              max(slo:sli_error:ratio_rate6h{sloth_id="myservice-requests-availability"} > (1 * 0.0009999999999999))
              and
              max(slo:sli_error:ratio_rate3d{sloth_id="myservice-requests-availability"} > (1 * 0.0009999999999999))
            )
          labels:
            sloth_id: "myservice-requests-availability"
            sloth_service: "myservice"
            sloth_slo: "requests-availability"
            sloth_severity: "ticket"
            severity: "warning"
            slack_channel: "#platform-reliability"
          annotations:
            summary: "High error rate on myservice"
            runbook_url: "https://wiki.example.com/runbooks/myservice-error-rate"

The recording rules in group 1 are what make the alerting rules lightweight. Prometheus evaluates slo:sli_error:ratio_rate5m every 30 seconds as a precomputed value; the alert rule just compares it to a threshold rather than recomputing the full rate expression on every evaluation.

Sloth v0.16 New Features

The v0.16.0 release (April 2026) adds two significant capabilities:

K8s Transformer Plugins let you customize the Kubernetes objects Sloth produces without depending on the prometheus-operator PrometheusRule CRD. The built-in sloth.dev/k8stransform/prom-operator-prometheus-rule/v1 plugin handles the standard case; you can write custom plugins for Thanos Ruler, VictoriaMetrics, or any other Prometheus-compatible backend.

Built-in Web UI via sloth server --prometheus-url=http://prometheus:9090 launches a UI at http://localhost:8080 that shows service listings, SLO state, SLI charts, error budget burn-in-period charts, and alert firing status. It reads directly from Prometheus using the sloth_* labels on the generated recording rules. This is experimental as of v0.16 but a welcome addition for teams not yet running Grafana dashboards.

Sloth Plugin System

Sloth’s plugin architecture allows you to define custom SLI types beyond events and raw. For example, a plugin could handle Kubernetes resource quota utilization, AWS SQS queue depth, or database transaction success rates, wrapping the metric-specific PromQL into a reusable plugin callable from any SLO spec:

1
2
3
4
5
6
7

sli:
  plugin:
    id: "myorg/http-errors-by-route/v1"
    options:
      job: "myservice"
      route_pattern: "/api/v2/.*"
      error_codes: "5..|429"

Plugins are loaded from a directory at startup with --slo-plugins-path.

Part 4: Pyrra — SLO Operator with Built-in UI

What Pyrra Is

Pyrra (v0.10.0, released April 2026) takes a different approach to the same problem. Where Sloth is primarily a generation tool — you run it, it outputs rules — Pyrra is a Kubernetes operator with an API server and UI built in. It watches ServiceLevelObjective CRDs, generates PrometheusRule resources continuously, and serves a dashboard that shows error budget state in real time.

Pyrra’s Two Modes

Kubernetes Operator Mode: Deploy Pyrra as a controller alongside prometheus-operator. Pyrra watches for ServiceLevelObjective CRDs and reconciles PrometheusRule objects. This is the primary mode.

Filesystem Mode: Two containers — one runs the API server, one runs the file reconciler that reads SLO YAML from disk and writes generated rules to a directory Prometheus scrapes. Useful for non-Kubernetes environments or when you want to commit generated rules to git.

Pyrra’s ServiceLevelObjective Spec

Availability (ratio) SLO:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

apiVersion: pyrra.dev/v1alpha1
kind: ServiceLevelObjective
metadata:
  name: myservice-availability
  namespace: monitoring
  labels:
    prometheus: k8s
    role: alert-rules
    pyrra.dev/team: platform
    pyrra.dev/slack_channel: "#platform-reliability"
spec:
  target: "99.9"
  window: 30d
  description: "99.9% of HTTP requests to myservice return non-5xx responses."
  indicator:
    ratio:
      errors:
        metric: http_requests_total{job="myservice", code=~"5.."}
      total:
        metric: http_requests_total{job="myservice"}

Latency SLO (histogram bucket approach):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

apiVersion: pyrra.dev/v1alpha1
kind: ServiceLevelObjective
metadata:
  name: myservice-latency-p95
  namespace: monitoring
  labels:
    prometheus: k8s
    role: alert-rules
    pyrra.dev/team: platform
spec:
  target: "95"
  window: 30d
  description: "95% of successful requests to myservice complete in under 200ms."
  indicator:
    latency:
      success:
        metric: http_request_duration_seconds_bucket{job="myservice", le="0.2", code!~"5.."}
      total:
        metric: http_request_duration_seconds_count{job="myservice", code!~"5.."}

Latency SLO (native histogram, Prometheus v3):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

apiVersion: pyrra.dev/v1alpha1
kind: ServiceLevelObjective
metadata:
  name: myservice-latency-native
  namespace: monitoring
spec:
  target: "95"
  window: 30d
  indicator:
    latencyNative:
      success:
        metric: http_request_duration_seconds{job="myservice", code!~"5..", le="0.2"}
      total:
        metric: http_request_duration_seconds_count{job="myservice", code!~"5.."}

Notable Pyrra-specific details:

• target is a string (not a float), a Kubebuilder CRD limitation
• Labels prefixed with pyrra.dev/ propagate onto generated Prometheus rules with the prefix stripped
• window accepts Prometheus duration strings: 30d, 28d, 2w, 7d
• The indicator.ratio.grouping field generates separate burn rates per label dimension, which is powerful for per-route or per-region SLOs

Pyrra’s Generated Recording Rules

Pyrra generates burn rates at seven time windows: 3m, 15m, 30m, 1h, 3h, 12h, 2d. For a 99.9% availability SLO named myservice-availability:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87

groups:
  - name: myservice-availability-generic
    interval: 30s
    rules:
      - record: pyrra_objective
        expr: "0.999"
        labels:
          slo: myservice-availability

      - record: pyrra_window
        expr: "2592000"   # 30 days in seconds
        labels:
          slo: myservice-availability

  - name: myservice-availability-burnrates
    interval: 30s
    rules:
      - record: http_requests_total:burnrate3m
        expr: |
          sum(rate(http_requests_total{job="myservice",code=~"5.."}[3m]))
          /
          sum(rate(http_requests_total{job="myservice"}[3m]))
        labels:
          slo: myservice-availability

      - record: http_requests_total:burnrate15m
        expr: |
          sum(rate(http_requests_total{job="myservice",code=~"5.."}[15m]))
          /
          sum(rate(http_requests_total{job="myservice"}[15m]))
        labels:
          slo: myservice-availability

      - record: http_requests_total:burnrate30m
        expr: |
          sum(rate(http_requests_total{job="myservice",code=~"5.."}[30m]))
          /
          sum(rate(http_requests_total{job="myservice"}[30m]))
        labels:
          slo: myservice-availability

      # ... similarly for 1h, 3h, 12h, 2d

  - name: myservice-availability-alerts
    rules:
      - alert: ErrorBudgetBurn
        expr: |
          http_requests_total:burnrate3m{slo="myservice-availability"} > (14.4 * (1-0.999))
          and
          http_requests_total:burnrate1h{slo="myservice-availability"} > (14.4 * (1-0.999))
        for: 2m
        labels:
          slo: myservice-availability
          severity: critical
          pyrra_url: "https://pyrra.example.com/objectives/myservice-availability"
        annotations:
          summary: "High error budget burn rate for myservice-availability"

      - alert: ErrorBudgetBurn
        expr: |
          http_requests_total:burnrate30m{slo="myservice-availability"} > (6 * (1-0.999))
          and
          http_requests_total:burnrate6h{slo="myservice-availability"} > (6 * (1-0.999))
        for: 15m
        labels:
          slo: myservice-availability
          severity: critical

      - alert: ErrorBudgetBurn
        expr: |
          http_requests_total:burnrate2h{slo="myservice-availability"} > (3 * (1-0.999))
          and
          http_requests_total:burnrate1d{slo="myservice-availability"} > (3 * (1-0.999))
        for: 1h
        labels:
          slo: myservice-availability
          severity: warning

      - alert: ErrorBudgetBurn
        expr: |
          http_requests_total:burnrate6h{slo="myservice-availability"} > (1 * (1-0.999))
          and
          http_requests_total:burnrate3d{slo="myservice-availability"} > (1 * (1-0.999))
        for: 3h
        labels:
          slo: myservice-availability
          severity: warning

Pyrra v0.10 Notable Features

The v0.10.0 release (April 2026) introduces:

• Performance Mode: Subquery-based SLO calculations instead of recording rules, trading some precision for significantly reduced Prometheus storage and rule evaluation overhead. Enable with --performance-mode on the operator.
• Alert severity overrides: Configure critical vs. warning thresholds directly in the SLO spec.
• Prometheus v3 histogram compatibility: Correct handling of the le label normalization difference between Prometheus v2 and v3 native histograms.
• pyrra_url annotation: All generated alerts include a direct link back to the Pyrra UI page for that SLO, included automatically in the alert annotation. This is an underrated feature for on-call response — the alert fires and the link takes you directly to the error budget chart.
• Redesigned UI: Migrated from Bootstrap to shadcn/ui with Tailwind CSS.

Sloth vs. Pyrra: When to Use Each

Choose Sloth if you want full control over alert routing and annotation, you need custom SLI plugins, you have a GitOps workflow that pre-generates rules, or you are using a non-Kubernetes Prometheus setup.

Choose Pyrra if you want a purpose-built SLO UI with live error budget visualization, you need per-label-dimension SLOs (e.g., separate SLOs per route), your team wants to see error budgets without setting up Grafana dashboards, or you prefer Kubernetes-native reconciliation over a generation step.

Both tools are production-ready and actively maintained. Many teams end up evaluating both and picking based on whether the built-in UI justifies Pyrra’s lower alerting configurability.

Part 5: Grafana SLO Dashboards

The Grafana SLO Plugin (Cloud-First)

The native Grafana SLO feature — available as the grafana-slo-app plugin — is primarily a Grafana Cloud offering. It provides a guided UI for creating SLOs, generates 10–12 Prometheus recording rules per SLO, and creates pre-built dashboards automatically. It supports fast-burn and slow-burn alerts and includes an error budget burndown view.

Important caveat: at the time of writing, the full Grafana SLO application with dashboard reporting is available in Grafana Cloud paid plans and Grafana Enterprise. If you are running self-hosted Grafana OSS, you will need to use community dashboards or import Sloth’s generated dashboards.

Community Dashboards for Self-Hosted Grafana

Two official Sloth dashboards are available on Grafana.com’s dashboard library:

• High-level Sloth SLOs (Dashboard ID 14643): Overview across all services — bargauge, stat, state-timeline, table, and timeseries panels showing SLO status at a glance.
• SLO / Detail (Dashboard ID 19148): Per-SLO detail view with error budget remaining, burn rate over time, and SLI trend.

Import these via Grafana’s “Import dashboard” UI using the dashboard IDs, or download the JSON and commit it to your dashboard-as-code repository.

Key Dashboard Panels and Their Queries

These panels cover the essential SLO visibility requirements. All queries assume Sloth’s sloth_id label identifies the SLO.

Error budget remaining (gauge + time series)

1
2
3
4

# Current error budget remaining as a percentage
(
  1 - slo:period_burn_rate:ratio{sloth_id="myservice-requests-availability"}
) * 100

Display as a gauge with thresholds: green above 50%, yellow 25–50%, red below 25%.

Current burn rate (normalized)

1
2

# Current burn rate as a multiple of steady state (1.0 = exactly on target)
slo:current_burn_rate:ratio{sloth_id="myservice-requests-availability"}

A value of 1.0 means you will use the budget by end of window. Above 14.4 means you will exhaust it in about 2 days.

SLI over time

1
2

# Error ratio over time (lower is better)
slo:sli_error:ratio_rate5m{sloth_id="myservice-requests-availability"}

Plot as a time series. Overlay the SLO threshold as a horizontal reference line: 1 - slo:objective:ratio{sloth_id="myservice-requests-availability"}.

Multi-window burn rates (combined panel)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# 5-minute window burn rate
slo:sli_error:ratio_rate5m{sloth_id="myservice-requests-availability"}
  / on(sloth_id) group_left slo:error_budget:ratio{sloth_id="myservice-requests-availability"}

# 1-hour window burn rate  
slo:sli_error:ratio_rate1h{sloth_id="myservice-requests-availability"}
  / on(sloth_id) group_left slo:error_budget:ratio{sloth_id="myservice-requests-availability"}

# 6-hour window burn rate
slo:sli_error:ratio_rate6h{sloth_id="myservice-requests-availability"}
  / on(sloth_id) group_left slo:error_budget:ratio{sloth_id="myservice-requests-availability"}

Plot all three on one graph. The divergence between short and long windows is the diagnostic signal: a spike in 5m with flat 6h means a transient issue; rising 6h with flat 5m means recovery is underway.

Error budget burn alert timeline

1
2
3
4
5

# Show which burn-rate alerts are currently firing
ALERTS{
  alertname=~"MyServiceHighErrorRate|MyServiceHighLatency",
  alertstate="firing"
}

Use a state-timeline panel to show alert firing history over the current SLO window.

Projected budget exhaustion

1
2
3
4
5

# Days until budget exhaustion at current burn rate
(
  slo:period_error_budget_remaining:ratio{sloth_id="myservice-requests-availability"}
  / slo:current_burn_rate:ratio{sloth_id="myservice-requests-availability"}
) * slo:time_period:days{sloth_id="myservice-requests-availability"}

Pyrra’s Built-in UI

Pyrra’s UI (accessible after deployment at http://pyrra-service:9099) provides:

• Objective list view with status indicators and remaining budget percentages
• Per-SLO detail page with error budget burndown chart, burn rate at each window, and firing alert state
• Direct alert-to-SLO linking via pyrra_url annotations (v0.10+)

For teams already running Pyrra, this is often sufficient for day-to-day SLO visibility without Grafana dashboard setup. Grafana’s deeper integration (cross-service views, custom annotations, historical analysis) still has advantages for larger organizations.

Part 6: What Your Application Needs to Expose

For the availability and latency SLOs in this post to work, your application needs to expose these Prometheus metrics:

# Request counter — total requests by status code
http_requests_total{job="myservice", code="200"} 42314
http_requests_total{job="myservice", code="500"} 17

# Request duration histogram — required for latency SLOs
# Expose buckets that cover your SLO threshold (e.g., le="0.2" for 200ms)
http_request_duration_seconds_bucket{job="myservice", le="0.05", code="200"} 12000
http_request_duration_seconds_bucket{job="myservice", le="0.1", code="200"} 28000
http_request_duration_seconds_bucket{job="myservice", le="0.2", code="200"} 40000
http_request_duration_seconds_bucket{job="myservice", le="0.5", code="200"} 42000
http_request_duration_seconds_bucket{job="myservice", le="+Inf", code="200"} 42314
http_request_duration_seconds_sum{job="myservice", code="200"} 4231.4
http_request_duration_seconds_count{job="myservice", code="200"} 42314

Critical: you must have a histogram bucket at le="0.2" (or whatever your SLO threshold is) for histogram quantile-based latency SLOs to work correctly. If the bucket does not exist, the SLI will always be 0 or 1 and will not track real latency distribution. Choose your histogram bucket boundaries before writing the SLO spec.

Useful Standalone PromQL Queries

Current availability over the last 30 days:

1
2
3
4
5

1 - (
  sum(increase(http_requests_total{job="myservice", code=~"5.."}[30d]))
  /
  sum(increase(http_requests_total{job="myservice"}[30d]))
)

Current p95 latency:

1
2
3
4
5

histogram_quantile(0.95,
  sum by (le) (
    rate(http_request_duration_seconds_bucket{job="myservice"}[5m])
  )
)

Error budget remaining (manual calculation):

1
2
3
4
5
6
7
8
9

1 - (
  (1 - (
    sum(rate(http_requests_total{job="myservice", code!~"5.."}[30d]))
    /
    sum(rate(http_requests_total{job="myservice"}[30d]))
  ))
  /
  (1 - 0.999)
)

Burn rate at a given window:

1
2
3
4
5
6
7

# Burn rate over last 1h (expressed as multiple of budget rate)
(
  sum(rate(http_requests_total{job="myservice", code=~"5.."}[1h]))
  /
  sum(rate(http_requests_total{job="myservice"}[1h]))
)
/ (1 - 0.999)

Part 7: Error Budget Policy

What It Is

An error budget policy is a written agreement that defines what the organization does at each level of budget consumption. Without it, the error budget is just a number in Prometheus that nobody acts on. With it, teams have a pre-negotiated framework for trading off feature velocity against reliability work.

Policy Template

The following template is adapted from the Google SRE Workbook’s example policy:

Error Budget Policy — <Service Name>

Owner: <engineering team> Stakeholders: <product, infra, SRE> SLO Window: Rolling 30 days Review cadence: Monthly SLO review meeting

Status Tiers

Escalation

Any single incident consuming more than 20% of the monthly error budget requires a post-mortem with at least one committed action item. Recurring incident classes consuming 20%+ of monthly budget require planned sprint work to address root causes.

Exceptions

• Planned maintenance windows: excluded from budget consumption with advance notice and customer communication
• Infrastructure failures outside team control (network, cloud provider): exempted at the discretion of the on-call engineer with documented reasoning
• Out-of-scope traffic (scrapers, abuse): may be excluded with filtering rules

Disputes

Budget calculation disputes escalate to <engineering manager / SRE lead / CTO as appropriate>.

The Organizational Conversation

The policy document is the easy part. The harder conversations are:

Who sets the SLO? The right answer is: engineers who understand the technical constraints and product managers who understand user tolerance, together. SLOs set without product input become engineering exercises with no business meaning. SLOs set without engineering input become aspirational numbers that ignore infrastructure reality.

Who owns the error budget? The team that can act on it. If your app team cannot deploy without infra approval, and infra outages consume the app team’s budget, you have a governance problem that no SLO tool can fix. The team that owns the budget must have the authority and capability to protect it.

What happens when infra causes app SLO misses? This is the most common point of conflict. The clean answer is: the app team’s SLO measures what the user experiences, regardless of cause. The infra team has its own SLO on the infrastructure services. If the infra SLO misses cause app SLO misses, that is a prioritization signal, not a blame assignment. Track the source in post-mortems; do not exclude events from the SLO window after the fact.

The too-tight SLO trap. A 100% SLO makes every deployment a potential SLO breach. Teams respond by either deploying less frequently (reducing velocity) or ignoring the SLO entirely (destroying trust in the metric). Neither is useful. Start with a realistic SLO based on historical performance with some headroom. You can tighten it later as reliability improves. It is much harder to loosen an SLO after you have told customers you target 99.99%.

Part 8: Common Mistakes

Setting SLOs Too Tight (or Too Loose)

The 100% trap is well-known but 99.9% is often too tight for teams that are still building reliability infrastructure. Look at your last 90 days of actual availability and set the SLO slightly below the measured floor. If you have been achieving 99.7% naturally, a 99.5% SLO gives you a meaningful error budget and a realistic bar to hold. Once you have the tooling, runbooks, and culture in place, move to 99.9%.

Measuring at the Wrong Point

Server-side metrics miss client-side failures: DNS resolution failures, TLS handshake timeouts, and load balancer drops never appear in your application metrics. Client-side measurement (via synthetic monitoring or real user monitoring) gives a more accurate picture of what users experience. The ideal setup: server-side SLOs as your primary signal (fast, cheap, high resolution), synthetic monitoring as a secondary check, and real user monitoring for the ground truth.

Confusing Availability with Reliability

A service that returns 200 OK with a corrupted response body is “available” by error-rate metrics but not reliable. Define your SLI carefully: “non-5xx responses” is not always sufficient. Consider whether slow responses should count as failures (they usually should for latency SLOs), whether partial failures count, and whether specific error codes (like 429 Rate Limited) should be included in the error count.

SLOs Without Consequences

The most common SLO failure mode is: team implements Sloth, dashboards go green, nobody looks at them until something breaks. SLOs only produce value when they drive decisions. The error budget policy is the mechanism that connects the number to behavior. If your weekly sprint planning does not reference the error budget, your SLOs are decoration.

Missing Latency SLOs

Most teams start with availability SLOs because they are easier to reason about. But a service that returns 200 OK in 30 seconds is not meeting user expectations. Add latency SLOs early. Track p95 and p99, not just average — the average can look fine while the 99th percentile makes the service unusable for a meaningful fraction of users.

Not Excluding Planned Maintenance

Prometheus does not automatically know about maintenance windows. Without alertmanager silences or explicit recording rule modifications, your error budget will be consumed by planned downtime. Set AlertManager silences before maintenance windows, and consider whether your SLO window should use avg_over_time with a presence-weighted approach for services with legitimate planned downtime.

Alert Fatigue from Poorly Tuned Burn Rates

The tier 1 page alert (14.4x burn rate) is aggressive. If your service has frequent transient spikes that resolve within minutes, you will get frequent false pages. Track your alert-to-incident ratio for the first month. If tier 1 pages more than twice without a real incident, raise the burn rate threshold to 28 or add a for: 2m clause to require the condition to hold for two minutes before alerting. The Workbook defaults are a starting point, not a law.

Part 9: Integrating with Incident Management

Routing Burn-Rate Alerts

Configure Alertmanager to route SLO alerts by sloth_severity (Sloth) or severity (Pyrra):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

# alertmanager.yml (excerpt)
route:
  receiver: "default"
  group_by: ["alertname", "sloth_service"]
  routes:
    - match:
        sloth_severity: "page"
      receiver: "pagerduty-oncall"
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 1h

    - match:
        sloth_severity: "ticket"
      receiver: "slack-tickets"
      group_wait: 5m
      group_interval: 30m
      repeat_interval: 24h

receivers:
  - name: "pagerduty-oncall"
    pagerduty_configs:
      - routing_key: "<PAGERDUTY_KEY>"
        description: '{{ template "pagerduty.default.description" . }}'
        details:
          slo: '{{ .Labels.sloth_slo }}'
          service: '{{ .Labels.sloth_service }}'
          burn_rate: '{{ .Labels.sloth_severity }}'
          runbook: '{{ .Annotations.runbook_url }}'

Linking SLO Dashboards in Runbooks

Every page_alert or ticket_alert should include a runbook_url annotation pointing to the relevant runbook. The runbook itself should link to:

1. The Grafana SLO detail dashboard for the service (filtered to the relevant sloth_id)
2. The Pyrra SLO page (if using Pyrra; the pyrra_url annotation handles this automatically in v0.10+)
3. The error budget burn chart with a pre-set time range covering the current SLO window

The SLO Review Meeting

Run a monthly SLO review with these agenda items:

1. Budget report: Which services are green/yellow/red? Any that were exhausted?
2. Top contributors: What incidents consumed the most budget? (Source from post-mortems)
3. Target review: Are current SLO targets still appropriate? Too tight? Too loose?
4. Action items: What reliability work is committed for next month based on current budget state?
5. Policy compliance: Did we follow the error budget policy when budget was low?

The review cadence matters. Monthly is the minimum. For high-traffic services with fast budget consumption, weekly reviews of the burn rate trend (not the full review) are worthwhile.

Using Error Budget Depletion as Escalation Signal

Consider a secondary escalation: when the period burn rate indicates the budget will be exhausted within 72 hours at the current rate, send a Slack notification to the engineering manager and product manager — not just the on-call engineer. This creates the organizational pressure to have the error budget policy conversation before the budget actually hits zero.

1
2
3
4
5

# Alert: budget will exhaust within 72 hours at current rate
(
  slo:period_error_budget_remaining:ratio{sloth_id="myservice-requests-availability"}
  / slo:current_burn_rate:ratio{sloth_id="myservice-requests-availability"}
) * slo:time_period:days{sloth_id="myservice-requests-availability"} < 3

Putting It All Together: GitOps Workflow

The recommended workflow with Sloth:

1. Engineer writes SLO spec YAML in /slos/myservice.yml
2. CI pipeline runs: sloth validate -i slos/myservice.yml
3. On merge, CI runs: sloth generate -i slos/myservice.yml -o rules/myservice-slo-rules.yml
4. Generated PrometheusRules YAML committed to git (or applied directly to cluster)
5. Prometheus-operator picks up the PrometheusRule CRD
6. Prometheus evaluates recording rules every 30s; alerting rules reference pre-computed metrics
7. Alertmanager routes page vs. ticket alerts
8. Sloth server / Grafana dashboard show error budget state
9. Monthly SLO review uses burn rate history to drive reliability prioritization

With Pyrra, steps 2–4 are replaced by kubectl apply -f slos/myservice-crd.yml and the operator handles reconciliation continuously.

The key principle: the SLO spec is the source of truth, committed to version control, reviewed like code, and validated before merging. The Prometheus rules are an artifact, not a hand-written config.

Version Reference

Sources

• Google SRE Workbook — Alerting on SLOs
• Google SRE Workbook — Error Budget Policy
• Sloth GitHub repository (slok/sloth)
• Sloth official documentation
• Sloth v0.16.0 release discussion
• Pyrra GitHub repository (pyrra-dev/pyrra)
• Pyrra official site
• Grafana SLO plugin documentation
• Grafana Cloud SLO documentation
• Service Level Objectives made easy with Sloth and Pyrra (0xdc.me)
• The SLO Toolkit: Setup & Alerting with Pyrra (tb.lx insider)
• Using Sloth to Monitor SLOs Easily (KINTO Tech Blog)
• How to implement multi-window multi-burn-rate alerts with Grafana Cloud
• High level Sloth SLOs — Grafana Dashboard 14643
• SLO / Detail — Grafana Dashboard 19148
• Common SLO pitfalls and how to avoid them (Dynatrace)
• Error Budget Policy for Service Reliability (Google SRE)
• SLO Reporting Frameworks: Pyrra vs. SloK (TECHVZERO)