The USE and RED Methods: Systematic Performance Investigation

April 20, 2026 13 min read The Physics of Computing

sreobservabilityperformancemonitoringmethodology

Contents

The worst kind of performance investigation is one where you’re looking at a Grafana dashboard full of graphs and you don’t know which one to read first. You click around for a while, pull some metrics, notice something that looks suspicious, follow the thread, hit a dead end, start again. Forty minutes later, your understanding of the system is slightly richer and the user is still waiting.

The USE and RED methods are antidotes. They are checklists — methodologies, if you want the fancier word — that tell you exactly what to look at first, in what order, for any performance investigation. Neither is complicated. Both are under-adopted. Together they cover the two most important questions: “is my resource healthy?” (USE, from Brendan Gregg) and “is my service healthy?” (RED, from Tom Wilkie). Applied consistently, they turn a large fraction of performance investigations from improvisation into routine.

This post covers both methods, how they complement each other, how to implement them in a modern observability stack, and where they stop working.

The USE method: resources

Brendan Gregg introduced the USE method around 2012 as a checklist for analyzing systems performance. The premise: for every resource, check three things.

Utilization — the percent of time the resource was busy servicing work.
Saturation — the degree to which the resource has extra work it can’t service (queue length, run queue, pending I/O).
Errors — counts of error events emitted by the resource.

The insight is that these three metrics together describe every plausible resource problem. A resource that’s at 100% utilization with zero saturation is fully used but not overloaded. A resource at 100% utilization with high saturation is a bottleneck. A resource with errors has hardware or driver issues regardless of utilization. And a resource at low utilization with no saturation and no errors is probably not your problem.

The “for every resource” is the discipline. The USE method is not “look at CPU.” It’s “look at U, S, E for every resource in the system.” On a typical Linux host that’s a long list:

CPU — per-core utilization, run-queue length, context-switch-error events.
Memory — utilization, swap-in/swap-out rate (saturation), OOM kill events (errors).
Disk I/O — per-device utilization (iostat’s %util), queue depth (saturation), block-layer errors.
Network interfaces — RX/TX utilization, drop/overrun counters (saturation and errors combined).
Network controllers — interrupt rates (saturation indicator on shared NICs).
Storage controllers / HBAs — bandwidth utilization, queue depths.
Kernel locks — held time (utilization), wait queues (saturation).
File descriptors — current count vs limit, failed opens.
Processes / threads — count vs limit, failed forks.
GPU — compute utilization, memory utilization, throttle events.

The first time you write this list out, you realize how many resources you were implicitly ignoring. Most dashboards look at CPU, memory, and disk — maybe network. A USE-method dashboard for a host looks at 15–20 signals, one row per resource.

Interpreting USE

The utility isn’t just in the checklist — it’s in the patterns of U/S/E readings.

Low U, low S, no E — the resource is healthy. Not your problem; move on.

High U, low S, no E — the resource is fully used but keeping up. Depending on the resource, this is either fine or a warning:

CPU at 100% with zero run queue: you’re CPU-bound, everything that wants CPU is getting it. Increase parallelism or optimize code.
Disk at 100% util with queue depth of 1: single outstanding I/O at a time. Could increase with parallelism.

High U, high S, no E — classic bottleneck. Work is queuing up faster than the resource can service it. This is where latency regressions live. Options: reduce load, speed up the resource, add more of it.

Any U, any S, errors > 0 — errors always deserve investigation regardless of utilization. A disk with correctable errors is fine for now; a disk with uncorrectable errors is dying. Network drops at 0.01% are typically background noise; at 1% they’re killing throughput. Errors are never “normal.”

Low U, high S — oddly common. Usually indicates lock contention, misconfigured concurrency limits, or saturation in a resource you’re not directly measuring. Example: low CPU util but a long run queue means cores are going idle waiting for something (often I/O or a synchronization primitive).

Implementing USE on a Linux host

A concrete set of tools and metrics:

CPU:

U: top/vmstat %us + %sy + %si (avoid %id), or Prometheus 100 - rate(node_cpu_seconds_total{mode="idle"}[1m]) * 100.
S: vmstat r column (run-queue length), uptime load average, Prometheus node_load1.
E: rare in practice but: thermal throttling events, correctable ECC errors on CPU (edac-util, mcelog).

Memory:

U: free -m, node_memory_MemTotal - node_memory_MemAvailable.
S: swap activity (vmstat si/so, node_vmstat_pswpin, node_vmstat_pswpout), major page faults.
E: OOM kills (dmesg, node_vmstat_oom_kill).

Disk:

U: iostat -x %util per device, node_disk_io_time_seconds_total.
S: iostat aqu-sz (average queue size), await (latency).
E: dmesg I/O errors, SMART errors, node_disk_io_now.

Network:

U: ip -s link, sar -n DEV, node_network_transmit_bytes_total and receive_bytes_total / link capacity.
S: netstat -s drops, overruns, node_network_receive_drop_total, TX queue length.
E: framing errors, checksum errors, carrier errors.

A USE dashboard shows U/S/E for every resource in rows. Per resource, a red indicator lights when any column is anomalous. You can glance at it and know whether the host’s resources are healthy without clicking into any individual graph.

USE for non-host resources

USE generalizes to everything. The trick is defining the resource correctly.

Databases — the database itself is a resource with:

U: connection pool usage, buffer pool hit rate (high hit rate = low utilization of the slow storage path), CPU time.
S: connections queueing, lock waits, queries queued in the query scheduler.
E: failed queries, deadlocks, connection errors.

Kafka topics — a topic partition is a resource:

U: partition bytes per second vs broker capacity.
S: consumer lag.
E: offset commit failures, ISR shrinks.

External APIs you depend on — yes, USE applies:

U: requests per second / rate limit.
S: requests that hit retry-after or 429 responses.
E: 5xx responses, network-level failures.

Thinking in USE forces you to confront what “saturation” means for each resource. For many software resources, it means “requests are waiting in a queue” — which is often invisible unless the queue is instrumented. The discipline of asking “what is saturation for this resource?” often reveals that you’re missing the most important metric.

The RED method: services

The RED method, promoted by Tom Wilkie (of Weaveworks and later Grafana Labs), approaches observability from the other end: not “what is this resource doing?” but “what is this service doing?”

For every service endpoint:

Rate — requests per second.
Errors — failed requests per second.
Duration — latency distribution (p50, p95, p99).

That’s it. Three metrics per endpoint. The beauty is that these metrics describe the service from the user’s perspective — not “is the CPU 80% utilized?” but “how many users are being served, how many are failing, and how long are the successful ones waiting?”

The mapping to RED is direct from HTTP access logs, RPC layer metrics, or service mesh telemetry. A typical RED dashboard:

One row per service.
Three panels per row: requests/sec (with a line per status-code group), errors/sec (with a breakdown by error type), p95 and p99 latency (with a line per endpoint).

The point of RED is that you know what “healthy” looks like for any service instantly. Rate up? Traffic increased. Errors up? Something broke. Duration up? Something is slow. A single glance at a RED row answers “is this service OK right now?” — and that’s enough for 80% of operational questions.

RED and Prometheus

The canonical RED implementation in a Prometheus + histogram setup:

# Rate
sum(rate(http_requests_total{service="checkout"}[5m])) by (method, status)

# Errors
sum(rate(http_requests_total{service="checkout", status=~"5.."}[5m])) by (method)

# Error rate (percentage)
sum(rate(http_requests_total{service="checkout", status=~"5.."}[5m]))
  / sum(rate(http_requests_total{service="checkout"}[5m]))

# Duration p95
histogram_quantile(0.95,
  sum(rate(http_request_duration_seconds_bucket{service="checkout"}[5m]))
    by (le, method))

# Duration p99
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket{service="checkout"}[5m]))
    by (le, method))

Key points:

Use histograms, not summaries. Summaries compute quantiles per instance, which you can’t aggregate. Histograms expose bucket counts that aggregate cleanly across replicas.
Pick bucket boundaries carefully. Default Prometheus buckets (5ms to 10s, widely spaced) are too coarse for web services. Set buckets tuned to your SLO — if your target p99 is 200ms, have buckets at 50/100/150/200/250/300ms, not 500ms jumps.
Label on status and method, sparingly on endpoint. High-cardinality paths (e.g., /user/{id}) break Prometheus unless you template them. Most frameworks have a way to tag routes with a pattern, not the literal URL.

RED and service meshes

If you have a service mesh (Istio, Linkerd, Consul Connect, Cilium), you get RED metrics for free from the sidecar. No application instrumentation required. This is one of the less-celebrated benefits of service meshes: even without mTLS or traffic policies, the RED signals from the sidecar cover every service-to-service call.

Linkerd’s linkerd viz UI is the cleanest example: every service has a page with exactly RED metrics, and the “success rate” tile is effectively 1 - error_rate. Grafana Tempo + span metrics produces similar data from OpenTelemetry traces.

Where USE and RED meet

USE is about resources. RED is about services. They’re complementary.

When a service has a RED alert:

Rate up unexpectedly → traffic analysis (is this a DDoS, a retry storm, a runaway client?).
Errors up → application-level debugging (logs, traces).
Duration up → USE on the resources the service depends on (CPU, memory, downstream services, database).

The typical failure scenario:

RED alert: service p99 latency spiked.
Check USE on the hosts running the service: CPU saturated? Memory swap? Disk queue depth?
Check USE on downstream resources: database query time up? Cache hit rate down? Message queue backlog?
Either way, you’ve narrowed the cause from “something is slow” to “this specific resource is saturated.”

This pipeline — RED surfaces the symptom, USE identifies the cause — is what makes systematic performance investigation work.

The four golden signals (and why they coexist with RED)

Google’s SRE book introduced the “four golden signals” for service monitoring, which look suspiciously like RED with an extra metric:

Latency — same as Duration.
Traffic — same as Rate.
Errors — same as Errors.
Saturation — a measure of how full the service is.

Saturation is the addition. Google’s point is that a service can have nominal R/E/D but still be on the edge of overload — and the ability to absorb additional traffic matters. For a CPU-bound service, saturation is CPU utilization. For a queue-backed worker, it’s queue depth. For a database, it’s connection pool usage.

In practice, RED and the golden signals converge: most modern dashboards show rate, errors, duration, and a saturation indicator. Call it whichever you want; the important thing is that you have all four.

Adding USE and RED to an existing stack

A realistic adoption path:

RED for top services. Start with the most-user-facing endpoints. Get rate, errors, duration dashboards from your existing telemetry (access logs, histograms, service mesh). Aim to answer “is this service healthy?” in one glance. This is usually the highest-leverage observability work a team can do.
USE dashboards for hosts. The node_exporter Prometheus exporter covers most host resources. Build a USE-organized dashboard (one row per resource, U/S/E columns). Do not try to show every metric; show the U, S, and E for each resource.
USE for key infrastructure. Database (connection pool util, query queue, errors), message queue (utilization vs throughput ceiling, lag, errors), cache (hit rate as inverse util, eviction rate as saturation, errors). This is where problem-finding actually lives.
Unified on-call view. A single page that shows RED for services + USE for resources. When something is wrong, this is the first page on-call opens.
SLOs per service. Once RED is reliable, use the duration and error data to define SLOs (e.g., 99.9% of checkout requests complete under 300ms). Burn-rate alerts on SLOs replace noisy threshold alerts.

Common mistakes

Confusing utilization with load. Utilization is the percentage of time busy. A system at 100% utilization is not necessarily overloaded — it’s just fully engaged. Overload is saturation: work that can’t be serviced. Distinguish these carefully.

Measuring wrong. %util from iostat on SSDs is a lie; modern SSDs can service many concurrent I/Os, and %util hits 100% well below actual saturation. Use queue depth and latency as primary signals on fast storage.

Missing saturation entirely. Most off-the-shelf dashboards show utilization and ignore saturation. A CPU graph showing 70% utilization tells you nothing about whether the run queue is 0 or 500. Always pair utilization with a saturation metric.

Measuring E as a gauge. Errors should be rates, not current counts. “We have 47,000 errors total” means nothing; “we’re seeing 15 errors/sec, up from 0.1/sec an hour ago” is actionable.

Dashboards that show everything. A USE/RED dashboard should be opinionated — a handful of carefully chosen signals. Dashboards with 40 panels are looked at by no one. If every panel matters, split into multiple dashboards.

Alerting on single metrics. Alert on outcomes (SLO burn rate) or multi-signal compound conditions, not “CPU > 80%.” CPU at 80% is often fine; CPU at 80% while user-facing latency is up is not.

An opinionated dashboard template

For a service with RED + host USE:

Top row (RED):

Requests per second, by status code group (stacked area).
Error rate percentage (single number panel + sparkline).
p50 / p95 / p99 latency (line chart with three series).

Middle row (upstream RED):

Same three panels, but for the dependencies: database query rate/error/duration, downstream service RED from traces.

Bottom rows (USE by resource):

CPU: utilization per core, run queue length, throttle events.
Memory: used bytes, swap activity, OOM events.
Network: bytes/sec in/out vs link capacity, drops/errors.
Disk: per-volume util, queue depth, errors.
Service-specific: thread pool utilization, GC pauses (JVM), event loop lag (Node), GIL contention (Python).

Total: 5–8 panels in RED section, 10–15 in USE. Everything on one page. This is the dashboard your on-call opens first.

When the methods don’t apply

USE and RED are checklists, not laws. They’re insufficient or wrong in some cases:

Batch systems. A nightly batch job doesn’t have a rate/error/duration in the RED sense. It has a different set of signals: duration of the run, input size, output correctness.
Stateful systems where “errors” are ambiguous. A database that rejects an invalid query has “errored,” but not in the operational sense. Carefully separate client errors from server errors.
Pipeline systems. A data pipeline’s useful signals are end-to-end latency and backlog at each stage, not service-level RED.
Truly asynchronous work. A job queue worker doesn’t have a “rate” of requests in the same way. Queue depth, processing rate, retry rate are the right signals.

The underlying mental model — “resources have U/S/E; services have R/E/D” — generalizes. For any new system, ask: what are the resources? What are the services? What is utilization, saturation, errors for each resource? What is rate, errors, duration for each service? The specific metric names matter less than the discipline of answering those questions.

Where this gets you

The combination of USE and RED is unusually effective because it’s fast to apply. A developer new to a service who has a RED dashboard in front of them can answer “is this service healthy?” in seconds. If the answer is no, they can answer “what resource is likely to blame?” in under a minute via USE.

This isn’t a replacement for deep investigation. Profilers, traces, logs, and expert knowledge are still necessary. But it’s a startling reduction in how much time is wasted figuring out where to look. The default investigation path is: RED alert → which of R, E, D? → which resource does that signal implicate? → USE on that resource → root cause.

Most incident post-mortems contain a version of “it took us 20 minutes to figure out this was a database problem.” A team with reliable USE + RED dashboards collapses that to 2 minutes. Multiplied across every on-call rotation for the next five years, that’s a career’s worth of hours reclaimed. Build the dashboards.