Pyroscope: Continuous Profiling in Production

Metrics tell you something is slow. Traces tell you where in the request the slowness happened. But neither tells you why the code in that span is slow — which function is burning CPU, where memory is leaking, which lock is causing contention. That’s what profiling is for.

The traditional approach is on-demand profiling: trigger a pprof capture when something goes wrong, stare at the flame graph, try to reproduce the problem. This works for obvious bugs in development but fails in production, where the worst issues are transient, non-reproducible under artificial load, or only visible at specific traffic patterns.

Continuous profiling solves this by collecting profiles constantly — low-overhead snapshots every few seconds, stored with timestamps. When a performance regression happens, you already have the data. No need to reproduce the problem with a profiler attached.

Grafana Pyroscope is the open-source platform for this. It’s the product of Grafana Labs acquiring the original Pyroscope project in March 2023 and merging it with their own in-house continuous profiling database, Grafana Phlare. The result is a mature, production-ready profiling backend with 11,000+ GitHub stars, full Grafana integration, and support for Go, Java, Python, Ruby, .NET, Node.js, and Rust.

Continuous vs On-Demand Profiling

The key difference is data availability:

The overhead comparison is the key insight. A pprof capture is safe to run briefly but expensive to run constantly. Modern continuous profilers use adaptive sampling and efficient encoding to keep overhead at 1–3% CPU and under 50 MB of memory per application instance. For eBPF-based profiling (no application instrumentation required), overhead drops below 1%.

The ROI case is straightforward: if continuous profiling reveals a 5% CPU optimization, it pays for itself in the first week. More importantly, it often finds 20–40% optimizations that were invisible without it.

Architecture

Pyroscope has three components that can be mixed and matched:

The Pyroscope server ingests profiles, stores them in a time-series database, and serves the query/visualization UI. It runs in two modes:

• Monolithic — single binary running all components (development, homelab, small teams)
• Microservices — Distributor, Ingester, Querier, Compactor split across separate processes (production scale)

Application SDKs (push model) — your application imports a language SDK that periodically samples its own runtime and pushes profiles to the Pyroscope server. Full control, custom labels, works anywhere.

Grafana Alloy (pull and eBPF models) — an agent that either scrapes profiling endpoints from applications (pyroscope.scrape) or uses Linux eBPF to capture profiles from any process without code changes (pyroscope.ebpf).

Most production setups use a combination: SDKs for languages that need custom labels or aren’t well-supported by eBPF, and Alloy’s eBPF profiler for infrastructure-level visibility.

Supported Languages

Push model (application SDKs):

eBPF model (via Grafana Alloy, zero instrumentation): Works best for compiled languages: C/C++, Go, Rust. Python works with frame pointers enabled. Ruby and JavaScript/Node.js have limited fidelity — the eBPF profiler captures the interpreter’s stack frames rather than the actual application functions.

Requirements for eBPF: Linux kernel 5.8+, Alloy running as root with access to the host PID namespace.

Deploying Pyroscope

Docker Compose (Homelab / Development)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

services:
  pyroscope:
    image: grafana/pyroscope:latest
    container_name: pyroscope
    ports:
      - "4040:4040"
    command:
      - -config.file=/etc/pyroscope/config.yaml
      - -config.expand-env=true
    volumes:
      - ./pyroscope-config.yaml:/etc/pyroscope/config.yaml
      - pyroscope-data:/var/lib/pyroscope
    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3000:3000"
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    volumes:
      - grafana-data:/var/lib/grafana
    restart: unless-stopped

volumes:
  pyroscope-data:
  grafana-data:

Minimal pyroscope-config.yaml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

storage:
  backend: filesystem
  filesystem:
    dir: /var/lib/pyroscope

compactor:
  blocks_retention_period: 720h  # 30 days

server:
  http_listen_port: 4040

self_profiling:
  disable_push: true  # Don't profile Pyroscope itself

Pyroscope’s web UI is at http://localhost:4040. Add it as a Grafana datasource (type: Grafana Pyroscope) at http://pyroscope:4040.

With S3 Storage

For persistent production storage backed by MinIO or AWS S3:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

storage:
  backend: s3
  s3:
    bucket: pyroscope-profiles
    endpoint: ${S3_ENDPOINT}
    region: ${S3_REGION}
    access_key_id: ${S3_ACCESS_KEY}
    secret_access_key: ${S3_SECRET_KEY}
    insecure: false
    force_path_style: true  # Required for MinIO

compactor:
  blocks_retention_period: 2160h  # 90 days

Use -config.expand-env=true to expand ${VARIABLE} references from environment variables.

Kubernetes with Helm

1
2
3
4
5
6

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm install pyroscope grafana/pyroscope \
  --namespace monitoring \
  --create-namespace \
  -f values.yaml

values.yaml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

pyroscope:
  structuredConfig:
    compactor:
      blocks_retention_period: 720h
    storage:
      backend: s3
      s3:
        bucket: pyroscope-profiles
        endpoint: s3.amazonaws.com
        region: us-east-1
  extraEnvVars:
    - name: S3_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: pyroscope-s3
          key: access_key
    - name: S3_SECRET_KEY
      valueFrom:
        secretKeyRef:
          name: pyroscope-s3
          key: secret_key

alloy:
  enabled: true  # Deploy Grafana Alloy as a collector

grafana:
  enabled: false  # Use your existing Grafana instance

Instrumenting Applications

Go: Push Model with SDK

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

package main

import (
    "context"
    "log"
    "os"

    "github.com/grafana/pyroscope-go"
)

func main() {
    profiler, err := pyroscope.Start(pyroscope.Config{
        ApplicationName: "my-service",
        ServerAddress:   "http://pyroscope:4040",
        Logger:          pyroscope.StandardLogger,
        Tags: map[string]string{
            "env":     os.Getenv("ENV"),
            "version": os.Getenv("VERSION"),
            "region":  os.Getenv("REGION"),
        },
        ProfileTypes: []pyroscope.ProfileType{
            pyroscope.ProfileCPU,
            pyroscope.ProfileAllocObjects,
            pyroscope.ProfileAllocSpace,
            pyroscope.ProfileInuseObjects,
            pyroscope.ProfileInuseSpace,
            pyroscope.ProfileGoroutines,
            pyroscope.ProfileMutexCount,
            pyroscope.ProfileMutexDuration,
            pyroscope.ProfileBlockCount,
            pyroscope.ProfileBlockDuration,
        },
    })
    if err != nil {
        log.Fatal(err)
    }
    defer profiler.Stop()

    // Your application code here
}

Tags are labels that let you filter profiles in the UI. env, version, and region are the most useful — they let you compare profiles across deployments.

Dynamic labeling — tag profiles with request-specific context:

1
2
3
4
5
6

func handleRequest(ctx context.Context, userID string) {
    defer pyroscope.TagWrapper(ctx, pyroscope.Labels("user_id", userID), func(ctx context.Context) {
        // Code in this block is profiled with the user_id label
        processRequest(ctx)
    })()
}

Java: Push Model

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

import io.pyroscope.javaagent.PyroscopeAgent;
import io.pyroscope.javaagent.config.Config;
import io.pyroscope.lang.Pyroscope;

PyroscopeAgent.start(
    new Config.Builder()
        .setApplicationName("my-java-service")
        .setProfilingEvent(EventType.ITIMER)
        .setServerAddress("http://pyroscope:4040")
        .setLabels(Map.of(
            "env", System.getenv("ENV"),
            "version", System.getenv("VERSION")
        ))
        .build()
);

Or as a JVM agent (no code changes):

java -javaagent:pyroscope.jar \
     -Dpyroscope.application.name=my-service \
     -Dpyroscope.server.address=http://pyroscope:4040 \
     -jar myapp.jar

Python: Push Model

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

import pyroscope

pyroscope.configure(
    application_name="my-python-service",
    server_address="http://pyroscope:4040",
    tags={
        "env": os.environ.get("ENV", "dev"),
        "version": os.environ.get("VERSION", "unknown"),
    }
)

Pull Model via Grafana Alloy (No Code Changes)

For Go applications that expose /debug/pprof (which they do by default when you import net/http/pprof):

// In your Alloy config.alloy

discovery.kubernetes "pods" {
  role = "pod"
}

// Filter to pods with the profiling annotation
discovery.relabel "profiling" {
  targets = discovery.kubernetes.pods.targets
  rule {
    source_labels = ["__meta_kubernetes_pod_annotation_profiles_grafana_com_cpu_scrape"]
    action        = "keep"
    regex         = "true"
  }
  rule {
    source_labels = ["__meta_kubernetes_namespace"]
    target_label  = "namespace"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_name"]
    target_label  = "pod"
  }
}

pyroscope.scrape "go_pods" {
  targets    = discovery.relabel.profiling.output
  forward_to = [pyroscope.write.default.receiver]

  profiling_config {
    profile.cpu { enabled = true }
    profile.memory { enabled = true }
    profile.goroutine { enabled = true }
    profile.mutex { enabled = true }
    profile.block { enabled = true }
  }
}

pyroscope.write "default" {
  endpoint {
    url = env("PYROSCOPE_URL")
  }
}

Annotate your Kubernetes pods to opt into profiling:

1
2
3
4

metadata:
  annotations:
    profiles.grafana.com/cpu.scrape: "true"
    profiles.grafana.com/memory.scrape: "true"

eBPF Model via Grafana Alloy (Zero Instrumentation)

Profiles every process on the host without any code changes:

pyroscope.ebpf "system" {
  forward_to = [pyroscope.write.default.receiver]

  targets_only         = false  // Profile all processes
  collect_interval     = "15s"
  sample_rate          = 97     // Stack traces per second
  demangle             = "full" // Demangle C++ symbols

  // Optionally filter to specific processes
  // targets = [{"__process_pid__" = "1234"}]
}

pyroscope.write "default" {
  endpoint {
    url = env("PYROSCOPE_URL")
  }
}

The eBPF profiler requires Alloy to run as root with hostPID: true in Kubernetes:

1
2
3
4
5
6
7

# In Helm values.yaml
alloy:
  controller:
    type: DaemonSet
  securityContext:
    privileged: true
  hostPID: true

Reading Flame Graphs

The flame graph is the primary visualization. Reading it:

• Width of a bar — proportional to time spent in that function (wider = more time)
• Stack depth — Y-axis shows the call hierarchy; bottom is the root, top is the leaf where time is actually spent
• Click to zoom — click any function to focus on that subtree
• Search — highlight all occurrences of a function across the call graph

The profile type changes what the width represents:

Detecting a memory leak: Compare the in-use heap profile at two points in time. Functions whose bars widen are allocating memory that isn’t being freed. The flame graph shows exactly where the allocation originates.

Detecting goroutine leaks: The goroutine profile counts active goroutines and shows where they’re blocked. If the goroutine count grows over time and the profile shows goroutines stuck in select{} or channel receive, you have a leak.

Correlating Profiles with Traces

This is the most powerful feature of the Grafana observability stack: clicking a slow trace span and seeing the flame graph for exactly that span’s duration.

How it works:

1. Your application propagates trace context (OpenTelemetry trace ID) into both the trace (sent to Tempo) and the profile labels (sent to Pyroscope)
2. When you view a slow trace in Grafana, Pyroscope is queried for profiles matching that trace ID and time window
3. The resulting flame graph shows only the CPU/memory consumed during that specific request

Go configuration:

1
2
3
4
5
6
7
8

import (
    otelpyroscope "github.com/grafana/otel-profiling-go"
    "go.opentelemetry.io/otel"
)

// Wrap your OTel tracer provider
tp = otelpyroscope.NewTracerProvider(tp)
otel.SetTracerProvider(tp)

With this in place, the trace ID from each request is embedded as a label in the corresponding profile segment.

Grafana configuration:

In your Tempo datasource settings, add a “Trace to profiles” link pointing at your Pyroscope datasource. When you’re browsing a trace, a “Profiles” button appears on each span — click it to see the flame graph for that span’s exact duration.

The workflow in practice:

1. Latency alert fires — P99 response time exceeds SLO
2. Open Tempo: find a slow trace from the alert time window
3. Identify the slow span (say, a database query function taking 3 seconds)
4. Click “Profiles” → flame graph shows that 80% of time is in JSON marshaling before the query, not in the query itself
5. The trace said “database was slow” — the profile says “actually, it was JSON encoding”

This kind of root cause identification would have taken hours of guesswork before. With correlated profiles and traces, it’s a few clicks.

Correlating with Metrics and Logs

Profiles → Metrics: When you see a CPU profile spike at 14:32, switch to your metrics dashboard and check request rate, error rate, and resource utilization at 14:32. The profile tells you what code was running; the metrics tell you what was happening at the application level.

Metrics → Profiles: Conversely, when a dashboard shows a memory usage spike, click through to Pyroscope at that timestamp to see which functions allocated the memory.

Logs → Profiles: When logs show elevated error rates, profile data at that time window often shows unusual code paths being executed — request types that were rare before suddenly becoming common.

The Grafana “Explore Profiles” UI makes this cross-signal navigation natural — you can switch between signals while keeping the same time range pinned.

Storage and Retention

For homelab or small team deployments, local disk storage is straightforward:

1
2
3
4
5
6
7
8

storage:
  backend: filesystem
  filesystem:
    dir: /var/lib/pyroscope

compactor:
  blocks_retention_period: 720h   # 30 days
  deletion_delay: 2h              # Grace period before deleting marked blocks

For production, S3-compatible storage decouples data from the Pyroscope process:

1
2
3
4
5
6
7
8

storage:
  backend: s3
  s3:
    bucket: pyroscope-profiles
    endpoint: s3.us-east-1.amazonaws.com
    region: us-east-1
    access_key_id: ${S3_ACCESS_KEY}
    secret_access_key: ${S3_SECRET_KEY}

MinIO works for self-hosted S3-compatible storage:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

storage:
  backend: s3
  s3:
    bucket: pyroscope
    endpoint: http://minio:9000
    region: us-east-1
    access_key_id: ${MINIO_ACCESS_KEY}
    secret_access_key: ${MINIO_SECRET_KEY}
    insecure: true
    force_path_style: true  # Required for MinIO

Multi-Tenancy

Enable isolation between teams or environments:

1

multitenancy_enabled: true

When enabled, all requests to Pyroscope must include an X-Scope-OrgID header identifying the tenant. Without it, requests are rejected. Authentication and authorization (deciding who gets which tenant ID) are handled by a reverse proxy — Pyroscope only checks that the header exists.

Common pattern: Traefik middleware reads a JWT, extracts the team name, and adds X-Scope-OrgID: team-name before forwarding to Pyroscope. Each team sees only their own profiles.

Comparison with Alternatives

Choose Pyroscope when you’re already in the Grafana stack (Loki, Tempo, Mimir), need multi-language support, or want trace-to-profile correlation without vendor lock-in.

Choose Parca if you want the lightest possible overhead and only need infrastructure-level (C/C++/Go/Rust) profiling with eBPF — it’s the CNCF-incubating alternative.

Use async-profiler directly for deep JVM profiling during incident investigation — it’s not continuous, but it provides more detail than continuous profilers for Java-specific issues.

Performance Overhead

The numbers from Grafana Labs’ production measurements:

The eBPF profiler’s memory is higher because it caches symbol tables for every profiled process in /tmp/symb-cache. First startup is slower while the cache is populated; subsequent restarts are fast.

For Go applications specifically, enabling all profile types adds about 1–2% CPU overhead over baseline. This is the cost of having goroutine, mutex, and block profiles always available — each has independent value and is hard to enable retrospectively during an incident.

Safety mechanisms: The Pyroscope SDK never blocks the application if the Pyroscope server is unavailable. If the backend is down or overloaded, new profiles are discarded rather than buffered indefinitely. The profiler will not cause your application to OOM.

Gotchas

eBPF doesn’t work well for JIT-compiled languages. Ruby and JavaScript/Node.js show the interpreter’s internal function names rather than your application code. Use the SDK push model for these languages.

Python eBPF profiling requires frame pointers. CPython must be compiled with --enable-profiling or you need a distribution that includes frame pointers. Many system-package Python builds don’t. Check with python3 -c "import sys; print(sys.version)" — if it’s a stripped build, the eBPF profiler shows [unknown] frames.

Multi-tenant queries are per-tenant only. You can’t aggregate across tenants in a single query. If you need cross-team visibility, keep everything in the same tenant or maintain a separate aggregation layer.

Only the last two minor versions receive security fixes. Plan to upgrade regularly. Running Pyroscope 1.3 while 1.7 is current means known vulnerabilities won’t be patched for you.

Trace-to-profile correlation requires explicit SDK setup. You need to import otel-profiling-go (or equivalent for your language) and wrap your tracer provider. The standard Pyroscope SDK alone doesn’t embed trace IDs into profiles — the OTel integration does.

The Pyroscope server UI is separate from Grafana. The built-in UI at :4040 is useful for quick checks, but most teams do their profiling work inside Grafana using the Pyroscope datasource and the Profiles Drilldown app. Configure Grafana from the start — it’s where the trace correlation and cross-signal navigation lives.

The Bottom Line

Continuous profiling is the practical answer to “we know this is slow but we don’t know why.” The data is already there when the incident happens, the overhead is low enough to justify always-on collection, and with Pyroscope integrated into the Grafana stack, you can navigate from a metrics alert to a trace to a flame graph in a single session.

The eBPF collection model removes the last major barrier — you don’t need to instrument every application. Deploy Alloy with pyroscope.ebpf as a DaemonSet and you get profiles from every process on every node with one config change.

For Go services specifically, the SDK model with all profile types enabled is worth the small overhead. Goroutine profiles and mutex profiles catch entire classes of concurrency bugs that are essentially invisible to any other observability tool.