Distributed Tracing with Tempo and Grafana: From Zero to TraceQL

March 26, 2026 23 min read DevOps

observabilitytracinggrafanatempoopentelemetrykubernetes

Contents

Distributed Tracing with Tempo and Grafana: From Zero to TraceQL

Distributed tracing answers the question every on-call engineer eventually asks: “What actually happened to that specific request?” Logs tell you events occurred. Metrics tell you how many times. Traces tell you where the time went and how services talked to each other for a single request’s journey through your system.

Grafana Tempo is the purpose-built trace backend that integrates natively with Loki (logs) and Prometheus (metrics), enabling the “click a trace ID in a log line, jump to the full trace, click a metric exemplar, land on the same trace” workflow that makes incident investigation dramatically faster.

This guide covers Tempo’s storage architecture, full deployment on Kubernetes, instrumenting services with OpenTelemetry, writing TraceQL queries, and correlating the three pillars of observability into a unified workflow.

Why Tempo?

Before Tempo, teams typically chose between:

Jaeger: Feature-rich but storage can be complex (Cassandra/Elasticsearch backends)
Zipkin: Simple but limited query capabilities
Elastic APM: Powerful but expensive at scale and vendor-locked to Elasticsearch

Tempo’s design philosophy:

Property	Tempo approach
Storage	Object storage only (S3, GCS, Azure Blob) — no Elasticsearch
Index	No traditional index — lookup by trace ID only (or TraceQL search)
Cost	Dramatically cheaper than index-based systems at scale
Integration	Native Grafana datasource, links from Loki and Prometheus
Cardinality	No cardinality limits — any span attribute is searchable via TraceQL

The trade-off: Tempo trades real-time arbitrary attribute searches for cost efficiency. With TraceQL and its pipeline search, you can still find traces by any attribute — it just queries object storage rather than an inverted index.

Tempo Architecture

Understanding Tempo’s internals helps you size it correctly and debug problems.

                    ┌─────────────────┐
Instrumented Apps → │   Distributor   │ ← receives traces (OTLP/Jaeger/Zipkin)
                    └────────┬────────┘
                             │ routes by trace ID hash
                    ┌────────▼────────┐
                    │    Ingester     │ ← holds recent traces in memory + WAL
                    └────────┬────────┘
                             │ flushes every ~5 minutes
                    ┌────────▼────────┐
                    │  Compactor      │ ← merges blocks, applies retention
                    └────────┬────────┘
                             │
                    ┌────────▼────────┐
                    │  Object Store   │ ← S3, GCS, Azure Blob, or filesystem
                    └────────┬────────┘
                             │
                    ┌────────▼────────┐
  Grafana ←────── │   Querier       │ ← searches object store blocks
                    └─────────────────┘

Distributor receives trace data via OTLP (gRPC/HTTP), Jaeger Thrift/gRPC, or Zipkin. It hashes the trace ID to route all spans for a trace to the same ingester, ensuring complete traces.

Ingester buffers spans in memory and a Write-Ahead Log (WAL) on disk. Every few minutes it flushes complete blocks to object storage. The WAL allows recovery if an ingester crashes before flushing.

Compactor merges small blocks into larger ones for query efficiency, applies retention policy (deletes old blocks), and builds the bloom filter index used for tag-based search.

Querier handles search requests. For trace-ID lookups, it checks ingesters (recent data) and the object store. For TraceQL attribute searches, it uses bloom filters to skip irrelevant blocks, then reads matching blocks from object storage.

Storage Format

Tempo stores data in blocks on object storage. Each block contains:

meta.json — block metadata (start/end time, tenant, span count)
traces — columnar Parquet files containing spans
bloom — bloom filter per column for fast attribute searching
index — sparse index mapping trace IDs to byte offsets

The columnar format (Parquet) is what enables efficient TraceQL queries — Tempo reads only the columns relevant to your query predicates.

Deploying Tempo on Kubernetes

Helm Installation

1
2
3
4
5
6
7
8


helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

# Install in monolithic mode (all components in one pod) for small deployments
helm install tempo grafana/tempo \
  --namespace monitoring \
  --create-namespace \
  --values tempo-values.yaml

Monolithic Mode (Small/Medium Scale)

For clusters handling up to ~1M spans/minute, monolithic mode is simpler to operate:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56


# tempo-values.yaml
tempo:
  storage:
    trace:
      backend: s3
      s3:
        bucket: my-tempo-traces
        endpoint: s3.us-east-1.amazonaws.com
        region: us-east-1
        # Use IRSA or environment variables for credentials
        access_key: ""
        secret_key: ""

  retention: 720h  # 30 days

  # Enable search (TraceQL)
  search:
    enabled: true

  # Receiver configuration — what formats Tempo accepts
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: "0.0.0.0:4317"
        http:
          endpoint: "0.0.0.0:4318"
    jaeger:
      protocols:
        thrift_http:
          endpoint: "0.0.0.0:14268"
        grpc:
          endpoint: "0.0.0.0:14250"

  # Limits
  ingester:
    max_block_duration: 5m

  querier:
    max_concurrent_queries: 20

serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/tempo-s3-role

resources:
  requests:
    cpu: 500m
    memory: 1Gi
  limits:
    cpu: 2
    memory: 4Gi

persistence:
  enabled: true
  size: 10Gi  # For WAL and local blocks before flush

Distributed Mode (Large Scale)

For high-throughput environments, deploy components separately:

1
2
3


helm install tempo grafana/tempo-distributed \
  --namespace monitoring \
  --values tempo-distributed-values.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50


# tempo-distributed-values.yaml
global:
  clusterDomain: cluster.local

storage:
  trace:
    backend: s3
    s3:
      bucket: my-tempo-traces
      region: us-east-1

distributor:
  replicas: 3
  resources:
    requests:
      cpu: 500m
      memory: 512Mi

ingester:
  replicas: 3
  persistence:
    enabled: true
    size: 20Gi
  resources:
    requests:
      cpu: 1
      memory: 2Gi

querier:
  replicas: 3
  resources:
    requests:
      cpu: 1
      memory: 2Gi

compactor:
  replicas: 1  # Only one compactor needed
  resources:
    requests:
      cpu: 500m
      memory: 1Gi

# Enable metrics generation from traces (RED metrics)
metricsGenerator:
  enabled: true
  replicas: 1
  config:
    storage:
      remote_write:
        - url: http://prometheus:9090/api/v1/write

S3 IAM Policy for Tempo

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject",
        "s3:ListBucket",
        "s3:GetBucketLocation"
      ],
      "Resource": [
        "arn:aws:s3:::my-tempo-traces",
        "arn:aws:s3:::my-tempo-traces/*"
      ]
    }
  ]
}

The OpenTelemetry Collector as the Trace Pipeline

Route traces through the OTel Collector rather than sending directly to Tempo. This gives you batching, sampling, enrichment, and the ability to fan-out to multiple backends.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86


# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024

  # Add Kubernetes metadata to spans
  k8sattributes:
    auth_type: serviceAccount
    passthrough: false
    extract:
      metadata:
        - k8s.pod.name
        - k8s.pod.uid
        - k8s.deployment.name
        - k8s.namespace.name
        - k8s.node.name
      labels:
        - tag_name: app.version
          key: app.kubernetes.io/version
          from: pod

  # Add resource attributes
  resource:
    attributes:
      - key: environment
        value: production
        action: upsert

  # Tail-based sampling — keep all errors, sample the rest
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    policies:
      - name: errors-policy
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: slow-traces-policy
        type: latency
        latency: {threshold_ms: 1000}
      - name: probabilistic-policy
        type: probabilistic
        probabilistic: {sampling_percentage: 10}

  memory_limiter:
    limit_mib: 512
    spike_limit_mib: 128
    check_interval: 5s

exporters:
  otlp:
    endpoint: tempo:4317
    tls:
      insecure: true

  # Also export to Jaeger for teams still using it
  jaeger:
    endpoint: jaeger-collector:14250
    tls:
      insecure: true

  # Debug — log traces to stdout (disable in production)
  debug:
    verbosity: basic

extensions:
  health_check:
    endpoint: 0.0.0.0:13133
  pprof:
    endpoint: 0.0.0.0:1777

service:
  extensions: [health_check, pprof]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, resource, batch, tail_sampling]
      exporters: [otlp]

Instrumenting Services

Python (FastAPI)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58


# tracing.py
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource, SERVICE_NAME
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor

def configure_tracing(service_name: str, otlp_endpoint: str = "http://otel-collector:4317"):
    resource = Resource.create({SERVICE_NAME: service_name})

    exporter = OTLPSpanExporter(endpoint=otlp_endpoint, insecure=True)
    processor = BatchSpanProcessor(exporter)

    provider = TracerProvider(resource=resource)
    provider.add_span_processor(processor)
    trace.set_tracer_provider(provider)

    # Auto-instrument common libraries
    FastAPIInstrumentor.instrument()
    HTTPXClientInstrumentor.instrument()
    SQLAlchemyInstrumentor.instrument()

# main.py
from fastapi import FastAPI, HTTPException
from opentelemetry import trace
import httpx

app = FastAPI()
configure_tracing("order-service")
tracer = trace.get_tracer(__name__)

@app.post("/orders")
async def create_order(order: OrderRequest):
    with tracer.start_as_current_span("create-order") as span:
        span.set_attribute("order.customer_id", order.customer_id)
        span.set_attribute("order.item_count", len(order.items))

        # Child span for inventory check
        with tracer.start_as_current_span("check-inventory") as inv_span:
            available = await check_inventory(order.items)
            inv_span.set_attribute("inventory.available", available)
            if not available:
                inv_span.set_status(trace.Status(trace.StatusCode.ERROR, "Out of stock"))
                raise HTTPException(status_code=409, detail="Items out of stock")

        # Call payment service — context propagates automatically via httpx instrumentation
        async with httpx.AsyncClient() as client:
            payment_resp = await client.post(
                "http://payment-service/charge",
                json={"amount": order.total, "customer_id": order.customer_id}
            )

        order_id = await save_order(order)
        span.set_attribute("order.id", str(order_id))
        return {"order_id": order_id}

Go

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90


// tracing/tracing.go
package tracing

import (
    "context"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/sdk/resource"
    "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
    "google.golang.org/grpc"
    "google.golang.org/grpc/credentials/insecure"
)

func InitTracer(ctx context.Context, serviceName, endpoint string) (func(context.Context) error, error) {
    conn, err := grpc.DialContext(ctx, endpoint,
        grpc.WithTransportCredentials(insecure.NewCredentials()),
        grpc.WithBlock(),
    )
    if err != nil {
        return nil, err
    }

    exporter, err := otlptracegrpc.New(ctx, otlptracegrpc.WithGRPCConn(conn))
    if err != nil {
        return nil, err
    }

    res, _ := resource.New(ctx,
        resource.WithAttributes(semconv.ServiceName(serviceName)),
        resource.WithFromEnv(),
        resource.WithProcess(),
    )

    tp := trace.NewTracerProvider(
        trace.WithBatcher(exporter),
        trace.WithResource(res),
        trace.WithSampler(trace.ParentBased(trace.TraceIDRatioBased(0.1))), // 10% head sampling
    )

    otel.SetTracerProvider(tp)
    return tp.Shutdown, nil
}

// handler.go
package main

import (
    "context"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/codes"
    "net/http"
)

var tracer = otel.Tracer("inventory-service")

func GetInventoryHandler(w http.ResponseWriter, r *http.Request) {
    ctx, span := tracer.Start(r.Context(), "get-inventory")
    defer span.End()

    itemID := r.URL.Query().Get("item_id")
    span.SetAttributes(
        attribute.String("item.id", itemID),
        attribute.String("http.method", r.Method),
    )

    inventory, err := fetchFromDB(ctx, itemID)
    if err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, err.Error())
        http.Error(w, "DB error", http.StatusInternalServerError)
        return
    }

    span.SetAttributes(attribute.Int("inventory.quantity", inventory.Quantity))
    // Write response...
}

func fetchFromDB(ctx context.Context, itemID string) (*Inventory, error) {
    _, span := tracer.Start(ctx, "db-query")
    defer span.End()

    span.SetAttributes(
        attribute.String("db.system", "postgresql"),
        attribute.String("db.statement", "SELECT * FROM inventory WHERE item_id = $1"),
    )
    // Execute query...
    return &Inventory{}, nil
}

Node.js (Express)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55


// tracing.js — load before everything else
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { Resource } = require('@opentelemetry/resources');
const { SEMRESATTRS_SERVICE_NAME } = require('@opentelemetry/semantic-conventions');

const sdk = new NodeSDK({
  resource: new Resource({
    [SEMRESATTRS_SERVICE_NAME]: 'frontend-service',
  }),
  traceExporter: new OTLPTraceExporter({
    url: 'http://otel-collector:4317',
  }),
  instrumentations: [
    getNodeAutoInstrumentations({
      '@opentelemetry/instrumentation-fs': { enabled: false }, // Too noisy
    }),
  ],
});

sdk.start();

// Graceful shutdown
process.on('SIGTERM', () => {
  sdk.shutdown().finally(() => process.exit(0));
});

// app.js — manual instrumentation for custom spans
const { trace, context, propagation } = require('@opentelemetry/api');
const tracer = trace.getTracer('frontend-service');

app.get('/products/:id', async (req, res) => {
  const span = tracer.startSpan('get-product-details', {
    attributes: {
      'product.id': req.params.id,
      'user.id': req.user?.id,
    },
  });

  // Attach span to context for child spans
  await context.with(trace.setSpan(context.active(), span), async () => {
    try {
      const product = await productService.getById(req.params.id);
      span.setAttributes({ 'product.name': product.name, 'product.category': product.category });
      res.json(product);
    } catch (err) {
      span.recordException(err);
      span.setStatus({ code: SpanStatusCode.ERROR, message: err.message });
      res.status(500).json({ error: 'Internal server error' });
    } finally {
      span.end();
    }
  });
});

Configuring Grafana Tempo Datasource

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71


# grafana-datasources.yaml (ConfigMap for Grafana)
apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-datasources
  namespace: monitoring
data:
  datasources.yaml: |
    apiVersion: 1
    datasources:
      - name: Tempo
        type: tempo
        url: http://tempo:3100
        uid: tempo
        jsonData:
          httpMethod: GET
          # Link traces to logs in Loki
          tracesToLogs:
            datasourceUid: loki
            filterByTraceID: true
            filterBySpanID: false
            mapTagNamesEnabled: true
            mappedTags:
              - key: service.name
                value: app
            lokiSearch: true
          # Link traces to metrics in Prometheus
          tracesToMetrics:
            datasourceUid: prometheus
            tags:
              - key: service.name
                value: job
            queries:
              - name: "Request rate"
                query: "rate(http_server_duration_count{$$__tags}[5m])"
              - name: "Error rate"
                query: "rate(http_server_duration_count{$$__tags, status_code=~\"5..\"}[5m])"
          # Enable service graph
          serviceMap:
            datasourceUid: prometheus
          # Node graph panel
          nodeGraph:
            enabled: true
          # TraceQL search
          search:
            hide: false
          # Span bar (what to show as the colored bar per span)
          spanBar:
            type: duration

      - name: Loki
        type: loki
        url: http://loki:3100
        uid: loki
        jsonData:
          # Link logs to traces in Tempo
          derivedFields:
            - matcherRegex: '"trace_id":"(\w+)"'
              name: TraceID
              url: "$${__value.raw}"
              datasourceUid: tempo
              urlDisplayLabel: "View Trace"

      - name: Prometheus
        type: prometheus
        url: http://prometheus:9090
        uid: prometheus
        jsonData:
          exemplarTraceIdDestinations:
            - name: trace_id
              datasourceUid: tempo

TraceQL: The Query Language for Traces

TraceQL lets you search traces by any span attribute, duration, status, or structural property. It’s analogous to LogQL for traces.

Basic Syntax

{ <span-selectors> } | <pipeline>

A query selects spans matching conditions in {}, then optionally applies pipeline operations.

Attribute Selectors

# Find spans from the order-service with errors
{ .service.name = "order-service" && status = error }

# Find slow database spans
{ span.db.system = "postgresql" && duration > 500ms }

# Find spans with a specific HTTP status code
{ span.http.status_code >= 500 }

# Regex match on span name
{ name =~ ".*checkout.*" }

# Find spans that have a specific attribute (any value)
{ span.user.id != nil }

Structural Operators

# Find traces where the root span is slow (total trace duration)
{ rootName = "POST /api/orders" && traceDuration > 2s }

# Find traces containing an error span anywhere
{ .service.name = "payment-service" } && { status = error }

# Trace-level filter: any span with error in this service
{ rootService = "frontend" } && { .service.name = "database" && status = error }

Pipeline Operations

# Count spans matching criteria
{ .service.name = "api-gateway" } | count() > 5

# Filter by child count (fan-out detection)
{ name = "process-batch" } | count() > 100

# Select only specific attributes in results
{ status = error } | select(span.http.url, span.http.status_code, duration)

Aggregate by TraceQL

Find the slowest services:

{ } | rate()  # spans per second by service

# Average duration by service (metrics view)
{ } | avg(duration) by (.service.name)

Practical TraceQL Recipes

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


# All errors in the last hour
{ status = error }

# Slow checkout flows (any span in the checkout flow > 1s)
{ .service.name =~ "checkout.*" && duration > 1s }

# Find which user had the slow request
{ traceDuration > 5s } | select(span.user.id, traceDuration)

# Database queries without indexes (sequential scans)
{ span.db.statement =~ ".*Seq Scan.*" }

# Traces that hit more than 10 microservices
{ } | count() by (trace_id) | count() > 10

# HTTP 504 gateway timeouts
{ span.http.status_code = 504 }

# Find all spans calling a specific external API
{ span.http.url =~ ".*stripe\.com.*" }

# Spans that generated an exception
{ span.exception.type != nil }

# Trace ID lookup (direct navigation)
{ traceID = "4bf92f3577b34da6a3ce929d0e0e4736" }

Metrics Generator: RED Metrics from Traces

Tempo’s metrics generator derives service-level metrics from trace data automatically — no instrumentation code changes needed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


# In tempo config
metrics_generator:
  storage:
    path: /var/tempo/generator/wal
    remote_write:
      - url: http://prometheus:9090/api/v1/write
        send_exemplars: true

  processor:
    service_graphs:
      dimensions:
        - http.method
        - http.status_code
    span_metrics:
      dimensions:
        - service.name
        - span.name
        - http.method
        - http.status_code
      intrinsic_dimensions:
        service: true
        span_name: true
        span_kind: true
        status_code: true

This generates metrics like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


# Request rate (from traces)
rate(traces_spanmetrics_calls_total{service_name="order-service"}[5m])

# Error rate
rate(traces_spanmetrics_calls_total{service_name="order-service", status_code="STATUS_CODE_ERROR"}[5m])

# Latency percentiles
histogram_quantile(0.99,
  rate(traces_spanmetrics_duration_seconds_bucket{service_name="order-service"}[5m])
)

# Service graph — upstream dependency calls
rate(traces_service_graph_request_total{client="frontend", server="order-service"}[5m])

Prometheus Alert Using Trace-Derived Metrics

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


groups:
  - name: service_slos
    rules:
      - alert: HighErrorRate
        expr: |
          (
            rate(traces_spanmetrics_calls_total{status_code="STATUS_CODE_ERROR"}[5m])
            /
            rate(traces_spanmetrics_calls_total{}[5m])
          ) > 0.01
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.service_name }} error rate {{ $value | humanizePercentage }}"
          runbook_url: "https://runbooks.example.com/high-error-rate"

      - alert: SlowP99Latency
        expr: |
          histogram_quantile(0.99,
            rate(traces_spanmetrics_duration_seconds_bucket{}[5m])
          ) > 1.0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.service_name }} P99 latency above 1s"

Correlating Traces, Logs, and Metrics

The real power of Tempo is navigating between the three pillars seamlessly.

Structured Logs with Trace ID

Inject the current trace ID into every log line:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


# Python with structlog
import structlog
from opentelemetry import trace

def add_trace_context(logger, method, event_dict):
    span = trace.get_current_span()
    if span.is_recording():
        ctx = span.get_span_context()
        event_dict["trace_id"] = format(ctx.trace_id, "032x")
        event_dict["span_id"] = format(ctx.span_id, "016x")
    return event_dict

structlog.configure(
    processors=[
        add_trace_context,
        structlog.processors.JSONRenderer(),
    ]
)

log = structlog.get_logger()
log.info("order.created", order_id="ord_123", amount=99.99)
# Output: {"event": "order.created", "order_id": "ord_123", "amount": 99.99,
#          "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "span_id": "a2fb4a1d1a96d312"}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


// Go with slog
import (
    "log/slog"
    "go.opentelemetry.io/otel/trace"
)

func withTrace(ctx context.Context) *slog.Logger {
    span := trace.SpanFromContext(ctx)
    if !span.IsRecording() {
        return slog.Default()
    }
    sc := span.SpanContext()
    return slog.With(
        "trace_id", sc.TraceID().String(),
        "span_id", sc.SpanID().String(),
    )
}

// In handler:
log := withTrace(ctx)
log.Info("processing payment", "customer_id", customerID, "amount", amount)

Prometheus Exemplars

Exemplars attach a trace ID to a metric data point, enabling “click this spike, show me the trace”:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34


// Go — add exemplar to histogram observation
import (
    "github.com/prometheus/client_golang/prometheus"
    "go.opentelemetry.io/otel/trace"
)

var requestDuration = prometheus.NewHistogramVec(prometheus.HistogramOpts{
    Name:    "http_request_duration_seconds",
    Help:    "HTTP request latency",
    Buckets: prometheus.DefBuckets,
}, []string{"method", "status"})

func metricsMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()
        wrapped := &responseWriter{ResponseWriter: w}
        next.ServeHTTP(wrapped, r)
        duration := time.Since(start).Seconds()

        // Add trace ID as exemplar
        span := trace.SpanFromContext(r.Context())
        labels := prometheus.Labels{"method": r.Method, "status": strconv.Itoa(wrapped.status)}

        if span.IsRecording() {
            traceID := span.SpanContext().TraceID().String()
            requestDuration.With(labels).(prometheus.ExemplarObserver).ObserveWithExemplar(
                duration,
                prometheus.Labels{"trace_id": traceID},
            )
        } else {
            requestDuration.With(labels).Observe(duration)
        }
    })
}

Enable exemplar storage in Prometheus:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# prometheus.yml
global:
  scrape_interval: 15s

# Enable exemplar storage (required for Prometheus 2.25+)
storage:
  exemplars:
    max_exemplars: 100000

# Scrape jobs must use OpenMetrics format to receive exemplars
scrape_configs:
  - job_name: 'my-service'
    scrape_protocols: [OpenMetricsText1.0.0, OpenMetricsText0.0.1, PrometheusText0.0.4]
    static_configs:
      - targets: ['my-service:8080']

The Correlation Workflow in Grafana

Start in a dashboard — you notice P99 latency spikes at 14:32
Click on the spike — exemplar dot appears → click “Query with exemplar”
Land in Explore with the trace — full waterfall showing which service was slow
Identify the slow span — postgres:query took 4.2s
Click the log icon on that span → jumps to Loki, filtered to that trace ID
See the log line: "slow query detected: sequential scan on orders table"
Fix: add an index, deploy, verify the trace duration drops

Grafana Dashboards for Tracing

Service Map Panel

The service map (Node Graph panel) shows your entire microservice topology derived from trace data. Enable it in the Tempo datasource config with serviceMap.datasourceUid.

It shows:

All services and their dependencies (automatically discovered)
Request rate and error rate per edge
Click any node to see its traces

RED Dashboard Using Trace-Derived Metrics

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


{
  "panels": [
    {
      "title": "Request Rate by Service",
      "type": "timeseries",
      "targets": [{
        "expr": "sum(rate(traces_spanmetrics_calls_total[5m])) by (service_name)",
        "legendFormat": "{{service_name}}"
      }]
    },
    {
      "title": "Error Rate by Service",
      "type": "timeseries",
      "targets": [{
        "expr": "sum(rate(traces_spanmetrics_calls_total{status_code=\"STATUS_CODE_ERROR\"}[5m])) by (service_name) / sum(rate(traces_spanmetrics_calls_total[5m])) by (service_name)",
        "legendFormat": "{{service_name}}"
      }]
    },
    {
      "title": "P99 Duration by Service",
      "type": "timeseries",
      "targets": [{
        "expr": "histogram_quantile(0.99, sum(rate(traces_spanmetrics_duration_seconds_bucket[5m])) by (le, service_name))",
        "legendFormat": "{{service_name}} P99"
      }]
    }
  ]
}

Sampling Strategies

Sampling determines which traces you actually store. Get it wrong and you miss the bugs that matter.

Head-Based Sampling

The decision is made at the start of the trace. Simple and low-overhead.

1
2
3
4
5
6


// Go SDK — 10% sampling
tp := trace.NewTracerProvider(
    trace.WithSampler(trace.ParentBased(
        trace.TraceIDRatioBased(0.10),
    )),
)

Problems with head sampling: By definition, you can’t keep “all errors” because you don’t know the trace will error when you start it.

Tail-Based Sampling (Recommended)

The OTel Collector buffers complete traces and samples based on the outcome:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


# Already shown above, but key policy combinations:
processors:
  tail_sampling:
    decision_wait: 30s  # Wait this long to see if the trace completes
    num_traces: 100000  # Max in-memory traces
    policies:
      # Always keep errors
      - name: keep-errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      # Always keep slow traces
      - name: keep-slow
        type: latency
        latency: {threshold_ms: 2000}
      # Always keep traces with user IDs (for debugging specific users)
      - name: keep-vip-users
        type: string_attribute
        string_attribute:
          key: user.tier
          values: [enterprise, vip]
      # Sample 5% of healthy fast traces
      - name: sample-normal
        type: probabilistic
        probabilistic: {sampling_percentage: 5}

Adaptive Sampling

Automatically adjust sampling rates to hit a target volume:

1
2
3
4
5
6


processors:
  tail_sampling:
    policies:
      - name: adaptive
        type: rate_limiting
        rate_limiting: {spans_per_second: 1000}

Production Operations

Retention and Storage Sizing

Estimate your storage requirements:

Storage (bytes/day) =
  spans_per_second × 86400 × avg_span_size_bytes × (1 - sample_rate)

Typical numbers:

Average span size (compressed): ~300 bytes
10,000 spans/sec at 10% sampling = 1,000 spans/sec stored
Daily storage: 1,000 × 86,400 × 300 bytes ≈ 26 GB/day
30-day retention: ~780 GB on S3 (≈$18/month at $0.023/GB)

1
2
3
4
5
6
7


# Tempo retention config
compactor:
  compaction:
    block_retention: 720h  # 30 days
    compacted_block_retention: 1h
    # Retention for specific tenants (multi-tenant mode)
    per_tenant_override_config: /etc/tempo/overrides.yaml

1
2
3
4
5
6


# /etc/tempo/overrides.yaml
overrides:
  tenant-a:
    block_retention: 2160h  # 90 days for enterprise tenant
  tenant-b:
    block_retention: 168h   # 7 days for dev tenant

Monitoring Tempo Itself

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


# Prometheus rules for Tempo health
groups:
  - name: tempo
    rules:
      - alert: TempoIngesterNotReady
        expr: tempo_ingester_bytes_received_total == 0
        for: 5m
        annotations:
          summary: "Tempo ingester receiving no spans"

      - alert: TempoCompactorNotRunning
        expr: time() - tempo_compactor_last_successful_run_timestamp_seconds > 3600
        for: 15m
        annotations:
          summary: "Tempo compactor hasn't run successfully in 1 hour"

      - alert: TempoQuerierLatencyHigh
        expr: histogram_quantile(0.99, rate(tempo_query_frontend_query_range_duration_seconds_bucket[5m])) > 10
        for: 5m
        annotations:
          summary: "Tempo query P99 latency above 10s"

      - alert: TempoBlocksFailing
        expr: rate(tempodb_backend_failed_write_total[5m]) > 0
        for: 5m
        annotations:
          summary: "Tempo failing to write blocks to object storage"

Multi-Tenancy

Isolate traces per team or environment:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


# Tempo config
distributor:
  receivers:
    otlp:
      protocols:
        http:
          endpoint: "0.0.0.0:4318"

auth_enabled: true  # Require X-Scope-OrgID header

# OTel Collector — set tenant header
exporters:
  otlp:
    endpoint: tempo:4317
    headers:
      X-Scope-OrgID: production
    tls:
      insecure: true

Debugging Trace Pipeline Issues

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


# Check if Tempo is receiving spans
kubectl exec -n monitoring deploy/tempo -- wget -qO- localhost:3100/metrics | grep tempo_distributor_spans

# Query trace directly by ID (bypass Grafana)
curl "http://tempo:3100/api/traces/4bf92f3577b34da6a3ce929d0e0e4736"

# List recent blocks
curl "http://tempo:3100/api/v2/search/tags"

# Check OTel Collector is exporting
kubectl exec -n monitoring deploy/otel-collector -- wget -qO- localhost:8888/metrics | grep otelcol_exporter_sent_spans

# TraceQL via API
curl -G "http://tempo:3100/api/search" \
  --data-urlencode 'q={ status = error }' \
  --data-urlencode 'start=1700000000' \
  --data-urlencode 'end=1700003600' \
  --data-urlencode 'limit=20'

Grafana Explore: Tracing Workflow

The Explore view is where you’ll spend most of your debugging time:

Search Mode

Use the search UI (no TraceQL knowledge required):

Service name dropdown
Span name
Tags as key=value pairs
Min/Max duration
Time range

TraceQL Mode

Switch to TraceQL in Explore for complex queries:

# Slowest 10 traces touching the checkout service
{ .service.name = "checkout-service" } | select(traceDuration) | sort(traceDuration desc) | limit(10)

Trace View

The trace waterfall shows:

Horizontal bars = span duration (timeline)
Nesting = parent-child relationships
Color coding by service
Click a span → attributes panel (all span attributes, events, links)
“Logs for this span” button → Loki filtered by trace+span ID
“Metrics for this span” button → Prometheus metrics for the service

Common Tracing Pitfalls

Not propagating context across async boundaries

1
2
3
4
5
6


# Wrong — loses trace context in background task
asyncio.create_task(send_notification(order_id))

# Right — copy current context into the task
ctx = context.copy()
asyncio.create_task(send_notification(order_id), context=ctx)

Sampling at the wrong level

If you sample at the SDK level (head sampling) AND at the Collector (tail sampling), you might drop important traces twice. Use one or the other — prefer tail sampling in the Collector.

Spans without meaningful attributes

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


# Unhelpful
with tracer.start_as_current_span("database"):
    result = db.query(sql)

# Helpful
with tracer.start_as_current_span("database") as span:
    span.set_attributes({
        "db.system": "postgresql",
        "db.name": "orders",
        "db.statement": sql,  # Be careful with PII
        "db.rows_returned": len(result),
    })
    result = db.query(sql)

Missing error recording

1
2
3
4
5
6


try:
    result = risky_operation()
except Exception as e:
    span.record_exception(e)  # Records exception type, message, and stack trace
    span.set_status(StatusCode.ERROR, str(e))
    raise

Clock skew between services

If services have unsynchronized clocks, trace waterfalls show negative durations or overlapping spans that don’t make sense. Ensure NTP is running everywhere and use W3C traceparent propagation (built into all OTel SDKs) rather than vendor-specific propagation.

Quick Reference

TraceQL Cheat Sheet

Query	What it finds
`{ status = error }`	All error spans
`{ duration > 1s }`	Slow spans
`{ traceDuration > 5s }`	Slow traces (root span)
`{ .service.name = "api" }`	Spans from api service
`{ span.http.status_code >= 500 }`	HTTP 5xx spans
`{ name =~ ".payment." }`	Spans with “payment” in name
`{ span.user.id = "u123" }`	Spans for a specific user
`{ rootService = "frontend" }`	Traces originating from frontend
`{ } \| count() > 50`	Traces with many spans (fan-out)

Key Endpoints

1
2
3
4
5
6
7


# Tempo API
GET /api/traces/{traceID}        # Fetch trace by ID
GET /api/search                  # TraceQL search
GET /api/v2/search/tags          # List all tag names
GET /api/v2/search/tag/{name}/values  # List values for a tag
GET /metrics                     # Prometheus metrics
GET /ready                       # Readiness probe

Summary

Grafana Tempo provides cost-effective distributed tracing built for the cloud-native stack. The key design decisions that make it practical at scale:

Object storage backend keeps costs manageable — no Elasticsearch cluster to size and operate
TraceQL gives you SQL-like power over trace data without sacrificing the columnar storage efficiency
Metrics generator derives RED metrics from traces automatically, closing the loop with Prometheus
Native Grafana integration makes the traces-logs-metrics correlation workflow genuinely fast to navigate

Start with the monolithic deployment, send traces through the OTel Collector with tail-based sampling, and wire up the Grafana datasource cross-links. The first time you click a latency spike in a Prometheus graph and land directly on the exact trace that caused it — with the log lines right there — you’ll understand why distributed tracing is worth the instrumentation investment.

Distributed Tracing with Tempo and Grafana: From Zero to TraceQL

Why Tempo?

Tempo Architecture

Storage Format

Deploying Tempo on Kubernetes

Helm Installation

Monolithic Mode (Small/Medium Scale)

Distributed Mode (Large Scale)

S3 IAM Policy for Tempo

The OpenTelemetry Collector as the Trace Pipeline

Instrumenting Services

Python (FastAPI)

Go

Node.js (Express)

Configuring Grafana Tempo Datasource

TraceQL: The Query Language for Traces

Basic Syntax

Attribute Selectors

Structural Operators

Pipeline Operations

Aggregate by TraceQL

Practical TraceQL Recipes

Metrics Generator: RED Metrics from Traces

Prometheus Alert Using Trace-Derived Metrics

Correlating Traces, Logs, and Metrics

Structured Logs with Trace ID

Prometheus Exemplars

The Correlation Workflow in Grafana

Grafana Dashboards for Tracing

Service Map Panel

RED Dashboard Using Trace-Derived Metrics

Sampling Strategies

Head-Based Sampling

Tail-Based Sampling (Recommended)

Adaptive Sampling

Production Operations

Retention and Storage Sizing

Monitoring Tempo Itself

Multi-Tenancy

Debugging Trace Pipeline Issues

Grafana Explore: Tracing Workflow

Search Mode

TraceQL Mode

Trace View

Common Tracing Pitfalls

Quick Reference

TraceQL Cheat Sheet

Key Endpoints

Summary

Comments