The Prometheus + Grafana Stack: Metrics, Alerting, and Dashboards

March 25, 2026 29 min read DevOps

prometheusgrafanamonitoringobservabilityalertingmetricshomelabdevops

Contents

Something is wrong with your server at 3am. Response times climbed for 45 minutes before the on-call page fired. Now you’re staring at a terminal and asking yourself: was it CPU? Memory pressure? A disk filling up? A container that kept OOM-crashing and restarting? Did it start before or after that deploy?

Without metrics, you are guessing. With Prometheus and Grafana, you scroll back in time, pull up the exact 45-minute window, and the answer is sitting right there in a graph — CPU saturation on one core, caused by a container that had no resource limits and decided to consume everything it could reach.

This guide builds that capability from scratch. You’ll get a complete Docker Compose stack, working configuration files, real PromQL queries, alerting rules that fire on the things that actually matter, and dashboards you can use immediately.

Part 1: Why Prometheus + Grafana?

The modern observability space organizes around three signals: metrics (numerical measurements over time), logs (timestamped event records), and traces (request journeys through distributed systems). Prometheus owns the metrics tier. It is the most widely deployed metrics system in existence, and for good reason.

Prometheus is a pull-based metrics collector and time-series database built at SoundCloud in 2012 and donated to the CNCF in 2016. Rather than having agents push data to a central server, Prometheus scrapes HTTP endpoints — called targets — on a configurable interval. Every target exposes its metrics in a plain-text format at /metrics. Prometheus fetches that page, parses it, and stores the values in its time-series database (TSDB) with nanosecond-precision timestamps.

The pull model has a subtle but important advantage: Prometheus knows when a target goes silent. If a scrape fails, that’s a data point — the target is down. Push-based systems have to infer absence, which is harder to alert on reliably.

PromQL (Prometheus Query Language) is the query language that turns raw time-series data into answers. It is a functional, expression-based language purpose-built for time-series math. You can compute rates over windows, aggregate across label dimensions, predict future values with linear regression, and correlate metrics from different sources. Once you understand PromQL, the data becomes genuinely useful.

Grafana is the visualization layer. It connects to Prometheus (and dozens of other data sources), renders panels and dashboards, and ships its own alerting engine. The combination of Prometheus as the data store and Grafana as the UI has become the default observability stack for Kubernetes clusters, homelabs, and production infrastructure alike.

The exporter ecosystem is the final piece. Because Prometheus scrapes a simple HTTP text format, anyone can write an exporter. Today there are battle-tested exporters for Linux hosts, Docker containers, PostgreSQL, MySQL, Redis, Nginx, HAProxy, Elasticsearch, SNMP devices, and hundreds of other systems. You do not need to instrument anything to get immediate value — install the exporter, point Prometheus at it, and you have metrics within seconds.

By the end of this guide you will be able to answer:

Is my server overloaded right now, and has it been overloaded in the past week?
Why did my application slow down at 3am on Tuesday?
Which container is consuming the most CPU?
When will this disk fill up, given the current growth rate?
Are all my HTTP endpoints responding, and what is the error rate?

Part 2: Architecture Overview

Before touching configuration files, a clear picture of how the components connect will save you debugging time later.

┌─────────────────────────────────────────────────────────────────┐
│                         Your Infrastructure                      │
│                                                                  │
│  ┌──────────────┐  ┌──────────────┐  ┌───────────────────────┐  │
│  │ Node Exporter│  │   cAdvisor   │  │  Blackbox Exporter    │  │
│  │ (host metrics│  │  (container  │  │  (HTTP/TCP/ICMP probes│  │
│  │  :9100/metrics│  │  metrics)    │  │   :9115/metrics)      │  │
│  │             │  │  :8080/metrics│  │                       │  │
│  └──────┬───────┘  └──────┬───────┘  └──────────┬────────────┘  │
│         │                 │                      │               │
└─────────┼─────────────────┼──────────────────────┼───────────────┘
          │  scrape         │  scrape               │  scrape
          ▼                 ▼                       ▼
┌─────────────────────────────────────────────────────────────────┐
│                    Prometheus Server (:9090)                      │
│                                                                  │
│  ┌───────────────┐  ┌─────────────┐  ┌───────────────────────┐  │
│  │  Scrape Engine│  │  TSDB Store │  │  Query Engine (PromQL)│  │
│  │  (pull-based) │  │  (on disk)  │  │                       │  │
│  └───────────────┘  └─────────────┘  └───────────────────────┘  │
│                                                                  │
│  ┌───────────────────────────────────────────────────────────┐   │
│  │  Alerting Rules Evaluation  →  fires to Alertmanager      │   │
│  └───────────────────────────────────────────────────────────┘   │
└────────────────────────────────┬────────────────────────────────┘
                                 │  query
                                 ▼
              ┌──────────────────────────────────┐
              │         Grafana (:3000)           │
              │  Dashboards │ Panels │ Alerting   │
              └──────────────────────────────────┘

                    Prometheus ──alerts──▶ Alertmanager (:9093)
                                               │
                                ┌──────────────┼──────────────┐
                                ▼              ▼              ▼
                             Slack           Email        PagerDuty

Prometheus server is the core. It runs the scrape engine (fetching /metrics from targets at configurable intervals), the TSDB (a local time-series database that compresses and indexes metric data on disk), the PromQL query engine, and the alerting rule evaluator.

Exporters are the bridge between your systems and Prometheus. Each exporter runs as a sidecar or standalone process next to the thing it monitors, reads metrics from that system’s native interface, and presents them in Prometheus text format on an HTTP endpoint. Prometheus scrapes that endpoint. Exporters do not push — they wait to be scraped.

Alertmanager handles alert routing, deduplication, grouping, silencing, and notification delivery. Prometheus evaluates alerting rules and fires alerts at Alertmanager. Alertmanager decides who gets notified, when, and through which channel. This separation keeps alerting logic out of Prometheus.

Grafana connects to Prometheus as a datasource and renders dashboards. It also ships a built-in alerting engine (Grafana Unified Alerting) that can complement or replace Alertmanager for simpler setups.

Pushgateway is a short-lived intermediary for batch jobs and ephemeral workloads that do not live long enough to be scraped. A cron job pushes its completion metrics to the Pushgateway, and Prometheus scrapes the Pushgateway. For always-running services, you almost never need it.

Part 3: Installation with Docker Compose

The full stack runs cleanly in Docker Compose. This configuration includes Prometheus, Grafana, Node Exporter, cAdvisor, and Alertmanager with proper data persistence and a dedicated network.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106


# docker-compose.yml
version: "3.8"

networks:
  monitoring:
    driver: bridge

volumes:
  prometheus_data: {}
  grafana_data: {}
  alertmanager_data: {}

services:

  prometheus:
    image: prom/prometheus:v2.51.0
    container_name: prometheus
    restart: unless-stopped
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./prometheus/rules:/etc/prometheus/rules:ro
      - prometheus_data:/prometheus
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--storage.tsdb.retention.time=30d"
      - "--web.console.libraries=/etc/prometheus/console_libraries"
      - "--web.console.templates=/etc/prometheus/consoles"
      - "--web.enable-lifecycle"
      - "--web.enable-admin-api"
    ports:
      - "9090:9090"
    networks:
      - monitoring
    depends_on:
      - alertmanager

  grafana:
    image: grafana/grafana:10.4.0
    container_name: grafana
    restart: unless-stopped
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=changeme
      - GF_USERS_ALLOW_SIGN_UP=false
      - GF_SERVER_ROOT_URL=https://grafana.yourdomain.com
      - GF_SMTP_ENABLED=false
    ports:
      - "3000:3000"
    networks:
      - monitoring
    depends_on:
      - prometheus

  node-exporter:
    image: prom/node-exporter:v1.7.0
    container_name: node-exporter
    restart: unless-stopped
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - "--path.procfs=/host/proc"
      - "--path.rootfs=/rootfs"
      - "--path.sysfs=/host/sys"
      - "--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)"
    ports:
      - "9100:9100"
    networks:
      - monitoring

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.49.1
    container_name: cadvisor
    restart: unless-stopped
    privileged: true
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker:/var/lib/docker:ro
      - /dev/disk:/dev/disk:ro
    ports:
      - "8080:8080"
    networks:
      - monitoring

  alertmanager:
    image: prom/alertmanager:v0.27.0
    container_name: alertmanager
    restart: unless-stopped
    volumes:
      - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
      - alertmanager_data:/alertmanager
    command:
      - "--config.file=/etc/alertmanager/alertmanager.yml"
      - "--storage.path=/alertmanager"
      - "--web.external-url=https://alertmanager.yourdomain.com"
    ports:
      - "9093:9093"
    networks:
      - monitoring

Create the Grafana datasource auto-provisioning file so Prometheus is wired up automatically on first boot:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


# grafana/provisioning/datasources/prometheus.yml
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    jsonData:
      httpMethod: POST
      prometheusType: Prometheus
      prometheusVersion: 2.51.0
      exemplarTraceIdDestinations:
        - name: traceID
          datasourceUid: tempo
    editable: true

Bring the stack up with:

1

docker compose up -d

Prometheus will be available at http://localhost:9090, Grafana at http://localhost:3000.

Part 4: Prometheus Configuration

The prometheus.yml file controls everything: where to scrape, how often, which rules to evaluate, and where to send alerts.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66


# prometheus/prometheus.yml
global:
  scrape_interval: 15s          # How often to scrape targets
  evaluation_interval: 15s      # How often to evaluate alerting rules
  scrape_timeout: 10s           # Timeout for individual scrapes
  external_labels:
    cluster: homelab
    env: production

# Alertmanager connection
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:9093

# Rule files for recording rules and alert rules
rule_files:
  - "rules/node_alerts.yml"
  - "rules/container_alerts.yml"
  - "rules/recording_rules.yml"

scrape_configs:

  # Prometheus itself
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  # Linux host metrics via Node Exporter
  - job_name: "node"
    static_configs:
      - targets:
          - "node-exporter:9100"
        labels:
          host: homelab-01

  # Docker container metrics via cAdvisor
  - job_name: "cadvisor"
    static_configs:
      - targets: ["cadvisor:8080"]

  # Blackbox Exporter — HTTP probing
  - job_name: "blackbox_http"
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://yourdomain.com
          - https://grafana.yourdomain.com
          - https://prometheus.yourdomain.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

  # File-based service discovery — add targets without reloading
  - job_name: "file_sd_nodes"
    file_sd_configs:
      - files:
          - "/etc/prometheus/targets/*.yml"
        refresh_interval: 30s

Hot reload lets you apply config changes without restarting Prometheus. The --web.enable-lifecycle flag enables the reload endpoint:

1

curl -X POST http://localhost:9090/-/reload

File-based service discovery (file_sd_configs) is the practical choice for homelabs and small fleets. Drop a YAML file in the targets directory and Prometheus picks it up within 30 seconds, no restart needed:

1
2
3
4
5
6
7


# prometheus/targets/additional_nodes.yml
- targets:
    - "192.168.1.10:9100"
    - "192.168.1.11:9100"
  labels:
    job: node
    env: homelab

Part 5: Key Exporters

Node Exporter

Node Exporter is the standard Linux host metrics exporter. It exposes CPU, memory, disk, network, filesystem, and dozens of other system metrics via /proc and /sys. The key metrics you will use constantly:

Metric	Description
`node_cpu_seconds_total`	CPU time by mode (idle, user, system, iowait, etc.)
`node_memory_MemAvailable_bytes`	Available memory (kernel estimate)
`node_memory_MemTotal_bytes`	Total installed RAM
`node_filesystem_avail_bytes`	Filesystem space available to non-root users
`node_filesystem_size_bytes`	Total filesystem capacity
`node_network_receive_bytes_total`	Network bytes received by interface
`node_network_transmit_bytes_total`	Network bytes transmitted by interface
`node_load1`, `node_load5`, `node_load15`	System load averages
`node_disk_read_bytes_total`	Disk bytes read by device

cAdvisor

cAdvisor (Container Advisor) exposes resource usage metrics for every running Docker container. It reads from the Docker API and cgroups. Essential metrics:

Metric	Description
`container_cpu_usage_seconds_total`	Cumulative CPU usage per container
`container_memory_usage_bytes`	Current memory usage
`container_memory_working_set_bytes`	Memory working set (more meaningful than usage)
`container_network_receive_bytes_total`	Container network receive
`container_fs_usage_bytes`	Container filesystem usage

Blackbox Exporter

Blackbox Exporter probes external endpoints over HTTP, HTTPS, TCP, and ICMP. It answers: is this URL up? What is the response time? Is the TLS certificate about to expire?

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


# blackbox/blackbox.yml
modules:
  http_2xx:
    prober: http
    timeout: 10s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: []  # Defaults to 2xx
      method: GET
      follow_redirects: true
      preferred_ip_protocol: ip4
      tls_config:
        insecure_skip_verify: false

  http_post_2xx:
    prober: http
    http:
      method: POST
      headers:
        Content-Type: application/json

  tcp_connect:
    prober: tcp
    timeout: 5s

  icmp_ping:
    prober: icmp
    timeout: 5s
    icmp:
      preferred_ip_protocol: ip4

Key Blackbox metrics:

probe_success — 1 if the probe succeeded, 0 if it failed
probe_duration_seconds — how long the probe took
probe_http_status_code — the HTTP status code returned
probe_ssl_earliest_cert_expiry — Unix timestamp when the TLS cert expires

Database Exporters

For common databases, drop-in exporters handle instrumentation:

mysqld_exporter — MySQL/MariaDB: query throughput, connection pool, replication lag
postgres_exporter — PostgreSQL: transaction rates, cache hit ratios, bloat, replication
redis_exporter — Redis: commands per second, memory usage, keyspace hits/misses, replication

Application Instrumentation

For custom applications, Prometheus client libraries make instrumentation straightforward. A Go service exposing a counter and histogram:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


import "github.com/prometheus/client_golang/prometheus"

var httpRequests = prometheus.NewCounterVec(
    prometheus.CounterOpts{
        Name: "http_requests_total",
        Help: "Total HTTP requests processed.",
    },
    []string{"method", "path", "status"},
)

var requestDuration = prometheus.NewHistogramVec(
    prometheus.HistogramOpts{
        Name:    "http_request_duration_seconds",
        Help:    "HTTP request duration in seconds.",
        Buckets: prometheus.DefBuckets,
    },
    []string{"method", "path"},
)

Client libraries exist for Go, Python (prometheus_client), Java (simpleclient), Node.js (prom-client), Ruby, Rust, and many more. Register your metrics, instrument your code paths, and expose the default /metrics handler — that’s all Prometheus needs.

Part 6: PromQL Fundamentals

PromQL is the query language that makes Prometheus useful. It takes getting used to, but once it clicks, you can answer almost any infrastructure question from a terminal or dashboard.

Data Types

Instant vector — a set of time series, each with a single sample at the current time: node_memory_MemTotal_bytes
Range vector — a set of time series with a range of samples over a time window: node_cpu_seconds_total[5m]
Scalar — a single floating point number
String — a string literal (rarely used directly)

Label Matchers

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# Exact match
node_cpu_seconds_total{mode="idle"}

# Not equal
node_cpu_seconds_total{mode!="idle"}

# Regex match — all modes except idle and iowait
node_cpu_seconds_total{mode=~"user|system|softirq"}

# Negative regex
node_cpu_seconds_total{mode!~"idle|iowait"}

Essential Functions

rate(v[d]) — per-second average rate of increase over duration d. Use this for counters on dashboards. Always use rate() over [5m] or longer; never over less than two scrape intervals.

irate(v[d]) — instantaneous rate, looking at only the last two data points. More responsive but spikier. Useful for alerting on sudden bursts.

increase(v[d]) — total increase in a counter over duration d. Good for “requests in the last hour”.

sum(v) by (label) — aggregate by summing across the given label dimension.

avg(v) by (label), max(v) by (label), min(v) by (label) — other aggregations.

predict_linear(v[d], t) — linear regression: given the trend over d, what will the value be in t seconds? Essential for disk capacity planning.

histogram_quantile(φ, v) — compute the φ-quantile (e.g., 0.95 for p95) from a histogram.

Common Patterns You Will Use Every Day

CPU usage percentage per host:

1
2
3
4
5


100 - (
  avg by (instance) (
    rate(node_cpu_seconds_total{mode="idle"}[5m])
  ) * 100
)

This takes the idle CPU rate, averages across all cores for each instance, converts to a percentage, then inverts to get usage.

Memory usage percentage:

1

(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100

Use MemAvailable rather than MemFree. Available accounts for reclaimable cache, which is what the kernel actually has to offer new processes.

Disk usage percentage for root filesystem:

1
2
3
4


(
  1 - node_filesystem_avail_bytes{mountpoint="/", fstype!="tmpfs"}
    / node_filesystem_size_bytes{mountpoint="/", fstype!="tmpfs"}
) * 100

Disk filling up prediction — how many hours until full:

1
2
3
4


predict_linear(
  node_filesystem_avail_bytes{mountpoint="/"}[6h],
  24 * 3600
) < 0

This fires when the linear projection based on the last 6 hours of growth predicts the disk will be full within 24 hours.

Network receive rate in bytes/sec:

1

rate(node_network_receive_bytes_total{device!~"lo|veth.*|docker.*|br-.*"}[5m])

The label filter excludes loopback, veth (Docker internal), and bridge interfaces to show only physical/WAN traffic.

HTTP error rate (5xx):

1
2


rate(http_requests_total{status=~"5.."}[5m])
  / rate(http_requests_total[5m])

Container CPU usage percentage:

1
2
3


sum by (name) (
  rate(container_cpu_usage_seconds_total{name!=""}[5m])
) * 100

HTTP probe uptime (Blackbox):

1

probe_success{job="blackbox_http"}

TLS certificate expiry in days:

1

(probe_ssl_earliest_cert_expiry - time()) / 86400

p95 request latency from histogram:

1
2
3
4
5
6


histogram_quantile(
  0.95,
  sum by (le, job) (
    rate(http_request_duration_seconds_bucket[5m])
  )
)

Part 7: Recording Rules and Alerting Rules

Recording Rules

Some PromQL expressions are expensive — they fan out across many time series and take hundreds of milliseconds to evaluate. Running these on every dashboard refresh is wasteful. Recording rules pre-compute these expressions and store the result as a new metric that is cheap to query.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34


# prometheus/rules/recording_rules.yml
groups:
  - name: node_recording_rules
    interval: 1m
    rules:

      # Pre-compute CPU usage per instance
      - record: instance:node_cpu_utilization:rate5m
        expr: |
          100 - (
            avg by (instance) (
              rate(node_cpu_seconds_total{mode="idle"}[5m])
            ) * 100
          )

      # Pre-compute memory usage per instance
      - record: instance:node_memory_utilization:ratio
        expr: |
          1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes

      # Pre-compute disk usage per mountpoint
      - record: instance:node_disk_utilization:ratio
        expr: |
          1 - (
            node_filesystem_avail_bytes{fstype!~"tmpfs|fuse.lxcfs"}
            / node_filesystem_size_bytes{fstype!~"tmpfs|fuse.lxcfs"}
          )

      # Network receive rate excluding internal interfaces
      - record: instance:node_network_receive_bytes:rate5m
        expr: |
          sum by (instance) (
            rate(node_network_receive_bytes_total{device!~"lo|veth.*|docker.*|br-.*"}[5m])
          )

Dashboard panels referencing instance:node_cpu_utilization:rate5m will return instantly because Prometheus already has the answer computed.

Alerting Rules

Alert rules evaluate a PromQL expression on each evaluation interval. When the expression returns results, Prometheus puts the alert into pending state. If it remains pending for the duration specified in for, Prometheus fires the alert to Alertmanager.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110


# prometheus/rules/node_alerts.yml
groups:
  - name: node_alerts
    rules:

      # Host is down — scrape failed
      - alert: HostDown
        expr: up{job="node"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Host {{ $labels.instance }} is down"
          description: "{{ $labels.instance }} has been unreachable for more than 1 minute."
          runbook_url: "https://wiki.yourdomain.com/runbooks/host-down"

      # CPU saturation
      - alert: HighCPUUsage
        expr: instance:node_cpu_utilization:rate5m > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU on {{ $labels.instance }}"
          description: "CPU usage is {{ $value | printf \"%.1f\" }}% on {{ $labels.instance }} (threshold: 85%)."

      - alert: CriticalCPUUsage
        expr: instance:node_cpu_utilization:rate5m > 95
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Critical CPU on {{ $labels.instance }}"
          description: "CPU usage is {{ $value | printf \"%.1f\" }}% on {{ $labels.instance }}."

      # Memory pressure
      - alert: HighMemoryUsage
        expr: instance:node_memory_utilization:ratio > 0.90
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High memory usage on {{ $labels.instance }}"
          description: "Memory is {{ $value | humanizePercentage }} used on {{ $labels.instance }}."

      # Disk usage
      - alert: DiskSpaceLow
        expr: |
          instance:node_disk_utilization:ratio{mountpoint="/"} > 0.85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Disk space low on {{ $labels.instance }}"
          description: "Disk {{ $labels.mountpoint }} is {{ $value | humanizePercentage }} full on {{ $labels.instance }}."

      - alert: DiskSpaceCritical
        expr: |
          instance:node_disk_utilization:ratio{mountpoint="/"} > 0.95
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Disk space critical on {{ $labels.instance }}"
          description: "Disk {{ $labels.mountpoint }} is {{ $value | humanizePercentage }} full on {{ $labels.instance }}."

      # Disk filling up — predict_linear
      - alert: DiskFillingUp
        expr: |
          predict_linear(
            node_filesystem_avail_bytes{
              mountpoint="/",
              fstype!~"tmpfs|fuse.lxcfs"
            }[6h],
            24 * 3600
          ) < 0
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "Disk will fill up within 24 hours on {{ $labels.instance }}"
          description: "At the current fill rate, {{ $labels.mountpoint }} on {{ $labels.instance }} will be full within 24 hours."

      # High system load
      - alert: HighLoadAverage
        expr: |
          node_load5
            / on(instance) group_left()
          count by (instance) (
            node_cpu_seconds_total{mode="idle"}
          ) > 2.0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High load average on {{ $labels.instance }}"
          description: "Load average (5m) per CPU is {{ $value | printf \"%.2f\" }} on {{ $labels.instance }}."

      # High iowait — usually disk bottleneck
      - alert: HighIOWait
        expr: |
          avg by (instance) (
            rate(node_cpu_seconds_total{mode="iowait"}[5m])
          ) * 100 > 20
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High I/O wait on {{ $labels.instance }}"
          description: "iowait CPU is {{ $value | printf \"%.1f\" }}% on {{ $labels.instance }}, indicating disk saturation."

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71


# prometheus/rules/container_alerts.yml
groups:
  - name: container_alerts
    rules:

      # Container restarting repeatedly
      - alert: ContainerRestarting
        expr: |
          increase(kube_pod_container_status_restarts_total[1h]) > 3
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Container {{ $labels.container }} is restarting"
          description: "Container {{ $labels.container }} in pod {{ $labels.pod }} has restarted {{ $value }} times in the last hour."

      # Container OOM killed
      - alert: ContainerOOMKilled
        expr: |
          kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1
        for: 0m
        labels:
          severity: warning
        annotations:
          summary: "Container OOM killed: {{ $labels.container }}"
          description: "Container {{ $labels.container }} was OOM killed. Consider increasing memory limits."

      # cAdvisor-based: container using >90% of its memory limit
      - alert: ContainerHighMemory
        expr: |
          (
            container_memory_working_set_bytes{name!=""}
            / container_spec_memory_limit_bytes{name!=""}
          ) > 0.9
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Container {{ $labels.name }} near memory limit"
          description: "Container {{ $labels.name }} is using {{ $value | humanizePercentage }} of its memory limit."

      # HTTP endpoint down (Blackbox)
      - alert: EndpointDown
        expr: probe_success{job="blackbox_http"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Endpoint {{ $labels.instance }} is down"
          description: "HTTP probe for {{ $labels.instance }} has been failing for 2 minutes."

      # TLS certificate expiring soon
      - alert: TLSCertExpiringSoon
        expr: |
          (probe_ssl_earliest_cert_expiry - time()) / 86400 < 21
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "TLS cert for {{ $labels.instance }} expires soon"
          description: "Certificate for {{ $labels.instance }} expires in {{ $value | printf \"%.0f\" }} days."

      - alert: TLSCertExpiringCritical
        expr: |
          (probe_ssl_earliest_cert_expiry - time()) / 86400 < 7
        for: 1h
        labels:
          severity: critical
        annotations:
          summary: "TLS cert for {{ $labels.instance }} expires in < 7 days"
          description: "Certificate for {{ $labels.instance }} expires in {{ $value | printf \"%.0f\" }} days. Renew immediately."

Validate your rule files before applying them:

1
2


promtool check rules prometheus/rules/*.yml
promtool check config prometheus/prometheus.yml

Part 8: Alertmanager Configuration

Alertmanager receives alerts from Prometheus and decides what to do with them. Its job is routing, deduplication, grouping, silencing, and delivery. The configuration is a tree of routes where each alert flows down until it matches a receiver.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113


# alertmanager/alertmanager.yml
global:
  resolve_timeout: 5m
  smtp_from: "alertmanager@yourdomain.com"
  smtp_smarthost: "smtp.yourdomain.com:587"
  smtp_auth_username: "alertmanager@yourdomain.com"
  smtp_auth_password: "your-smtp-password"
  smtp_require_tls: true
  slack_api_url: "https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK"

route:
  # Default grouping and timing
  receiver: "slack-warnings"
  group_by: ["alertname", "instance"]
  group_wait: 30s        # Wait before sending the first notification in a group
  group_interval: 5m     # Wait before sending updated notifications for existing groups
  repeat_interval: 4h    # Re-notify if still firing after this interval

  routes:
    # Critical alerts get immediate Slack notification
    - match:
        severity: critical
      receiver: "slack-critical"
      group_wait: 10s
      repeat_interval: 1h
      continue: true   # Also fall through to the next matching route

    # Critical alerts also page via email
    - match:
        severity: critical
      receiver: "email-oncall"
      group_wait: 10s
      repeat_interval: 2h

    # Warning alerts go to the standard Slack channel
    - match:
        severity: warning
      receiver: "slack-warnings"
      repeat_interval: 4h

    # Suppress disk-filling alerts if disk-critical is also firing
    # (inhibit_rules handles this better — see below)

receivers:
  - name: "slack-critical"
    slack_configs:
      - channel: "#alerts-critical"
        send_resolved: true
        title: '{{ template "slack.title" . }}'
        text: |
          {{ range .Alerts }}
          *Alert:* {{ .Annotations.summary }}
          *Severity:* `{{ .Labels.severity }}`
          *Instance:* `{{ .Labels.instance }}`
          *Description:* {{ .Annotations.description }}
          {{ if .Annotations.runbook_url }}*Runbook:* {{ .Annotations.runbook_url }}{{ end }}
          {{ end }}
        color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}'

  - name: "slack-warnings"
    slack_configs:
      - channel: "#alerts-warnings"
        send_resolved: true
        title: '[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .GroupLabels.alertname }}'
        text: |
          {{ range .Alerts }}
          *{{ .Annotations.summary }}*
          {{ .Annotations.description }}
          {{ end }}
        color: '{{ if eq .Status "firing" }}warning{{ else }}good{{ end }}'

  - name: "email-oncall"
    email_configs:
      - to: "oncall@yourdomain.com"
        send_resolved: true
        headers:
          subject: '[{{ .Status | toUpper }}] {{ .GroupLabels.alertname }} on {{ .GroupLabels.instance }}'
        html: |
          <h2>{{ .GroupLabels.alertname }}</h2>
          {{ range .Alerts }}
          <p><strong>{{ .Annotations.summary }}</strong></p>
          <p>{{ .Annotations.description }}</p>
          {{ end }}

  - name: "null"
    # A do-nothing receiver for suppressed alerts

# Inhibition rules suppress lower-severity alerts when higher-severity fires
inhibit_rules:
  # If a host is down, suppress all other alerts from that host
  - source_match:
      alertname: HostDown
    target_match_re:
      alertname: ".+"
    equal:
      - instance

  # If disk is critical, suppress the disk-low warning
  - source_match:
      alertname: DiskSpaceCritical
    target_match:
      alertname: DiskSpaceLow
    equal:
      - instance
      - mountpoint

  # If CPU is critical, suppress the CPU warning
  - source_match:
      alertname: CriticalCPUUsage
    target_match:
      alertname: HighCPUUsage
    equal:
      - instance

Managing Alerts with amtool

amtool is the CLI for interacting with Alertmanager. Useful commands:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


# List currently firing alerts
amtool alert --alertmanager.url=http://localhost:9093

# Create a silence for 4 hours (maintenance window)
amtool silence add \
  --alertmanager.url=http://localhost:9093 \
  --duration=4h \
  --comment="Scheduled maintenance" \
  alertname="HostDown" instance="homelab-01:9100"

# List active silences
amtool silence --alertmanager.url=http://localhost:9093

# Expire a silence by ID
amtool silence expire --alertmanager.url=http://localhost:9093 <silence-id>

# Check Alertmanager config
amtool check-config alertmanager/alertmanager.yml

Part 9: Grafana Dashboards

Navigate to http://localhost:3000 and log in with the credentials you set in docker-compose.yml (admin / changeme). Change the admin password immediately.

If you used the datasource provisioning file from Part 3, Prometheus is already wired up. Otherwise, go to Configuration → Data Sources → Add data source, select Prometheus, set the URL to http://prometheus:9090 (the Docker service name), and click Save & Test.

Community Dashboards Worth Importing

Rather than building everything from scratch, import these battle-tested dashboards from the Grafana dashboard catalogue (grafana.com/grafana/dashboards):

Dashboard	ID	What it shows
Node Exporter Full	1860	Complete host metrics: CPU, memory, disk, network, filesystem
cAdvisor	14282	Container resource usage with per-container breakdown
Blackbox Exporter	7587	Endpoint uptime, latency, TLS expiry
Prometheus Stats	2	Prometheus’s own performance and health
Alertmanager	9578	Alert routing and notification stats

Import via Dashboards → Import → Enter dashboard ID, select your Prometheus datasource, and click Import.

Building a “Host Overview” Dashboard

To understand how Grafana works, build a simple host overview dashboard from scratch. Go to Dashboards → New Dashboard → Add visualization.

Dashboard Variables make dashboards reusable. Add a variable: Settings → Variables → New variable:

Name: instance
Type: Query
Query: label_values(node_uname_info, instance)
Multi-value: enabled
Include All: enabled

Now every panel can use $instance in its query, and the dropdown at the top of the dashboard lets you switch between hosts.

CPU Usage Time Series Panel:

Query:

1
2
3
4
5


100 - (
  avg by (instance) (
    rate(node_cpu_seconds_total{mode="idle", instance=~"$instance"}[5m])
  ) * 100
)

Panel type: Time series
Unit: Percent (0-100)
Add threshold at 85 (warning yellow) and 95 (critical red)

Memory Usage Stat Panel:

Query:

1
2


(1 - node_memory_MemAvailable_bytes{instance=~"$instance"}
  / node_memory_MemTotal_bytes{instance=~"$instance"}) * 100

Panel type: Stat
Unit: Percent (0-100)
Color mode: Background, thresholds at 80 (yellow) and 90 (red)

Disk Usage Gauge Panel:

Query:

1
2


(1 - node_filesystem_avail_bytes{instance=~"$instance", mountpoint="/", fstype!="tmpfs"}
  / node_filesystem_size_bytes{instance=~"$instance", mountpoint="/", fstype!="tmpfs"}) * 100

Panel type: Gauge
Min: 0, Max: 100
Thresholds: 80 (yellow), 95 (red)

Network I/O Time Series Panel:

Query A (Receive):

1
2
3
4


rate(node_network_receive_bytes_total{
  instance=~"$instance",
  device!~"lo|veth.*|docker.*|br-.*"
}[5m])

Query B (Transmit, negated for the classic in/out display):

1
2
3
4


-rate(node_network_transmit_bytes_total{
  instance=~"$instance",
  device!~"lo|veth.*|docker.*|br-.*"
}[5m])

Panel type: Time series
Unit: bytes/sec

System Load Panel:

Query:

1
2
3


node_load5{instance=~"$instance"}
  / on(instance) group_left()
count by (instance) (node_cpu_seconds_total{mode="idle", instance=~"$instance"})

This normalizes load by CPU count so the alert threshold of 1.0 means “fully saturated”.

Dashboard Provisioning (GitOps for Dashboards)

Rather than clicking through the Grafana UI and hoping your dashboards survive a container rebuild, provision them from JSON files:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


# grafana/provisioning/dashboards/dashboards.yml
apiVersion: 1

providers:
  - name: "default"
    orgId: 1
    folder: "Provisioned"
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    allowUiUpdates: true
    options:
      path: /etc/grafana/provisioning/dashboards/json

Export a dashboard as JSON from Dashboard Settings → JSON Model, save it to grafana/provisioning/dashboards/json/host-overview.json, and mount the provisioning directory into the Grafana container (already done in the docker-compose.yml above). Grafana picks up JSON files automatically. Now your dashboards live in git and survive container rebuilds.

Grafana Unified Alerting

Grafana ships its own alerting engine (Unified Alerting) that can query any datasource — not just Prometheus. For teams that want all alerting in one place:

Go to Alerting → Contact Points → New contact point, configure Slack or email
Go to Alerting → Notification policies, set the default policy to your contact point
Go to Alerting → Alert rules → New alert rule, write a PromQL expression, set evaluation interval, and configure thresholds

Grafana Unified Alerting is convenient but less powerful than Prometheus alerting rules + Alertmanager for complex routing and inhibition. For homelab use, either works. For production multi-tenant setups, the Prometheus + Alertmanager path scales better.

Part 10: Long-Term Storage and Scaling

Local Storage Limits

By default, Prometheus stores data on local disk. With 15-second scrape intervals and a modest number of targets, you can expect roughly 1-3 GB per day. The --storage.tsdb.retention.time=30d flag keeps 30 days of data on disk.

The tradeoff: local storage is fast and simple, but it does not survive disk failures, it cannot be queried across multiple Prometheus instances, and 30 days of history is limiting for capacity planning.

Remote Write to Long-Term Storage

Prometheus supports remote_write — shipping samples to an external storage backend in real time while keeping local storage for fast recent queries. Add to prometheus.yml:

1
2
3
4
5
6


remote_write:
  - url: "http://victoriametrics:8428/api/v1/write"
    queue_config:
      capacity: 10000
      max_samples_per_send: 5000
      batch_send_deadline: 5s

VictoriaMetrics is the most popular remote write target for homelabs and small teams. It is a drop-in Prometheus-compatible TSDB with 10x better compression than Prometheus, faster ingestion, and a compatible query API. A single-node VictoriaMetrics instance can handle hundreds of thousands of active time series on modest hardware.

Thanos is the production-grade option for multi-Prometheus setups. The Thanos sidecar attaches to each Prometheus instance, uploads blocks to object storage (S3, GCS, Azure Blob), and the Thanos Store component serves historical data back to Grafana with a unified query interface. This gives you unlimited retention at object storage prices and global querying across multiple Prometheus instances.

Grafana Mimir (formerly Cortex) is the cloud-native, horizontally scalable option — designed for thousands of tenants and billions of active series. Overkill for a homelab, but the architecture Grafana Cloud runs on.

Part 11: Tips and Best Practices

Cardinality: The Number One Footgun

Cardinality is the number of unique time series in Prometheus. Each unique combination of metric name + label set is one time series. High cardinality is the most common way to accidentally fill a Prometheus instance’s memory and crash it.

What causes high cardinality:

Using user IDs, email addresses, or session tokens as label values
Including URL paths with IDs in them (e.g., /api/user/12345)
Putting IP addresses or UUIDs in labels without aggregation

The rule: label values should have a bounded, small set of possible values. status_code is fine (200, 404, 500 — handful of values). user_id is never fine.

1
2
3
4
5


# Bad — creates a time series per user
http_requests_total{user_id="12345", path="/api/user/12345/settings"}

# Good — aggregate before labeling
http_requests_total{status="200", endpoint="/api/user/:id/settings"}

Check the top cardinality metrics in the Prometheus UI under Status → TSDB Status.

Scrape Interval vs Evaluation Interval

Your scrape_interval (how often Prometheus fetches metrics) and evaluation_interval (how often rules are evaluated) should be the same value or multiples of each other. The scrape interval determines your resolution. 15 seconds is a good default — low enough to catch short spikes, high enough to keep cardinality manageable.

For evaluation intervals, 15-30 seconds is standard. Setting for: 5m on an alert with a 15-second evaluation interval means the condition must hold for at least 20 consecutive evaluations.

Label Naming Conventions

Follow the Prometheus naming conventions to avoid confusion:

Metric names: snake_case, with the unit as a suffix (_bytes, _seconds, _total)
Always use base units — bytes not kilobytes, seconds not milliseconds
Counters must end in _total
Histograms automatically get _bucket, _count, and _sum suffixes

Validate Before Deploying

Always run promtool before applying changes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# Validate prometheus.yml
promtool check config prometheus/prometheus.yml

# Validate rule files
promtool check rules prometheus/rules/*.yml

# Test a PromQL query against a live server
promtool query instant http://localhost:9090 'up'

# Run unit tests for alerting rules
promtool test rules prometheus/tests/*.yml

Keep Dashboards in Git

Grafana dashboards are JSON. Export them, commit them, provision them via the filesystem. This means:

Dashboard changes are reviewed like code (pull requests, diff)
Dashboards survive container rebuilds and data loss
You can reproduce the entire observability stack from git in minutes

The single biggest operational mistake in Grafana setups is treating dashboards as ephemeral UI artifacts. They are configuration. Treat them accordingly.

Putting It All Together

Start with the Docker Compose stack and Node Exporter. Within five minutes you will have CPU, memory, disk, and network graphs in Grafana. Import dashboard 1860 (Node Exporter Full) and you will have a comprehensive host overview immediately.

Add the alerting rules from Part 7. Run promtool check rules to validate them, hot-reload Prometheus, and watch the Alertmanager silence rules page for the next few days to tune the thresholds for your environment. What fires constantly in a quiet homelab will be signal in a production environment, and vice versa.

Add cAdvisor for container visibility. Import dashboard 14282. Now you can see which container is consuming the most CPU or memory at any moment.

Add Blackbox Exporter for uptime monitoring. Import dashboard 7587. Configure it to probe every public-facing URL. Add the TLS certificate expiry alert. You will never be surprised by an expired certificate again.

From there, the stack grows with your needs. Add database exporters as you deploy databases. Add remote_write to VictoriaMetrics when you want longer history. Add Thanos when you want to query across multiple sites.

The Prometheus + Grafana stack is genuinely one of the highest-leverage tools in infrastructure. A few hours of setup pays for itself the first time something breaks and you know exactly where to look.

For the logs side of the observability stack — shipping, querying, and correlating with metrics — see the upcoming post on Loki for Log Aggregation.

Part 1: Why Prometheus + Grafana?

Part 2: Architecture Overview

Part 3: Installation with Docker Compose

Part 4: Prometheus Configuration

Part 5: Key Exporters

Node Exporter

cAdvisor

Blackbox Exporter

Database Exporters

Application Instrumentation

Part 6: PromQL Fundamentals

Data Types

Label Matchers

Essential Functions

Common Patterns You Will Use Every Day

Part 7: Recording Rules and Alerting Rules

Recording Rules

Alerting Rules

Part 8: Alertmanager Configuration

Managing Alerts with amtool

Part 9: Grafana Dashboards

First Login and Datasource Setup

Community Dashboards Worth Importing

Building a “Host Overview” Dashboard

Dashboard Provisioning (GitOps for Dashboards)

Grafana Unified Alerting

Part 10: Long-Term Storage and Scaling

Local Storage Limits

Remote Write to Long-Term Storage

Part 11: Tips and Best Practices

Cardinality: The Number One Footgun

Scrape Interval vs Evaluation Interval

Label Naming Conventions

Validate Before Deploying

Keep Dashboards in Git

Putting It All Together

Comments