Grafana + Prometheus Homelab Stack

The first time your NAS fills up and silently drops writes, or your UPS battery dies the night before a power outage takes out your server, you learn the lesson: the homelab is not a toy environment, it just doesn’t have an on-call team. The machines don’t care that you’re asleep. The disk doesn’t warn you it’s 98% full before it starts corrupting files, and the switch port that went half-duplex three weeks ago isn’t going to file a ticket about the 40% packet loss you haven’t noticed yet.

Observability is the answer, but “observability” is a word that gets thrown around in a lot of contexts. What it means practically for a homelab is this: you want to be paged when something breaks, ideally before something else breaks as a consequence. You want to open a dashboard and immediately understand the state of your infrastructure. You want to look at a graph from two weeks ago and understand what was happening when that backup job started failing.

This guide builds a full observability stack from scratch: Prometheus for metric collection and storage, a suite of exporters to instrument every layer of the stack, Alertmanager for routing notifications, Grafana for visualization and provisioned dashboards, and options for long-term storage when 15 days of retention stops being enough.

Why Metrics, Why Prometheus

The “three pillars of observability” framing — metrics, logs, traces — is accurate but can obscure priorities. For a homelab, traces are largely irrelevant: you’re not debugging distributed transaction latency across microservices. Logs are useful but reactive; you read them after something breaks. Metrics with alerting is where you get the most value per unit of operational effort. A handful of well-chosen alerts on node disk usage, memory pressure, service availability, and certificate expiry covers the failure modes you’re actually likely to hit.

Prometheus is the right tool for this job. It was built from the ground up for exactly this use case — instrumenting infrastructure and services, collecting metrics on a pull schedule, and evaluating alert conditions against those metrics. The ecosystem of exporters is unmatched. The query language (PromQL) is expressive enough to compute anything interesting from the raw data. And the entire stack runs comfortably on a single machine with a few gigabytes of RAM.

The Prometheus data model is worth understanding before you start writing scrape configs. Every piece of data in Prometheus is a time series: a unique combination of a metric name and a set of key-value label pairs, with a sequence of timestamped float64 samples. The metric node_cpu_seconds_total{cpu="0",mode="idle",instance="homeserver:9100",job="node"} is one time series. Change any label value and you have a different time series. This is powerful — it lets you write a single query that aggregates across all CPU cores, all instances, or all jobs — but it also means that labels with high cardinality (many unique values, like container IDs or request paths) can explode your time series count.

Prometheus has four metric types:

• Counter: A monotonically increasing value. Resets to zero on restart. Used for request counts, bytes transferred, errors. Always query with rate() or increase(), never the raw value.
• Gauge: A value that can go up or down. Temperature, memory usage, queue depth. Query directly or with avg_over_time().
• Histogram: Samples observations into configurable buckets. Used for latency and request size distributions. Enables histogram_quantile() for percentile calculations.
• Summary: Pre-computes quantiles on the client side. Less flexible than histograms for aggregation. Generally prefer histograms for new instrumentation.

The pull model is what distinguishes Prometheus from older monitoring systems like Graphite or InfluxDB (in its traditional push configuration). Prometheus scrapes HTTP /metrics endpoints on a schedule. The targets don’t send data to Prometheus; Prometheus comes to them. This means the monitoring system has a complete, authoritative list of what it’s monitoring. If a target disappears from the scrape config, Prometheus stops collecting data for it. If a target’s /metrics endpoint stops responding, Prometheus records a scrape failure. There’s no silent data gap caused by a metrics agent that crashed quietly on the monitored host.

The Full Stack

Before diving into each component, here’s the architecture we’re building:

                         HOMELAB OBSERVABILITY STACK
  ─────────────────────────────────────────────────────────────────────

  Physical / VM Hosts                  Network Devices
  ┌─────────────────────┐              ┌──────────────────────┐
  │  node_exporter      │              │  Router / Switch      │
  │  :9100/metrics      │              │  NAS (Synology)       │
  │  (CPU, mem, disk,   │              │  UPS (APC)            │
  │   net, fs, sensors) │              │  speak SNMP only      │
  └──────────┬──────────┘              └──────────┬───────────┘
             │ scrape                              │ SNMP poll
  ┌──────────▼──────────┐              ┌──────────▼───────────┐
  │  cAdvisor           │              │  snmp_exporter        │
  │  :8080/metrics      │              │  :9116/metrics        │
  │  (per-container     │              │  (interface stats,    │
  │   CPU/mem/net)      │              │   UPS battery, etc.)  │
  └──────────┬──────────┘              └──────────┬───────────┘
             │                                    │
             │         HTTP probe targets         │ scrape
             │   ┌─────────────────────────┐      │
             │   │  blackbox_exporter      │      │
             │   │  :9115/probe            │      │
             │   │  (HTTP, ICMP, DNS,      │      │
             │   │   TCP, TLS cert expiry) │      │
             │   └──────────┬──────────────┘      │
             │              │                     │
  ┌──────────▼──────────────▼─────────────────────▼──────────┐
  │                      PROMETHEUS                           │
  │              :9090   /metrics  /-/reload                  │
  │                                                           │
  │  scrape_interval: 15s    evaluation_interval: 15s         │
  │  TSDB: /prometheus  (15-30 day local retention)           │
  │                                                           │
  │  alerting rules ─────────────┐                           │
  └──────────────────────────────┼───────────────────────────┘
                                 │ firing alerts
                    ┌────────────▼────────────┐
                    │     ALERTMANAGER         │
                    │     :9093               │
                    │  route → receivers      │
                    │  group / dedup / silence │
                    └──────┬──────────────────┘
                           │
              ┌────────────┴──────────────┐
              │                           │
         ┌────▼─────┐              ┌──────▼──────┐
         │  Slack   │              │   Email     │
         │  webhook │              │   SMTP      │
         └──────────┘              └─────────────┘

                    ┌─────────────────────────┐
                    │        GRAFANA           │
                    │        :3000            │
                    │  datasource: Prometheus  │
                    │  provisioned dashboards  │
                    │  unified alerting        │
                    └─────────────────────────┘

Every component in this diagram runs as a Docker container. The entire stack is defined in a single docker-compose.yml, which means you can tear it down and rebuild it in under two minutes, move it to a different host, or version-control the entire configuration in git.

Deploying with Docker Compose

Version pinning is non-negotiable for a stable homelab stack. Using latest tags means your stack can break on a Monday morning when the upstream image is updated over the weekend. Pin to specific versions and upgrade deliberately.

As of late May 2026, the relevant versions are: Prometheus 3.x (v3.11.3 is the latest patch, v3.5.x is the current LTS branch), Grafana 13.0.1, Alertmanager 0.32.1, Node Exporter 1.11.1, cAdvisor 0.57.0. The Compose file below uses these versions.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134

# docker-compose.yml
# Grafana + Prometheus observability stack
# All config files are mounted from ./config/ subdirectories

networks:
  monitoring:
    driver: bridge

volumes:
  prometheus_data: {}
  grafana_data: {}
  alertmanager_data: {}

services:

  prometheus:
    image: prom/prometheus:v3.11.3
    container_name: prometheus
    restart: unless-stopped
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=30d'
      - '--storage.tsdb.retention.size=20GB'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      - '--web.enable-lifecycle'
    volumes:
      - ./config/prometheus:/etc/prometheus:ro
      - prometheus_data:/prometheus
    ports:
      - "9090:9090"
    networks:
      - monitoring
    depends_on:
      - node-exporter
      - cadvisor
      - blackbox-exporter

  alertmanager:
    image: prom/alertmanager:v0.32.1
    container_name: alertmanager
    restart: unless-stopped
    command:
      - '--config.file=/etc/alertmanager/alertmanager.yml'
      - '--storage.path=/alertmanager'
      - '--cluster.advertise-address=0.0.0.0:9093'
    volumes:
      - ./config/alertmanager:/etc/alertmanager:ro
      - alertmanager_data:/alertmanager
    ports:
      - "9093:9093"
    networks:
      - monitoring

  grafana:
    image: grafana/grafana:13.0.1
    container_name: grafana
    restart: unless-stopped
    environment:
      - GF_SECURITY_ADMIN_USER=${GRAFANA_ADMIN_USER:-admin}
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD}
      - GF_USERS_ALLOW_SIGN_UP=false
      - GF_SERVER_ROOT_URL=https://grafana.yourdomain.local
      - GF_SMTP_ENABLED=false
    volumes:
      - grafana_data:/var/lib/grafana
      - ./config/grafana/provisioning:/etc/grafana/provisioning:ro
    ports:
      - "3000:3000"
    networks:
      - monitoring
    depends_on:
      - prometheus

  node-exporter:
    image: quay.io/prometheus/node-exporter:v1.11.1
    container_name: node-exporter
    restart: unless-stopped
    command:
      - '--path.rootfs=/host'
      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
      - '--collector.textfile.directory=/var/lib/node_exporter/textfile_collector'
    network_mode: host
    pid: host
    volumes:
      - /:/host:ro,rslave
      - ./config/node-exporter/textfile:/var/lib/node_exporter/textfile_collector:ro

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.57.0
    container_name: cadvisor
    restart: unless-stopped
    privileged: true
    devices:
      - /dev/kmsg
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker:/var/lib/docker:ro
      - /dev/disk:/dev/disk:ro
    ports:
      - "8080:8080"
    networks:
      - monitoring

  blackbox-exporter:
    image: prom/blackbox-exporter:v0.25.0
    container_name: blackbox-exporter
    restart: unless-stopped
    cap_add:
      - NET_RAW
    command:
      - '--config.file=/etc/blackbox/blackbox.yml'
    volumes:
      - ./config/blackbox:/etc/blackbox:ro
    ports:
      - "9115:9115"
    networks:
      - monitoring

  snmp-exporter:
    image: prom/snmp-exporter:v0.26.0
    container_name: snmp-exporter
    restart: unless-stopped
    command:
      - '--config.file=/etc/snmp_exporter/snmp.yml'
    volumes:
      - ./config/snmp:/etc/snmp_exporter:ro
    ports:
      - "9116:9116"
    networks:
      - monitoring

A few design decisions worth noting. Node Exporter uses network_mode: host because it needs to see the host’s network interfaces accurately — running it behind Docker’s bridge NAT gives you wrong interface metrics. The pid: host flag is needed so it can read /proc data for host processes rather than the container’s process namespace. cAdvisor requires privileged: true and /dev/kmsg to read cgroup statistics for all containers.

Grafana’s admin password is read from an environment variable, not hardcoded. Keep a .env file in the stack directory:

1
2

# .env — do not commit to git
GRAFANA_ADMIN_PASSWORD=your-strong-password-here

Prometheus is exposed on port 9090 without authentication. This is fine if you’re behind a firewall, but if Grafana is internet-facing, put both Prometheus and Alertmanager behind a reverse proxy (Caddy or Traefik) with basic auth or OAuth. Exposing an unauthenticated Prometheus endpoint to the public internet means anyone can read all your metrics, which includes hostnames, service names, and IP addresses from your internal network.

The --web.enable-lifecycle flag on Prometheus enables the /-/reload endpoint, letting you reload the configuration without restarting the container:

1

curl -X POST http://localhost:9090/-/reload

Always validate the config before reloading:

1

docker exec prometheus promtool check config /etc/prometheus/prometheus.yml

A failed reload leaves the old configuration running. A failed restart leaves Prometheus down. Always check before reloading.

Prometheus Configuration

The prometheus.yml is the nerve center of the stack. It defines how often to scrape, where to send alerts, and what to scrape. Here is a complete configuration for the stack above:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128

# config/prometheus/prometheus.yml

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  scrape_timeout: 10s
  external_labels:
    cluster: homelab
    environment: home

# Alerting configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:9093
      timeout: 10s

# Load alerting rules from separate files
rule_files:
  - "rules/*.yml"

scrape_configs:

  # Prometheus itself
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']
        labels:
          instance: prometheus

  # Node Exporter — host-level system metrics
  # node-exporter uses host networking, so target the Docker host IP
  - job_name: node
    static_configs:
      - targets:
          - '192.168.1.10:9100'   # main server (adjust to your host IPs)
          - '192.168.1.11:9100'   # secondary node
        labels:
          env: homelab
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
        regex: '(.+):\d+'
        replacement: '${1}'

  # cAdvisor — per-container metrics
  - job_name: cadvisor
    static_configs:
      - targets: ['cadvisor:8080']
        labels:
          instance: docker-host
    metric_relabel_configs:
      # Drop high-cardinality metrics we don't need
      - source_labels: [__name__]
        regex: 'container_tasks_state|container_memory_failures_total'
        action: drop

  # Blackbox Exporter — HTTP probe targets
  - job_name: blackbox_http
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://grafana.yourdomain.local
          - https://prometheus.yourdomain.local
          - https://homeassistant.yourdomain.local:8123
          - https://nextcloud.yourdomain.local
          - http://192.168.1.1           # router admin
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

  # Blackbox Exporter — ICMP ping probes
  - job_name: blackbox_icmp
    metrics_path: /probe
    params:
      module: [icmp]
    static_configs:
      - targets:
          - 192.168.1.1      # gateway
          - 8.8.8.8          # Google DNS (external connectivity check)
          - 192.168.1.20     # NAS
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

  # SNMP Exporter — network devices
  - job_name: snmp_ubiquiti
    metrics_path: /snmp
    params:
      module: [ubiquiti_unifi]
    static_configs:
      - targets:
          - 192.168.1.1    # UniFi gateway
          - 192.168.1.2    # managed switch
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: snmp-exporter:9116

  # SNMP Exporter — UPS (APC)
  - job_name: snmp_apc
    metrics_path: /snmp
    params:
      module: [apcups]
    static_configs:
      - targets:
          - 192.168.1.30    # APC Smart-UPS
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: snmp-exporter:9116

The multi-target pattern used by Blackbox Exporter and SNMP Exporter deserves explanation because it’s non-obvious. These exporters don’t scrape themselves — they act as HTTP-to-probe translators. When Prometheus scrapes /probe?target=https://grafana.yourdomain.local&module=http_2xx, the blackbox exporter performs that HTTP probe and returns the result as a metrics response. The relabel configuration in the scrape config rewrites __address__ (which Prometheus uses as the scrape URL host) to point at the exporter, while copying the original target address into __param_target. The instance label ends up as the probed URL, not the exporter address, which is what you want for dashboards and alerts.

Recording rules let you pre-compute expensive queries. Instead of computing a 5-minute CPU rate across all cores every time Grafana loads a dashboard panel, you compute it once per evaluation interval and store the result as a new time series. This is especially valuable for aggregations that span many series.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

# config/prometheus/rules/recording.yml
groups:
  - name: node_recording_rules
    interval: 1m
    rules:

      # Per-instance CPU utilization (0-100)
      - record: instance:node_cpu_utilization:rate5m
        expr: |
          100 - (
            avg by(instance) (
              rate(node_cpu_seconds_total{mode="idle"}[5m])
            ) * 100
          )

      # Per-instance memory utilization (0-100)
      - record: instance:node_memory_utilization:ratio
        expr: |
          (1 - (
            node_memory_MemAvailable_bytes /
            node_memory_MemTotal_bytes
          )) * 100

      # Per-device disk utilization for root filesystem
      - record: instance:node_filesystem_utilization:ratio
        expr: |
          100 - (
            node_filesystem_avail_bytes{mountpoint="/", fstype!="tmpfs"} /
            node_filesystem_size_bytes{mountpoint="/", fstype!="tmpfs"} * 100
          )

Node Exporter

Node Exporter is the canonical way to get Linux host metrics into Prometheus. It exposes everything the kernel knows about the system: CPU utilization broken down by mode and core, memory in extraordinary detail (total, free, available, buffers, cached, swap), disk I/O per block device (reads, writes, IOPS, latency, utilization), network statistics per interface (bytes, packets, errors, drops), filesystem usage per mount point (bytes, inodes), system load averages, NTP clock offset, and hardware sensor readings via hwmon (temperatures, fan speeds, voltages).

Running Node Exporter as a Docker container works but requires careful volume mounts to see the host’s actual metrics rather than the container’s view. The complete set of required mounts:

/:/host:ro,rslave        → rootfs, accessed via --path.rootfs=/host
/proc → (via rootfs mount)
/sys  → (via rootfs mount)
network_mode: host       → see actual network interfaces
pid: host                → see actual host PIDs for process metrics

Running it as a systemd service is simpler and avoids these mount complications. For a homelab where the host runs a stable distro:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

# Download and install
wget https://github.com/prometheus/node_exporter/releases/download/v1.11.1/node_exporter-1.11.1.linux-amd64.tar.gz
tar xvfz node_exporter-1.11.1.linux-amd64.tar.gz
sudo cp node_exporter-1.11.1.linux-amd64/node_exporter /usr/local/bin/

# Create systemd unit
sudo tee /etc/systemd/system/node_exporter.service <<EOF
[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter \
  --collector.textfile.directory=/var/lib/node_exporter/textfile_collector \
  --collector.systemd \
  --collector.processes
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter

The textfile collector is one of Node Exporter’s most underappreciated features. Any .prom file you drop into the configured textfile directory gets included in the /metrics output. This means any script can export custom metrics without running its own HTTP server. RAID status, backup job results, disk SMART data, ZFS pool health — anything a script can determine can be a Prometheus metric:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

#!/bin/bash
# /usr/local/bin/zfs-health-metrics.sh
# Run via cron every 5 minutes, write to textfile collector dir

OUTFILE=/var/lib/node_exporter/textfile_collector/zfs_health.prom
TMP=$(mktemp)

zpool list -H -o name,health,size,alloc,free,capacity | while read name health size alloc free cap; do
  healthy=0
  [ "$health" = "ONLINE" ] && healthy=1
  cap_num=${cap%\%}
  echo "zfs_pool_healthy{pool=\"$name\"} $healthy" >> $TMP
  echo "zfs_pool_capacity_percent{pool=\"$name\"} $cap_num" >> $TMP
done

mv $TMP $OUTFILE

Essential PromQL queries for Node Exporter data:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

# CPU utilization across all cores for a single instance
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle", instance="homeserver"}[5m])) * 100)

# Memory available as percentage
node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100

# Disk space remaining on root filesystem
100 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} * 100)

# Disk read throughput (bytes/sec)
rate(node_disk_read_bytes_total[5m])

# Disk write throughput (bytes/sec)
rate(node_disk_written_bytes_total[5m])

# Network receive throughput
rate(node_network_receive_bytes_total{device!~"lo|veth.*|docker.*|br.*"}[5m])

# Load average relative to CPU count
node_load1 / count by(instance) (count by(instance, cpu) (node_cpu_seconds_total))

The network filter device!~"lo|veth.*|docker.*|br.*" deserves a note. On a Docker host, Node Exporter will expose metrics for every Docker bridge interface and every veth pair. A system with dozens of containers will have dozens of virtual interfaces. Filter them out at query time or with metric relabeling in the scrape config to avoid polluting your dashboards.

cAdvisor

cAdvisor (Container Advisor) is Google’s tool for collecting per-container resource metrics. It automatically discovers all running Docker containers by watching the Docker socket and reads their cgroup statistics directly. Without cAdvisor, you know your Docker host is using 80% of its memory but you don’t know which container is responsible.

The key metrics from cAdvisor:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# CPU usage per container (rate of CPU seconds consumed)
rate(container_cpu_usage_seconds_total{name!=""}[5m])

# Memory usage per container (bytes)
container_memory_usage_bytes{name!=""}

# Memory working set (excludes reclaimable cache — closer to "real" memory pressure)
container_memory_working_set_bytes{name!=""}

# Network receive bytes per container
rate(container_network_receive_bytes_total{name!=""}[5m])

# Top 5 memory consumers
topk(5, container_memory_working_set_bytes{name!=""})

When using Docker Compose, the container_label_com_docker_compose_service label gives you the service name, which is more readable than the container name or container ID:

1
2
3
4
5
6

# Memory by Compose service
sort_desc(
  sum by(container_label_com_docker_compose_service) (
    container_memory_working_set_bytes{container_label_com_docker_compose_service!=""}
  )
)

Cardinality warning: cAdvisor generates a lot of time series. Each container gets metrics for each of its network interfaces, each of its filesystems, and CPU in fine-grained detail. On a host running 50 containers, cAdvisor can easily produce 5,000-10,000 time series. Most of this is legitimate data you want, but be aware that short-lived containers (CI runners, batch jobs) will create time series that sit in the TSDB until the retention window expires, consuming memory even after the container is gone. The metric_relabel_configs drop rules in the Prometheus scrape config above are the right place to prune metrics you’ll never use.

Blackbox Exporter

Blackbox Exporter shifts your monitoring perspective from inside-out to outside-in. Node Exporter tells you the CPU is fine and the process is running. Blackbox Exporter tells you whether the service is actually reachable and returning valid responses from the perspective of the network path Prometheus sits on.

This matters in the homelab because you can have a container that is running fine (good CPU, good memory, process alive) but is not serving requests because a configuration file is malformed, a dependency is down, or a reverse proxy misconfiguration is returning 502s. Node Exporter and cAdvisor won’t see that. Blackbox Exporter will.

The blackbox.yml configuration defines probe modules:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

# config/blackbox/blackbox.yml

modules:

  http_2xx:
    prober: http
    timeout: 10s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: []  # defaults to 2xx
      method: GET
      follow_redirects: true
      preferred_ip_protocol: ip4
      tls_config:
        insecure_skip_verify: false

  http_2xx_insecure:
    prober: http
    timeout: 10s
    http:
      valid_status_codes: [200]
      tls_config:
        insecure_skip_verify: true  # for self-signed certs in internal services

  icmp:
    prober: icmp
    timeout: 5s
    icmp:
      preferred_ip_protocol: ip4

  tcp_connect:
    prober: tcp
    timeout: 5s

  dns_lookup:
    prober: dns
    timeout: 5s
    dns:
      transport_protocol: udp
      preferred_ip_protocol: ip4
      query_name: grafana.yourdomain.local
      query_type: A

The ICMP module requires either --privileged or the CAP_NET_RAW capability. The cap_add: [NET_RAW] in the Compose file above handles this without full privilege escalation.

Key metrics from Blackbox Exporter:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# Is the target up? (1 = probe succeeded, 0 = failed)
probe_success{job="blackbox_http"}

# HTTP response time in seconds
probe_duration_seconds{job="blackbox_http"}

# SSL certificate expiry — days until expiry
(probe_ssl_earliest_cert_expiry - time()) / 86400

# DNS resolution time
probe_dns_lookup_time_seconds

# ICMP round-trip time
probe_icmp_duration_seconds{phase="rtt"}

The SSL certificate expiry metric is one of the most practical alerts you can configure. A certificate expiring in 14 days is not an emergency but it is a thing you want to know about before users start seeing browser warnings. The alert for this is in the Alertmanager section below.

SNMP Exporter

SNMP is the protocol that lets you monitor devices you can’t install agents on — routers, managed switches, NASes, and UPS units. These devices speak SNMP natively, and the SNMP Exporter acts as a translator: Prometheus scrapes the SNMP Exporter, which in turn polls the target device via SNMP and translates the response into Prometheus metrics.

SNMP versions: v1 and v2c use a community string (typically “public” or something custom) as the only authentication mechanism. This community string is transmitted in cleartext. Use v3 if your devices support it — v3 adds authentication (MD5/SHA) and encryption (DES/AES). Most modern devices (UniFi, Cisco, Synology) support SNMPv3. The snmp.yml configuration file handles this per-module.

The snmp.yml file is not written by hand. The SNMP Exporter ships with a generator that takes MIB files and a generator config, and produces a snmp.yml that contains the OID-to-metric mappings. For common devices, pre-generated files exist in the community or the snmp_exporter repository:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# Using the pre-built snmp.yml from the community
# Available at: https://github.com/prometheus/snmp_exporter/tree/main/generator

# For custom vendor MIBs (e.g., Synology):
# 1. Download vendor MIB files
# 2. Write generator.yml specifying which OIDs to expose
# 3. Run the generator
docker run --rm \
  -v "${PWD}/mibs:/home/generator/mibs" \
  -v "${PWD}/generator.yml:/home/generator/generator.yml" \
  ghcr.io/prometheus/snmp-generator:latest \
  generate

The scrape configuration follows the same multi-target pattern as Blackbox Exporter. The key metrics from the if_mib module (standard interface stats available on virtually all SNMP-capable devices):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

# Inbound traffic per interface (bytes/sec)
rate(ifHCInOctets{job="snmp_ubiquiti"}[5m]) * 8  # multiply by 8 for bits/sec

# Outbound traffic per interface
rate(ifHCOutOctets{job="snmp_ubiquiti"}[5m]) * 8

# Interface operational status (1=up, 2=down, 3=testing)
ifOperStatus{job="snmp_ubiquiti"}

# UPS battery charge percentage
upsAdvBatteryCapacity{job="snmp_apc"}

# UPS estimated runtime remaining (seconds)
upsAdvBatteryRunTimeRemaining{job="snmp_apc"}

# UPS output load percentage
upsAdvOutputLoad{job="snmp_apc"}

# UPS input voltage
upsAdvInputLineVoltage{job="snmp_apc"}

The UPS metrics deserve special attention. An alert on upsAdvBatteryCapacity < 50 means you know when the battery is degrading before a power event reveals it by failing to provide runtime. An alert on upsAdvBatteryRunTimeRemaining < 300 (less than 5 minutes) during an actual power outage gives you time to initiate a clean shutdown.

Alertmanager

Prometheus evaluates alerting rules on each evaluation interval and sends any firing alerts to Alertmanager. Alertmanager’s job is everything that happens next: deduplication (the same alert firing from multiple sources shouldn’t generate multiple pages), grouping (ten services going down at once is one incident, not ten pages), routing (critical disk alerts go to one receiver, informational alerts go to another), silencing (mute everything for the next two hours while you’re doing planned maintenance), and inhibition (don’t send CPU high alerts for a host that’s already been paged as unreachable).

The alerting rules file:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141

# config/prometheus/rules/alerts.yml

groups:
  - name: host_alerts
    rules:

      # Any scrape target has been unreachable for 2 minutes
      - alert: TargetDown
        expr: up == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Target {{ $labels.instance }} is down"
          description: "{{ $labels.job }}/{{ $labels.instance }} has been down for more than 2 minutes."

      # CPU usage above 85% for 10 minutes
      - alert: HighCPUUsage
        expr: instance:node_cpu_utilization:rate5m > 85
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          description: "CPU utilization is {{ $value | humanize }}% on {{ $labels.instance }}."

      # Less than 10% disk space remaining
      - alert: LowDiskSpace
        expr: |
          (
            node_filesystem_avail_bytes{fstype!~"tmpfs|fuse.lxcfs|squashfs|vfat"} /
            node_filesystem_size_bytes{fstype!~"tmpfs|fuse.lxcfs|squashfs|vfat"}
          ) * 100 < 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Low disk space on {{ $labels.instance }}:{{ $labels.mountpoint }}"
          description: "Filesystem {{ $labels.mountpoint }} on {{ $labels.instance }} has {{ $value | humanize }}% free."

      # Critically low disk space — less than 5%
      - alert: CriticalDiskSpace
        expr: |
          (
            node_filesystem_avail_bytes{fstype!~"tmpfs|fuse.lxcfs|squashfs|vfat"} /
            node_filesystem_size_bytes{fstype!~"tmpfs|fuse.lxcfs|squashfs|vfat"}
          ) * 100 < 5
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Critical disk space on {{ $labels.instance }}:{{ $labels.mountpoint }}"
          description: "Filesystem {{ $labels.mountpoint }} on {{ $labels.instance }} has {{ $value | humanize }}% free. Immediate action required."

      # Memory usage above 90% for 5 minutes
      - alert: HighMemoryUsage
        expr: instance:node_memory_utilization:ratio > 90
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High memory usage on {{ $labels.instance }}"
          description: "Memory utilization is {{ $value | humanize }}% on {{ $labels.instance }}."

      # System load above 2x CPU count for 15 minutes
      - alert: HighSystemLoad
        expr: |
          node_load15 /
          count by(instance) (count by(instance, cpu) (node_cpu_seconds_total)) > 2
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "High system load on {{ $labels.instance }}"
          description: "15-minute load average is {{ $value | humanize }}x CPU count on {{ $labels.instance }}."

  - name: service_alerts
    rules:

      # HTTP probe failed
      - alert: ServiceDown
        expr: probe_success{job=~"blackbox_.*"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Service {{ $labels.instance }} is unreachable"
          description: "Blackbox probe to {{ $labels.instance }} has been failing for 2 minutes."

      # SSL certificate expires in less than 14 days
      - alert: SSLCertExpiringSoon
        expr: (probe_ssl_earliest_cert_expiry - time()) / 86400 < 14
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "SSL certificate expiring soon for {{ $labels.instance }}"
          description: "Certificate for {{ $labels.instance }} expires in {{ $value | humanize }} days."

      # SSL certificate expires in less than 3 days
      - alert: SSLCertExpiringCritical
        expr: (probe_ssl_earliest_cert_expiry - time()) / 86400 < 3
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "SSL certificate expiring in less than 3 days for {{ $labels.instance }}"
          description: "Certificate for {{ $labels.instance }} expires in {{ $value | humanize }} days."

      # Container restart loop — restart count increased by more than 3 in 10 minutes
      - alert: ContainerRestartLoop
        expr: increase(container_restart_count{name!=""}[10m]) > 3
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Container {{ $labels.name }} is restart-looping"
          description: "Container {{ $labels.name }} has restarted {{ $value | humanize }} times in the last 10 minutes."

  - name: ups_alerts
    rules:

      # UPS battery below 50%
      - alert: UPSBatteryLow
        expr: upsAdvBatteryCapacity < 50
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "UPS battery low on {{ $labels.instance }}"
          description: "UPS {{ $labels.instance }} battery is at {{ $value }}%. Battery may need replacement."

      # UPS on battery (power outage)
      - alert: UPSOnBattery
        expr: upsBasicOutputStatus == 2  # 2 = onBattery
        for: 30s
        labels:
          severity: critical
        annotations:
          summary: "UPS {{ $labels.instance }} is running on battery"
          description: "Mains power lost. UPS {{ $labels.instance }} is running on battery with {{ $value }} minutes remaining."

The complete alertmanager.yml with Slack and email receivers:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106

# config/alertmanager/alertmanager.yml

global:
  resolve_timeout: 5m
  smtp_smarthost: 'smtp.gmail.com:587'
  smtp_from: 'alerts@yourdomain.com'
  smtp_auth_username: 'alerts@yourdomain.com'
  smtp_auth_password: 'your-app-password'
  smtp_require_tls: true
  slack_api_url: 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK'

templates:
  - '/etc/alertmanager/templates/*.tmpl'

# The routing tree.
# Alertmanager walks the tree top-to-bottom, and routes the alert
# to the first matching receiver.
route:
  receiver: 'slack-homelab'     # default receiver
  group_by: ['alertname', 'instance', 'severity']
  group_wait: 30s               # wait for more alerts before sending the first notification
  group_interval: 5m            # wait between notifications for the same group
  repeat_interval: 4h           # re-notify if alert is still firing after 4 hours

  routes:
    # Critical alerts go to both Slack and email
    - match:
        severity: critical
      receiver: slack-and-email
      group_wait: 10s
      repeat_interval: 1h

    # UPS alerts are critical — page immediately and frequently
    - match:
        alertname: UPSOnBattery
      receiver: slack-and-email
      group_wait: 0s
      repeat_interval: 15m

    # Warning-level alerts Slack only, less frequent
    - match:
        severity: warning
      receiver: slack-homelab
      repeat_interval: 6h

# Inhibition rules: suppress symptoms when you've already paged the cause.
inhibit_rules:
  # If a host is completely unreachable (TargetDown), suppress
  # all other alerts from that same instance.
  - source_match:
      alertname: TargetDown
    target_match_re:
      instance: .*
    equal: ['instance']

  # If ServiceDown fires, suppress SSLCertExpiringSoon for the same instance
  # (can't check the cert if the service is down)
  - source_match:
      alertname: ServiceDown
    target_match:
      alertname: SSLCertExpiringSoon
    equal: ['instance']

receivers:
  - name: 'slack-homelab'
    slack_configs:
      - channel: '#homelab-alerts'
        send_resolved: true
        icon_url: 'https://avatars3.githubusercontent.com/u/3380462'
        title: '{{ template "slack.default.title" . }}'
        text: >-
          {{ range .Alerts }}
          *Alert:* {{ .Annotations.summary }}
          *Details:* {{ .Annotations.description }}
          *Severity:* {{ .Labels.severity }}
          *Instance:* {{ .Labels.instance }}
          {{ end }}
        actions:
          - type: button
            text: 'Prometheus'
            url: 'http://prometheus.yourdomain.local:9090'
          - type: button
            text: 'Silence'
            url: '{{ template "slack.default.alertmanagerURL" . }}'

  - name: 'slack-and-email'
    slack_configs:
      - channel: '#homelab-critical'
        send_resolved: true
        title: 'CRITICAL: {{ template "slack.default.title" . }}'
        text: >-
          {{ range .Alerts }}
          *Alert:* {{ .Annotations.summary }}
          *Details:* {{ .Annotations.description }}
          {{ end }}
    email_configs:
      - to: 'you@youremail.com'
        send_resolved: true
        headers:
          Subject: '[HOMELAB ALERT] {{ .GroupLabels.alertname }}'
        html: |
          <h2>{{ .GroupLabels.alertname }}</h2>
          {{ range .Alerts }}
          <p><strong>{{ .Annotations.summary }}</strong><br>
          {{ .Annotations.description }}</p>
          {{ end }}

Inhibition rules are the feature most people discover after getting paged fifteen times for the same underlying failure. The source_match defines the alert that suppresses, target_match defines what gets suppressed, and equal defines which label values must match between source and target. When TargetDown fires for instance="homeserver", all other alerts with instance="homeserver" are inhibited. You get one page for the root cause, not a cascade of symptom alerts.

Silences are temporary mutes that you create through the Alertmanager UI or API. During planned maintenance — OS updates, hardware swaps, network reconfiguration — create a silence for the affected instance for the duration of the maintenance window:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# Create a silence via API
curl -X POST http://localhost:9093/api/v2/silences \
  -H 'Content-Type: application/json' \
  -d '{
    "matchers": [
      {"name": "instance", "value": "homeserver", "isRegex": false}
    ],
    "startsAt": "2026-05-29T18:00:00.000Z",
    "endsAt": "2026-05-29T22:00:00.000Z",
    "comment": "OS upgrade maintenance window",
    "createdBy": "admin"
  }'

Grafana Configuration and Provisioning

The first mistake most people make with Grafana is configuring everything through the UI and then losing it all when they recreate the container. Grafana’s persistent volume stores everything — if you’ve mounted a volume properly, your data survives. But you want something better than a binary blob in a Docker volume: you want your configuration in version-controlled YAML files.

Grafana’s provisioning system lets you define datasources, dashboards, alert rules, and notification policies as YAML files that Grafana reads at startup. Mount these files into /etc/grafana/provisioning/ and Grafana configures itself automatically.

Datasource provisioning:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# config/grafana/provisioning/datasources/prometheus.yaml

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    jsonData:
      httpMethod: POST
      prometheusType: Prometheus
      prometheusVersion: 3.11.3
      queryTimeout: 30s
      timeInterval: 15s        # should match Prometheus scrape_interval
    editable: false            # prevent UI edits from overriding provisioning

Dashboard provisioning works in two parts: a folder configuration file that defines where dashboards live, and the dashboard JSON files themselves.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

# config/grafana/provisioning/dashboards/homelab.yaml

apiVersion: 1

providers:
  - name: 'homelab'
    orgId: 1
    folder: 'Homelab'
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30   # check for updated JSON files every 30 seconds
    allowUiUpdates: false       # UI changes won't stick — edit the JSON file instead
    options:
      path: /etc/grafana/provisioning/dashboards/json
      foldersFromFilesStructure: true

Place your dashboard JSON files in config/grafana/provisioning/dashboards/json/. To use a community dashboard, download the JSON from grafana.com or use the ID import:

Node Exporter Full        : https://grafana.com/grafana/dashboards/1860
cAdvisor Exporter         : https://grafana.com/grafana/dashboards/14282
Prometheus Blackbox Exporter : https://grafana.com/grafana/dashboards/7587

These IDs are stable and remain valid. Download the JSON, save it to your dashboards directory, and Grafana will load it automatically.

Dashboard variables make dashboards dynamic. Instead of hardcoding instance="homeserver" in every query, define a $instance variable that populates from a Prometheus label query. Edit the dashboard settings (gear icon) → Variables → New:

Type: Query
Name: instance
Label: Instance
Data source: Prometheus
Query: label_values(node_uname_info, instance)
Refresh: On time range change

Now every panel in the dashboard can use {instance="$instance"} in its PromQL, and the dropdown at the top of the dashboard lets you switch between hosts without modifying any queries.

Grafana Unified Alerting vs Prometheus alerting rules: Grafana 8+ introduced its own alerting system that evaluates rules against any datasource. Prometheus alerting rules (evaluated by Prometheus itself) are better for infrastructure-level alerts — they fire even when Grafana is down, they’re colocated with the rest of your Prometheus config, and they’re easier to test with promtool. Grafana alerts are better for dashboard-level alerts that involve multi-datasource queries or when you want alert annotations to appear directly on panel graphs. For a homelab, keep infrastructure alerts in Prometheus and use Grafana alerts sparingly, if at all.

Exporter Reference

Dashboard-as-Code with Grafonnet

Once you have more than a handful of dashboards, the pain of managing them as exported JSON becomes apparent. JSON is not diffable in any meaningful sense — a single panel change produces a multi-hundred-line diff. You can’t write a function that generates a row of panels, one per host. You can’t abstract a common panel configuration and reuse it across dashboards. Grafonnet addresses this by letting you define dashboards in Jsonnet — a data templating language that is a strict superset of JSON with functions, imports, and arithmetic.

Grafonnet is generated from the Grafana Foundation SDK, which means it tracks the current Grafana schema. The original grafonnet-lib is deprecated; use the new grafonnet library:

1
2
3
4
5
6
7
8

# Install jsonnet and jb (jsonnet-bundler)
go install github.com/google/go-jsonnet/cmd/jsonnet@latest
go install github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb@latest

# Initialize a project
mkdir my-dashboards && cd my-dashboards
jb init
jb install github.com/grafana/grafonnet/gen/grafonnet-latest@main

A minimal dashboard with a time series panel and a stat panel:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

// dashboards/node-overview.jsonnet
local g = import 'github.com/grafana/grafonnet/gen/grafonnet-latest/main.libsonnet';

local cpuPanel =
  g.panel.timeSeries.new('CPU Utilization')
  + g.panel.timeSeries.queryOptions.withTargets([
    g.query.prometheus.new(
      'Prometheus',
      'instance:node_cpu_utilization:rate5m{instance="$instance"}'
    )
    + g.query.prometheus.withLegendFormat('CPU %'),
  ])
  + g.panel.timeSeries.standardOptions.withUnit('percent')
  + g.panel.timeSeries.gridPos.withW(12)
  + g.panel.timeSeries.gridPos.withH(8);

local memPanel =
  g.panel.stat.new('Memory Available')
  + g.panel.stat.queryOptions.withTargets([
    g.query.prometheus.new(
      'Prometheus',
      'node_memory_MemAvailable_bytes{instance="$instance"} / node_memory_MemTotal_bytes{instance="$instance"} * 100'
    ),
  ])
  + g.panel.stat.standardOptions.withUnit('percent')
  + g.panel.stat.gridPos.withW(4)
  + g.panel.stat.gridPos.withH(4);

g.dashboard.new('Node Overview')
+ g.dashboard.withUid('node-overview-v1')
+ g.dashboard.withDescription('Host-level metrics from node_exporter')
+ g.dashboard.withRefresh('30s')
+ g.dashboard.withPanels([cpuPanel, memPanel])
+ g.dashboard.withTemplating([
  g.dashboard.variable.query.new('instance')
  + g.dashboard.variable.query.queryTypes.withLabelValues(
    'instance',
    'node_uname_info'
  ),
])

Build the dashboard JSON and place it in your Grafana provisioning directory:

1
2

jsonnet -J vendor dashboards/node-overview.jsonnet > \
  config/grafana/provisioning/dashboards/json/node-overview.json

Alternatives to Grafonnet:

• grafana-foundation-sdk: The official Grafana SDK, available in multiple languages (Go, TypeScript, Python). Generates the same output as Grafonnet but in your language of choice. More verbose but no Jsonnet dependency.
• Grizzly (grr): A CLI tool for pushing/pulling Grafana resources (dashboards, datasources, alert rules) as YAML or Jsonnet. Good for keeping dashboards in sync across environments.
• Terraform + Grafana provider: Works well if you’re already managing infrastructure with Terraform. The grafana_dashboard resource takes a JSON template. Less elegant for parameterization than Grafonnet but integrates naturally with IaC workflows.

Honestly, for a solo homelab, Grafonnet is overkill unless you enjoy it or have more than ~20 dashboards. The provisioning workflow — download community dashboard JSON, put it in the directory, done — covers most needs. Grafonnet pays off at the team level, when multiple people are modifying dashboards and code review is part of the workflow.

Long-Term Storage

Prometheus’s local TSDB has a hard limit: it’s designed for short-to-medium-term data on a single machine. The default retention is 15 days. You can extend this to 30, 60, or 90 days with --storage.tsdb.retention.time, but you’re still on a single node with no replication, and memory usage grows proportionally with the number of active time series multiplied by the retention window. At some point you want years of data — for capacity planning, trend analysis, or just answering “was my disk I/O this bad last winter?”

The four realistic options are local Prometheus with extended retention, Thanos, Grafana Mimir, and VictoriaMetrics.

Thanos

Thanos wraps existing Prometheus instances. You don’t replace Prometheus; you add Thanos components around it. The sidecar runs alongside each Prometheus instance, reads the local TSDB blocks that Prometheus produces every two hours, and uploads them to object storage (S3, GCS, or MinIO for self-hosting). The Store Gateway exposes those uploaded blocks via Thanos’s gRPC store API. The Querier aggregates queries across multiple Prometheus instances and the Store Gateway, providing a unified query interface that spans years of data.

                    THANOS ARCHITECTURE

  ┌─────────────────────────────────────────────────────────┐
  │                   Grafana / Query Frontend               │
  └───────────────────────────┬─────────────────────────────┘
                              │ PromQL queries
                    ┌─────────▼──────────┐
                    │   Thanos Querier   │
                    │  (global view,     │
                    │  deduplication)    │
                    └──┬────────────┬────┘
                       │            │
          ┌────────────▼──┐  ┌──────▼─────────────┐
          │ Thanos Sidecar│  │ Thanos Store Gateway│
          │ (recent 2h)   │  │ (historical blocks) │
          └────────┬──────┘  └──────────┬──────────┘
                   │                    │ reads blocks
          ┌────────▼──────┐  ┌──────────▼──────────┐
          │  Prometheus   │  │   Object Storage     │
          │  (scraping,   │  │   (MinIO / S3)       │
          │  local TSDB)  │  │                      │
          └───────────────┘  │  [2h blocks, compacted│
                             │   1d, 2h downsampled,│
                             │   5m downsampled]    │
                   ┌─────────┴──────────┐
                   │  Thanos Compactor  │
                   │  (offline, runs    │
                   │   periodically)    │
                   └────────────────────┘

The Compactor runs periodically (not continuously) and does two things: it merges small 2-hour blocks into larger 1-day blocks (compaction), and it creates downsampled versions at 5-minute and 1-hour resolution (downsampling). Downsampling means a 2-year query doesn’t have to process millions of individual 15-second samples — Grafana uses the appropriate resolution for the time range.

The Thanos Compose setup is more complex but not overwhelming:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

  thanos-sidecar:
    image: quay.io/thanos/thanos:v0.37.2
    container_name: thanos-sidecar
    command:
      - sidecar
      - --tsdb.path=/prometheus
      - --prometheus.url=http://prometheus:9090
      - --objstore.config-file=/etc/thanos/objstore.yml
      - --http-address=0.0.0.0:10902
      - --grpc-address=0.0.0.0:10901
    volumes:
      - prometheus_data:/prometheus:ro
      - ./config/thanos:/etc/thanos:ro
    networks:
      - monitoring

  thanos-store:
    image: quay.io/thanos/thanos:v0.37.2
    container_name: thanos-store
    command:
      - store
      - --objstore.config-file=/etc/thanos/objstore.yml
      - --http-address=0.0.0.0:10904
      - --grpc-address=0.0.0.0:10903
    volumes:
      - ./config/thanos:/etc/thanos:ro
      - thanos_store_cache:/thanos-store
    networks:
      - monitoring

  thanos-query:
    image: quay.io/thanos/thanos:v0.37.2
    container_name: thanos-query
    command:
      - query
      - --http-address=0.0.0.0:9091
      - --endpoint=thanos-sidecar:10901
      - --endpoint=thanos-store:10903
      - --query.replica-label=replica
    ports:
      - "9091:9091"
    networks:
      - monitoring

The object store configuration:

1
2
3
4
5
6
7
8

# config/thanos/objstore.yml
type: S3
config:
  bucket: thanos-homelab
  endpoint: minio:9000
  access_key: minioadmin
  secret_key: minioadmin-secret
  insecure: true  # for local MinIO without TLS

Update Grafana’s datasource to point at the Thanos Querier (http://thanos-query:9091) instead of Prometheus directly, and you get a unified query interface with full historical access.

Grafana Mimir

Mimir is Grafana Labs’ horizontally scalable, multi-tenant, Prometheus-compatible TSDB. It’s what powers Grafana Cloud’s metrics backend. In monolithic mode (single binary, single process), it runs comfortably on a single machine and is simple enough for a homelab that wants years of retention without the sidecar complexity of Thanos.

Configure Prometheus to remote-write to Mimir:

1
2
3
4
5
6

# In prometheus.yml
remote_write:
  - url: http://mimir:9009/api/v1/push
    queue_config:
      max_samples_per_send: 10000
      capacity: 50000

Mimir stores blocks in object storage (MinIO works). You don’t need a separate sidecar, store gateway, or querier — Mimir handles all of this internally. The tradeoff is that you’re duplicating your storage: Prometheus keeps its local TSDB for fast recent queries, and Mimir has its own copy in object storage for long-term retention. With Thanos, Prometheus is still the source of truth and Thanos uploads its existing blocks; with Mimir remote_write, you’re writing the data twice.

VictoriaMetrics

VictoriaMetrics is the option that most people overlook and many people end up wishing they’d started with. It is a drop-in replacement for Prometheus: it accepts Prometheus remote_write, speaks PromQL (with extensions), and Grafana connects to it as a Prometheus datasource. It stores data roughly 7-10x more compactly than Prometheus and queries faster.

Single-node VictoriaMetrics is genuinely simple:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

  victoriametrics:
    image: victoriametrics/victoria-metrics:v1.115.0
    container_name: victoriametrics
    restart: unless-stopped
    command:
      - '-storageDataPath=/storage'
      - '-retentionPeriod=2y'         # two years of retention
      - '-httpListenAddr=:8428'
    volumes:
      - vm_data:/storage
    ports:
      - "8428:8428"
    networks:
      - monitoring

Point Prometheus remote_write at http://victoriametrics:8428/api/v1/write and point Grafana at http://victoriametrics:8428 as a Prometheus datasource. You get two years of metrics with no object storage, no sidecar, no compactor, and no Querier — just one container. The cluster version scales horizontally for multi-node deployments, but for a homelab that’s unnecessary.

Choosing Your Long-Term Storage

The honest recommendation: if you have a single Prometheus instance and you want years of retention, start with VictoriaMetrics. If you later add a second Prometheus instance on a different node and want a global view across both, consider Thanos at that point. Mimir makes sense if you’re running something close to production scale, need strict multi-tenancy, or want exactly the same stack as Grafana Cloud.

Retention, Cardinality, and the Costs You Don’t See Coming

A few pitfalls that will bite you if you’re not expecting them.

Cardinality explosion: Prometheus memory usage is dominated by the number of active time series, not the number of samples. Each unique label combination is a separate time series. If you accidentally scrape a metric with a high-cardinality label — request path, user ID, container image digest, Kubernetes pod UID — you can go from 50,000 time series to 5,000,000 overnight. Common culprits are cAdvisor (container IDs before they’re cleaned up), application exporters that include request parameters in labels, and anything that uses UUIDs as label values.

Use topk(10, count by(__name__)({__name__=~".+"})) to find your highest-cardinality metrics, and use metric relabeling (metric_relabel_configs in the scrape config) to drop or relabel them before they reach the TSDB.

Prometheus storage sizing: The rule of thumb is roughly 1-2 bytes per sample (after TSDB compression). At 15s scrape interval, a single time series generates 4 samples per minute, 240 per hour, 5,760 per day. With 100,000 time series at 1.5 bytes/sample over 30 days, that’s roughly 26 GB. Budget your storage accordingly, and set both --storage.tsdb.retention.time and --storage.tsdb.retention.size so you have two bounds.

Query performance on long ranges: PromQL over long time ranges (weeks, months) on a local TSDB is slow because Prometheus must scan every sample in the range. Recording rules that aggregate frequently-queried expressions into pre-computed time series are the primary mitigation. For long-term historical queries, Thanos’s downsampling or VictoriaMetrics’s efficient storage are the architectural answers.

Scrape interval selection: 15 seconds is appropriate for most homelab metrics. Some metrics (disk space, UPS battery, NTP offset) change slowly enough that 60-second intervals are fine. For per-container metrics from cAdvisor, 30 seconds is usually enough resolution. Reducing scrape intervals increases load on both the scraped targets and the Prometheus TSDB. Don’t go below 5 seconds without a specific reason.

Directory Structure

The complete configuration layout for this stack:

.
├── docker-compose.yml
├── .env                    (GRAFANA_ADMIN_PASSWORD=...)
└── config/
    ├── prometheus/
    │   ├── prometheus.yml
    │   └── rules/
    │       ├── alerts.yml
    │       └── recording.yml
    ├── alertmanager/
    │   └── alertmanager.yml
    ├── blackbox/
    │   └── blackbox.yml
    ├── snmp/
    │   └── snmp.yml        (generated, not handwritten)
    ├── node-exporter/
    │   └── textfile/       (*.prom files from custom scripts)
    ├── thanos/             (optional)
    │   └── objstore.yml
    └── grafana/
        └── provisioning/
            ├── datasources/
            │   └── prometheus.yaml
            └── dashboards/
                ├── homelab.yaml
                └── json/
                    ├── node-exporter-full.json
                    ├── cadvisor.json
                    └── blackbox.json

Everything in config/ is version-controlled. The only file that stays outside git is .env. Running git clone on a new machine and docker compose up -d gives you a fully operational observability stack in minutes.

Putting It Together

The stack described in this guide covers every layer of a typical homelab: the bare metal host via Node Exporter, containers via cAdvisor, services via Blackbox Exporter probes, and network gear via SNMP Exporter. Alertmanager routes notifications with intelligent grouping and inhibition so you get paged for causes, not symptoms. Grafana displays it all with provisioned dashboards that survive container recreation. Long-term storage via VictoriaMetrics or Thanos extends retention from weeks to years.

The operational overhead after initial setup is low. The main recurring tasks are: adjusting alert thresholds as you learn what’s normal in your environment (the first few weeks will produce alerts that turn out to be false positives), rotating Grafana credentials, and occasionally updating image versions. promtool check config and amtool check-config are your friends for catching configuration errors before they become outages.

Start with Node Exporter, cAdvisor, and the basic alerting rules. Get paged for a real disk-full event or a service outage and you’ll immediately understand why this setup is worth maintaining. Add Blackbox Exporter probing once the basics are stable. Add SNMP Exporter when you get tired of logging into your router’s web UI to see whether the WAN link is saturated. Add long-term storage when you start wanting to answer questions like “how much has my NAS usage grown over the past year?”

The homelab doesn’t have an SLA or on-call rotation. This stack is the next best thing.