Monitoring and Observability: From the Golden Signals to a Complete Self-Hosted Stack

March 13, 2026 32 min read DevOps

monitoringobservabilityprometheusgrafanalokidockerhomelabdevopssrealerting

Contents

Monitoring isn’t about collecting metrics. It’s about understanding your systems well enough to know when something is wrong and why. You’ve set up your VPS, configured your Docker containers, reverse proxy, and TLS. Services are running. Then one day something breaks — and you have no idea when it started, what caused it, or whether it’s actually fixed. You’re flying blind.

This guide does two things. First it covers what to measure and why: the frameworks (RED, USE, the Four Golden Signals), percentiles, SLOs, and alerting that doesn’t burn you out. Then it builds a complete, self-hosted observability stack from scratch using the open-source gold standard — Prometheus for metrics, Loki for logs, Grafana for visualization, and Alertmanager for notifications. Everything runs in Docker, works equally well on a single VPS or a homelab cluster, and costs nothing beyond your server time.

Two Frameworks: RED and USE

Before you wire up a single exporter, decide what’s worth measuring. Two complementary frameworks cover almost everything.

RED Method (for services)

For request-driven services (APIs, web servers), measure:

Rate: requests per second
Errors: failed requests per second
Duration: time per request (latency)

1
2
3
4
5
6
7
8


# Rate
sum(rate(http_requests_total[5m]))

# Errors
sum(rate(http_requests_total{status=~"5.."}[5m]))

# Duration (95th percentile)
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

USE Method (for resources)

For infrastructure components (CPU, memory, disk, network), measure:

Utilization: percentage of the resource used
Saturation: work queued, waiting for the resource
Errors: error events

Resource	Utilization	Saturation	Errors
CPU	CPU usage %	Load average, run queue	-
Memory	Used memory %	Swap usage, OOM events	ECC errors
Disk	Disk usage %	IO wait, queue depth	Read/write errors
Network	Bandwidth %	Dropped packets, retransmits	Interface errors

The Four Golden Signals

Google’s SRE book distills all of this into four signals:

Latency — how long requests take
Traffic — how much demand exists
Errors — rate of failed requests
Saturation — how “full” the service is

If you can only monitor four things, monitor these.

The Three Pillars of Observability

Frameworks tell you what to measure. Observability is built from three types of telemetry data that answer different questions.

Metrics

Numerical measurements sampled over time. CPU usage at 14:03. HTTP requests per second. Memory used. Queue depth. Metrics are cheap to store and fast to query — great for answering “is something wrong?” and for alerting. They’re weak at explaining why.

Logs

Timestamped records of events. Application errors. HTTP access logs. Authentication attempts. Logs are verbose and expensive at scale, but essential for understanding exactly what happened. They’re the primary tool for debugging.

Traces

Records of a single request’s path through distributed services — showing how long each component took and where latency was introduced. Traces matter most in microservice architectures where one user action fans out across many services. For a typical VPS or homelab, traces are optional but increasingly relevant as your stack grows.

Grafana Labs now identifies a fourth signal: Profiles — continuous performance profiling (CPU flame graphs, memory allocation traces) that gives you the “why” behind metric anomalies. For most self-hosted setups, metrics + logs + (optionally) traces gets you 90% of the value.

What Else Is Worth Measuring

Technical metrics don’t tell the whole story. Two categories are easy to forget:

Business metrics — user signups per hour, orders completed per minute, payment failures, search queries returning no results. These often matter more than CPU usage.

Dependency health — your service depends on others, so watch them too:

1
2
3
4
5
6
7
8


# Database connection pool
db_pool_connections_active
db_pool_connections_idle
db_pool_connections_waiting

# External API
external_api_request_duration_seconds
external_api_errors_total

The Stack We’re Building

┌────────────────── Data Collection ──────────────────┐
│                                                     │
│  Node Exporter        cAdvisor          Promtail    │
│  (host metrics)   (container metrics) (log shipping)│
│       │                  │                  │       │
└───────┼──────────────────┼──────────────────┼───────┘
        │                  │                  │
        ▼                  ▼                  ▼
   ┌─────────┐        ┌─────────┐        ┌──────┐
   │Prometheus│        │Prometheus│        │ Loki │
   │ (metrics)│        │ (metrics)│        │(logs)│
   └────┬────┘        └────┬────┘        └──┬───┘
        │                  │               │
        └──────────┬────────┘               │
                   ▼                        │
             ┌──────────┐                  │
             │ Grafana  │◄─────────────────┘
             │(dashboard│
             │& alerts) │
             └────┬─────┘
                  │
                  ▼
           ┌──────────────┐
           │ Alertmanager │
           │(Slack, email,│
           │  PagerDuty)  │
           └──────────────┘

Component Reference

Component	What it does	Port
Prometheus	Scrapes and stores metrics in a time-series database	9090
Grafana	Visualization, dashboards, alerting UI	3000
Loki	Log aggregation and storage (indexes labels, not content)	3100
Grafana Alloy	Modern log and metrics agent (replaces Promtail)	12345
Promtail	Legacy log shipping agent (simpler, still widely used)	—
Node Exporter	Exports host hardware/OS metrics (CPU, memory, disk, network)	9100
cAdvisor	Exports per-container resource metrics	8080
Alertmanager	Routes Prometheus alerts to notification channels	9093

Percentiles, SLOs, and Error Budgets

Averages Lie — Use Percentiles

Average latency: 50ms
p50 latency:     20ms
p95 latency:    100ms
p99 latency:   2000ms

The average looks fine, but 1% of users wait two seconds. Which percentiles to track:

p50 — the typical experience
p95 — bad but common
p99 — the worst common experience
p99.9 — outliers (only worth it at scale)

SLOs and Error Budgets

A Service Level Objective sets the target, and the error budget is what’s left over:

SLO:          99.9% of requests complete in < 500ms
Error budget: 0.1% = ~43 minutes of failure per month

Alert when you’re burning the budget too fast, not on every individual blip:

1
2
3
4
5
6
7
8


# Fire if the current hourly error rate would exhaust the monthly budget quickly
(
  1 - (
    sum(rate(http_requests_total{status="200"}[1h]))
    /
    sum(rate(http_requests_total[1h]))
  )
) > 0.001 * 24   # More than 24x the hourly budget

Error budgets turn reliability into a number you can spend deliberately — ship features while there’s budget, slow down and harden when you’re burning it.

Full Stack Docker Compose

This single docker-compose.yml deploys the complete monitoring stack. It’s designed to live alongside your existing services on the same host — VPS or homelab.

Create a monitoring directory and build the file structure:

monitoring/
├── docker-compose.yml
├── prometheus/
│   ├── prometheus.yml
│   └── rules/
│       └── alerts.yml
├── loki/
│   └── loki-config.yml
├── promtail/
│   └── promtail-config.yml
├── alertmanager/
│   └── alertmanager.yml
└── grafana/
    └── provisioning/
        ├── datasources/
        │   └── datasources.yml
        └── dashboards/
            └── dashboards.yml

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134


# monitoring/docker-compose.yml
version: "3.9"

networks:
  monitoring:
    driver: bridge
  # If your app services use an external network, add it here:
  # app_network:
  #   external: true

volumes:
  prometheus_data:
  grafana_data:
  loki_data:

services:

  # ─── Metrics: Prometheus ────────────────────────────────────
  prometheus:
    image: prom/prometheus:v2.51.0
    container_name: prometheus
    restart: unless-stopped
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--storage.tsdb.retention.time=30d"   # Keep 30 days of metrics
      - "--web.enable-lifecycle"               # Allow config reload via HTTP
      - "--web.enable-admin-api"
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./prometheus/rules:/etc/prometheus/rules:ro
      - prometheus_data:/prometheus
    ports:
      - "127.0.0.1:9090:9090"  # Bind to localhost — expose via reverse proxy
    networks:
      - monitoring

  # ─── Alerting: Alertmanager ──────────────────────────────────
  alertmanager:
    image: prom/alertmanager:v0.27.0
    container_name: alertmanager
    restart: unless-stopped
    command:
      - "--config.file=/etc/alertmanager/alertmanager.yml"
      - "--storage.path=/alertmanager"
    volumes:
      - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
    ports:
      - "127.0.0.1:9093:9093"
    networks:
      - monitoring

  # ─── Visualization: Grafana ──────────────────────────────────
  grafana:
    image: grafana/grafana:10.4.0
    container_name: grafana
    restart: unless-stopped
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD:-changeme}
      - GF_USERS_ALLOW_SIGN_UP=false
      - GF_SERVER_ROOT_URL=https://grafana.yourdomain.com  # Change this
      - GF_SMTP_ENABLED=false  # Configure if you want email alerts
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
    ports:
      - "127.0.0.1:3000:3000"  # Expose via reverse proxy (Traefik or Nginx)
    networks:
      - monitoring
    depends_on:
      - prometheus
      - loki

  # ─── Logs: Loki ──────────────────────────────────────────────
  loki:
    image: grafana/loki:3.0.0
    container_name: loki
    restart: unless-stopped
    command: -config.file=/etc/loki/loki-config.yml
    volumes:
      - ./loki/loki-config.yml:/etc/loki/loki-config.yml:ro
      - loki_data:/loki
    ports:
      - "127.0.0.1:3100:3100"
    networks:
      - monitoring

  # ─── Log Shipping: Promtail ──────────────────────────────────
  promtail:
    image: grafana/promtail:3.0.0
    container_name: promtail
    restart: unless-stopped
    volumes:
      - ./promtail/promtail-config.yml:/etc/promtail/config.yml:ro
      - /var/log:/var/log:ro              # Host system logs
      - /var/lib/docker/containers:/var/lib/docker/containers:ro  # Docker logs
      - /run/docker.sock:/run/docker.sock:ro  # Docker socket (for label discovery)
    command: -config.file=/etc/promtail/config.yml
    networks:
      - monitoring
    depends_on:
      - loki

  # ─── Host Metrics: Node Exporter ────────────────────────────
  node-exporter:
    image: prom/node-exporter:v1.7.0
    container_name: node-exporter
    restart: unless-stopped
    command:
      - "--path.rootfs=/host"
      - "--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)"
    network_mode: host  # Must use host networking for accurate network metrics
    pid: host           # Must share host PID namespace for process metrics
    volumes:
      - /:/host:ro,rslave

  # ─── Container Metrics: cAdvisor ────────────────────────────
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.49.1
    container_name: cadvisor
    restart: unless-stopped
    privileged: true
    devices:
      - /dev/kmsg:/dev/kmsg
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker:/var/lib/docker:ro
      - /cgroup:/cgroup:ro
    ports:
      - "127.0.0.1:8080:8080"
    networks:
      - monitoring

Set the Grafana password before starting:

1
2


echo "GRAFANA_ADMIN_PASSWORD=your-strong-password-here" > .env
docker compose up -d

Prometheus Configuration

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81


# prometheus/prometheus.yml
global:
  scrape_interval: 15s       # How often to collect metrics
  evaluation_interval: 15s   # How often to evaluate alert rules
  scrape_timeout: 10s

  # Labels applied to all time series
  external_labels:
    monitor: 'homelab'
    environment: 'production'

# Alertmanager connection
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:9093

# Load alerting rules
rule_files:
  - "/etc/prometheus/rules/*.yml"

scrape_configs:
  # Prometheus itself
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  # Host machine metrics
  - job_name: "node-exporter"
    static_configs:
      - targets: ["node-exporter:9100"]
    # node-exporter uses host networking, so use the Docker bridge gateway IP
    # On most systems this is 172.17.0.1 — check with: ip route | grep docker
    # static_configs:
    #   - targets: ["172.17.0.1:9100"]

  # Docker container metrics
  - job_name: "cadvisor"
    static_configs:
      - targets: ["cadvisor:8080"]
    metric_relabel_configs:
      # Drop high-cardinality container metrics you don't need
      - source_labels: [__name__]
        regex: "container_(tasks_state|memory_failures_total)"
        action: drop

  # Alertmanager self-monitoring
  - job_name: "alertmanager"
    static_configs:
      - targets: ["alertmanager:9093"]

  # Loki self-monitoring
  - job_name: "loki"
    static_configs:
      - targets: ["loki:3100"]

  # ── Add your application targets below ──
  # Example: a web app exposing /metrics on port 8000
  # - job_name: "my-web-app"
  #   static_configs:
  #     - targets: ["my-web-app:8000"]
  #   metrics_path: /metrics

  # Example: Traefik reverse proxy metrics
  # - job_name: "traefik"
  #   static_configs:
  #     - targets: ["traefik:8082"]

  # Example: Docker daemon metrics (enable in /etc/docker/daemon.json first)
  # - job_name: "docker-daemon"
  #   static_configs:
  #     - targets: ["172.17.0.1:9323"]

  # ── Scrape additional hosts on your network ──
  # - job_name: "second-vps"
  #   static_configs:
  #     - targets: ["10.0.0.2:9100"]  # Node Exporter on another machine
  #       labels:
  #         instance: "second-vps"
  #         role: "database"

Instrumenting Your Own Application

Host and container metrics come for free from the exporters. To get the RED metrics that actually matter to users, instrument the application itself. Here’s the pattern with Python’s prometheus_client:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


from prometheus_client import Counter, Histogram

REQUEST_COUNT = Counter(
    'app_requests_total',
    'Total requests',
    ['method', 'endpoint', 'status']
)

REQUEST_LATENCY = Histogram(
    'app_request_duration_seconds',
    'Request latency',
    ['method', 'endpoint']
)

def handle_request(request):
    with REQUEST_LATENCY.labels(
        method=request.method,
        endpoint=request.path
    ).time():
        response = process_request(request)
        REQUEST_COUNT.labels(
            method=request.method,
            endpoint=request.path,
            status=response.status_code
        ).inc()
        return response

The Histogram automatically produces the _bucket series that histogram_quantile() needs for p95/p99 latency. Expose /metrics, point a scrape job at it, and your application’s RED metrics flow straight into Prometheus.

Adding Your Application Services to Prometheus

If your Docker services expose a Prometheus /metrics endpoint, add them to scrape_configs. Many popular tools do this natively:

Service	Default metrics port	Notes
Traefik	`:8082/metrics`	Enable via `--metrics.prometheus=true`
Nginx (with exporter)	`:9113/metrics`	Requires nginx-prometheus-exporter sidecar
PostgreSQL	`:9187/metrics`	Requires postgres_exporter
Redis	`:9121/metrics`	Requires redis_exporter
MySQL/MariaDB	`:9104/metrics`	Requires mysqld_exporter
Nextcloud	`:9205/metrics`	Requires nextcloud-exporter
Gitea	`:3000/metrics`	Built-in, enable in app.ini
Blackbox Exporter	`:9115/probe`	HTTP/HTTPS/DNS/TCP endpoint probing

For apps without native Prometheus support, the Blackbox Exporter lets you probe any HTTP endpoint and get availability and latency metrics without touching the application code.

Loki Configuration

Loki indexes only log labels (like app, host, level) — not the full log content. This makes it far cheaper to run than Elasticsearch while still making logs fully searchable via LogQL.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54


# loki/loki-config.yml
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  instance_addr: 127.0.0.1
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://alertmanager:9093

# Limits — tune these for your server's RAM
limits_config:
  retention_period: 30d
  ingestion_rate_mb: 16
  ingestion_burst_size_mb: 32
  max_query_series: 5000
  max_query_lookback: 90d

compactor:
  working_directory: /loki/boltdb-shipper-compactor
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150
  delete_request_store: filesystem

Promtail Configuration

Promtail is the log shipping agent that reads logs and sends them to Loki. It discovers Docker containers automatically and attaches meaningful labels.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77


# promtail/promtail-config.yml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:

  # ── Docker container logs (auto-discovery via Docker socket) ──
  - job_name: docker
    docker_sd_configs:
      - host: unix:///run/docker.sock
        refresh_interval: 5s
        filters:
          - name: status
            values: ["running"]
    relabel_configs:
      # Use container name as the instance label
      - source_labels: ["__meta_docker_container_name"]
        regex: "/(.*)"
        target_label: "container"
      # Use Docker Compose service name as the app label
      - source_labels: ["__meta_docker_container_label_com_docker_compose_service"]
        target_label: "app"
      # Use Docker Compose project name
      - source_labels: ["__meta_docker_container_label_com_docker_compose_project"]
        target_label: "project"
      # Keep the image name for context
      - source_labels: ["__meta_docker_container_image"]
        target_label: "image"

  # ── Host system logs (systemd journal) ──
  - job_name: systemd
    journal:
      max_age: 12h
      labels:
        job: systemd-journal
        host: ${HOSTNAME}
    relabel_configs:
      - source_labels: ["__journal__systemd_unit"]
        target_label: "unit"
      - source_labels: ["__journal_priority_keyword"]
        target_label: "level"

  # ── Nginx access logs (if running Nginx directly on host) ──
  - job_name: nginx
    static_configs:
      - targets: ["localhost"]
        labels:
          job: nginx
          __path__: /var/log/nginx/*.log
    pipeline_stages:
      - regex:
          expression: '^(?P<remote_addr>\S+) - (?P<remote_user>\S+) \[(?P<time_local>[^\]]+)\] "(?P<method>\S+) (?P<request_uri>\S+) \S+" (?P<status>\d+) (?P<body_bytes_sent>\d+)'
      - labels:
          method:
          status:
      - metrics:
          http_nginx_response_total:
            type: Counter
            description: "Total nginx responses by status code"
            source: status
            config:
              action: inc

  # ── Traefik access logs ──
  - job_name: traefik
    static_configs:
      - targets: ["localhost"]
        labels:
          job: traefik
          __path__: /var/log/traefik/access.log

Opt-in Logging with Docker Labels

For cleaner control over which containers get shipped to Loki, use an opt-in label approach. Update Promtail to only collect containers with a logging=promtail label:

1
2
3
4
5
6
7


# In promtail-config.yml, add a filter to docker_sd_configs:
docker_sd_configs:
  - host: unix:///run/docker.sock
    refresh_interval: 5s
    filters:
      - name: label
        values: ["logging=promtail"]

Then label containers you want monitored in your app’s docker-compose.yml:

1
2
3
4
5
6
7


# In your application's docker-compose.yml
services:
  my-app:
    image: my-app:latest
    labels:
      logging: "promtail"
      logging_jobname: "my-app"

Alertmanager Configuration

Alertmanager receives fired alerts from Prometheus and routes them to the right destination — Slack, email, PagerDuty, webhooks. It handles deduplication, grouping, and silencing.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81


# alertmanager/alertmanager.yml
global:
  # Default time to wait before sending a notification about resolved alerts
  resolve_timeout: 5m

  # Slack webhook (get from Slack Apps → Incoming Webhooks)
  slack_api_url: "https://hooks.slack.com/services/YOUR/WEBHOOK/URL"

route:
  # Default receiver for unmatched alerts
  receiver: "slack-notifications"

  # Group alerts that share these labels into a single notification
  group_by: ["alertname", "instance"]

  # Wait this long for more alerts before sending the first notification
  group_wait: 30s

  # Wait this long before sending a second notification for a group
  group_interval: 5m

  # Resend notification for an alert that is still firing
  repeat_interval: 4h

  routes:
    # Critical alerts: page immediately
    - match:
        severity: critical
      receiver: "slack-critical"
      group_wait: 10s
      repeat_interval: 1h

    # Warning alerts: batch to a lower-priority channel
    - match:
        severity: warning
      receiver: "slack-warnings"
      repeat_interval: 6h

    # DeadMansSwitch: always fires to confirm Alertmanager is alive
    # Route to null so it doesn't create noise
    - match:
        alertname: DeadMansSwitch
      receiver: "null"

receivers:
  - name: "null"

  - name: "slack-notifications"
    slack_configs:
      - channel: "#alerts"
        title: '{{ template "slack.title" . }}'
        text: '{{ template "slack.text" . }}'
        send_resolved: true
        color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}'

  - name: "slack-critical"
    slack_configs:
      - channel: "#alerts-critical"
        title: "CRITICAL: {{ .GroupLabels.alertname }}"
        text: |
          *Alert:* {{ .GroupLabels.alertname }}
          *Instance:* {{ .GroupLabels.instance }}
          *Description:* {{ range .Alerts }}{{ .Annotations.description }}{{ end }}
        send_resolved: true

  - name: "slack-warnings"
    slack_configs:
      - channel: "#alerts"
        title: "Warning: {{ .GroupLabels.alertname }}"
        text: "{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}"
        send_resolved: true

# Alternative: email notifications
# - name: "email"
#   email_configs:
#     - to: "you@yourdomain.com"
#       from: "alertmanager@yourdomain.com"
#       smarthost: "smtp.yourdomain.com:587"
#       auth_username: "alertmanager@yourdomain.com"
#       auth_password: "your-smtp-password"
#       send_resolved: true

Alerting Rules

Good alerts fire when users are affected and when someone can act on them. A rule that fires for a condition nobody can or should fix just creates noise.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148


# prometheus/rules/alerts.yml
groups:

  # ── Host Health ──────────────────────────────────────────────
  - name: host
    rules:

      # CPU sustained above 90% for 10 minutes
      - alert: HighCPUUsage
        expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High CPU on {{ $labels.instance }}"
          description: "CPU usage is {{ printf \"%.1f\" $value }}% on {{ $labels.instance }} for more than 10 minutes."
          runbook: "https://your-wiki/runbooks/high-cpu"

      # Memory available below 10%
      - alert: LowMemory
        expr: (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 < 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Low memory on {{ $labels.instance }}"
          description: "Only {{ printf \"%.1f\" $value }}% memory available on {{ $labels.instance }}."

      # Critical memory — below 5%
      - alert: CriticalMemory
        expr: (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 < 5
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "CRITICAL: Memory nearly exhausted on {{ $labels.instance }}"
          description: "Only {{ printf \"%.1f\" $value }}% memory available. OOM kills imminent."

      # Disk usage above 80%
      - alert: DiskSpaceWarning
        expr: 100 - ((node_filesystem_avail_bytes{mountpoint="/",fstype!="rootfs"} / node_filesystem_size_bytes{mountpoint="/",fstype!="rootfs"}) * 100) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Disk space warning on {{ $labels.instance }}"
          description: "Disk usage is {{ printf \"%.1f\" $value }}% on {{ $labels.instance }}:{{ $labels.mountpoint }}."

      # Disk usage above 90% — act now
      - alert: DiskSpaceCritical
        expr: 100 - ((node_filesystem_avail_bytes{mountpoint="/",fstype!="rootfs"} / node_filesystem_size_bytes{mountpoint="/",fstype!="rootfs"}) * 100) > 90
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "CRITICAL: Disk nearly full on {{ $labels.instance }}"
          description: "{{ printf \"%.1f\" $value }}% disk used on {{ $labels.instance }}. Immediate action required."

      # System has been rebooted recently
      - alert: HostRestarted
        expr: node_time_seconds - node_boot_time_seconds < 600
        for: 0m
        labels:
          severity: info
        annotations:
          summary: "Host {{ $labels.instance }} just restarted"
          description: "System uptime is less than 10 minutes — recent reboot or crash."

  # ── Container Health ──────────────────────────────────────────
  - name: containers
    rules:

      # Container using more than 80% of its memory limit
      - alert: ContainerHighMemory
        expr: |
          (container_memory_working_set_bytes{name!=""} /
           container_spec_memory_limit_bytes{name!=""} * 100) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Container {{ $labels.name }} high memory usage"
          description: "Container {{ $labels.name }} is using {{ printf \"%.1f\" $value }}% of its memory limit."

      # Container has been restarting repeatedly
      - alert: ContainerRestarting
        expr: increase(container_restart_count{name!=""}[15m]) > 3
        for: 0m
        labels:
          severity: warning
        annotations:
          summary: "Container {{ $labels.name }} is crash-looping"
          description: "Container {{ $labels.name }} has restarted {{ $value }} times in the last 15 minutes."

      # Container is down (not in running state)
      - alert: ContainerDown
        expr: absent(container_last_seen{name="your-critical-service"})
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Critical container is down"
          description: "Container 'your-critical-service' has not been seen for more than 1 minute."

  # ── Service Availability ─────────────────────────────────────
  - name: availability
    rules:

      # Prometheus target is down
      - alert: TargetDown
        expr: up == 0
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Scrape target {{ $labels.instance }} is down"
          description: "Prometheus cannot reach {{ $labels.job }}/{{ $labels.instance }}."

      # Dead man's switch — fires continuously to confirm the alerting pipeline works
      - alert: DeadMansSwitch
        expr: vector(1)
        labels:
          severity: none
        annotations:
          summary: "Alerting pipeline is alive"

  # ── Prometheus Self-Monitoring ────────────────────────────────
  - name: prometheus
    rules:

      # Prometheus config reload failed
      - alert: PrometheusConfigReloadFailed
        expr: prometheus_config_last_reload_successful != 1
        for: 0m
        labels:
          severity: warning
        annotations:
          summary: "Prometheus config reload failed"
          description: "Prometheus failed to reload its configuration. Check the config file."

      # Prometheus storage running out of room
      - alert: PrometheusTSDBStorageWarning
        expr: (1 - (prometheus_tsdb_head_chunks_storage_size_bytes / prometheus_tsdb_storage_blocks_bytes)) < 0.2
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Prometheus TSDB storage filling up"

The Awesome Prometheus Alerts Collection

Rather than writing all rules from scratch, the community-maintained Awesome Prometheus Alerts collection has production-ready rules for nearly every exporter imaginable: PostgreSQL, Redis, Nginx, MySQL, Elasticsearch, Kafka, RabbitMQ, and hundreds more. Copy, adapt, and use.

Alerting Best Practices

Alert fatigue is real and dangerous: engineers start ignoring notifications because there are too many, too often, or about things they can’t fix. Alerting done well means every notification demands attention.

Alert on Symptoms, Not Causes

Bad: “CPU usage > 80%” or “PostgreSQL connection count > 50” — these might be normal behavior. Good: “Request latency p95 > 500ms” or “API error rate > 5% for 5 minutes” — these mean users are affected.

High CPU might be fine. Slow requests definitely aren’t. Users don’t care that a container restarted; they care when the app is slow or returning errors.

The Alert Hierarchy

Match urgency to the response you actually want:

Page (wake someone up):
- Service completely down
- Error rate > 10% for 5 minutes
- p99 latency > 5 seconds for 5 minutes

Ticket (fix during business hours):
- Error rate > 1% for 30 minutes
- Disk usage > 80%
- Certificate expires in < 14 days

Log (investigate when convenient):
- Single request failures
- Individual host issues
- Approaching thresholds

This maps directly onto the severity: critical / warning / info labels routed in the Alertmanager config above.

Every Alert Must Be Actionable

If you receive an alert and there’s nothing you can or should do about it, delete it and use a dashboard panel instead.

Checklist for each alert:

Is there an action I can take right now?
Is this impacting, or likely to impact, users?
Does this alert have a runbook link?
Is this alert going to the right person?

Then keep the set healthy: group related alerts so one incident is one notification, automate anything that doesn’t need human judgment, and review regularly — delete alerts nobody acts on.

The DeadMansSwitch Pattern

Add an alert that fires continuously and is routed to a null receiver locally. Then configure an external service (like Healthchecks.io) to expect a regular ping from this alert and notify you if it stops. This catches failures in your alerting pipeline itself — if Prometheus, Alertmanager, or your notification channel breaks, you’ll know.

Silence During Maintenance

Use Alertmanager’s silence feature when doing planned maintenance to prevent spam:

1
2
3
4
5


# Create a silence for all alerts on a specific instance for 2 hours
amtool silence add --alertmanager.url http://localhost:9093 \
  --comment "Planned maintenance" \
  --duration 2h \
  instance="your-server"

Grafana Setup and Dashboards

Auto-Provisioned Data Sources

Instead of clicking through the Grafana UI, provision data sources automatically at startup:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


# grafana/provisioning/datasources/datasources.yml
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false

  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
    editable: false
    jsonData:
      maxLines: 1000

Importing Community Dashboards

Grafana’s community dashboard library at grafana.com/grafana/dashboards has thousands of pre-built dashboards. Import by ID — no need to build from scratch.

Essential dashboards to import immediately:

Dashboard	ID	What it shows
Node Exporter Full	`1860`	Complete host metrics: CPU, memory, disk I/O, network
Docker & System (cAdvisor)	`16310`	Per-container CPU, memory, network stats
Loki Dashboard	`13639`	Log volume, error rates, label breakdown
Prometheus Stats	`2`	Prometheus server health and performance
Alertmanager	`9578`	Alert status and history
Traefik v3	`17346`	Request rates, response times, error rates (requires Traefik metrics)

To import: Grafana UI → Dashboards → Import → enter dashboard ID → Load → select your Prometheus data source → Import.

Designing Your Own Dashboards

Resist the urge to graph everything. A useful dashboard tells a story, and three of them cover almost every need:

Overview — high-level health at a glance: RED metrics per service, traffic patterns, recent deployments marked.
Service — a deep dive into one service: every endpoint broken down, dependency health, resource usage.
Investigation — ad-hoc exploration: log correlation, trace sampling, custom queries.

When you build a custom dashboard, structure it around the RED method or the Four Golden Signals. A practical layout for a web service:

Row 1: Overview (stat panels)
  - Requests/sec (last 5m)   - Error rate %   - p99 latency   - Uptime

Row 2: Traffic
  - Requests/sec over time (graph)
  - Response status codes (stacked bar)

Row 3: Errors
  - Error rate over time
  - Recent error log lines from Loki (inline log panel)

Row 4: Saturation
  - CPU usage %   - Memory usage %   - Active connections

Key Grafana tips:

Put the most important metrics at the top, and use consistent colors (red = bad, green = good).
Show context: overlay lines for SLOs and deployment markers. Prefer graphs over single numbers — trends matter.
Use template variables for $instance and $service so one dashboard works across all your hosts and services.
Link metric panels to Loki log panels — clicking a CPU spike should surface logs from that time window automatically.
Set panel links to your runbooks so on-call engineers have context right on the dashboard.
Enable Grafana Alerting (built in) as an alternative to Alertmanager for simpler setups that don’t need complex routing.

Querying: PromQL and LogQL Essentials

PromQL (Prometheus Query Language)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


# CPU usage percentage per instance
100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory used (not available)
node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes

# Disk usage percentage
100 - ((node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100)

# HTTP request rate (requests per second, 5-minute window)
rate(http_requests_total[5m])

# HTTP error rate as a percentage
rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) * 100

# Container memory usage in MiB
container_memory_working_set_bytes{name!=""} / 1024 / 1024

# Network egress per container in Mbps
irate(container_network_transmit_bytes_total{name!=""}[5m]) * 8 / 1024 / 1024

# Top 5 containers by CPU usage
topk(5, sum by (name) (irate(container_cpu_usage_seconds_total{name!=""}[5m])))

LogQL (Loki Query Language)

# All logs from a specific container
{container="my-app"}

# Error logs from all containers in a Docker Compose project
{project="my-project"} |= "error"

# Filter by log level label
{app="nginx"} | logfmt | level="error"

# Count error log lines per minute (metric query)
sum(rate({app="my-app"} |= "ERROR" [1m]))

# Parse JSON logs and filter by field
{container="api"} | json | status_code >= 500

# Most common error messages in the last hour
topk(10, sum by (msg) (count_over_time({container="api"} |= "error" | json | __error__="" [1h])))

# Latency from structured logs
{container="api"} | json | duration_ms > 1000

# Logs across all containers matching a pattern
{project=~".+"} |~ "exception|panic|fatal"

Integrating Monitoring with Your Existing Docker Services

If you’re already running Docker services on a VPS or homelab (Traefik, databases, web apps), the monitoring stack plugs in with minimal changes.

Connecting to Existing Docker Networks

Your monitoring stack and application services need to be on the same Docker network for Prometheus to scrape app metrics. Add your app network to the monitoring Compose file:

1
2
3
4
5
6


# In monitoring/docker-compose.yml — add your app network
networks:
  monitoring:
    driver: bridge
  app_network:         # The network your apps use
    external: true     # Already created by your app's docker-compose.yml

Then attach Prometheus to both networks:

1
2
3
4
5


services:
  prometheus:
    networks:
      - monitoring
      - app_network    # Can now reach containers on this network

And add your app services to the Prometheus scrape config:

1
2
3
4
5


# In prometheus/prometheus.yml
scrape_configs:
  - job_name: "my-api"
    static_configs:
      - targets: ["my-api:8000"]  # Container name resolves on shared network

Traefik Integration

If you’re running Traefik (covered in our Traefik Complete Guide), enable its built-in Prometheus metrics and log shipping:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


# In traefik.yml — enable Prometheus metrics
metrics:
  prometheus:
    buckets:
      - 0.1
      - 0.3
      - 1.2
      - 5.0
    entryPoint: metrics   # Serve metrics on a dedicated entrypoint

# Enable access logs for Loki
accessLog:
  filePath: "/var/log/traefik/access.log"
  format: json           # JSON format is much easier to parse with Loki

# Traefik entrypoints (add a metrics entrypoint)
entryPoints:
  web:
    address: ":80"
  websecure:
    address: ":443"
  metrics:
    address: ":8082"     # Prometheus will scrape this port

1
2
3
4


# Add to prometheus/prometheus.yml
- job_name: "traefik"
  static_configs:
    - targets: ["traefik:8082"]

Now the Traefik v3 Grafana dashboard (ID 17346) shows real-time request rates, response times, error rates, and per-service breakdowns sourced directly from your reverse proxy.

Database Monitoring with Exporters

Add exporters as sidecar containers alongside your databases:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


# In your app's docker-compose.yml
services:
  postgres:
    image: postgres:16
    environment:
      POSTGRES_DB: myapp
      POSTGRES_USER: user
      POSTGRES_PASSWORD: ${DB_PASSWORD}
    networks:
      - app_network

  # Postgres metrics exporter
  postgres-exporter:
    image: prometheuscommunity/postgres-exporter:v0.15.0
    environment:
      DATA_SOURCE_NAME: "postgresql://user:${DB_PASSWORD}@postgres:5432/myapp?sslmode=disable"
    networks:
      - app_network
      - monitoring_network  # Must be reachable by Prometheus

  redis:
    image: redis:7-alpine
    networks:
      - app_network

  # Redis metrics exporter
  redis-exporter:
    image: oliver006/redis_exporter:v1.58.0
    environment:
      REDIS_ADDR: "redis://redis:6379"
    networks:
      - app_network
      - monitoring_network

1
2
3
4
5
6
7
8


# Add to prometheus/prometheus.yml
- job_name: "postgres"
  static_configs:
    - targets: ["postgres-exporter:9187"]

- job_name: "redis"
  static_configs:
    - targets: ["redis-exporter:9121"]

Monitoring Services on Multiple Machines

If you have more than one server (a second VPS, a homelab cluster, a separate database server), deploy Node Exporter on each and add them to Prometheus’s scrape config:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


# On each additional machine, run node-exporter:
docker run -d \
  --name node-exporter \
  --restart unless-stopped \
  --net="host" \
  --pid="host" \
  -v "/:/host:ro,rslave" \
  prom/node-exporter:v1.7.0 \
  --path.rootfs=/host

# Then in prometheus/prometheus.yml on your monitoring host:
- job_name: "remote-hosts"
  static_configs:
    - targets:
        - "10.0.0.2:9100"   # Second VPS or homelab node
        - "10.0.0.3:9100"   # Third node
        - "10.0.0.4:9100"   # etc.
      labels:
        env: "production"

For cross-network monitoring (e.g., homelab machines monitored from a cloud VPS), use Tailscale mesh networking — each machine gets a stable 100.x.x.x IP reachable from anywhere on your tailnet. No firewall rules to open, no separate VPN to configure.

Exposing Grafana via Traefik

Never expose Grafana (or Prometheus/Loki) directly on a public port. Route them through your existing Traefik reverse proxy with TLS:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# Add to your monitoring/docker-compose.yml grafana service
services:
  grafana:
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.grafana.rule=Host(`grafana.yourdomain.com`)"
      - "traefik.http.routers.grafana.entrypoints=websecure"
      - "traefik.http.routers.grafana.tls.certresolver=letsencrypt"
      - "traefik.http.services.grafana.loadbalancer.server.port=3000"
      # Optional: restrict to your IP only
      # - "traefik.http.routers.grafana.middlewares=ip-allowlist@file"
    networks:
      - monitoring
      - traefik_proxy   # Must be on Traefik's network

Traefik auto-discovers the label and creates a TLS-secured route to your Grafana instance at https://grafana.yourdomain.com.

Operational Tips

Reloading Configuration Without Restart

Prometheus, Alertmanager, and Loki all support live config reloads:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# Reload Prometheus config (requires --web.enable-lifecycle flag)
curl -X POST http://localhost:9090/-/reload

# Reload Alertmanager config
curl -X POST http://localhost:9093/-/reload

# Verify Prometheus config before reloading
docker exec prometheus promtool check config /etc/prometheus/prometheus.yml

# Verify alert rules
docker exec prometheus promtool check rules /etc/prometheus/rules/alerts.yml

Checking Metrics Are Being Scraped

1
2
3
4
5
6
7
8


# See all active scrape targets and their status
open http://localhost:9090/targets

# Query metrics in the Prometheus expression browser
open http://localhost:9090/graph

# Check what labels exist on a metric
curl http://localhost:9090/api/v1/label/job/values

Storage and Retention Management

Prometheus stores data locally by default. Monitor disk usage and tune retention:

1
2
3
4
5


# Check current Prometheus storage usage
docker exec prometheus du -sh /prometheus

# Current data in the Prometheus TSDB
curl http://localhost:9090/api/v1/status/tsdb | python3 -m json.tool

For long-term storage beyond 30 days without significant disk growth, integrate Grafana Mimir (Prometheus-compatible, scales horizontally) or use a remote_write target like Grafana Cloud’s free tier.

Loki Log Retention

Loki’s retention_period controls how long logs are kept. Set it based on your disk space and compliance needs:

1
2
3


# In loki-config.yml
limits_config:
  retention_period: 30d   # Keep 30 days; adjust to 7d on small VPS disks

1
2


# Check Loki storage usage
docker exec loki du -sh /loki

Cost Estimates: How Much Disk Does This Take?

For a typical small VPS or homelab machine with a few Docker services:

Component	Storage per day	Per month
Prometheus metrics (5 scrape targets, 15s interval)	~50–150 MB/day	~1.5–4.5 GB
Loki logs (moderate log volume)	~100–500 MB/day	~3–15 GB
Grafana (dashboards, alert state)	Minimal	~100 MB
Total (conservative)	~200 MB/day	~6 GB
Total (busy server)	~700 MB/day	~21 GB

Adjust --storage.tsdb.retention.time in Prometheus and retention_period in Loki to fit your available disk. On a 25 GB VPS, 14-day retention for both is a safe default.

Quick Start: From Zero to Dashboard in 10 Minutes

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


# 1. Create the directory structure
mkdir -p monitoring/{prometheus/rules,loki,promtail,alertmanager,grafana/provisioning/{datasources,dashboards}}

# 2. Drop all the config files from this guide into the right directories

# 3. Set your admin password
echo "GRAFANA_ADMIN_PASSWORD=your-strong-password" > monitoring/.env

# 4. Start the stack
cd monitoring && docker compose up -d

# 5. Verify everything is running
docker compose ps

# 6. Check Prometheus targets (wait ~30 seconds for initial scrape)
curl http://localhost:9090/api/v1/targets | python3 -m json.tool

# 7. Open Grafana — login admin / your-strong-password
echo "Open http://localhost:3000 (or your domain)"

# 8. Import the Node Exporter Full dashboard (ID 1860)
# 9. Import the Docker / cAdvisor dashboard (ID 16310)
# Grafana → Dashboards → Import → enter ID → Load

Quick Wins If You’re Starting Smaller

You don’t need the full stack to get value on day one:

Add a /health endpoint — returns 200 if the service is working.
Log request durations — even before metrics, this helps.
Monitor error rates — the simplest useful metric.
Mark deploys on your graphs — most incidents correlate with a change.
Set up a status page — communicate with users when something breaks.

Going Further: The Full LGTM Stack

Once your foundational metrics and logs stack is solid, consider extending it:

Grafana Tempo (distributed tracing) — when you have multiple services calling each other, traces show exactly where latency is introduced. Instrument apps with OpenTelemetry, send traces to Tempo, and correlate them with metrics and logs in unified dashboards.

Grafana Alloy (next-gen agent) — the successor to both Promtail (logs) and the Grafana Agent (metrics/traces). One config file collects logs, metrics, and traces, with native OpenTelemetry support. If you’re starting fresh, consider Alloy over Promtail.

Grafana Mimir (long-term metrics) — a horizontally scalable, Prometheus-compatible TSDB for months or years of retention. It can use S3-compatible object storage (Backblaze B2, MinIO, AWS S3) for cheap long-term storage.

Blackbox Exporter (uptime monitoring) — probes any HTTP/HTTPS endpoint from outside your containers, verifying services are reachable and responding correctly from an external perspective:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


# Add to prometheus/prometheus.yml
- job_name: "blackbox-http"
  metrics_path: /probe
  params:
    module: [http_2xx]
  static_configs:
    - targets:
        - "https://yourdomain.com"
        - "https://api.yourdomain.com/health"
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: blackbox-exporter:9115

Verdict

Good monitoring answers two questions: “Is it broken?” and “Why is it broken?” Start with the concepts — RED for services, USE for infrastructure, the Four Golden Signals when you can only watch four things — because no amount of tooling rescues a stack that measures the wrong things. Then build the implementation: Prometheus for metrics, Loki for logs, Grafana to see it all, Alertmanager to tell you when it matters.

The whole stack runs in Docker on a single VPS or homelab box for roughly 6–21 GB of disk a month and zero licensing cost. You don’t have to deploy all of it at once — a /health endpoint and an error-rate graph beat flying blind. But the day something breaks at 2 a.m., the difference between “I have no idea when this started” and “the 502s began three minutes after my last deploy, here are the logs” is the difference between an hour of guessing and a five-minute fix. Build it before you need it.