Loki for Log Aggregation: Ship, Query, and Correlate Your Logs

Your Prometheus dashboards are clean. CPU, memory, disk, request rates — all graphed and alerting. Then something breaks at 2am and the metrics tell you that something happened, but not why. The error rate spiked. Memory climbed. A container restarted three times. What actually went wrong? The answer is in the logs — and if your logs are scattered across a dozen hosts with no central place to query them, you are SSH-ing into boxes and grepping files while the incident is still open.

Loki fixes this. It gives you a single place to ship, store, and query logs from every host, container, and service in your infrastructure. And because it was built by the same team as Grafana and Prometheus, it integrates cleanly with everything you already have. One query interface, one dashboard, logs and metrics side by side.

This guide builds a complete Loki stack from scratch — Docker Compose configuration, log shipping with Promtail and Alloy, LogQL query language, Grafana integration, and alerting rules. By the end, you will be able to query logs across your entire infrastructure from a single browser tab and correlate them with Prometheus metrics in the same dashboard.

Part 1: What Is Loki and Why Use It?

Loki is often described as “Prometheus, but for logs.” That framing is useful because it captures the key design decision: Loki does not index log content. It only indexes the labels attached to log streams.

When a log line arrives at Loki, the label set is indexed (things like host, job, container_name, level). The actual text of the log line is compressed and stored as a chunk on disk or in object storage. When you query, Loki first uses the label index to find the relevant chunks, then decompresses and scans the log content to match your query. This is fundamentally different from Elasticsearch, which tokenizes and inverts every word in every log line into a massive index.

The practical consequence: Loki is dramatically cheaper to run. An Elasticsearch cluster that indexes 10GB of logs per day needs serious CPU and RAM to maintain the inverted index. A Loki installation handling the same volume runs on a modest single machine because it is mostly doing I/O — fetching chunks and scanning them — rather than index maintenance.

The tradeoff: If you rely on full-text search across arbitrary fields without knowing label context, Elasticsearch is faster. Google-style “find any log containing this string anywhere across all logs ever” is not Loki’s sweet spot. Loki is better framed as: you know what service you care about, you know roughly when the event happened, and you want to filter and extract structured data from the relevant log lines.

The Grafana LGTM Stack

Loki is one component in the broader Grafana observability ecosystem, sometimes called the LGTM stack:

• L — Loki (logs)
• G — Grafana (visualization)
• T — Tempo (distributed traces)
• M — Mimir or Prometheus (metrics)

All four are designed to interoperate. In Grafana you can jump from a Prometheus alert to the relevant Loki logs in one click. A log line containing a trace ID can link directly into Tempo. The stack is modular — you can adopt Loki without changing your existing Prometheus and Grafana setup.

When Loki Is Not the Right Choice

Loki is the right choice for most operational log aggregation. It is not the right choice if:

• You need full-text search across log content without knowing which service or host the log came from
• Your compliance requirements demand indexed, searchable audit logs that can be queried by security teams without knowing label structure
• You are ingesting structured application logs that already have dozens of fields and you want to query any of them like a database

For those cases, Elasticsearch or OpenSearch remains the better option. For the vast majority of homelab and small-team production use cases — centralizing Docker logs, monitoring auth events, analyzing web server access, alerting on application errors — Loki is efficient, cost-effective, and integrates cleanly with the stack you already have.

Part 2: Architecture

Core Components

Loki is composed of several internal components. In the default monolithic deployment they all run inside a single binary, which is what you want for homelab and small production deployments.

Distributor — Receives incoming log streams from agents. Validates labels, applies rate limiting, and fans out to ingesters using consistent hashing so the same stream always hits the same ingester.

Ingester — Holds recent log chunks in memory. Flushes completed chunks to the storage backend. Replication across ingesters provides durability before a flush completes.

Querier — Handles query and query_range API requests. Fetches chunks from storage (and from ingesters for very recent data), decompresses them, and evaluates the LogQL expression against the content.

Query Frontend — Optional component that sits in front of queriers. Splits large queries into smaller time shards, executes them in parallel, caches results, and merges responses. Important for performance at scale.

Compactor — Runs as a background process, merging small chunks into larger ones and enforcing retention policies. Reduces storage costs and speeds up queries over older data.

Ruler — Evaluates LogQL alerting and recording rules on a schedule. Sends firing alerts to Alertmanager, exactly like Prometheus does for metric-based alerts.

Monolithic vs Microservices Mode

Monolithic mode (single binary with -target=all) runs all components in one process. This is the right choice for everything up to tens of gigabytes of logs per day. Configuration is simpler, the operational footprint is minimal, and vertical scaling takes you a long way.

Microservices mode deploys each component separately (or in small logical groups called “simple scalable” mode). This enables horizontal scaling of individual bottleneck components — typically ingesters and queriers under high load. Kubernetes operators and Helm charts exist for this deployment model. For anything running on a single machine or a small cluster, you will not need it.

Storage Backends

The filesystem backend is perfectly fine for a homelab with adequate disk space. For anything you care about losing, use object storage — even a small MinIO instance on a NAS is better than chunks on a Docker volume.

Log Streams and Labels

A log stream is the fundamental unit in Loki. It is defined entirely by its set of labels. Any two log lines that share the exact same label set belong to the same stream. This is exactly how Prometheus time series work — the label set is the identity.

stream: {job="nginx", host="web01", env="prod"}
stream: {job="nginx", host="web02", env="prod"}

These are two different streams, even though they both have job="nginx". Each stream is stored as a sequence of compressed chunks, ordered by timestamp. Within a stream, log lines must arrive in timestamp order (or very close to it — Loki allows configurable out-of-order tolerance).

Architecture Diagram

┌─────────────────────────────────────────────────────────────────┐
│                      Log Sources                                 │
│                                                                  │
│  ┌────────────┐  ┌────────────┐  ┌────────────┐  ┌───────────┐  │
│  │  /var/log  │  │  Docker    │  │  Systemd   │  │  App HTTP │  │
│  │  files     │  │  containers│  │  Journal   │  │  /push    │  │
│  └─────┬──────┘  └─────┬──────┘  └─────┬──────┘  └─────┬─────┘  │
└────────┼───────────────┼───────────────┼────────────────┼────────┘
         │               │               │                │
         └───────────────┴───────────────┴────────────────┘
                                 │
                                 ▼
              ┌──────────────────────────────────┐
              │    Promtail / Grafana Alloy       │
              │  (label assignment, pipeline      │
              │   stages, HTTP push to Loki)      │
              └──────────────────┬───────────────┘
                                 │  HTTP POST /loki/api/v1/push
                                 ▼
              ┌──────────────────────────────────┐
              │           Loki (:3100)            │
              │                                   │
              │  Distributor → Ingester → Chunks  │
              │  Compactor (background)           │
              │  Ruler (alerting rules)           │
              │  Storage: filesystem / S3 / MinIO │
              └──────────────────┬───────────────┘
                                 │  LogQL query
                                 ▼
              ┌──────────────────────────────────┐
              │         Grafana (:3000)           │
              │  Explore │ Dashboards │ Alerting  │
              │  (Loki datasource)                │
              └──────────────────────────────────┘
                                 │
                        Ruler ──►│ Alertmanager (:9093)
                                 │  PagerDuty / Slack / Email

Part 3: Installation with Docker Compose

The following stack brings up Loki, Promtail, and Grafana with Loki auto-provisioned as a datasource. Create a project directory and place these files inside it.

Directory Layout

loki-stack/
├── docker-compose.yml
├── loki-config.yaml
├── promtail-config.yaml
└── grafana/
    └── provisioning/
        └── datasources/
            └── loki.yaml

docker-compose.yml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63

version: "3.8"

networks:
  loki:
    driver: bridge

volumes:
  loki_data:
  grafana_data:

services:
  loki:
    image: grafana/loki:2.9.4
    container_name: loki
    restart: unless-stopped
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yaml:/etc/loki/config.yaml:ro
      - loki_data:/loki
    command: -config.file=/etc/loki/config.yaml
    networks:
      - loki
    healthcheck:
      test: ["CMD-SHELL", "wget -q --spider http://localhost:3100/ready || exit 1"]
      interval: 10s
      timeout: 5s
      retries: 5

  promtail:
    image: grafana/promtail:2.9.4
    container_name: promtail
    restart: unless-stopped
    volumes:
      - ./promtail-config.yaml:/etc/promtail/config.yaml:ro
      - /var/log:/var/log:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /run/docker.sock:/run/docker.sock:ro
      - /etc/machine-id:/etc/machine-id:ro
    command: -config.file=/etc/promtail/config.yaml
    networks:
      - loki
    depends_on:
      loki:
        condition: service_healthy

  grafana:
    image: grafana/grafana:10.3.3
    container_name: grafana
    restart: unless-stopped
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=changeme
      - GF_USERS_ALLOW_SIGN_UP=false
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
    networks:
      - loki
    depends_on:
      - loki

loki-config.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096
  log_level: warn

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://alertmanager:9093
  storage:
    type: local
    local:
      directory: /loki/rules
  rule_path: /loki/rules-temp
  enable_api: true

limits_config:
  # Ingestion
  ingestion_rate_mb: 16
  ingestion_burst_size_mb: 32
  max_streams_per_user: 10000
  max_line_size: 256kb
  # Retention
  retention_period: 30d
  # Query limits
  max_query_length: 721h
  max_query_parallelism: 32
  max_entries_limit_per_query: 5000

compactor:
  working_directory: /loki/compactor
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150
  delete_request_store: filesystem

ingester:
  chunk_idle_period: 3m
  chunk_block_size: 262144
  chunk_retain_period: 1m
  max_transfer_retries: 0
  wal:
    enabled: true
    dir: /loki/wal

Grafana datasource provisioning (grafana/provisioning/datasources/loki.yaml)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

apiVersion: 1

datasources:
  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
    isDefault: true
    jsonData:
      maxLines: 1000
      derivedFields:
        - datasourceUid: tempo
          matcherRegex: '"trace_id":"(\w+)"'
          name: TraceID
          url: "$${__value.raw}"
    version: 1
    editable: true

  # Uncomment if you have Prometheus in the same stack
  # - name: Prometheus
  #   type: prometheus
  #   access: proxy
  #   url: http://prometheus:9090
  #   isDefault: false

Bring the stack up:

1
2
3
4
5

docker compose up -d
docker compose ps
# Verify Loki is ready
curl http://localhost:3100/ready
# Should return: ready

Part 4: Log Shipping with Promtail

Promtail is the official Loki log shipping agent. It is a lightweight binary (or container) that tails log files, reads from the systemd journal, discovers Docker container logs, processes each line through a pipeline, and ships the results to Loki via HTTP.

Promtail Config Anatomy

A Promtail configuration has four top-level sections:

server:        # Promtail's own HTTP server (for metrics, health)
positions:     # File tracking state — which byte offset we've read to
clients:       # Where to ship logs (Loki URL + auth)
scrape_configs: # What to tail and how to label it

Complete promtail-config.yaml

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147

server:
  http_listen_port: 9080
  grpc_listen_port: 0
  log_level: warn

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push
    tenant_id: fake   # Required when auth_enabled: false

scrape_configs:

  # ─── System Logs ───────────────────────────────────────────────
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: syslog
          host: __HOSTNAME__
          __path__: /var/log/syslog

  - job_name: auth
    static_configs:
      - targets:
          - localhost
        labels:
          job: auth
          host: __HOSTNAME__
          __path__: /var/log/auth.log

  # ─── Docker Container Logs ─────────────────────────────────────
  - job_name: docker
    docker_sd_configs:
      - host: unix:///run/docker.sock
        refresh_interval: 5s
        filters:
          - name: status
            values: ["running"]
    relabel_configs:
      # Use container name as the job label
      - source_labels: [__meta_docker_container_name]
        regex: "/(.*)"
        target_label: container
      # Expose compose project and service as labels
      - source_labels: [__meta_docker_container_label_com_docker_compose_project]
        target_label: compose_project
      - source_labels: [__meta_docker_container_label_com_docker_compose_service]
        target_label: compose_service
      - source_labels: [__meta_docker_container_label_com_docker_compose_service]
        target_label: job
      # Static host label
      - replacement: __HOSTNAME__
        target_label: host
    pipeline_stages:
      # Try to parse the log line as JSON (for apps that emit JSON logs)
      - json:
          expressions:
            level: level
            msg: msg
            ts: ts
      # If the json stage found a level field, promote it to a label
      - labels:
          level:
      # Drop DEBUG logs to save storage (remove this if you need them)
      - drop:
          expression: '.*'
          if: '{{ eq .level "debug" }}'

  # ─── Nginx Access Logs ─────────────────────────────────────────
  - job_name: nginx
    static_configs:
      - targets:
          - localhost
        labels:
          job: nginx
          host: __HOSTNAME__
          __path__: /var/log/nginx/access.log
    pipeline_stages:
      # Parse nginx combined log format
      - regex:
          expression: >-
            ^(?P<remote_addr>\S+) - (?P<remote_user>\S+) \[(?P<time_local>[^\]]+)\]
            "(?P<method>\S+) (?P<path>\S+) \S+" (?P<status>\d+) (?P<body_bytes_sent>\d+)
            "(?P<http_referer>[^"]*)" "(?P<http_user_agent>[^"]*)"
            (?P<request_time>\d+\.\d+)?
      # Promote HTTP status code to a label (low cardinality — only ~10 unique values)
      - labels:
          status:
          method:
      # Reformat the log line to be more readable in Grafana
      - output:
          source: method

  # ─── Traefik Access Logs ───────────────────────────────────────
  - job_name: traefik
    static_configs:
      - targets:
          - localhost
        labels:
          job: traefik
          host: __HOSTNAME__
          __path__: /var/log/traefik/access.log
    pipeline_stages:
      - json:
          expressions:
            level: level
            ClientHost: ClientHost
            RequestMethod: RequestMethod
            RequestPath: RequestPath
            DownstreamStatus: DownstreamStatus
            Duration: Duration
      - labels:
          level:
          DownstreamStatus:

  # ─── Kubernetes / K3s Pod Logs (when running on a node) ────────
  - job_name: kubernetes-pods
    static_configs:
      - targets:
          - localhost
        labels:
          job: kubernetes
          host: __HOSTNAME__
          __path__: /var/log/pods/*/*/*.log
    pipeline_stages:
      # K3s/containerd log format wraps the actual log in JSON
      - json:
          expressions:
            log: log
            stream: stream
            time: time
      - timestamp:
          source: time
          format: RFC3339Nano
      - output:
          source: log
      # Extract namespace and pod name from the file path
      - regex:
          expression: '/var/log/pods/(?P<namespace>[^_]+)_(?P<pod_name>[^_]+)_[^/]+/(?P<container_name>[^/]+)/\d+\.log'
          source: filename
      - labels:
          namespace:
          pod_name:
          container_name:

Pipeline Stages Explained

Promtail processes each log line through a sequence of pipeline stages before sending it to Loki. Stages run in order and can mutate the log line or its labels.

Multiline example — grouping Java stack traces into a single log entry:

1
2
3
4
5

pipeline_stages:
  - multiline:
      firstline: '^\d{4}-\d{2}-\d{2}'  # Line starting with a date is a new entry
      max_wait_time: 3s
      max_lines: 128

Part 5: Grafana Alloy — The Modern Replacement for Promtail

Grafana Alloy is the unified successor to both Promtail and Grafana Agent. Released in 2024, it combines log shipping (Promtail’s functionality), metrics collection, and trace forwarding into a single binary with a composable configuration language.

Should You Use Alloy or Promtail?

• New deployments: Use Alloy. It is the supported path forward and supports everything Promtail does, plus metrics scraping and trace forwarding.
• Existing Promtail installations: Promtail continues to work and receive bug fixes. There is no urgent need to migrate, but plan to do so when it makes sense.
• Kubernetes: Alloy’s Helm chart is the recommended deployment mechanism for K8s log collection going forward.

Alloy Configuration Basics

Alloy uses a configuration language called Alloy config (formerly called River). It is a declarative, component-based language where each block is a typed component that exposes exports and accepts arguments.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66

// alloy-config.alloy

// Discover Docker containers
discovery.docker "containers" {
  host = "unix:///run/docker.sock"
}

// Relabel to get container names as job labels
discovery.relabel "docker_relabel" {
  targets = discovery.docker.containers.targets

  rule {
    source_labels = ["__meta_docker_container_name"]
    regex         = "/(.*)"
    target_label  = "container"
  }

  rule {
    source_labels = ["__meta_docker_container_label_com_docker_compose_service"]
    target_label  = "job"
  }
}

// Tail Docker container logs
loki.source.docker "docker_logs" {
  host    = "unix:///run/docker.sock"
  targets = discovery.relabel.docker_relabel.output
  forward_to = [loki.write.default.receiver]

  labels = {
    host = env("HOSTNAME"),
  }
}

// Tail log files
local.file_match "system_logs" {
  path_targets = [
    {__path__ = "/var/log/syslog", job = "syslog", host = env("HOSTNAME")},
    {__path__ = "/var/log/auth.log", job = "auth", host = env("HOSTNAME")},
  ]
}

loki.source.file "system_files" {
  targets    = local.file_match.system_logs.targets
  forward_to = [loki.write.default.receiver]
}

// Process logs through a pipeline (equivalent to Promtail pipeline stages)
loki.process "parse_json_logs" {
  forward_to = [loki.write.default.receiver]

  stage.json {
    expressions = {level = "level", msg = "message"}
  }

  stage.labels {
    values = {level = null}
  }
}

// Ship to Loki
loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

Run Alloy in Docker:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# In your docker-compose.yml, replace the promtail service with:
  alloy:
    image: grafana/alloy:v1.1.0
    container_name: alloy
    restart: unless-stopped
    volumes:
      - ./alloy-config.alloy:/etc/alloy/config.alloy:ro
      - /var/log:/var/log:ro
      - /run/docker.sock:/run/docker.sock:ro
    command: run /etc/alloy/config.alloy --server.http.listen-addr=0.0.0.0:12345
    ports:
      - "12345:12345"  # Alloy UI
    networks:
      - loki

Part 6: Other Log Shipping Options

Docker Logging Driver

The Loki Docker logging driver sends container logs directly from Docker without a separate agent. Install it once on the host:

1
2
3

docker plugin install grafana/loki-docker-driver:latest \
  --alias loki \
  --grant-all-permissions

Configure it per container in your Compose file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

services:
  myapp:
    image: myapp:latest
    logging:
      driver: loki
      options:
        loki-url: "http://localhost:3100/loki/api/v1/push"
        loki-external-labels: "job=myapp,env=prod,host=${HOSTNAME}"
        loki-batch-size: "400"
        loki-retries: "3"

Or set it as the default Docker logging driver in /etc/docker/daemon.json:

1
2
3
4
5
6
7

{
  "log-driver": "loki",
  "log-opts": {
    "loki-url": "http://localhost:3100/loki/api/v1/push",
    "loki-external-labels": "host=myserver"
  }
}

Note: The Docker logging driver does not buffer to disk. If Loki is unreachable, logs are lost. For production, prefer Promtail or Alloy with their disk-based position tracking.

Vector.dev

Vector is a high-performance observability data pipeline that can collect from many sources and route to many sinks, including Loki:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

# vector.toml
[sources.docker]
type = "docker_logs"

[transforms.add_labels]
type = "remap"
inputs = ["docker"]
source = '''
.container_name = .container_name
.host = get_hostname!()
'''

[sinks.loki]
type = "loki"
inputs = ["add_labels"]
endpoint = "http://loki:3100"
encoding.codec = "text"
labels.job = "{{ container_name }}"
labels.host = "{{ host }}"

Fluent Bit

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# fluent-bit.conf
[INPUT]
    Name              tail
    Path              /var/log/syslog
    Tag               syslog

[OUTPUT]
    Name              loki
    Match             *
    Host              loki
    Port              3100
    Labels            job=syslog
    Auto_Kubernetes_Labels on

Direct HTTP Push

Any application can push logs directly to Loki’s HTTP API:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Push a single log line via curl
curl -s -X POST http://localhost:3100/loki/api/v1/push \
  -H 'Content-Type: application/json' \
  -d '{
    "streams": [{
      "stream": {"job": "myapp", "env": "prod"},
      "values": [
        ["'"$(date +%s%N)"'", "Application started successfully"]
      ]
    }]
  }'

This is useful for scripts, one-off events, or applications that cannot run a sidecar agent.

Part 7: LogQL Fundamentals

LogQL is Loki’s query language. It has two types of queries: log queries (return log lines) and metric queries (compute aggregate values from log streams, returning time series). The syntax is intentionally similar to PromQL to reduce the learning curve for Prometheus users.

Log Stream Selectors

Every LogQL query begins with a stream selector — a set of label matchers in curly braces:

{job="nginx"}
{job="nginx", host="web01"}
{job=~"nginx|traefik"}           # Regex match
{job!="auth"}                    # Negative exact match
{container=~"app-.*", env="prod"} # Regex on one label, exact on another

The stream selector must match at least one label. You cannot query Loki without a stream selector (unlike Elasticsearch where you can search everything).

Log Pipeline Operators

After the stream selector, you pipe the log lines through filter expressions:

# Contains string (case-sensitive)
{job="app"} |= "ERROR"

# Does not contain string
{job="app"} != "healthcheck"

# Regex match
{job="auth"} |~ "Failed (password|publickey)"

# Regex not match
{job="nginx"} !~ "^(10\.|192\.168\.)"

Multiple filters chain with implicit AND:

{job="app"} |= "ERROR" != "connection refused" |~ "user_id=\d+"

Parser Expressions

Parsers extract structured fields from log lines, making them available for further filtering and metric queries:

# Parse entire line as JSON
{job="app"} | json

# Parse specific JSON fields with aliases
{job="app"} | json msg="message", lvl="level", ts="timestamp"

# Parse key=value (logfmt) format
{job="app"} | logfmt

# Named capture groups from regex
{job="nginx"} | regexp `(?P<method>\w+) (?P<path>\S+) HTTP/\d\.\d" (?P<status>\d+)`

# Loki pattern parser (simpler syntax for fixed-format logs)
{job="nginx"} | pattern `<ip> - - [<_>] "<method> <path> <_>" <status> <bytes>`

After parsing, use | label_filter to filter on extracted fields:

# HTTP 5xx errors from nginx (status is a string after json parse)
{job="nginx"} | json | status >= 500

# Slow requests over 2 seconds
{job="nginx"} | json | response_time > 2.0

# Specific user
{job="app"} | json | user_id = "12345"

Line Format

Reformat the output line using extracted fields — useful in dashboards:

{job="nginx"} | json | line_format "{{.method}} {{.path}} → {{.status}} ({{.response_time}}s)"

Metric Queries

Metric queries wrap log queries in aggregation functions to produce time series:

# Log lines per second over a 5-minute rolling window
rate({job="nginx"}[5m])

# Total log volume
count_over_time({job="app"}[1h])

# Bytes ingested per second
bytes_rate({job="app"}[5m])

# Sum across label dimensions (like PromQL's sum by())
sum by (status) (rate({job="nginx"} | json [5m]))

Practical LogQL Examples

All errors in the last hour:

{job="app"} |= "ERROR"

Error rate as a metric (errors per second):

sum(rate({job="app"} |= "ERROR" [5m]))

SSH brute force detection — failed password attempts per 5 minutes per source IP:

sum by (source_ip) (
  count_over_time(
    {job="auth"} |~ "Failed password" | regexp `from (?P<source_ip>\d+\.\d+\.\d+\.\d+)` [5m]
  )
)

Nginx 5xx error rate:

sum(rate({job="nginx"} | json | status >= 500 [5m])) by (status)

Top paths by request count (for a rate table in Grafana):

topk(10, sum by (path) (
  count_over_time({job="nginx"} | json [5m])
))

Slow request analysis — 99th percentile response time:

quantile_over_time(0.99,
  {job="nginx"}
  | json
  | unwrap response_time [5m]
) by (path)

Container logs filtered by container name:

{container="myapp"} |= "panic"

All logs from a Kubernetes namespace:

{namespace="production"} |= "error"

OOM killer events on any host:

{job="syslog"} |~ "Out of memory|oom_kill_process|Killed process"

Recent auth events for a specific user:

{job="auth"} |~ "user admin" | line_format "{{__line__}}"

Log volume by job — useful for identifying noisy sources:

sum by (job) (bytes_rate({host="web01"}[5m]))

logcli — Querying Loki from the Terminal

logcli is the official command-line client for Loki. Install it on any machine that needs terminal access to your log data:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# Install
curl -sLO https://github.com/grafana/loki/releases/latest/download/logcli-linux-amd64.zip
unzip logcli-linux-amd64.zip
sudo mv logcli-linux-amd64 /usr/local/bin/logcli
sudo chmod +x /usr/local/bin/logcli

# Set the Loki endpoint
export LOKI_ADDR=http://localhost:3100

# Query logs
logcli query '{job="nginx"}' --limit=50 --since=1h

# Stream live logs (like tail -f)
logcli query '{job="auth"}' --tail

# List all labels
logcli labels

# List label values
logcli labels job

# Instant metric query
logcli instant-query 'sum(rate({job="nginx"}[5m]))'

Part 8: Grafana Integration

Adding Loki as a Datasource

If you used the provisioning file from Part 3, Loki is already configured. To add it manually:

1. In Grafana, go to Connections → Data Sources → Add data source
2. Select Loki
3. Set URL to http://loki:3100 (or your Loki address)
4. Click Save & Test — you should see “Data source connected and labels found”

Explore View

The Explore view is your primary interface for ad-hoc log investigation. Select the Loki datasource and use the query builder or switch to raw LogQL mode.

Key features in Explore:

• Live tailing — click the “Live” button to stream logs in real time, like tail -f in a browser
• Derived fields — clickable links extracted from log content (trace IDs, request IDs)
• Time range picker — zoom into any window to investigate an incident
• Log context — click any log line to see lines immediately before and after it

Dashboard Panels for Logs

Logs panel — Displays a scrollable, filterable list of log lines. Use it to surface the most important log streams on an operational dashboard. In panel JSON:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

{
  "type": "logs",
  "options": {
    "showTime": true,
    "showLabels": false,
    "showCommonLabels": true,
    "wrapLogMessage": true,
    "sortOrder": "Descending",
    "dedupStrategy": "none"
  }
}

Stat panel with log metrics — Use a LogQL metric query to drive a stat or time series panel. For example, total error count in the last hour:

sum(count_over_time({job="app"} |= "ERROR" [1h]))

Dashboard Variables with Label Values

Use Loki label values as dashboard template variables to create filterable dashboards:

1. In dashboard settings, add a variable:
   • Type: Query
   • Data source: Loki
   • Query: label_values(job) — populates the list from all available job label values
2. Reference the variable in your queries:

{job=~"$job"} |= "ERROR"

Correlating Logs and Metrics in the Same Dashboard

This is where the LGTM stack shines. A well-constructed dashboard has:

• Row 1: Prometheus metrics — request rate, error rate, latency (time series panels)
• Row 2: Loki log metrics — error count, log volume by level (stat + bar chart panels)
• Row 3: Loki logs panel — the actual log lines for the selected service and time range

When an alert fires on the Prometheus error rate, you scroll down to the logs panel — same time range, same service — and the relevant log lines are right there. No context switching, no SSH.

Time range synchronization: All panels in a Grafana dashboard share the global time range by default. When you zoom into the spike on the error rate graph, the logs panel zooms to the same window automatically.

Derived Fields — Trace ID Links to Tempo

If your application logs include a trace ID, you can configure Loki to turn that ID into a clickable link to Tempo:

In the Loki datasource settings, under Derived Fields:

Name: TraceID
Regex: "trace_id":"(\w+)"
Query: ${__value.raw}
URL: (select your Tempo datasource)

Now when a log line contains "trace_id":"abc123", the trace ID is highlighted and clicking it opens the corresponding trace in Tempo. This is the logs-to-traces correlation that makes the full LGTM stack valuable.

Part 9: Alerting with Loki Ruler

Loki’s Ruler component evaluates LogQL expressions on a schedule and fires alerts to Alertmanager — the exact same Alertmanager you use for Prometheus alerts.

Ruler Configuration

The ruler section is already included in the loki-config.yaml from Part 3. Create an alert rules directory:

1

mkdir -p /path/to/loki-stack/loki-rules/fake

The fake subdirectory name corresponds to the tenant ID (auth_enabled: false uses the tenant fake).

Alert Rules File

Create /path/to/loki-stack/loki-rules/fake/alert-rules.yaml:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106

groups:
  - name: system.alerts
    interval: 1m
    rules:

      # OOM Killer triggered
      - alert: OOMKillerTriggered
        expr: |
          sum(count_over_time(
            {job="syslog"} |~ "oom_kill_process|Out of memory: Kill process" [5m]
          )) > 0
        for: 0m
        labels:
          severity: critical
          team: ops
        annotations:
          summary: "OOM killer triggered on {{ $labels.host }}"
          description: >
            The Linux OOM killer has terminated a process on {{ $labels.host }}.
            Check memory usage immediately. View logs: {job="syslog",host="{{ $labels.host }}"} |~ "oom_kill"

      # SSH root login attempt
      - alert: SSHRootLoginDetected
        expr: |
          sum(count_over_time(
            {job="auth"} |~ "Accepted .* for root from" [5m]
          )) > 0
        for: 0m
        labels:
          severity: critical
          team: security
        annotations:
          summary: "SSH root login on {{ $labels.host }}"
          description: "A successful SSH login as root was detected on {{ $labels.host }}. Investigate immediately."

      # SSH brute force — many failed attempts in a short window
      - alert: SSHBruteForceDetected
        expr: |
          sum by (host) (
            count_over_time(
              {job="auth"} |= "Failed password" [5m]
            )
          ) > 20
        for: 2m
        labels:
          severity: warning
          team: security
        annotations:
          summary: "SSH brute force on {{ $labels.host }}"
          description: "More than 20 failed SSH password attempts in 5 minutes on {{ $labels.host }}."

      # Application error rate spike
      - alert: AppErrorRateHigh
        expr: |
          sum by (job) (
            rate({job=~"app|backend|api"} |= "ERROR" [5m])
          ) > 0.5
        for: 5m
        labels:
          severity: warning
          team: dev
        annotations:
          summary: "High error rate in {{ $labels.job }}"
          description: "Error rate exceeds 0.5 errors/sec in {{ $labels.job }} for the past 5 minutes."

      # Disk full warning in logs
      - alert: DiskFullWarningInLogs
        expr: |
          sum(count_over_time(
            {job="syslog"} |~ "No space left on device|disk full|ENOSPC" [10m]
          )) > 0
        for: 0m
        labels:
          severity: warning
          team: ops
        annotations:
          summary: "Disk full error on {{ $labels.host }}"
          description: "Log messages indicate a disk-full condition on {{ $labels.host }}."

      # Nginx 5xx error spike
      - alert: NginxHighErrorRate
        expr: |
          sum(rate({job="nginx"} | json | status >= 500 [5m])) > 1
        for: 3m
        labels:
          severity: warning
          team: ops
        annotations:
          summary: "Nginx 5xx error rate above threshold"
          description: "More than 1 HTTP 5xx error per second from nginx for the past 3 minutes."

      # Container crash loop (frequent restarts appear in logs)
      - alert: ContainerCrashLoop
        expr: |
          sum by (container) (
            count_over_time(
              {job="syslog"} |~ "container .* exited with code [^0]" [10m]
            )
          ) > 3
        for: 0m
        labels:
          severity: warning
          team: ops
        annotations:
          summary: "Container crash loop detected for {{ $labels.container }}"
          description: "Container {{ $labels.container }} has exited with a non-zero exit code more than 3 times in 10 minutes."

Mount this directory into your Loki container by adding to the volumes in docker-compose.yml:

1
2
3
4
5

  loki:
    volumes:
      - ./loki-config.yaml:/etc/loki/config.yaml:ro
      - loki_data:/loki
      - ./loki-rules:/loki/rules:ro  # Add this line

Verify that Loki is loading your rules:

1

curl http://localhost:3100/loki/api/v1/rules

Alerts flow from Loki Ruler → Alertmanager → your notification channels (Slack, PagerDuty, email) — the same path as Prometheus alerts. If you have Alertmanager already configured for Prometheus, your routing rules and receivers apply to Loki alerts too.

Part 10: Label Strategy and Performance

Label design has a bigger impact on Loki’s performance and cost than almost any other configuration decision.

High Cardinality: The Core Problem

Each unique combination of label values creates a new log stream, which requires its own index entry and chunk storage. If you put a high-cardinality value like a user ID or request ID into a label, you will create millions of tiny streams — each with a handful of log lines. Loki performs very poorly in this situation because it must open and scan enormous numbers of tiny chunks for every query.

Good labels (low cardinality — a handful of distinct values):

• host — server hostname (a dozen servers = a dozen values)
• job — service or component name
• container — container name
• env — prod, staging, dev
• level — info, warn, error, debug
• namespace — Kubernetes namespace
• status — HTTP status code class (200, 404, 500)

Bad labels (high cardinality — unbounded unique values):

• user_id — potentially millions of distinct values
• request_id / trace_id — unique per request, never repeat
• ip_address — unbounded
• session_id — unique per session
• timestamp as a label — each nanosecond is a new value

Put high-cardinality values in the log line content, not in labels. Extract them at query time with | json, | regexp, or | pattern.

limits_config Tuning

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

limits_config:
  # Per-user (per-tenant) ingestion limits
  ingestion_rate_mb: 16          # Max MB/s per tenant
  ingestion_burst_size_mb: 32    # Burst allowance above the rate limit
  max_streams_per_user: 10000    # Cap on active stream count — increase if you see "too many streams" errors
  max_line_size: 256kb           # Reject individual log lines larger than this

  # Per-stream retention (override global default)
  # Requires retention_enabled: true in compactor
  retention_period: 30d

  # Per-tenant retention override example
  # Put this in a per-tenant override section if needed:
  # per_tenant_override_config: /etc/loki/overrides.yaml

Retention Per Stream

You can set different retention periods for different streams using per-tenant overrides or by configuring the compactor with stream selectors. For homelab, global retention is usually sufficient. For production, you might want:

• Security/auth logs: 90 days
• Debug logs: 7 days
• Application error logs: 30 days

Index and Cache Performance

For a single-node deployment with moderate volume, the embedded cache in query_range.results_cache is sufficient. For higher query loads, external Memcached significantly reduces query latency for repeated time range queries:

1
2
3
4
5
6
7

query_range:
  results_cache:
    cache:
      memcached_client:
        host: memcached:11211
        service: memcache
        timeout: 500ms

Stream Sharding for High-Volume Sources

If a single log source generates more than ~5MB/s, a single stream becomes a bottleneck because Loki serializes writes to each stream. Add an artificial sharding label to distribute the load:

In Promtail’s relabel config, add a label that cycles through a small set of values:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

relabel_configs:
  - source_labels: [__journal_systemd_unit]
    target_label: job
  # Shard high-volume streams across 4 buckets based on hash of source labels
  - action: hashmod
    source_labels: [job, host]
    modulus: 4
    target_label: __shard__
  - source_labels: [__shard__]
    target_label: shard

Part 11: Practical Homelab Use Cases

Centralizing Logs from Multiple Docker Hosts

Run Promtail on each host, shipping to a central Loki instance. Add a host label with the machine’s hostname:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# In promtail-config.yaml on each host:
clients:
  - url: http://loki.internal:3100/loki/api/v1/push

scrape_configs:
  - job_name: docker
    docker_sd_configs:
      - host: unix:///run/docker.sock
    relabel_configs:
      - replacement: myserver01  # or use __HOSTNAME__
        target_label: host

Query all hosts at once:

{job="nginx"} |= "ERROR"

Or filter to a specific host:

{job="nginx", host="myserver01"} |= "ERROR"

Auth Log Monitoring for Intrusion Detection

A Grafana dashboard row showing:

1. Stat panel — failed SSH attempts in last 24h: sum(count_over_time({job="auth"} |= "Failed password" [24h]))
2. Time series — failed attempts over time: sum(rate({job="auth"} |= "Failed password" [5m]))
3. Logs panel — raw auth log lines: {job="auth"} |~ "Failed|Invalid|error"
4. Table — top attacking IPs (using LogQL + regex to extract IP, then topk)

K3s / Kubernetes Pod Logs

On each K3s node, run Promtail or Alloy targeting /var/log/pods. Labels are extracted from the pod log path:

# All logs from the production namespace
{namespace="production"}

# Logs from a specific pod
{namespace="production", pod_name=~"myapp-.*"}

# Errors from all containers in a deployment
{namespace="production", container_name="myapp"} |= "error"

The DaemonSet Helm chart for Promtail handles all of this automatically on Kubernetes:

1
2
3
4

helm repo add grafana https://grafana.github.io/helm-charts
helm install promtail grafana/promtail \
  --set config.clients[0].url=http://loki:3100/loki/api/v1/push \
  --namespace monitoring

Nginx/Traefik Access Log Analysis

With nginx logs parsed and a status label extracted:

# Error rate by status code
sum by (status) (rate({job="nginx"} | json [5m]))

# Top 404 paths (content that doesn't exist)
topk(10, sum by (path) (
  count_over_time({job="nginx"} | json | status = 404 [1h])
))

# Bandwidth by path
sum by (path) (
  bytes_over_time({job="nginx"} | json [1h])
)

Systemd Journal Forwarding

Promtail can read the systemd journal directly without needing log files:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

scrape_configs:
  - job_name: journal
    journal:
      max_age: 12h
      labels:
        job: journal
        host: __HOSTNAME__
    relabel_configs:
      # Add systemd unit as a label
      - source_labels: [__journal__systemd_unit]
        target_label: unit
      # Add syslog identifier (process name)
      - source_labels: [__journal_syslog_identifier]
        target_label: syslog_identifier

This captures everything journald sees — kernel messages, systemd unit output, any application using the journal API — without needing any log files on disk.

Putting It All Together

The Loki stack gives you the third leg of the observability triangle alongside Prometheus metrics and (optionally) Tempo traces. The value compounds as you add more sources: once you have a Grafana dashboard with CPU/memory graphs from Prometheus and log panels from Loki both responding to the same dashboard variables, incident investigation becomes qualitatively different. You stop jumping between terminals and start reasoning about systems from a single view.

Start with the Docker Compose stack from Part 3, get your Docker container logs flowing, then layer in system logs and alerting rules. Once you have those working and you understand what your log volume actually looks like, tune your label strategy and retention settings. The default configuration will handle years of homelab operation without changes.

For the full picture, add Tempo for distributed traces and wire up the derived fields so trace IDs in your logs link directly to trace waterfall views. At that point you have the complete LGTM stack and the investigative capability that previously required enterprise tooling and a dedicated SRE team — running at home on hardware you already own.

Quick Reference

Essential LogQL Patterns

# All errors, any service
{job=~".+"} |= "ERROR" | json

# Error rate for alerting
sum by (job) (rate({job=~".+"} |= "ERROR" [5m]))

# SSH failures in last 10 minutes
{job="auth"} |= "Failed password" | regexp `from (?P<ip>\S+) port`

# Nginx 5xx last hour
{job="nginx"} | json | status >= 500

# Container logs by name
{container="traefik"} |= "level=error"

# OOM events
{job="syslog"} |~ "oom_kill|Out of memory"

# K8s pod logs
{namespace="prod", pod_name=~"api-.*"} | json | level = "error"

Loki API Endpoints

logcli Quick Reference

1
2
3
4
5
6
7

export LOKI_ADDR=http://localhost:3100

logcli labels                           # List all label names
logcli labels job                       # List values for 'job' label
logcli query '{job="nginx"}' --since=1h # Query last hour
logcli query '{job="auth"}' --tail      # Live tail
logcli query --limit=100 '{job="app"} |= "ERROR"' --output=raw