Modern Logging Architecture: Loki, Splunk, Elasticsearch, ClickHouse, and the Cost-Per-GB That Decides Everything

At homelab scale, every logging backend works and the choice is aesthetic. At production scale, the choice is financial, and the feature comparison everyone starts with turns out to be the wrong document entirely. The number that actually governs a logging platform is cost-per-gigabyte at your ingest volume and retention, and that number is not a pricing-page line item — it is a consequence of an architectural bet each backend made about how to store and index a log line. Splunk and Elasticsearch bet on indexing everything, which makes any search instant and makes storage and ingest expensive. Loki bet on indexing almost nothing, which makes storage cheap and full-text search a brute-force scan. ClickHouse bet on columns, which makes aggregation and compression extraordinary and makes you design a schema. None of these is wrong; they are optimized for different query patterns and different budgets, and choosing the one that mismatches your workload is how teams end up either with a six-figure logging bill or a system that cannot answer the question they bought it to answer. And underneath all of it sits a lever more powerful than any backend choice: the cheapest, fastest log to store and search is the one you never ingested.

The Economics Nobody Reads Until the Invoice

Logging cost is dominated by two variables: how many gigabytes per day you ingest, and how long you keep them. Query cost matters, but it is a rounding error next to ingest and retention, because you pay to write and store every line whether or not anyone ever reads it — and the brutal truth of production logging is that almost nobody ever reads almost any of it. The median log line is written, replicated, indexed, compressed, stored, tiered, and finally expired, having been queried exactly zero times.

This is why the historically traumatic logging bills came from Splunk’s per-GB-ingested licensing: it priced the exact thing teams were worst at controlling. You did not pay for value extracted; you paid for volume produced, and volume produced grows with every new service, every added debug line, every chatty access log, with no natural ceiling. The same dynamic exists on every backend, just expressed differently — Elasticsearch charges you in storage and CPU for the index it builds over that volume, and even the cheap backends charge you in object-storage bytes. The first principle of modern logging architecture is therefore not “which database” but “how do I ingest less,” and any conversation that starts with the backend has already skipped the most important decision.

Four Storage Bets

Everything downstream — cost, query latency, which queries are even possible — follows from how the backend stores and indexes a line. There are three distinct bets, and four major products spread across them.

The inverted-index camp — Splunk and Elasticsearch — builds, for every field and often every token, a map from value to the documents that contain it. That is what makes “find the one request with this trace ID across a billion logs” return instantly without knowing where to look. The cost is that the index is enormous, frequently rivaling or exceeding the size of the raw data, and building it consumes CPU on every ingested line. You are paying, continuously, for the ability to search any field instantly, whether or not you ever search most of them.

Loki made the opposite bet: index only a small set of labels (service, environment, level) and store the actual log lines as compressed chunks in object storage, unindexed. Storage becomes cheap — object storage is cheap and the chunks compress well — and ingest becomes cheap because there is almost no index to build. The cost surfaces at query time: searching log content is a scan over the chunks selected by your labels, so a query is only as fast as the label selection is narrow.

ClickHouse made a third bet entirely: store each field as a compressed column, sorted by a primary key, with no inverted index by default. Columnar storage compresses logs ferociously — ten to thirty times is routine — and aggregations over columns are blisteringly fast, but there is no free per-field lookup, so content search is a scan accelerated by the primary-key sort order and optional skip indexes. You get the lowest cost-per-GB at high volume and the best analytics, in exchange for owning a schema and a database.

Loki: Index the Labels, Grep the Rest

Loki is “Prometheus for logs,” and the analogy is load-bearing. A log stream is identified by a set of labels, exactly like a metric series, and only those labels are indexed. The log lines themselves are batched into chunks, compressed, and written to object storage (S3, GCS, or a local equivalent), with a small TSDB index mapping label sets to chunk locations. LogQL queries first select streams by label, then optionally filter line content with a scan over the matched chunks. The mechanics, shipping, and LogQL details are covered in the Loki log-aggregation guide; the architectural point here is the consequence.

The consequence is that Loki is cheap and that its single failure mode is label cardinality. Every unique combination of label values creates a separate stream, and high-cardinality labels — request_id, user_id, pod hash, anything unbounded — explode the number of streams, bloat the index Loki was supposed to keep tiny, and fragment storage into millions of microscopic chunks that destroy both ingest and query performance. The discipline is absolute: labels are for low-cardinality dimensions you slice by (service, env, level, region), and high-cardinality identifiers live in the log line (or in Loki’s structured-metadata feature), never as labels.

1
2
3
4
5
6

# promtail/alloy: GOOD label strategy — low cardinality only
- labels:
    service: checkout
    env: prod
    level:                 # parsed from the line, bounded set
# request_id, user_id, trace_id stay in the line body, NOT here

Loki is the right default when you are already in the Grafana/Prometheus world, your dominant query is “show me this service’s logs around this time,” and you are cost-sensitive. It is a poor fit when your dominant query is an unpredictable needle-in-haystack across every field at very large scale, because that is a full scan — the experimental bloom-filter accelerators help, but they are not a substitute for an inverted index when arbitrary instant search is the actual requirement.

ClickHouse: Columnar Logs and the Schema You Must Design

ClickHouse treats logs as what they often are — high-volume, semi-structured events you want to filter, aggregate, and retain cheaply. You store them in a MergeTree table, sorted by a primary key chosen for your access pattern, with per-column compression codecs and a TTL that tiers old data to object storage and eventually deletes it. The ClickHouse-for-observability deep dive covers operations; the schema is where the cost-per-GB is won or lost:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

CREATE TABLE logs
(
    timestamp  DateTime64(3) CODEC(DoubleDelta, ZSTD(1)),
    service    LowCardinality(String),
    level      LowCardinality(String),
    trace_id   String,
    message    String CODEC(ZSTD(3)),
    attributes Map(LowCardinality(String), String)
)
ENGINE = MergeTree
PARTITION BY toDate(timestamp)
ORDER BY (service, level, timestamp)          -- primary key = your filter pattern
TTL toDateTime(timestamp) + INTERVAL 14 DAY TO VOLUME 'cold',
    toDateTime(timestamp) + INTERVAL 90 DAY DELETE
SETTINGS index_granularity = 8192;

-- optional: make content search fast without a full inverted index
ALTER TABLE logs ADD INDEX msg_tokens message TYPE tokenbf_v1(32768, 3, 0) GRANULARITY 4;

LowCardinality dictionary-encodes repetitive fields, DoubleDelta crushes monotonic timestamps, ZSTD handles message text, the ORDER BY gives you a sparse primary index that turns time-and-service queries into narrow range scans, and the tokenbf_v1 skip index lets content searches prune granules instead of scanning everything. Materialized views can pre-aggregate the access-log firehose into per-minute rollups so you keep the counts forever and the raw lines for two weeks. The payoff is the lowest cost-per-GB available and SQL-grade analytics over your logs. The price is honest: you own the schema, the merges, the cluster, and the learning curve. ClickHouse rewards volume and punishes teams that want turnkey.

Elasticsearch and Splunk: The Index-Everything Camp

There are workloads where instant search over any field genuinely is the requirement, and for those the inverted-index camp earns its cost. Security and SIEM use cases, incident forensics where you cannot predict which field you will need to pivot on, and ad-hoc investigation across heterogeneous sources all want the property that every field is searchable now, without having designed for that query in advance. That is exactly what Elasticsearch and Splunk sell, and it is a real capability the cheap backends do not match.

The way you survive the cost is lifecycle management. Elasticsearch’s Index Lifecycle Management moves indices through hot, warm, cold, and frozen tiers, where frozen uses searchable snapshots that keep data queryable while it lives on cheap object storage rather than fast local disk:

1
2
3
4
5
6
7
8

{
  "policy": { "phases": {
    "hot":    { "actions": { "rollover": { "max_primary_shard_size": "50gb", "max_age": "1d" } } },
    "warm":   { "min_age": "2d",  "actions": { "shrink": { "number_of_shards": 1 }, "forcemerge": { "max_num_segments": 1 } } },
    "cold":   { "min_age": "7d",  "actions": { "searchable_snapshot": { "snapshot_repository": "s3-logs" } } },
    "delete": { "min_age": "30d", "actions": { "delete": {} } }
  }}
}

Splunk gives you the most powerful search-and-correlation language in the category (SPL) and a mature ecosystem, at the highest price in it. A note on the ecosystem split: after Elastic’s 2021 license change, AWS forked the project as OpenSearch, which is API-compatible and the common choice when you want the model without Elastic’s licensing or pricing. The decision to join this camp should be deliberate: you are paying a continuous premium for instant arbitrary search, and it is worth it precisely when that capability is the job and a brute-force scan would be too slow to be useful during an incident.

The Real Lever: Reduce Before You Ingest

Every backend choice is dominated by a discipline that applies to all of them: ship less. The cheapest and fastest log is the one that never entered the pipeline, and a collector tier — Vector, the OpenTelemetry Collector, or Fluent Bit — is where you enforce that before paying any backend to store it.

SOURCES                 COLLECTOR (Vector / OTel)              BACKENDS (by tier)
app / k8s / syslog ---> parse -> drop DEBUG -> redact PII      hot:  Loki / ES   (7-14d)
                        -> route + sample by level               |
                              |                                   v
                              +-- errors:  keep 100% --------> cold: object store / CH TTL
                              +-- access:  keep 1 in 10           archive: S3 (compliance)

The concrete moves, in order of return:

Log structured, not stringly. Emit JSON or logfmt so fields are parsed once at the source, not scraped with fragile regex at query time forever. The observability-by-design patterns cover this at the application layer; structured logs are the precondition for everything below.

Drop and sample at the collector. Production does not need DEBUG. High-volume access and INFO logs can be sampled aggressively while errors are kept in full. In Vector’s remap language:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

[transforms.clean]
type = "remap"
inputs = ["app_logs"]
source = '''
  if .level == "debug" { abort }                       # never ship debug in prod
  .msg = replace(.msg, r'(?i)(bearer\s+)\S+', "$1[REDACTED]")   # redact secrets
  del(.kubernetes.pod_labels)                          # drop a chatty nested field
'''

[transforms.sample]
type = "sample"
inputs = ["clean"]
rate = 10                                               # keep 1 in 10...
exclude = '.level == "error" || .status >= 500'         # ...but keep ALL errors

Control cardinality and route by tier. Keep unbounded identifiers out of indexed labels (the Loki rule, but it saves money everywhere), and route by value: errors to the searchable hot tier at full fidelity, sampled access logs to cheap storage, compliance-mandated audit logs to immutable archive.

Use metrics for counting, logs for context. The single most expensive anti-pattern is logging an event you only ever count. If you want “requests per second by status,” that is a counter, not a billion log lines you later aggregate. Emit the metric, log the exceptions.

Tier retention deliberately. Searchable hot retention of one to two weeks covers the overwhelming majority of real queries; everything older goes to object storage or a ClickHouse cold volume and is deleted on a schedule the moment compliance allows. Paying hot-tier prices for 90-day retention because nobody set a policy is the most common logging-bill autopsy finding.

A team that does these well can switch from Splunk to almost anything and watch the bill fall by an order of magnitude, because the savings came from ingesting a tenth as much, not from the new backend.

Choosing, Honestly

The decision is a function of query pattern, scale, and how much database you want to operate.

If you are already running Grafana and Prometheus, your scale is homelab-to-mid, you are cost-sensitive, and your dominant question is “show me this service’s logs around this incident,” choose Loki. It is cheap, it correlates natively with your metrics and traces, and label discipline is the only skill it demands.

If your volume is large, your logs are structured, you want serious aggregation and the lowest cost-per-GB, and you are willing to own a schema and a cluster, choose ClickHouse. It is the analytics-grade answer and it rewards the investment with bills the other camps cannot touch at scale.

If you need instant arbitrary full-text and field search across everything — SIEM, security, unpredictable forensics — and the budget exists, choose Elasticsearch/OpenSearch or Splunk, and live or die by lifecycle management to keep the index-everything bet affordable.

And whichever you choose, build the collector tier first, because it is the part that determines whether any of them is affordable.

Verdict

The logging-backend decision masquerades as a feature comparison and is actually an economics decision, because cost-per-GB at your volume and retention dominates everything and is fixed by the storage bet each backend made. Splunk and Elasticsearch index everything, which buys instant search over any field and charges a continuous premium in storage and CPU that only the SIEM-grade workloads truly need. Loki indexes only labels and leans on cheap object storage, which is the right cost-conscious default for the Grafana-native team that mostly browses logs by service and time, provided it never lets a high-cardinality field become a label. ClickHouse stores columns and compresses them into the ground, delivering the lowest cost-per-GB and the best analytics at high volume in exchange for the schema and cluster you have to own. There is no universally correct pick; there is the pick that matches your dominant query pattern, your scale, and your appetite for operating a database.

But the backend is the second decision, not the first. The first is to stop ingesting logs nobody will ever read — structure them, drop DEBUG, sample the firehose while keeping every error, redact and route at the collector, count with metrics instead of lines, and tier retention so hot storage holds two weeks and not two quarters. Teams that do this well find the backend choice becomes forgiving and the bill becomes an order of magnitude smaller, because the cheapest, fastest, most private log line in any architecture is still the one you had the discipline never to ship.

Sources

• Grafana Loki documentation
• Loki label best practices and cardinality
• ClickHouse: using ClickHouse for observability and logs
• ClickHouse MergeTree and compression codecs
• Elasticsearch Index Lifecycle Management (ILM)
• Elasticsearch searchable snapshots and the frozen tier
• OpenSearch project
• Splunk: about pricing and ingest-based licensing
• Vector: the observability data pipeline
• OpenTelemetry Collector — processors