Redis Architecture and Persistence

Redis is one of those systems that engineers tend to underestimate until they actually run it in production at scale. On the surface it looks like a fast key-value store — throw a string in, get it back later. In practice, Redis is a data structure server with persistence options, replication topologies, a cluster mode, eviction semantics, and diagnostic tooling sophisticated enough to serve as the backbone for rate limiters, session stores, leaderboards, pub/sub pipelines, and lightweight event streams. The gap between “I spun up Redis in Docker” and “I understand what Redis guarantees and when it will fail me” is significant, and this post intends to close most of it.

As of this writing, Redis Open Source is at version 8.6.x (8.6 introduced major throughput and observability improvements). The Redis licensing change in 2024 — when Redis Ltd. moved to the Business Source License — prompted the Linux Foundation to fork the project under the name Valkey, which has since reached version 9.1 and is now the default redis package in Ubuntu 24.04 and is available in Debian’s testing/unstable channels. Functionally, Valkey 7.2 was a drop-in replacement for Redis 7.2; Valkey 8+ and 9+ have diverged on their own roadmap with multi-threading improvements that deliver over a billion RPS in cluster benchmarks. This post uses “Redis” to refer to both unless a distinction is relevant, and notes where Valkey diverges meaningfully.

What Redis Is and Is Not

Redis describes itself as an “in-memory data structure store.” Every key and value lives in RAM. The persistence layer (covered in depth below) writes snapshots or command logs to disk, but reads always come from RAM. This is the source of Redis’s sub-millisecond latency — there is no disk seek on the hot path.

The critical architectural distinction that drives every configuration decision is whether Redis is acting as a cache or as a database.

When Redis is a cache, it sits in front of a source of truth (PostgreSQL, MySQL, an API). Every value in Redis is derivable from that source. If Redis loses data — due to a crash, a misconfigured eviction policy, or a failover with replication lag — the system keeps working; it just gets slower as it warms up again. For a cache, persistence is optional, aggressive eviction is correct, and a minor data loss on failover is entirely acceptable.

When Redis is a database — the system of record, not a cache — it holds data that exists nowhere else: session tokens, rate limit counters that cannot be recalculated, distributed locks, queued jobs. Here, losing data is an application bug. Persistence must be configured carefully, eviction policy must be noeviction rather than LRU, and failover behavior must be tuned to minimize the replication lag window.

Most production deployments mix both roles in the same Redis instance, which is where configuration goes wrong. The answer is usually to run separate Redis instances with separate persistence and eviction configurations per role, even if they share the same hardware.

The Single-Threaded Event Loop

Redis processes commands in a single thread. There is one event loop. Commands execute sequentially, one at a time, with no locking, no concurrency primitives, and no race conditions inside the command processing path. This sounds like a performance limitation; in practice it is a simplicity advantage. Redis commands are predominantly O(1) or O(log n) against in-memory data structures — the CPU is almost never the bottleneck. Network I/O is.

Redis 6.0 introduced I/O threads (io-threads 4 in redis.conf) which handle reading from and writing to client sockets in parallel, while keeping command execution single-threaded. This dramatically improves throughput on multi-core machines by removing the I/O bottleneck without introducing concurrency bugs in command processing. Valkey 8+ goes further with fully multi-threaded command execution on an opt-in basis, which is one of its main differentiators from Redis.

The important implication of single-threaded command processing is that any O(N) command on a large dataset blocks all other clients for its duration. KEYS * on an instance with 10 million keys might take 200ms. During those 200ms, every other client is waiting. This is not theoretical — KEYS * in a production monitoring script has caused outages. Use SCAN instead.

Data Structures and Use Cases

Redis is not a key-value store in the narrow sense. It is a collection of data structures, each with its own memory representation and operational guarantees. Choosing the right structure is the difference between an elegant solution and a Lua script nightmare.

String

The fundamental type. A Redis string is a binary-safe byte sequence up to 512MB. Despite the name, integers and floats are stored as strings but Redis understands them for atomic arithmetic.

SET session:abc123 '{"user_id":42,"role":"admin"}' EX 3600
GET session:abc123

SET rate:ip:10.0.0.1 0
INCR rate:ip:10.0.0.1          # atomic increment, returns new value
EXPIRE rate:ip:10.0.0.1 60

# Distributed lock: SET only if key does not exist, with TTL
SET lock:resource:42 <uuid> EX 30 NX

The SET key value EX seconds NX pattern is the canonical Redis distributed lock. NX (set if Not eXists) is atomic — there is no window where two callers both succeed. EX ensures the lock expires even if the holder crashes. The UUID value is the lock token; the holder must verify it still owns the lock before releasing (via a Lua script to make read-then-delete atomic).

Use strings for: simple caching of serialized objects, rate limiting counters, session tokens, feature flags, distributed locks, and any counter that needs atomic increment/decrement.

List

A doubly-linked list. O(1) push and pop from both ends. O(N) access by index. Supports blocking pops.

LPUSH jobs:queue '{"id":1,"type":"email"}'   # push to head
RPUSH jobs:queue '{"id":2,"type":"sms"}'     # push to tail
RPOP  jobs:queue                              # pop from tail (FIFO with RPUSH/LPOP)
LRANGE jobs:queue 0 -1                        # all elements (O(N) — use carefully)
BLPOP jobs:queue 0                            # blocking pop, wait indefinitely

BLPOP is a genuinely useful primitive: it blocks the calling client (not the server — other clients are unaffected) until an element is available, then pops and returns it atomically. This enables simple job queue patterns without polling.

Use lists for: job queues, activity feeds (prepend new events, trim with LTRIM), recent items lists, message passing between processes on the same Redis instance.

Set

An unordered collection of unique strings. O(1) membership test. O(N) for full enumeration. Set algebra is built in.

SADD tags:post:42 redis databases caching
SISMEMBER tags:post:42 redis             # 1
SMEMBERS tags:post:42                    # {"redis","databases","caching"}
SUNION tags:post:42 tags:post:43         # union of tags
SINTER followers:user:1 followers:user:2 # mutual followers
SDIFF  followers:user:1 followers:user:2 # unique to user 1

Use sets for: unique visitor tracking (add user IDs, SCARD for count), tagging systems, friend/follower graphs where set operations are useful, real-time membership checks, de-duplication.

Sorted Set (ZSet)

Each member has a floating-point score. Members are ordered by score, ascending, with O(log n) insertion and O(log n) range queries. This is one of Redis’s most powerful data structures.

ZADD leaderboard 9842.5 "player:alice"
ZADD leaderboard 7231.0 "player:bob"
ZINCRBY leaderboard 500.0 "player:alice"   # atomic score increment
ZRANGE leaderboard 0 9 REV WITHSCORES      # top 10, highest score first
ZRANK  leaderboard "player:alice"          # rank (0-indexed)
ZRANGEBYSCORE leaderboard 7000 10000       # all players scoring 7000-10000

# Sliding window rate limiter: score = timestamp in ms
ZADD rate:user:42 1717200000000 "req:uuid1"
ZREMRANGEBYSCORE rate:user:42 0 <(now - window_ms)>
ZCARD rate:user:42                          # requests in the window

The sliding window rate limiter with a sorted set is elegant: add each request with score=timestamp, remove expired entries, count remaining. It is more accurate than a fixed-window counter but more memory-intensive.

Use sorted sets for: leaderboards, priority queues (score = priority), time-series event indexing (score = timestamp), scheduled job execution (score = run-at timestamp), sliding window rate limiting.

Hash

A map of field-value pairs within a single key. More memory-efficient than storing each field as its own top-level key because Redis uses a compact internal encoding (ziplist/listpack) for small hashes.

HSET user:42 name "Alice" email "alice@example.com" plan "pro" login_count 0
HGET user:42 name                          # "Alice"
HMGET user:42 name email                   # ["Alice","alice@example.com"]
HGETALL user:42                            # all fields and values
HINCRBY user:42 login_count 1              # atomic field increment
HDEL user:42 plan

Use hashes for: user profiles, configuration objects, session data with multiple fields, any object with well-defined fields that you need to access individually (avoids deserializing an entire JSON blob to update one field).

The memory efficiency note is real: a hash with 100 fields uses significantly less memory than 100 separate string keys with a common prefix, because each top-level key has a fixed overhead (~64 bytes plus the key string), while hash fields share the key overhead.

Stream

An append-only log with consumer group semantics. Added in Redis 5.0. The closest thing Redis has to Kafka topics.

# Producer
XADD events:orders * order_id 12345 user_id 42 amount 99.99
# Returns a stream ID like: 1717200000000-0

# Simple consumer (no group)
XREAD COUNT 10 STREAMS events:orders 0    # read from beginning
XREAD COUNT 10 BLOCK 0 STREAMS events:orders $  # block for new entries

# Consumer group
XGROUP CREATE events:orders processors $ MKSTREAM
XREADGROUP GROUP processors worker-1 COUNT 5 STREAMS events:orders >
XACK events:orders processors 1717200000000-0   # acknowledge processed

# Inspect
XLEN events:orders
XRANGE events:orders - +   # all entries
XPENDING events:orders processors - + 10  # unacknowledged entries

The consumer group model provides at-least-once delivery: entries are delivered to consumers, tracked as pending until acknowledged with XACK, and can be reclaimed if a consumer dies. This is a legitimate alternative to Kafka for low-throughput event pipelines that do not need Kafka’s retention or multi-partition parallelism.

Redis 8.2 added XACKDEL (acknowledge and delete in one command) and XDELEX (delete with consumer group handling) — useful for streams where old data should be discarded immediately after processing.

Use streams for: event sourcing, activity logs, audit trails, lightweight async job processing, sensor data ingestion, replacing Kafka when you do not need multi-partition parallelism or long-term log retention.

HyperLogLog

A probabilistic data structure for cardinality estimation. PFADD adds elements; PFCOUNT returns an approximate distinct count with roughly 0.81% standard error. Uses a fixed 12KB of memory regardless of how many distinct elements have been added — you could track billions of distinct user IDs and it still takes 12KB.

PFADD visitors:2026-05-31 user:101 user:202 user:303
PFADD visitors:2026-05-31 user:101   # duplicate, no effect on count
PFCOUNT visitors:2026-05-31          # ~3 (or 2 or 4 — probabilistic)
PFMERGE visitors:week visitors:2026-05-31 visitors:2026-06-01

Use HyperLogLog for: unique visitor counts, distinct user IDs, any “how many unique X” question where exact precision is not required and memory matters.

Bitmap

Bit operations on a string. Each string can hold up to 2^32 bits. SETBIT key offset 1 and GETBIT key offset. BITCOUNT counts set bits.

# User 12345 was active today
SETBIT active:2026-05-31 12345 1

# How many users were active today?
BITCOUNT active:2026-05-31

# Which users were active both days?
BITOP AND active:both active:2026-05-31 active:2026-06-01

Use bitmaps for: daily active users (one bit per user ID, one key per day), feature flag tracking per user, compact boolean arrays.

Data Structure Reference

Persistence Modes

Persistence is where Redis configuration gets dangerous. The defaults are not safe for all workloads, and the trade-offs between durability, performance, and restart time are non-trivial.

No Persistence

Set save "" in redis.conf to disable all snapshotting and omit appendonly yes. Redis operates as a pure cache: all data is lost on process exit or restart. Zero persistence overhead, maximum throughput.

This is correct when Redis is a cache and your application can tolerate a cold-start period after restart. It is catastrophically wrong when Redis holds data with no other copy.

RDB — Point-in-Time Snapshots

RDB (Redis Database Backup) takes periodic snapshots of the entire dataset. The canonical config:

save 900 1        # snapshot if >= 1 key changed in last 15 minutes
save 300 10       # snapshot if >= 10 keys changed in last 5 minutes
save 60 10000     # snapshot if >= 10000 keys changed in last 60 seconds
dbfilename dump.rdb
dir /var/lib/redis

BGSAVE triggers a manual snapshot. The mechanism is a Unix fork(): Redis forks a child process, the child writes the full dataset to a new dump.rdb file, and the parent continues serving requests. When the child finishes, the new file atomically replaces the old one.

The fork cost. On a 50GB Redis instance, fork() must copy the page table — the kernel data structure that maps virtual addresses to physical pages. On modern Linux with huge pages disabled, this can take hundreds of milliseconds. During the fork, the parent stalls. Redis 7+ uses copy-on-write aggressively, but the page table copy itself is unavoidable. Transparent huge pages (THP) make this worse, not better — THP increases page granularity, increasing page table size, increasing fork latency. The Redis documentation explicitly recommends disabling THP:

1

echo never > /sys/kernel/mm/transparent_hugepage/enabled

RDB advantages: compact binary file, fast to restore (binary format, no command replay), minimal runtime overhead between snapshots, easy to ship offsite. RDB disadvantages: up to the full snapshot interval of data loss on crash (typically 60-900 seconds), fork latency spikes, not suitable as sole persistence for a Redis-as-database use case.

AOF — Append-Only File

AOF logs every write command to an append-only file. On restart, Redis replays the file to reconstruct state.

appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec   # options: always | everysec | no

The appendfsync setting controls the fsync call:

• always: fsync after every command. Maximum durability — at most one command lost on crash. Throughput penalty is severe: every write waits for a disk fsync, typically 1-10ms. Appropriate only for Redis-as-database with very low write rates.
• everysec: fsync every second in a background thread. At most ~1 second of data loss. This is the right default for most production use cases — good durability, minimal latency impact.
• no: never call fsync, let the OS flush when it wants (typically every 30 seconds on Linux). Maximum write throughput, data loss risk up to the OS flush interval.

AOF rewrite. AOF grows without bound as commands accumulate. A key set to “foo” then “bar” then deleted appears as three lines, but the final state is “key does not exist.” Redis periodically rewrites the AOF to produce the minimal representation of current state:

auto-aof-rewrite-percentage 100    # rewrite when AOF doubles in size
auto-aof-rewrite-min-size 64mb     # but only if it's at least this big

The rewrite process:

Parent Process                     Child Process
      |                                  |
      |-- fork() -------------------->   |
      |                           Reads current dataset
      |                           Writes new AOF (minimal commands)
      |                                  |
      | (continues serving writes)       |
      | New writes go to:                |
      |   1. Old AOF file (safety)       |
      |   2. In-memory rewrite buffer    |
      |                                  |
      |             <-- signals done --  |
      |                                  |
      | Appends rewrite buffer to        |
      | child's new AOF file             |
      |                                  |
      | Atomically renames new AOF       |
      | over old AOF                     |
      |                                  |
      v                                  v
New AOF is now the active file    Child exits

The rewrite buffer is critical: writes that arrive while the child is working are buffered in memory so they can be appended to the new AOF before it goes live. The old AOF continues receiving writes as a safety fallback.

BGREWRITEAOF triggers a manual rewrite.

Hybrid Persistence

Since Redis 4.0, the aof-use-rdb-preamble yes option (enabled by default) makes AOF rewrites write an RDB snapshot at the front of the AOF file, followed by incremental AOF commands since the snapshot. On restart, Redis loads the fast RDB preamble, then replays only the incremental commands — dramatically faster restart than pure AOF, with the durability of AOF.

This is the recommended configuration for most production Redis-as-database deployments.

appendonly yes
aof-use-rdb-preamble yes
appendfsync everysec

Persistence Decision Table

Eviction Policies

When maxmemory is set and Redis reaches that limit, new writes must either fail or evict existing data. The right policy depends entirely on whether Redis is a cache or a database.

maxmemory 4gb
maxmemory-policy allkeys-lru
maxmemory-samples 10

Policy Reference

LRU vs LFU. LRU evicts the key that was accessed least recently. If you have a key accessed once per day and another accessed millions of times but not in the last hour, LRU evicts the frequently-accessed key. LFU (available since Redis 4.0) tracks access frequency with an exponential decay model — it evicts the key that is accessed least often over time, not just most recently. For workloads with Zipf-distributed access patterns (which describes most real applications: a small number of hot keys, a large number of cold keys), LFU makes better eviction decisions.

maxmemory-samples controls how many keys Redis examines when choosing what to evict. The default is 5 — Redis samples 5 keys and evicts the least recent/frequent among them. This is approximate LRU/LFU rather than exact (exact would require a sorted list of all keys by access time, which is expensive). Increase to 10 for better accuracy at a modest CPU cost.

Redis LFU can also feed the --hotkeys diagnostic:

redis-cli --hotkeys   # requires maxmemory-policy to be an LFU policy

Redis Sentinel — High Availability

A single Redis instance is a single point of failure. Redis Sentinel provides automatic failover without requiring Redis Cluster’s operational complexity.

What Sentinel Does

Sentinel is a separate process (or set of processes) that:

1. Monitors a Redis primary and its replicas
2. Detects primary failure using a quorum vote
3. Elects a new primary from the replica pool
4. Reconfigures replicas to follow the new primary
5. Provides service discovery — clients ask Sentinel “who is the current primary?” rather than hardcoding an IP

Topology

                    +-------------+
                    |  Sentinel 1 |
                    |  (port 26379)|
                    +------+------+
                           |
          +----------------+----------------+
          |                                 |
  +-------+------+                 +--------+-----+
  |  Sentinel 2  |                 |  Sentinel 3  |
  |  (port 26379)|                 |  (port 26379)|
  +-------+------+                 +--------+-----+
          |                                 |
          +----------------+----------------+
                           |
                    +------+-------+
                    |   PRIMARY    |   <--- clients write here
                    |  10.0.0.1   |
                    |  port 6379  |
                    +------+-------+
                     /           \
          +---------+--+       +--+---------+
          |  Replica 1  |       |  Replica 2 |
          | 10.0.0.2    |       | 10.0.0.3   |
          | port 6379   |       | port 6379  |
          +-------------+       +------------+

Minimum deployment: 3 Sentinel instances. With 3 Sentinels, quorum of 2 means a majority is still reachable during a single Sentinel failure or network partition. Running only 2 Sentinels provides no fault tolerance — both must agree, so any partition prevents failover.

Sentinels can run on the same machines as Redis (common in small deployments) or on dedicated hosts.

Configuration

# /etc/redis/sentinel.conf

# Monitor "mymaster" at 10.0.0.1:6379, quorum = 2
sentinel monitor mymaster 10.0.0.1 6379 2

# Declare primary subjectively down after 5s of no response
sentinel down-after-milliseconds mymaster 5000

# Allow 60s for failover to complete
sentinel failover-timeout mymaster 60000

# Reconfigure up to 1 replica at a time (avoids all replicas syncing simultaneously)
sentinel parallel-syncs mymaster 1

# Optional: run notification script on events
# sentinel notification-script mymaster /var/redis/notify.sh

Start Sentinel with: redis-server /etc/redis/sentinel.conf --sentinel

Failover Mechanics

1. A Sentinel observes that the primary is not responding for down-after-milliseconds. It marks the primary as subjectively down (SDOWN).
2. The Sentinel asks other Sentinels if they also see the primary as down. If a quorum of Sentinels agree, the primary is declared objectively down (ODOWN).
3. Sentinels elect a leader among themselves using a Raft-like vote (each Sentinel can vote once per epoch).
4. The leader selects the best replica to promote — selection criteria: replication offset (how up-to-date), slave-priority config value, run ID as tiebreaker.
5. The leader sends SLAVEOF NO ONE to the chosen replica, promoting it to primary.
6. The leader reconfigures other replicas with SLAVEOF <new-primary-ip> <port>.
7. The old primary, if it recovers, is reconfigured as a replica.
8. Sentinel publishes +switch-master on its pub/sub channel. Clients subscribed to this channel receive the new primary address.

Client Integration

Clients must implement the Sentinel protocol rather than connecting directly to the primary IP. At startup, the client connects to any Sentinel and asks for the current primary:

redis-cli -p 26379 SENTINEL get-master-addr-by-name mymaster
# Returns: ["10.0.0.1","6379"]

redis-cli -p 26379 SENTINEL masters
# Shows all monitored primaries and their state

redis-cli -p 26379 SENTINEL replicas mymaster
# Shows all replicas

All major Redis client libraries (redis-py, Jedis, ioredis, go-redis, StackExchange.Redis) have Sentinel support. Sentinel is not a proxy — the client uses Sentinel for discovery and then connects directly to the primary.

Sentinel Limitations

Sentinel does not prevent data loss during failover. Because Redis replication is asynchronous, the new primary may be missing the last few writes the old primary acknowledged before it crashed. The min-replicas-to-write and min-replicas-max-lag options can reduce this window by requiring writes to be confirmed by at least N replicas within M seconds before the primary acknowledges them — at the cost of write latency and availability.

Split-brain is possible: if a network partition isolates a minority of Sentinels that believe they can reach the primary, they will not trigger failover. The majority partition will. Once the partition heals, the old primary becomes a replica and discards any writes it took while isolated. Use min-replicas-to-write 1 to prevent a partitioned primary from accepting writes when it cannot reach any replica.

Redis Cluster — Horizontal Scaling

Sentinel gives you high availability for a single dataset. Redis Cluster solves the next problem: what happens when your dataset exceeds the RAM of a single machine, or your write rate exceeds what a single CPU thread can handle?

Hash Slots

Redis Cluster divides the keyspace into 16384 hash slots. Each key is assigned to a slot by CRC16(key) % 16384. Hash slots are assigned to primary nodes — each primary owns a contiguous (or non-contiguous) range of slots.

# Slot distribution for a 3-node cluster (approximately):
Node A: slots 0 - 5460       (~5461 slots)
Node B: slots 5461 - 10922   (~5462 slots)
Node C: slots 10923 - 16383  (~5461 slots)

Minimum viable cluster: 3 primaries + 3 replicas (6 nodes total). With only 1 primary per shard, a single node failure takes down 1/3 of the keyspace. The standard recommendation is 1 replica per primary minimum.

Cluster Topology

  Clients
    |
    +-- smart client (handles MOVED/ASK redirects)
    |
    +--------------------+--------------------+
    |                    |                    |
+---+------+        +----+-----+        +-----+----+
| Primary A |        | Primary B |        | Primary C |
| slots 0-5460|      |slots 5461 |        |slots 10923|
|             |      |   -10922  |        |  -16383   |
+---+------+  |      +----+-----+        +-----+----+
    |          |           |                    |
+---+------+   |      +----+-----+        +-----+----+
| Replica A |  |      | Replica B |        | Replica C |
| (hot standby)|      |(hot standby)       |(hot standby)
+------------+        +-----------+        +-----------+

Bootstrapping a Cluster

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Create a 6-node cluster (3 primaries, 3 replicas)
redis-cli --cluster create \
  10.0.0.1:6379 10.0.0.2:6379 10.0.0.3:6379 \
  10.0.0.4:6379 10.0.0.5:6379 10.0.0.6:6379 \
  --cluster-replicas 1

# Check cluster state
redis-cli -c -h 10.0.0.1 CLUSTER INFO

# Show all nodes with slot assignments
redis-cli -c -h 10.0.0.1 CLUSTER NODES

CLUSTER INFO output you should understand:

cluster_enabled:1
cluster_state:ok                 # "fail" if too many nodes are down
cluster_slots_assigned:16384     # all slots must be assigned
cluster_slots_ok:16384
cluster_known_nodes:6
cluster_size:3                   # number of primaries

If cluster_state is fail, writes are rejected cluster-wide. This happens if any slot has no reachable primary — the cluster refuses to serve partial data rather than silently returning wrong results.

Resharding

Adding a node and redistributing slots is done online — the cluster keeps serving traffic while keys migrate:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# Add a new primary node
redis-cli --cluster add-node 10.0.0.7:6379 10.0.0.1:6379

# Reshard: move 4096 slots from existing nodes to the new node
redis-cli --cluster reshard 10.0.0.1:6379 \
  --cluster-from all \
  --cluster-to <new-node-id> \
  --cluster-slots 4096 \
  --cluster-yes

# Rebalance evenly across all primaries
redis-cli --cluster rebalance 10.0.0.1:6379

During resharding, a slot’s keys migrate from source to destination one at a time. The slot is in migrating state on the source and importing state on the destination. Clients receive MOVED redirects for keys already migrated and ASK redirects (temporary, single-command) for keys mid-migration. Smart clients handle both transparently; dumb clients need to follow the redirect manually.

Hash Tags and Multi-Key Operations

Redis Cluster cannot execute multi-key commands (MSET, SUNION, EVAL) across keys in different slots — those operations would require cross-node coordination, which breaks the cluster’s performance model.

The solution is hash tags: if a key contains {...}, only the content inside the braces is used for slot assignment.

# These two keys hash to the same slot because {user:42} is identical
SET {user:42}:profile '{"name":"Alice"}'
SET {user:42}:session 'abc123'

# Now you can run multi-key operations
MGET {user:42}:profile {user:42}:session

# Lua script touching both keys works because they're on the same node
EVAL "return redis.call('GET', KEYS[1])" 2 {user:42}:profile {user:42}:session

Use hash tags deliberately. Overusing them (e.g., using the same tag for everything) defeats sharding by concentrating all keys on one node.

Cluster Limitations

• No cross-slot transactions: MULTI/EXEC transactions can only span keys in the same slot. Lua scripts similarly must declare all keys upfront and they must share a slot.
• No SELECT: Redis Cluster only supports database 0. The SELECT n command is disabled. Logical separation must use key prefixes.
• All nodes must be reachable: Cluster requires a majority of nodes per shard to be reachable. If more than half the primaries fail simultaneously without replicas taking over, the cluster enters fail state.
• Operational complexity: managing a 6+ node cluster, handling node failures, resharding, and monitoring slot coverage is significantly more complex than Sentinel. Do not adopt Cluster until you genuinely need horizontal scale. A single well-tuned Redis instance on modern hardware with large RAM can handle very high throughput — Cluster is not the first answer to a performance problem.

Diagnostic Commands

Production Redis problems usually fall into one of three categories: high latency, memory pressure, or slow commands. Redis has excellent built-in tooling for all three.

Latency Diagnostics

1
2
3
4
5
6
7
8

# Measure round-trip latency (runs continuously, ctrl+C to stop)
redis-cli --latency -h redis-host -p 6379

# Sample latency over time (shows min/max/avg per interval)
redis-cli --latency-history -h redis-host -p 6379

# Intrinsic latency (kernel-level, not Redis) — run on Redis server itself
redis-cli --intrinsic-latency 10

The gap between --intrinsic-latency (kernel scheduler jitter) and --latency (round-trip to Redis) indicates Redis-specific latency. If both are high, you have a kernel/VM issue. If only --latency is high, the problem is in Redis or the network.

Enable latency monitoring in redis.conf:

latency-monitor-threshold 25    # track events > 25ms
latency-tracking yes            # Redis 7.0+

LATENCY LATEST          # most recent latency events by type
LATENCY HISTORY event   # full history for a specific event type
LATENCY RESET           # clear latency history

Memory Diagnostics

1
2
3
4
5
6
7
8

# Scan for large keys (samples the keyspace — can be slow)
redis-cli --bigkeys -h redis-host

# Per-key memory usage (Redis 7.0+, samples keyspace)
redis-cli --memkeys -h redis-host

# Most-accessed keys (requires LFU eviction policy)
redis-cli --hotkeys -h redis-host

Inside redis-cli:

MEMORY USAGE mykey               # bytes used by this key including overhead
MEMORY USAGE mykey SAMPLES 0    # exact count (slower for nested structures)
MEMORY DOCTOR                   # Redis self-diagnosis: fragmentation, keys near expiry, etc.

INFO memory is the most important single command for memory health:

INFO memory
# Key fields:
# used_memory_human: 4.23G       -- what Redis thinks it's using
# used_memory_rss_human: 6.10G   -- what the OS allocated to the Redis process
# mem_fragmentation_ratio: 1.44  -- rss / used_memory; >1.5 is concerning
# rdb_last_bgsave_status: ok
# aof_last_write_status: ok

A mem_fragmentation_ratio above 1.5 indicates significant allocator fragmentation — memory the OS allocated to Redis that Redis’s allocator has not returned to the OS. This typically improves on its own as keys expire and are overwritten, or you can trigger active defragmentation:

CONFIG SET activedefrag yes

Slow Command Diagnostics

SLOWLOG GET 25         # last 25 commands that exceeded slowlog-log-slower-than
SLOWLOG LEN            # number of entries in the slow log
SLOWLOG RESET          # clear the slow log

Configure the threshold:

CONFIG SET slowlog-log-slower-than 10000   # microseconds (10ms)
CONFIG SET slowlog-max-len 256

Each slow log entry includes: a unique ID, timestamp, execution duration in microseconds, the command and arguments, client IP, and client name. This is where you discover that HGETALL on a 50,000-field hash is taking 80ms, or that KEYS * is being called by a monitoring script every 30 seconds.

Real-Time Monitoring

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# Full server stats
redis-cli INFO all

# Just replication status
redis-cli INFO replication

# Connected clients
redis-cli CLIENT LIST

# Real-time command stream — WARNING: significant performance impact
# Each MONITOR call adds a constant overhead proportional to traffic
# Use for seconds to debug, not as continuous monitoring
redis-cli MONITOR

CLIENT LIST shows every connected client with their IP, last command, idle time, and memory usage. Use it to find clients leaking connections or running persistent slow commands.

SCAN is the safe alternative to KEYS * for iterating the keyspace:

# Iterate all keys matching "user:*" in batches of ~100
SCAN 0 MATCH user:* COUNT 100
# Returns [next_cursor, [key1, key2, ...]]
# Continue with returned cursor until cursor == 0

SCAN is O(1) per call (it does not scan all keys at once) and does not block other clients. It may return keys multiple times and may miss keys added/deleted during iteration, but for most operational tasks (finding large keys, auditing TTLs) it is the right tool.

When Redis Is the Wrong Tool

Redis is excellent at what it does, but it is not appropriate for every problem that involves fast data access.

When your working set does not fit in RAM. Redis is in-memory. On cloud instances, RAM is expensive — a 256GB instance costs significantly more than an equivalent disk-backed store. If your dataset is 2TB of rarely-accessed user records and you only need fast access to the hot 5GB, Redis is a poor choice as the primary store. Consider a tiered approach: Redis for the hot tier, with PostgreSQL, ScyllaDB, or DynamoDB as the backing store.

When you need complex queries. Redis has no secondary indexes, no SQL, no aggregation pipeline (beyond what you can implement in Lua or with sorted sets). Every access pattern must be pre-modeled in data structures at write time. If you find yourself maintaining five separate sorted sets to support five query patterns on the same data, or writing 200-line Lua scripts to implement multi-step logic, a document database or a relational database with proper indexing is likely the better fit.

When you need strong consistency. Redis replication is asynchronous by default. A failover (whether Sentinel or Cluster) can lose the last few writes the old primary acknowledged. If you need the guarantee that a committed write will survive any single-node failure, use the WAIT command after writes:

SET account:balance 10000
WAIT 1 500   # wait for >= 1 replica to acknowledge, timeout 500ms

WAIT blocks the client until N replicas acknowledge the write or the timeout expires. This provides synchronous-ish replication at the cost of write latency and reduced availability when replicas lag. If you need true synchronous replication and ACID guarantees, use PostgreSQL.

When persistence is the primary concern. If you are reaching for Redis primarily because you want fast writes with persistence, and data durability is more important than latency, PostgreSQL with synchronous_commit = on or a purpose-built durable store will serve you better. Redis persistence is a feature, not the foundation.

When operational simplicity matters more than performance. Redis Cluster’s operational surface — 6+ nodes, hash slot management, resharding procedures, hash tag discipline, no cross-slot transactions, no SELECT — is substantial. Before adopting Cluster, honestly assess whether a single large Redis instance, a read replica, or a Sentinel setup can handle the load. A single Redis instance on a 384GB machine with NVMe-backed AOF persistence handles extraordinary throughput. Cluster is the right answer when you have genuinely exhausted single-instance capacity, not as an architectural choice made early.

Valkey

For new deployments on Ubuntu 24.04 LTS or Debian testing/unstable, the redis package in official repositories now resolves to Valkey — the Linux Foundation fork created after Redis Ltd. moved to the Business Source License in early 2024. Valkey is wire-compatible with Redis 7.2 for all standard commands and data structures. All major client libraries work against it without changes.

Valkey 9.1 (released May 2026) has diverged meaningfully from Redis with native multi-threaded command execution, hash field expiration (set TTLs on individual hash fields without storing them as separate keys), and substantial throughput improvements — the Valkey team claims 1 billion+ RPS in cluster benchmarks with multi-threading enabled. For most organizations, Valkey is now the better open-source choice: actively maintained by a community-governed foundation, moving faster on performance improvements, and available in standard package repositories without licensing ambiguity.

Redis Open Source 8.6.x remains a strong option, particularly if you use the Redis Stack data types (RedisSearch, RedisJSON, TimeSeries, Bloom filters) that are bundled with Redis but not yet fully replicated in Valkey. The architectural principles in this post apply equally to both.

Production Configuration Reference

A reasonable starting point for a production Redis instance used as a cache:

# /etc/redis/redis.conf

bind 127.0.0.1 ::1
protected-mode yes
port 6379

# Memory
maxmemory 8gb
maxmemory-policy allkeys-lfu
maxmemory-samples 10

# Persistence: RDB snapshots only (cache — AOF not needed)
save 900 1
save 300 10
save 60 10000
dbfilename dump.rdb
dir /var/lib/redis

# Disable AOF for pure cache
appendonly no

# Performance
io-threads 4
io-threads-do-reads yes
tcp-backlog 511
hz 20

# Transparency
latency-monitor-threshold 25
latency-tracking yes
slowlog-log-slower-than 10000
slowlog-max-len 256

# Disable dangerous commands in production
rename-command KEYS ""
rename-command FLUSHALL ""
rename-command FLUSHDB ""
rename-command DEBUG ""

For Redis as a database (system of record):

# Persistence: hybrid AOF+RDB
appendonly yes
aof-use-rdb-preamble yes
appendfsync everysec
no-appendfsync-on-rewrite no

# Never evict — return errors when full
maxmemory-policy noeviction

# Minimize replication lag risk
min-replicas-to-write 1
min-replicas-max-lag 10

# Transparent huge pages: disable at the OS level
# echo never > /sys/kernel/mm/transparent_hugepage/enabled

Redis rewards engineers who understand its data model deeply and punishes those who treat it as a generic fast store. The data structures are expressive enough to model most caching and coordination patterns directly. The persistence layer has real trade-offs that must match your durability requirements. Sentinel and Cluster both work well when deployed correctly and with honest assessment of what each provides. The diagnostic tooling is comprehensive if you know where to look. And for the cases where Redis is not the right answer — complex queries, large cold datasets, strong consistency requirements — there are better tools, and recognizing those boundaries is part of using Redis well.

Sources:

• Releases · redis/redis
• Redis End of Life Dates
• Redis 8.2 Release Notes
• Valkey 2025 Year-End Review
• Valkey Releases
• Valkey is now included in Ubuntu
• Valkey Debian Package Tracker