Most engineers encounter Redis as a cache: store a serialized object, set a TTL, move on. That’s a fine use of Redis, but it’s the equivalent of buying a Swiss Army knife and only using the blade. Redis is a data structure server — not just a key-value store — with native support for lists, sorted sets, streams, pub/sub, geospatial indexes, probabilistic data structures, and with modules, full-text search and JSON documents.
This guide goes well past the cache use case: pub/sub for lightweight messaging, Streams for durable event logs with consumer groups, RedisJSON for document storage, RediSearch for full-text search, and the operational questions — persistence modes, Sentinel for HA, and Cluster for horizontal scaling — that matter when Redis becomes a critical dependency.
Data Structures: The Foundation
Before the advanced features, the core data structures deserve more than a list — understanding their time complexities clarifies when to reach for each.
Strings
The fundamental type. Strings hold bytes up to 512MB — text, serialized JSON, binary data, integers, floats.
1
2
3
4
5
6
7
8
9
10
11
12
13
|
SET user:1000:session "eyJhbGci..." EX 3600 NX
# EX: expire in 3600 seconds
# NX: only set if key does not exist (atomic conditional write)
INCR api:rate:user:1000:2024-03-27
# Atomic increment — perfect for counters, rate limiting
# Returns new value; no read-modify-write race
# Atomic get-and-set (useful for cache warming)
GETDEL cache:stale:key
# Distributed lock (atomic SET with NX and EX)
SET lock:order:42 $random_token EX 30 NX
|
Lists
Doubly-linked list. O(1) push/pop from both ends. The canonical queue or stack.
1
2
3
4
5
6
7
8
9
10
11
12
|
# Producer: push jobs to the right end
RPUSH job:queue '{"type":"email","to":"user@example.com"}'
# Consumer: blocking pop from left end (waits up to 30s if empty)
BLPOP job:queue 30
# Reliable queue: atomically move to "processing" list before consuming
LMOVE job:queue job:processing LEFT RIGHT
# Capped log: keep last 1000 entries
RPUSH access:log "2024-03-27T10:00:00Z GET /api/users 200"
LTRIM access:log -1000 -1
|
Sorted Sets
The most versatile data structure: a set where each member has a float score, ordered by score. O(log N) add/update, O(log N + M) range queries.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
# Leaderboard
ZADD leaderboard 9850.5 "player:alice"
ZADD leaderboard 8200.0 "player:bob"
ZREVRANGE leaderboard 0 9 WITHSCORES # Top 10
# Time-series event store (score = Unix timestamp)
ZADD events:user:1000 1711497600 "login"
ZADD events:user:1000 1711501200 "purchase:42"
ZRANGEBYSCORE events:user:1000 1711490000 1711510000 # Window query
# Sliding window rate limiting
ZADD rate:user:1000 1711497600 "req:unique-id-1"
ZREMRANGEBYSCORE rate:user:1000 -inf "(now - window)"
ZCARD rate:user:1000 # Count in window
|
HyperLogLog
Probabilistic cardinality estimation using ~12KB regardless of N. Error rate ~0.81%.
1
2
3
4
5
6
7
8
9
10
|
# Count unique visitors without storing every user ID
PFADD visitors:2024-03-27 "user:1000" "user:1001" "user:1002"
PFCOUNT visitors:2024-03-27 # Returns approximate count
# Merge HLLs across days for monthly uniques
PFMERGE visitors:march \
visitors:2024-03-01 \
visitors:2024-03-02 \
visitors:2024-03-31
PFCOUNT visitors:march
|
Pub/Sub: Lightweight Messaging
Redis pub/sub is fire-and-forget messaging. Publishers send to channels; subscribers receive in real time. No persistence — offline subscribers miss messages.
1
2
3
4
5
6
7
8
9
|
# Terminal 1: subscriber
SUBSCRIBE notifications:user:1000 notifications:global
# Terminal 2: publisher
PUBLISH notifications:user:1000 '{"type":"like","post_id":42}'
PUBLISH notifications:global '{"type":"maintenance","starts_at":"2024-03-28T02:00:00Z"}'
# Pattern subscribe (all user notification channels)
PSUBSCRIBE notifications:user:*
|
Go Pub/Sub
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
|
import (
"context"
"encoding/json"
"github.com/redis/go-redis/v9"
)
type Notification struct {
Type string `json:"type"`
PostID int `json:"post_id,omitempty"`
}
func subscribeToNotifications(ctx context.Context, rdb *redis.Client, userID string) {
channel := fmt.Sprintf("notifications:user:%s", userID)
sub := rdb.Subscribe(ctx, channel, "notifications:global")
defer sub.Close()
for msg := range sub.Channel() {
var notif Notification
if err := json.Unmarshal([]byte(msg.Payload), ¬if); err != nil {
continue
}
handleNotification(notif)
}
}
func publishNotification(ctx context.Context, rdb *redis.Client, userID string, n Notification) error {
data, _ := json.Marshal(n)
return rdb.Publish(ctx, fmt.Sprintf("notifications:user:%s", userID), data).Err()
}
|
When Pub/Sub Falls Short
Pub/sub is simple but limited:
- No persistence: offline subscribers miss messages permanently
- No consumer groups: all subscribers receive every message (fan-out only)
- No acknowledgement: no way to confirm delivery
- No backpressure: slow subscribers can’t throttle producers
For these requirements, use Redis Streams.
Redis Streams: Durable Event Logs
Streams (added in Redis 5.0) are an append-only log — like Kafka in miniature. Every message is durably stored with an auto-generated ID. Consumer groups enable competing consumers where each message goes to exactly one worker.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
# Append an event (auto-generated ID: <timestamp>-<sequence>)
XADD orders:events * \
type "order.placed" \
order_id "42" \
user_id "1000" \
amount "59.99"
# Returns: "1711497600123-0"
# Read the last 10 events
XRANGE orders:events - + COUNT 10
# Read events after a specific ID
XRANGE orders:events 1711497600123-0 +
# Trim stream to last 10,000 events (approximate for performance)
XTRIM orders:events MAXLEN ~ 10000
|
Consumer Groups
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
|
# Create a consumer group ($ = start from new messages only)
XGROUP CREATE orders:events email-processor $ MKSTREAM
# Worker reads next undelivered message (blocking wait up to 5s)
XREADGROUP GROUP email-processor worker-1 \
COUNT 1 BLOCK 5000 \
STREAMS orders:events >
# ">" means messages not yet delivered to this group
# Acknowledge successful processing
XACK orders:events email-processor 1711497600123-0
# View pending (unacknowledged) messages for crash recovery
XPENDING orders:events email-processor - + 10
# Claim messages stuck in pending > 5 minutes (dead worker recovery)
XAUTOCLAIM orders:events email-processor worker-2 300000 0-0
|
Go Stream Consumer
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
|
package events
import (
"context"
"time"
"github.com/redis/go-redis/v9"
)
type StreamConsumer struct {
rdb *redis.Client
stream string
group string
consumer string
}
func (c *StreamConsumer) Run(ctx context.Context, handler func(map[string]interface{}) error) {
for {
select {
case <-ctx.Done():
return
default:
}
// Claim any messages pending > 5 minutes from dead workers
pending, _ := c.rdb.XAutoClaim(ctx, &redis.XAutoClaimArgs{
Stream: c.stream,
Group: c.group,
Consumer: c.consumer,
MinIdle: 5 * time.Minute,
Start: "0-0",
Count: 10,
}).Result()
for _, msg := range pending.Messages {
if err := handler(msg.Values); err == nil {
c.rdb.XAck(ctx, c.stream, c.group, msg.ID)
}
}
// Read new messages
streams, err := c.rdb.XReadGroup(ctx, &redis.XReadGroupArgs{
Group: c.group,
Consumer: c.consumer,
Streams: []string{c.stream, ">"},
Count: 10,
Block: 5 * time.Second,
}).Result()
if err == redis.Nil {
continue
}
if err != nil {
time.Sleep(time.Second)
continue
}
for _, stream := range streams {
for _, msg := range stream.Messages {
if err := handler(msg.Values); err != nil {
// Don't ACK — stays pending for retry
continue
}
c.rdb.XAck(ctx, c.stream, c.group, msg.ID)
}
}
}
}
// Producer
func PublishOrder(ctx context.Context, rdb *redis.Client, values map[string]interface{}) (string, error) {
return rdb.XAdd(ctx, &redis.XAddArgs{
Stream: "orders:events",
MaxLen: 100000,
Approx: true,
Values: values,
}).Result()
}
|
Streams vs Pub/Sub vs Lists
| Feature |
Pub/Sub |
Lists (BLPOP) |
Streams |
| Persistence |
None |
Yes |
Yes |
| Message history |
No |
No |
Yes |
| Consumer groups |
No |
No |
Yes |
| Multiple consumers |
Fan-out only |
Competing |
Both |
| Acknowledgement |
No |
No |
Yes |
| Redelivery on failure |
No |
No |
Yes (XAUTOCLAIM) |
| Ordering |
No guarantee |
FIFO |
Strict (by ID) |
| Best for |
Real-time broadcast |
Simple queue |
Event log, reliable queue |
Use pub/sub for real-time notifications where missing a message is acceptable (presence indicators, cache invalidation). Use Streams when you need durability, competing consumers, and at-least-once delivery.
RedisJSON: Document Storage
The RedisJSON module stores and queries JSON documents natively. Unlike serializing JSON to a string, RedisJSON allows partial updates and path-based queries without loading the entire document.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
|
# Store a JSON document
JSON.SET user:1000 $ \
'{"name":"Alice","email":"alice@example.com","address":{"city":"Portland","zip":"97201"},"tags":["pro","beta"]}'
# Get the whole document
JSON.GET user:1000
# Get a nested path (JSONPath syntax)
JSON.GET user:1000 $.address.city
# Returns: ["Portland"]
# Update a specific field (no need to fetch-modify-store)
JSON.SET user:1000 $.email '"alice.smith@example.com"'
# Append to an array
JSON.ARRAPPEND user:1000 $.tags '"enterprise"'
# Atomic numeric increment
JSON.NUMINCRBY user:1000 $.login_count 1
# Delete a field
JSON.DEL user:1000 $.address.zip
# Get array length
JSON.ARRLEN user:1000 $.tags
|
Python with RedisJSON
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
|
import redis
from redis.commands.json.path import Path
r = redis.Redis(host='localhost', decode_responses=True)
# Store a document
r.json().set('product:42', Path.root_path(), {
'name': 'Mechanical Keyboard',
'price': 149.99,
'stock': 50,
'tags': ['electronics', 'keyboards'],
'specs': {'switches': 'Cherry MX Blue', 'layout': 'TKL'}
})
# Partial update — only touch what changed
r.json().set('product:42', '$.price', 129.99)
r.json().numincrby('product:42', '$.stock', -1)
# Multi-path read
result = r.json().get('product:42', '$.name', '$.price', '$.stock')
# {'$.name': ['Mechanical Keyboard'], '$.price': [129.99], '$.stock': [49]}
|
RediSearch: Full-Text Search and Indexing
RediSearch creates secondary indexes over Redis Hashes or JSON documents, enabling full-text search, numeric ranges, tag filtering, geo queries, and aggregations — without a separate search engine.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
|
# Create an index over JSON documents with prefix "product:"
FT.CREATE idx:products \
ON JSON \
PREFIX 1 product: \
SCHEMA \
$.name AS name TEXT WEIGHT 2.0 \
$.price AS price NUMERIC SORTABLE \
$.tags[*] AS tags TAG \
$.specs.switches AS switches TEXT \
$.stock AS stock NUMERIC
# Full-text search
FT.SEARCH idx:products "mechanical keyboard"
# Filtered search: tag + price range
FT.SEARCH idx:products "@tags:{electronics} @price:[50 200]"
# Full-text + filter + sort + pagination
FT.SEARCH idx:products "cherry mx @price:[100 300]" \
RETURN 3 $.name $.price $.stock \
SORTBY price ASC \
LIMIT 0 10
# Fuzzy search (typo tolerance — % = one edit distance)
FT.SEARCH idx:products "%keybard%"
# Aggregation: average price per tag
FT.AGGREGATE idx:products "*" \
GROUPBY 1 @tags \
REDUCE AVG 1 @price AS avg_price \
SORTBY 2 @avg_price DESC
|
Python with RediSearch
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
|
from redis.commands.search.query import Query
from redis.commands.search.field import TextField, NumericField, TagField
from redis.commands.search.indexDefinition import IndexDefinition, IndexType
r = redis.Redis(host='localhost', decode_responses=True)
# Create index
r.ft('idx:products').create_index(
[
TextField('$.name', as_name='name', weight=2.0),
NumericField('$.price', as_name='price', sortable=True),
TagField('$.tags[*]', as_name='tags'),
NumericField('$.stock', as_name='stock'),
],
definition=IndexDefinition(
prefix=['product:'],
index_type=IndexType.JSON
)
)
# The index updates automatically when JSON documents change
# Search
results = r.ft('idx:products').search(
Query('@tags:{electronics} @price:[50 150]')
.sort_by('price', asc=True)
.return_fields('$.name', '$.price', '$.stock')
.paging(0, 10)
)
for doc in results.docs:
print(doc['$.name'], doc['$.price'])
|
Vector Search (Semantic Search)
Redis 7.2+ with RediSearch supports vector similarity search — store embeddings alongside data and find semantically similar items:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
|
import numpy as np
# Index with a vector field
r.ft('idx:articles').create_index([
TextField('$.title', as_name='title'),
r.ft().VectorField(
'$.embedding', 'HNSW',
{'TYPE': 'FLOAT32', 'DIM': 1536, 'DISTANCE_METRIC': 'COSINE'},
as_name='embedding'
)
], definition=IndexDefinition(prefix=['article:'], index_type=IndexType.JSON))
# Store article with its embedding vector
embedding = get_embedding("Redis is a fast in-memory database")
r.json().set('article:1', Path.root_path(), {
'title': 'Redis Performance Guide',
'embedding': embedding.tolist()
})
# Find top-5 semantically similar articles
query_vec = np.array(get_embedding("fast caching databases"), dtype=np.float32).tobytes()
results = r.ft('idx:articles').search(
Query('*=>[KNN 5 @embedding $vec AS score]')
.sort_by('score', asc=True)
.return_fields('title', 'score')
.dialect(2),
query_params={'vec': query_vec}
)
|
Redis as a Primary Database
Right Use Cases
Redis works excellently as a primary store for:
- Session storage: naturally key-value with TTLs; most frameworks have Redis backends
- Real-time leaderboards: sorted sets with
ZADD/ZREVRANGE are purpose-built; updating and querying 10M entries is O(log N)
- Rate limiting state: atomic
INCR and Lua scripts prevent races that would corrupt SQL-based rate limiters
- Feature flags: hash of flag → JSON config with pub/sub for real-time propagation
- High-velocity counters: API call counts, event metrics — no SQL database matches Redis ingestion rates
- Work queues and ephemeral state: shopping carts, temporary computation results, job queues
When Redis Is Wrong
- Data requiring strong durability without any data loss window (use PostgreSQL)
- Complex relational queries and JOINs
- Datasets much larger than available RAM
- Workloads needing arbitrary schema enforcement
Persistence Modes
RDB — Point-in-Time Snapshots
Periodic binary snapshots written to disk. Compact, fast startup, minimal I/O overhead during operation.
1
2
3
4
5
6
7
8
|
# redis.conf
save 900 1 # Snapshot if ≥1 key changed in 900s
save 300 10 # Snapshot if ≥10 keys changed in 300s
save 60 10000 # Snapshot if ≥10000 keys changed in 60s
rdbcompression yes
dbfilename dump.rdb
dir /var/lib/redis
|
Cons: you lose all writes since the last snapshot (up to several minutes).
AOF — Append Only File
Every write command is appended to a log. On restart, Redis replays the log.
1
2
3
4
5
6
7
8
9
10
11
12
|
appendonly yes
appendfilename "appendonly.aof"
# fsync policy:
# always — sync after every write (slowest, no data loss)
# everysec — sync every second (recommended: ≤1 second data loss)
# no — OS decides (fastest, unpredictable data loss)
appendfsync everysec
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
no-appendfsync-on-rewrite yes # Avoid I/O spikes during rewrite
|
Recommended for primary storage: enable both RDB and AOF. On restart, Redis uses the AOF (more complete); RDB provides faster disaster recovery restores.
1
2
3
4
|
# Primary database: hybrid persistence
save 3600 1
appendonly yes
appendfsync everysec
|
1
2
3
|
# Pure cache: discard everything on restart
save ""
appendonly no
|
High Availability: Sentinel vs Cluster
Redis Sentinel
Sentinel provides automatic failover for a primary-replica setup. Three (or more) Sentinel processes monitor the primary; if it fails, they elect a replica to become the new primary.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
|
# docker-compose.yml
services:
redis-primary:
image: redis:7-alpine
command: redis-server --requirepass ${REDIS_PASSWORD} --appendonly yes
redis-replica:
image: redis:7-alpine
command: >
redis-server
--replicaof redis-primary 6379
--requirepass ${REDIS_PASSWORD}
--masterauth ${REDIS_PASSWORD}
--appendonly yes
sentinel-1: &sentinel
image: redis:7-alpine
command: redis-sentinel /etc/redis/sentinel.conf
volumes:
- ./sentinel.conf:/etc/redis/sentinel.conf
sentinel-2:
<<: *sentinel
sentinel-3:
<<: *sentinel
|
1
2
3
4
5
6
7
8
|
# sentinel.conf
sentinel monitor mymaster redis-primary 6379 2
# "2" = quorum: 2 of 3 sentinels must agree before failover
sentinel auth-pass mymaster changeme
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 60000
sentinel parallel-syncs mymaster 1
|
1
2
3
4
5
6
7
8
9
10
11
|
// Go client with Sentinel support
rdb := redis.NewFailoverClient(&redis.FailoverOptions{
MasterName: "mymaster",
SentinelAddrs: []string{
"sentinel-1:26379",
"sentinel-2:26379",
"sentinel-3:26379",
},
Password: os.Getenv("REDIS_PASSWORD"),
SentinelPassword: os.Getenv("REDIS_PASSWORD"),
})
|
Sentinel provides automatic failover with ~30-60 seconds of downtime during a primary failure. The entire dataset lives on one node — no sharding. Right choice for most deployments.
Redis Cluster
Cluster distributes data across multiple shards (nodes) using 16,384 hash slots. Each shard holds a subset of slots with its own replica.
1
2
3
4
5
6
|
# Create a 6-node cluster (3 primaries + 3 replicas)
redis-cli --cluster create \
node1:6379 node2:6379 node3:6379 \
node4:6379 node5:6379 node6:6379 \
--cluster-replicas 1 \
-a ${REDIS_PASSWORD}
|
Hash tags force related keys to the same shard — required for multi-key operations:
1
2
3
4
5
6
7
|
# BAD: these keys land on different shards
MSET user:1000 "alice" order:42 "pending"
# GOOD: hash tag {user:1000} determines the slot for both keys
SET {user:1000}:profile "alice"
SET {user:1000}:orders:42 "pending"
MGET {user:1000}:profile {user:1000}:orders:42 # Works — same shard
|
Cluster limitations:
MGET/MSET only across same-shard keys (use hash tags)
- Lua scripts can only access same-shard keys
SELECT (database selection) not supported — only DB 0
SCAN must be run per-node
Choose Sentinel: dataset fits on one server, simplicity preferred, automatic failover with acceptable downtime.
Choose Cluster: dataset exceeds single-server RAM, need >100k ops/second write throughput, or horizontal scaling required.
Operational Essentials
Memory Management
1
2
3
4
5
6
7
8
9
10
11
12
13
|
# Memory overview
redis-cli INFO memory
# Key metrics:
# used_memory_human — actual data stored
# used_memory_rss_human — what the OS reports (includes fragmentation)
# mem_fragmentation_ratio — healthy: 1.0–1.5; above 1.5 = fragmentation
# Per-key memory usage
redis-cli MEMORY USAGE user:1000
# Find largest keys (slow — use on replicas or off-peak)
redis-cli --bigkeys
|
1
2
3
4
|
# redis.conf — memory policy
maxmemory 4gb
maxmemory-policy allkeys-lru # Cache: evict LRU when full
# maxmemory-policy noeviction # Primary DB: return error when full
|
| Policy |
Behavior |
noeviction |
Return error when memory full (primary DB) |
allkeys-lru |
Evict least-recently-used from all keys |
allkeys-lfu |
Evict least-frequently-used (better for skewed access) |
volatile-lru |
Evict LRU from keys with TTL only |
volatile-ttl |
Evict keys closest to expiration |
Monitoring
1
2
3
4
5
6
7
8
9
10
11
12
|
# Real-time stats
redis-cli --stat
# Slow query log (commands slower than 10ms)
redis-cli CONFIG SET slowlog-log-slower-than 10000
redis-cli SLOWLOG GET 25
# Keyspace info
redis-cli INFO keyspace
# Replication lag
redis-cli INFO replication
|
Lua Scripts for Atomicity
When an operation requires multiple commands to execute atomically, use Lua — Redis executes scripts without interleaving other commands:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
-- Sliding window rate limiter (atomic)
-- KEYS[1] = rate key, ARGV[1] = window seconds, ARGV[2] = limit
local key = KEYS[1]
local window = tonumber(ARGV[1])
local limit = tonumber(ARGV[2])
local now = tonumber(redis.call('TIME')[1])
redis.call('ZREMRANGEBYSCORE', key, 0, now - window)
local count = redis.call('ZCARD', key)
if count < limit then
redis.call('ZADD', key, now, now .. '-' .. math.random(100000))
redis.call('EXPIRE', key, window)
return 1 -- allowed
end
return 0 -- rate limited
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
var rateLimitScript = redis.NewScript(`
local now = tonumber(redis.call('TIME')[1])
redis.call('ZREMRANGEBYSCORE', KEYS[1], 0, now - ARGV[1])
local count = redis.call('ZCARD', KEYS[1])
if count < tonumber(ARGV[2]) then
redis.call('ZADD', KEYS[1], now, now .. math.random())
redis.call('EXPIRE', KEYS[1], ARGV[1])
return 1
end
return 0
`)
func isAllowed(ctx context.Context, rdb *redis.Client, userID string) (bool, error) {
result, err := rateLimitScript.Run(ctx, rdb,
[]string{fmt.Sprintf("rate:%s", userID)},
60, 100, // 100 requests per 60 seconds
).Int()
return result == 1, err
}
|
Redis is most valuable when you use it for what it’s genuinely good at — not as a relational database, not as a blob store for large files, but as a high-throughput, low-latency data structure server for the specific patterns (leaderboards, rate limiting, messaging, real-time state, event logs, search) where no other technology comes close.
Comments