Apache Kafka Deep Dive

March 26, 2026 16 min read Data & Databases

kafkastreamingmessagingdistributed-systemskubernetesdata-engineeringbackend

Apache Kafka is one of those systems that looks simple on the surface — a distributed log, messages in, messages out — until you run it in production and discover that the gap between “it works” and “it works correctly at scale under failure conditions” is vast. Kafka is built on a deceptively powerful abstraction: the immutable, ordered, replicated log. Everything else flows from that.

This guide covers Kafka from the ground up: the internal architecture, producing and consuming messages with real code, partition strategy and consumer groups, the subtle correctness guarantees that separate at-most-once from exactly-once delivery, and deploying Kafka on Kubernetes with Strimzi.

The Core Abstraction: The Log

A Kafka topic is a named, ordered, append-only log. Messages are written to the end of the log and never modified. Consumers read from any position in the log by tracking an offset — a monotonically increasing integer that identifies a message’s position.

This is fundamentally different from a traditional message queue:

Queue	Kafka Log
Message consumed → message deleted	Message consumed → message stays (until retention expires)
One consumer processes each message	Multiple independent consumer groups each read the full log
Queue depth = backlog metric	Offset lag = consumer backlog metric
Hard to replay	Replay by resetting offset
Ordered within single queue	Ordered within partition

The log model means Kafka is simultaneously a message bus, an event store, and a stream processing platform. Multiple teams can independently consume the same events without coordination, and any consumer can re-read historical events up to the retention window.

Architecture

┌────────────────────────────────────────────────────────────┐
│                        Kafka Cluster                        │
│                                                            │
│  ┌──────────────────────────────────────────────────────┐  │
│  │  Topic: orders  (6 partitions, replication factor 3) │  │
│  │                                                      │  │
│  │  Partition 0: [msg0][msg1][msg2][msg3]...            │  │
│  │  Partition 1: [msg0][msg1][msg2]...                  │  │
│  │  Partition 2: [msg0][msg1][msg2][msg3][msg4]...      │  │
│  │  ...                                                 │  │
│  └──────────────────────────────────────────────────────┘  │
│                                                            │
│  Broker 1          Broker 2          Broker 3             │
│  Leader: P0,P3     Leader: P1,P4     Leader: P2,P5        │
│  Follower: P1,P4   Follower: P2,P5   Follower: P0,P3      │
│                                                            │
│  KRaft Controller (replaces ZooKeeper in Kafka 3.x)       │
└────────────────────────────────────────────────────────────┘

Partitions are the unit of parallelism and ordering. A topic is split into N partitions, each of which is a fully independent ordered log. Messages within a partition are strictly ordered. Messages across partitions are not.

Replication provides durability. Each partition has one leader and N-1 followers. Producers write to the leader; followers replicate asynchronously. If the leader fails, a follower is elected as the new leader.

KRaft (Kafka Raft) replaced ZooKeeper as the metadata store in Kafka 3.3+ and is now the default. It removes the operational complexity of running a separate ZooKeeper cluster.

Producers

A producer writes messages to Kafka topics. Each message has:

An optional key — used for partition assignment (same key → same partition)
A value — the message payload (bytes — you choose the serialization)
Optional headers — metadata key-value pairs
An optional timestamp

Java Producer

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
    StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
    StringSerializer.class.getName());

// Reliability settings
props.put(ProducerConfig.ACKS_CONFIG, "all");         // wait for all ISR replicas
props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true); // exactly-once per partition

// Throughput tuning
props.put(ProducerConfig.LINGER_MS_CONFIG, 5);        // batch for up to 5ms
props.put(ProducerConfig.BATCH_SIZE_CONFIG, 65536);   // 64KB batches
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4");

KafkaProducer<String, String> producer = new KafkaProducer<>(props);

// Async send with callback
ProducerRecord<String, String> record = new ProducerRecord<>(
    "orders",           // topic
    "order-12345",      // key — routes to consistent partition
    "{\"id\":\"12345\",\"amount\":99.99}"  // value
);

producer.send(record, (metadata, exception) -> {
    if (exception != null) {
        log.error("Failed to send message", exception);
    } else {
        log.info("Sent to partition {} offset {}",
            metadata.partition(), metadata.offset());
    }
});

// Flush before shutdown
producer.flush();
producer.close();

Python Producer (confluent-kafka)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


from confluent_kafka import Producer
import json

producer = Producer({
    'bootstrap.servers': 'kafka:9092',
    'acks': 'all',
    'retries': 2147483647,
    'enable.idempotence': True,
    'compression.type': 'lz4',
    'linger.ms': 5,
    'batch.size': 65536,
})

def delivery_report(err, msg):
    if err is not None:
        print(f'Delivery failed for record {msg.key()}: {err}')
    else:
        print(f'Record delivered to partition {msg.partition()} '
              f'at offset {msg.offset()}')

# Produce a message
order = {'id': '12345', 'amount': 99.99, 'user_id': 42}
producer.produce(
    topic='orders',
    key='order-12345',      # same key → same partition every time
    value=json.dumps(order).encode('utf-8'),
    callback=delivery_report,
)

# Poll for delivery reports and block until all sent
producer.flush()

Partition Assignment

The producer assigns messages to partitions based on the key:

partition = hash(key) % num_partitions

No key → round-robin across partitions (maximises throughput but loses ordering).

Key-based assignment gives you:

Ordering guarantee: all messages with the same key land in the same partition, so consumers see them in order
Locality: related messages (all events for order-12345) are co-located and processed by the same consumer instance

Design your keys around your ordering requirements:

user_id → all events for a user are ordered per-user
order_id → all events for an order are ordered per-order
device_id → sensor readings from a device stay in order

Consumers and Consumer Groups

A consumer reads messages from one or more partitions, tracking its position with an offset. A consumer group is a set of consumers that cooperatively consume a topic — each partition is assigned to exactly one consumer in the group at a time.

Topic: orders (6 partitions)

Consumer Group A (3 consumers — full parallelism):
  Consumer A1: Partition 0, 1
  Consumer A2: Partition 2, 3
  Consumer A3: Partition 4, 5

Consumer Group B (1 consumer — sequential):
  Consumer B1: Partition 0, 1, 2, 3, 4, 5

Consumer Group C (6 consumers — max parallelism):
  Consumer C1: Partition 0
  Consumer C2: Partition 1
  ...
  Consumer C6: Partition 5

Key insight: consumer groups are independent. Group A and Group B both read the full topic independently. Kafka doesn’t care how many consumer groups exist.

Java Consumer

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "order-processor");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
    StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
    StringDeserializer.class.getName());

// Offset management
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); // start from beginning if no offset
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);      // manual commit for reliability

// Performance
props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 500);
props.put(ConsumerConfig.FETCH_MIN_BYTES_CONFIG, 1024);
props.put(ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG, 500);

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(List.of("orders"));

try {
    while (true) {
        ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));

        for (ConsumerRecord<String, String> record : records) {
            // Process the record
            processOrder(record.key(), record.value());

            log.info("Processed: topic={} partition={} offset={} key={}",
                record.topic(), record.partition(), record.offset(), record.key());
        }

        // Commit after successful processing (at-least-once)
        consumer.commitSync();
    }
} finally {
    consumer.close();
}

Python Consumer

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


from confluent_kafka import Consumer, KafkaError
import json

consumer = Consumer({
    'bootstrap.servers': 'kafka:9092',
    'group.id': 'order-processor',
    'auto.offset.reset': 'earliest',
    'enable.auto.commit': False,    # manual commits
    'max.poll.interval.ms': 300000, # 5 minutes max processing time
})

consumer.subscribe(['orders'])

try:
    while True:
        msg = consumer.poll(timeout=1.0)

        if msg is None:
            continue
        if msg.error():
            if msg.error().code() == KafkaError._PARTITION_EOF:
                continue   # end of partition, not an error
            raise Exception(f'Consumer error: {msg.error()}')

        # Process the message
        order = json.loads(msg.value().decode('utf-8'))
        process_order(msg.key().decode('utf-8'), order)

        # Commit only after successful processing
        consumer.commit(msg)

finally:
    consumer.close()

Offset Management

Offset management is where most Kafka bugs live. The three delivery semantics:

At-most-once (commit before processing):

1
2


consumer.commit(msg)    # commit first
process_order(order)    # if this crashes, message is lost

At-least-once (commit after processing — the usual choice):

1
2
3


process_order(order)    # process first
consumer.commit(msg)    # if crash here, message replays on restart
# Your processing must be idempotent — handle duplicate delivery

Exactly-once (transactional, covered below):

1

# Uses Kafka transactions — atomically process + commit offset

For at-least-once, make your consumers idempotent: processing the same message twice produces the same result as processing it once. Common techniques:

Upsert instead of insert (use message key as DB primary key)
Check-and-set with a processed message ID table
Natural idempotency (e.g., setting a field to a value is idempotent)

Partition Strategy: The Most Important Design Decision

How many partitions should a topic have? This decision is hard to change later (increasing partitions breaks key-based ordering for existing keys).

Rules of thumb:

Throughput: if you need N MB/s and each partition handles ~10 MB/s, you need N/10 partitions
Consumer parallelism: you can never have more parallelism than partitions (extra consumers sit idle)
Broker count: partitions distribute across brokers; uneven distribution causes hotspots
Start higher than you think: 6, 12, 24, 48 — powers of 2 or multiples of broker count

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


# Create a topic with 12 partitions and replication factor 3
kafka-topics.sh --bootstrap-server kafka:9092 \
  --create \
  --topic orders \
  --partitions 12 \
  --replication-factor 3 \
  --config retention.ms=604800000 \   # 7 days retention
  --config min.insync.replicas=2       # require 2 replicas in sync before ack

# Check partition distribution across brokers
kafka-topics.sh --bootstrap-server kafka:9092 \
  --describe --topic orders

# Check consumer group lag
kafka-consumer-groups.sh --bootstrap-server kafka:9092 \
  --describe --group order-processor
# GROUP           TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
# order-processor orders  0          14230           14231           1
# order-processor orders  1          18940           18940           0

Exactly-Once Semantics

Kafka’s exactly-once semantics (EOS) guarantees that each message is processed and its effects are committed exactly once, even under failures. It requires two components:

Idempotent producer (prevents duplicate messages from producer retries):

1
2
3


props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
// Each producer gets a PID; each message gets a sequence number
// Broker deduplicates retries within the producer session

Transactions (atomic multi-topic writes + offset commits):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "order-processor-1");
// Unique per producer instance — survives restarts with same ID

KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.initTransactions();

// Read-process-write loop with transactions
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));

producer.beginTransaction();
try {
    for (ConsumerRecord<String, String> record : records) {
        // Process: produce output to another topic
        ProducerRecord<String, String> output = processOrder(record);
        producer.send(output);
    }

    // Atomically commit: output messages + consumer offsets
    // Either both commit or both are rolled back
    producer.sendOffsetsToTransaction(
        getOffsets(records),
        consumer.groupMetadata()
    );
    producer.commitTransaction();

} catch (Exception e) {
    producer.abortTransaction();
    throw e;
}

Consumers reading from transactionally-written topics must set isolation level:

1
2
3


props.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
// Only reads messages from committed transactions
// Default "read_uncommitted" reads everything including aborted transactions

EOS is powerful but adds latency and complexity. Use it when duplicate processing would cause incorrect business outcomes (double-charging, double-booking). For many use cases, at-least-once with idempotent consumers is simpler and sufficient.

Schema Management with Schema Registry

Raw bytes are fragile. A producer changing its message format silently breaks consumers. Schema Registry (Confluent’s open-source component, or Apicurio) manages Avro, Protobuf, or JSON Schema schemas and enforces compatibility.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


from confluent_kafka import Producer
from confluent_kafka.schema_registry import SchemaRegistryClient
from confluent_kafka.schema_registry.avro import AvroSerializer
from confluent_kafka.serialization import SerializationContext, MessageField

schema_str = """
{
  "type": "record",
  "name": "Order",
  "namespace": "io.lunarops.orders",
  "fields": [
    {"name": "id", "type": "string"},
    {"name": "user_id", "type": "long"},
    {"name": "amount", "type": "double"},
    {"name": "currency", "type": "string", "default": "USD"},
    {"name": "created_at", "type": "long", "logicalType": "timestamp-millis"}
  ]
}
"""

schema_registry_client = SchemaRegistryClient({'url': 'http://schema-registry:8081'})
avro_serializer = AvroSerializer(schema_registry_client, schema_str)

producer = Producer({'bootstrap.servers': 'kafka:9092'})

order = {
    'id': 'order-12345',
    'user_id': 42,
    'amount': 99.99,
    'currency': 'USD',
    'created_at': int(time.time() * 1000)
}

producer.produce(
    topic='orders',
    key='order-12345',
    value=avro_serializer(order, SerializationContext('orders', MessageField.VALUE)),
)

Schema compatibility modes prevent breaking changes:

BACKWARD: new schema can read data written with old schema (add fields with defaults, remove fields)
FORWARD: old schema can read data written with new schema
FULL: both directions (safest — required for zero-downtime rolling upgrades)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Set compatibility mode for a subject
curl -X PUT http://schema-registry:8081/config/orders-value \
  -H "Content-Type: application/json" \
  -d '{"compatibility": "FULL"}'

# List registered schemas
curl http://schema-registry:8081/subjects

# Get schema versions
curl http://schema-registry:8081/subjects/orders-value/versions

Kafka Streams: Stream Processing Without a Separate Cluster

Kafka Streams is a Java library for stateful stream processing that runs inside your application — no separate processing cluster required.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39


StreamsBuilder builder = new StreamsBuilder();

// Read from input topic
KStream<String, Order> orders = builder.stream("orders",
    Consumed.with(Serdes.String(), orderSerde));

// Filter — only paid orders
KStream<String, Order> paidOrders = orders
    .filter((key, order) -> order.getStatus().equals("PAID"));

// Enrich — join with user profile from a KTable
KTable<String, UserProfile> users = builder.table("user-profiles",
    Consumed.with(Serdes.String(), userProfileSerde));

KStream<String, EnrichedOrder> enriched = paidOrders
    .join(users,
        (order, profile) -> new EnrichedOrder(order, profile),
        Joined.with(Serdes.String(), orderSerde, userProfileSerde));

// Aggregate — count orders per user per hour
KTable<Windowed<String>, Long> ordersPerUser = paidOrders
    .groupByKey()
    .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofHours(1)))
    .count(Materialized.as("orders-per-user-store"));

// Write results to output topics
enriched.to("enriched-orders",
    Produced.with(Serdes.String(), enrichedOrderSerde));

ordersPerUser.toStream()
    .map((windowedKey, count) ->
        KeyValue.pair(windowedKey.key(), count))
    .to("user-order-counts",
        Produced.with(Serdes.String(), Serdes.Long()));

// Build and start
KafkaStreams streams = new KafkaStreams(builder.build(), props);
streams.start();
Runtime.getRuntime().addShutdownHook(new Thread(streams::close));

Running Kafka in Kubernetes with Strimzi

Strimzi is the CNCF-graduated Kafka operator for Kubernetes. It manages the full Kafka lifecycle: deployment, scaling, rolling upgrades, TLS, authentication, and topic/user management.

Installing Strimzi

1
2
3
4
5
6
7
8
9


# Install the Strimzi operator
kubectl create namespace kafka
kubectl apply -f "https://strimzi.io/install/latest?namespace=kafka" -n kafka

# Wait for operator pod
kubectl wait pod -n kafka \
  -l name=strimzi-cluster-operator \
  --for=condition=Ready \
  --timeout=300s

Deploying a Kafka Cluster

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76


# kafka-cluster.yaml
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: lunarops-kafka
  namespace: kafka
spec:
  kafka:
    version: 3.7.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
        authentication:
          type: tls
      - name: external
        port: 9094
        type: loadbalancer
        tls: true
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: "3.7"
      # Retention
      log.retention.hours: 168          # 7 days
      log.retention.bytes: 107374182400 # 100 GB per partition
      log.segment.bytes: 1073741824     # 1 GB segments
    storage:
      type: jbod
      volumes:
        - id: 0
          type: persistent-claim
          size: 500Gi
          class: fast-ssd
          deleteClaim: false
    resources:
      requests:
        memory: 4Gi
        cpu: "1"
      limits:
        memory: 8Gi
        cpu: "2"
    jvmOptions:
      -Xms: 2048m
      -Xmx: 4096m
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: kafka-metrics-config.yml

  # KRaft mode (no ZooKeeper)
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 10Gi
      class: fast-ssd
    resources:
      requests:
        memory: 1Gi
        cpu: 500m

  entityOperator:
    topicOperator: {}
    userOperator: {}

Managing Topics and Users Declaratively

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48


# kafka-topic.yaml
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: orders
  namespace: kafka
  labels:
    strimzi.io/cluster: lunarops-kafka
spec:
  partitions: 12
  replicas: 3
  config:
    retention.ms: 604800000   # 7 days
    min.insync.replicas: "2"
    compression.type: lz4
---
# kafka-user.yaml — TLS client with ACLs
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: order-processor
  namespace: kafka
  labels:
    strimzi.io/cluster: lunarops-kafka
spec:
  authentication:
    type: tls
  authorization:
    type: simple
    acls:
      - resource:
          type: topic
          name: orders
          patternType: literal
        operations: [Read, Describe]
        host: "*"
      - resource:
          type: group
          name: order-processor
          patternType: literal
        operations: [Read]
        host: "*"
      - resource:
          type: topic
          name: enriched-orders
          patternType: literal
        operations: [Write, Describe, Create]
        host: "*"

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


kubectl apply -f kafka-cluster.yaml
kubectl apply -f kafka-topic.yaml
kubectl apply -f kafka-user.yaml

# Watch Kafka pods come up
kubectl get pods -n kafka -w

# Check cluster status
kubectl get kafka -n kafka lunarops-kafka -o jsonpath='{.status.conditions}'

# Test connectivity from inside the cluster
kubectl run kafka-test -it --rm \
  --image=quay.io/strimzi/kafka:latest-kafka-3.7.0 \
  --restart=Never \
  -n kafka \
  -- bin/kafka-console-producer.sh \
    --bootstrap-server lunarops-kafka-kafka-bootstrap:9092 \
    --topic orders

Monitoring Kafka

The critical metrics to watch:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# Key Kafka broker metrics (expose via JMX → Prometheus)
kafka_server_brokertopicmetrics_messagesin_total          # inbound message rate
kafka_server_brokertopicmetrics_bytesin_total             # inbound bytes/sec
kafka_server_brokertopicmetrics_bytesout_total            # outbound bytes/sec
kafka_server_replicamanager_underreplicatedpartitions     # ALERT if > 0
kafka_server_replicamanager_offlinereplicacount           # ALERT if > 0
kafka_controller_kafkacontroller_activecontrollercount    # ALERT if != 1
kafka_network_requestmetrics_totaltimems                  # request latency

# Consumer group lag — the most important operational metric
# Monitor with kafka-consumer-groups.sh or Confluent's kafka-lag-exporter

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32


# PrometheusRule for Kafka alerting
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: kafka-alerts
  namespace: kafka
spec:
  groups:
  - name: kafka
    rules:
    - alert: KafkaUnderReplicatedPartitions
      expr: kafka_server_replicamanager_underreplicatedpartitions > 0
      for: 10m
      labels:
        severity: critical
      annotations:
        summary: "Kafka has under-replicated partitions"

    - alert: KafkaConsumerGroupLag
      expr: |
        sum(kafka_consumergroup_lag) by (consumergroup, topic) > 10000
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Consumer group {{ $labels.consumergroup }} is lagging on {{ $labels.topic }}"

    - alert: KafkaNoActiveController
      expr: kafka_controller_kafkacontroller_activecontrollercount != 1
      for: 1m
      labels:
        severity: critical

Quick Reference

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


# Topic management
kafka-topics.sh --bootstrap-server kafka:9092 --list
kafka-topics.sh --bootstrap-server kafka:9092 --describe --topic orders
kafka-topics.sh --bootstrap-server kafka:9092 --create --topic orders \
  --partitions 12 --replication-factor 3

# Produce test messages
kafka-console-producer.sh --bootstrap-server kafka:9092 --topic orders \
  --property "key.separator=:" --property "parse.key=true"

# Consume messages (from beginning)
kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic orders \
  --from-beginning --property print.key=true

# Consumer group management
kafka-consumer-groups.sh --bootstrap-server kafka:9092 --list
kafka-consumer-groups.sh --bootstrap-server kafka:9092 \
  --describe --group order-processor
kafka-consumer-groups.sh --bootstrap-server kafka:9092 \
  --reset-offsets --to-earliest \
  --group order-processor --topic orders --execute

# Get partition offsets
kafka-get-offsets.sh --bootstrap-server kafka:9092 --topic orders

# Check broker config
kafka-configs.sh --bootstrap-server kafka:9092 \
  --describe --entity-type brokers --entity-default

Kafka rewards the time you invest in understanding it. The partition model, offset management, and delivery guarantee semantics are genuinely subtle — getting them wrong produces bugs that only appear under load or after failures. But once you understand the immutable log abstraction and how everything builds on it, Kafka’s behaviour becomes predictable. Events become the lingua franca between services, replay becomes a debugging superpower, and the consumer group model makes horizontal scaling of event processing almost trivial. That’s a foundation worth understanding deeply.