Kafka Internals

Apache Kafka is a distributed, replicated commit log. Producers append records to topics, and consumers read them by position. Unlike a queue that deletes a message once acknowledged, Kafka retains records for a configured time or size, so many consumers can read independently and re-read old data. This makes it suitable for event streaming, log aggregation, and as the backbone of event-driven architectures.

Topics, partitions, and offsets

A topic is a named stream of records. It is split into partitions. Each partition is an ordered, append-only sequence of records. Every record in a partition has an offset, a sequential integer that identifies its position in that partition.

graph LR
    subgraph topic["Topic: orders (3 partitions)"]
        P0["Partition 0: offsets 0 1 2 3 4"]
        P1["Partition 1: offsets 0 1 2 3"]
        P2["Partition 2: offsets 0 1 2 3 4 5"]
    end
    PR["Producer"] --> P0
    PR --> P1
    PR --> P2

• A record has an optional key, a value, a timestamp, and headers.
• The producer chooses the partition. If a key is present, the default partitioner hashes it, so records with the same key go to the same partition. Without a key, the producer spreads records across partitions.
• Partitions are the unit of parallelism, storage distribution, and ordering. Changing the partition count later changes the key-to-partition mapping for new records.
• Partition count sizing example: if a topic must carry 100 MB/s and one partition sustains about 10 MB/s of writes (an assumed figure, measured per deployment), at least 10 partitions are needed. The count also caps consumer parallelism, so it is chosen from the expected peak consumer count as well.

Storage: the log on disk

Each partition is a directory of log segments. Records are appended to the active segment file. When a segment reaches a size or age limit, it is closed and a new one starts. Each segment has companion index files that map offsets (and timestamps) to positions in the file, so a read at an offset finds its position with a binary search over a sparse index.

Kafka's speed comes from sequential I/O:

• Writes append to the end of a file. Sequential disk access is far faster than random access.
• It relies on the operating system page cache for reads and writes instead of a separate in-process cache.
• Consumers reading recent data are served from the page cache, and the broker can send data from the file to the network socket without copying through user space (zero-copy transfer, sendfile).
• Records are batched and can be compressed by the producer, and stay compressed on the broker.

Replication and in-sync replicas

Each partition has a replication factor, commonly 3. One replica is the leader, and the others are followers. Producers and consumers talk to the leader. Followers fetch records from the leader, as consumers do, and append them to their own logs.

The in-sync replica set (ISR) is the set of replicas (including the leader) that are caught up with the leader. A follower stays in the ISR while it keeps fetching and has reached the leader's log end within a configured time limit (replica.lag.time.max.ms). A record is committed when all replicas in the ISR have it. Consumers only see committed records (up to the high watermark).

sequenceDiagram
    participant P as Producer
    participant L as Leader
    participant F1 as Follower 1
    participant F2 as Follower 2
    P->>L: produce record
    L->>L: append to log
    F1->>L: fetch
    L-->>F1: record
    F2->>L: fetch
    L-->>F2: record
    Note over L: all ISR replicas have it, high watermark advances
    L-->>P: ack (acks=all)

If the leader fails, the controller elects a new leader from the ISR. Because every ISR member has all committed records, no committed data is lost. A follower that falls behind is removed from the ISR, and rejoins after catching up. Leader election by default excludes out-of-sync replicas (unclean leader election is disabled by default), which favors consistency over availability. Cluster metadata and controller election are handled by a Raft-based controller quorum (KRaft) in current versions, which replaced ZooKeeper.

Producer acknowledgements

The producer's acks setting controls when a write is acknowledged.

With acks=all, the broker/topic setting min.insync.replicas sets the minimum ISR size required to accept a write. With replication factor 3 and min.insync.replicas=2, writes tolerate one replica failure without loss, and are rejected when only the leader remains.

Producers retry on transient errors, which can produce duplicates and reordering. The idempotent producer (enable.idempotence) attaches a producer id and sequence numbers per partition, so the broker discards duplicates from retries and preserves order within a partition. Recent Kafka versions enable it by default. Transactions allow atomically writing to multiple partitions and committing consumer offsets together, which supports exactly-once processing in stream applications. See delivery semantics and idempotency.

Consumers and consumer groups

A consumer group is a set of consumers sharing a group id that cooperate to read a topic. Each partition is assigned to exactly one consumer in the group at a time. A consumer may hold several partitions.

graph LR
    subgraph topic["Topic partitions"]
        P0["P0"]
        P1["P1"]
        P2["P2"]
        P3["P3"]
    end
    subgraph g1["Group A"]
        A1["Consumer 1"]
        A2["Consumer 2"]
    end
    subgraph g2["Group B"]
        B1["Consumer 1"]
    end
    P0 --> A1
    P1 --> A1
    P2 --> A2
    P3 --> A2
    P0 --> B1
    P1 --> B1
    P2 --> B1
    P3 --> B1

• Different groups read the same topic independently, each with its own offsets. This gives pub/sub across groups and competing consumers within a group.
• The maximum useful parallelism of a group equals the partition count. Extra consumers beyond that stay idle.
• A group coordinator on a broker tracks membership. When a consumer joins, leaves, or is declared dead (missed heartbeats or exceeded poll interval), the group rebalances and partitions are reassigned. Processing pauses for the affected partitions during a rebalance (less so with cooperative rebalancing).
• Each consumer commits offsets: the position up to which it has processed, stored in an internal Kafka topic. After a restart or rebalance the new owner resumes from the committed offset. The consumer lag of a partition is the log end offset minus the group's committed offset; it measures how far the group is behind. For example, with a log end offset of 1,500 and a committed offset of 1,200, the lag is 300 records, and records 1,200 to 1,499 are reprocessed if the consumer restarts. Committing after processing gives at-least-once behavior. Committing before gives at-most-once.

Retention and compaction

Records are not deleted on consumption. Retention policies delete old segments:

• Time-based (for example, keep 7 days) or size-based deletion of whole segments.
• Log compaction (cleanup.policy=compact): the log keeps at least the latest record for each key, and older records with the same key are removed by a background cleaner. A record with a key and a null value (a tombstone) marks the key for deletion. A compacted topic holds the latest state per key, used for tables in stream processing and for configuration or entity snapshots.

Ordering guarantees

• Order is guaranteed within a partition: consumers read records in offset order.
• There is no ordering guarantee across partitions of a topic.
• To keep all events for an entity ordered, use the entity id as the key.
• With retries and multiple in-flight requests, order can break unless idempotence is enabled (or in-flight requests limited to one).
• Changing the number of partitions changes where a key maps to, so ordering for a key across that change is not preserved.

When Kafka fits

• High-throughput event ingestion, log and metric pipelines, change data capture streams.
• Multiple independent consumers of the same events, and replay of history.
• Stream processing with Kafka Streams or Flink. See stream vs batch processing.

It is a poorer fit for per-message routing rules, per-message priorities, delayed delivery, or small task queues where individual message acknowledgement and retry semantics matter. See RabbitMQ, SQS, and broker comparison.

Interview questions

• Why can Kafka sustain high throughput? It appends sequentially to partitioned logs, batches and compresses records, uses the OS page cache, and serves consumers with zero-copy transfers. Partitions let load spread across brokers and consumers.
• What does acks=all guarantee? The write is acknowledged only after all in-sync replicas have it. Combined with min.insync.replicas of at least 2, a committed record survives the failure of the leader.
• How many consumers in a group can read a topic with 6 partitions? Up to six do useful work, since each partition goes to one consumer per group. A seventh stays idle.
• How does Kafka preserve per-entity ordering? Use the entity id as the record key so its records hash to one partition, and enable the idempotent producer to keep order across retries.
• What is log compaction for? It retains the latest value per key, so a topic can serve as a changelog from which current state is rebuilt without keeping every historical update.