How Kafka Works End to End

1. You set up a Kafka cluster - it’s made up of multiple brokers (servers).

2. You create topics - like channels or folders where messages will go.

3. Producers are apps or services that send messages to Kafka topics.

4. Each message is just a piece of data (like a log, event, or JSON).

5. Kafka stores these messages inside partitions - each topic is split into multiple partitions for scalability.

6. Messages in a partition are ordered - always appended at the end.

7. Kafka stores the data durably on disk and keeps it for a configured time (e.g., 7 days), even if it’s already read.

8. Consumers are apps or services that read messages from topics.

9. Consumers track their own position (called offset) to know what they’ve already read.

10. Kafka doesn’t delete messages just because they’re read - different consumers can read at their own pace.

11. Multiple consumers can read the same topic in consumer groups - Kafka ensures each message goes to only one consumer in a group.

12. If a consumer crashes, Kafka reassigns its share of work to others in the group.

13. Kafka can handle millions of messages per second - it’s built for speed and scale.

14. It’s used for decoupling systems - producer doesn’t need to know who the consumer is.

15. Kafka is great for real-time pipelines, event sourcing, activity tracking, log aggregation, etc.

16. Optional: You can add Kafka Connect to plug in data sources/sinks like DBs, or Kafka Streams for processing on the fly.

17. Kafka ensures high durability, fault tolerance, and horizontal scalability out of the box.

18. Kafka automatically replicates data across multiple brokers - so even if one server dies, your messages are safe.

Kafka Architecture Deep Dive

Broker Internals

A Kafka broker is a server in the Kafka cluster that:

1. Manages Storage: Maintains partition commit logs
2. Handles Network Protocol: Serves producer/consumer requests
3. Replication Management: Coordinates replicas between brokers
4. Group Coordination: For consumer group management (leader brokers)
5. Controller Functions: One broker acts as a controller for the cluster

Brokers are identified by unique IDs and store partition data in segments:

• Log Segments: The actual files containing message data
• Indexes: Offset and time-based indexes for fast message lookup
• Active Segment: The currently writable segment

Partitioning Mechanics

Partitions are the fundamental unit of parallelism in Kafka:

1. Partitioning Strategy:

   • Default: Round-robin or hash of the message key
   • Custom: Producers can implement custom partitioners
   • Sticky: Batch messages to same partition for efficiency

2. Partition Leaders and Followers:

   • Each partition has one leader and multiple followers (replicas)
   • Leaders handle all reads and writes
   • Followers passively replicate the leader

3. Partition Reassignment:

   • Rebalancing partitions across brokers
   • Automatically happens when brokers join/leave
   • Can be manually triggered for load balancing

Replication Protocol

Kafka uses a leader-follower replication model:

1. In-Sync Replicas (ISR):

   • Set of replicas that are fully caught up with the leader
   • Maintained by the controller
   • Min.insync.replicas setting controls durability guarantees

2. Replication Flow:

   • Producer sends message to leader
   • Leader appends to local log
   • Followers fetch from leader (pull-based)
   • Message is considered “committed” when all ISRs have replicated it

3. Leader Election:

   • If a leader fails, a follower from ISR becomes the new leader
   • Unclean leader election (allowing non-ISR followers) is configurable

Message Format

Kafka messages have a sophisticated binary format:

1. Message Structure:

   • Timestamp: When message was created
   • Key (optional): For partitioning and compaction
   • Value: The actual payload (can be null)
   • Headers: Optional metadata (key-value pairs)

2. Record Batches:

   • Messages are grouped into batches for efficiency
   • Compression happens at batch level (gzip, snappy, lz4, or zstd)
   • Each batch shares metadata like producer ID and timestamp base

3. Message Versions:

   • v0, v1: Legacy formats
   • v2: Current format with better performance and features

Producer Internals

Kafka producers are highly optimized:

1. Buffering and Batching:

   • Messages are queued in memory before sending
   • Batched by partition for network efficiency
   • Controlled by batch.size and linger.ms

2. Acknowledgment Modes (acks):

   • acks=0: Fire and forget (no wait for acknowledgment)
   • acks=1: Wait for leader acknowledgment
   • acks=all: Wait for all in-sync replicas to acknowledge

3. Retries and Idempotence:

   • Automatic retries on transient failures
   • Idempotent producer uses sequence numbers to prevent duplicates
   • Transactions allow atomic multi-partition writes

Consumer Internals

Kafka consumers have sophisticated features:

1. Offset Management:

   • Stored in special topic __consumer_offsets
   • auto.offset.reset controls behavior for new consumers
   • Can be committed automatically or manually

2. Consumer Groups:

   • Group Coordinator broker manages membership
   • Rebalance Protocol:
     • Eager: Stop all consumers during rebalance
     • Cooperative: Incremental reassignment

3. Fetch Protocol:

   • Consumers pull data from brokers (not push)
   • Controlled by max.poll.records and fetch.max.bytes
   • Backpressure handled naturally by consumer pace

Kafka’s Storage Engine

Kafka’s storage is optimized for sequential operations:

1. Log Structure:

   • Append-only commit log
   • Immutable segments (files)
   • Sequential writes for high performance

2. Retention Policies:

   • Time-based: Delete segments older than N days
   • Size-based: Keep only N bytes per partition
   • Compaction: Keep only latest value per key

3. Zero-Copy:

   • Uses sendfile() system call
   • Data goes directly from disk to network
   • Bypasses application buffer for efficiency

Zookeeper’s Role (Pre-KRaft)

Traditionally, Kafka relied on Zookeeper for:

1. Cluster Coordination:

   • Broker registration
   • Topic configuration
   • Access control lists

2. Leader Election:

   • Managing controller elections
   • Tracking partition leadership

3. KRaft Mode:

   • Newer versions can run without Zookeeper
   • Uses Raft consensus protocol built into Kafka
   • Improves scalability and simplifies architecture

Advanced Features

1. Kafka Streams:

   • Stream processing library built on Kafka
   • Stateful and stateless transformations
   • Exactly-once semantics with processing guarantees
   • Windows, joins, aggregations on streaming data

2. Schema Registry:

   • Central repository for message schemas
   • Schema evolution with compatibility checks
   • Common formats: Avro, Protobuf, JSON Schema

3. MirrorMaker 2:

   • Cross-cluster replication
   • Geo-replication for disaster recovery
   • Active-active multi-datacenter setups

4. Kafka Connect:

   • Data integration framework
   • Source connectors: Import data to Kafka
   • Sink connectors: Export data from Kafka
   • Distributed or standalone mode
   • Offset tracking and exactly-once delivery

Performance Tuning

1. Broker Tuning:

   • num.network.threads and num.io.threads
   • log.flush.interval.messages and log.flush.interval.ms
   • replica.fetch.max.bytes for replication throughput

2. Producer Tuning:

   • buffer.memory for producer memory allocation
   • compression.type for message compression
   • batch.size and linger.ms for batching control

3. Consumer Tuning:

   • fetch.min.bytes and fetch.max.wait.ms
   • max.partition.fetch.bytes for batch size
   • heartbeat.interval.ms for group membership

4. Topic Tuning:

   • Partition count for parallelism
   • Replication factor for reliability
   • min.insync.replicas for durability guarantees