29 min · theory + hands-on practical

Brokers, controllers and partitions

Lesson 1.1 treated Kafka as one box that stores event logs. It's really a small group of servers. This lesson names the parts, shows how one topic is spread over them, and explains how a client finds the right server, which is where many first outages come from.

Theory

Example: the gateway that couldn't start

At 06:00 last Tuesday, an engineer rebooted the server kafka-2 to install an operating-system patch. At 06:01 the charger gateway was redeployed. It didn't start: it crashed, restarted and crashed again for four minutes, until kafka-2 was back. Meanwhile 240 chargers kept their meter readings in memory, about 9,600 of them.

Billing ran through the whole reboot without a single error. Both programs talk to the same Kafka cluster. The only difference was one line of configuration:

# gateway
bootstrap.servers=kafka-2:9092
# billing
bootstrap.servers=kafka-1:9092,kafka-2:9092,kafka-3:9092

To see why that line matters, you need the parts of a cluster.

Brokers and the cluster

A broker is a Kafka server process. It stores topic data on its disk and answers clients that write and read. Brokers that work together form a cluster. Production clusters have at least three brokers, so one can fail, or be patched like kafka-2, without losing data or stopping service.

Partitions and replicas

A topic is split into partitions. Each partition is its own ordered log with its own offsets 0, 1, 2… Splitting a topic lets several brokers share its traffic, and lets several consumers read it in parallel (module 3). Brightlane's charging.telemetry has 6 partitions.

Each partition is stored on several brokers. Each copy is a replica, and the number of copies is the replication factor (3 at Brightlane, and in the lab). One replica is the leader: producers write to it and consumers read from it. The others are followers: they copy the leader's new records continuously. Followers that are fully caught up are in sync, and the list of them is the partition's ISR (in-sync replicas).

Three nodes, kafka-1 to kafka-3. Each runs a controller and a broker. The controllers form a quorum with one active controller. Each broker holds a replica of all six partitions; two partitions per broker are leaders.

What to notice: every broker holds a copy of every partition here, but each one leads only two of them, so the work is shared.

If a leader's broker stops, an in-sync follower on another broker becomes the new leader within seconds. Lesson 4.1 looks at that hand-over in detail.

Controllers and KRaft

Somebody has to decide which broker leads which partition, notice when a broker disappears, and record which topics exist. That's the job of the controllers. Everything they decide is the cluster's metadata: the list of brokers, topics, partitions, leaders and settings.

Controllers keep the metadata in a log of their own and agree on it with KRaft, Kafka's built-in version of the Raft agreement protocol. Three (or five) controllers form a quorum. One of them is the active controller: it makes decisions and writes them to the metadata log. The others are standbys that copy that log, ready to take over. A change counts as decided once a majority has stored it: 2 out of 3. So three controllers survive one failure, and five survive two.

Until Kafka 3.x, this job was done by a separate system, Apache ZooKeeper. Kafka 4.0 removed ZooKeeper support completely. Any guide that tells you to start ZooKeeper is out of date.

In the lab, each node runs both a broker and a controller. This is combined mode. It saves memory on a laptop. In production, controllers run on their own small servers. Then a busy broker can't slow down the cluster's decisions, and you can add brokers without changing the quorum.

How a client finds the right broker

A client can't ask "Kafka" in general. It has to send records to the leader of each partition. It finds the leaders in two steps:

  1. It connects to one of the addresses in its bootstrap servers list and asks for the metadata. Any broker can answer.
  2. The answer lists every broker and the leader of every partition. From then on, the client talks to the leaders directly.

The gateway asks kafka-1, one of its bootstrap brokers, for metadata. kafka-1 answers with all brokers and the leader of each partition. The gateway then sends its records for partition 1 straight to kafka-2, the leader.

What to notice: the bootstrap list is only used to get started. If every address in it is down, the client never learns that other brokers exist. That's what happened to the gateway.

The addresses in the metadata answer are each broker's advertised listener: the address the broker tells clients to use. They must be reachable from where the client runs. The lab brokers have two listeners each: kafka-1:19092 for containers inside Docker, and 127.0.0.1:9092 for your Mac. If a broker advertised kafka-1:19092 to your Mac, step 1 would work and step 2 would fail. That is the most common "Kafka in Docker" problem.

In production

Practical

Note

Notebook: code/module-01/02-brokers-controllers-and-partitions.ipynb · Lab services: kafka-1, kafka-2, kafka-3 · Time: ~17 min

What you'll do: look at the cluster as a client sees it and as an operator sees it, map a 6-partition topic onto the brokers, then stop kafka-2 and see which client survives.

Step 1: Ask the cluster who is in it

code/module-01/02-brokers-controllers-and-partitions.ipynb
admin = AdminClient({"bootstrap.servers": settings.KAFKA_BOOTSTRAP})
cluster = admin.describe_cluster(request_timeout=10).result()

print("cluster id:", cluster.cluster_id)
cluster id: ZpMZEQqGS768XpROns7_Ug
broker  advertised address
------  ------------------
1       127.0.0.1:9092
2       127.0.0.1:9093
3       127.0.0.1:9094

These are the advertised addresses for your Mac.

Step 2: Find the active controller

The notebook runs Kafka's quorum tool inside the kafka-1 container (lab.kafka_cli prints each command before running it):

code/module-01/02-brokers-controllers-and-partitions.ipynb
print(lab.kafka_cli("kafka-metadata-quorum.sh", "--bootstrap-server", "localhost:19092",
                    "describe", "--replication"))

Output (shortened; the leader and the numbers vary):

kafka-1$ kafka-metadata-quorum.sh --bootstrap-server localhost:19092 describe --replication
NodeId  LogEndOffset  Lag  ...  Status
3       1167          0    ...  Leader
1       1167          0    ...  Follower
2       1167          0    ...  Follower

Node 3 is the active controller here. All three have the same LogEndOffset: the standbys have copied the whole metadata log.

Step 3: Create a topic with 6 partitions and 3 replicas

code/module-01/02-brokers-controllers-and-partitions.ipynb
admin.create_topics([NewTopic(TOPIC, num_partitions=6, replication_factor=3)])[TOPIC].result()

show_partitions() then calls admin.describe_topics() and prints one row per partition. Output (which broker leads which partition varies):

partition  leader  replicas   isr
---------  ------  ---------  ---------
0          1       [1, 2, 3]  [1, 2, 3]
1          2       [2, 3, 1]  [2, 3, 1]
2          3       [3, 1, 2]  [3, 1, 2]
3          3       [3, 1, 2]  [3, 1, 2]
4          1       [1, 2, 3]  [1, 2, 3]
5          2       [2, 3, 1]  [2, 3, 1]

Step 4: Count the leaders per broker

{'broker 1': 2, 'broker 2': 2, 'broker 3': 2}

Each broker leads two of the six partitions.

Break it: a client that only knows one broker

Predict first: you stop kafka-2. Client A was configured with only 127.0.0.1:9093, which is kafka-2. Client B has all three addresses. Which one can still start?

code/module-01/02-brokers-controllers-and-partitions.ipynb
lab.compose("stop", "kafka-2")

for name, bootstrap in [("A", "127.0.0.1:9093"), ("B", settings.KAFKA_BOOTSTRAP)]:
    client = AdminClient({"bootstrap.servers": bootstrap, "logger": silent})
    try:
        brokers = sorted(client.list_topics(timeout=8).brokers)
        print(f"client {name}: connected, sees brokers {brokers}")
    except Exception as e:
        print(f"client {name}: failed to start: {e.args[0].str()}")

Output (you may also see a log line from the Step 1 client noting that kafka-2 disconnected):

$ docker compose stop kafka-2
client A: failed to start: Failed to get metadata: Local: Broker transport failure
client B: connected, sees brokers [1, 3]

Then the notebook shows the topic again:

partition  leader  replicas   isr
---------  ------  ---------  ------
0          1       [1, 2, 3]  [1, 3]
1          3       [2, 3, 1]  [3, 1]
2          3       [3, 1, 2]  [3, 1]
3          3       [3, 1, 2]  [3, 1]
4          1       [1, 2, 3]  [1, 3]
5          3       [2, 3, 1]  [3, 1]

What just happened

Step 1 showed the cluster as a client sees it: three brokers, each at its advertised address. Step 2 showed the controller quorum: one active controller, two standbys with a full copy of the metadata log. Steps 3 and 4 showed a topic split into partitions, each with a leader and two followers, and the leaders spread evenly. In the break-it step, client A failed exactly like the gateway: its only bootstrap address was down, so it never received the metadata that names the other brokers. Client B started through kafka-1 or kafka-3. The controllers noticed kafka-2 was gone and moved its two partition leaderships to the remaining brokers. Broker 2 also dropped out of every ISR list. The topic kept working with two copies of each partition.

Clean up

The last cell starts kafka-2 again (docker compose up -d --wait kafka-2) and deletes the topic. Broker 2 rejoins the ISR lists within seconds. It takes back its leaderships a few minutes later, when Kafka rebalances leaders automatically (lesson 4.1). If anything gets stuck, docker compose down -v in the code/ folder resets the whole lab.

Key takeaways

Next: Records, offsets and replay