32 min · theory + hands-on practical

Leaders, followers and failover

Lesson 2.2 showed what a producer needs so that a broker failure loses nothing. This lesson looks at the failure from the cluster's side: how followers keep up, how a new leader is chosen, why a crash hurts more than a planned stop, and what happens when there aren't enough controllers left.

Theory

Example: two restarts, two very different minutes

On a Tuesday at 18:10, at the evening peak, the power supply of the server running kafka-2 failed. For about 10 seconds the charger gateway got no confirmation for any reading, on any partition, not only the two partitions kafka-2 led. Then everything resumed, and no reading was lost.

The next week, the platform team restarted kafka-1 for a planned upgrade. The gateway's dashboard showed nothing at all. The same cluster, the same traffic, one broker gone each time. This lesson explains the difference.

Followers and the ISR

Every partition has one leader and, at Brightlane, two followers (lesson 1.2). A follower copies the leader the way a consumer would: it sends fetch requests, "give me everything from offset 209". Each fetch also tells the leader how far that follower has got.

Partition 0 has three replicas. kafka-2 is the leader with records 200 to 209. Follower kafka-1 has records up to 208 and fetches from 209; follower kafka-3 has records up to 206 and fetches from 207. The high watermark is 207: every in-sync replica has the records before it, so consumers can read up to 206.

What to notice: the high watermark follows the slowest in-sync follower, so one slow follower holds back what consumers can read.

The ISR is the list of replicas that are caught up. A follower that hasn't caught up with the leader for replica.lag.time.max.ms (30 seconds) is dropped from it, and joins again when it has caught up. The high watermark is the first offset that not every in-sync replica has yet. Consumers only read below it, and acks=all writes are confirmed once it has passed them.

How a new leader is chosen

The active controller (lesson 1.2) decides who leads each partition. Every broker sends it a heartbeat every 2 seconds (broker.heartbeat.interval.ms). A broker that stays silent for broker.session.timeout.ms (9 seconds) is fenced: the controller treats it as gone, removes it from every ISR, and gives each partition it led a new leader from the remaining ISR. Clients learn about the new leader from their next metadata request, after the old leader stops answering or answers NOT_LEADER_OR_FOLLOWER. The idempotent producer retries without creating duplicates.

How long that takes depends on how the broker went away:

Top: a clean stop. kafka-2 asks the controller to move its leaders, the controller makes kafka-1 and kafka-3 leaders, then kafka-2 exits; writes pause for about 0.1 seconds. Bottom: a crash. kafka-2 dies; acks=all writes wait on every partition with a replica on kafka-2; after 9 seconds without heartbeats the controller fences kafka-2; new leaders are elected and the ISRs shrink; writes pause for about 9 to 10 seconds.

What to notice: during a crash, the dead broker is still in every ISR, so acks=all writes wait even on partitions it doesn't lead.

When no in-sync replica is left

A new leader taken from the ISR has every acknowledged record. That's a clean election. But if every ISR member is gone, the controller has only out-of-sync replicas to choose from. With unclean.leader.election.enable=false (the default), it waits and the partition stays offline. With true, it elects an out-of-sync replica: the partition is back at once, but acknowledged records that replica never received are lost for good.

Kafka 4.1 added a safer middle way, eligible leader replicas (ELR, KIP-966), on by default for clusters created with 4.1 or later. When the ISR shrinks below min.insync.replicas, the replicas that leave it go on a second list, the ELR. Since the high watermark stops moving in that state (lesson 2.2), they still have every acknowledged record. If the last ISR member then fails, the controller can elect one of them without losing anything.

Preferred leaders

The first replica in a partition's replica list is its preferred leader. When a topic is created, Kafka spreads the preferred leaders evenly over the brokers. After a failover and restart, the restarted broker leads nothing, and the other brokers do its share of the work.

The controller fixes that by itself: with auto.leader.rebalance.enable=true (the default), it checks every 5 minutes (leader.imbalance.check.interval.seconds) and moves each partition back to its preferred leader if that replica is in sync. (Older guides mention a 10% threshold, leader.imbalance.per.broker.percentage; Kafka 4.0 removed it.) You can trigger the same preferred-leader election at once with kafka-leader-election.sh or the admin client.

When the controllers lose their majority

Every change to the metadata needs a majority of the controllers: 2 of 3. That includes electing a leader, shrinking an ISR and creating a topic. With only one controller left, nothing can change. Brokers keep the metadata they last had. Partitions led by the dead brokers have no working leader, and on the others acks=all writes wait, because no ISR can shrink. As soon as a second controller is back, the cluster continues.

In production

Practical

Note

Script: code/module-04/steady_writer.py (the notebook starts it) · Notebook: code/module-04/01-leaders-followers-and-failover.ipynb · Lab services: kafka-1, kafka-2, kafka-3 · Time: ~18 min (about 3 minutes of waiting)

What you'll do: keep a writer running at 50 readings a second, stop a leader cleanly, crash another, move the leaders back, and take away the controllers' majority. Then count every reading.

Step 1: A steady writer, and who leads what

The writer uses the course's safe producer settings, with a 2-minute delivery timeout:

code/module-04/steady_writer.py
    producer = Producer({**settings.PRODUCER_DEFAULTS, "message.timeout.ms": args.timeout_ms})

layout() asks for the topic's metadata, which any broker can answer, even without a controller:

code/module-04/01-leaders-followers-and-failover.ipynb
def layout():
    """Per partition: the leader, the preferred leader (first replica) and the in-sync replicas."""
    partitions = admin.list_topics(TOPIC, timeout=10).topics[TOPIC].partitions   # any broker answers
    return {p.id: {"leader": p.leader, "preferred": p.replicas[0], "isr": sorted(p.isrs)}
            for p in sorted(partitions.values(), key=lambda p: p.id)}
0 {'leader': 2, 'preferred': 2, 'isr': [1, 2, 3]}
1 {'leader': 3, 'preferred': 3, 'isr': [1, 2, 3]}
2 {'leader': 1, 'preferred': 1, 'isr': [1, 2, 3]}

writer_report(since) prints the writer's log lines that weren't normal: fewer than 40 readings delivered in a second, or one slower than 100 ms.

Step 2: Stop a leader cleanly

code/module-04/01-leaders-followers-and-failover.ipynb
lab.compose("stop", f"kafka-{leader}")
$ docker compose stop kafka-2
0 {'leader': 3, 'preferred': 2, 'isr': [1, 3]}
1 {'leader': 3, 'preferred': 3, 'isr': [1, 3]}
2 {'leader': 1, 'preferred': 1, 'isr': [1, 3]}
  17:52:06 sent=51 delivered=51 waiting=1 slowest=141ms
seconds with no delivery: 0   slowest delivery: 141 ms

Step 3: Bring it back, then move the leaders back

After the restart, kafka-2 is in every ISR again but leads nothing:

0 {'leader': 3, 'preferred': 2, 'isr': [1, 2, 3]}
code/module-04/01-leaders-followers-and-failover.ipynb
result = admin.elect_leaders(ElectionType.PREFERRED, [TopicPartition(TOPIC, p) for p in range(3)]).result()
{0: 'ok', 1: 'Leader election not needed for topic partition.', 2: 'Leader election not needed for topic partition.'}
0 {'leader': 2, 'preferred': 2, 'isr': [1, 2, 3]}

If the controller's 5-minute check happened to run first, partition 0 also says "not needed".

Step 4: Crash a leader

Predict first: docker compose kill ends the broker at once, like a power cut. Does the pause look like step 2?

code/module-04/01-leaders-followers-and-failover.ipynb
lab.compose("kill", f"kafka-{leader}")

The layout whenever it changed, then the writer's report:

+ 0.2s {0: (2, [1, 2, 3]), 1: (3, [1, 2, 3]), 2: (1, [1, 2, 3])}
+ 9.8s {0: (3, [1, 3]), 1: (3, [1, 3]), 2: (1, [1, 3])}
  17:52:21 sent=51 delivered=35 waiting=17 slowest=24ms
  17:52:22 sent=51 delivered=0 waiting=68 slowest=0ms
  ...
  17:52:30 sent=50 delivered=0 waiting=472 slowest=0ms
  17:52:31 sent=51 delivered=522 waiting=1 slowest=9721ms
seconds with no delivery: 9   slowest delivery: 9721 ms

Break it: lose the controllers' majority

Predict first: you kill two of the three nodes, so one controller is left. What still works?

code/module-04/01-leaders-followers-and-failover.ipynb
lab.compose("kill", "kafka-2", "kafka-3")
create topic: KafkaError{code=_TIMED_OUT,val=-185,str="Failed while waiting for response from broker: Local: Timed out"}
  17:52:46 sent=51 delivered=0 waiting=55 slowest=0ms
  ...
  17:53:09 sent=51 delivered=0 waiting=1217 slowest=0ms
seconds with no delivery: 24   slowest delivery: 24 ms

Then the notebook restarts kafka-2:

$ docker compose up -d --wait kafka-2
deliveries resumed 18 s after the restart command

Step 5: Was anything lost or duplicated?

17:53:35 stopped: sent=4657 delivered=4657 failed=0 not answered=0
records in the topic: 4657   different readings: 4657

What just happened

The clean stop was a controlled shutdown: kafka-2 handed its leaderships over before it exited, so the worst delivery took 141 ms. After the restart, kafka-2 was back in the ISRs but led nothing, until the preferred-leader election moved partition 0 back. The crash was different. For 9.8 seconds the metadata still showed kafka-2 as leader of partition 0 and in every ISR, and not one reading was delivered on any partition: acks=all waited for a replica that was gone. When the controller fenced kafka-2, partition 0 got a new leader, every ISR shrank, and the 472 waiting readings went through at once. Killing two nodes left one controller without a majority. Nothing could be decided, so no topic could be created and no ISR could shrink, and all writes stopped until kafka-2 was back. Every outage was shorter than the writer's 2-minute delivery timeout, so all 4,657 readings were stored exactly once.

One more thing the last cell shows: m04l01-new-topic, the topic whose creation timed out during the break-it step, exists. The client gave up after 10 seconds, but the request was still pending in the cluster and went through once the controllers had a majority again. A timeout means "no answer in time", not "nothing happened": check before you retry an admin operation.

Clean up

The last cell deletes the topics. If anything gets stuck, docker compose down -v in the code/ folder resets the whole lab.

Key takeaways

Next: Segments, retention and deleting data