Leaders, followers and failover
Lesson 2.2 showed what a producer needs so that a broker failure loses nothing. This lesson looks at the failure from the cluster's side: how followers keep up, how a new leader is chosen, why a crash hurts more than a planned stop, and what happens when there aren't enough controllers left.
Theory
Example: two restarts, two very different minutes
On a Tuesday at 18:10, at the evening peak, the power supply of the server running kafka-2 failed. For about 10 seconds the charger gateway got no confirmation for any reading, on any partition, not only the two partitions kafka-2 led. Then everything resumed, and no reading was lost.
The next week, the platform team restarted kafka-1 for a planned upgrade. The gateway's dashboard showed nothing at all. The same cluster, the same traffic, one broker gone each time. This lesson explains the difference.
Followers and the ISR
Every partition has one leader and, at Brightlane, two followers (lesson 1.2). A follower copies the leader the way a consumer would: it sends fetch requests, "give me everything from offset 209". Each fetch also tells the leader how far that follower has got.
What to notice: the high watermark follows the slowest in-sync follower, so one slow follower holds back what consumers can read.
The ISR is the list of replicas that are caught up. A follower that hasn't caught up with the
leader for replica.lag.time.max.ms (30 seconds) is dropped from it, and joins again when it has
caught up. The high watermark is the first offset that not every in-sync replica has yet.
Consumers only read below it, and acks=all writes are confirmed once it has passed them.
How a new leader is chosen
The active controller (lesson 1.2) decides who leads each partition. Every broker sends it a
heartbeat every 2 seconds (broker.heartbeat.interval.ms). A broker that stays silent for
broker.session.timeout.ms (9 seconds) is fenced: the controller treats it as gone, removes it
from every ISR, and gives each partition it led a new leader from the remaining ISR. Clients learn
about the new leader from their next metadata request, after the old leader stops answering or
answers NOT_LEADER_OR_FOLLOWER. The idempotent producer retries without creating duplicates.
How long that takes depends on how the broker went away:
What to notice: during a crash, the dead broker is still in every ISR, so acks=all writes wait
even on partitions it doesn't lead.
- A clean stop (a polite shutdown signal,
SIGTERM, asdocker compose stopor Kubernetes sends) starts a controlled shutdown. The broker asks the controller to move its leaderships first, then exits. Clients barely notice. - A crash (power loss,
kill -9, a frozen machine) gives nobody any warning. For the next 9 seconds the dead broker is still a leader and still in every ISR. Partitions it led accept nothing, and on the othersacks=allwrites wait for a follower that will never answer.
When no in-sync replica is left
A new leader taken from the ISR has every acknowledged record. That's a clean election. But if
every ISR member is gone, the controller has only out-of-sync replicas to choose from. With
unclean.leader.election.enable=false (the default), it waits and the partition stays offline. With
true, it elects an out-of-sync replica: the partition is back at once, but acknowledged records
that replica never received are lost for good.
Kafka 4.1 added a safer middle way, eligible leader replicas (ELR, KIP-966), on by default for
clusters created with 4.1 or later. When the ISR shrinks below min.insync.replicas, the replicas
that leave it go on a second list, the ELR. Since the high watermark stops moving in that state
(lesson 2.2), they still have every acknowledged record. If the last ISR member then fails, the
controller can elect one of them without losing anything.
Preferred leaders
The first replica in a partition's replica list is its preferred leader. When a topic is created, Kafka spreads the preferred leaders evenly over the brokers. After a failover and restart, the restarted broker leads nothing, and the other brokers do its share of the work.
The controller fixes that by itself: with auto.leader.rebalance.enable=true (the default), it
checks every 5 minutes (leader.imbalance.check.interval.seconds) and moves each partition back to
its preferred leader if that replica is in sync. (Older guides mention a 10% threshold,
leader.imbalance.per.broker.percentage; Kafka 4.0 removed it.) You can trigger the same
preferred-leader election at once with kafka-leader-election.sh or the admin client.
When the controllers lose their majority
Every change to the metadata needs a majority of the controllers: 2 of 3. That includes electing a
leader, shrinking an ISR and creating a topic. With only one controller left, nothing can change.
Brokers keep the metadata they last had. Partitions led by the dead brokers have no working leader,
and on the others acks=all writes wait, because no ISR can shrink. As soon as a second controller
is back, the cluster continues.
In production
- Restart brokers one at a time, with a clean stop. Before the next one, wait until no partition is
under-replicated (
kafka-topics.sh --describe --under-replicated-partitionsprints nothing). - Keep
unclean.leader.election.enable=falsefor any topic whose data you can't lose. - Give producers a delivery timeout longer than any failover you expect:
message.timeout.msin the Python client (5 minutes by default). Handle the readings that still fail. - Run 3 or 5 controllers on separate machines, ideally in separate zones. Set
broker.rackso the replicas of each partition land in different racks or zones. - Alert on under-replicated and offline partitions, on an active controller count other than 1, and on leader imbalance (lesson 8.3).
Practical
Note
Script: code/module-04/steady_writer.py (the notebook starts it)
· Notebook: code/module-04/01-leaders-followers-and-failover.ipynb
· Lab services: kafka-1, kafka-2, kafka-3 · Time: ~18 min (about 3 minutes of waiting)
What you'll do: keep a writer running at 50 readings a second, stop a leader cleanly, crash another, move the leaders back, and take away the controllers' majority. Then count every reading.
Step 1: A steady writer, and who leads what
The writer uses the course's safe producer settings, with a 2-minute delivery timeout:
producer = Producer({**settings.PRODUCER_DEFAULTS, "message.timeout.ms": args.timeout_ms})
layout() asks for the topic's metadata, which any broker can answer, even without a controller:
def layout():
"""Per partition: the leader, the preferred leader (first replica) and the in-sync replicas."""
partitions = admin.list_topics(TOPIC, timeout=10).topics[TOPIC].partitions # any broker answers
return {p.id: {"leader": p.leader, "preferred": p.replicas[0], "isr": sorted(p.isrs)}
for p in sorted(partitions.values(), key=lambda p: p.id)}
0 {'leader': 2, 'preferred': 2, 'isr': [1, 2, 3]}
1 {'leader': 3, 'preferred': 3, 'isr': [1, 2, 3]}
2 {'leader': 1, 'preferred': 1, 'isr': [1, 2, 3]}
writer_report(since) prints the writer's log lines that weren't normal: fewer than 40 readings
delivered in a second, or one slower than 100 ms.
Step 2: Stop a leader cleanly
lab.compose("stop", f"kafka-{leader}")
$ docker compose stop kafka-2
0 {'leader': 3, 'preferred': 2, 'isr': [1, 3]}
1 {'leader': 3, 'preferred': 3, 'isr': [1, 3]}
2 {'leader': 1, 'preferred': 1, 'isr': [1, 3]}
17:52:06 sent=51 delivered=51 waiting=1 slowest=141ms
seconds with no delivery: 0 slowest delivery: 141 ms
Step 3: Bring it back, then move the leaders back
After the restart, kafka-2 is in every ISR again but leads nothing:
0 {'leader': 3, 'preferred': 2, 'isr': [1, 2, 3]}
result = admin.elect_leaders(ElectionType.PREFERRED, [TopicPartition(TOPIC, p) for p in range(3)]).result()
{0: 'ok', 1: 'Leader election not needed for topic partition.', 2: 'Leader election not needed for topic partition.'}
0 {'leader': 2, 'preferred': 2, 'isr': [1, 2, 3]}
If the controller's 5-minute check happened to run first, partition 0 also says "not needed".
Step 4: Crash a leader
Predict first: docker compose kill ends the broker at once, like a power cut. Does the pause look
like step 2?
lab.compose("kill", f"kafka-{leader}")
The layout whenever it changed, then the writer's report:
+ 0.2s {0: (2, [1, 2, 3]), 1: (3, [1, 2, 3]), 2: (1, [1, 2, 3])}
+ 9.8s {0: (3, [1, 3]), 1: (3, [1, 3]), 2: (1, [1, 3])}
17:52:21 sent=51 delivered=35 waiting=17 slowest=24ms
17:52:22 sent=51 delivered=0 waiting=68 slowest=0ms
...
17:52:30 sent=50 delivered=0 waiting=472 slowest=0ms
17:52:31 sent=51 delivered=522 waiting=1 slowest=9721ms
seconds with no delivery: 9 slowest delivery: 9721 ms
Break it: lose the controllers' majority
Predict first: you kill two of the three nodes, so one controller is left. What still works?
lab.compose("kill", "kafka-2", "kafka-3")
create topic: KafkaError{code=_TIMED_OUT,val=-185,str="Failed while waiting for response from broker: Local: Timed out"}
17:52:46 sent=51 delivered=0 waiting=55 slowest=0ms
...
17:53:09 sent=51 delivered=0 waiting=1217 slowest=0ms
seconds with no delivery: 24 slowest delivery: 24 ms
Then the notebook restarts kafka-2:
$ docker compose up -d --wait kafka-2
deliveries resumed 18 s after the restart command
Step 5: Was anything lost or duplicated?
17:53:35 stopped: sent=4657 delivered=4657 failed=0 not answered=0
records in the topic: 4657 different readings: 4657
What just happened
The clean stop was a controlled shutdown: kafka-2 handed its leaderships over before it exited,
so the worst delivery took 141 ms. After the restart, kafka-2 was back in the ISRs but led nothing,
until the preferred-leader election moved partition 0 back. The crash was different. For 9.8
seconds the metadata still showed kafka-2 as leader of partition 0 and in every ISR, and not one
reading was delivered on any partition: acks=all waited for a replica that was gone. When the
controller fenced kafka-2, partition 0 got a new leader, every ISR shrank, and the 472 waiting
readings went through at once. Killing two nodes left one controller without a majority. Nothing
could be decided, so no topic could be created and no ISR could shrink, and all writes stopped until
kafka-2 was back. Every outage was shorter than the writer's 2-minute delivery timeout, so all 4,657
readings were stored exactly once.
One more thing the last cell shows: m04l01-new-topic, the topic whose creation timed out during
the break-it step, exists. The client gave up after 10 seconds, but the request was still pending in
the cluster and went through once the controllers had a majority again. A timeout means "no answer
in time", not "nothing happened": check before you retry an admin operation.
Clean up
The last cell deletes the topics. If anything gets stuck, docker compose down -v in the code/
folder resets the whole lab.
Key takeaways
- Followers fetch from the leader; one that stays behind for 30 s leaves the ISR, and the high watermark follows the slowest in-sync replica.
- A clean stop moves leaders first (a pause of milliseconds); a crash stalls
acks=allwrites until the broker is fenced after 9 s. - New leaders come from the ISR. Unclean election trades acknowledged records for availability; ELR (4.1+) avoids many of those cases safely.
- After restarts, preferred-leader elections spread the work again, automatically every 5 minutes or on demand.
- Without a controller majority nothing changes: no elections, no ISR changes, no new topics, and
acks=allwrites stop.