Download 01-leaders-followers-and-failover.ipynb
Open it in JupyterLab or VS Code from the lab folder to run it. This page is a read-only view.
Leaders, followers and failover — practical
A writer sends 50 meter readings a second while you stop a leader cleanly, crash another one, move the leaders back, and finally take away the controllers' majority. At the end you count whether a single reading was lost or written twice.
Lab services: kafka-1, kafka-2, kafka-3 · Script: module-04/steady_writer.py
· Lesson: modules/04-inside-the-cluster/01-leaders-followers-and-failover.md
# Setup: find the lab's shared helpers and check the lab is running import sys, pathlib lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / "common").is_dir()) sys.path.insert(0, str(lab_root)) import json, re, time from datetime import datetime from common import settings, lab lab.require_kafka() lab.print_versions()
Step 1: A steady writer, and who leads what
The topic has 3 partitions with 3 replicas each. The writer runs in the background with the safe
settings (acks=all, idempotence on) and logs one line per second.
from confluent_kafka import TopicPartition, ElectionType from confluent_kafka.admin import AdminClient TOPIC = "m04l01-telemetry" admin = AdminClient({"bootstrap.servers": settings.KAFKA_BOOTSTRAP}) lab.recreate_topic(admin, TOPIC, partitions=3) writer = lab.start_script("module-04/steady_writer.py", "--topic", TOPIC, "--rate", "50", name="m04l01-writer") def layout(): """Per partition: the leader, the preferred leader (first replica) and the in-sync replicas.""" partitions = admin.list_topics(TOPIC, timeout=10).topics[TOPIC].partitions # any broker answers return {p.id: {"leader": p.leader, "preferred": p.replicas[0], "isr": sorted(p.isrs)} for p in sorted(partitions.values(), key=lambda p: p.id)} time.sleep(5) for partition, info in layout().items(): print(partition, info)
Each partition has a different leader, and right now each leader is also its preferred leader.
writer_report(since) reads the writer's log from a moment on and prints the seconds that weren't
normal: fewer than 40 deliveries, or a delivery slower than 100 ms.
LINE = re.compile(r"^(\d\d:\d\d:\d\d) sent=(\d+) delivered=(\d+) waiting=(\d+) slowest=(\d+)ms(.*)") def writer_report(since): lines = [m for m in (LINE.match(l) for l in writer.lines()) if m and m.group(1) >= since] odd = [m.group(0) for m in lines if int(m.group(3)) < 40 or int(m.group(5)) > 100] for line in odd: print(" " + line) stalled = sum(1 for m in lines if int(m.group(3)) == 0) slowest = max((int(m.group(5)) for m in lines), default=0) print(f"seconds with no delivery: {stalled} slowest delivery: {slowest} ms") now = lambda: datetime.now().strftime("%H:%M:%S")
Step 2: Stop a leader cleanly
docker compose stop sends the broker a polite shutdown signal, like a planned restart.
Predict first: how long do writes to partition 0 pause?
leader = layout()[0]["leader"] since = now() lab.compose("stop", f"kafka-{leader}") for partition, info in layout().items(): print(partition, info) time.sleep(5) writer_report(since)
Every partition already has a new leader and the stopped broker has left the ISRs. The writer barely noticed: a few hundred milliseconds at most.
Step 3: Bring it back, then move the leaders back
The broker restarts and catches up. Is it the leader of anything?
lab.compose("up", "-d", "--wait", f"kafka-{leader}") for partition, info in layout().items(): print(partition, info)
It's back in every ISR, but (unless the controller's 5-minute balancing check happened to run) it leads nothing: its partitions' leader is not the preferred one. A preferred-leader election moves leadership back:
result = admin.elect_leaders(ElectionType.PREFERRED, [TopicPartition(TOPIC, p) for p in range(3)]).result() print({tp.partition: (err.str() if err else "ok") for tp, err in result.items()}) for partition, info in layout().items(): print(partition, info)
A partition whose leader was already the preferred one reports Election not needed.
Step 4: Crash a leader
Now docker compose kill: the broker process dies at once, like a power cut. Predict first: does
the pause look like step 2?
leader = layout()[0]["leader"] since, crashed = now(), time.monotonic() lab.compose("kill", f"kafka-{leader}") last = None while time.monotonic() - crashed < 15: current = layout() if current != last: print(f"+{time.monotonic() - crashed:4.1f}s", {p: (i["leader"], i["isr"]) for p, i in current.items()}) last = current time.sleep(0.5) writer_report(since)
For about 9 seconds the cluster still lists the dead broker as a leader and as in sync, and no
reading is delivered, on any partition. Then the controller fences the broker (it has missed
heartbeats for broker.session.timeout.ms, 9 s), elects new leaders and shrinks the ISRs.
Writes resume, and the readings that waited are delivered.
Restart the broker and move the leaders back:
lab.compose("up", "-d", "--wait", f"kafka-{leader}") time.sleep(3) admin.elect_leaders(ElectionType.PREFERRED, [TopicPartition(TOPIC, p) for p in range(3)]).result() for partition, info in layout().items(): print(partition, info)
Break it: lose the controllers' majority
Each lab node also runs a controller. Kill two of the three nodes: one controller is left, with no majority. Predict first: what still works?
since = now() lab.compose("kill", "kafka-2", "kafka-3") time.sleep(15) from confluent_kafka.admin import NewTopic try: admin.create_topics([NewTopic("m04l01-new-topic", 3, 3)], request_timeout=10)["m04l01-new-topic"].result() except Exception as e: print("create topic:", e) writer_report(since)
Nothing works: no topic can be created, and no reading is delivered, not even to the partitions kafka-1 leads. Restart one node, and the controllers have a majority again (2 of 3):
since, restarted = now(), time.monotonic() lab.compose("up", "-d", "--wait", "kafka-2") lab.wait_until(lambda: any(m and m.group(1) > since and int(m.group(3)) > 0 for m in (LINE.match(l) for l in writer.lines())), timeout=60, what="deliveries") print(f"deliveries resumed {time.monotonic() - restarted:.0f} s after the restart command") lab.compose("up", "-d", "--wait", "kafka-3") print("all three nodes are back")
Step 5: Was anything lost or duplicated?
Stop the writer. Its last line says how many readings it sent and how many were confirmed. Then read the whole topic and check the sequence numbers.
from confluent_kafka import Consumer lab.stop_script(writer) print(writer.lines("stopped")[-1]) consumer = Consumer({"bootstrap.servers": settings.KAFKA_BOOTSTRAP, "group.id": "m04l01-check", "enable.auto.commit": False}) ends = {p: consumer.get_watermark_offsets(TopicPartition(TOPIC, p), timeout=10)[1] for p in range(3)} consumer.assign([TopicPartition(TOPIC, p, 0) for p in range(3)]) seqs = [] while len(seqs) < sum(ends.values()): msg = consumer.poll(10) if msg is None: break if not msg.error(): seqs.append(json.loads(msg.value())["seq"]) consumer.close() print(f"records in the topic: {len(seqs)} different readings: {len(set(seqs))}")
You should see the same number three times: sent, confirmed and in the topic, with no duplicates. Two leader changes and a frozen cluster cost time, not data, because every outage was shorter than the writer's delivery timeout (120 s).
Clean up
Delete the topic. If anything gets stuck, docker compose down -v in the code/ folder resets
the whole lab.
admin = AdminClient({"bootstrap.servers": settings.KAFKA_BOOTSTRAP}) # a fresh client after the outage print("the topic from the break-it step exists now:", "m04l01-new-topic" in admin.list_topics(timeout=10).topics) lab.delete_topics(admin, [TOPIC, "m04l01-new-topic"]) admin.delete_consumer_groups(["m04l01-check"]) print("cleaned up")