Notebook · code/module-04/01-leaders-followers-and-failover.ipynb

Download 01-leaders-followers-and-failover.ipynb

Open it in JupyterLab or VS Code from the lab folder to run it. This page is a read-only view.

Leaders, followers and failover — practical

A writer sends 50 meter readings a second while you stop a leader cleanly, crash another one, move the leaders back, and finally take away the controllers' majority. At the end you count whether a single reading was lost or written twice.

Lab services: kafka-1, kafka-2, kafka-3 · Script: module-04/steady_writer.py · Lesson: modules/04-inside-the-cluster/01-leaders-followers-and-failover.md

# Setup: find the lab's shared helpers and check the lab is running
import sys, pathlib
lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / "common").is_dir())
sys.path.insert(0, str(lab_root))

import json, re, time
from datetime import datetime
from common import settings, lab

lab.require_kafka()
lab.print_versions()

Step 1: A steady writer, and who leads what

The topic has 3 partitions with 3 replicas each. The writer runs in the background with the safe settings (acks=all, idempotence on) and logs one line per second.

from confluent_kafka import TopicPartition, ElectionType
from confluent_kafka.admin import AdminClient

TOPIC = "m04l01-telemetry"
admin = AdminClient({"bootstrap.servers": settings.KAFKA_BOOTSTRAP})
lab.recreate_topic(admin, TOPIC, partitions=3)
writer = lab.start_script("module-04/steady_writer.py", "--topic", TOPIC, "--rate", "50", name="m04l01-writer")

def layout():
    """Per partition: the leader, the preferred leader (first replica) and the in-sync replicas."""
    partitions = admin.list_topics(TOPIC, timeout=10).topics[TOPIC].partitions   # any broker answers
    return {p.id: {"leader": p.leader, "preferred": p.replicas[0], "isr": sorted(p.isrs)}
            for p in sorted(partitions.values(), key=lambda p: p.id)}

time.sleep(5)
for partition, info in layout().items():
    print(partition, info)

Each partition has a different leader, and right now each leader is also its preferred leader.

writer_report(since) reads the writer's log from a moment on and prints the seconds that weren't normal: fewer than 40 deliveries, or a delivery slower than 100 ms.

LINE = re.compile(r"^(\d\d:\d\d:\d\d) sent=(\d+) delivered=(\d+) waiting=(\d+) slowest=(\d+)ms(.*)")

def writer_report(since):
    lines = [m for m in (LINE.match(l) for l in writer.lines()) if m and m.group(1) >= since]
    odd = [m.group(0) for m in lines if int(m.group(3)) < 40 or int(m.group(5)) > 100]
    for line in odd:
        print("  " + line)
    stalled = sum(1 for m in lines if int(m.group(3)) == 0)
    slowest = max((int(m.group(5)) for m in lines), default=0)
    print(f"seconds with no delivery: {stalled}   slowest delivery: {slowest} ms")

now = lambda: datetime.now().strftime("%H:%M:%S")

Step 2: Stop a leader cleanly

docker compose stop sends the broker a polite shutdown signal, like a planned restart. Predict first: how long do writes to partition 0 pause?

leader = layout()[0]["leader"]
since = now()
lab.compose("stop", f"kafka-{leader}")
for partition, info in layout().items():
    print(partition, info)
time.sleep(5)
writer_report(since)

Every partition already has a new leader and the stopped broker has left the ISRs. The writer barely noticed: a few hundred milliseconds at most.

Step 3: Bring it back, then move the leaders back

The broker restarts and catches up. Is it the leader of anything?

lab.compose("up", "-d", "--wait", f"kafka-{leader}")
for partition, info in layout().items():
    print(partition, info)

It's back in every ISR, but (unless the controller's 5-minute balancing check happened to run) it leads nothing: its partitions' leader is not the preferred one. A preferred-leader election moves leadership back:

result = admin.elect_leaders(ElectionType.PREFERRED, [TopicPartition(TOPIC, p) for p in range(3)]).result()
print({tp.partition: (err.str() if err else "ok") for tp, err in result.items()})
for partition, info in layout().items():
    print(partition, info)

A partition whose leader was already the preferred one reports Election not needed.

Step 4: Crash a leader

Now docker compose kill: the broker process dies at once, like a power cut. Predict first: does the pause look like step 2?

leader = layout()[0]["leader"]
since, crashed = now(), time.monotonic()
lab.compose("kill", f"kafka-{leader}")
last = None
while time.monotonic() - crashed < 15:
    current = layout()
    if current != last:
        print(f"+{time.monotonic() - crashed:4.1f}s", {p: (i["leader"], i["isr"]) for p, i in current.items()})
        last = current
    time.sleep(0.5)
writer_report(since)

For about 9 seconds the cluster still lists the dead broker as a leader and as in sync, and no reading is delivered, on any partition. Then the controller fences the broker (it has missed heartbeats for broker.session.timeout.ms, 9 s), elects new leaders and shrinks the ISRs. Writes resume, and the readings that waited are delivered.

Restart the broker and move the leaders back:

lab.compose("up", "-d", "--wait", f"kafka-{leader}")
time.sleep(3)
admin.elect_leaders(ElectionType.PREFERRED, [TopicPartition(TOPIC, p) for p in range(3)]).result()
for partition, info in layout().items():
    print(partition, info)

Break it: lose the controllers' majority

Each lab node also runs a controller. Kill two of the three nodes: one controller is left, with no majority. Predict first: what still works?

since = now()
lab.compose("kill", "kafka-2", "kafka-3")
time.sleep(15)
from confluent_kafka.admin import NewTopic
try:
    admin.create_topics([NewTopic("m04l01-new-topic", 3, 3)], request_timeout=10)["m04l01-new-topic"].result()
except Exception as e:
    print("create topic:", e)
writer_report(since)

Nothing works: no topic can be created, and no reading is delivered, not even to the partitions kafka-1 leads. Restart one node, and the controllers have a majority again (2 of 3):

since, restarted = now(), time.monotonic()
lab.compose("up", "-d", "--wait", "kafka-2")
lab.wait_until(lambda: any(m and m.group(1) > since and int(m.group(3)) > 0
                           for m in (LINE.match(l) for l in writer.lines())), timeout=60, what="deliveries")
print(f"deliveries resumed {time.monotonic() - restarted:.0f} s after the restart command")
lab.compose("up", "-d", "--wait", "kafka-3")
print("all three nodes are back")

Step 5: Was anything lost or duplicated?

Stop the writer. Its last line says how many readings it sent and how many were confirmed. Then read the whole topic and check the sequence numbers.

from confluent_kafka import Consumer

lab.stop_script(writer)
print(writer.lines("stopped")[-1])
consumer = Consumer({"bootstrap.servers": settings.KAFKA_BOOTSTRAP, "group.id": "m04l01-check",
                     "enable.auto.commit": False})
ends = {p: consumer.get_watermark_offsets(TopicPartition(TOPIC, p), timeout=10)[1] for p in range(3)}
consumer.assign([TopicPartition(TOPIC, p, 0) for p in range(3)])
seqs = []
while len(seqs) < sum(ends.values()):
    msg = consumer.poll(10)
    if msg is None:
        break
    if not msg.error():
        seqs.append(json.loads(msg.value())["seq"])
consumer.close()
print(f"records in the topic: {len(seqs)}   different readings: {len(set(seqs))}")

You should see the same number three times: sent, confirmed and in the topic, with no duplicates. Two leader changes and a frozen cluster cost time, not data, because every outage was shorter than the writer's delivery timeout (120 s).

Clean up

Delete the topic. If anything gets stuck, docker compose down -v in the code/ folder resets the whole lab.

admin = AdminClient({"bootstrap.servers": settings.KAFKA_BOOTSTRAP})   # a fresh client after the outage
print("the topic from the break-it step exists now:", "m04l01-new-topic" in admin.list_topics(timeout=10).topics)
lab.delete_topics(admin, [TOPIC, "m04l01-new-topic"])
admin.delete_consumer_groups(["m04l01-check"])
print("cleaned up")