Download 02-durable-writes.ipynb
Open it in JupyterLab or VS Code from the lab folder to run it. This page is a read-only view.
Durable writes: acks, replicas and idempotence — practical
You write meter readings with delivery reports, then take a broker away and watch what
acks=all, min.insync.replicas, acks=1 and acks=0 each do, including what they don't tell you.
Lab services: kafka-1, kafka-2, kafka-3 · Lesson: modules/02-producers/02-durable-writes.md
# Setup: find the lab's shared helpers and check the lab is running import sys, pathlib lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / "common").is_dir()) sys.path.insert(0, str(lab_root)) import json, time from collections import Counter from common import settings, lab from common.scenario import Brightlane, to_json lab.require_kafka() lab.print_versions()
Step 1: Write with acks=all and an idempotent producer
One partition, three replicas. The delivery report callback tells you, for every record,
whether it was stored and at which offset. message.timeout.ms is how long the client keeps
trying before it gives up: 5 minutes by default, 10 seconds here so the notebook doesn't wait.
from confluent_kafka import Producer, Consumer, TopicPartition, TopicCollection from confluent_kafka.admin import AdminClient, ConfigResource, ConfigEntry, AlterConfigOpType TOPIC = "m02l02-readings" admin = AdminClient({"bootstrap.servers": settings.KAFKA_BOOTSTRAP}) lab.recreate_topic(admin, TOPIC, partitions=1, replication=3) readings = [e for _, e in Brightlane().stream(sessions=20) if "type" not in e] def write(records, **config): """Write records with the given settings; return a Counter of outcomes and the offsets used.""" outcomes, offsets = Counter(), [] def report(err, msg): outcomes[err.name() if err else "delivered"] += 1 if not err and msg.offset() is not None: offsets.append(msg.offset()) p = Producer({"bootstrap.servers": settings.KAFKA_BOOTSTRAP, "message.timeout.ms": 10000, **config}) for r in records: p.produce(TOPIC, key=r["charger_id"], value=to_json(r), on_delivery=report) p.flush(20) return dict(outcomes), (min(offsets), max(offsets)) if offsets else None safe = {"acks": "all", "enable.idempotence": True} print(write(readings[:100], **safe))
You should see ({'delivered': 100}, (0, 99)).
Is the producer really idempotent? Its statistics (lesson 8.3) show the producer id the cluster gave it. The broker uses that id and a sequence number per batch to drop duplicates.
stats = [] p = Producer({"bootstrap.servers": settings.KAFKA_BOOTSTRAP, **safe, "statistics.interval.ms": 500, "stats_cb": stats.append}) p.produce(TOPIC, key="BL-N03-07", value=b"{}") p.flush(10) p.poll(1.0) # give the client time to emit one statistics report eos = json.loads(stats[-1])["eos"] print("idempotence state:", eos["idemp_state"], "| producer id:", eos["producer_id"], "| epoch:", eos["producer_epoch"])
You should see idempotence state: Assigned and a producer id (the number varies).
Step 2: Lose one broker
Stop a broker that holds a follower copy of the partition (not the leader), then write 100 more readings with the same settings.
def describe(): part = admin.describe_topics(TopicCollection([TOPIC]))[TOPIC].result().partitions[0] return part.leader.id, sorted(r.id for r in part.isr) leader, isr = describe() follower = next(b for b in (1, 2, 3) if b != leader) lab.compose("stop", f"kafka-{follower}") time.sleep(3) print("leader:", describe()[0], "| in-sync replicas:", describe()[1]) print(write(readings[100:200], **safe))
You should see two in-sync replicas and ({'delivered': 100}, (101, 200)). With
min.insync.replicas=2 (the lab's default, like production), two in-sync copies are enough.
Step 3: Demand three in-sync copies
Raise the topic's min.insync.replicas to 3, with only two replicas in sync, and write 20 more.
def set_min_isr(n): res = ConfigResource("topic", TOPIC, incremental_configs=[ ConfigEntry("min.insync.replicas", str(n), incremental_operation=AlterConfigOpType.SET)]) admin.incremental_alter_configs([res])[res].result() set_min_isr(3) time.sleep(2) start = time.monotonic() print(write(readings[200:220], **safe), f"after {time.monotonic() - start:.0f} s")
Every record fails with _MSG_TIMED_OUT, and only after 10 seconds. The broker refused each
attempt, but the reason was retriable, so the client kept retrying until its timeout ran out.
To see the broker's actual answer, write one record with retries switched off:
print(write(readings[220:221], **{"acks": "all", "enable.idempotence": False, "retries": 0}))
You should see NOT_ENOUGH_REPLICAS: fewer replicas are in sync than the topic demands.
Step 4: acks=1 in the same situation
Now write 20 readings with acks=1: only the leader has to store them. Then ask the partition
how far consumers can read (its high watermark).
c = Consumer({"bootstrap.servers": settings.KAFKA_BOOTSTRAP, "group.id": "m02l02-check"}) tp = TopicPartition(TOPIC, 0) print("readable up to (before):", c.get_watermark_offsets(tp, timeout=10)[1]) print(write(readings[221:241], acks="1")) print("readable up to (after): ", c.get_watermark_offsets(tp, timeout=10)[1])
The producer was told the 20 records were stored, at offsets 201 to 220. But the readable end didn't move: consumers can't see records until enough replicas have them. Bring the broker back and look again.
lab.compose("up", "-d", "--wait", f"kafka-{follower}") lab.wait_until(lambda: len(describe()[1]) == 3, timeout=60, what="all replicas in sync") print("readable up to (broker back):", c.get_watermark_offsets(tp, timeout=10)[1]) set_min_isr(2)
You should see the readable end jump to 221 once the stopped broker has caught up.
Break it: acks=0 never hears "no"
Predict first: the topic's maximum record size is lowered to 1,000 bytes. A producer with
acks=0 sends a 5,000-byte record. What does its delivery report say?
res = ConfigResource("topic", TOPIC, incremental_configs=[ ConfigEntry("max.message.bytes", "1000", incremental_operation=AlterConfigOpType.SET)]) admin.incremental_alter_configs([res])[res].result() time.sleep(2) big = [dict(readings[0], padding="x" * 5000)] end_before = c.get_watermark_offsets(tp, timeout=10)[1] print("acks=0: ", write(big, acks="0"), "| readable end:", end_before, "->", c.get_watermark_offsets(tp, timeout=10)[1]) print("acks=all:", write(big, **safe))
With acks=0 the report says delivered, but the readable end didn't move: the broker
rejected the record and nobody told the producer. With acks=all the same record fails with
MSG_SIZE_TOO_LARGE, and your code can react.
Clean up
All brokers are running again. Close the consumer and delete the topic. If anything gets stuck,
docker compose down -v in the code/ folder resets the whole lab.
c.close() lab.delete_topics(admin, [TOPIC]) admin.delete_consumer_groups(["m02l02-check"]) print("cleaned up")