Download 02-committing-offsets.ipynb
Open it in JupyterLab or VS Code from the lab folder to run it. This page is a read-only view.
Committing offsets: where processing resumes — practical
Billing charges 400 sessions into a ledger. You crash the billing worker halfway (kill -9),
restart it, and count the charges, once per commit strategy. Then you make the ledger
idempotent and get exactly 400.
Lab services: kafka-1, kafka-2, kafka-3 · Script: module-03/billing_worker.py
· Lesson: modules/03-consumers/02-committing-offsets.md
# Setup: find the lab's shared helpers and check the lab is running import sys, pathlib lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / "common").is_dir()) sys.path.insert(0, str(lab_root)) import sqlite3, time from common import settings, lab from common.scenario import Brightlane, to_json lab.require_kafka() lab.print_versions()
Step 1: Yesterday's sessions, waiting to be billed
400 sessions, so 400 session_stopped events among 800 events. A correct billing run writes
exactly 400 charges.
from confluent_kafka import Producer from confluent_kafka.admin import AdminClient TOPIC = "m03l02-sessions" admin = AdminClient({"bootstrap.servers": settings.KAFKA_BOOTSTRAP}) lab.recreate_topic(admin, TOPIC, partitions=3) producer = Producer(settings.PRODUCER_DEFAULTS) for charger_id, event in Brightlane().stream(sessions=400, readings=False): producer.produce(TOPIC, key=charger_id, value=to_json(event)) print("not yet stored:", producer.flush(30))
Step 2: A crash test
crash_test() starts a billing worker with a fresh group and an empty SQLite ledger. The worker
fetches 50 events at a time and spends 15 ms on each. When 150 charges are in the ledger, the
test kills it with kill -9, starts it again, waits until the group has caught up, and counts.
The workers use session.timeout.ms=6000, so the group notices the crashed member after 6
seconds instead of 45 (lesson 3.3 explains this setting).
DATA = lab_root / ".lab-data" DATA.mkdir(exist_ok=True) def crash_test(commit, idempotent=False): group = f"m03l02-{commit}" + ("-idempotent" if idempotent else "") admin.delete_consumer_groups([group]) ledger = DATA / f"{group}.db" ledger.unlink(missing_ok=True) args = ["--name", "W", "--topic", TOPIC, "--group", group, "--commit", commit, "--batch", "50", "--work-ms", "15", "--ledger", f".lab-data/{group}.db", "--session-timeout-ms", "6000"] args += ["--idempotent"] if idempotent else [] charges = lambda: sqlite3.connect(ledger).execute("SELECT count(*) FROM charges").fetchone()[0] if ledger.exists() else 0 first = lab.start_script("module-03/billing_worker.py", *args, name=f"{group}-1") lab.wait_until(lambda: charges() >= 150, timeout=60, interval=0.1, what="150 charges") lab.kill_script(first) # crash: no close(), no final commit second = lab.start_script("module-03/billing_worker.py", *args, name=f"{group}-2") lab.wait_until(lambda: sum(r[2] for r in lab.group_lag(admin, group, TOPIC, 3).values()) == 0, timeout=90, interval=1, what="the group to catch up") lab.stop_script(second) rows, distinct = sqlite3.connect(ledger).execute( "SELECT count(*), count(DISTINCT session_id) FROM charges").fetchone() return {"commit": commit + (" + idempotent ledger" if idempotent else ""), "charges": rows, "sessions charged": distinct, "duplicates": rows - distinct, "missing": 400 - distinct}
Step 3: The client's default: auto-commit
With the defaults, the client marks each event as done as soon as consume() hands it to you,
and commits those marks in the background every 5 seconds.
results = [crash_test("auto")] lab.table([list(r.values()) for r in results], list(results[0].keys()))
You should see duplicates: the events handled after the last background commit were handled again after the restart. The counts vary from run to run.
Step 4: Commit after processing each batch
Now the worker turns auto-commit off and calls commit() itself, after each batch of 50 has
been processed.
results.append(crash_test("after")) lab.table([list(r.values()) for r in results], list(results[0].keys()))
Still duplicates, but never more than one batch, and nothing is missing. This is at-least-once processing.
Break it: commit before processing
Predict first: the worker commits each batch as soon as it has fetched it, then processes it. What happens to the batch that was in progress when it crashed?
results.append(crash_test("before")) lab.table([list(r.values()) for r in results], list(results[0].keys()))
No duplicates, but sessions are missing: the crashed batch was already committed, so the restarted worker skipped it. This is at-most-once processing: those drivers were never charged.
Step 5: Make the ledger idempotent
Keep at-least-once, and make a second charge for the same session harmless. The worker's
--idempotent option makes session_id the ledger's primary key and inserts with INSERT OR
IGNORE. The store mode is the Python client's recommended pattern: mark each event done
after processing it (store_offsets), and let the background commit send the marks.
results.append(crash_test("store")) results.append(crash_test("store", idempotent=True)) lab.table([list(r.values()) for r in results], list(results[0].keys()))
With the idempotent ledger you should see exactly 400 charges, 0 duplicates, 0 missing.
Clean up
Delete the topic, the groups and the ledgers. If anything gets stuck, docker compose down -v
in the code/ folder resets the whole lab.
lab.delete_topics(admin, [TOPIC]) admin.delete_consumer_groups([f"m03l02-{m}" for m in ("auto", "after", "before", "store", "store-idempotent")]) for f in DATA.glob("m03l02-*.db*"): f.unlink() print("cleaned up")