Notebook · code/module-03/02-committing-offsets.ipynb

Download 02-committing-offsets.ipynb

Open it in JupyterLab or VS Code from the lab folder to run it. This page is a read-only view.

Committing offsets: where processing resumes — practical

Billing charges 400 sessions into a ledger. You crash the billing worker halfway (kill -9), restart it, and count the charges, once per commit strategy. Then you make the ledger idempotent and get exactly 400.

Lab services: kafka-1, kafka-2, kafka-3 · Script: module-03/billing_worker.py · Lesson: modules/03-consumers/02-committing-offsets.md

# Setup: find the lab's shared helpers and check the lab is running
import sys, pathlib
lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / "common").is_dir())
sys.path.insert(0, str(lab_root))

import sqlite3, time
from common import settings, lab
from common.scenario import Brightlane, to_json

lab.require_kafka()
lab.print_versions()

Step 1: Yesterday's sessions, waiting to be billed

400 sessions, so 400 session_stopped events among 800 events. A correct billing run writes exactly 400 charges.

from confluent_kafka import Producer
from confluent_kafka.admin import AdminClient

TOPIC = "m03l02-sessions"
admin = AdminClient({"bootstrap.servers": settings.KAFKA_BOOTSTRAP})
lab.recreate_topic(admin, TOPIC, partitions=3)
producer = Producer(settings.PRODUCER_DEFAULTS)
for charger_id, event in Brightlane().stream(sessions=400, readings=False):
    producer.produce(TOPIC, key=charger_id, value=to_json(event))
print("not yet stored:", producer.flush(30))

Step 2: A crash test

crash_test() starts a billing worker with a fresh group and an empty SQLite ledger. The worker fetches 50 events at a time and spends 15 ms on each. When 150 charges are in the ledger, the test kills it with kill -9, starts it again, waits until the group has caught up, and counts.

The workers use session.timeout.ms=6000, so the group notices the crashed member after 6 seconds instead of 45 (lesson 3.3 explains this setting).

DATA = lab_root / ".lab-data"
DATA.mkdir(exist_ok=True)

def crash_test(commit, idempotent=False):
    group = f"m03l02-{commit}" + ("-idempotent" if idempotent else "")
    admin.delete_consumer_groups([group])
    ledger = DATA / f"{group}.db"
    ledger.unlink(missing_ok=True)
    args = ["--name", "W", "--topic", TOPIC, "--group", group, "--commit", commit, "--batch", "50",
            "--work-ms", "15", "--ledger", f".lab-data/{group}.db", "--session-timeout-ms", "6000"]
    args += ["--idempotent"] if idempotent else []
    charges = lambda: sqlite3.connect(ledger).execute("SELECT count(*) FROM charges").fetchone()[0] if ledger.exists() else 0
    first = lab.start_script("module-03/billing_worker.py", *args, name=f"{group}-1")
    lab.wait_until(lambda: charges() >= 150, timeout=60, interval=0.1, what="150 charges")
    lab.kill_script(first)                                   # crash: no close(), no final commit
    second = lab.start_script("module-03/billing_worker.py", *args, name=f"{group}-2")
    lab.wait_until(lambda: sum(r[2] for r in lab.group_lag(admin, group, TOPIC, 3).values()) == 0,
                   timeout=90, interval=1, what="the group to catch up")
    lab.stop_script(second)
    rows, distinct = sqlite3.connect(ledger).execute(
        "SELECT count(*), count(DISTINCT session_id) FROM charges").fetchone()
    return {"commit": commit + (" + idempotent ledger" if idempotent else ""), "charges": rows,
            "sessions charged": distinct, "duplicates": rows - distinct, "missing": 400 - distinct}

Step 3: The client's default: auto-commit

With the defaults, the client marks each event as done as soon as consume() hands it to you, and commits those marks in the background every 5 seconds.

results = [crash_test("auto")]
lab.table([list(r.values()) for r in results], list(results[0].keys()))

You should see duplicates: the events handled after the last background commit were handled again after the restart. The counts vary from run to run.

Step 4: Commit after processing each batch

Now the worker turns auto-commit off and calls commit() itself, after each batch of 50 has been processed.

results.append(crash_test("after"))
lab.table([list(r.values()) for r in results], list(results[0].keys()))

Still duplicates, but never more than one batch, and nothing is missing. This is at-least-once processing.

Break it: commit before processing

Predict first: the worker commits each batch as soon as it has fetched it, then processes it. What happens to the batch that was in progress when it crashed?

results.append(crash_test("before"))
lab.table([list(r.values()) for r in results], list(results[0].keys()))

No duplicates, but sessions are missing: the crashed batch was already committed, so the restarted worker skipped it. This is at-most-once processing: those drivers were never charged.

Step 5: Make the ledger idempotent

Keep at-least-once, and make a second charge for the same session harmless. The worker's --idempotent option makes session_id the ledger's primary key and inserts with INSERT OR IGNORE. The store mode is the Python client's recommended pattern: mark each event done after processing it (store_offsets), and let the background commit send the marks.

results.append(crash_test("store"))
results.append(crash_test("store", idempotent=True))
lab.table([list(r.values()) for r in results], list(results[0].keys()))

With the idempotent ledger you should see exactly 400 charges, 0 duplicates, 0 missing.

Clean up

Delete the topic, the groups and the ledgers. If anything gets stuck, docker compose down -v in the code/ folder resets the whole lab.

lab.delete_topics(admin, [TOPIC])
admin.delete_consumer_groups([f"m03l02-{m}" for m in ("auto", "after", "before", "store", "store-idempotent")])
for f in DATA.glob("m03l02-*.db*"):
    f.unlink()
print("cleaned up")