32 min · theory + hands-on practical

Committing offsets: where processing resumes

A consumer group remembers how far it has read, so a restarted worker continues where the old one stopped. When that memory is updated decides what a crash costs: work done twice, or work never done. This lesson measures both and shows how to make the repeats harmless.

Theory

Example: the morning after the power cut

On Tuesday at 18:22 the server running Brightlane's billing worker lost power. It restarted a minute later and carried on. On Wednesday, 14 drivers complained: they had been charged twice for one session.

A developer traced it to the consumer's default settings and "fixed" it: the worker now committed its position as soon as it had fetched a batch, before charging anything. Three weeks later the server crashed again. This time nobody was charged twice, but 9 sessions were never billed. Finance noticed a month later.

Both incidents come from one choice: the moment the worker tells Kafka "I'm done up to here".

Committed offsets

For each partition it reads, a consumer group stores a committed offset: the offset of the next record it should process. Kafka keeps these in an internal topic, __consumer_offsets. When a member starts, or takes over a partition in a rebalance, it continues from the committed offset. (Only a group with no committed offset uses auto.offset.reset.)

Committing is a separate step from reading. Reading moves the consumer's position in memory; committing saves it. A crash loses everything that was read but not yet committed, and the next reader starts from the last commit.

What the Python client does by default

With the default settings, three things happen without your code asking:

So after a crash, up to 5 seconds of processed records are processed again: duplicates. And if the timer fires while you're halfway through a batch, it commits records you haven't processed. If you then crash, those are lost.

Before or after: the two guarantees

What a crash costs depends on the order of two steps, process and commit:

Two timelines for one batch of 50 events. Commit after processing: a crash after 30 events means all 50 are processed again, 30 of them twice. Commit before processing: a crash after 30 events means the other 20 are never processed.

What to notice: the same crash at the same point costs duplicates in one order and lost work in the other. There's no third order of two separate steps.

Processing exactly once needs the two steps to happen atomically: either both or neither. If the results go to Kafka, transactions do that (lesson 7.1). If they go to a database, store the offset in the same database transaction as the result.

Idempotent processing

The practical answer for most consumers: choose at-least-once and make repeats harmless. Processing is idempotent when doing it twice has the same effect as doing it once.

Billing has a natural key: each session is charged once, so session_id can be the ledger's primary key. A repeated session_stopped then hits the key and is ignored (INSERT OR IGNORE in SQLite, INSERT … ON CONFLICT DO NOTHING in PostgreSQL). Where there is no natural key, have the producer put a unique event id in each record and keep a table of processed ids.

The Python client's recommended pattern for at-least-once is to keep auto-commit but turn off the automatic marking. Mark each record yourself after processing it, and the background timer commits only processed records:

consumer = Consumer({"bootstrap.servers": BOOTSTRAP, "group.id": "billing",
                     "enable.auto.offset.store": False})       # auto-commit stays on
consumer.subscribe(["charging.sessions"])
for msg in consumer.consume(num_messages=50, timeout=0.5):
    handle(msg)                                                  # your processing
    consumer.store_offsets(message=msg)                          # done: may be committed now

Calling commit() after every record also works, but it costs a round trip to the broker each time. After every batch is a good middle ground.

In production

Practical

Note

Script: code/module-03/billing_worker.py (the notebook starts and crashes it) · Notebook: code/module-03/02-committing-offsets.ipynb · Lab services: kafka-1, kafka-2, kafka-3 · Time: ~16 min (about 2 minutes of waiting)

What you'll do: bill 400 sessions with a worker that crashes halfway, once per commit strategy, and count the charges in its ledger. Then make the ledger idempotent.

Step 1: Yesterday's sessions, waiting to be billed

The notebook writes 400 sessions (800 events) to m03l02-sessions. A correct run writes exactly 400 charges.

Step 2: A crash test

crash_test(commit, idempotent=False) runs the worker from lesson 3.1 with a fresh group and an empty SQLite ledger, using these options:

code/module-03/02-committing-offsets.ipynb
    args = ["--name", "W", "--topic", TOPIC, "--group", group, "--commit", commit, "--batch", "50",
            "--work-ms", "15", "--ledger", f".lab-data/{group}.db", "--session-timeout-ms", "6000"]

The worker fetches 50 events at a time and spends 15 ms on each. When 150 charges are in the ledger, the test kills it with kill -9, so no close() and no final commit run. Then it starts a new worker in the same group and waits until the group has caught up:

code/module-03/02-committing-offsets.ipynb
    lab.kill_script(first)                                   # crash: no close(), no final commit
    second = lab.start_script("module-03/billing_worker.py", *args, name=f"{group}-2")

The --session-timeout-ms 6000 option makes the group notice the dead worker after 6 seconds instead of 45. Lesson 3.3 explains that setting.

In the worker, --commit chooses where committing happens:

code/module-03/billing_worker.py
            if args.commit == "before":                    # commit first, then process
                consumer.commit(asynchronous=False)
            # ...
                if args.commit == "store":                 # mark as done; auto-commit sends it later
                    consumer.store_offsets(message=msg)
            if args.commit == "after":                     # process first, then commit
                consumer.commit(asynchronous=False)

Step 3: The client's default: auto-commit

Output (all counts in this lesson vary from run to run):

commit  charges  sessions charged  duplicates  missing
------  -------  ----------------  ----------  -------
auto    420      400               20          0

Step 4: Commit after processing each batch

after   419      400               19          0

Break it: commit before processing

Predict first: the worker commits each batch as soon as it has fetched it. What happens to the batch in progress when it crashes?

before  393      393               0           7

Step 5: Make the ledger idempotent

The worker's --idempotent option gives the ledger a primary key on session_id and switches the insert:

code/module-03/billing_worker.py
    key = "PRIMARY KEY" if idempotent else ""
code/module-03/billing_worker.py
                        verb = "INSERT OR IGNORE" if args.idempotent else "INSERT"

The notebook runs the store pattern twice, without and with the idempotent ledger:

commit                     charges  sessions charged  duplicates  missing
-------------------------  -------  ----------------  ----------  -------
auto                       420      400               20          0
after                      419      400               19          0
before                     393      393               0           7
store                      419      400               19          0
store + idempotent ledger  400      400               0           0

What just happened

Every crash happened at the same point: after about 150 charges, in the middle of a batch. With auto-commit, the restarted worker went back to the last background commit and charged about 20 sessions again. Committing after each batch is at-least-once: nothing was lost, but up to one batch was repeated. Committing before each batch is at-most-once: no repeats, but 7 sessions in the crashed batch were never charged. That's the developer's "fix" from the example. The store pattern is at-least-once too, so it repeated work like the others. Only the idempotent ledger turned the repeats into no-ops: exactly 400 charges, with at-least-once delivery underneath.

Clean up

The last cell deletes the topic, the five groups and the ledger files in code/.lab-data/. If anything gets stuck, docker compose down -v in the code/ folder resets the whole lab.

Key takeaways

Next: Rebalances and the new group protocol