Committing offsets: where processing resumes
A consumer group remembers how far it has read, so a restarted worker continues where the old one stopped. When that memory is updated decides what a crash costs: work done twice, or work never done. This lesson measures both and shows how to make the repeats harmless.
Theory
Example: the morning after the power cut
On Tuesday at 18:22 the server running Brightlane's billing worker lost power. It restarted a minute later and carried on. On Wednesday, 14 drivers complained: they had been charged twice for one session.
A developer traced it to the consumer's default settings and "fixed" it: the worker now committed its position as soon as it had fetched a batch, before charging anything. Three weeks later the server crashed again. This time nobody was charged twice, but 9 sessions were never billed. Finance noticed a month later.
Both incidents come from one choice: the moment the worker tells Kafka "I'm done up to here".
Committed offsets
For each partition it reads, a consumer group stores a committed offset: the offset of the next
record it should process. Kafka keeps these in an internal topic, __consumer_offsets. When a
member starts, or takes over a partition in a rebalance, it continues from the committed offset.
(Only a group with no committed offset uses auto.offset.reset.)
Committing is a separate step from reading. Reading moves the consumer's position in memory; committing saves it. A crash loses everything that was read but not yet committed, and the next reader starts from the last commit.
What the Python client does by default
With the default settings, three things happen without your code asking:
enable.auto.offset.store=true: as soon aspoll()orconsume()hands you a record, the client marks its offset to be committed, before you have processed it.enable.auto.commit=true: a background timer commits the marked offsets every 5 seconds (auto.commit.interval.ms).close()commits the marked offsets one last time.
So after a crash, up to 5 seconds of processed records are processed again: duplicates. And if the timer fires while you're halfway through a batch, it commits records you haven't processed. If you then crash, those are lost.
Before or after: the two guarantees
What a crash costs depends on the order of two steps, process and commit:
- Commit after processing: a crash repeats the records since the last commit, but loses none. This is at-least-once processing.
- Commit before processing: a crash skips the records that were committed but not yet processed, but repeats none. This is at-most-once processing.
What to notice: the same crash at the same point costs duplicates in one order and lost work in the other. There's no third order of two separate steps.
Processing exactly once needs the two steps to happen atomically: either both or neither. If the results go to Kafka, transactions do that (lesson 7.1). If they go to a database, store the offset in the same database transaction as the result.
Idempotent processing
The practical answer for most consumers: choose at-least-once and make repeats harmless. Processing is idempotent when doing it twice has the same effect as doing it once.
Billing has a natural key: each session is charged once, so session_id can be the ledger's
primary key. A repeated session_stopped then hits the key and is ignored (INSERT OR IGNORE in
SQLite, INSERT … ON CONFLICT DO NOTHING in PostgreSQL). Where there is no natural key, have the
producer put a unique event id in each record and keep a table of processed ids.
The Python client's recommended pattern for at-least-once is to keep auto-commit but turn off the automatic marking. Mark each record yourself after processing it, and the background timer commits only processed records:
consumer = Consumer({"bootstrap.servers": BOOTSTRAP, "group.id": "billing",
"enable.auto.offset.store": False}) # auto-commit stays on
consumer.subscribe(["charging.sessions"])
for msg in consumer.consume(num_messages=50, timeout=0.5):
handle(msg) # your processing
consumer.store_offsets(message=msg) # done: may be committed now
Calling commit() after every record also works, but it costs a round trip to the broker each
time. After every batch is a good middle ground.
In production
- Never run billing-like consumers on the defaults. Use
enable.auto.offset.store=falsewithstore_offsets(), or commit after each batch, and make the processing idempotent. - Call
close()on shutdown (handleSIGTERMin containers), so the last positions are committed and the group rebalances at once. - If processing fails for one record, don't store or commit past it silently. Retry, or park it in a dead-letter topic (lesson 5.3).
- A group that is inactive for 7 days loses its committed offsets (
offsets.retention.minutes). When it returns,auto.offset.resetdecides where it starts, so choose that setting on purpose.
Practical
Note
Script: code/module-03/billing_worker.py (the notebook starts and crashes it)
· Notebook: code/module-03/02-committing-offsets.ipynb
· Lab services: kafka-1, kafka-2, kafka-3 · Time: ~16 min (about 2 minutes of waiting)
What you'll do: bill 400 sessions with a worker that crashes halfway, once per commit strategy, and count the charges in its ledger. Then make the ledger idempotent.
Step 1: Yesterday's sessions, waiting to be billed
The notebook writes 400 sessions (800 events) to m03l02-sessions. A correct run writes exactly
400 charges.
Step 2: A crash test
crash_test(commit, idempotent=False) runs the worker from lesson 3.1 with a fresh group and an
empty SQLite ledger, using these options:
args = ["--name", "W", "--topic", TOPIC, "--group", group, "--commit", commit, "--batch", "50",
"--work-ms", "15", "--ledger", f".lab-data/{group}.db", "--session-timeout-ms", "6000"]
The worker fetches 50 events at a time and spends 15 ms on each. When 150 charges are in the
ledger, the test kills it with kill -9, so no close() and no final commit run. Then it starts a
new worker in the same group and waits until the group has caught up:
lab.kill_script(first) # crash: no close(), no final commit
second = lab.start_script("module-03/billing_worker.py", *args, name=f"{group}-2")
The --session-timeout-ms 6000 option makes the group notice the dead worker after 6 seconds
instead of 45. Lesson 3.3 explains that setting.
In the worker, --commit chooses where committing happens:
if args.commit == "before": # commit first, then process
consumer.commit(asynchronous=False)
# ...
if args.commit == "store": # mark as done; auto-commit sends it later
consumer.store_offsets(message=msg)
if args.commit == "after": # process first, then commit
consumer.commit(asynchronous=False)
Step 3: The client's default: auto-commit
Output (all counts in this lesson vary from run to run):
commit charges sessions charged duplicates missing
------ ------- ---------------- ---------- -------
auto 420 400 20 0
Step 4: Commit after processing each batch
after 419 400 19 0
Break it: commit before processing
Predict first: the worker commits each batch as soon as it has fetched it. What happens to the batch in progress when it crashes?
before 393 393 0 7
Step 5: Make the ledger idempotent
The worker's --idempotent option gives the ledger a primary key on session_id and switches the
insert:
key = "PRIMARY KEY" if idempotent else ""
verb = "INSERT OR IGNORE" if args.idempotent else "INSERT"
The notebook runs the store pattern twice, without and with the idempotent ledger:
commit charges sessions charged duplicates missing
------------------------- ------- ---------------- ---------- -------
auto 420 400 20 0
after 419 400 19 0
before 393 393 0 7
store 419 400 19 0
store + idempotent ledger 400 400 0 0
What just happened
Every crash happened at the same point: after about 150 charges, in the middle of a batch. With
auto-commit, the restarted worker went back to the last background commit and charged about 20
sessions again. Committing after each batch is at-least-once: nothing was lost, but up to one
batch was repeated. Committing before each batch is at-most-once: no repeats, but 7 sessions in the
crashed batch were never charged. That's the developer's "fix" from the example. The store
pattern is at-least-once too, so it repeated work like the others. Only the idempotent ledger turned
the repeats into no-ops: exactly 400 charges, with at-least-once delivery underneath.
Clean up
The last cell deletes the topic, the five groups and the ledger files in code/.lab-data/. If
anything gets stuck, docker compose down -v in the code/ folder resets the whole lab.
Key takeaways
- A group's committed offset is where processing resumes after a restart or a rebalance.
- By default, the Python client marks records as done when it hands them to you and commits every 5 s: crashes repeat work.
- Commit after processing for at-least-once; commit before for at-most-once. There is no third order.
- Make processing idempotent (a natural key plus an upsert) so at-least-once repeats are harmless.
- The recommended pattern:
enable.auto.offset.store=falseplusstore_offsets()after each record.