Exactly-once billing with transactions
A service that reads from Kafka and writes to Kafka has two things to save after each batch: the records it wrote, and how far it read. If a crash comes between the two, work is repeated or lost. Kafka transactions save both together. This lesson shows the difference in charges, what readers see, and what happens when two copies of a service share one identity.
Theory
Example: 14 drivers charged twice
Billing reads charging.sessions and writes one charge per finished session to
billing.session-charges, which finance and the driver app read. On a Wednesday evening the billing
server rebooted halfway through a batch. The charges it had already written for that batch were in
the topic, but its consumer offsets weren't committed. After the restart it read the same batch
again and wrote those charges a second time: 14 drivers were charged twice, and support spent the
morning issuing refunds.
Lesson 3.2 made a database write idempotent. Here the output is a Kafka topic that other teams read, and "ignore the duplicate" would be their problem, not billing's.
Consume, transform, produce
Most stream processing has this shape: read records from input topics, compute something, write records to output topics. After each batch the service must save two things:
- its output records, in the output topics;
- its position, the consumer offsets of the input topics.
Done separately, a crash between them repeats the batch (output first) or loses it (offsets first). That's lesson 3.2's at-least-once and at-most-once, again.
Kafka transactions
A transaction groups writes to several partitions, and the consumer's offsets, so that they all become visible together or not at all. The steps in the Python client:
init_transactions(), once at start-up, with atransactional.idthat stays the same across restarts of this instance.- Per batch:
begin_transaction(), produce the output records,send_offsets_to_transaction()with the consumer's positions, thencommit_transaction().
A broker acts as the transaction coordinator and keeps each transaction's state in the internal
topic __transaction_state. On commit, it writes a small commit marker into every partition the
transaction touched, including __consumer_offsets. If the transaction is aborted, or its producer
dies, it writes an abort marker instead.
What to notice: the charges reach the topic before the commit. What makes them count is the marker, and only readers that wait for it get exactly once.
Readers: read_committed
The output records are in the partition from the moment they're written. What a consumer does with
them depends on its isolation.level:
read_committedreturns only records of committed transactions, and skips aborted ones. It reads up to the last stable offset: an open transaction holds back everything after it until it ends. This is the Python client's default. The Java client's default isread_uncommitted.read_uncommittedreturns everything, including records of transactions that were later aborted.
Exactly once holds only for readers that use read_committed.
Fencing zombies
What if the old instance isn't dead but only frozen, and wakes up after the new one has started? It
would happily commit its stale batch. To prevent that, init_transactions() raises the
transactional.id's epoch, a counter, and aborts anything the old instance left open. From then
on the coordinator and the brokers reject the old instance: it is fenced, and its next request
fails with a fatal error. Since Kafka 4.0 (transactions version 2, KIP-890), the epoch also rises with
every transaction, which closes a few rarer gaps.
So each running instance needs its own transactional.id, kept the same across its restarts, such
as billing-1, billing-2, or the pod name.
What transactions don't cover
Exactly once covers what stays inside Kafka: reading from Kafka, writing to Kafka, and the offsets. A payment provider called during the batch, an email sent, a row written to a database: none of these are rolled back by an aborted transaction. For those, use idempotency keys (lesson 3.2), or let the side effect happen downstream, from the committed output topic.
In production
- Handle errors in three groups. An abortable error (
txn_requires_abort()) means: abort, seek back to the committed offsets, and try again. A fatal error (fenced, for example) means: close the producer and stop, or restart the process. A retriable error on commit can be retried. - Keep transactions short.
transaction.timeout.ms(60 seconds by default) aborts a transaction left open too long, and an open transaction delays everyread_committedreader of its partitions. - Make all readers of transactional topics use
read_committed, Java consumers included. - Transactions cost a little: markers in every partition, and a round trip to the coordinator per batch. Commit every few hundred milliseconds or few hundred records, not per record.
- Kafka Streams offers the same guarantee as one setting,
processing.guarantee=exactly_once_v2.
Practical
Note
Script: code/module-07/billing_service.py (the notebook starts and crashes it)
· Notebook: code/module-07/01-exactly-once-billing.ipynb
· Lab services: kafka-1, kafka-2, kafka-3 · Time: ~16 min
What you'll do: crash the billing service halfway through 400 sessions, once committing offsets separately and once with transactions, and count the charges as an auditor would. Then start two copies with the same transactional id.
Step 1: 400 finished sessions
800 events, a start and a stop per session, in m07l01-sessions. A correct run writes exactly 400
charges.
Step 2: A crash, with offsets committed separately
In plain mode the service writes the charges of a batch, then commits its offsets:
producer.flush(10)
consumer.commit(asynchronous=False)
crash_test() starts it with --crash-after 150: right after the 150th charge reached the broker,
the process dies. Then it runs again until the group has caught up, and audit() counts:
20:23:13.995 billing CRASH after 150 charges
audit: 414 charges for 400 sessions
Step 3: The same crash, with transactions
In txn mode, each batch is one transaction:
producer = Producer({**settings.PRODUCER_DEFAULTS, "transactional.id": args.txn_id})
producer.init_transactions(30) # fences any older instance with the same id
producer.send_offsets_to_transaction(consumer.position(consumer.assignment()),
consumer.consumer_group_metadata())
producer.commit_transaction() # charges and offsets: both or neither
20:23:25.347 billing CRASH after 150 charges
audit (read_committed): 400 charges for 400 sessions
audit (read_uncommitted): 414 charges for 400 sessions
Break it: two copies with one transactional id
Predict first: a second copy starts with the same configuration, so with the same
transactional.id. What happens to the first one, and to the charges?
20:23:39.257 A started: mode=txn transactional.id=billing-1
20:23:43.885 A FATAL _FENCED: ProduceRequest for m07l01-charges [0] with 1 message(s) failed: Broker: Producer attempted an operation with an old epoch (broker 2 PID{Id:25002,Epoch:20}, base seq 29): transactional producer fenced by newer producer instance
20:23:43.887 A stopped: charged=78
20:24:00.733 B stopped: charged=337
audit (read_committed): 400 charges for 400 sessions
What just happened
In plain mode the crash came after the batch's charges had reached the broker but before the
offsets were committed. The restarted service read the batch again and charged 14 sessions a second
time, the Wednesday incident. In txn mode the same crash left a transaction open. When the service
started again, init_transactions() aborted it and fenced the dead instance. The batch was processed
again, in a new transaction. A read_committed auditor saw exactly 400 charges. A read_uncommitted
one also saw the 14 charges of the aborted transaction: they are in the partitions, marked aborted.
In the break-it step, B's init_transactions() raised the epoch, and A's next write was rejected as
coming from an old epoch. A stopped with a fatal error, its open transaction was aborted, and B took
over its partitions. A and B together wrote 78 + 337 = 415 charges, and readers saw exactly 400.
Clean up
The last cell deletes the topics and groups. If anything gets stuck, docker compose down -v in the
code/ folder resets the whole lab.
Key takeaways
- A consume-transform-produce service must save its output and its offsets together, or crashes repeat or lose work.
- A Kafka transaction makes output records and consumer offsets visible together, through commit or abort markers.
- Exactly once holds for
read_committedreaders (the Python default, not the Java one); others also see aborted records. - Give each instance a stable
transactional.id; a new instance fences the old one, which fails with a fatal error. - Transactions cover Kafka only: side effects in other systems still need idempotency.