34 min · theory + hands-on practical

Exactly-once billing with transactions

A service that reads from Kafka and writes to Kafka has two things to save after each batch: the records it wrote, and how far it read. If a crash comes between the two, work is repeated or lost. Kafka transactions save both together. This lesson shows the difference in charges, what readers see, and what happens when two copies of a service share one identity.

Theory

Example: 14 drivers charged twice

Billing reads charging.sessions and writes one charge per finished session to billing.session-charges, which finance and the driver app read. On a Wednesday evening the billing server rebooted halfway through a batch. The charges it had already written for that batch were in the topic, but its consumer offsets weren't committed. After the restart it read the same batch again and wrote those charges a second time: 14 drivers were charged twice, and support spent the morning issuing refunds.

Lesson 3.2 made a database write idempotent. Here the output is a Kafka topic that other teams read, and "ignore the duplicate" would be their problem, not billing's.

Consume, transform, produce

Most stream processing has this shape: read records from input topics, compute something, write records to output topics. After each batch the service must save two things:

  1. its output records, in the output topics;
  2. its position, the consumer offsets of the input topics.

Done separately, a crash between them repeats the batch (output first) or loses it (offsets first). That's lesson 3.2's at-least-once and at-most-once, again.

Kafka transactions

A transaction groups writes to several partitions, and the consumer's offsets, so that they all become visible together or not at all. The steps in the Python client:

  1. init_transactions(), once at start-up, with a transactional.id that stays the same across restarts of this instance.
  2. Per batch: begin_transaction(), produce the output records, send_offsets_to_transaction() with the consumer's positions, then commit_transaction().

A broker acts as the transaction coordinator and keeps each transaction's state in the internal topic __transaction_state. On commit, it writes a small commit marker into every partition the transaction touched, including __consumer_offsets. If the transaction is aborted, or its producer dies, it writes an abort marker instead.

The billing service registers its transactional id with the transaction coordinator once, which fences any older instance with the same id. For each batch it begins a transaction, writes the charges to the charges topic, adds the consumer's offsets to the transaction, and commits. The coordinator then writes commit markers into the charges partitions and into __consumer_offsets. A read_committed consumer only sees the charges once the marker is there. If the service crashes before the commit, the coordinator writes abort markers instead, and read_committed consumers never see those charges.

What to notice: the charges reach the topic before the commit. What makes them count is the marker, and only readers that wait for it get exactly once.

Readers: read_committed

The output records are in the partition from the moment they're written. What a consumer does with them depends on its isolation.level:

Exactly once holds only for readers that use read_committed.

Fencing zombies

What if the old instance isn't dead but only frozen, and wakes up after the new one has started? It would happily commit its stale batch. To prevent that, init_transactions() raises the transactional.id's epoch, a counter, and aborts anything the old instance left open. From then on the coordinator and the brokers reject the old instance: it is fenced, and its next request fails with a fatal error. Since Kafka 4.0 (transactions version 2, KIP-890), the epoch also rises with every transaction, which closes a few rarer gaps.

So each running instance needs its own transactional.id, kept the same across its restarts, such as billing-1, billing-2, or the pod name.

What transactions don't cover

Exactly once covers what stays inside Kafka: reading from Kafka, writing to Kafka, and the offsets. A payment provider called during the batch, an email sent, a row written to a database: none of these are rolled back by an aborted transaction. For those, use idempotency keys (lesson 3.2), or let the side effect happen downstream, from the committed output topic.

In production

Practical

Note

Script: code/module-07/billing_service.py (the notebook starts and crashes it) · Notebook: code/module-07/01-exactly-once-billing.ipynb · Lab services: kafka-1, kafka-2, kafka-3 · Time: ~16 min

What you'll do: crash the billing service halfway through 400 sessions, once committing offsets separately and once with transactions, and count the charges as an auditor would. Then start two copies with the same transactional id.

Step 1: 400 finished sessions

800 events, a start and a stop per session, in m07l01-sessions. A correct run writes exactly 400 charges.

Step 2: A crash, with offsets committed separately

In plain mode the service writes the charges of a batch, then commits its offsets:

code/module-07/billing_service.py
                producer.flush(10)
                consumer.commit(asynchronous=False)

crash_test() starts it with --crash-after 150: right after the 150th charge reached the broker, the process dies. Then it runs again until the group has caught up, and audit() counts:

20:23:13.995 billing CRASH after 150 charges
audit: 414 charges for 400 sessions

Step 3: The same crash, with transactions

In txn mode, each batch is one transaction:

code/module-07/billing_service.py
        producer = Producer({**settings.PRODUCER_DEFAULTS, "transactional.id": args.txn_id})
        producer.init_transactions(30)               # fences any older instance with the same id
code/module-07/billing_service.py
                producer.send_offsets_to_transaction(consumer.position(consumer.assignment()),
                                                     consumer.consumer_group_metadata())
                producer.commit_transaction()            # charges and offsets: both or neither
20:23:25.347 billing CRASH after 150 charges
audit (read_committed):   400 charges for 400 sessions
audit (read_uncommitted): 414 charges for 400 sessions

Break it: two copies with one transactional id

Predict first: a second copy starts with the same configuration, so with the same transactional.id. What happens to the first one, and to the charges?

20:23:39.257 A started: mode=txn transactional.id=billing-1
20:23:43.885 A FATAL _FENCED: ProduceRequest for m07l01-charges [0] with 1 message(s) failed: Broker: Producer attempted an operation with an old epoch (broker 2 PID{Id:25002,Epoch:20}, base seq 29): transactional producer fenced by newer producer instance
20:23:43.887 A stopped: charged=78
20:24:00.733 B stopped: charged=337
audit (read_committed): 400 charges for 400 sessions

What just happened

In plain mode the crash came after the batch's charges had reached the broker but before the offsets were committed. The restarted service read the batch again and charged 14 sessions a second time, the Wednesday incident. In txn mode the same crash left a transaction open. When the service started again, init_transactions() aborted it and fenced the dead instance. The batch was processed again, in a new transaction. A read_committed auditor saw exactly 400 charges. A read_uncommitted one also saw the 14 charges of the aborted transaction: they are in the partitions, marked aborted. In the break-it step, B's init_transactions() raised the epoch, and A's next write was rejected as coming from an old epoch. A stopped with a fatal error, its open transaction was aborted, and B took over its partitions. A and B together wrote 78 + 337 = 415 charges, and readers saw exactly 400.

Clean up

The last cell deletes the topics and groups. If anything gets stuck, docker compose down -v in the code/ folder resets the whole lab.

Key takeaways

Next: State: turning a stream into a table