Module 3 solutions

Solution 1: Charge each session once

The solution worker and notebook are in code/module-03/exercise-1/solution/.

Approach. Make the ledger the only place that remembers the position. First, turn off Kafka's auto-commit, and on assignment start each partition where the ledger says:

code/module-03/exercise-1/solution/ledger_worker.py
    config["enable.auto.commit"] = False                 # the ledger, not Kafka, remembers the position

    def on_assign(consumer, partitions):
        for p in partitions:
            row = db.execute("SELECT next_offset FROM positions WHERE topic = ? AND partition = ?",
                             (p.topic, p.partition)).fetchone()
            if row:
                p.offset = row[0]                        # resume where the ledger says we got to
        consumer.assign(partitions)

Then wrap each batch in a transaction that writes the charges and the next offset of every partition in the batch:

code/module-03/exercise-1/solution/ledger_worker.py
            db.execute("INSERT INTO positions VALUES (?, ?, ?) ON CONFLICT (topic, partition) "
                       "DO UPDATE SET next_offset = excluded.next_offset",
                       (msg.topic(), msg.partition(), msg.offset() + 1))
        db.execute("COMMIT")                             # charges and positions: both or neither

Output of the check:

charges: 400  sessions charged: 400  duplicates: 0  missing: 0
exactly once: True
positions: {0: 292, 1: 262, 2: 246}  covers all 800 events: True
charges table has no unique key: True

The worker logs show where each restart began. The second worker started partition 1 at offset 262 (already finished) and partition 2 at 88; the third started partition 0 at 142:

16:12:15.054 ASSIGNED [(0, -1001), (1, 262), (2, 88)]
16:12:24.092 ASSIGNED [(0, 142), (1, 262), (2, 246)]

-1001 means "no stored position": the consumer falls back to auto.offset.reset, here the start of the partition.

Why it works. A kill -9 can stop the worker at any point. If it stops before COMMIT, SQLite rolls back the whole batch when the ledger is next opened: no charges and no new positions, so the next worker processes that batch again, once. If it stops after COMMIT, both are stored, and the next worker starts after the batch. There is no moment where one is saved without the other, so the repeats of at-least-once can't happen. This is exactly-once processing into a database, without a unique key on the results.

Two things to know before you use this pattern. Kafka now has no committed offsets for the group, so lag tools show nothing. You can still call commit(asynchronous=True) after the database commit, for monitoring only. And with the cooperative or KIP-848 protocol, set the positions with incremental_assign() instead, because those protocols hand out partitions in increments.

Solution 2: Size the billing group for the peak

The solution notebook is in code/module-03/exercise-2/solution/.

Part 1: 6 workers. One run of 4, 5 and 6 workers:

worker  partitions  events/s in  lag change in 60 s
------  ----------  -----------  ------------------
A       [0, 1]      32.7         573
B       [2, 3]      29.1         359
C       [4]         14.0         -403
D       [5]         14.2         -417
4 workers keep up: False   idle workers: 0
...
A       [0, 1]      32.2         542
5 workers keep up: False   idle workers: 0
...
A       [0]         17.4         -347
6 workers keep up: True   idle workers: 0

Each partition is read by one worker, so the number that matters is the load on the busiest worker, not the total. With 4 workers, A and B each owned two partitions, about 30 events a second, more than the 25 one worker can handle. Their lag grew while C and D had spare time. With 5 workers, A still owned two partitions. With 6, the busiest worker got 17.4 a second and the whole backlog shrank. A total-capacity check (4 × 25 > 90) hides two things: the partitions don't get equal traffic, and a group can't split a partition's load between workers.

Part 2: 12 partitions and 12 workers.

 6 partitions at 160/s: busiest  30.2/s, quietest  21.8/s
 8 partitions at 160/s: busiest  24.3/s, quietest  17.4/s
10 partitions at 160/s: busiest  22.7/s, quietest  11.0/s
12 partitions at 160/s: busiest  16.6/s, quietest   9.6/s
16 partitions at 160/s: busiest  13.2/s, quietest   6.8/s

With 6 partitions, the busiest one would get 30 events a second, and no number of workers can fix that. 8 and 10 partitions still leave one partition above 20. With 12, the busiest gets about 17, so 12 workers each stay below 80% busy. Fewer workers than 12 would put two partitions on some of them, about 26 a second. 16 partitions would also work and leave room to grow, at the cost of more partitions for the brokers to manage.

Adding partitions to the live topic changes where keys go (lesson 2.1). A charger's new events land in a different partition from its old ones, so for a while its events have no single order. Clients also notice new partitions only at their next metadata refresh, every 5 minutes by default (topic.metadata.refresh.interval.ms). Add them at a quiet time, after the group has caught up. The other lever is a faster worker, for example one that calls the payment provider for several sessions at once.