Module 7 solutions
Solution 1: Charge and receipt in one transaction
The solution script and notebook are in code/module-07/exercise-1/solution/.
Approach. Drop the second producer and write the receipts with the transactional one:
producer.produce(args.receipts, key=event["session_id"], value=to_json( # same transaction
charges: 400 for 400 sessions
receipts: 400 for 400 sessions
exactly one receipt per charge: True
With the start script:
charges: 400 for 400 sessions
receipts: 414 for 400 sessions
exactly one receipt per charge: False
Where the extra receipts came from. The crash aborted the open transaction, so its 14 charges
disappeared for read_committed readers, and the batch was processed again. But the receipts of that
batch had gone out through a plain producer. They were ordinary, committed records, so the second run
added 14 more. Why it works: a transaction covers every record its producer writes between
begin_transaction() and commit_transaction(), in any number of topics, plus the offsets. Written
by the same producer, the charge, the receipt and the position commit or abort together.
In a real system the receipt is usually an email or a push message, not a Kafka record. Then the
same idea applies one step later: let the notification service read the committed receipts topic, with
read_committed, and send only what's there, using the session id as an idempotency key.
Solution 2: Peak kW per site per window
The solution notebook is in code/module-07/exercise-2/solution/.
Approach. Sum per site, window and 10-second slot; take the maximum per site and window:
t = parse_iso(reading["ts"]).timestamp() # event time, as seconds
site = reading["charger_id"].replace("BL-", "site-")[:8]
slot = (site, int(t // 300 * 300), int(t // 10)) # site, 5-minute window, 10-second slot
state[slot] = state.get(slot, 0.0) + reading["power_kw"]
for (site, window, _), load in state.items():
key = (site, iso(datetime.fromtimestamp(window, tz=timezone.utc)))
peaks[key] = max(peaks.get(key, 0.0), load)
site-N03 18:15: expected 296.0 kW, got 296.0
...
windows: 326 (expected 326)
highest: ('site-C05', '18:25', 323.8) (expected site-C05 18:25 323.8)
all right: True
Why it works. A charger sends one reading per 10 seconds, so each charger contributes at most one reading to a 10-second slot. The slot's sum is the site's total load at that moment. The window's peak is then the largest of its 30 slot sums. The aggregation has two levels, sum then maximum, and the order matters: the maximum of single readings would only give the strongest charger, not the site's load.
Keeping the state small. This version keeps one number per site and slot, about 36 × 8,640 = 311,000 entries a day, and never drops any. A long-running service keeps only open windows. For each (site, window) it keeps that window's slot sums and the peak so far. When the window closes (its end plus the grace period, lesson 7.3), it emits the peak and forgets the window. Then the state holds a few windows per site, whatever the length of the stream. Frameworks do exactly this: windowed state expires after the window's retention.