Module 7 solutions

Solution 1: Charge and receipt in one transaction

The solution script and notebook are in code/module-07/exercise-1/solution/.

Approach. Drop the second producer and write the receipts with the transactional one:

code/module-07/exercise-1/solution/billing_with_receipts.py
            producer.produce(args.receipts, key=event["session_id"], value=to_json(   # same transaction
charges: 400 for 400 sessions
receipts: 400 for 400 sessions
exactly one receipt per charge: True

With the start script:

charges: 400 for 400 sessions
receipts: 414 for 400 sessions
exactly one receipt per charge: False

Where the extra receipts came from. The crash aborted the open transaction, so its 14 charges disappeared for read_committed readers, and the batch was processed again. But the receipts of that batch had gone out through a plain producer. They were ordinary, committed records, so the second run added 14 more. Why it works: a transaction covers every record its producer writes between begin_transaction() and commit_transaction(), in any number of topics, plus the offsets. Written by the same producer, the charge, the receipt and the position commit or abort together.

In a real system the receipt is usually an email or a push message, not a Kafka record. Then the same idea applies one step later: let the notification service read the committed receipts topic, with read_committed, and send only what's there, using the session id as an idempotency key.

Solution 2: Peak kW per site per window

The solution notebook is in code/module-07/exercise-2/solution/.

Approach. Sum per site, window and 10-second slot; take the maximum per site and window:

code/module-07/exercise-2/solution/peak-site-load.ipynb
    t = parse_iso(reading["ts"]).timestamp()                # event time, as seconds
    site = reading["charger_id"].replace("BL-", "site-")[:8]
    slot = (site, int(t // 300 * 300), int(t // 10))        # site, 5-minute window, 10-second slot
    state[slot] = state.get(slot, 0.0) + reading["power_kw"]
code/module-07/exercise-2/solution/peak-site-load.ipynb
    for (site, window, _), load in state.items():
        key = (site, iso(datetime.fromtimestamp(window, tz=timezone.utc)))
        peaks[key] = max(peaks.get(key, 0.0), load)
site-N03 18:15: expected  296.0 kW, got 296.0
...
windows: 326 (expected 326)
highest: ('site-C05', '18:25', 323.8) (expected site-C05 18:25 323.8)
all right: True

Why it works. A charger sends one reading per 10 seconds, so each charger contributes at most one reading to a 10-second slot. The slot's sum is the site's total load at that moment. The window's peak is then the largest of its 30 slot sums. The aggregation has two levels, sum then maximum, and the order matters: the maximum of single readings would only give the strongest charger, not the site's load.

Keeping the state small. This version keeps one number per site and slot, about 36 × 8,640 = 311,000 entries a day, and never drops any. A long-running service keeps only open windows. For each (site, window) it keeps that window's slot sums and the peak so far. When the window closes (its end plus the grace period, lesson 7.3), it emits the peak and forgets the window. Then the state holds a few windows per site, whatever the length of the stream. Frameworks do exactly this: windowed state expires after the window's retention.