Module 3 exercises

Start the lab first (docker compose up -d --wait in the code/ folder). Each exercise has a start notebook to work in and a solution notebook to compare with afterwards.

Exercise 1: Charge each session once

Goal: keep a consumer's position in the same transaction as its results. · Time: ~25 min · Starter: code/module-03/exercise-1/start/

Finance wants the billing ledger to be append-only: the charges table may not have a unique key, because later corrections will add rows for a session. They also want the ledger itself to know how far billing got. So the idempotent ledger from lesson 3.2 is not allowed here.

The starter folder holds ledger_worker.py, a billing worker with the client's default commit settings, and the notebook charge-once.ipynb. The notebook writes 400 sessions (800 events) to m03e1-sessions, then runs a crash test: it kills the worker with kill -9 after 150 and again after 300 charged sessions, and restarts it each time. Change ledger_worker.py (the three TODO comments) so that:

You're done when: the check cell prints exactly once: True, covers all 800 events: True and charges table has no unique key: True. With the unchanged starter, it reports duplicates (36 in our run; the count varies).

Hints

Hint 1 If the ledger remembers the position, Kafka's committed offsets would only compete with it. Look at `enable.auto.commit`.
Hint 2 In `on_assign`, you can set `p.offset` on each partition object and then call `consumer.assign(partitions)`: the consumer starts reading there.
Hint 3 The ledger connection runs in autocommit mode (`isolation_level=None`). Run `BEGIN` before a batch and `COMMIT` after it. SQLite's `INSERT … ON CONFLICT (topic, partition) DO UPDATE` updates a row that exists and inserts one that doesn't.

Exercise 2: Size the billing group for the peak

Goal: choose the number of consumers and partitions from measured per-partition load. · Time: ~20 min · Starter: code/module-03/exercise-2/start/

After the takeover, the lab-speed evening peak is 90 session events a second on the 6-partition topic m03e2-sessions. A billing worker spends 40 ms on an event, so it handles about 25 a second. The capacity planner says "4 workers: 4 × 25 = 100, more than 90". The notebook's run_billing(n) starts n workers, then shows each worker's partitions, the events per second arriving in them, and how its lag changed in 60 seconds.

  1. Find the smallest number of workers that keeps up. In a markdown cell, explain why 4 isn't enough.
  2. Next year's peak is 160 events a second. Use partition_rates(partitions, rate), which spreads the feed's real keys over any partition count, to choose a partition count and a number of workers that keep each worker below 20 events a second (80% busy). Write down your choice, the numbers behind it, and what changes when you add partitions to a live topic.

You're done when: run_billing() prints keeps up: True with no idle workers for your number, and your notes give a partition count and worker count for 160 events a second with the busiest partition's rate.

Hints

Hint 1 Look at the "events/s in" column for the workers that fall behind. How many partitions do they own?
Hint 2 A partition is read by one worker of the group. If one partition alone gets more than 20 events a second, no number of workers brings that partition below 80% busy.