Module 3 exercises
Start the lab first (docker compose up -d --wait in the code/ folder). Each exercise has a
start notebook to work in and a solution notebook to compare with afterwards.
Exercise 1: Charge each session once
Goal: keep a consumer's position in the same transaction as its results. · Time: ~25 min · Starter: code/module-03/exercise-1/start/
Finance wants the billing ledger to be append-only: the charges table may not have a unique key,
because later corrections will add rows for a session. They also want the ledger itself to know how
far billing got. So the idempotent ledger from lesson 3.2 is not allowed here.
The starter folder holds ledger_worker.py, a billing worker with the client's default commit
settings, and the notebook charge-once.ipynb. The notebook writes 400 sessions (800 events) to
m03e1-sessions, then runs a crash test: it kills the worker with kill -9 after 150 and again
after 300 charged sessions, and restarts it each time. Change ledger_worker.py (the three TODO
comments) so that:
- each partition's next offset to process is stored in the ledger's
positionstable; - the charges of a batch and the positions are written in one SQLite transaction;
- a restarted worker continues each partition from the offset in
positions.
You're done when: the check cell prints exactly once: True, covers all 800 events: True and
charges table has no unique key: True. With the unchanged starter, it reports duplicates (36 in
our run; the count varies).
Hints
Hint 1
If the ledger remembers the position, Kafka's committed offsets would only compete with it. Look at `enable.auto.commit`.Hint 2
In `on_assign`, you can set `p.offset` on each partition object and then call `consumer.assign(partitions)`: the consumer starts reading there.Hint 3
The ledger connection runs in autocommit mode (`isolation_level=None`). Run `BEGIN` before a batch and `COMMIT` after it. SQLite's `INSERT … ON CONFLICT (topic, partition) DO UPDATE` updates a row that exists and inserts one that doesn't.Exercise 2: Size the billing group for the peak
Goal: choose the number of consumers and partitions from measured per-partition load. · Time: ~20 min · Starter: code/module-03/exercise-2/start/
After the takeover, the lab-speed evening peak is 90 session events a second on the 6-partition
topic m03e2-sessions. A billing worker spends 40 ms on an event, so it handles about 25 a second.
The capacity planner says "4 workers: 4 × 25 = 100, more than 90". The notebook's run_billing(n)
starts n workers, then shows each worker's partitions, the events per second arriving in them,
and how its lag changed in 60 seconds.
- Find the smallest number of workers that keeps up. In a markdown cell, explain why 4 isn't enough.
- Next year's peak is 160 events a second. Use
partition_rates(partitions, rate), which spreads the feed's real keys over any partition count, to choose a partition count and a number of workers that keep each worker below 20 events a second (80% busy). Write down your choice, the numbers behind it, and what changes when you add partitions to a live topic.
You're done when: run_billing() prints keeps up: True with no idle workers for your number,
and your notes give a partition count and worker count for 160 events a second with the busiest
partition's rate.
Hints