Module 4 exercises

Start the lab first (docker compose up -d --wait in the code/ folder). Each exercise has a start notebook to work in and a solution notebook to compare with afterwards.

Exercise 1: Gone within 90 seconds

Goal: configure retention for a deletion deadline and a minimum history. · Time: ~20 min · Starter: code/module-04/exercise-1/start/

The privacy rule says no meter reading may be kept longer than 72 hours, and the grid team needs at least 36 hours of history for replays. In the lab, time runs 2,880 times faster: no reading may be readable after 90 seconds, and at least 45 seconds of history must always be there. The start notebook creates m04e1-telemetry like the topic the audit found, with only retention.ms set, writes 5 readings a second, and measures the oldest readable reading every 5 seconds for 2 minutes, after a 100-second warm-up.

Choose MY_SETTINGS, run the check, and adjust until it passes. Then, in a markdown cell, translate your settings to the real rule (72 hours, 36 hours, a 5-minute retention check), and list what could still hold a reading after Kafka has deleted it.

You're done when: the check prints gone within 90 s: True and at least 45 s of history: True, and your note gives production values for retention.ms and segment.ms with the worst case they allow.

Hints

Hint 1 The worst case is `retention.ms` + `segment.ms` + the check interval (10 seconds in the lab). The best case, right after a deletion, is about `retention.ms`.
Hint 2 Leave a few seconds of margin: a segment only closes when the next reading arrives, and the check runs on its own clock.

Exercise 2: The live charger map

Goal: rebuild state from a compacted topic correctly, then keep it up to date. · Time: ~20 min · Starter: code/module-04/exercise-2/start/

The live map service must load m04e2-charger-status completely before it reports "ready", and then apply new status changes as they arrive. The start notebook writes yesterday's 3,000 changes and tombstones for 5 retired chargers. After your code has loaded the map, it starts the status feed and gives your code 30 seconds of live changes. Then it stops the feed, catches up, and compares your map with the expected one.

Write two functions:

You're done when: the notebook prints ready map is right: True, live map is right: True and retired chargers in the map: [].

Hints

Hint 1 `consumer.get_watermark_offsets(TopicPartition(TOPIC, p), timeout=10)` returns a partition's `(start, end)`. A partition is loaded once you've seen offset `end - 1`, or a later one.
Hint 2 Use `consumer.assign()` with `OFFSET_BEGINNING`, not `subscribe()`: the map service needs every partition, not a share of them.
Hint 3 Don't stop loading at the first empty `poll()`. Wait longer, and if nothing comes at all, raise an error instead of reporting "ready".