Module 4 solutions
Solution 1: Gone within 90 seconds
The solution notebook is in code/module-04/exercise-1/solution/.
Approach. Work backwards from the worst case. The oldest readable reading is at most
retention.ms + segment.ms + the check interval old, and right after a deletion it is about
retention.ms old. So retention.ms must be at least 45 seconds, and the three together at most 90:
MY_SETTINGS = {
"retention.ms": "50000", # a segment goes once its NEWEST reading is 50 s old ...
"segment.ms": "15000", # ... and holds at most 15 s of readings: oldest <= 50 + 15 + 10 (check)
}
oldest readable reading: between 51 and 70 seconds old
gone within 90 s: True
at least 45 s of history: True
With the start settings (retention.ms only), the topic has one active segment that never closes in
4 minutes, so nothing is deleted and the check fails: the oldest reading ages past 200 seconds.
Why it works. Retention deletes closed segments, once their newest reading is older than
retention.ms. A 15-second segment means a segment's oldest reading is at most 15 seconds older
than its newest, and the 10-second check adds the rest. Other pairs work too, for example 60 + 15,
as long as the sum stays below 90 with some margin.
Production. 72 hours with 36 hours of history: retention.ms = 60 hours (216,000,000) and
segment.ms = 6 hours (21,600,000). With the 5-minute check, a reading is gone after at most 66
hours 5 minutes, and at least 60 hours are always there. Deleted files stay on disk one more minute
(log.segment.delete.delay.ms).
The rule also covers copies outside Kafka: the grid team's database, the archive job's files, backups or snapshots of the brokers' disks, a MirrorMaker copy in another cluster, and tiered storage if the topic uses it. Each needs its own deletion, or the readings need crypto-shredding.
Solution 2: The live charger map
The solution notebook is in code/module-04/exercise-2/solution/.
Approach. One function applies a record; the loader remembers the end offsets first and reads until every partition has reached its own:
def apply(status, msg):
"""Apply one status record (or tombstone) to the map."""
charger_id = msg.key().decode()
if msg.value() is None:
status.pop(charger_id, None) # tombstone: retired
else:
status[charger_id] = json.loads(msg.value())["status"]
ends = {p: consumer.get_watermark_offsets(TopicPartition(TOPIC, p), timeout=10)[1] for p in range(PARTITIONS)}
consumer.assign([TopicPartition(TOPIC, p, OFFSET_BEGINNING) for p in range(PARTITIONS)])
if msg.offset() >= ends[msg.partition()] - 1:
waiting.discard(msg.partition()) # this partition is loaded
Output:
ready: 235 chargers from 3005 records in 0.01 s
ready map is right: True
feed sent 152 changes; live map has 235 chargers
live map is right: True
retired chargers in the map: []
Why it works. The end offsets taken at the start define "everything that existed when the service
started". Once each partition has reached its end, the map holds the latest status of every charger,
and only then does the service report "ready". The >= matters on a compacted topic: the record at
end - 1 may already have been removed, and then the next record you see has a higher offset. The
same apply() serves both loading and the live updates, so tombstones are handled in both. Here the
loader read 3,005 records because the cleaner hadn't run yet (nothing had closed the segments). On a
compacted topic in steady state, it reads little more than one record per charger.
assign() instead of subscribe() is deliberate. Every instance of the map service needs every
partition, and a consumer group would split them between instances. Since the consumer never joins
the group, nothing is committed either: the service rebuilds from the beginning at every start.