Module 1 solutions
Solution 1: Answer a dispute
The solution notebook is in code/module-01/exercise-1/solution/.
Approach. Don't scan from offset 0. Ask the broker where 18:00 starts in each partition, then read forward and stop at 18:10.
from_ms, to_ms = int(FROM.timestamp() * 1000), int(TO.timestamp() * 1000)
starts = c.offsets_for_times([TopicPartition(TOPIC, p, from_ms) for p in range(3)], timeout=10)
# ...
c.assign(starts)
The read loop keeps records whose key is BL-C08-02 and that are meter readings (no type
field). It marks a partition as done at its first record stamped at or after 18:10:
if msg.timestamp()[1] >= to_ms:
done.add(msg.partition())
continue
Output:
{0: 1669, 1: 2372, 2: 1864}
found 60 readings, all in partition(s) [1]
BL-C08-02: 60 readings between 18:00 and 18:10, 11.23 kWh delivered
matches the gateway's own records: True
Why it works. Each partition has a time index, so offsets_for_times jumps straight to the
first record at or after 18:00 without reading the ~1,700 records before it. Because the key
decides the partition, BL-C08-02's readings all sit in one partition (1 here); the other two
partitions are read only up to 18:10 and contribute nothing. Stopping on the timestamp is safe
here because the gateway wrote this topic in time order. With late records (lesson 1.3's change-it
step) a few could sit after the first 18:10 record. A production tool would read a little further
and filter by timestamp.
The charger delivered 11.23 kWh in those ten minutes. That's about 67 kW on average, so the driver's complaint isn't supported by the data.
Solution 2: A cluster health report
The solution notebook is in code/module-01/exercise-2/solution/.
Approach. Combine two admin calls: describe_cluster() for the live brokers, and
describe_topics() for each partition's leader, replicas and ISR. Count per broker, and compare
each partition's ISR with its replica list.
for p in desc.partitions:
isr = {r.id for r in p.isr}
if p.leader is not None:
rows[p.leader.id][0] += 1
for r in p.replicas:
rows[r.id][1 if r.id in isr else 2] += 1
if len(isr) < len(p.replicas):
under.append(p.id)
Output, during the stop:
broker state leads in sync out of sync
------ ------- ----- ------- -----------
1 online 3 6 0
2 online 3 6 0
3 OFFLINE 0 0 6
under-replicated partitions: [0, 1, 2, 3, 4, 5]
After the restart, broker 3 is back in sync for all 6 replicas within seconds, but it leads 0
partitions: leaders return to it only at the next automatic leader rebalance (lesson 4.1).
Bootstrap lists. The gateway in Docker uses the internal listener:
kafka-1:19092,kafka-2:19092,kafka-3:19092. A notebook on the Mac uses the host listener:
127.0.0.1:9092,127.0.0.1:9093,127.0.0.1:9094. They differ because each broker advertises a
different address per listener, and each client must use the listener whose addresses it can
reach.
Why it works. Under-replication is the first signal operators watch (module 8 turns it into an alert). Every partition here has a replica on every broker. So losing any one broker makes all 6 partitions under-replicated, but none of them unavailable: each still has two in-sync copies and a leader.