Module 1 solutions

Solution 1: Answer a dispute

The solution notebook is in code/module-01/exercise-1/solution/.

Approach. Don't scan from offset 0. Ask the broker where 18:00 starts in each partition, then read forward and stop at 18:10.

code/module-01/exercise-1/solution/answer-a-dispute.ipynb
from_ms, to_ms = int(FROM.timestamp() * 1000), int(TO.timestamp() * 1000)
starts = c.offsets_for_times([TopicPartition(TOPIC, p, from_ms) for p in range(3)], timeout=10)
# ...
c.assign(starts)

The read loop keeps records whose key is BL-C08-02 and that are meter readings (no type field). It marks a partition as done at its first record stamped at or after 18:10:

code/module-01/exercise-1/solution/answer-a-dispute.ipynb
        if msg.timestamp()[1] >= to_ms:
            done.add(msg.partition())
            continue

Output:

{0: 1669, 1: 2372, 2: 1864}
found 60 readings, all in partition(s) [1]
BL-C08-02: 60 readings between 18:00 and 18:10, 11.23 kWh delivered
matches the gateway's own records: True

Why it works. Each partition has a time index, so offsets_for_times jumps straight to the first record at or after 18:00 without reading the ~1,700 records before it. Because the key decides the partition, BL-C08-02's readings all sit in one partition (1 here); the other two partitions are read only up to 18:10 and contribute nothing. Stopping on the timestamp is safe here because the gateway wrote this topic in time order. With late records (lesson 1.3's change-it step) a few could sit after the first 18:10 record. A production tool would read a little further and filter by timestamp.

The charger delivered 11.23 kWh in those ten minutes. That's about 67 kW on average, so the driver's complaint isn't supported by the data.

Solution 2: A cluster health report

The solution notebook is in code/module-01/exercise-2/solution/.

Approach. Combine two admin calls: describe_cluster() for the live brokers, and describe_topics() for each partition's leader, replicas and ISR. Count per broker, and compare each partition's ISR with its replica list.

code/module-01/exercise-2/solution/cluster-health.ipynb
    for p in desc.partitions:
        isr = {r.id for r in p.isr}
        if p.leader is not None:
            rows[p.leader.id][0] += 1
        for r in p.replicas:
            rows[r.id][1 if r.id in isr else 2] += 1
        if len(isr) < len(p.replicas):
            under.append(p.id)

Output, during the stop:

broker  state    leads  in sync  out of sync
------  -------  -----  -------  -----------
1       online   3      6        0
2       online   3      6        0
3       OFFLINE  0      0        6
under-replicated partitions: [0, 1, 2, 3, 4, 5]

After the restart, broker 3 is back in sync for all 6 replicas within seconds, but it leads 0 partitions: leaders return to it only at the next automatic leader rebalance (lesson 4.1).

Bootstrap lists. The gateway in Docker uses the internal listener: kafka-1:19092,kafka-2:19092,kafka-3:19092. A notebook on the Mac uses the host listener: 127.0.0.1:9092,127.0.0.1:9093,127.0.0.1:9094. They differ because each broker advertises a different address per listener, and each client must use the listener whose addresses it can reach.

Why it works. Under-replication is the first signal operators watch (module 8 turns it into an alert). Every partition here has a replica on every broker. So losing any one broker makes all 6 partitions under-replicated, but none of them unavailable: each still has two in-sync copies and a leader.