Module 2 solutions

Solution 1: A key for driver notifications

The solution notebook is in code/module-02/exercise-1/solution/.

Approach. Order matters per driver, so the key is the driver id:

code/module-02/exercise-1/solution/key-for-notifications.ipynb
def key_for(note):
    """Return the record key for one notification (a str), or None for no key."""
    return note["driver_id"]   # one driver's notifications must stay in order

Output:

drivers who saw notifications out of order: 0 of 299
records per partition: [(0, 260), (1, 188), (2, 148), (3, 172), (4, 228), (5, 204)]
busiest / quietest partition: 1.76

With the start notebook's None key, the same check reports about 270 of 299 drivers out of order.

Why it works. Every notification of one driver hashes to the same partition, so they're read in the order they were written. A session id keeps one session in order, but a driver with two sessions could see the second session's "charging started" before the first one's "receipt ready". A charger id fails differently: a driver's sessions on two chargers go to two partitions. The 299 drivers spread over 6 partitions about evenly. No single driver can be a hot key, because one person receives only a few notifications per session.

Solution 2: Tune the replay producer

The solution notebook is in code/module-02/exercise-2/solution/.

Approach. Measure first, then choose. One run of the codec comparison:

defaults: {'readings/s': 863880, 'p99 ms': 41.3, 'stored MB': 46.0}
gzip    {'readings/s': 717770, 'p99 ms': 73.0, 'stored MB': 4.5}
snappy  {'readings/s': 912939, 'p99 ms': 50.6, 'stored MB': 9.5}
lz4     {'readings/s': 997947, 'p99 ms': 39.7, 'stored MB': 10.0}
zstd    {'readings/s': 818687, 'p99 ms': 34.1, 'stored MB': 4.7}

All four codecs more than halve the stored bytes. gzip and zstd shrink the data about tenfold, snappy and lz4 about fivefold. The throughput numbers move by 10–20% between runs; in other runs zstd was the fastest. The solution keeps zstd for its size, and adds a 20 ms linger for the live stream:

code/module-02/exercise-2/solution/tune-the-replay.ipynb
my_settings = {
    "compression.type": "zstd",   # JSON compresses ~10x; fewer bytes to send, replicate and store
    "linger.ms": 20,              # helps the live stream; in a burst batches fill anyway
}
yours:    {'readings/s': 915372, 'p99 ms': 48.1, 'stored MB': 4.7}
at least as fast:   True
at most half bytes: True
p99 under 250 ms:   True

Why it works. In a burst, batches already fill up to the 1 MB batch.size, so batching is not the lever. Compression is: it runs once per batch in the producer and then saves bytes on the network, on replication to two followers, on three disks and on every consumer's fetch. zstd gives gzip's ratio at a much lower CPU cost. lz4 is a good choice when the producer's CPU is the scarce resource and the doubled disk use is acceptable. For Brightlane's projected 260 GB of telemetry, zstd means about 26 GB.

The linger.ms value doesn't change the replay much. In the live stream at 2,500 readings a second, it cuts produce requests roughly tenfold compared with linger.ms=0 (lesson 2.3 measured 22 records per request at 50 ms), for about 20 ms of extra latency, well within the grid report's needs.