Notebook · code/module-04/03-compacted-topics.ipynb

Download 03-compacted-topics.ipynb

Open it in JupyterLab or VS Code from the lab folder to run it. This page is a read-only view.

Compacted topics: the latest value per key — practical

Yesterday's 3,000 charger status changes go into a compacted topic, a live feed keeps adding new ones, and the cleaner throws away everything but the latest status per charger. You rebuild the live map from the topic, retire a charger with a tombstone, and check the map against the truth.

Lab services: kafka-1, kafka-2, kafka-3 · Script: module-04/status_feed.py · Lesson: modules/04-inside-the-cluster/03-compacted-topics.md

# Setup: find the lab's shared helpers and check the lab is running
import sys, pathlib
lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / "common").is_dir())
sys.path.insert(0, str(lab_root))

import json, time
from collections import Counter
from common import settings, lab
from common.scenario import Brightlane, to_json

lab.require_kafka()
lab.print_versions()

Step 1: A compacted topic

cleanup.policy=compact turns on compaction. The other three settings make the cleaner work at lab speed: close a segment every 10 seconds, clean as soon as 1% of the log is outdated, and keep tombstones for 20 seconds. (Production defaults: 7 days, 50% and 24 hours.)

from confluent_kafka import Consumer, Producer, TopicPartition
from confluent_kafka.admin import AdminClient

TOPIC = "m04l03-charger-status"
admin = AdminClient({"bootstrap.servers": settings.KAFKA_BOOTSTRAP})
lab.recreate_topic(admin, TOPIC, partitions=3, config={
    "cleanup.policy": "compact", "segment.ms": "10000",
    "min.cleanable.dirty.ratio": "0.01", "delete.retention.ms": "20000"})
print("created", TOPIC)

Step 2: Yesterday's status changes

3,000 changes for the network's 240 chargers, keyed by charger id.

changes = Brightlane().status_changes(3000)
producer = Producer(settings.PRODUCER_DEFAULTS)
for change in changes:
    producer.produce(TOPIC, key=change["charger_id"], value=to_json(change))
producer.flush(30)

def read_all():
    """Every record still in the topic, from each partition's start to its end offset of now."""
    consumer = Consumer({"bootstrap.servers": settings.KAFKA_BOOTSTRAP, "group.id": "m04l03-reader",
                         "enable.auto.commit": False})
    bounds = {p: consumer.get_watermark_offsets(TopicPartition(TOPIC, p), timeout=10) for p in range(3)}
    consumer.assign([TopicPartition(TOPIC, p, start) for p, (start, end) in bounds.items()])
    left = {p: end for p, (start, end) in bounds.items() if end > start}
    records = []
    while left:
        msg = consumer.poll(10)
        if msg is None:
            break
        if msg.error() or msg.partition() not in left:
            continue
        if msg.offset() < left[msg.partition()]:
            records.append(msg)
        if msg.offset() >= left[msg.partition()] - 1:
            del left[msg.partition()]                    # reached the end offset we started with
    consumer.close()
    return records

records = read_all()
print(f"{len(records)} records for {len({m.key() for m in records})} chargers")

Step 3: The live feed, and the cleaner at work

The status feed keeps writing new changes, 2 a second, as the charger gateway would. Watch the number of records in the topic.

feed = lab.start_script("module-04/status_feed.py", "--topic", TOPIC, "--rate", "2", name="m04l03-feed")
rows = []
for _ in range(6):
    time.sleep(10)
    records = read_all()
    per_partition = Counter(m.partition() for m in records)
    rows.append((time.strftime("%H:%M:%S"), len(records), len({m.key() for m in records}),
                 [per_partition[p] for p in range(3)]))
lab.table(rows, ["time", "records", "chargers", "records per partition"])

The record count falls towards the number of chargers, one partition at a time: a partition is cleaned once its full segment has closed. The feed's newest records sit in each partition's active segment, which is never cleaned, so a few chargers appear twice.

Step 4: Rebuild the live map

This is what the map service does when it starts: read the topic from the start and keep the last status per charger.

def load_map():
    status = {}
    for msg in read_all():
        status[msg.key().decode()] = json.loads(msg.value())["status"]
    return status

started = time.monotonic()
live_map = load_map()
print(f"{len(live_map)} chargers loaded in {time.monotonic() - started:.2f} s:", dict(Counter(live_map.values())))

Break it: retire a charger

Brightlane removes a charger from service. A tombstone, a record with the charger's key and no value, tells every reader to forget it. We pick a charger the feed won't touch during this notebook, so the feed doesn't bring it back. Predict first: what does load_map() do now?

feed_keys = {c["charger_id"] for c in Brightlane().status_changes(3800)[3000:]}
retired = next(c["charger_id"] for c in Brightlane().chargers if c["charger_id"] not in feed_keys)
producer.produce(TOPIC, key=retired, value=None)                  # the tombstone
producer.flush(10)
try:
    load_map()
except Exception as e:
    print(f"load_map() failed: {type(e).__name__}: {e}")

The map loader assumed every record has a value. A loader for a compacted topic must treat a missing value as "delete this key":

def load_map():
    status = {}
    for msg in read_all():
        if msg.value() is None:
            status.pop(msg.key().decode(), None)           # tombstone: the charger is gone
        else:
            status[msg.key().decode()] = json.loads(msg.value())["status"]
    return status

print(len(load_map()), "chargers;", retired, "in the map:", retired in load_map())

Now watch the retired charger's records in the topic while the cleaner works:

for _ in range(7):
    kept = [(m.offset(), "tombstone" if m.value() is None else json.loads(m.value())["status"])
            for m in read_all() if m.key().decode() == retired]
    print(time.strftime("%H:%M:%S"), retired, kept)
    time.sleep(10)

First the charger's old statuses go, at the next cleaning after the tombstone's segment closed. Then, once the tombstone is older than delete.retention.ms (20 seconds here), the next cleaning removes the tombstone too, and the key has no records at all.

Step 5: Does the map match the truth?

Stop the feed. Every change it sent is known (it plays the scenario's changes 3,001 onwards), so the expected map can be computed without Kafka.

lab.stop_script(feed)
sent = int(feed.lines("stopped")[-1].split("sent=")[1].split()[0])
expected = {}
for change in Brightlane().status_changes(3000 + sent):
    expected[change["charger_id"]] = change["status"]
expected.pop(retired)
live_map = load_map()
print(f"feed sent {sent} changes; map has {len(live_map)} chargers; matches: {live_map == expected}")
print(f"records read to build it: {len(read_all())}, out of {3000 + sent + 1} ever written")

Clean up

Delete the topic. If anything gets stuck, docker compose down -v in the code/ folder resets the whole lab.

lab.delete_topics(admin, [TOPIC])
admin.delete_consumer_groups(["m04l03-reader"])
print("cleaned up")