Notebook · code/module-02/01-keys-decide-order.ipynb

Download 01-keys-decide-order.ipynb

Open it in JupyterLab or VS Code from the lab folder to run it. This page is a read-only view.

Keys decide order — practical

You write the evening's session events into a 6-partition topic three ways (no key, keyed by charger, keyed by region) and check what each does to order and to the load on each partition. Then you compare Python's and Java's partitioners, and add partitions to a live topic.

Lab services: kafka-1, kafka-2, kafka-3 · Lesson: modules/02-producers/01-keys-decide-order.md

# Setup: find the lab's shared helpers and check the lab is running
import sys, pathlib
lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / "common").is_dir())
sys.path.insert(0, str(lab_root))

from collections import Counter, defaultdict
from common import settings, lab
from common.scenario import Brightlane, to_json, from_json

lab.require_kafka()
lab.print_versions()

Step 1: Write session events without a key

200 sessions make 400 events: a session_started and a session_stopped each, in time order. A helper writes them to a fresh 6-partition topic and reads them back with one consumer, the way billing would.

from confluent_kafka import Producer, Consumer, TopicPartition
from confluent_kafka.admin import AdminClient

admin = AdminClient({"bootstrap.servers": settings.KAFKA_BOOTSTRAP})
events = Brightlane().stream(sessions=200, readings=False)

def write_and_read(topic, key_of, producer_config):
    """Write all events with key_of(charger_id) as key, then read the topic back in one pass."""
    lab.recreate_topic(admin, topic, partitions=6)
    producer = Producer({"bootstrap.servers": settings.KAFKA_BOOTSTRAP, **producer_config})
    for charger_id, event in events:
        producer.produce(topic, key=key_of(charger_id), value=to_json(event))
    producer.flush(30)
    c = Consumer({"bootstrap.servers": settings.KAFKA_BOOTSTRAP, "group.id": f"{topic}-billing",
                  "enable.auto.commit": False})
    c.assign([TopicPartition(topic, p, 0) for p in range(6)])
    got = []
    for _ in range(100):                      # bounded: stop when all 400 are back
        got += [(m.partition(), from_json(m.value())) for m in c.consume(500, 0.5) if not m.error()]
        if len(got) >= len(events):
            break
    c.close()
    return got

Now the unkeyed write. A charger gateway spreads its events over the evening, so the client would send each one to a random partition. The notebook writes all 400 in a burst, so it turns off the client's habit of filling one partition for 10 ms at a time (sticky.partitioning.linger.ms).

def check_order(got):
    """Count sessions whose stop was read before their start, and partitions per charger."""
    started, out_of_order, parts = set(), 0, defaultdict(set)
    for partition, e in got:
        parts[e["charger_id"]].add(partition)
        if e["type"] == "session_started":
            started.add(e["session_id"])
        elif e["session_id"] not in started:
            out_of_order += 1
    spread = Counter(len(p) for p in parts.values())
    print(f"sessions read stop-before-start: {out_of_order} of {len(started)}")
    print("chargers by number of partitions their events landed in:", dict(sorted(spread.items())))

no_key = write_and_read("m02l01-no-key", lambda charger: None, {"sticky.partitioning.linger.ms": 0})
check_order(no_key)

You should see dozens of sessions read stop-before-start, and chargers whose events landed in 2, 3 or more partitions. The exact numbers vary from run to run.

Step 2: Key by charger id

Same events, now keyed by charger_id. The partitioner hashes the key, so every event of one charger goes to the same partition.

by_charger = write_and_read("m02l01-by-charger", lambda charger: charger,
                            {"partitioner": "murmur2_random"})
check_order(by_charger)
print("records per partition:", sorted(Counter(p for p, _ in by_charger).items()))

You should see 0 of 200 sessions out of order, every charger in exactly 1 partition, and the records spread over all six partitions.

Step 3: A key with too few values

Someone suggests keying by region (N, C or V) "to keep each region together". Count the records per partition.

by_region = write_and_read("m02l01-by-region", lambda charger: charger.split("-")[1][0],
                           {"partitioner": "murmur2_random"})
print("records per partition:", sorted(Counter(p for p, _ in by_region).items()))

You should see only two partitions with records: regions N and V hash to the same partition. Order is still kept, but one partition carries most of the traffic and four sit idle.

Break it: two languages, two partitions

Predict first: a Java service and a Python service both write events keyed BL-N03-07 to the same 6-partition topic. Do they land in the same partition?

Kafka's console producer is a Java program. The notebook runs it inside kafka-1, then writes the same key from Python with the client's default partitioner and with murmur2_random.

T = "m02l01-by-charger"
lab.kafka_cli("kafka-console-producer.sh", "--bootstrap-server", "localhost:19092", "--topic", T,
              "--property", "parse.key=true", "--property", "key.separator=:",
              stdin="BL-N03-07:from-java\n")

for label, config in [("python default", {}), ("python murmur2_random", {"partitioner": "murmur2_random"})]:
    p = Producer({"bootstrap.servers": settings.KAFKA_BOOTSTRAP, **config})
    p.produce(T, key="BL-N03-07", value=label.encode())
    p.flush(10)

c = Consumer({"bootstrap.servers": settings.KAFKA_BOOTSTRAP, "group.id": "m02l01-check"})
c.assign([TopicPartition(T, p, 0) for p in range(6)])
for m in [m for _ in range(20) for m in c.consume(500, 0.5) if not m.error()]:
    if m.value() in (b"from-java", b"python default", b"python murmur2_random"):
        print(f"{m.value().decode():<22} -> partition {m.partition()}")
c.close()

You should see Java and murmur2_random in partition 2, and the Python default in partition 5. The two default hash functions differ, so the same charger's events would sit in two partitions with no order between them.

Change it: add partitions to a live topic

Brightlane plans to grow, and someone raises m02l01-by-charger from 6 to 8 partitions. Predict first: how many of the 240 chargers will now write to a different partition?

from confluent_kafka import murmur2
from confluent_kafka.admin import NewPartitions

admin.create_partitions([NewPartitions(T, 8)])[T].result()
chargers = [c["charger_id"] for c in Brightlane().chargers]
moved = [ch for ch in chargers if murmur2(ch.encode(), 6) != murmur2(ch.encode(), 8)]
print(f"{len(moved)} of {len(chargers)} chargers change partition")
for ch in ["BL-N03-07", "BL-C08-02"]:
    print(f"{ch}: partition {murmur2(ch.encode(), 6)} -> {murmur2(ch.encode(), 8)}")

You should see 172 of 240 chargers change partition. BL-N03-07 stays in partition 2; BL-C08-02 moves from 4 to 0. For the moved chargers, older events sit in one partition and new ones in another, so a reader can see them out of order during the switch.

Clean up

Delete the three topics. If anything gets stuck, docker compose down -v in the code/ folder resets the whole lab.

lab.delete_topics(admin, ["m02l01-no-key", "m02l01-by-charger", "m02l01-by-region"])
admin.delete_consumer_groups(["m02l01-check"])
print("cleaned up")