Download 01-keys-decide-order.ipynb
Open it in JupyterLab or VS Code from the lab folder to run it. This page is a read-only view.
Keys decide order — practical
You write the evening's session events into a 6-partition topic three ways (no key, keyed by charger, keyed by region) and check what each does to order and to the load on each partition. Then you compare Python's and Java's partitioners, and add partitions to a live topic.
Lab services: kafka-1, kafka-2, kafka-3 · Lesson: modules/02-producers/01-keys-decide-order.md
# Setup: find the lab's shared helpers and check the lab is running import sys, pathlib lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / "common").is_dir()) sys.path.insert(0, str(lab_root)) from collections import Counter, defaultdict from common import settings, lab from common.scenario import Brightlane, to_json, from_json lab.require_kafka() lab.print_versions()
Step 1: Write session events without a key
200 sessions make 400 events: a session_started and a session_stopped each, in time order.
A helper writes them to a fresh 6-partition topic and reads them back with one consumer, the
way billing would.
from confluent_kafka import Producer, Consumer, TopicPartition from confluent_kafka.admin import AdminClient admin = AdminClient({"bootstrap.servers": settings.KAFKA_BOOTSTRAP}) events = Brightlane().stream(sessions=200, readings=False) def write_and_read(topic, key_of, producer_config): """Write all events with key_of(charger_id) as key, then read the topic back in one pass.""" lab.recreate_topic(admin, topic, partitions=6) producer = Producer({"bootstrap.servers": settings.KAFKA_BOOTSTRAP, **producer_config}) for charger_id, event in events: producer.produce(topic, key=key_of(charger_id), value=to_json(event)) producer.flush(30) c = Consumer({"bootstrap.servers": settings.KAFKA_BOOTSTRAP, "group.id": f"{topic}-billing", "enable.auto.commit": False}) c.assign([TopicPartition(topic, p, 0) for p in range(6)]) got = [] for _ in range(100): # bounded: stop when all 400 are back got += [(m.partition(), from_json(m.value())) for m in c.consume(500, 0.5) if not m.error()] if len(got) >= len(events): break c.close() return got
Now the unkeyed write. A charger gateway spreads its events over the evening, so the client
would send each one to a random partition. The notebook writes all 400 in a burst, so it turns
off the client's habit of filling one partition for 10 ms at a time (sticky.partitioning.linger.ms).
def check_order(got): """Count sessions whose stop was read before their start, and partitions per charger.""" started, out_of_order, parts = set(), 0, defaultdict(set) for partition, e in got: parts[e["charger_id"]].add(partition) if e["type"] == "session_started": started.add(e["session_id"]) elif e["session_id"] not in started: out_of_order += 1 spread = Counter(len(p) for p in parts.values()) print(f"sessions read stop-before-start: {out_of_order} of {len(started)}") print("chargers by number of partitions their events landed in:", dict(sorted(spread.items()))) no_key = write_and_read("m02l01-no-key", lambda charger: None, {"sticky.partitioning.linger.ms": 0}) check_order(no_key)
You should see dozens of sessions read stop-before-start, and chargers whose events landed in 2, 3 or more partitions. The exact numbers vary from run to run.
Step 2: Key by charger id
Same events, now keyed by charger_id. The partitioner hashes the key, so every event of one
charger goes to the same partition.
by_charger = write_and_read("m02l01-by-charger", lambda charger: charger, {"partitioner": "murmur2_random"}) check_order(by_charger) print("records per partition:", sorted(Counter(p for p, _ in by_charger).items()))
You should see 0 of 200 sessions out of order, every charger in exactly 1 partition, and
the records spread over all six partitions.
Step 3: A key with too few values
Someone suggests keying by region (N, C or V) "to keep each region together". Count the records per partition.
by_region = write_and_read("m02l01-by-region", lambda charger: charger.split("-")[1][0], {"partitioner": "murmur2_random"}) print("records per partition:", sorted(Counter(p for p, _ in by_region).items()))
You should see only two partitions with records: regions N and V hash to the same partition. Order is still kept, but one partition carries most of the traffic and four sit idle.
Break it: two languages, two partitions
Predict first: a Java service and a Python service both write events keyed BL-N03-07 to the
same 6-partition topic. Do they land in the same partition?
Kafka's console producer is a Java program. The notebook runs it inside kafka-1, then writes
the same key from Python with the client's default partitioner and with murmur2_random.
T = "m02l01-by-charger" lab.kafka_cli("kafka-console-producer.sh", "--bootstrap-server", "localhost:19092", "--topic", T, "--property", "parse.key=true", "--property", "key.separator=:", stdin="BL-N03-07:from-java\n") for label, config in [("python default", {}), ("python murmur2_random", {"partitioner": "murmur2_random"})]: p = Producer({"bootstrap.servers": settings.KAFKA_BOOTSTRAP, **config}) p.produce(T, key="BL-N03-07", value=label.encode()) p.flush(10) c = Consumer({"bootstrap.servers": settings.KAFKA_BOOTSTRAP, "group.id": "m02l01-check"}) c.assign([TopicPartition(T, p, 0) for p in range(6)]) for m in [m for _ in range(20) for m in c.consume(500, 0.5) if not m.error()]: if m.value() in (b"from-java", b"python default", b"python murmur2_random"): print(f"{m.value().decode():<22} -> partition {m.partition()}") c.close()
You should see Java and murmur2_random in partition 2, and the Python default in
partition 5. The two default hash functions differ, so the same charger's events would sit in
two partitions with no order between them.
Change it: add partitions to a live topic
Brightlane plans to grow, and someone raises m02l01-by-charger from 6 to 8 partitions.
Predict first: how many of the 240 chargers will now write to a different partition?
from confluent_kafka import murmur2 from confluent_kafka.admin import NewPartitions admin.create_partitions([NewPartitions(T, 8)])[T].result() chargers = [c["charger_id"] for c in Brightlane().chargers] moved = [ch for ch in chargers if murmur2(ch.encode(), 6) != murmur2(ch.encode(), 8)] print(f"{len(moved)} of {len(chargers)} chargers change partition") for ch in ["BL-N03-07", "BL-C08-02"]: print(f"{ch}: partition {murmur2(ch.encode(), 6)} -> {murmur2(ch.encode(), 8)}")
You should see 172 of 240 chargers change partition. BL-N03-07 stays in partition 2;
BL-C08-02 moves from 4 to 0. For the moved chargers, older events sit in one partition and new
ones in another, so a reader can see them out of order during the switch.
Clean up
Delete the three topics. If anything gets stuck, docker compose down -v in the code/ folder
resets the whole lab.
lab.delete_topics(admin, ["m02l01-no-key", "m02l01-by-charger", "m02l01-by-region"]) admin.delete_consumer_groups(["m02l01-check"]) print("cleaned up")