31 min · theory + hands-on practical

Keys decide order

Module 1 wrote events without thinking about where they went. This lesson shows how a producer picks a partition for each record, what that means for order, and how to choose a key and a partition count you won't regret.

Theory

Example: the stops that came first

Brightlane's gateway team wanted to spread the load evenly over the six partitions of charging.sessions. They removed the record key: "the partitioner will balance it". The next evening billing logged 31 warnings like this one:

WARN billing: session_stopped for unknown session ses-7f3a19 (charger BL-N03-04), skipped

Billing builds a session when it reads session_started and charges it when it reads session_stopped. For 31 sessions it read the stop before the start, so it skipped them, and 31 drivers charged for free. Nothing was lost: both events were in the topic. They were just in different partitions, and Kafka keeps no order between partitions.

How the producer picks a partition

The producer, not the broker, decides which partition each record goes to. The part of the client that decides is the partitioner:

Records keyed BL-N03-07 go through the partitioner to partition 2, records keyed BL-C08-02 to partition 4. Records without a key can land in any partition.

What to notice: both of BL-N03-07's records go to the same partition. The record with no key could go anywhere.

There's a trap in the hash. The Java client hashes keys with murmur2. The Python client, like every client built on librdkafka (Go, .NET, Node.js), uses CRC32 by default (partitioner=consistent_random). The same key then lands in different partitions depending on the language of the producer. Set partitioner=murmur2_random in librdkafka clients and they agree with Java. All the course's code does.

Order is per partition

Kafka keeps records in the order they arrive within one partition. A consumer reads each partition in that order. Between partitions there is no order at all: a consumer may read partition 4 before partition 1, or interleave them.

So the rule is: records that must stay in order must share a partition. With one producer and the same key, they do. Lesson 2.2 adds one condition: without idempotence, a retried batch can overtake the next one, even within a partition.

Choosing a key

Pick as key the thing whose events must stay in order. For Brightlane that's the charger: a charger's session start, readings, status changes and stop must be read in the order they happened. A session id would keep each session in order, but not a charger's status changes between sessions.

Then check two more things:

If order doesn't matter for an event, no key is fine: the partitioner spreads the load best.

Partition count, now and later

You choose the number of partitions when you create a topic. Two facts make that choice stick:

With 6 partitions, BL-C08-02's events go to partition 4. After the topic grows to 8 partitions its new events go to partition 0, while its older events stay in partition 4.

What to notice: the events at 17:53 and 17:54 now sit in different partitions, with nothing to keep them in order.

Size a topic for the parallelism you expect to need: a partition is read by at most one member of a consumer group (module 3), so 6 partitions cap a group at 6 workers. Leave room to grow, but don't create thousands "just in case". Every partition costs open files and memory on the brokers and in each client.

In production

Practical

Note

Notebook: code/module-02/01-keys-decide-order.ipynb · Lab services: kafka-1, kafka-2, kafka-3 · Time: ~15 min

What you'll do: write 400 session events into a 6-partition topic without a key, keyed by charger, and keyed by region, and check order and load each time. Then compare Python's and Java's partitioners, and grow a topic from 6 to 8 partitions.

Step 1: Write session events without a key

A helper, write_and_read(topic, key_of, producer_config), recreates a 6-partition topic, writes the 400 events of 200 sessions with the key key_of(charger_id), and reads them back with one consumer. check_order() counts sessions whose stop was read before their start.

The real gateway sends events spread over the evening, so each one would land in a random partition. The notebook writes all 400 in a burst, so it turns off the 10 ms "stickiness":

code/module-02/01-keys-decide-order.ipynb
no_key = write_and_read("m02l01-no-key", lambda charger: None, {"sticky.partitioning.linger.ms": 0})
check_order(no_key)

Output (the numbers vary from run to run):

sessions read stop-before-start: 78 of 200
chargers by number of partitions their events landed in: {1: 23, 2: 124, 3: 19, 4: 5}

Step 2: Key by charger id

code/module-02/01-keys-decide-order.ipynb
by_charger = write_and_read("m02l01-by-charger", lambda charger: charger,
                            {"partitioner": "murmur2_random"})
check_order(by_charger)
sessions read stop-before-start: 0 of 200
chargers by number of partitions their events landed in: {1: 171}
records per partition: [(0, 84), (1, 62), (2, 46), (3, 70), (4, 66), (5, 72)]

Every one of the 171 chargers in these sessions sits in exactly one partition, and every partition gets a share.

Step 3: A key with too few values

code/module-02/01-keys-decide-order.ipynb
by_region = write_and_read("m02l01-by-region", lambda charger: charger.split("-")[1][0],
                           {"partitioner": "murmur2_random"})
records per partition: [(0, 250), (4, 150)]

Regions N and V hash to the same partition, so partition 0 carries 62% of the traffic and four partitions get nothing.

Break it: two languages, two partitions

Predict first: a Java service and a Python service both write events keyed BL-N03-07. Do they land in the same partition?

The notebook runs Kafka's console producer, a Java program, inside kafka-1:

code/module-02/01-keys-decide-order.ipynb
lab.kafka_cli("kafka-console-producer.sh", "--bootstrap-server", "localhost:19092", "--topic", T,
              "--property", "parse.key=true", "--property", "key.separator=:",
              stdin="BL-N03-07:from-java\n")

Then it writes the same key from Python, once with the default partitioner and once with murmur2_random, and reads back where each record went:

from-java              -> partition 2
python murmur2_random  -> partition 2
python default         -> partition 5

Change it: add partitions to a live topic

Predict first: the keyed topic grows from 6 to 8 partitions. How many of the 240 chargers will now write to a different partition?

code/module-02/01-keys-decide-order.ipynb
admin.create_partitions([NewPartitions(T, 8)])[T].result()
chargers = [c["charger_id"] for c in Brightlane().chargers]
moved = [ch for ch in chargers if murmur2(ch.encode(), 6) != murmur2(ch.encode(), 8)]

murmur2() is the client's own copy of Java's partitioning function. Output:

172 of 240 chargers change partition
BL-N03-07: partition 2 -> 2
BL-C08-02: partition 4 -> 0

You also see a log line from the producer still open in the notebook: partition count changed from 6 to 8. Clients notice the new partitions on their next metadata refresh.

What just happened

Without a key, the partitioner spread each charger's events over two to four partitions, and the consumer read 78 sessions' stops before their starts: order holds only within a partition. Keyed by charger, every charger stayed in one partition and nothing was out of order, while the load still spread over all six. Keyed by region, order held but only two partitions did any work: too few key values. The break-it step showed the partitioner trap: the Java producer and Python's murmur2_random agreed on partition 2, but Python's default CRC32 put the same key in partition 5. Finally, growing the topic moved 172 of 240 chargers to new partitions. Their old and new events now sit apart, which is why partition counts are chosen up front.

Clean up

The last cell deletes the three topics and the consumer group. If anything gets stuck, docker compose down -v in the code/ folder resets the whole lab.

Key takeaways

Next: Durable writes: acks, replicas and idempotence