Keys decide order
Module 1 wrote events without thinking about where they went. This lesson shows how a producer picks a partition for each record, what that means for order, and how to choose a key and a partition count you won't regret.
Theory
Example: the stops that came first
Brightlane's gateway team wanted to spread the load evenly over the six partitions of
charging.sessions. They removed the record key: "the partitioner will balance it". The next
evening billing logged 31 warnings like this one:
WARN billing: session_stopped for unknown session ses-7f3a19 (charger BL-N03-04), skipped
Billing builds a session when it reads session_started and charges it when it reads
session_stopped. For 31 sessions it read the stop before the start, so it skipped them, and
31 drivers charged for free. Nothing was lost: both events were in the topic. They were just in
different partitions, and Kafka keeps no order between partitions.
How the producer picks a partition
The producer, not the broker, decides which partition each record goes to. The part of the client that decides is the partitioner:
- With a key, it hashes the key and takes the result modulo the number of partitions. The same key always gives the same partition, as long as the partition count doesn't change.
- Without a key, it spreads records over the partitions. The Python client picks a partition
at random, and keeps using it for about 10 ms (
sticky.partitioning.linger.ms) so batches fill up. The Java client does something similar. - Your code can also name a partition explicitly. You rarely should.
What to notice: both of BL-N03-07's records go to the same partition. The record with no key could go anywhere.
There's a trap in the hash. The Java client hashes keys with murmur2. The Python client, like
every client built on librdkafka (Go, .NET, Node.js), uses CRC32 by default
(partitioner=consistent_random). The same key then lands in different partitions depending on
the language of the producer. Set partitioner=murmur2_random in librdkafka clients and they
agree with Java. All the course's code does.
Order is per partition
Kafka keeps records in the order they arrive within one partition. A consumer reads each partition in that order. Between partitions there is no order at all: a consumer may read partition 4 before partition 1, or interleave them.
So the rule is: records that must stay in order must share a partition. With one producer and the same key, they do. Lesson 2.2 adds one condition: without idempotence, a retried batch can overtake the next one, even within a partition.
Choosing a key
Pick as key the thing whose events must stay in order. For Brightlane that's the charger: a charger's session start, readings, status changes and stop must be read in the order they happened. A session id would keep each session in order, but not a charger's status changes between sessions.
Then check two more things:
- Enough distinct values. With 240 chargers and 6 partitions, each partition gets about 40 chargers. With 3 regions as the key, at most 3 partitions get any data at all.
- Evenly busy values. A key that produces most of the traffic (a hot key) makes one partition the bottleneck, however many partitions you have. One busy site's 10 chargers are fine. A single key for "all of site N03" would be hot.
If order doesn't matter for an event, no key is fine: the partitioner spreads the load best.
Partition count, now and later
You choose the number of partitions when you create a topic. Two facts make that choice stick:
- You can add partitions later, but never remove them.
- Adding partitions changes
hash % partitions, so most keys move to a new partition. A key's old records stay where they were, and its new ones go elsewhere, so readers can see them out of order during the change.
What to notice: the events at 17:53 and 17:54 now sit in different partitions, with nothing to keep them in order.
Size a topic for the parallelism you expect to need: a partition is read by at most one member of a consumer group (module 3), so 6 partitions cap a group at 6 workers. Leave room to grow, but don't create thousands "just in case". Every partition costs open files and memory on the brokers and in each client.
In production
- Set
partitioner=murmur2_randomin every librdkafka-based producer, whatever its language, so all producers agree with Java clients and with Kafka Streams. - Create topics deliberately, with a partition count you chose, never through auto-creation.
- If a keyed topic must grow, plan it: either accept a short window of disorder, or create a new topic with more partitions, copy the data, and move producers and consumers over.
- Watch records per partition. A partition far busier than the others means a hot key.
Practical
Note
Notebook: code/module-02/01-keys-decide-order.ipynb
· Lab services: kafka-1, kafka-2, kafka-3 · Time: ~15 min
What you'll do: write 400 session events into a 6-partition topic without a key, keyed by charger, and keyed by region, and check order and load each time. Then compare Python's and Java's partitioners, and grow a topic from 6 to 8 partitions.
Step 1: Write session events without a key
A helper, write_and_read(topic, key_of, producer_config), recreates a 6-partition topic, writes
the 400 events of 200 sessions with the key key_of(charger_id), and reads them back with one
consumer. check_order() counts sessions whose stop was read before their start.
The real gateway sends events spread over the evening, so each one would land in a random partition. The notebook writes all 400 in a burst, so it turns off the 10 ms "stickiness":
no_key = write_and_read("m02l01-no-key", lambda charger: None, {"sticky.partitioning.linger.ms": 0})
check_order(no_key)
Output (the numbers vary from run to run):
sessions read stop-before-start: 78 of 200
chargers by number of partitions their events landed in: {1: 23, 2: 124, 3: 19, 4: 5}
Step 2: Key by charger id
by_charger = write_and_read("m02l01-by-charger", lambda charger: charger,
{"partitioner": "murmur2_random"})
check_order(by_charger)
sessions read stop-before-start: 0 of 200
chargers by number of partitions their events landed in: {1: 171}
records per partition: [(0, 84), (1, 62), (2, 46), (3, 70), (4, 66), (5, 72)]
Every one of the 171 chargers in these sessions sits in exactly one partition, and every partition gets a share.
Step 3: A key with too few values
by_region = write_and_read("m02l01-by-region", lambda charger: charger.split("-")[1][0],
{"partitioner": "murmur2_random"})
records per partition: [(0, 250), (4, 150)]
Regions N and V hash to the same partition, so partition 0 carries 62% of the traffic and four partitions get nothing.
Break it: two languages, two partitions
Predict first: a Java service and a Python service both write events keyed BL-N03-07. Do they
land in the same partition?
The notebook runs Kafka's console producer, a Java program, inside kafka-1:
lab.kafka_cli("kafka-console-producer.sh", "--bootstrap-server", "localhost:19092", "--topic", T,
"--property", "parse.key=true", "--property", "key.separator=:",
stdin="BL-N03-07:from-java\n")
Then it writes the same key from Python, once with the default partitioner and once with
murmur2_random, and reads back where each record went:
from-java -> partition 2
python murmur2_random -> partition 2
python default -> partition 5
Change it: add partitions to a live topic
Predict first: the keyed topic grows from 6 to 8 partitions. How many of the 240 chargers will now write to a different partition?
admin.create_partitions([NewPartitions(T, 8)])[T].result()
chargers = [c["charger_id"] for c in Brightlane().chargers]
moved = [ch for ch in chargers if murmur2(ch.encode(), 6) != murmur2(ch.encode(), 8)]
murmur2() is the client's own copy of Java's partitioning function. Output:
172 of 240 chargers change partition
BL-N03-07: partition 2 -> 2
BL-C08-02: partition 4 -> 0
You also see a log line from the producer still open in the notebook: partition count changed
from 6 to 8. Clients notice the new partitions on their next metadata refresh.
What just happened
Without a key, the partitioner spread each charger's events over two to four partitions, and the
consumer read 78 sessions' stops before their starts: order holds only within a partition. Keyed
by charger, every charger stayed in one partition and nothing was out of order, while the load
still spread over all six. Keyed by region, order held but only two partitions did any work: too
few key values. The break-it step showed the partitioner trap: the Java producer and Python's
murmur2_random agreed on partition 2, but Python's default CRC32 put the same key in partition 5.
Finally, growing the topic moved 172 of 240 chargers to new partitions. Their old and new events
now sit apart, which is why partition counts are chosen up front.
Clean up
The last cell deletes the three topics and the consumer group. If anything gets stuck,
docker compose down -v in the code/ folder resets the whole lab.
Key takeaways
- The producer's partitioner picks the partition:
hash(key) % partitions, or a spread when there's no key. - Kafka keeps order only within a partition, so records that must stay in order need the same key.
- Choose as key the entity whose events must stay ordered, with many, evenly busy values.
- Set
partitioner=murmur2_randomin Python (and every librdkafka client) to match Java. - Partitions can be added but never removed, and adding them moves most keys. Size topics up front.