Records, offsets and replay
You've written and read events. This lesson looks closer at one event as Kafka stores it, at offsets, and at reading from any point you choose: a position, or a time.
Theory
Example: a disputed charge
On Monday morning a driver, drv-13280, writes to support. Her receipt for session ses-f2873b on charger BL-N03-07 says 47.1 kWh. She says she unplugged at about 18:00, but she was billed until 18:16.
Support needs every event that BL-N03-07 sent from 17:40 onwards: the session start, every 10-second meter reading and the stop. The evening's telemetry topic holds about 9,700 events from 40 sessions on dozens of chargers. Support doesn't want to read all of them. They want to jump straight to 17:40 on one charger, read forward, and be able to do it again tomorrow and get the same answer.
What a record carries
Kafka calls each stored event a record. A producer sets four parts:
- Key: optional bytes that say what the record is about. Brightlane uses the charger id. Lesson 2.1 shows that the key decides the partition.
- Value: the event itself, as bytes. Kafka never reads or checks it. Turning an object into bytes is serializing; Brightlane serializes to JSON here, and to Avro from module 5.
- Headers: optional name–value pairs for technical metadata such as a trace id or the firmware version. Keep business data in the value, where every reader expects it.
- Timestamp: milliseconds since 1970. By default the producer's clock sets it (type
CreateTime). A topic can be set toLogAppendTime, where the broker's clock overwrites it when the record is stored.
When the broker stores the record, it adds the partition and the offset.
What to notice: the left side belongs to your code. Kafka only adds the two fields on the right.
Offsets in an append-only log
Each partition numbers its records 0, 1, 2… in the order they arrive. The number is the record's offset. Offsets are never reused and records are never changed. Old ones disappear only when retention removes them (module 4). So "partition 2, offset 461" names one record for as long as it exists, and reading it again gives the same bytes.
Each partition has an end offset: the offset the next record will get. The difference between a reader's position and the end offset is how far behind that reader is.
Offsets give the order of arrival, not the order of the timestamps. If a charger's mobile link drops for five minutes and it then sends a 17:35 reading, that reading gets the next offset at the end, after records stamped 17:41. Lesson 7.3 deals with such late events.
Reading by position and by time
Lesson 1.1 used subscribe: you name a topic, and Kafka's group mechanism hands you partitions and remembers your position. For one-off jobs like support's, there's a second way: assign. You name the exact partitions and the offset to start from yourself. Kafka doesn't store a position for you and doesn't share the partitions with anyone.
To start from a time instead of an offset, ask the broker with offsets_for_times: "what is
the first offset in partition 2 whose timestamp is at or after 17:40?" Each partition keeps a
time index for exactly this question. Then you assign that offset and read forward.
What to notice: the support reader and billing read the same partition from different places, and the late reading sits at the end despite its early timestamp.
In production
- Record size. Brokers accept batches up to
message.max.bytes, about 1 MB by default. Keep events small and send links to large files. - Future timestamps are rejected. Since Kafka 4.0 a broker refuses records stamped more than
one hour ahead of its own clock (
message.timestamp.after.max.ms). A charger with a wrong clock getsINVALID_TIMESTAMPerrors. Keep the device's time as a field in the value, and set record timestamps only from clocks you trust. - Old timestamps can disappear early. Time-based retention (module 4) looks at record timestamps. Records produced today but stamped three weeks ago are deleted as soon as the broker checks, with the default 7-day retention.
- Replay is cheap but not free. Re-reading a day of data puts real load on the brokers and the network. Plan big replays like any batch job.
Practical
Note
Notebook: code/module-01/03-records-offsets-and-replay.ipynb
· Lab services: kafka-1, kafka-2, kafka-3 · Time: ~17 min
What you'll do: take one record apart, write an evening of charger events, and answer the driver's dispute by reading one charger from 17:40, twice. Then add a late reading and see where it lands.
Step 1: Write one record and look at all its parts
producer = Producer({"bootstrap.servers": settings.KAFKA_BOOTSTRAP})
producer.produce(ANATOMY, key="BL-N03-07",
value=to_json({"charger_id": "BL-N03-07", "power_kw": 0.0, "ts": iso(BASE_TIME)}),
headers={"firmware": "4.2.1", "trace-id": "a91f3c"})
print("not yet stored:", producer.flush(10))
The notebook reads it back with read_all(), a helper that assigns partitions and reads up to
their current end offsets:
topic m01l03-anatomy | partition 2 | offset 0
key b'BL-N03-07'
value b'{"charger_id":"BL-N03-07","power_kw":0.0,"ts":"2026-10-05T17:30:00Z"}'
headers [('firmware', b'4.2.1'), ('trace-id', b'a91f3c')]
timestamp 1791279777591 (type 1 = CreateTime, set by the producer)
The key, value and header values come back as bytes: Kafka doesn't know they were text. We didn't set a timestamp, so the producer used "now". Your number will differ.
Step 2: Write an evening of meter readings
The gateway uses each charger's reading time as the record timestamp:
for charger_id, event in events:
ts_ms = int(parse_iso(event["ts"]).timestamp() * 1000)
producer.produce(TOPIC, key=charger_id, value=to_json(event), timestamp=ts_ms)
producer.poll(0) # serve delivery events while producing
wrote 9713 records, not yet stored: 0
Step 3: Find the disputed charger's partition
BL-N03-07 -> partition 2: 227 records
partition 2 holds offsets 0 to 2866 (2867 records in total)
All 227 records from BL-N03-07 are in partition 2, mixed with records from other chargers that share the partition.
Step 4: Jump to 17:40 by time
at_1740 = int((BASE_TIME + timedelta(minutes=10)).timestamp() * 1000) # BASE_TIME is 17:30 yesterday
found = c.offsets_for_times([TopicPartition(TOPIC, partition, at_1740)], timeout=10)[0]
The notebook then reads partition 2 from found.offset and keeps BL-N03-07's records:
first offset at or after 17:40 in partition 2: 456
461 2026-10-05T17:40:05Z reading 2.091
473 2026-10-05T17:40:15Z reading 2.302
486 2026-10-05T17:40:25Z reading 2.515
... 217 records for BL-N03-07 since 17:40
Offset 456 belongs to another charger's 17:40 reading. BL-N03-07's first reading after 17:40 is at offset 461. Its meter already showed 2.091 kWh, because the session started at 17:38:25.
Step 5: Replay: read the same range again
second read: 217 records, identical to the first: True
Change it: a reading that arrives late
Predict first: a 17:35 reading from BL-N03-07 reaches the gateway only now. Where is it stored, and does a time search for 17:30 still find the start of the evening?
producer.produce(TOPIC, key="BL-N03-07", value=to_json(late),
timestamp=int(parse_iso(late["ts"]).timestamp() * 1000))
late record stored at offset 2867 with timestamp 2026-10-05T17:35:00Z
time search for 17:30 starts at offset 0
What just happened
Step 1 showed a record's four producer-set parts, and the partition and offset the broker added.
Kafka stored the value as opaque bytes. Steps 3 and 4 used the two ways to address data in Kafka: a
partition plus an offset, and a timestamp turned into an offset by offsets_for_times. Support
assigned the partition directly, so no group position was stored. Step 5 re-read the same offsets
and got identical records: nothing is changed or removed by reading, which is what makes replay
safe. In the change-it step the 17:35 reading got offset 2867, the end of the log, after records
stamped later: offsets follow arrival order, timestamps don't. The time search for 17:30 still
starts at offset 0, and reading forward from there reaches the late record too.
Clean up
The last cell deletes both topics. If anything gets stuck, docker compose down -v in the code/
folder resets the whole lab.
Key takeaways
- A record has a key, a value (bytes), headers and a timestamp; the broker adds partition and offset.
- Offsets number records per partition in arrival order. They are never reused or changed.
subscribelets a group track your position;assignlets you read exact partitions from an offset you choose.offsets_for_timesturns a time into a starting offset, for replays from a point in time.- Since Kafka 4.0, brokers reject records stamped more than an hour in the future.
Next: Module 1 exercises