28 min · theory + hands-on practical

Segments, retention and deleting data

Kafka keeps records for a set time and then deletes them by itself. That sounds simple, until an audit finds data that should have been gone days ago. This lesson shows how a partition is stored on disk, when records really disappear, and what you can do when someone asks you to delete data now.

Theory

Example: the 9-day-old readings

Brightlane's meter readings are linked to sessions, and sessions to drivers. So the privacy officer set a rule: telemetry must be gone after 72 hours. The platform team set retention.ms on charging.telemetry to 72 hours (259,200,000 ms) and closed the ticket.

Two months later, an audit read the oldest reading in each partition. Some were 9 days old. Nobody had changed the setting, and the brokers were working normally. The setting was right; the assumption about what it means was wrong.

Segments

Each replica of a partition is stored in its own folder on the broker, as a series of files called segments. Kafka only ever appends to the newest one, the active segment. Each segment file is named after the first offset it holds: 00000000000000000000.log, then 00000000000000000103.log, and so on, each with small index files next to it.

Kafka closes the active segment and starts a new one when the segment reaches segment.bytes (1 GiB by default), or when its first record is older than segment.ms (7 days by default). It checks this when the next record arrives.

How retention really works

With cleanup.policy=delete (the default), each broker looks for expired segments every log.retention.check.interval.ms (5 minutes). It deletes a closed segment once the newest record in it is older than retention.ms (7 days by default). It also deletes the oldest segments while the partition is bigger than retention.bytes (no limit by default; the limit is per partition).

Top: with the default segment.ms of 7 days and retention.ms of 72 hours, a quiet partition has one active segment holding days 1 to 9 of readings; it is still being written, so it is never deleted. Bottom: with segment.ms of 6 hours and retention.ms of 60 hours, the partition has many closed segments, each labelled with the age of its newest reading; the segment whose newest reading is 66 hours old is deleted, the others and the active segment stay. The oldest readable reading is at most retention.ms plus segment.ms plus the check interval old.

What to notice: retention deletes whole segments, never the active one, and counts from a segment's newest record.

Three things follow:

The first offset still readable is the partition's log start offset. A consumer group whose committed offset falls below it, because it stopped for longer than the retention, gets "offset out of range" from the broker. The client then silently applies auto.offset.reset and continues from the log start (earliest) or the end (latest). The records in between are skipped without an error.

Deleting records before their time

delete_records() in the admin client (or kafka-delete-records.sh) moves a partition's log start offset forward to an offset you choose. Every record before it is unreadable at once, and segments that only held such records are deleted at the next check.

You can only cut off the beginning of a partition. Kafka has no way to delete one record from the middle of a topic with the delete policy. When a single driver asks to be forgotten, you have three tools:

Tiered storage

Tiered storage (generally available since Kafka 3.9) copies closed segments to object storage such as S3, and keeps only the recent ones on the brokers' disks. Consumers can still read the old segments; they are just slower to fetch. You turn it on per broker (remote.log.storage.system.enable) and per topic (remote.storage.enable), and choose the local part with local.retention.ms. retention.ms still decides when data is gone everywhere. Kafka itself ships no production plug-in for the object store, so you add one, such as Aiven's open-source plug-in. Compacted topics can't use tiered storage. The lab doesn't run it.

In production

Practical

Note

Script: code/module-04/steady_writer.py (the notebook starts two copies) · Notebook: code/module-04/02-segments-retention-and-deleting-data.ipynb · Lab services: kafka-1, kafka-2, kafka-3 · Time: ~16 min (about 4 minutes of waiting)

What you'll do: watch two topics with the same 60-second retention, one with 20-second segments and one with the default. Then fix the second one, bring back a consumer that stopped for too long, and delete records on request.

The lab's brokers look for expired segments every 10 seconds, not every 5 minutes, so you see retention within the lesson. That's set in compose.yaml:

code/compose.yaml
  KAFKA_LOG_RETENTION_CHECK_INTERVAL_MS: "10000"

Step 1: Two topics, two writers, one archive job

code/module-04/02-segments-retention-and-deleting-data.ipynb
lab.recreate_topic(admin, SMALL, partitions=1, config={"retention.ms": "60000", "segment.ms": "20000"})
lab.recreate_topic(admin, DEFAULT, partitions=1, config={"retention.ms": "60000"})

A writer sends 5 readings a second to each. An archive job reads the first 20 readings of the first topic, commits, and stops:

archive job stopped after offset 19

Step 2: Watch segments come and go

segments(topic) lists the segment files on the partition leader's disk, and oldest(topic) returns the age of the oldest record still readable:

code/module-04/02-segments-retention-and-deleting-data.ipynb
    listing = lab.compose("exec", "-T", f"kafka-{leader}", "ls", "-l", f"/var/lib/kafka/data/{topic}-0", echo=False)

Every 10 seconds for 2 minutes:

time      segments (first offsets)  oldest s  default: segments  default: oldest s
--------  ------------------------  --------  -----------------  -----------------
17:54:46  [0]                       14        1                  14
17:54:56  [0, 103]                  24        1                  24
...
17:55:37  [0, 103, 204, 304]        65        1                  65
17:55:47  [0, 103, 204, 304]        75        1                  75
17:55:58  [103, 204, 304, 405]      65        1                  86
17:56:08  [103, 204, 304, 405]      76        1                  96
17:56:19  [204, 304, 405, 505]      66        1                  107
...
17:56:39  [304, 405, 505, 606]      66        1                  127

Step 3: Change it: give the default topic smaller segments

code/module-04/02-segments-retention-and-deleting-data.ipynb
fix = ConfigResource("topic", DEFAULT, incremental_configs=[
    ConfigEntry("segment.ms", "20000", incremental_operation=AlterConfigOpType.SET)])
time      segments (first offsets)  oldest s  default: segments  default: oldest s
--------  ------------------------  --------  -----------------  -----------------
17:56:49  [304, 405, 505, 606]      76        2                  137
17:56:59  [405, 505, 606, 706]      66        3                  148
...
17:57:40  [606, 706, 807, 907]      67        5                  188
17:57:51  [606, 706, 807, 907]      77        4                  71
17:58:01  [706, 807, 907, 1007]     68        5                  82

Break it: the archive job comes back

Predict first: the archive job committed offset 20 about four minutes ago. Where does it continue?

committed offset: 20   log start offset now: 706   first reading it got: offset 706
readings the archive job never saw: 686

Step 4: Delete records on request

A privacy request: delete every reading in the first topic written before now.

code/module-04/02-segments-retention-and-deleting-data.ipynb
result = admin.delete_records([TopicPartition(SMALL, 0, end)])
new log start offset: 1099
segment files right after: [(706, 22513), (807, 22292), (907, 22301), (1007, 21048)]
segment files 15 s later: [(1007, 22616), (1108, 15452)]

What just happened

On the first topic, a new segment started about every 20 seconds, and the oldest segment went once its newest reading was older than 60 seconds. So the oldest readable reading was never exactly 60 seconds old: it swung between about 65 and 80 seconds, within the 90 that retention.ms + segment.ms + the check interval allows. The second topic had one active segment, which retention never deletes, so its oldest reading just kept ageing: that's the audit's 9-day-old reading. After segment.ms was set, the big segment closed at the next write, but it was only deleted about 60 seconds later, because its newest reading had only just been written. The archive job's committed offset 20 no longer existed. Its consumer quietly fell back to auto.offset.reset=earliest and skipped 686 readings without an error. Finally, delete_records() made every older reading unreadable at once. The files that only held deleted readings were gone at the next check, while segment 1007, which also held newer readings, stayed.

Clean up

The last cell stops the writers and deletes both topics and the groups. If anything gets stuck, docker compose down -v in the code/ folder resets the whole lab.

Key takeaways

Next: Compacted topics: the latest value per key