Download 02-segments-retention-and-deleting-data.ipynb
Open it in JupyterLab or VS Code from the lab folder to run it. This page is a read-only view.
Segments, retention and deleting data — practical
Two telemetry topics keep readings for 60 seconds. One rolls a new segment every 20 seconds, the other uses the default segment settings. You watch the segment files on the broker, measure how old the oldest reading really is, fix the second topic, watch a late consumer lose readings, and delete records on request.
Lab services: kafka-1, kafka-2, kafka-3 · Script: module-04/steady_writer.py
· Lesson: modules/04-inside-the-cluster/02-segments-retention-and-deleting-data.md
# Setup: find the lab's shared helpers and check the lab is running import sys, pathlib lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / "common").is_dir()) sys.path.insert(0, str(lab_root)) import time from datetime import datetime from common import settings, lab lab.require_kafka() lab.print_versions()
Step 1: Two topics, two writers, one archive job
Both topics keep readings for 60 seconds (retention.ms). Only the first one also closes its
segment every 20 seconds (segment.ms). A writer sends 5 readings a second to each. (The lab's
brokers check for expired segments every 10 seconds instead of every 5 minutes.)
from confluent_kafka import Consumer, TopicPartition from confluent_kafka.admin import AdminClient SMALL, DEFAULT = "m04l02-telemetry", "m04l02-telemetry-default" admin = AdminClient({"bootstrap.servers": settings.KAFKA_BOOTSTRAP}) lab.recreate_topic(admin, SMALL, partitions=1, config={"retention.ms": "60000", "segment.ms": "20000"}) lab.recreate_topic(admin, DEFAULT, partitions=1, config={"retention.ms": "60000"}) admin.delete_consumer_groups(["m04l02-archive"]) writers = [lab.start_script("module-04/steady_writer.py", "--topic", t, "--rate", "5", name=f"{t}-writer") for t in (SMALL, DEFAULT)]
An archive job reads the first 20 readings of the first topic, commits its position and stops for "maintenance". It comes back in the break-it step.
archive = Consumer({"bootstrap.servers": settings.KAFKA_BOOTSTRAP, "group.id": "m04l02-archive", "auto.offset.reset": "earliest", "enable.auto.commit": False}) archive.subscribe([SMALL]) archived = 0 while archived < 20: msg = archive.poll(5) if msg is not None and not msg.error(): archived += 1 archive.commit(message=msg, asynchronous=False) archive.close() print("archive job stopped after offset", msg.offset())
Step 2: Watch segments come and go
segments(topic) lists the partition's segment files on its leader: each file is named after the
first offset it holds. oldest(topic) reads the first record still in the partition and returns
its age in seconds.
def segments(topic): """(first offset, size in bytes) of each segment file of partition 0, on the leader's disk.""" leader = admin.list_topics(topic, timeout=10).topics[topic].partitions[0].leader listing = lab.compose("exec", "-T", f"kafka-{leader}", "ls", "-l", f"/var/lib/kafka/data/{topic}-0", echo=False) return [(int(line.split()[-1].split(".")[0]), int(line.split()[4])) for line in listing.splitlines() if line.endswith(".log")] peek = Consumer({"bootstrap.servers": settings.KAFKA_BOOTSTRAP, "group.id": "m04l02-peek", "enable.auto.commit": False}) def oldest(topic): """(log start offset, end offset, age in seconds of the oldest record still readable).""" start, end = peek.get_watermark_offsets(TopicPartition(topic, 0), timeout=10) peek.assign([TopicPartition(topic, 0, start)]) msg = peek.poll(10) return start, end, round(time.time() - msg.timestamp()[1] / 1000) def watch(seconds, every=10): rows = [] for _ in range(seconds // every): time.sleep(every) rows.append((datetime.now().strftime("%H:%M:%S"), [first for first, _ in segments(SMALL)], oldest(SMALL)[2], len(segments(DEFAULT)), oldest(DEFAULT)[2])) lab.table(rows, ["time", "segments (first offsets)", "oldest s", "default: segments", "default: oldest s"]) watch(120)
On the first topic a new segment appears every 20 seconds, and the oldest segment disappears once its newest reading is older than 60 seconds. So the oldest reading's age climbs to about 80–90 seconds, then drops. On the second topic there is one segment, and nothing is ever deleted: the oldest reading just gets older.
Step 3: Change it: give the default topic smaller segments
Set segment.ms on the second topic too, and watch for another 90 seconds.
from confluent_kafka.admin import ConfigResource, ConfigEntry, AlterConfigOpType fix = ConfigResource("topic", DEFAULT, incremental_configs=[ ConfigEntry("segment.ms", "20000", incremental_operation=AlterConfigOpType.SET)]) admin.incremental_alter_configs([fix])[fix].result() watch(90)
The big segment closed at the next write, but it was only deleted about 60 seconds later: retention counts from the newest record in a segment, and that one had just been written.
Break it: the archive job comes back
The archive job committed offset 20 about four minutes ago. Predict first: where does it continue?
archive = Consumer({"bootstrap.servers": settings.KAFKA_BOOTSTRAP, "group.id": "m04l02-archive", "auto.offset.reset": "earliest", "enable.auto.commit": False}) committed = archive.committed([TopicPartition(SMALL, 0)], timeout=10)[0].offset archive.subscribe([SMALL]) msg = None while msg is None or msg.error(): msg = archive.poll(5) archive.close() start, end, age = oldest(SMALL) print(f"committed offset: {committed} log start offset now: {start} first reading it got: offset {msg.offset()}") print(f"readings the archive job never saw: {msg.offset() - committed}")
Its committed offset no longer exists. The consumer got an "offset out of range" answer, fell back
to auto.offset.reset (here earliest, the log start offset), and continued from there. With
latest it would have skipped everything up to the newest reading. Either way, no error reached
your code.
Step 4: Delete records on request
A privacy request: delete every reading in the first topic that was written before now.
delete_records() moves the partition's log start offset.
start, end, _ = oldest(SMALL) result = admin.delete_records([TopicPartition(SMALL, 0, end)]) for tp, future in result.items(): print("new log start offset:", future.result().low_watermark) print("segment files right after:", segments(SMALL)) time.sleep(15) print("segment files 15 s later:", segments(SMALL))
Consumers can't read below the new log start offset at once. The segment files that only held deleted readings go at the next retention check. A segment that also holds newer readings stays until retention removes it.
Clean up
Stop the writers and delete both topics and the groups. If anything gets stuck,
docker compose down -v in the code/ folder resets the whole lab.
for w in writers: lab.stop_script(w) peek.close() lab.delete_topics(admin, [SMALL, DEFAULT]) admin.delete_consumer_groups(["m04l02-archive", "m04l02-peek"]) print("cleaned up")