Notebook · code/module-04/02-segments-retention-and-deleting-data.ipynb

Download 02-segments-retention-and-deleting-data.ipynb

Open it in JupyterLab or VS Code from the lab folder to run it. This page is a read-only view.

Segments, retention and deleting data — practical

Two telemetry topics keep readings for 60 seconds. One rolls a new segment every 20 seconds, the other uses the default segment settings. You watch the segment files on the broker, measure how old the oldest reading really is, fix the second topic, watch a late consumer lose readings, and delete records on request.

Lab services: kafka-1, kafka-2, kafka-3 · Script: module-04/steady_writer.py · Lesson: modules/04-inside-the-cluster/02-segments-retention-and-deleting-data.md

# Setup: find the lab's shared helpers and check the lab is running
import sys, pathlib
lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / "common").is_dir())
sys.path.insert(0, str(lab_root))

import time
from datetime import datetime
from common import settings, lab

lab.require_kafka()
lab.print_versions()

Step 1: Two topics, two writers, one archive job

Both topics keep readings for 60 seconds (retention.ms). Only the first one also closes its segment every 20 seconds (segment.ms). A writer sends 5 readings a second to each. (The lab's brokers check for expired segments every 10 seconds instead of every 5 minutes.)

from confluent_kafka import Consumer, TopicPartition
from confluent_kafka.admin import AdminClient

SMALL, DEFAULT = "m04l02-telemetry", "m04l02-telemetry-default"
admin = AdminClient({"bootstrap.servers": settings.KAFKA_BOOTSTRAP})
lab.recreate_topic(admin, SMALL, partitions=1, config={"retention.ms": "60000", "segment.ms": "20000"})
lab.recreate_topic(admin, DEFAULT, partitions=1, config={"retention.ms": "60000"})
admin.delete_consumer_groups(["m04l02-archive"])
writers = [lab.start_script("module-04/steady_writer.py", "--topic", t, "--rate", "5", name=f"{t}-writer")
           for t in (SMALL, DEFAULT)]

An archive job reads the first 20 readings of the first topic, commits its position and stops for "maintenance". It comes back in the break-it step.

archive = Consumer({"bootstrap.servers": settings.KAFKA_BOOTSTRAP, "group.id": "m04l02-archive",
                    "auto.offset.reset": "earliest", "enable.auto.commit": False})
archive.subscribe([SMALL])
archived = 0
while archived < 20:
    msg = archive.poll(5)
    if msg is not None and not msg.error():
        archived += 1
archive.commit(message=msg, asynchronous=False)
archive.close()
print("archive job stopped after offset", msg.offset())

Step 2: Watch segments come and go

segments(topic) lists the partition's segment files on its leader: each file is named after the first offset it holds. oldest(topic) reads the first record still in the partition and returns its age in seconds.

def segments(topic):
    """(first offset, size in bytes) of each segment file of partition 0, on the leader's disk."""
    leader = admin.list_topics(topic, timeout=10).topics[topic].partitions[0].leader
    listing = lab.compose("exec", "-T", f"kafka-{leader}", "ls", "-l", f"/var/lib/kafka/data/{topic}-0", echo=False)
    return [(int(line.split()[-1].split(".")[0]), int(line.split()[4]))
            for line in listing.splitlines() if line.endswith(".log")]

peek = Consumer({"bootstrap.servers": settings.KAFKA_BOOTSTRAP, "group.id": "m04l02-peek",
                 "enable.auto.commit": False})

def oldest(topic):
    """(log start offset, end offset, age in seconds of the oldest record still readable)."""
    start, end = peek.get_watermark_offsets(TopicPartition(topic, 0), timeout=10)
    peek.assign([TopicPartition(topic, 0, start)])
    msg = peek.poll(10)
    return start, end, round(time.time() - msg.timestamp()[1] / 1000)

def watch(seconds, every=10):
    rows = []
    for _ in range(seconds // every):
        time.sleep(every)
        rows.append((datetime.now().strftime("%H:%M:%S"), [first for first, _ in segments(SMALL)],
                     oldest(SMALL)[2], len(segments(DEFAULT)), oldest(DEFAULT)[2]))
    lab.table(rows, ["time", "segments (first offsets)", "oldest s", "default: segments", "default: oldest s"])

watch(120)

On the first topic a new segment appears every 20 seconds, and the oldest segment disappears once its newest reading is older than 60 seconds. So the oldest reading's age climbs to about 80–90 seconds, then drops. On the second topic there is one segment, and nothing is ever deleted: the oldest reading just gets older.

Step 3: Change it: give the default topic smaller segments

Set segment.ms on the second topic too, and watch for another 90 seconds.

from confluent_kafka.admin import ConfigResource, ConfigEntry, AlterConfigOpType

fix = ConfigResource("topic", DEFAULT, incremental_configs=[
    ConfigEntry("segment.ms", "20000", incremental_operation=AlterConfigOpType.SET)])
admin.incremental_alter_configs([fix])[fix].result()
watch(90)

The big segment closed at the next write, but it was only deleted about 60 seconds later: retention counts from the newest record in a segment, and that one had just been written.

Break it: the archive job comes back

The archive job committed offset 20 about four minutes ago. Predict first: where does it continue?

archive = Consumer({"bootstrap.servers": settings.KAFKA_BOOTSTRAP, "group.id": "m04l02-archive",
                    "auto.offset.reset": "earliest", "enable.auto.commit": False})
committed = archive.committed([TopicPartition(SMALL, 0)], timeout=10)[0].offset
archive.subscribe([SMALL])
msg = None
while msg is None or msg.error():
    msg = archive.poll(5)
archive.close()
start, end, age = oldest(SMALL)
print(f"committed offset: {committed}   log start offset now: {start}   first reading it got: offset {msg.offset()}")
print(f"readings the archive job never saw: {msg.offset() - committed}")

Its committed offset no longer exists. The consumer got an "offset out of range" answer, fell back to auto.offset.reset (here earliest, the log start offset), and continued from there. With latest it would have skipped everything up to the newest reading. Either way, no error reached your code.

Step 4: Delete records on request

A privacy request: delete every reading in the first topic that was written before now. delete_records() moves the partition's log start offset.

start, end, _ = oldest(SMALL)
result = admin.delete_records([TopicPartition(SMALL, 0, end)])
for tp, future in result.items():
    print("new log start offset:", future.result().low_watermark)
print("segment files right after:", segments(SMALL))
time.sleep(15)
print("segment files 15 s later:", segments(SMALL))

Consumers can't read below the new log start offset at once. The segment files that only held deleted readings go at the next retention check. A segment that also holds newer readings stays until retention removes it.

Clean up

Stop the writers and delete both topics and the groups. If anything gets stuck, docker compose down -v in the code/ folder resets the whole lab.

for w in writers:
    lab.stop_script(w)
peek.close()
lab.delete_topics(admin, [SMALL, DEFAULT])
admin.delete_consumer_groups(["m04l02-archive", "m04l02-peek"])
print("cleaned up")