{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c00-baf6439b",
   "metadata": {},
   "source": [
    "# Segments, retention and deleting data — practical\n",
    "\n",
    "Two telemetry topics keep readings for 60 seconds. One rolls a new segment every 20 seconds, the\n",
    "other uses the default segment settings. You watch the segment files on the broker, measure how\n",
    "old the oldest reading really is, fix the second topic, watch a late consumer lose readings, and\n",
    "delete records on request.\n",
    "\n",
    "**Lab services:** kafka-1, kafka-2, kafka-3 · **Script:** `module-04/steady_writer.py`\n",
    "· **Lesson:** `modules/04-inside-the-cluster/02-segments-retention-and-deleting-data.md`"
   ]
  },
  {
   "cell_type": "code",
   "id": "c01-6b0b4947",
   "metadata": {},
   "source": [
    "# Setup: find the lab's shared helpers and check the lab is running\n",
    "import sys, pathlib\n",
    "lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / \"common\").is_dir())\n",
    "sys.path.insert(0, str(lab_root))\n",
    "\n",
    "import time\n",
    "from datetime import datetime\n",
    "from common import settings, lab\n",
    "\n",
    "lab.require_kafka()\n",
    "lab.print_versions()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c02-1f046ec7",
   "metadata": {},
   "source": [
    "## Step 1: Two topics, two writers, one archive job\n",
    "Both topics keep readings for 60 seconds (`retention.ms`). Only the first one also closes its\n",
    "segment every 20 seconds (`segment.ms`). A writer sends 5 readings a second to each. (The lab's\n",
    "brokers check for expired segments every 10 seconds instead of every 5 minutes.)"
   ]
  },
  {
   "cell_type": "code",
   "id": "c03-c3a2b5b0",
   "metadata": {},
   "source": [
    "from confluent_kafka import Consumer, TopicPartition\n",
    "from confluent_kafka.admin import AdminClient\n",
    "\n",
    "SMALL, DEFAULT = \"m04l02-telemetry\", \"m04l02-telemetry-default\"\n",
    "admin = AdminClient({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP})\n",
    "lab.recreate_topic(admin, SMALL, partitions=1, config={\"retention.ms\": \"60000\", \"segment.ms\": \"20000\"})\n",
    "lab.recreate_topic(admin, DEFAULT, partitions=1, config={\"retention.ms\": \"60000\"})\n",
    "admin.delete_consumer_groups([\"m04l02-archive\"])\n",
    "writers = [lab.start_script(\"module-04/steady_writer.py\", \"--topic\", t, \"--rate\", \"5\", name=f\"{t}-writer\")\n",
    "           for t in (SMALL, DEFAULT)]"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c04-0f2908fd",
   "metadata": {},
   "source": [
    "An archive job reads the first 20 readings of the first topic, commits its position and stops\n",
    "for \"maintenance\". It comes back in the break-it step."
   ]
  },
  {
   "cell_type": "code",
   "id": "c05-a8b6114e",
   "metadata": {},
   "source": [
    "archive = Consumer({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP, \"group.id\": \"m04l02-archive\",\n",
    "                    \"auto.offset.reset\": \"earliest\", \"enable.auto.commit\": False})\n",
    "archive.subscribe([SMALL])\n",
    "archived = 0\n",
    "while archived < 20:\n",
    "    msg = archive.poll(5)\n",
    "    if msg is not None and not msg.error():\n",
    "        archived += 1\n",
    "archive.commit(message=msg, asynchronous=False)\n",
    "archive.close()\n",
    "print(\"archive job stopped after offset\", msg.offset())"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c06-5a26e5fd",
   "metadata": {},
   "source": [
    "## Step 2: Watch segments come and go\n",
    "`segments(topic)` lists the partition's segment files on its leader: each file is named after the\n",
    "first offset it holds. `oldest(topic)` reads the first record still in the partition and returns\n",
    "its age in seconds."
   ]
  },
  {
   "cell_type": "code",
   "id": "c07-6489f6a7",
   "metadata": {},
   "source": [
    "def segments(topic):\n",
    "    \"\"\"(first offset, size in bytes) of each segment file of partition 0, on the leader's disk.\"\"\"\n",
    "    leader = admin.list_topics(topic, timeout=10).topics[topic].partitions[0].leader\n",
    "    listing = lab.compose(\"exec\", \"-T\", f\"kafka-{leader}\", \"ls\", \"-l\", f\"/var/lib/kafka/data/{topic}-0\", echo=False)\n",
    "    return [(int(line.split()[-1].split(\".\")[0]), int(line.split()[4]))\n",
    "            for line in listing.splitlines() if line.endswith(\".log\")]\n",
    "\n",
    "peek = Consumer({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP, \"group.id\": \"m04l02-peek\",\n",
    "                 \"enable.auto.commit\": False})\n",
    "\n",
    "def oldest(topic):\n",
    "    \"\"\"(log start offset, end offset, age in seconds of the oldest record still readable).\"\"\"\n",
    "    start, end = peek.get_watermark_offsets(TopicPartition(topic, 0), timeout=10)\n",
    "    peek.assign([TopicPartition(topic, 0, start)])\n",
    "    msg = peek.poll(10)\n",
    "    return start, end, round(time.time() - msg.timestamp()[1] / 1000)\n",
    "\n",
    "def watch(seconds, every=10):\n",
    "    rows = []\n",
    "    for _ in range(seconds // every):\n",
    "        time.sleep(every)\n",
    "        rows.append((datetime.now().strftime(\"%H:%M:%S\"), [first for first, _ in segments(SMALL)],\n",
    "                     oldest(SMALL)[2], len(segments(DEFAULT)), oldest(DEFAULT)[2]))\n",
    "    lab.table(rows, [\"time\", \"segments (first offsets)\", \"oldest s\", \"default: segments\", \"default: oldest s\"])\n",
    "\n",
    "watch(120)"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c08-a344d2b5",
   "metadata": {},
   "source": [
    "On the first topic a new segment appears every 20 seconds, and the oldest segment disappears once\n",
    "its newest reading is older than 60 seconds. So the oldest reading's age climbs to about 80–90\n",
    "seconds, then drops. On the second topic there is one segment, and nothing is ever deleted: the\n",
    "oldest reading just gets older.\n",
    "\n",
    "## Step 3: Change it: give the default topic smaller segments\n",
    "Set `segment.ms` on the second topic too, and watch for another 90 seconds."
   ]
  },
  {
   "cell_type": "code",
   "id": "c09-0e736c7b",
   "metadata": {},
   "source": [
    "from confluent_kafka.admin import ConfigResource, ConfigEntry, AlterConfigOpType\n",
    "\n",
    "fix = ConfigResource(\"topic\", DEFAULT, incremental_configs=[\n",
    "    ConfigEntry(\"segment.ms\", \"20000\", incremental_operation=AlterConfigOpType.SET)])\n",
    "admin.incremental_alter_configs([fix])[fix].result()\n",
    "watch(90)"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c10-69f91834",
   "metadata": {},
   "source": [
    "The big segment closed at the next write, but it was only deleted about 60 seconds later: retention\n",
    "counts from the **newest** record in a segment, and that one had just been written.\n",
    "\n",
    "## Break it: the archive job comes back\n",
    "The archive job committed offset 20 about four minutes ago. Predict first: where does it continue?"
   ]
  },
  {
   "cell_type": "code",
   "id": "c11-a061ce73",
   "metadata": {},
   "source": [
    "archive = Consumer({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP, \"group.id\": \"m04l02-archive\",\n",
    "                    \"auto.offset.reset\": \"earliest\", \"enable.auto.commit\": False})\n",
    "committed = archive.committed([TopicPartition(SMALL, 0)], timeout=10)[0].offset\n",
    "archive.subscribe([SMALL])\n",
    "msg = None\n",
    "while msg is None or msg.error():\n",
    "    msg = archive.poll(5)\n",
    "archive.close()\n",
    "start, end, age = oldest(SMALL)\n",
    "print(f\"committed offset: {committed}   log start offset now: {start}   first reading it got: offset {msg.offset()}\")\n",
    "print(f\"readings the archive job never saw: {msg.offset() - committed}\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c12-5486c12a",
   "metadata": {},
   "source": [
    "Its committed offset no longer exists. The consumer got an \"offset out of range\" answer, fell back\n",
    "to `auto.offset.reset` (here `earliest`, the log start offset), and continued from there. With\n",
    "`latest` it would have skipped everything up to the newest reading. Either way, no error reached\n",
    "your code.\n",
    "\n",
    "## Step 4: Delete records on request\n",
    "A privacy request: delete every reading in the first topic that was written before now.\n",
    "`delete_records()` moves the partition's log start offset."
   ]
  },
  {
   "cell_type": "code",
   "id": "c13-ab9b12f5",
   "metadata": {},
   "source": [
    "start, end, _ = oldest(SMALL)\n",
    "result = admin.delete_records([TopicPartition(SMALL, 0, end)])\n",
    "for tp, future in result.items():\n",
    "    print(\"new log start offset:\", future.result().low_watermark)\n",
    "print(\"segment files right after:\", segments(SMALL))\n",
    "time.sleep(15)\n",
    "print(\"segment files 15 s later:\", segments(SMALL))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c14-154c6ed6",
   "metadata": {},
   "source": [
    "Consumers can't read below the new log start offset at once. The segment files that only held\n",
    "deleted readings go at the next retention check. A segment that also holds newer readings stays\n",
    "until retention removes it.\n",
    "\n",
    "## Clean up\n",
    "Stop the writers and delete both topics and the groups. If anything gets stuck,\n",
    "`docker compose down -v` in the `code/` folder resets the whole lab."
   ]
  },
  {
   "cell_type": "code",
   "id": "c15-dfc52c64",
   "metadata": {},
   "source": [
    "for w in writers:\n",
    "    lab.stop_script(w)\n",
    "peek.close()\n",
    "lab.delete_topics(admin, [SMALL, DEFAULT])\n",
    "admin.delete_consumer_groups([\"m04l02-archive\", \"m04l02-peek\"])\n",
    "print(\"cleaned up\")"
   ],
   "execution_count": null,
   "outputs": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
