{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c00-b24a89ea",
   "metadata": {},
   "source": [
    "# Day-2 operations: change without downtime — practical\n",
    "\n",
    "Brightlane's topics live in `module-08/topics.yaml`. You apply the file to the cluster, catch a\n",
    "change someone made by hand, add partitions for the growth plan and see which chargers move.\n",
    "Then you restart all three brokers while a writer runs, read the cluster's feature versions,\n",
    "and move replicas off kafka-1 with a throttle.\n",
    "\n",
    "**Lab services:** kafka-1..3 (`docker compose up -d --wait`)\n",
    "· **Lesson:** `modules/08-security-and-operations/04-day-2-operations.md`"
   ]
  },
  {
   "cell_type": "code",
   "id": "c01-4747fd45",
   "metadata": {},
   "source": [
    "# Setup: find the lab's shared helpers\n",
    "import sys, pathlib\n",
    "lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / \"common\").is_dir())\n",
    "sys.path.insert(0, str(lab_root))\n",
    "\n",
    "import json, logging, os, re, time\n",
    "import yaml\n",
    "from confluent_kafka import Producer\n",
    "from confluent_kafka.admin import (AlterConfigOpType, ConfigEntry, ConfigResource, ConfigSource,\n",
    "                                   NewPartitions, NewTopic, ResourceType)\n",
    "from common import lab, settings\n",
    "from common.scenario import Brightlane, to_json\n",
    "\n",
    "quiet = logging.getLogger(\"quiet\")            # brokers restart in this lesson: mute the clients' reconnect log\n",
    "quiet.addHandler(logging.NullHandler())\n",
    "quiet.propagate = False\n",
    "\n",
    "lab.require_kafka()\n",
    "admin = lab.admin_client(logger=quiet)\n",
    "desired = yaml.safe_load((lab_root / \"module-08\" / \"topics.yaml\").read_text())[\"topics\"]\n",
    "ARCHIVE = \"charging.telemetry-archive\"\n",
    "lab.delete_topics(admin, [*desired, ARCHIVE])          # start from a cluster without them\n",
    "print(list(desired))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c02-0c5711b1",
   "metadata": {},
   "source": [
    "## Step 1: Plan, then apply\n",
    "`plan()` compares the file with the cluster and lists the changes, like a dry run. Nothing\n",
    "changes until `apply()` runs."
   ]
  },
  {
   "cell_type": "code",
   "id": "c03-08287975",
   "metadata": {},
   "source": [
    "def plan(desired):\n",
    "    \"\"\"The changes that make the cluster match `desired`, as (change, topic, setting, detail) rows.\"\"\"\n",
    "    current = admin.list_topics(timeout=10).topics\n",
    "    existing = [name for name in desired if name in current]\n",
    "    configs = {}\n",
    "    if existing:\n",
    "        futures = admin.describe_configs([ConfigResource(ResourceType.TOPIC, name) for name in existing])\n",
    "        configs = {resource.name: future.result() for resource, future in futures.items()}\n",
    "    changes = []\n",
    "    for name, spec in desired.items():\n",
    "        if name not in current:\n",
    "            changes.append((\"create\", name, \"\", f\"{spec['partitions']} partitions, RF {spec['replication_factor']}\"))\n",
    "            continue\n",
    "        have = len(current[name].partitions)\n",
    "        if spec[\"partitions\"] > have:\n",
    "            changes.append((\"add partitions\", name, \"\", f\"{have} -> {spec['partitions']}\"))\n",
    "        elif spec[\"partitions\"] < have:\n",
    "            changes.append((\"can't apply\", name, \"\", f\"{have} -> {spec['partitions']}: partitions can't be removed\"))\n",
    "        if spec[\"replication_factor\"] != len(current[name].partitions[0].replicas):\n",
    "            changes.append((\"can't apply\", name, \"\", \"replication factor: needs a reassignment (step 5)\"))\n",
    "        wanted = {key: str(value) for key, value in spec.get(\"config\", {}).items()}\n",
    "        for key, value in wanted.items():\n",
    "            if configs[name][key].value != value:\n",
    "                changes.append((\"set\", name, key, f\"{configs[name][key].value} -> {value}\"))\n",
    "        for key, entry in configs[name].items():                 # set on the topic, but not in the file\n",
    "            if ConfigSource(entry.source) == ConfigSource.DYNAMIC_TOPIC_CONFIG and key not in wanted:\n",
    "                changes.append((\"remove\", name, key, f\"{entry.value} -> cluster default\"))\n",
    "    return changes\n",
    "\n",
    "changes = plan(desired)\n",
    "lab.table(changes, [\"change\", \"topic\", \"setting\", \"detail\"])"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c04-a977ab9c",
   "metadata": {},
   "source": [
    "`apply()` makes each change with the AdminClient: create topics, add partitions, set or remove\n",
    "a topic's settings."
   ]
  },
  {
   "cell_type": "code",
   "id": "c05-a96ed8c6",
   "metadata": {},
   "source": [
    "def apply(desired, changes):\n",
    "    for change, name, key, detail in changes:\n",
    "        spec = desired[name]\n",
    "        if change == \"create\":\n",
    "            topic = NewTopic(name, spec[\"partitions\"], spec[\"replication_factor\"],\n",
    "                             config={k: str(v) for k, v in spec.get(\"config\", {}).items()})\n",
    "            admin.create_topics([topic])[name].result()\n",
    "        elif change == \"add partitions\":\n",
    "            admin.create_partitions([NewPartitions(name, spec[\"partitions\"])])[name].result()\n",
    "        elif change in (\"set\", \"remove\"):\n",
    "            operation = AlterConfigOpType.SET if change == \"set\" else AlterConfigOpType.DELETE\n",
    "            entry = ConfigEntry(key, str(spec[\"config\"].get(key, \"\")), incremental_operation=operation)\n",
    "            resource = ConfigResource(ResourceType.TOPIC, name, incremental_configs=[entry])\n",
    "            admin.incremental_alter_configs([resource])[resource].result()\n",
    "        else:\n",
    "            print(f\"skipped  {name}: {detail}\")\n",
    "            continue\n",
    "        print(f\"applied  {change} {name} {key}\")\n",
    "\n",
    "apply(desired, changes)\n",
    "print(\"plan now:\", plan(desired))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c06-a1c50213",
   "metadata": {},
   "source": [
    "## Step 2: A change by hand, and the growth plan\n",
    "During a disk alert last night, the on-call engineer shortened `charging.sessions`' retention to\n",
    "one hour and raised its maximum record size, by hand. Today the growth plan needs telemetry to go\n",
    "from 6 to 12 partitions; in real life that's an edit to `topics.yaml` in a pull request."
   ]
  },
  {
   "cell_type": "code",
   "id": "c07-12a90f56",
   "metadata": {},
   "source": [
    "lab.kafka_cli(\"kafka-configs.sh\", \"--bootstrap-server\", \"localhost:19092\", \"--alter\", \"--entity-type\", \"topics\",\n",
    "              \"--entity-name\", \"charging.sessions\", \"--add-config\", \"retention.ms=3600000,max.message.bytes=10485760\")\n",
    "desired[\"charging.telemetry\"][\"partitions\"] = 12\n",
    "changes = plan(desired)\n",
    "lab.table(changes, [\"change\", \"topic\", \"setting\", \"detail\"])"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c08-15cccb9f",
   "metadata": {},
   "source": [
    "Before applying, write one reading per charger and note the partition each lands in. After the\n",
    "change, do it again."
   ]
  },
  {
   "cell_type": "code",
   "id": "c09-e2fdf965",
   "metadata": {},
   "source": [
    "chargers = [c[\"charger_id\"] for c in Brightlane().chargers]\n",
    "\n",
    "def partition_per_charger():\n",
    "    where = {}\n",
    "    producer = Producer({**settings.PRODUCER_DEFAULTS, \"logger\": quiet})\n",
    "    for charger in chargers:\n",
    "        producer.produce(\"charging.telemetry\", key=charger, value=to_json({\"charger_id\": charger, \"power_kw\": 11.0}),\n",
    "                         on_delivery=lambda err, msg: where.__setitem__(msg.key().decode(), msg.partition()))\n",
    "    producer.flush(15)\n",
    "    return where\n",
    "\n",
    "before = partition_per_charger()\n",
    "apply(desired, changes)\n",
    "time.sleep(2)\n",
    "after = partition_per_charger()\n",
    "moved = [c for c in chargers if before[c] != after[c]]\n",
    "print(f\"chargers in a different partition now: {len(moved)} of {len(chargers)}\")\n",
    "print(\"for example:\", \", \".join(f\"{c} {before[c]} -> {after[c]}\" for c in moved[:4]))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c10-bac08b19",
   "metadata": {},
   "source": [
    "## Step 3: A rolling restart with a writer running\n",
    "A patch upgrade means restarting every broker. The steady writer from lesson 4.1 writes 50\n",
    "readings a second to `charging.telemetry` while the brokers restart one by one. Before each\n",
    "restart the notebook waits until every replica is back in sync."
   ]
  },
  {
   "cell_type": "code",
   "id": "c11-1ec18d3a",
   "metadata": {},
   "source": [
    "def under_replicated():\n",
    "    \"\"\"Partitions with fewer in-sync replicas than replicas, from the cluster's metadata.\"\"\"\n",
    "    return sum(len(p.isrs) < len(p.replicas)\n",
    "               for t in admin.list_topics(timeout=10).topics.values() for p in t.partitions.values())\n",
    "\n",
    "def writer_report(writer):\n",
    "    lines = writer.lines()\n",
    "    slowest = max(int(m.group(1)) for line in lines if (m := re.search(r\"slowest=(\\d+)ms\", line)))\n",
    "    print(lines[-1].split(\" \", 1)[1])\n",
    "    print(f\"slowest delivery: {slowest} ms; seconds with nothing delivered: \"\n",
    "          f\"{sum(' delivered=0 ' in line for line in lines[1:-1])}\")\n",
    "\n",
    "writer = lab.start_script(\"module-04/steady_writer.py\", \"--topic\", \"charging.telemetry\", \"--rate\", \"50\",\n",
    "                          name=\"m08l04-writer\")\n",
    "time.sleep(5)\n",
    "for node in settings.KAFKA_NODES:\n",
    "    lab.wait_until(lambda: under_replicated() == 0, timeout=120, interval=1, what=\"all replicas in sync\")\n",
    "    started = time.monotonic()\n",
    "    lab.compose(\"restart\", node, echo=False)\n",
    "    restarted = time.monotonic() - started\n",
    "    lab.wait_until(lambda: under_replicated() == 0, timeout=120, interval=1, what=\"all replicas in sync\")\n",
    "    print(f\"{node}: restarted in {restarted:.0f} s, all replicas in sync after {time.monotonic() - started:.0f} s\")\n",
    "time.sleep(3)\n",
    "lab.stop_script(writer)\n",
    "writer_report(writer)"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c12-a1440b8f",
   "metadata": {},
   "source": [
    "## Step 4: Feature versions\n",
    "New broker software can come with new on-disk and protocol formats. They stay off until you\n",
    "raise the cluster's feature versions, after every broker runs the new software."
   ]
  },
  {
   "cell_type": "code",
   "id": "c13-150fcb17",
   "metadata": {},
   "source": [
    "print(lab.kafka_cli(\"kafka-features.sh\", \"--bootstrap-server\", \"localhost:19092\", \"describe\", echo=False))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c14-03251b2d",
   "metadata": {},
   "source": [
    "## Step 5: Move replicas off kafka-1, with a throttle\n",
    "kafka-1's disk reports errors and will be replaced. The archive topic has two replicas per\n",
    "partition and 60 MB of data. Move its replicas to kafka-2 and kafka-3, copying at most 2 MB/s,\n",
    "so the copy doesn't crowd out billing's traffic."
   ]
  },
  {
   "cell_type": "code",
   "id": "c15-b842fd02",
   "metadata": {},
   "source": [
    "admin.create_topics([NewTopic(ARCHIVE, num_partitions=3, replication_factor=-1,\n",
    "                              replica_assignment=[[1, 2], [2, 3], [3, 1]])])[ARCHIVE].result()\n",
    "time.sleep(2)\n",
    "producer = Producer({**settings.PRODUCER_DEFAULTS, \"linger.ms\": 50, \"logger\": quiet})\n",
    "for i in range(6000):                                   # 6,000 records of 10 KB: 60 MB, 20 MB per partition\n",
    "    producer.produce(ARCHIVE, key=f\"day-{i}\", value=os.urandom(10_000), partition=i % 3)\n",
    "    producer.poll(0)\n",
    "producer.flush(60)\n",
    "\n",
    "def replicas():\n",
    "    return {p.id: p.replicas for p in admin.list_topics(ARCHIVE, timeout=10).topics[ARCHIVE].partitions.values()}\n",
    "\n",
    "print(\"replicas now:\", replicas())"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "id": "c16-99b900d2",
   "metadata": {},
   "source": [
    "move = {\"version\": 1, \"partitions\": [{\"topic\": ARCHIVE, \"partition\": 0, \"replicas\": [2, 3]},\n",
    "                                     {\"topic\": ARCHIVE, \"partition\": 1, \"replicas\": [2, 3]},\n",
    "                                     {\"topic\": ARCHIVE, \"partition\": 2, \"replicas\": [3, 2]}]}\n",
    "lab.put_file(\"kafka-1\", \"/tmp/move.json\", json.dumps(move).encode())\n",
    "reassign = [\"kafka-reassign-partitions.sh\", \"--bootstrap-server\", \"localhost:19092\",\n",
    "            \"--reassignment-json-file\", \"/tmp/move.json\"]\n",
    "out = lab.kafka_cli(*reassign, \"--execute\", \"--throttle\", \"2000000\", echo=False)\n",
    "print(out.splitlines()[-1])\n",
    "started = time.monotonic()\n",
    "while time.monotonic() - started < 120:\n",
    "    time.sleep(3)\n",
    "    out = lab.kafka_cli(*reassign, \"--verify\", echo=False)\n",
    "    busy = \"still in progress\" in out\n",
    "    print(f\"+{time.monotonic() - started:3.0f} s  replicas: {replicas()}  {'copying' if busy else 'done'}\")\n",
    "    if not busy:\n",
    "        break\n",
    "print(\"\\n\".join(line for line in out.splitlines() if \"hrottle\" in line))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c17-51ce264c",
   "metadata": {},
   "source": [
    "## Break it: two brokers at once\n",
    "The same restart, but kafka-1 and kafka-2 restart together, as when someone restarts \"all the\n",
    "brokers on that host\" or doesn't wait for the cluster to catch up."
   ]
  },
  {
   "cell_type": "code",
   "id": "c18-732d68e6",
   "metadata": {},
   "source": [
    "writer = lab.start_script(\"module-04/steady_writer.py\", \"--topic\", \"charging.telemetry\", \"--rate\", \"50\",\n",
    "                          name=\"m08l04-writer-2\")\n",
    "time.sleep(5)\n",
    "started = time.monotonic()\n",
    "lab.compose(\"restart\", \"kafka-1\", \"kafka-2\", echo=False)\n",
    "lab.wait_until(lambda: under_replicated() == 0, timeout=180, interval=1, what=\"all replicas in sync\")\n",
    "print(f\"both back and in sync after {time.monotonic() - started:.0f} s\")\n",
    "time.sleep(3)\n",
    "lab.stop_script(writer)\n",
    "writer_report(writer)"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c19-dd2d7238",
   "metadata": {},
   "source": [
    "## Clean up"
   ]
  },
  {
   "cell_type": "code",
   "id": "c20-5bbe5d59",
   "metadata": {},
   "source": [
    "lab.delete_topics(admin, [*desired, ARCHIVE])\n",
    "print(\"cleaned up\")"
   ],
   "execution_count": null,
   "outputs": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
