{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c00-236b1b27",
   "metadata": {},
   "source": [
    "# Leaders, followers and failover — practical\n",
    "\n",
    "A writer sends 50 meter readings a second while you stop a leader cleanly, crash another one,\n",
    "move the leaders back, and finally take away the controllers' majority. At the end you count\n",
    "whether a single reading was lost or written twice.\n",
    "\n",
    "**Lab services:** kafka-1, kafka-2, kafka-3 · **Script:** `module-04/steady_writer.py`\n",
    "· **Lesson:** `modules/04-inside-the-cluster/01-leaders-followers-and-failover.md`"
   ]
  },
  {
   "cell_type": "code",
   "id": "c01-f1daf67f",
   "metadata": {},
   "source": [
    "# Setup: find the lab's shared helpers and check the lab is running\n",
    "import sys, pathlib\n",
    "lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / \"common\").is_dir())\n",
    "sys.path.insert(0, str(lab_root))\n",
    "\n",
    "import json, re, time\n",
    "from datetime import datetime\n",
    "from common import settings, lab\n",
    "\n",
    "lab.require_kafka()\n",
    "lab.print_versions()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c02-9ed24ada",
   "metadata": {},
   "source": [
    "## Step 1: A steady writer, and who leads what\n",
    "The topic has 3 partitions with 3 replicas each. The writer runs in the background with the safe\n",
    "settings (`acks=all`, idempotence on) and logs one line per second."
   ]
  },
  {
   "cell_type": "code",
   "id": "c03-f6be34ba",
   "metadata": {},
   "source": [
    "from confluent_kafka import TopicPartition, ElectionType\n",
    "from confluent_kafka.admin import AdminClient\n",
    "\n",
    "TOPIC = \"m04l01-telemetry\"\n",
    "admin = AdminClient({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP})\n",
    "lab.recreate_topic(admin, TOPIC, partitions=3)\n",
    "writer = lab.start_script(\"module-04/steady_writer.py\", \"--topic\", TOPIC, \"--rate\", \"50\", name=\"m04l01-writer\")\n",
    "\n",
    "def layout():\n",
    "    \"\"\"Per partition: the leader, the preferred leader (first replica) and the in-sync replicas.\"\"\"\n",
    "    partitions = admin.list_topics(TOPIC, timeout=10).topics[TOPIC].partitions   # any broker answers\n",
    "    return {p.id: {\"leader\": p.leader, \"preferred\": p.replicas[0], \"isr\": sorted(p.isrs)}\n",
    "            for p in sorted(partitions.values(), key=lambda p: p.id)}\n",
    "\n",
    "time.sleep(5)\n",
    "for partition, info in layout().items():\n",
    "    print(partition, info)"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c04-c507b33d",
   "metadata": {},
   "source": [
    "Each partition has a different leader, and right now each leader is also its preferred leader.\n",
    "\n",
    "`writer_report(since)` reads the writer's log from a moment on and prints the seconds that weren't\n",
    "normal: fewer than 40 deliveries, or a delivery slower than 100 ms."
   ]
  },
  {
   "cell_type": "code",
   "id": "c05-27b20b77",
   "metadata": {},
   "source": [
    "LINE = re.compile(r\"^(\\d\\d:\\d\\d:\\d\\d) sent=(\\d+) delivered=(\\d+) waiting=(\\d+) slowest=(\\d+)ms(.*)\")\n",
    "\n",
    "def writer_report(since):\n",
    "    lines = [m for m in (LINE.match(l) for l in writer.lines()) if m and m.group(1) >= since]\n",
    "    odd = [m.group(0) for m in lines if int(m.group(3)) < 40 or int(m.group(5)) > 100]\n",
    "    for line in odd:\n",
    "        print(\"  \" + line)\n",
    "    stalled = sum(1 for m in lines if int(m.group(3)) == 0)\n",
    "    slowest = max((int(m.group(5)) for m in lines), default=0)\n",
    "    print(f\"seconds with no delivery: {stalled}   slowest delivery: {slowest} ms\")\n",
    "\n",
    "now = lambda: datetime.now().strftime(\"%H:%M:%S\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c06-1ed202d0",
   "metadata": {},
   "source": [
    "## Step 2: Stop a leader cleanly\n",
    "`docker compose stop` sends the broker a polite shutdown signal, like a planned restart.\n",
    "Predict first: how long do writes to partition 0 pause?"
   ]
  },
  {
   "cell_type": "code",
   "id": "c07-d4c0b224",
   "metadata": {},
   "source": [
    "leader = layout()[0][\"leader\"]\n",
    "since = now()\n",
    "lab.compose(\"stop\", f\"kafka-{leader}\")\n",
    "for partition, info in layout().items():\n",
    "    print(partition, info)\n",
    "time.sleep(5)\n",
    "writer_report(since)"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c08-7f3c6671",
   "metadata": {},
   "source": [
    "Every partition already has a new leader and the stopped broker has left the ISRs. The writer\n",
    "barely noticed: a few hundred milliseconds at most.\n",
    "\n",
    "## Step 3: Bring it back, then move the leaders back\n",
    "The broker restarts and catches up. Is it the leader of anything?"
   ]
  },
  {
   "cell_type": "code",
   "id": "c09-c6551ccc",
   "metadata": {},
   "source": [
    "lab.compose(\"up\", \"-d\", \"--wait\", f\"kafka-{leader}\")\n",
    "for partition, info in layout().items():\n",
    "    print(partition, info)"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c10-fce9e2fb",
   "metadata": {},
   "source": [
    "It's back in every ISR, but (unless the controller's 5-minute balancing check happened to run)\n",
    "it leads nothing: its partitions' leader is not the preferred one. A preferred-leader election\n",
    "moves leadership back:"
   ]
  },
  {
   "cell_type": "code",
   "id": "c11-fa8ca235",
   "metadata": {},
   "source": [
    "result = admin.elect_leaders(ElectionType.PREFERRED, [TopicPartition(TOPIC, p) for p in range(3)]).result()\n",
    "print({tp.partition: (err.str() if err else \"ok\") for tp, err in result.items()})\n",
    "for partition, info in layout().items():\n",
    "    print(partition, info)"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c12-2944bf6e",
   "metadata": {},
   "source": [
    "A partition whose leader was already the preferred one reports `Election not needed`.\n",
    "\n",
    "## Step 4: Crash a leader\n",
    "Now `docker compose kill`: the broker process dies at once, like a power cut. Predict first: does\n",
    "the pause look like step 2?"
   ]
  },
  {
   "cell_type": "code",
   "id": "c13-6b42b6f3",
   "metadata": {},
   "source": [
    "leader = layout()[0][\"leader\"]\n",
    "since, crashed = now(), time.monotonic()\n",
    "lab.compose(\"kill\", f\"kafka-{leader}\")\n",
    "last = None\n",
    "while time.monotonic() - crashed < 15:\n",
    "    current = layout()\n",
    "    if current != last:\n",
    "        print(f\"+{time.monotonic() - crashed:4.1f}s\", {p: (i[\"leader\"], i[\"isr\"]) for p, i in current.items()})\n",
    "        last = current\n",
    "    time.sleep(0.5)\n",
    "writer_report(since)"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c14-1dfa4342",
   "metadata": {},
   "source": [
    "For about 9 seconds the cluster still lists the dead broker as a leader and as in sync, and **no**\n",
    "reading is delivered, on any partition. Then the controller fences the broker (it has missed\n",
    "heartbeats for `broker.session.timeout.ms`, 9 s), elects new leaders and shrinks the ISRs.\n",
    "Writes resume, and the readings that waited are delivered.\n",
    "\n",
    "Restart the broker and move the leaders back:"
   ]
  },
  {
   "cell_type": "code",
   "id": "c15-4fb0104a",
   "metadata": {},
   "source": [
    "lab.compose(\"up\", \"-d\", \"--wait\", f\"kafka-{leader}\")\n",
    "time.sleep(3)\n",
    "admin.elect_leaders(ElectionType.PREFERRED, [TopicPartition(TOPIC, p) for p in range(3)]).result()\n",
    "for partition, info in layout().items():\n",
    "    print(partition, info)"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c16-4a078b16",
   "metadata": {},
   "source": [
    "## Break it: lose the controllers' majority\n",
    "Each lab node also runs a controller. Kill two of the three nodes: one controller is left, with\n",
    "no majority. Predict first: what still works?"
   ]
  },
  {
   "cell_type": "code",
   "id": "c17-a361bbe0",
   "metadata": {},
   "source": [
    "since = now()\n",
    "lab.compose(\"kill\", \"kafka-2\", \"kafka-3\")\n",
    "time.sleep(15)\n",
    "from confluent_kafka.admin import NewTopic\n",
    "try:\n",
    "    admin.create_topics([NewTopic(\"m04l01-new-topic\", 3, 3)], request_timeout=10)[\"m04l01-new-topic\"].result()\n",
    "except Exception as e:\n",
    "    print(\"create topic:\", e)\n",
    "writer_report(since)"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c18-acdd213b",
   "metadata": {},
   "source": [
    "Nothing works: no topic can be created, and no reading is delivered, not even to the partitions\n",
    "kafka-1 leads. Restart one node, and the controllers have a majority again (2 of 3):"
   ]
  },
  {
   "cell_type": "code",
   "id": "c19-ba215856",
   "metadata": {},
   "source": [
    "since, restarted = now(), time.monotonic()\n",
    "lab.compose(\"up\", \"-d\", \"--wait\", \"kafka-2\")\n",
    "lab.wait_until(lambda: any(m and m.group(1) > since and int(m.group(3)) > 0\n",
    "                           for m in (LINE.match(l) for l in writer.lines())), timeout=60, what=\"deliveries\")\n",
    "print(f\"deliveries resumed {time.monotonic() - restarted:.0f} s after the restart command\")\n",
    "lab.compose(\"up\", \"-d\", \"--wait\", \"kafka-3\")\n",
    "print(\"all three nodes are back\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c20-2abb370e",
   "metadata": {},
   "source": [
    "## Step 5: Was anything lost or duplicated?\n",
    "Stop the writer. Its last line says how many readings it sent and how many were confirmed. Then\n",
    "read the whole topic and check the sequence numbers."
   ]
  },
  {
   "cell_type": "code",
   "id": "c21-2b91adf6",
   "metadata": {},
   "source": [
    "from confluent_kafka import Consumer\n",
    "\n",
    "lab.stop_script(writer)\n",
    "print(writer.lines(\"stopped\")[-1])\n",
    "consumer = Consumer({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP, \"group.id\": \"m04l01-check\",\n",
    "                     \"enable.auto.commit\": False})\n",
    "ends = {p: consumer.get_watermark_offsets(TopicPartition(TOPIC, p), timeout=10)[1] for p in range(3)}\n",
    "consumer.assign([TopicPartition(TOPIC, p, 0) for p in range(3)])\n",
    "seqs = []\n",
    "while len(seqs) < sum(ends.values()):\n",
    "    msg = consumer.poll(10)\n",
    "    if msg is None:\n",
    "        break\n",
    "    if not msg.error():\n",
    "        seqs.append(json.loads(msg.value())[\"seq\"])\n",
    "consumer.close()\n",
    "print(f\"records in the topic: {len(seqs)}   different readings: {len(set(seqs))}\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c22-4f644c95",
   "metadata": {},
   "source": [
    "You should see the same number three times: sent, confirmed and in the topic, with no duplicates.\n",
    "Two leader changes and a frozen cluster cost time, not data, because every outage was shorter\n",
    "than the writer's delivery timeout (120 s).\n",
    "\n",
    "## Clean up\n",
    "Delete the topic. If anything gets stuck, `docker compose down -v` in the `code/` folder resets\n",
    "the whole lab."
   ]
  },
  {
   "cell_type": "code",
   "id": "c23-ad519cc6",
   "metadata": {},
   "source": [
    "admin = AdminClient({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP})   # a fresh client after the outage\n",
    "print(\"the topic from the break-it step exists now:\", \"m04l01-new-topic\" in admin.list_topics(timeout=10).topics)\n",
    "lab.delete_topics(admin, [TOPIC, \"m04l01-new-topic\"])\n",
    "admin.delete_consumer_groups([\"m04l01-check\"])\n",
    "print(\"cleaned up\")"
   ],
   "execution_count": null,
   "outputs": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
