{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c00-26736d04",
   "metadata": {},
   "source": [
    "# Brokers, controllers and partitions — practical\n",
    "\n",
    "You look at the lab cluster the way a client and an operator see it: its brokers, its\n",
    "controller quorum, and how a 6-partition topic is spread over three brokers. Then you stop a\n",
    "broker and see which clients can still connect.\n",
    "\n",
    "**Lab services:** kafka-1, kafka-2, kafka-3 · **Lesson:** `modules/01-first-events/02-brokers-controllers-and-partitions.md`"
   ]
  },
  {
   "cell_type": "code",
   "id": "c01-20ed57cc",
   "metadata": {},
   "source": [
    "# Setup: find the lab's shared helpers and check the lab is running\n",
    "import sys, pathlib\n",
    "lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / \"common\").is_dir())\n",
    "sys.path.insert(0, str(lab_root))\n",
    "\n",
    "from common import settings, lab\n",
    "\n",
    "lab.require_kafka()\n",
    "lab.print_versions()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c02-aeafd164",
   "metadata": {},
   "source": [
    "## Step 1: Ask the cluster who is in it\n",
    "`describe_cluster` returns the cluster id and the brokers. Each broker is listed with the\n",
    "address it **advertises** to clients: the one your Mac can reach."
   ]
  },
  {
   "cell_type": "code",
   "id": "c03-ea20033e",
   "metadata": {},
   "source": [
    "from confluent_kafka.admin import AdminClient\n",
    "\n",
    "admin = AdminClient({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP})\n",
    "cluster = admin.describe_cluster(request_timeout=10).result()\n",
    "\n",
    "print(\"cluster id:\", cluster.cluster_id)\n",
    "lab.table([(n.id, f\"{n.host}:{n.port}\") for n in sorted(cluster.nodes, key=lambda n: n.id)],\n",
    "          [\"broker\", \"advertised address\"])"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c04-34e12269",
   "metadata": {},
   "source": [
    "You should see cluster id `ZpMZEQqGS768XpROns7_Ug` and brokers 1, 2 and 3 at `127.0.0.1:9092`,\n",
    "`:9093` and `:9094`.\n",
    "\n",
    "## Step 2: Find the active controller\n",
    "The controllers keep the cluster's metadata in a replicated log of their own. One of them is\n",
    "the active controller (the quorum's leader). The KRaft tool shows it, and how far each\n",
    "controller is behind."
   ]
  },
  {
   "cell_type": "code",
   "id": "c05-72620da0",
   "metadata": {},
   "source": [
    "print(lab.kafka_cli(\"kafka-metadata-quorum.sh\", \"--bootstrap-server\", \"localhost:19092\",\n",
    "                    \"describe\", \"--replication\"))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c06-564e6c29",
   "metadata": {},
   "source": [
    "You should see three rows, one `Leader` and two `Follower`s, with `Lag` 0 or close to it.\n",
    "Which node leads varies between runs.\n",
    "\n",
    "## Step 3: Create a topic with 6 partitions and 3 replicas\n",
    "Each partition gets one **leader** and two **followers** on different brokers. `isr` lists the\n",
    "replicas that are fully caught up."
   ]
  },
  {
   "cell_type": "code",
   "id": "c07-a543a86a",
   "metadata": {},
   "source": [
    "from confluent_kafka import TopicCollection\n",
    "from confluent_kafka.admin import NewTopic\n",
    "\n",
    "TOPIC = \"m01l02-sessions\"\n",
    "lab.delete_topics(admin, [TOPIC])\n",
    "admin.create_topics([NewTopic(TOPIC, num_partitions=6, replication_factor=3)])[TOPIC].result()\n",
    "\n",
    "def show_partitions():\n",
    "    desc = admin.describe_topics(TopicCollection([TOPIC]))[TOPIC].result()\n",
    "    lab.table([(p.id, p.leader.id if p.leader else \"none\",\n",
    "                [r.id for r in p.replicas], [r.id for r in p.isr]) for p in desc.partitions],\n",
    "              [\"partition\", \"leader\", \"replicas\", \"isr\"])\n",
    "    return desc\n",
    "\n",
    "desc = show_partitions()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c08-dc26a525",
   "metadata": {},
   "source": [
    "You should see partitions 0–5, each with three replicas and all three in `isr`. The exact\n",
    "brokers vary between runs.\n",
    "\n",
    "## Step 4: Count the leaders per broker\n",
    "Kafka spreads the leaders so each broker does a fair share of the reading and writing."
   ]
  },
  {
   "cell_type": "code",
   "id": "c09-50038f0a",
   "metadata": {},
   "source": [
    "from collections import Counter\n",
    "\n",
    "leaders = Counter(p.leader.id for p in desc.partitions)\n",
    "print({f\"broker {b}\": n for b, n in sorted(leaders.items())})"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c10-f80a375c",
   "metadata": {},
   "source": [
    "You should see `{'broker 1': 2, 'broker 2': 2, 'broker 3': 2}`.\n",
    "\n",
    "## Break it: a client that only knows one broker\n",
    "Predict first: you stop **kafka-2**. Client A was configured with only `127.0.0.1:9093`\n",
    "(kafka-2). Client B has all three addresses. Which one can still start?"
   ]
  },
  {
   "cell_type": "code",
   "id": "c11-02e7bbd1",
   "metadata": {},
   "source": [
    "import logging\n",
    "silent = logging.getLogger(\"silent\"); silent.addHandler(logging.NullHandler()); silent.propagate = False\n",
    "\n",
    "lab.compose(\"stop\", \"kafka-2\")\n",
    "\n",
    "for name, bootstrap in [(\"A\", \"127.0.0.1:9093\"), (\"B\", settings.KAFKA_BOOTSTRAP)]:\n",
    "    client = AdminClient({\"bootstrap.servers\": bootstrap, \"logger\": silent})\n",
    "    try:\n",
    "        brokers = sorted(client.list_topics(timeout=8).brokers)\n",
    "        print(f\"client {name}: connected, sees brokers {brokers}\")\n",
    "    except Exception as e:\n",
    "        print(f\"client {name}: failed to start: {e.args[0].str()}\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c12-83b0939f",
   "metadata": {},
   "source": [
    "Client A fails: its only bootstrap address is down, so it never learns about the others.\n",
    "Client B connects through kafka-1 or kafka-3 and sees brokers `[1, 3]`.\n",
    "\n",
    "Now look at the topic again. The partitions kafka-2 led have new leaders, and broker 2 has\n",
    "dropped out of every `isr` list."
   ]
  },
  {
   "cell_type": "code",
   "id": "c13-2353c07e",
   "metadata": {},
   "source": [
    "desc = show_partitions()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c14-8e29883c",
   "metadata": {},
   "source": [
    "## Clean up\n",
    "Start kafka-2 again and wait until it's healthy, then delete the topic. If anything gets stuck,\n",
    "`docker compose down -v` in the `code/` folder resets the whole lab."
   ]
  },
  {
   "cell_type": "code",
   "id": "c15-49e05301",
   "metadata": {},
   "source": [
    "lab.compose(\"up\", \"-d\", \"--wait\", \"kafka-2\")\n",
    "lab.delete_topics(admin, [TOPIC])\n",
    "print(\"cleaned up\")"
   ],
   "execution_count": null,
   "outputs": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
