{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c00-736a5d4a",
   "metadata": {},
   "source": [
    "# Rebalances and the new group protocol — practical\n",
    "\n",
    "Three billing workers share six partitions while a feed writes 40 events a second. You restart\n",
    "them one by one, like a deploy, under four set-ups, and measure how much each one disturbs the\n",
    "group. Then a worker gets stuck and is thrown out of the group.\n",
    "\n",
    "**Lab services:** kafka-1, kafka-2, kafka-3 · **Scripts:** `module-03/session_feed.py`,\n",
    "`module-03/billing_worker.py` · **Lesson:** `modules/03-consumers/03-rebalances-and-the-new-group-protocol.md`"
   ]
  },
  {
   "cell_type": "code",
   "id": "c01-d35b83d4",
   "metadata": {},
   "source": [
    "# Setup: find the lab's shared helpers and check the lab is running\n",
    "import sys, pathlib\n",
    "lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / \"common\").is_dir())\n",
    "sys.path.insert(0, str(lab_root))\n",
    "\n",
    "import re, time\n",
    "from datetime import datetime\n",
    "from common import settings, lab\n",
    "\n",
    "lab.require_kafka()\n",
    "lab.print_versions()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c02-d1162cb4",
   "metadata": {},
   "source": [
    "## Step 1: Start the feed"
   ]
  },
  {
   "cell_type": "code",
   "id": "c03-c2da37be",
   "metadata": {},
   "source": [
    "from confluent_kafka.admin import AdminClient\n",
    "\n",
    "TOPIC = \"m03l03-sessions\"\n",
    "admin = AdminClient({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP})\n",
    "lab.recreate_topic(admin, TOPIC, partitions=6)\n",
    "feed = lab.start_script(\"module-03/session_feed.py\", \"--topic\", TOPIC, \"--rate\", \"40\", name=\"m03l03-feed\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c04-a3e60b4e",
   "metadata": {},
   "source": [
    "## Step 2: A rolling restart, measured\n",
    "`rolling_restart()` starts workers A, B and C, lets them settle, then restarts them one at a time:\n",
    "stop A cleanly (Ctrl+C), start a new A, wait until it has partitions, then B, then C. Every worker\n",
    "logs each `ASSIGNED` and `REVOKED` with a timestamp (its `on_assign` and `on_revoke` callbacks).\n",
    "From those logs the function counts:\n",
    "- **partitions revoked**: how many times a partition was taken from a worker;\n",
    "- **seconds without owner**: summed over all partitions, how long each one had no worker at all."
   ]
  },
  {
   "cell_type": "code",
   "id": "c05-cbb4ee8a",
   "metadata": {},
   "source": [
    "EVENT = re.compile(r\"^(\\d\\d:\\d\\d:\\d\\d\\.\\d{3}) \\w+ (ASSIGNED|REVOKED|LOST) \\[(.*)\\]\")\n",
    "\n",
    "def disruption(scripts, since):\n",
    "    \"\"\"Partitions revoked, and partition-seconds without an owner, after `since` (time of day).\"\"\"\n",
    "    events = []\n",
    "    for s in scripts:\n",
    "        for line in s.lines():\n",
    "            m = EVENT.match(line)\n",
    "            if m:\n",
    "                t = datetime.strptime(m.group(1), \"%H:%M:%S.%f\").time()\n",
    "                events.append((t, m.group(2), [int(p) for p in m.group(3).split(\",\") if p.strip()]))\n",
    "    owners, free_since, revoked, unowned = {p: 0 for p in range(6)}, {}, 0, 0.0\n",
    "    secs = lambda t: t.hour * 3600 + t.minute * 60 + t.second + t.microsecond / 1e6\n",
    "    for t, kind, parts in sorted(events):\n",
    "        for p in parts:\n",
    "            if kind == \"ASSIGNED\":\n",
    "                if owners[p] == 0 and p in free_since and t >= since:\n",
    "                    unowned += secs(t) - max(secs(free_since[p]), secs(since))\n",
    "                owners[p] += 1\n",
    "            else:\n",
    "                owners[p] = max(0, owners[p] - 1)\n",
    "                revoked += t >= since\n",
    "                if owners[p] == 0:\n",
    "                    free_since[p] = t\n",
    "    return revoked, round(unowned, 1)"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c06-37b7d61a",
   "metadata": {},
   "source": [
    "The restart itself:"
   ]
  },
  {
   "cell_type": "code",
   "id": "c07-93893a08",
   "metadata": {},
   "source": [
    "def rolling_restart(label, *flags):\n",
    "    group = f\"m03l03-{label.replace(' ', '-')}\"\n",
    "    start = lambda n, tag: lab.start_script(\"module-03/billing_worker.py\", \"--name\", n, \"--topic\", TOPIC,\n",
    "                                            \"--group\", group, \"--work-ms\", \"5\", *flags, name=f\"{group}-{tag}\")\n",
    "    running = {n: start(n, n) for n in \"ABC\"}\n",
    "    every = list(running.values())\n",
    "    time.sleep(10)                                         # let the group settle\n",
    "    since = datetime.now().time()\n",
    "    for n in \"ABC\":\n",
    "        lab.stop_script(running[n])                        # a clean stop, like a deploy\n",
    "        running[n] = start(n, n + \"2\")\n",
    "        every.append(running[n])\n",
    "        lab.wait_until(lambda: any(re.search(r\"ASSIGNED \\[\\d\", l) for l in running[n].lines()),\n",
    "                       timeout=60, interval=0.2, what=f\"{n} to get partitions\")\n",
    "        time.sleep(2)\n",
    "    for s in running.values():\n",
    "        lab.stop_script(s)\n",
    "    revoked, unowned = disruption(every, since)\n",
    "    return {\"set-up\": label, \"partitions revoked\": revoked, \"seconds without owner\": unowned}\n",
    "\n",
    "results = [rolling_restart(\"classic eager\")]\n",
    "lab.table([list(r.values()) for r in results], list(results[0].keys()))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c08-2136acba",
   "metadata": {},
   "source": [
    "The worker's default is the classic protocol with the eager `range,roundrobin` assignors.\n",
    "Look at the log of worker B, which was never restarted itself, during A's restart:"
   ]
  },
  {
   "cell_type": "code",
   "id": "c09-c29c2f45",
   "metadata": {},
   "source": [
    "b_log = lab.Script(\"B\", None, lab.LOG_DIR / \"m03l03-classic-eager-B.log\")\n",
    "for line in b_log.lines()[:12]:\n",
    "    if \"ASSIGNED\" in line or \"REVOKED\" in line:\n",
    "        print(line)"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c10-2ccea546",
   "metadata": {},
   "source": [
    "B gives up **all** its partitions each time anyone leaves or joins, then gets them back. That's a\n",
    "stop-the-world rebalance.\n",
    "\n",
    "## Step 3: Cooperative rebalancing\n",
    "The same restart with `partition.assignment.strategy=cooperative-sticky`."
   ]
  },
  {
   "cell_type": "code",
   "id": "c11-1528870d",
   "metadata": {},
   "source": [
    "results.append(rolling_restart(\"classic cooperative\", \"--assignor\", \"cooperative-sticky\"))\n",
    "lab.table([list(r.values()) for r in results], list(results[0].keys()))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c12-d91e4c97",
   "metadata": {},
   "source": [
    "## Step 4: The new consumer protocol (KIP-848)\n",
    "The same restart with `group.protocol=consumer`: the broker computes the assignment."
   ]
  },
  {
   "cell_type": "code",
   "id": "c13-584e0e94",
   "metadata": {},
   "source": [
    "results.append(rolling_restart(\"consumer protocol\", \"--protocol\", \"consumer\"))\n",
    "lab.table([list(r.values()) for r in results], list(results[0].keys()))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c14-6da62350",
   "metadata": {},
   "source": [
    "## Step 5: Static membership\n",
    "Back to the classic protocol, but each worker has a fixed `group.instance.id` (its name) and a\n",
    "10-second session timeout. A worker that comes back with the same id within the timeout gets\n",
    "its old partitions back without a rebalance."
   ]
  },
  {
   "cell_type": "code",
   "id": "c15-0683f295",
   "metadata": {},
   "source": [
    "results.append(rolling_restart(\"static members\", \"--static\", \"--session-timeout-ms\", \"10000\"))\n",
    "lab.table([list(r.values()) for r in results], list(results[0].keys()))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c16-dd2987cf",
   "metadata": {},
   "source": [
    "Compare the four rows: the eager protocol revokes the most partitions; cooperative, the consumer\n",
    "protocol and static membership revoke only what has to move. Your numbers will vary a little.\n",
    "\n",
    "## Break it: a stuck worker\n",
    "Predict first: workers A and B share the partitions. After 100 events, A's handler stalls for 15\n",
    "seconds (a slow database call). A's `max.poll.interval.ms` is 8 seconds. What happens to A's\n",
    "partitions, and to A when it wakes up?"
   ]
  },
  {
   "cell_type": "code",
   "id": "c17-f4574e13",
   "metadata": {},
   "source": [
    "group = \"m03l03-stuck\"\n",
    "flags = [\"--topic\", TOPIC, \"--group\", group, \"--work-ms\", \"5\", \"--max-poll-ms\", \"8000\", \"--session-timeout-ms\", \"6000\"]\n",
    "b = lab.start_script(\"module-03/billing_worker.py\", \"--name\", \"B\", *flags, name=\"m03l03-stuck-B\")\n",
    "lab.wait_until(lambda: b.lines(\"ASSIGNED [0\"), timeout=30, what=\"B to get partitions\")\n",
    "a = lab.start_script(\"module-03/billing_worker.py\", \"--name\", \"A\", *flags, \"--stall-at\", \"100\", \"--stall-ms\", \"15000\", name=\"m03l03-stuck-A\")\n",
    "for _ in range(8):\n",
    "    time.sleep(3)\n",
    "    print(time.strftime(\"%H:%M:%S\"), lab.group_members(admin, group))\n",
    "for s in (a, b):\n",
    "    lab.stop_script(s)\n",
    "for s in (a, b):\n",
    "    print(f\"--- {s.name}\")\n",
    "    for line in s.lines():\n",
    "        if any(k in line for k in (\"ASSIGNED\", \"REVOKED\", \"LOST\", \"STALLING\", \"ERROR\")):\n",
    "            print(line)"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c18-4a7187c2",
   "metadata": {},
   "source": [
    "A stops calling `consume()` for 15 seconds. After 8 seconds its client gives up its membership,\n",
    "and B takes all six partitions. When A wakes up, its next `consume()` returns\n",
    "`_MAX_POLL_EXCEEDED`, it learns its partitions were **lost**, and it rejoins. The events A was\n",
    "working on when it stalled were handed to B as well.\n",
    "\n",
    "## Clean up\n",
    "Stop the feed and delete the topic and the groups. If anything gets stuck, `docker compose down -v`\n",
    "in the `code/` folder resets the whole lab."
   ]
  },
  {
   "cell_type": "code",
   "id": "c19-072cc8ba",
   "metadata": {},
   "source": [
    "lab.stop_script(feed)\n",
    "lab.delete_topics(admin, [TOPIC])\n",
    "admin.delete_consumer_groups([\"m03l03-classic-eager\", \"m03l03-classic-cooperative\",\n",
    "                              \"m03l03-consumer-protocol\", \"m03l03-static-members\", \"m03l03-stuck\"])\n",
    "print(\"cleaned up\")"
   ],
   "execution_count": null,
   "outputs": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
