{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c00-b32b6644",
   "metadata": {},
   "source": [
    "# Durable writes: acks, replicas and idempotence — practical\n",
    "\n",
    "You write meter readings with delivery reports, then take a broker away and watch what\n",
    "`acks=all`, `min.insync.replicas`, `acks=1` and `acks=0` each do, including what they don't tell you.\n",
    "\n",
    "**Lab services:** kafka-1, kafka-2, kafka-3 · **Lesson:** `modules/02-producers/02-durable-writes.md`"
   ]
  },
  {
   "cell_type": "code",
   "id": "c01-4d585d33",
   "metadata": {},
   "source": [
    "# Setup: find the lab's shared helpers and check the lab is running\n",
    "import sys, pathlib\n",
    "lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / \"common\").is_dir())\n",
    "sys.path.insert(0, str(lab_root))\n",
    "\n",
    "import json, time\n",
    "from collections import Counter\n",
    "from common import settings, lab\n",
    "from common.scenario import Brightlane, to_json\n",
    "\n",
    "lab.require_kafka()\n",
    "lab.print_versions()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c02-f282353c",
   "metadata": {},
   "source": [
    "## Step 1: Write with acks=all and an idempotent producer\n",
    "One partition, three replicas. The delivery report callback tells you, for every record,\n",
    "whether it was stored and at which offset. `message.timeout.ms` is how long the client keeps\n",
    "trying before it gives up: 5 minutes by default, 10 seconds here so the notebook doesn't wait."
   ]
  },
  {
   "cell_type": "code",
   "id": "c03-7a665d5c",
   "metadata": {},
   "source": [
    "from confluent_kafka import Producer, Consumer, TopicPartition, TopicCollection\n",
    "from confluent_kafka.admin import AdminClient, ConfigResource, ConfigEntry, AlterConfigOpType\n",
    "\n",
    "TOPIC = \"m02l02-readings\"\n",
    "admin = AdminClient({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP})\n",
    "lab.recreate_topic(admin, TOPIC, partitions=1, replication=3)\n",
    "readings = [e for _, e in Brightlane().stream(sessions=20) if \"type\" not in e]\n",
    "\n",
    "def write(records, **config):\n",
    "    \"\"\"Write records with the given settings; return a Counter of outcomes and the offsets used.\"\"\"\n",
    "    outcomes, offsets = Counter(), []\n",
    "    def report(err, msg):\n",
    "        outcomes[err.name() if err else \"delivered\"] += 1\n",
    "        if not err and msg.offset() is not None:\n",
    "            offsets.append(msg.offset())\n",
    "    p = Producer({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP, \"message.timeout.ms\": 10000, **config})\n",
    "    for r in records:\n",
    "        p.produce(TOPIC, key=r[\"charger_id\"], value=to_json(r), on_delivery=report)\n",
    "    p.flush(20)\n",
    "    return dict(outcomes), (min(offsets), max(offsets)) if offsets else None\n",
    "\n",
    "safe = {\"acks\": \"all\", \"enable.idempotence\": True}\n",
    "print(write(readings[:100], **safe))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c04-9a78c806",
   "metadata": {},
   "source": [
    "You should see `({'delivered': 100}, (0, 99))`.\n",
    "\n",
    "Is the producer really idempotent? Its statistics (lesson 8.3) show the **producer id** the\n",
    "cluster gave it. The broker uses that id and a sequence number per batch to drop duplicates."
   ]
  },
  {
   "cell_type": "code",
   "id": "c05-9846aaa4",
   "metadata": {},
   "source": [
    "stats = []\n",
    "p = Producer({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP, **safe,\n",
    "              \"statistics.interval.ms\": 500, \"stats_cb\": stats.append})\n",
    "p.produce(TOPIC, key=\"BL-N03-07\", value=b\"{}\")\n",
    "p.flush(10)\n",
    "p.poll(1.0)                                   # give the client time to emit one statistics report\n",
    "eos = json.loads(stats[-1])[\"eos\"]\n",
    "print(\"idempotence state:\", eos[\"idemp_state\"], \"| producer id:\", eos[\"producer_id\"], \"| epoch:\", eos[\"producer_epoch\"])"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c06-d13d455f",
   "metadata": {},
   "source": [
    "You should see `idempotence state: Assigned` and a producer id (the number varies).\n",
    "\n",
    "## Step 2: Lose one broker\n",
    "Stop a broker that holds a **follower** copy of the partition (not the leader), then write 100\n",
    "more readings with the same settings."
   ]
  },
  {
   "cell_type": "code",
   "id": "c07-ce864ad4",
   "metadata": {},
   "source": [
    "def describe():\n",
    "    part = admin.describe_topics(TopicCollection([TOPIC]))[TOPIC].result().partitions[0]\n",
    "    return part.leader.id, sorted(r.id for r in part.isr)\n",
    "\n",
    "leader, isr = describe()\n",
    "follower = next(b for b in (1, 2, 3) if b != leader)\n",
    "lab.compose(\"stop\", f\"kafka-{follower}\")\n",
    "time.sleep(3)\n",
    "print(\"leader:\", describe()[0], \"| in-sync replicas:\", describe()[1])\n",
    "print(write(readings[100:200], **safe))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c08-40eb01ff",
   "metadata": {},
   "source": [
    "You should see two in-sync replicas and `({'delivered': 100}, (101, 200))`. With\n",
    "`min.insync.replicas=2` (the lab's default, like production), two in-sync copies are enough.\n",
    "\n",
    "## Step 3: Demand three in-sync copies\n",
    "Raise the topic's `min.insync.replicas` to 3, with only two replicas in sync, and write 20 more."
   ]
  },
  {
   "cell_type": "code",
   "id": "c09-3a879bda",
   "metadata": {},
   "source": [
    "def set_min_isr(n):\n",
    "    res = ConfigResource(\"topic\", TOPIC, incremental_configs=[\n",
    "        ConfigEntry(\"min.insync.replicas\", str(n), incremental_operation=AlterConfigOpType.SET)])\n",
    "    admin.incremental_alter_configs([res])[res].result()\n",
    "\n",
    "set_min_isr(3)\n",
    "time.sleep(2)\n",
    "start = time.monotonic()\n",
    "print(write(readings[200:220], **safe), f\"after {time.monotonic() - start:.0f} s\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c10-4bf12a73",
   "metadata": {},
   "source": [
    "Every record fails with `_MSG_TIMED_OUT`, and only after 10 seconds. The broker refused each\n",
    "attempt, but the reason was retriable, so the client kept retrying until its timeout ran out.\n",
    "To see the broker's actual answer, write one record with retries switched off:"
   ]
  },
  {
   "cell_type": "code",
   "id": "c11-56c083f5",
   "metadata": {},
   "source": [
    "print(write(readings[220:221], **{\"acks\": \"all\", \"enable.idempotence\": False, \"retries\": 0}))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c12-f78058de",
   "metadata": {},
   "source": [
    "You should see `NOT_ENOUGH_REPLICAS`: fewer replicas are in sync than the topic demands.\n",
    "\n",
    "## Step 4: acks=1 in the same situation\n",
    "Now write 20 readings with `acks=1`: only the leader has to store them. Then ask the partition\n",
    "how far consumers can read (its high watermark)."
   ]
  },
  {
   "cell_type": "code",
   "id": "c13-af40699e",
   "metadata": {},
   "source": [
    "c = Consumer({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP, \"group.id\": \"m02l02-check\"})\n",
    "tp = TopicPartition(TOPIC, 0)\n",
    "print(\"readable up to (before):\", c.get_watermark_offsets(tp, timeout=10)[1])\n",
    "print(write(readings[221:241], acks=\"1\"))\n",
    "print(\"readable up to (after): \", c.get_watermark_offsets(tp, timeout=10)[1])"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c14-502c91bb",
   "metadata": {},
   "source": [
    "The producer was told the 20 records were stored, at offsets 201 to 220. But the readable end\n",
    "didn't move: consumers can't see records until enough replicas have them. Bring the broker back\n",
    "and look again."
   ]
  },
  {
   "cell_type": "code",
   "id": "c15-fafa0411",
   "metadata": {},
   "source": [
    "lab.compose(\"up\", \"-d\", \"--wait\", f\"kafka-{follower}\")\n",
    "lab.wait_until(lambda: len(describe()[1]) == 3, timeout=60, what=\"all replicas in sync\")\n",
    "print(\"readable up to (broker back):\", c.get_watermark_offsets(tp, timeout=10)[1])\n",
    "set_min_isr(2)"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c16-ae3e7a49",
   "metadata": {},
   "source": [
    "You should see the readable end jump to 221 once the stopped broker has caught up.\n",
    "\n",
    "## Break it: acks=0 never hears \"no\"\n",
    "Predict first: the topic's maximum record size is lowered to 1,000 bytes. A producer with\n",
    "`acks=0` sends a 5,000-byte record. What does its delivery report say?"
   ]
  },
  {
   "cell_type": "code",
   "id": "c17-180749fe",
   "metadata": {},
   "source": [
    "res = ConfigResource(\"topic\", TOPIC, incremental_configs=[\n",
    "    ConfigEntry(\"max.message.bytes\", \"1000\", incremental_operation=AlterConfigOpType.SET)])\n",
    "admin.incremental_alter_configs([res])[res].result()\n",
    "time.sleep(2)\n",
    "big = [dict(readings[0], padding=\"x\" * 5000)]\n",
    "end_before = c.get_watermark_offsets(tp, timeout=10)[1]\n",
    "print(\"acks=0:  \", write(big, acks=\"0\"), \"| readable end:\", end_before, \"->\", c.get_watermark_offsets(tp, timeout=10)[1])\n",
    "print(\"acks=all:\", write(big, **safe))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c18-fffb2611",
   "metadata": {},
   "source": [
    "With `acks=0` the report says **delivered**, but the readable end didn't move: the broker\n",
    "rejected the record and nobody told the producer. With `acks=all` the same record fails with\n",
    "`MSG_SIZE_TOO_LARGE`, and your code can react.\n",
    "\n",
    "## Clean up\n",
    "All brokers are running again. Close the consumer and delete the topic. If anything gets stuck,\n",
    "`docker compose down -v` in the `code/` folder resets the whole lab."
   ]
  },
  {
   "cell_type": "code",
   "id": "c19-89ac0183",
   "metadata": {},
   "source": [
    "c.close()\n",
    "lab.delete_topics(admin, [TOPIC])\n",
    "admin.delete_consumer_groups([\"m02l02-check\"])\n",
    "print(\"cleaned up\")"
   ],
   "execution_count": null,
   "outputs": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
