{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c00-7c392411",
   "metadata": {},
   "source": [
    "# Committing offsets: where processing resumes — practical\n",
    "\n",
    "Billing charges 400 sessions into a ledger. You crash the billing worker halfway (`kill -9`),\n",
    "restart it, and count the charges, once per commit strategy. Then you make the ledger\n",
    "idempotent and get exactly 400.\n",
    "\n",
    "**Lab services:** kafka-1, kafka-2, kafka-3 · **Script:** `module-03/billing_worker.py`\n",
    "· **Lesson:** `modules/03-consumers/02-committing-offsets.md`"
   ]
  },
  {
   "cell_type": "code",
   "id": "c01-81e7ccb4",
   "metadata": {},
   "source": [
    "# Setup: find the lab's shared helpers and check the lab is running\n",
    "import sys, pathlib\n",
    "lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / \"common\").is_dir())\n",
    "sys.path.insert(0, str(lab_root))\n",
    "\n",
    "import sqlite3, time\n",
    "from common import settings, lab\n",
    "from common.scenario import Brightlane, to_json\n",
    "\n",
    "lab.require_kafka()\n",
    "lab.print_versions()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c02-75058176",
   "metadata": {},
   "source": [
    "## Step 1: Yesterday's sessions, waiting to be billed\n",
    "400 sessions, so 400 `session_stopped` events among 800 events. A correct billing run writes\n",
    "exactly 400 charges."
   ]
  },
  {
   "cell_type": "code",
   "id": "c03-f9bd16f2",
   "metadata": {},
   "source": [
    "from confluent_kafka import Producer\n",
    "from confluent_kafka.admin import AdminClient\n",
    "\n",
    "TOPIC = \"m03l02-sessions\"\n",
    "admin = AdminClient({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP})\n",
    "lab.recreate_topic(admin, TOPIC, partitions=3)\n",
    "producer = Producer(settings.PRODUCER_DEFAULTS)\n",
    "for charger_id, event in Brightlane().stream(sessions=400, readings=False):\n",
    "    producer.produce(TOPIC, key=charger_id, value=to_json(event))\n",
    "print(\"not yet stored:\", producer.flush(30))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c04-5b086f8a",
   "metadata": {},
   "source": [
    "## Step 2: A crash test\n",
    "`crash_test()` starts a billing worker with a fresh group and an empty SQLite ledger. The worker\n",
    "fetches 50 events at a time and spends 15 ms on each. When 150 charges are in the ledger, the\n",
    "test kills it with `kill -9`, starts it again, waits until the group has caught up, and counts.\n",
    "\n",
    "The workers use `session.timeout.ms=6000`, so the group notices the crashed member after 6\n",
    "seconds instead of 45 (lesson 3.3 explains this setting)."
   ]
  },
  {
   "cell_type": "code",
   "id": "c05-fc2367fe",
   "metadata": {},
   "source": [
    "DATA = lab_root / \".lab-data\"\n",
    "DATA.mkdir(exist_ok=True)\n",
    "\n",
    "def crash_test(commit, idempotent=False):\n",
    "    group = f\"m03l02-{commit}\" + (\"-idempotent\" if idempotent else \"\")\n",
    "    admin.delete_consumer_groups([group])\n",
    "    ledger = DATA / f\"{group}.db\"\n",
    "    ledger.unlink(missing_ok=True)\n",
    "    args = [\"--name\", \"W\", \"--topic\", TOPIC, \"--group\", group, \"--commit\", commit, \"--batch\", \"50\",\n",
    "            \"--work-ms\", \"15\", \"--ledger\", f\".lab-data/{group}.db\", \"--session-timeout-ms\", \"6000\"]\n",
    "    args += [\"--idempotent\"] if idempotent else []\n",
    "    charges = lambda: sqlite3.connect(ledger).execute(\"SELECT count(*) FROM charges\").fetchone()[0] if ledger.exists() else 0\n",
    "    first = lab.start_script(\"module-03/billing_worker.py\", *args, name=f\"{group}-1\")\n",
    "    lab.wait_until(lambda: charges() >= 150, timeout=60, interval=0.1, what=\"150 charges\")\n",
    "    lab.kill_script(first)                                   # crash: no close(), no final commit\n",
    "    second = lab.start_script(\"module-03/billing_worker.py\", *args, name=f\"{group}-2\")\n",
    "    lab.wait_until(lambda: sum(r[2] for r in lab.group_lag(admin, group, TOPIC, 3).values()) == 0,\n",
    "                   timeout=90, interval=1, what=\"the group to catch up\")\n",
    "    lab.stop_script(second)\n",
    "    rows, distinct = sqlite3.connect(ledger).execute(\n",
    "        \"SELECT count(*), count(DISTINCT session_id) FROM charges\").fetchone()\n",
    "    return {\"commit\": commit + (\" + idempotent ledger\" if idempotent else \"\"), \"charges\": rows,\n",
    "            \"sessions charged\": distinct, \"duplicates\": rows - distinct, \"missing\": 400 - distinct}"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c06-325e3ed8",
   "metadata": {},
   "source": [
    "## Step 3: The client's default: auto-commit\n",
    "With the defaults, the client marks each event as done as soon as `consume()` hands it to you,\n",
    "and commits those marks in the background every 5 seconds."
   ]
  },
  {
   "cell_type": "code",
   "id": "c07-f2317713",
   "metadata": {},
   "source": [
    "results = [crash_test(\"auto\")]\n",
    "lab.table([list(r.values()) for r in results], list(results[0].keys()))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c08-13f22c5b",
   "metadata": {},
   "source": [
    "You should see **duplicates**: the events handled after the last background commit were handled\n",
    "again after the restart. The counts vary from run to run.\n",
    "\n",
    "## Step 4: Commit after processing each batch\n",
    "Now the worker turns auto-commit off and calls `commit()` itself, after each batch of 50 has\n",
    "been processed."
   ]
  },
  {
   "cell_type": "code",
   "id": "c09-c19a514c",
   "metadata": {},
   "source": [
    "results.append(crash_test(\"after\"))\n",
    "lab.table([list(r.values()) for r in results], list(results[0].keys()))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c10-04ffb7ec",
   "metadata": {},
   "source": [
    "Still duplicates, but never more than one batch, and nothing is missing. This is\n",
    "**at-least-once** processing.\n",
    "\n",
    "## Break it: commit before processing\n",
    "Predict first: the worker commits each batch as soon as it has fetched it, then processes it.\n",
    "What happens to the batch that was in progress when it crashed?"
   ]
  },
  {
   "cell_type": "code",
   "id": "c11-67371b14",
   "metadata": {},
   "source": [
    "results.append(crash_test(\"before\"))\n",
    "lab.table([list(r.values()) for r in results], list(results[0].keys()))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c12-f3dd8ebe",
   "metadata": {},
   "source": [
    "No duplicates, but sessions are **missing**: the crashed batch was already committed, so the\n",
    "restarted worker skipped it. This is **at-most-once** processing: those drivers were never charged.\n",
    "\n",
    "## Step 5: Make the ledger idempotent\n",
    "Keep at-least-once, and make a second charge for the same session harmless. The worker's\n",
    "`--idempotent` option makes `session_id` the ledger's primary key and inserts with `INSERT OR\n",
    "IGNORE`. The `store` mode is the Python client's recommended pattern: mark each event done\n",
    "after processing it (`store_offsets`), and let the background commit send the marks."
   ]
  },
  {
   "cell_type": "code",
   "id": "c13-a88f4f02",
   "metadata": {},
   "source": [
    "results.append(crash_test(\"store\"))\n",
    "results.append(crash_test(\"store\", idempotent=True))\n",
    "lab.table([list(r.values()) for r in results], list(results[0].keys()))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c14-dd7e2ebf",
   "metadata": {},
   "source": [
    "With the idempotent ledger you should see exactly 400 charges, 0 duplicates, 0 missing.\n",
    "\n",
    "## Clean up\n",
    "Delete the topic, the groups and the ledgers. If anything gets stuck, `docker compose down -v`\n",
    "in the `code/` folder resets the whole lab."
   ]
  },
  {
   "cell_type": "code",
   "id": "c15-ffd8068a",
   "metadata": {},
   "source": [
    "lab.delete_topics(admin, [TOPIC])\n",
    "admin.delete_consumer_groups([f\"m03l02-{m}\" for m in (\"auto\", \"after\", \"before\", \"store\", \"store-idempotent\")])\n",
    "for f in DATA.glob(\"m03l02-*.db*\"):\n",
    "    f.unlink()\n",
    "print(\"cleaned up\")"
   ],
   "execution_count": null,
   "outputs": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
