{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c00-491df005",
   "metadata": {},
   "source": [
    "# Poison pills and dead-letter topics — practical\n",
    "\n",
    "600 meter readings, 30 of them forwarded unchanged from legacy 3.9 chargers. A naive worker\n",
    "crashes on the first legacy one, again and again. A careful worker parks the bad records in a\n",
    "dead-letter topic and rides out a ledger outage. Then a worker that parks every error throws away\n",
    "good readings.\n",
    "\n",
    "**Lab services:** kafka-1, kafka-2, kafka-3, schema-registry (`docker compose --profile schemas up -d --wait`)\n",
    "· **Script:** `module-05/telemetry_worker.py` · **Lesson:** `modules/05-data-contracts/03-poison-pills-and-dead-letter-topics.md`"
   ]
  },
  {
   "cell_type": "code",
   "id": "c01-cd567beb",
   "metadata": {},
   "source": [
    "# Setup: find the lab's shared helpers and check the lab is running\n",
    "import sys, pathlib\n",
    "lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / \"common\").is_dir())\n",
    "sys.path.insert(0, str(lab_root))\n",
    "\n",
    "import sqlite3, time\n",
    "from collections import Counter\n",
    "from common import settings, lab\n",
    "from common.scenario import Brightlane, parse_iso\n",
    "\n",
    "lab.require_kafka()\n",
    "lab.require_http(\"Schema Registry\", settings.SCHEMA_REGISTRY_URL + \"/subjects\",\n",
    "                 start_hint=\"docker compose --profile schemas up -d --wait\")\n",
    "lab.print_versions()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c02-7fdc5d65",
   "metadata": {},
   "source": [
    "## Step 1: The readings, with the gateway's legacy bug\n",
    "The gateway serializes readings with the Avro schema, except on its legacy path: readings from\n",
    "3.9 chargers are forwarded as the charger sent them."
   ]
  },
  {
   "cell_type": "code",
   "id": "c03-aaf98146",
   "metadata": {},
   "source": [
    "from confluent_kafka import Producer, Consumer, TopicPartition\n",
    "from confluent_kafka.admin import AdminClient\n",
    "from confluent_kafka.schema_registry import SchemaRegistryClient\n",
    "from confluent_kafka.schema_registry.avro import AvroSerializer\n",
    "from confluent_kafka.serialization import SerializationContext, MessageField\n",
    "\n",
    "TOPIC, DLQ = \"m05l03-telemetry\", \"m05l03-telemetry.dlq\"\n",
    "admin = AdminClient({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP})\n",
    "lab.recreate_topic(admin, TOPIC, partitions=1)\n",
    "lab.recreate_topic(admin, DLQ, partitions=1)\n",
    "registry = SchemaRegistryClient({\"url\": settings.SCHEMA_REGISTRY_URL})\n",
    "serialize = AvroSerializer(registry, (lab_root / \"schemas\" / \"meter_reading.avsc\").read_text(),\n",
    "                           lambda r, ctx: {**r, \"ts\": parse_iso(r[\"ts\"])})\n",
    "context = SerializationContext(TOPIC, MessageField.VALUE)\n",
    "\n",
    "telemetry = Brightlane().telemetry_with_legacy(600)\n",
    "producer = Producer(settings.PRODUCER_DEFAULTS)\n",
    "for key, reading, legacy_bytes in telemetry:\n",
    "    producer.produce(TOPIC, key=key, value=legacy_bytes or serialize(reading, context))\n",
    "producer.flush(10)\n",
    "legacy = [b for _, _, b in telemetry if b]\n",
    "print(f\"{len(telemetry)} readings, {len(legacy)} of them legacy payloads, for example:\")\n",
    "print(legacy[0])\n",
    "print(legacy[2])"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c04-b057137b",
   "metadata": {},
   "source": [
    "## Step 2: A naive worker\n",
    "The naive worker lets any error stop it. The notebook restarts it three times, like a container\n",
    "platform would. `run_worker()` starts the script and waits until it has exited or has nothing\n",
    "left to read."
   ]
  },
  {
   "cell_type": "code",
   "id": "c05-24a139d7",
   "metadata": {},
   "source": [
    "DATA = lab_root / \".lab-data\"\n",
    "DATA.mkdir(exist_ok=True)\n",
    "\n",
    "def run_worker(name, mode, *extra):\n",
    "    group, ledger = f\"m05l03-{name}\", DATA / f\"m05l03-{name}.db\"\n",
    "    ledger.unlink(missing_ok=True)\n",
    "    args = [\"--mode\", mode, \"--topic\", TOPIC, \"--group\", group, \"--ledger\", f\".lab-data/{ledger.name}\", *extra]\n",
    "    worker = lab.start_script(\"module-05/telemetry_worker.py\", *args, name=f\"m05l03-{name}\")\n",
    "    lab.wait_until(lambda: not worker.running or worker.lines(\"idle:\"), timeout=120, interval=1,\n",
    "                   what=\"the worker to finish\")\n",
    "    lab.stop_script(worker)\n",
    "    return worker\n",
    "\n",
    "admin.delete_consumer_groups([\"m05l03-naive\"])\n",
    "for attempt in range(1, 4):\n",
    "    worker = run_worker(\"naive\", \"naive\")\n",
    "    print(f\"run {attempt}:\", worker.lines(\"CRASH\")[-1].split(\" \", 1)[1])\n",
    "print(\"lag:\", lab.group_lag(admin, \"m05l03-naive\", TOPIC, 1))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c06-0e0a285d",
   "metadata": {},
   "source": [
    "Every run dies on the same record, offset 19, and the lag never shrinks: the record can't be\n",
    "processed, and the worker never gets past it. That's a **poison pill**.\n",
    "\n",
    "## Step 3: A careful worker with a dead-letter topic\n",
    "The careful worker parks records it can't decode or validate in `m05l03-telemetry.dlq`, and\n",
    "retries when the ledger is unavailable. This run also has a 2-second ledger outage after 300\n",
    "readings."
   ]
  },
  {
   "cell_type": "code",
   "id": "c07-68df95b5",
   "metadata": {},
   "source": [
    "careful = run_worker(\"careful\", \"careful\", \"--outage-at\", \"300\", \"--outage-s\", \"2\")\n",
    "with sqlite3.connect(DATA / \"m05l03-careful.db\") as db:\n",
    "    in_ledger = db.execute(\"SELECT count(*) FROM readings\").fetchone()[0]\n",
    "print(\"readings in the ledger:\", in_ledger, \"  parked:\", len(careful.lines(\"PARKED\")))\n",
    "for line in careful.lines(\"RETRY\"):\n",
    "    print(line)"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c08-66dfbfa4",
   "metadata": {},
   "source": [
    "## Step 4: What's in the dead-letter topic?\n",
    "Each parked record keeps its original bytes, and its headers say where it came from and why it\n",
    "was parked."
   ]
  },
  {
   "cell_type": "code",
   "id": "c09-ab4a6e73",
   "metadata": {},
   "source": [
    "def read_dlq():\n",
    "    consumer = Consumer({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP, \"group.id\": \"m05l03-inspect\",\n",
    "                         \"enable.auto.commit\": False})\n",
    "    end = consumer.get_watermark_offsets(TopicPartition(DLQ, 0), timeout=10)[1]\n",
    "    consumer.assign([TopicPartition(DLQ, 0, 0)])\n",
    "    records = [consumer.poll(10) for _ in range(end)]\n",
    "    consumer.close()\n",
    "    return records\n",
    "\n",
    "parked = read_dlq()\n",
    "first = parked[0]\n",
    "for key, value in first.headers():\n",
    "    print(f\"{key:<22} {value.decode()}\")\n",
    "print(\"value:\", first.value())\n",
    "print(Counter(dict(r.headers())[\"dlq.error.message\"].decode() for r in parked))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c10-1a8da061",
   "metadata": {},
   "source": [
    "## Break it: park every error\n",
    "A shortcut many workers take: `except Exception: send it to the dead-letter topic`. Predict\n",
    "first: what happens to good readings during the same 2-second ledger outage?"
   ]
  },
  {
   "cell_type": "code",
   "id": "c11-528dbe79",
   "metadata": {},
   "source": [
    "lab.recreate_topic(admin, DLQ, partitions=1)\n",
    "sloppy = run_worker(\"park-all\", \"park-all\", \"--outage-at\", \"300\", \"--outage-s\", \"2\")\n",
    "with sqlite3.connect(DATA / \"m05l03-park-all.db\") as db:\n",
    "    in_ledger = db.execute(\"SELECT count(*) FROM readings\").fetchone()[0]\n",
    "reasons = Counter(dict(r.headers())[\"dlq.error.class\"].decode() for r in read_dlq())\n",
    "print(\"readings in the ledger:\", in_ledger, \"  parked:\", len(sloppy.lines(\"PARKED\")))\n",
    "print(\"parked, by error:\", dict(reasons))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c12-3f0ffe05",
   "metadata": {},
   "source": [
    "The worker parked good readings as if they were bad data. They are missing from the ledger now,\n",
    "and somebody has to find and replay them.\n",
    "\n",
    "## Clean up\n",
    "Delete the topics, the groups and the ledgers. If anything gets stuck, `docker compose down -v` in\n",
    "the `code/` folder resets the whole lab."
   ]
  },
  {
   "cell_type": "code",
   "id": "c13-f7151db6",
   "metadata": {},
   "source": [
    "lab.delete_topics(admin, [TOPIC, DLQ])\n",
    "registry.delete_subject(f\"{TOPIC}-value\")\n",
    "registry.delete_subject(f\"{TOPIC}-value\", permanent=True)\n",
    "admin.delete_consumer_groups([\"m05l03-naive\", \"m05l03-careful\", \"m05l03-park-all\", \"m05l03-inspect\"])\n",
    "for f in DATA.glob(\"m05l03-*.db*\"):\n",
    "    f.unlink()\n",
    "print(\"cleaned up\")"
   ],
   "execution_count": null,
   "outputs": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
