{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c00-4888d012",
   "metadata": {},
   "source": [
    "# Keys decide order — practical\n",
    "\n",
    "You write the evening's session events into a 6-partition topic three ways (no key, keyed by\n",
    "charger, keyed by region) and check what each does to order and to the load on each partition.\n",
    "Then you compare Python's and Java's partitioners, and add partitions to a live topic.\n",
    "\n",
    "**Lab services:** kafka-1, kafka-2, kafka-3 · **Lesson:** `modules/02-producers/01-keys-decide-order.md`"
   ]
  },
  {
   "cell_type": "code",
   "id": "c01-6a63f24b",
   "metadata": {},
   "source": [
    "# Setup: find the lab's shared helpers and check the lab is running\n",
    "import sys, pathlib\n",
    "lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / \"common\").is_dir())\n",
    "sys.path.insert(0, str(lab_root))\n",
    "\n",
    "from collections import Counter, defaultdict\n",
    "from common import settings, lab\n",
    "from common.scenario import Brightlane, to_json, from_json\n",
    "\n",
    "lab.require_kafka()\n",
    "lab.print_versions()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c02-0f6b4ec0",
   "metadata": {},
   "source": [
    "## Step 1: Write session events without a key\n",
    "200 sessions make 400 events: a `session_started` and a `session_stopped` each, in time order.\n",
    "A helper writes them to a fresh 6-partition topic and reads them back with one consumer, the\n",
    "way billing would."
   ]
  },
  {
   "cell_type": "code",
   "id": "c03-08f4f149",
   "metadata": {},
   "source": [
    "from confluent_kafka import Producer, Consumer, TopicPartition\n",
    "from confluent_kafka.admin import AdminClient\n",
    "\n",
    "admin = AdminClient({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP})\n",
    "events = Brightlane().stream(sessions=200, readings=False)\n",
    "\n",
    "def write_and_read(topic, key_of, producer_config):\n",
    "    \"\"\"Write all events with key_of(charger_id) as key, then read the topic back in one pass.\"\"\"\n",
    "    lab.recreate_topic(admin, topic, partitions=6)\n",
    "    producer = Producer({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP, **producer_config})\n",
    "    for charger_id, event in events:\n",
    "        producer.produce(topic, key=key_of(charger_id), value=to_json(event))\n",
    "    producer.flush(30)\n",
    "    c = Consumer({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP, \"group.id\": f\"{topic}-billing\",\n",
    "                  \"enable.auto.commit\": False})\n",
    "    c.assign([TopicPartition(topic, p, 0) for p in range(6)])\n",
    "    got = []\n",
    "    for _ in range(100):                      # bounded: stop when all 400 are back\n",
    "        got += [(m.partition(), from_json(m.value())) for m in c.consume(500, 0.5) if not m.error()]\n",
    "        if len(got) >= len(events):\n",
    "            break\n",
    "    c.close()\n",
    "    return got"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c04-d2b3ff32",
   "metadata": {},
   "source": [
    "Now the unkeyed write. A charger gateway spreads its events over the evening, so the client\n",
    "would send each one to a random partition. The notebook writes all 400 in a burst, so it turns\n",
    "off the client's habit of filling one partition for 10 ms at a time (`sticky.partitioning.linger.ms`)."
   ]
  },
  {
   "cell_type": "code",
   "id": "c05-b453d1fb",
   "metadata": {},
   "source": [
    "def check_order(got):\n",
    "    \"\"\"Count sessions whose stop was read before their start, and partitions per charger.\"\"\"\n",
    "    started, out_of_order, parts = set(), 0, defaultdict(set)\n",
    "    for partition, e in got:\n",
    "        parts[e[\"charger_id\"]].add(partition)\n",
    "        if e[\"type\"] == \"session_started\":\n",
    "            started.add(e[\"session_id\"])\n",
    "        elif e[\"session_id\"] not in started:\n",
    "            out_of_order += 1\n",
    "    spread = Counter(len(p) for p in parts.values())\n",
    "    print(f\"sessions read stop-before-start: {out_of_order} of {len(started)}\")\n",
    "    print(\"chargers by number of partitions their events landed in:\", dict(sorted(spread.items())))\n",
    "\n",
    "no_key = write_and_read(\"m02l01-no-key\", lambda charger: None, {\"sticky.partitioning.linger.ms\": 0})\n",
    "check_order(no_key)"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c06-385a0e2b",
   "metadata": {},
   "source": [
    "You should see dozens of sessions read stop-before-start, and chargers whose events landed in\n",
    "2, 3 or more partitions. The exact numbers vary from run to run.\n",
    "\n",
    "## Step 2: Key by charger id\n",
    "Same events, now keyed by `charger_id`. The partitioner hashes the key, so every event of one\n",
    "charger goes to the same partition."
   ]
  },
  {
   "cell_type": "code",
   "id": "c07-c6cd508c",
   "metadata": {},
   "source": [
    "by_charger = write_and_read(\"m02l01-by-charger\", lambda charger: charger,\n",
    "                            {\"partitioner\": \"murmur2_random\"})\n",
    "check_order(by_charger)\n",
    "print(\"records per partition:\", sorted(Counter(p for p, _ in by_charger).items()))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c08-cc095bce",
   "metadata": {},
   "source": [
    "You should see `0 of 200` sessions out of order, every charger in exactly 1 partition, and\n",
    "the records spread over all six partitions.\n",
    "\n",
    "## Step 3: A key with too few values\n",
    "Someone suggests keying by region (N, C or V) \"to keep each region together\". Count the\n",
    "records per partition."
   ]
  },
  {
   "cell_type": "code",
   "id": "c09-297f2b52",
   "metadata": {},
   "source": [
    "by_region = write_and_read(\"m02l01-by-region\", lambda charger: charger.split(\"-\")[1][0],\n",
    "                           {\"partitioner\": \"murmur2_random\"})\n",
    "print(\"records per partition:\", sorted(Counter(p for p, _ in by_region).items()))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c10-455d12d4",
   "metadata": {},
   "source": [
    "You should see only **two** partitions with records: regions N and V hash to the same partition.\n",
    "Order is still kept, but one partition carries most of the traffic and four sit idle.\n",
    "\n",
    "## Break it: two languages, two partitions\n",
    "Predict first: a Java service and a Python service both write events keyed `BL-N03-07` to the\n",
    "same 6-partition topic. Do they land in the same partition?\n",
    "\n",
    "Kafka's console producer is a Java program. The notebook runs it inside kafka-1, then writes\n",
    "the same key from Python with the client's default partitioner and with `murmur2_random`."
   ]
  },
  {
   "cell_type": "code",
   "id": "c11-885fba99",
   "metadata": {},
   "source": [
    "T = \"m02l01-by-charger\"\n",
    "lab.kafka_cli(\"kafka-console-producer.sh\", \"--bootstrap-server\", \"localhost:19092\", \"--topic\", T,\n",
    "              \"--property\", \"parse.key=true\", \"--property\", \"key.separator=:\",\n",
    "              stdin=\"BL-N03-07:from-java\\n\")\n",
    "\n",
    "for label, config in [(\"python default\", {}), (\"python murmur2_random\", {\"partitioner\": \"murmur2_random\"})]:\n",
    "    p = Producer({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP, **config})\n",
    "    p.produce(T, key=\"BL-N03-07\", value=label.encode())\n",
    "    p.flush(10)\n",
    "\n",
    "c = Consumer({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP, \"group.id\": \"m02l01-check\"})\n",
    "c.assign([TopicPartition(T, p, 0) for p in range(6)])\n",
    "for m in [m for _ in range(20) for m in c.consume(500, 0.5) if not m.error()]:\n",
    "    if m.value() in (b\"from-java\", b\"python default\", b\"python murmur2_random\"):\n",
    "        print(f\"{m.value().decode():<22} -> partition {m.partition()}\")\n",
    "c.close()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c12-6fc09707",
   "metadata": {},
   "source": [
    "You should see Java and `murmur2_random` in partition **2**, and the Python default in\n",
    "partition **5**. The two default hash functions differ, so the same charger's events would sit in\n",
    "two partitions with no order between them.\n",
    "\n",
    "## Change it: add partitions to a live topic\n",
    "Brightlane plans to grow, and someone raises `m02l01-by-charger` from 6 to 8 partitions.\n",
    "Predict first: how many of the 240 chargers will now write to a different partition?"
   ]
  },
  {
   "cell_type": "code",
   "id": "c13-82452e67",
   "metadata": {},
   "source": [
    "from confluent_kafka import murmur2\n",
    "from confluent_kafka.admin import NewPartitions\n",
    "\n",
    "admin.create_partitions([NewPartitions(T, 8)])[T].result()\n",
    "chargers = [c[\"charger_id\"] for c in Brightlane().chargers]\n",
    "moved = [ch for ch in chargers if murmur2(ch.encode(), 6) != murmur2(ch.encode(), 8)]\n",
    "print(f\"{len(moved)} of {len(chargers)} chargers change partition\")\n",
    "for ch in [\"BL-N03-07\", \"BL-C08-02\"]:\n",
    "    print(f\"{ch}: partition {murmur2(ch.encode(), 6)} -> {murmur2(ch.encode(), 8)}\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c14-bd1b3826",
   "metadata": {},
   "source": [
    "You should see `172 of 240 chargers change partition`. BL-N03-07 stays in partition 2;\n",
    "BL-C08-02 moves from 4 to 0. For the moved chargers, older events sit in one partition and new\n",
    "ones in another, so a reader can see them out of order during the switch.\n",
    "\n",
    "## Clean up\n",
    "Delete the three topics. If anything gets stuck, `docker compose down -v` in the `code/` folder\n",
    "resets the whole lab."
   ]
  },
  {
   "cell_type": "code",
   "id": "c15-5cbe99ee",
   "metadata": {},
   "source": [
    "lab.delete_topics(admin, [\"m02l01-no-key\", \"m02l01-by-charger\", \"m02l01-by-region\"])\n",
    "admin.delete_consumer_groups([\"m02l01-check\"])\n",
    "print(\"cleaned up\")"
   ],
   "execution_count": null,
   "outputs": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
