{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c00-de1ea453",
   "metadata": {},
   "source": [
    "# Why Brightlane needs an event log — practical\n",
    "\n",
    "You write five charging-session events once, then let three independent readers (billing, the\n",
    "grid report and a team that starts next week) each read all of them, at their own pace.\n",
    "\n",
    "**Lab services:** kafka-1, kafka-2, kafka-3 · **Lesson:** `modules/01-first-events/01-why-brightlane-needs-an-event-log.md`"
   ]
  },
  {
   "cell_type": "code",
   "id": "c01-35aaa581",
   "metadata": {},
   "source": [
    "# Setup: find the lab's shared helpers and check the lab is running\n",
    "import sys, pathlib\n",
    "lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / \"common\").is_dir())\n",
    "sys.path.insert(0, str(lab_root))\n",
    "\n",
    "from common import settings, lab\n",
    "from common.scenario import Brightlane, to_json, from_json\n",
    "\n",
    "lab.require_kafka()\n",
    "lab.print_versions()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c02-6d68dd27",
   "metadata": {},
   "source": [
    "## Step 1: Create the topic\n",
    "A topic is a named log. This one holds Brightlane's session events. One partition keeps the\n",
    "first lesson simple; lesson 1.2 explains partitions."
   ]
  },
  {
   "cell_type": "code",
   "id": "c03-5296692b",
   "metadata": {},
   "source": [
    "from confluent_kafka.admin import AdminClient, NewTopic\n",
    "\n",
    "TOPIC = \"m01l01-sessions\"\n",
    "admin = AdminClient({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP})\n",
    "lab.delete_topics(admin, [TOPIC])                                   # start clean on reruns\n",
    "admin.delete_consumer_groups([\"m01l01-billing\", \"m01l01-grid\", \"m01l01-app\", \"m01l01-latecomer\"])\n",
    "\n",
    "futures = admin.create_topics([NewTopic(TOPIC, num_partitions=1, replication_factor=3)])\n",
    "futures[TOPIC].result()                                             # raises if creation failed\n",
    "print(f\"created {TOPIC}\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c04-aede10c7",
   "metadata": {},
   "source": [
    "You should see `created m01l01-sessions`.\n",
    "\n",
    "## Step 2: The gateway writes five events\n",
    "The charger gateway is the **producer**. It appends each event to the end of the log. It doesn't\n",
    "know who will read the events."
   ]
  },
  {
   "cell_type": "code",
   "id": "c05-a4997a2e",
   "metadata": {},
   "source": [
    "from confluent_kafka import Producer\n",
    "\n",
    "bl = Brightlane()\n",
    "events = bl.stream(sessions=5, readings=False)[:5]     # the first five session events of the evening\n",
    "\n",
    "producer = Producer({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP})\n",
    "for charger_id, event in events:\n",
    "    producer.produce(TOPIC, key=charger_id, value=to_json(event))\n",
    "    print(f\"{event['ts']}  {event['type']:<16} {event['session_id']}  {charger_id}\")\n",
    "print(\"not yet stored:\", producer.flush(10))           # wait until Kafka has stored all of them"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c06-0465fc5d",
   "metadata": {},
   "source": [
    "You should see five `session_started` events between 17:30:07 and 17:30:46.\n",
    "\n",
    "## Step 3: Billing reads them\n",
    "Billing is a **consumer**. It polls the log for events after its own position. `group.id` names\n",
    "the reader: Kafka remembers each group's position separately.\n",
    "`auto.offset.reset=earliest` tells a brand-new group to start at the beginning of the log."
   ]
  },
  {
   "cell_type": "code",
   "id": "c07-45b70836",
   "metadata": {},
   "source": [
    "import time\n",
    "from confluent_kafka import Consumer\n",
    "\n",
    "def read_available(consumer, max_wait=10.0, quiet=2.0):\n",
    "    \"\"\"Poll until no new event arrives for `quiet` seconds (or `max_wait` passes).\"\"\"\n",
    "    got, deadline, last = [], time.monotonic() + max_wait, time.monotonic()\n",
    "    while time.monotonic() < deadline and time.monotonic() - last < quiet:\n",
    "        msg = consumer.poll(0.5)\n",
    "        if msg is None or msg.error():\n",
    "            continue\n",
    "        got.append(from_json(msg.value()))\n",
    "        last = time.monotonic()\n",
    "    return got\n",
    "\n",
    "def reader(group_id, start=\"earliest\"):\n",
    "    c = Consumer({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP, \"group.id\": group_id,\n",
    "                  \"auto.offset.reset\": start})\n",
    "    c.subscribe([TOPIC])\n",
    "    return c\n",
    "\n",
    "billing = reader(\"m01l01-billing\")\n",
    "print(\"billing read:\", [e[\"session_id\"] for e in read_available(billing)])"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c08-3f8c1f69",
   "metadata": {},
   "source": [
    "You should see billing read all five session ids, oldest first.\n",
    "\n",
    "## Step 4: The grid report reads the same events\n",
    "Reading didn't remove anything. A second reader with its own `group.id` gets all five too."
   ]
  },
  {
   "cell_type": "code",
   "id": "c09-550b666f",
   "metadata": {},
   "source": [
    "grid = reader(\"m01l01-grid\")\n",
    "print(\"grid read:   \", [e[\"session_id\"] for e in read_available(grid)])"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c10-9ec83da8",
   "metadata": {},
   "source": [
    "You should see the same five ids. Billing's read had no effect on what the grid report sees.\n",
    "\n",
    "## Step 5: Three more events arrive\n",
    "Each reader continues from where it stopped. Nobody gets the first five again."
   ]
  },
  {
   "cell_type": "code",
   "id": "c11-51b75448",
   "metadata": {},
   "source": [
    "for charger_id, event in bl.stream(sessions=8, readings=False)[5:8]:\n",
    "    producer.produce(TOPIC, key=charger_id, value=to_json(event))\n",
    "producer.flush()\n",
    "\n",
    "print(\"billing read:\", [e[\"session_id\"] for e in read_available(billing)])\n",
    "print(\"grid read:   \", [e[\"session_id\"] for e in read_available(grid)])"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c12-e4cd5ba4",
   "metadata": {},
   "source": [
    "You should see both readers get the same three new ids, and only those.\n",
    "\n",
    "## Step 6: A team that starts next week\n",
    "The driver-app team arrives later. The log still has every event, so they start from the\n",
    "beginning without asking anyone to resend data."
   ]
  },
  {
   "cell_type": "code",
   "id": "c13-e08b6a4d",
   "metadata": {},
   "source": [
    "app = reader(\"m01l01-app\")\n",
    "history = read_available(app)\n",
    "print(f\"app read {len(history)} events, first {history[0]['ts']}, last {history[-1]['ts']}\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c14-45afc9f2",
   "metadata": {},
   "source": [
    "You should see `app read 8 events`. The log kept them; it didn't matter that two other readers\n",
    "had already processed them.\n",
    "\n",
    "## Break it: a newcomer that starts at \"latest\"\n",
    "Predict first: a new group with `auto.offset.reset=latest` subscribes now. How many of the 8\n",
    "stored events will it read?"
   ]
  },
  {
   "cell_type": "code",
   "id": "c15-78e7f0df",
   "metadata": {},
   "source": [
    "latecomer = reader(\"m01l01-latecomer\", start=\"latest\")\n",
    "print(\"latecomer read:\", len(read_available(latecomer, max_wait=6)), \"events\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c16-67438715",
   "metadata": {},
   "source": [
    "It reads **0**. A new group has no position yet, and `latest` places it at the end of the log:\n",
    "it only sees events written from now on. This is the most common reason for \"my consumer\n",
    "shows nothing\". The Python client's default is `latest`.\n",
    "\n",
    "## Clean up\n",
    "Close the consumers (they leave their groups at once), then delete the topic and the groups.\n",
    "If anything gets stuck, `docker compose down -v` in the `code/` folder resets the whole lab."
   ]
  },
  {
   "cell_type": "code",
   "id": "c17-fa9aa20f",
   "metadata": {},
   "source": [
    "for c in (billing, grid, app, latecomer):\n",
    "    c.close()\n",
    "lab.delete_topics(admin, [TOPIC])\n",
    "admin.delete_consumer_groups([\"m01l01-billing\", \"m01l01-grid\", \"m01l01-app\", \"m01l01-latecomer\"])\n",
    "print(\"cleaned up\")"
   ],
   "execution_count": null,
   "outputs": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
