{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c00-574946fe",
   "metadata": {},
   "source": [
    "# Event time and windows — practical\n",
    "\n",
    "The grid report sums each site's energy in 5-minute windows. One hour of meter readings arrives,\n",
    "and site-N03's link drops for five minutes, so its readings arrive late. You build the report by\n",
    "arrival time and by event time, with different grace periods, and then meet a charger whose clock\n",
    "is three hours ahead.\n",
    "\n",
    "**Lab services:** kafka-1, kafka-2, kafka-3 · **Script:** `module-07/grid_report.py`\n",
    "· **Lesson:** `modules/07-stream-processing/03-event-time-and-windows.md`"
   ]
  },
  {
   "cell_type": "code",
   "id": "c01-73952a25",
   "metadata": {},
   "source": [
    "# Setup: find the lab's shared helpers and check the lab is running\n",
    "import sys, pathlib\n",
    "lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / \"common\").is_dir())\n",
    "sys.path.insert(0, str(lab_root))\n",
    "\n",
    "import subprocess\n",
    "from datetime import timedelta\n",
    "from common import settings, lab\n",
    "from common.scenario import Brightlane, BASE_TIME, iso, parse_iso, to_json\n",
    "\n",
    "lab.require_kafka()\n",
    "lab.print_versions()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c02-97d1d9dd",
   "metadata": {},
   "source": [
    "## Step 1: One hour of readings, and a link that drops\n",
    "The readings from 17:30 to 18:30. Site-N03's mobile link drops from 18:05 to 18:10: its readings\n",
    "from those five minutes arrive five minutes late. Each record's Kafka timestamp is the time it\n",
    "arrived at the gateway; the reading's own `ts` is the time the charger measured it."
   ]
  },
  {
   "cell_type": "code",
   "id": "c03-009f7298",
   "metadata": {},
   "source": [
    "from confluent_kafka import Producer\n",
    "from confluent_kafka.admin import AdminClient\n",
    "\n",
    "TOPIC = \"m07l03-telemetry\"\n",
    "admin = AdminClient({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP})\n",
    "readings = [(key, e) for key, e in Brightlane().stream(sessions=300)\n",
    "            if \"type\" not in e and parse_iso(e[\"ts\"]) < BASE_TIME + timedelta(hours=1)]\n",
    "outage = (BASE_TIME + timedelta(minutes=35), BASE_TIME + timedelta(minutes=40))     # 18:05 to 18:10\n",
    "arrivals = Brightlane.delay(readings, \"site-N03\", *outage, delay_s=300)\n",
    "\n",
    "def write(topic, rows):\n",
    "    lab.recreate_topic(admin, topic, partitions=1)\n",
    "    producer = Producer(settings.PRODUCER_DEFAULTS)\n",
    "    for arrived, key, reading in rows:\n",
    "        producer.produce(topic, key=key, value=to_json(reading), timestamp=int(arrived.timestamp() * 1000))\n",
    "    producer.flush(30)\n",
    "\n",
    "write(TOPIC, arrivals)\n",
    "late = [(a, e) for a, _, e in arrivals if a != parse_iso(e[\"ts\"])]\n",
    "print(f\"{len(arrivals)} readings; {len(late)} from site-N03 arrive late, for example:\")\n",
    "print(f\"  measured {late[0][1]['ts'][11:19]}, arrived {iso(late[0][0])[11:19]}\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c04-ca699c01",
   "metadata": {},
   "source": [
    "## Step 2: Windows by arrival time\n",
    "`report()` runs the grid report script and prints its summary: totals, then site-N03 window by\n",
    "window from 17:55 to 18:25."
   ]
  },
  {
   "cell_type": "code",
   "id": "c05-8406e5d9",
   "metadata": {},
   "source": [
    "def report(topic, *options):\n",
    "    result = subprocess.run([sys.executable, \"module-07/grid_report.py\", \"--topic\", topic, *options],\n",
    "                            cwd=lab_root, capture_output=True, text=True)\n",
    "    print(result.stdout.strip() or result.stderr[-800:])\n",
    "\n",
    "report(TOPIC, \"--time\", \"arrival\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c06-c487f6c7",
   "metadata": {},
   "source": [
    "Between 18:05 and 18:10, site-N03 seems to use no energy at all, and twice the usual amount in\n",
    "the next window.\n",
    "\n",
    "## Step 3: Windows by event time, with a 2-minute grace period\n",
    "Now each reading counts in the window of its own `ts`. A window is reported once the stream time,\n",
    "the latest event time seen, has passed its end by the grace period."
   ]
  },
  {
   "cell_type": "code",
   "id": "c07-862fc4bf",
   "metadata": {},
   "source": [
    "report(TOPIC, \"--time\", \"event\", \"--grace\", \"120\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c08-990fe5ee",
   "metadata": {},
   "source": [
    "The window 18:05–18:10 has part of its readings. The rest came after it was reported: they are\n",
    "**late**, counted but not added.\n",
    "\n",
    "## Change it: a 6-minute grace period"
   ]
  },
  {
   "cell_type": "code",
   "id": "c09-1f86a04a",
   "metadata": {},
   "source": [
    "report(TOPIC, \"--time\", \"event\", \"--grace\", \"360\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c10-6bc8acb9",
   "metadata": {},
   "source": [
    "Every window is complete, but every window is also reported 4 minutes later than before.\n",
    "\n",
    "## Break it: a clock three hours ahead\n",
    "One reading from a charger whose clock jumped three hours ahead arrives at 18:07. Predict first:\n",
    "what happens to the windows after it?"
   ]
  },
  {
   "cell_type": "code",
   "id": "c11-76488d6f",
   "metadata": {},
   "source": [
    "middle = len(arrivals) // 2\n",
    "arrived, key, reading = arrivals[middle]\n",
    "wrong_clock = (arrived, key, {**reading, \"ts\": iso(arrived + timedelta(hours=3))})\n",
    "write(f\"{TOPIC}-clock\", arrivals[:middle] + [wrong_clock] + arrivals[middle:])\n",
    "print(\"wrong clock at\", iso(arrived)[11:19], \"says\", wrong_clock[2][\"ts\"][11:19])\n",
    "report(f\"{TOPIC}-clock\", \"--time\", \"event\", \"--grace\", \"120\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c12-49cc6588",
   "metadata": {},
   "source": [
    "One reading pushed the stream time three hours ahead, every open window closed, and almost every\n",
    "reading after it was late. The guard: ignore event times that are far ahead of the arrival time."
   ]
  },
  {
   "cell_type": "code",
   "id": "c13-495ec432",
   "metadata": {},
   "source": [
    "report(f\"{TOPIC}-clock\", \"--time\", \"event\", \"--grace\", \"120\", \"--max-ahead\", \"300\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c14-76d8a65e",
   "metadata": {},
   "source": [
    "## Clean up\n",
    "Delete the topics. If anything gets stuck, `docker compose down -v` in the `code/` folder resets\n",
    "the whole lab."
   ]
  },
  {
   "cell_type": "code",
   "id": "c15-c2fa4da7",
   "metadata": {},
   "source": [
    "lab.delete_topics(admin, [t for t in admin.list_topics(timeout=10).topics if t.startswith(TOPIC)])\n",
    "admin.delete_consumer_groups([\"m07l03-report\"])\n",
    "print(\"cleaned up\")"
   ],
   "execution_count": null,
   "outputs": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
