{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c00-7639f522",
   "metadata": {},
   "source": [
    "# Records, offsets and replay — practical\n",
    "\n",
    "You look inside a single record, write an evening of charger events, and then answer a\n",
    "driver's dispute by reading one charger's events from a chosen point in time, twice.\n",
    "\n",
    "**Lab services:** kafka-1, kafka-2, kafka-3 · **Lesson:** `modules/01-first-events/03-records-offsets-and-replay.md`"
   ]
  },
  {
   "cell_type": "code",
   "id": "c01-84c8e9b7",
   "metadata": {},
   "source": [
    "# Setup: find the lab's shared helpers and check the lab is running\n",
    "import sys, pathlib\n",
    "lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / \"common\").is_dir())\n",
    "sys.path.insert(0, str(lab_root))\n",
    "\n",
    "from common import settings, lab\n",
    "from datetime import timedelta\n",
    "from common.scenario import Brightlane, BASE_TIME, iso, to_json, from_json, parse_iso\n",
    "\n",
    "lab.require_kafka()\n",
    "lab.print_versions()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c02-ec41ef31",
   "metadata": {},
   "source": [
    "## Step 1: Write one record and look at all its parts\n",
    "A record has a key, a value, optional headers and a timestamp. Kafka adds the partition and\n",
    "the offset when it stores the record."
   ]
  },
  {
   "cell_type": "code",
   "id": "c03-546af1e5",
   "metadata": {},
   "source": [
    "from confluent_kafka import Producer, Consumer, TopicPartition\n",
    "from confluent_kafka.admin import AdminClient, NewTopic\n",
    "\n",
    "ANATOMY, TOPIC = \"m01l03-anatomy\", \"m01l03-telemetry\"\n",
    "admin = AdminClient({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP})\n",
    "lab.delete_topics(admin, [ANATOMY, TOPIC])\n",
    "for name in (ANATOMY, TOPIC):\n",
    "    admin.create_topics([NewTopic(name, num_partitions=3, replication_factor=3)])[name].result()\n",
    "\n",
    "producer = Producer({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP})\n",
    "producer.produce(ANATOMY, key=\"BL-N03-07\",\n",
    "                 value=to_json({\"charger_id\": \"BL-N03-07\", \"power_kw\": 0.0, \"ts\": iso(BASE_TIME)}),\n",
    "                 headers={\"firmware\": \"4.2.1\", \"trace-id\": \"a91f3c\"})\n",
    "print(\"not yet stored:\", producer.flush(10))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c04-f15d385c",
   "metadata": {},
   "source": [
    "Read it back with a consumer that is assigned to partitions directly: no group membership,\n",
    "no commits, a plain reader. It reads up to each partition's current end and stops."
   ]
  },
  {
   "cell_type": "code",
   "id": "c05-cd97e728",
   "metadata": {},
   "source": [
    "def read_all(tps, max_wait=30.0):\n",
    "    \"\"\"Read the given partitions from their start offsets up to their current end.\"\"\"\n",
    "    import time\n",
    "    c = Consumer({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP, \"group.id\": \"m01l03-reader\",\n",
    "                  \"enable.auto.commit\": False})\n",
    "    ends = {tp.partition: c.get_watermark_offsets(tp, timeout=10)[1] for tp in tps}\n",
    "    todo = {tp.partition for tp in tps if ends[tp.partition] > tp.offset}\n",
    "    c.assign(tps)\n",
    "    out, deadline = [], time.monotonic() + max_wait\n",
    "    while todo and time.monotonic() < deadline:\n",
    "        for msg in c.consume(num_messages=1000, timeout=0.5):\n",
    "            if msg.error():\n",
    "                continue\n",
    "            out.append(msg)\n",
    "            if msg.offset() >= ends[msg.partition()] - 1:\n",
    "                todo.discard(msg.partition())\n",
    "    c.close()\n",
    "    return out\n",
    "\n",
    "msg = read_all([TopicPartition(ANATOMY, p, 0) for p in range(3)])[0]   # offset 0 = from the start\n",
    "ts_type, ts_ms = msg.timestamp()\n",
    "print(\"topic     \", msg.topic(), \"| partition\", msg.partition(), \"| offset\", msg.offset())\n",
    "print(\"key       \", msg.key())\n",
    "print(\"value     \", msg.value())\n",
    "print(\"headers   \", msg.headers())\n",
    "print(\"timestamp \", ts_ms, \"(type 1 = CreateTime, set by the producer)\" if ts_type == 1 else ts_type)"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c06-120565a6",
   "metadata": {},
   "source": [
    "You should see the partition and offset Kafka chose, the key and value as **bytes**, the two\n",
    "headers, and a timestamp in milliseconds since 1970 (now, because we didn't set one).\n",
    "\n",
    "## Step 2: Write an evening of meter readings\n",
    "The gateway relays every reading with the charger's own clock as the record timestamp.\n",
    "That's 40 sessions and their 10-second readings: several thousand records."
   ]
  },
  {
   "cell_type": "code",
   "id": "c07-bba5f9ad",
   "metadata": {},
   "source": [
    "bl = Brightlane()\n",
    "events = bl.stream(sessions=40)\n",
    "for charger_id, event in events:\n",
    "    ts_ms = int(parse_iso(event[\"ts\"]).timestamp() * 1000)\n",
    "    producer.produce(TOPIC, key=charger_id, value=to_json(event), timestamp=ts_ms)\n",
    "    producer.poll(0)                                  # serve delivery events while producing\n",
    "print(f\"wrote {len(events)} records, not yet stored: {producer.flush(30)}\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c08-534bc3b1",
   "metadata": {},
   "source": [
    "You should see `wrote 9123 records, not yet stored: 0`.\n",
    "\n",
    "## Step 3: Find the disputed charger's partition\n",
    "Records with the same key always go to the same partition, so one partition holds all of\n",
    "BL-N03-07's events. The partition's **offsets** run from 0 to its end."
   ]
  },
  {
   "cell_type": "code",
   "id": "c09-1ed3a1ce",
   "metadata": {},
   "source": [
    "everything = [TopicPartition(TOPIC, p, 0) for p in range(3)]\n",
    "records = read_all(everything)\n",
    "mine = [m for m in records if m.key() == b\"BL-N03-07\"]\n",
    "partition = mine[0].partition()\n",
    "low, high = Consumer({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP, \"group.id\": \"m01l03-reader\"}) \\\n",
    "    .get_watermark_offsets(TopicPartition(TOPIC, partition), timeout=10)\n",
    "print(f\"BL-N03-07 -> partition {partition}: {len(mine)} records\")\n",
    "print(f\"partition {partition} holds offsets {low} to {high - 1} ({high - low} records in total)\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c10-cd687c9d",
   "metadata": {},
   "source": [
    "You should see all of BL-N03-07's records in one partition, mixed with other chargers' records.\n",
    "The exact partition and counts are in the lesson.\n",
    "\n",
    "## Step 4: Jump to 17:40 by time\n",
    "Support only needs events from 17:40. `offsets_for_times` asks the broker for the first offset\n",
    "whose timestamp is at or after a given time."
   ]
  },
  {
   "cell_type": "code",
   "id": "c11-040d0c67",
   "metadata": {},
   "source": [
    "c = Consumer({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP, \"group.id\": \"m01l03-reader\"})\n",
    "at_1740 = int((BASE_TIME + timedelta(minutes=10)).timestamp() * 1000)   # BASE_TIME is 17:30 yesterday\n",
    "found = c.offsets_for_times([TopicPartition(TOPIC, partition, at_1740)], timeout=10)[0]\n",
    "c.close()\n",
    "print(f\"first offset at or after 17:40 in partition {partition}: {found.offset}\")\n",
    "\n",
    "since_1740 = [m for m in read_all([TopicPartition(TOPIC, partition, found.offset)]) if m.key() == b\"BL-N03-07\"]\n",
    "for m in since_1740[:3]:\n",
    "    e = from_json(m.value())\n",
    "    print(m.offset(), e[\"ts\"], e.get(\"type\", \"reading\"), e.get(\"energy_kwh\"))\n",
    "print(f\"... {len(since_1740)} records for BL-N03-07 since 17:40\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c12-267368b7",
   "metadata": {},
   "source": [
    "You should see the first three BL-N03-07 records after 17:40 and the total. The session started\n",
    "at 17:38:25, so the reading at 17:40:05 already shows about 2 kWh.\n",
    "\n",
    "## Step 5: Replay: read the same range again\n",
    "Reading is not destructive. Ask for the same offset again and you get the same records,\n",
    "with the same offsets, in the same order."
   ]
  },
  {
   "cell_type": "code",
   "id": "c13-ac00bc7e",
   "metadata": {},
   "source": [
    "again = [m for m in read_all([TopicPartition(TOPIC, partition, found.offset)]) if m.key() == b\"BL-N03-07\"]\n",
    "same = [(m.offset(), m.value()) for m in again] == [(m.offset(), m.value()) for m in since_1740]\n",
    "print(f\"second read: {len(again)} records, identical to the first: {same}\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c14-6375d510",
   "metadata": {},
   "source": [
    "You should see `identical to the first: True`.\n",
    "\n",
    "## Change it: a reading that arrives late\n",
    "Predict first: a 17:35 reading from BL-N03-07 reaches the gateway only now (the charger's\n",
    "mobile link was down). It is appended at the end of the partition. Where does it sit, and\n",
    "does a time search for 17:30 find it?"
   ]
  },
  {
   "cell_type": "code",
   "id": "c15-b2709ef3",
   "metadata": {},
   "source": [
    "late = {\"charger_id\": \"BL-N03-07\", \"session_id\": \"ses-test\", \"ts\": iso(BASE_TIME + timedelta(minutes=5)),\n",
    "        \"power_kw\": 7.0, \"energy_kwh\": 0.019, \"firmware\": \"4.2.1\"}\n",
    "producer.produce(TOPIC, key=\"BL-N03-07\", value=to_json(late),\n",
    "                 timestamp=int(parse_iso(late[\"ts\"]).timestamp() * 1000))\n",
    "producer.flush(10)\n",
    "tail = read_all([TopicPartition(TOPIC, partition, high)])\n",
    "print(\"late record stored at offset\", tail[0].offset(), \"with timestamp\", from_json(tail[0].value())[\"ts\"])\n",
    "\n",
    "c = Consumer({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP, \"group.id\": \"m01l03-reader\"})\n",
    "at_1730 = int(BASE_TIME.timestamp() * 1000)\n",
    "print(\"time search for 17:30 starts at offset\",\n",
    "      c.offsets_for_times([TopicPartition(TOPIC, partition, at_1730)], timeout=10)[0].offset)\n",
    "c.close()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c16-c65ccacd",
   "metadata": {},
   "source": [
    "The late record gets the next offset at the end of the log, even though its timestamp is\n",
    "earlier than its neighbours'. The time search still starts near the beginning: it returns the\n",
    "**first** offset with a timestamp at or after 17:30, and reading forward from there reaches the\n",
    "late record too. Offsets give order of arrival; timestamps can go backwards.\n",
    "\n",
    "## Clean up\n",
    "Delete the topic. If anything gets stuck, `docker compose down -v` in the `code/` folder resets\n",
    "the whole lab."
   ]
  },
  {
   "cell_type": "code",
   "id": "c17-4a74b687",
   "metadata": {},
   "source": [
    "lab.delete_topics(admin, [ANATOMY, TOPIC])\n",
    "print(\"cleaned up\")"
   ],
   "execution_count": null,
   "outputs": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
