{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c00-4707ecfe",
   "metadata": {},
   "source": [
    "# Schemas and the Schema Registry — practical\n",
    "\n",
    "Meter readings get an Avro schema. You register it, write readings with it, look at the bytes\n",
    "that end up in Kafka, and read them back. Then a producer sends a reading with a renamed field,\n",
    "and another one sends plain JSON into the topic.\n",
    "\n",
    "**Lab services:** kafka-1, kafka-2, kafka-3, schema-registry (`docker compose --profile schemas up -d --wait`)\n",
    "· **Lesson:** `modules/05-data-contracts/01-schemas-and-the-schema-registry.md`"
   ]
  },
  {
   "cell_type": "code",
   "id": "c01-15c35515",
   "metadata": {},
   "source": [
    "# Setup: find the lab's shared helpers and check the lab is running\n",
    "import sys, pathlib\n",
    "lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / \"common\").is_dir())\n",
    "sys.path.insert(0, str(lab_root))\n",
    "\n",
    "import json\n",
    "from common import settings, lab\n",
    "from common.scenario import Brightlane, parse_iso, to_json\n",
    "\n",
    "lab.require_kafka()\n",
    "lab.require_http(\"Schema Registry\", settings.SCHEMA_REGISTRY_URL + \"/subjects\",\n",
    "                 start_hint=\"docker compose --profile schemas up -d --wait\")\n",
    "lab.print_versions()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c02-0437b0f8",
   "metadata": {},
   "source": [
    "## Step 1: The schema\n",
    "The schema lives in the repository, next to the code: `code/schemas/meter_reading.avsc`."
   ]
  },
  {
   "cell_type": "code",
   "id": "c03-45d3e805",
   "metadata": {},
   "source": [
    "schema_str = (lab_root / \"schemas\" / \"meter_reading.avsc\").read_text()\n",
    "for field in json.loads(schema_str)[\"fields\"]:\n",
    "    print(f\"{field['name']:<11} {json.dumps(field['type']):<50} {field['doc']}\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c04-e0510f4a",
   "metadata": {},
   "source": [
    "## Step 2: Write readings with the schema\n",
    "`AvroSerializer` turns a dict into Avro bytes. The first time, it registers the schema under\n",
    "the subject `<topic>-value` and gets the schema's id back. `to_avro` converts the reading's ISO\n",
    "time text into a `datetime`, which the `timestamp-millis` field expects."
   ]
  },
  {
   "cell_type": "code",
   "id": "c05-1d4b4904",
   "metadata": {},
   "source": [
    "from confluent_kafka import Producer\n",
    "from confluent_kafka.admin import AdminClient\n",
    "from confluent_kafka.schema_registry import SchemaRegistryClient\n",
    "from confluent_kafka.schema_registry.avro import AvroSerializer\n",
    "from confluent_kafka.serialization import SerializationContext, MessageField\n",
    "\n",
    "TOPIC = \"m05l01-telemetry\"\n",
    "admin = AdminClient({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP})\n",
    "lab.recreate_topic(admin, TOPIC, partitions=3)\n",
    "registry = SchemaRegistryClient({\"url\": settings.SCHEMA_REGISTRY_URL})\n",
    "for subject in [s for s in registry.get_subjects() if s.startswith(TOPIC)]:\n",
    "    registry.delete_subject(subject)\n",
    "    registry.delete_subject(subject, permanent=True)\n",
    "\n",
    "to_avro = lambda reading, ctx: {**reading, \"ts\": parse_iso(reading[\"ts\"])}\n",
    "serialize = AvroSerializer(registry, schema_str, to_avro)\n",
    "context = SerializationContext(TOPIC, MessageField.VALUE)\n",
    "\n",
    "readings = [e for _, e in Brightlane().stream(sessions=10) if \"type\" not in e][:200]\n",
    "producer = Producer(settings.PRODUCER_DEFAULTS)\n",
    "for reading in readings:\n",
    "    producer.produce(TOPIC, key=reading[\"charger_id\"], value=serialize(reading, context))\n",
    "producer.flush(10)\n",
    "\n",
    "latest = registry.get_latest_version(f\"{TOPIC}-value\")\n",
    "print(\"subjects:\", [s for s in registry.get_subjects() if s.startswith(\"m05l01\")])\n",
    "print(f\"subject {latest.subject}: version {latest.version}, schema id {latest.schema_id}\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c06-6f5d3027",
   "metadata": {},
   "source": [
    "## Step 3: The bytes in Kafka\n",
    "A plain consumer, without a deserializer, sees the raw bytes of the first record."
   ]
  },
  {
   "cell_type": "code",
   "id": "c07-aff9ecdc",
   "metadata": {},
   "source": [
    "from confluent_kafka import Consumer, TopicPartition\n",
    "\n",
    "raw_reader = Consumer({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP, \"group.id\": \"m05l01-raw\",\n",
    "                       \"enable.auto.commit\": False})\n",
    "raw_reader.assign([TopicPartition(TOPIC, p, 0) for p in range(3)])\n",
    "msg = raw_reader.poll(10)\n",
    "raw = msg.value()\n",
    "print(\"first 5 bytes:\", raw[:5].hex(\" \"))\n",
    "print(\"magic byte:\", raw[0], \"  schema id:\", int.from_bytes(raw[1:5], \"big\"))\n",
    "print(\"the rest:\", raw[5:40], \"...\")\n",
    "avro_bytes = sum(len(serialize(r, context)) for r in readings)\n",
    "json_bytes = sum(len(to_json(r)) for r in readings)\n",
    "print(f\"all 200 readings as Avro: {avro_bytes:,} bytes; as JSON: {json_bytes:,} bytes\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c08-ebe3718b",
   "metadata": {},
   "source": [
    "## Step 4: Read them back\n",
    "`AvroDeserializer` reads the schema id from each record, fetches that schema from the registry\n",
    "once (then caches it), and decodes the rest."
   ]
  },
  {
   "cell_type": "code",
   "id": "c09-ff150c8b",
   "metadata": {},
   "source": [
    "from confluent_kafka.schema_registry.avro import AvroDeserializer\n",
    "\n",
    "deserialize = AvroDeserializer(registry)\n",
    "print(deserialize(raw, context))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c10-a27fe307",
   "metadata": {},
   "source": [
    "## Break it: a renamed field, and a producer without a schema\n",
    "The gateway's next release sends power in watts, as `power_w`. Predict first: what happens when\n",
    "it serializes such a reading with the registered schema?"
   ]
  },
  {
   "cell_type": "code",
   "id": "c11-a06ca3bc",
   "metadata": {},
   "source": [
    "renamed = {k: v for k, v in readings[0].items() if k != \"power_kw\"} | {\"power_w\": 21200}\n",
    "try:\n",
    "    serialize(renamed, context)\n",
    "except Exception as e:\n",
    "    print(f\"producer side: {type(e).__name__}: {str(e)[:120]}\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c12-39a6bc84",
   "metadata": {},
   "source": [
    "The producer can't even create the record: the mistake shows up at the source, not as zeros in\n",
    "a report. Now a quick script writes one reading as plain JSON into the same topic, and a reader\n",
    "with the Avro deserializer meets it:"
   ]
  },
  {
   "cell_type": "code",
   "id": "c13-b7ebd5d1",
   "metadata": {},
   "source": [
    "producer.produce(TOPIC, key=\"BL-N03-04\", value=to_json(readings[1]), partition=0)\n",
    "producer.flush(10)\n",
    "end = raw_reader.get_watermark_offsets(TopicPartition(TOPIC, 0), timeout=10)[1]\n",
    "raw_reader.assign([TopicPartition(TOPIC, 0, end - 1)])\n",
    "msg = raw_reader.poll(10)\n",
    "try:\n",
    "    deserialize(msg.value(), context)\n",
    "except Exception as e:\n",
    "    print(f\"consumer side, offset {msg.offset()}: {type(e).__name__}: {e}\")\n",
    "raw_reader.close()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c14-9af98362",
   "metadata": {},
   "source": [
    "The registry can't stop someone from writing other bytes into a topic: Kafka accepts any bytes.\n",
    "The reader finds out, record by record. Lesson 5.3 shows how to handle such a record.\n",
    "\n",
    "## Clean up\n",
    "Delete the topic and the subject. If anything gets stuck, `docker compose down -v` in the `code/`\n",
    "folder resets the whole lab."
   ]
  },
  {
   "cell_type": "code",
   "id": "c15-58bfbdc8",
   "metadata": {},
   "source": [
    "lab.delete_topics(admin, [TOPIC])\n",
    "registry.delete_subject(f\"{TOPIC}-value\")\n",
    "registry.delete_subject(f\"{TOPIC}-value\", permanent=True)\n",
    "admin.delete_consumer_groups([\"m05l01-raw\"])\n",
    "print(\"cleaned up\")"
   ],
   "execution_count": null,
   "outputs": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
