{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c00-e9a26ffd",
   "metadata": {},
   "source": [
    "# Evolving a schema without breaking readers — practical\n",
    "\n",
    "The grid team wants a new field in meter readings, while billing still runs its old code. You\n",
    "add the field in a compatible way, read old and new readings with old and new readers, have a\n",
    "breaking change refused by the registry, and see what happens when the check is switched off.\n",
    "\n",
    "**Lab services:** kafka-1, kafka-2, kafka-3, schema-registry (`docker compose --profile schemas up -d --wait`)\n",
    "· **Lesson:** `modules/05-data-contracts/02-evolving-a-schema.md`"
   ]
  },
  {
   "cell_type": "code",
   "id": "c01-8415e75a",
   "metadata": {},
   "source": [
    "# Setup: find the lab's shared helpers and check the lab is running\n",
    "import sys, pathlib\n",
    "lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / \"common\").is_dir())\n",
    "sys.path.insert(0, str(lab_root))\n",
    "\n",
    "import json\n",
    "from common import settings, lab\n",
    "from common.scenario import Brightlane, parse_iso\n",
    "\n",
    "lab.require_kafka()\n",
    "lab.require_http(\"Schema Registry\", settings.SCHEMA_REGISTRY_URL + \"/subjects\",\n",
    "                 start_hint=\"docker compose --profile schemas up -d --wait\")\n",
    "lab.print_versions()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c02-f0c47b3d",
   "metadata": {},
   "source": [
    "## Step 1: Version 1, and 50 readings written with it"
   ]
  },
  {
   "cell_type": "code",
   "id": "c03-2575ab37",
   "metadata": {},
   "source": [
    "from confluent_kafka import Producer, Consumer, TopicPartition\n",
    "from confluent_kafka.admin import AdminClient\n",
    "from confluent_kafka.schema_registry import SchemaRegistryClient, Schema\n",
    "from confluent_kafka.schema_registry.avro import AvroSerializer, AvroDeserializer\n",
    "from confluent_kafka.serialization import SerializationContext, MessageField\n",
    "\n",
    "TOPIC = \"m05l02-telemetry\"\n",
    "SUBJECT = f\"{TOPIC}-value\"\n",
    "admin = AdminClient({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP})\n",
    "lab.recreate_topic(admin, TOPIC, partitions=1)\n",
    "registry = SchemaRegistryClient({\"url\": settings.SCHEMA_REGISTRY_URL})\n",
    "if SUBJECT in registry.get_subjects():\n",
    "    registry.delete_subject(SUBJECT)\n",
    "    registry.delete_subject(SUBJECT, permanent=True)\n",
    "try:\n",
    "    registry.delete_config(SUBJECT)          # forget a compatibility rule left by an earlier run\n",
    "except Exception:\n",
    "    pass\n",
    "\n",
    "V1 = (lab_root / \"schemas\" / \"meter_reading.avsc\").read_text()\n",
    "context = SerializationContext(TOPIC, MessageField.VALUE)\n",
    "readings = [e for _, e in Brightlane().stream(sessions=10) if \"type\" not in e][:150]\n",
    "producer = Producer(settings.PRODUCER_DEFAULTS)\n",
    "\n",
    "def write(schema_str, batch, **extra):\n",
    "    \"\"\"Write readings with a given writer schema, registering it if needed.\"\"\"\n",
    "    serialize = AvroSerializer(registry, schema_str, lambda r, ctx: {**r, \"ts\": parse_iso(r[\"ts\"]), **extra})\n",
    "    for reading in batch:\n",
    "        producer.produce(TOPIC, key=reading[\"charger_id\"], value=serialize(reading, context))\n",
    "    producer.flush(10)\n",
    "\n",
    "write(V1, readings[:50])\n",
    "print(\"versions:\", registry.get_versions(SUBJECT), \" compatibility:\", registry.get_compatibility())"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c04-509a30cf",
   "metadata": {},
   "source": [
    "The registry's default rule is `BACKWARD`: a new version must be able to read data written with\n",
    "the previous one.\n",
    "\n",
    "## Step 2: Version 2 adds an optional field\n",
    "The grid team wants `phase_count`: how many phases the charger uses. Old readings don't have it,\n",
    "so the field may be empty (`null`) and its default is `null`."
   ]
  },
  {
   "cell_type": "code",
   "id": "c05-43b813c4",
   "metadata": {},
   "source": [
    "def changed(schema_str, edit):\n",
    "    schema = json.loads(schema_str)\n",
    "    edit(schema[\"fields\"])\n",
    "    return json.dumps(schema)\n",
    "\n",
    "V2 = changed(V1, lambda fields: fields.append({\"name\": \"phase_count\", \"type\": [\"null\", \"int\"], \"default\": None}))\n",
    "print(\"compatible with version 1:\", registry.test_compatibility(SUBJECT, Schema(V2, \"AVRO\")))\n",
    "write(V2, readings[50:100], phase_count=3)\n",
    "print(\"versions:\", registry.get_versions(SUBJECT))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c06-30573df4",
   "metadata": {},
   "source": [
    "## Step 3: Old and new readers\n",
    "Billing still reads with version 1, the grid report already with version 2. Giving the\n",
    "deserializer a schema makes it the **reader schema**: Avro translates each record from the schema\n",
    "it was written with into that one."
   ]
  },
  {
   "cell_type": "code",
   "id": "c07-ee57c6b7",
   "metadata": {},
   "source": [
    "def read_all(reader_schema=None):\n",
    "    deserialize = AvroDeserializer(registry, reader_schema)\n",
    "    consumer = Consumer({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP, \"group.id\": \"m05l02-reader\",\n",
    "                         \"enable.auto.commit\": False})\n",
    "    end = consumer.get_watermark_offsets(TopicPartition(TOPIC, 0), timeout=10)[1]\n",
    "    consumer.assign([TopicPartition(TOPIC, 0, 0)])\n",
    "    out = []\n",
    "    while len(out) < end:\n",
    "        msg = consumer.poll(10)\n",
    "        try:\n",
    "            out.append(deserialize(msg.value(), context))\n",
    "        except Exception as e:\n",
    "            out.append(f\"offset {msg.offset()}: {type(e).__name__}: {e}\")\n",
    "    consumer.close()\n",
    "    return out\n",
    "\n",
    "billing, grid = read_all(V1), read_all(V2)\n",
    "print(\"billing (v1) reads\", len(billing), \"readings; fields of the newest:\", sorted(billing[-1]))\n",
    "print(\"grid (v2) on an old reading: phase_count =\", grid[0][\"phase_count\"])\n",
    "print(\"grid (v2) on a new reading:  phase_count =\", grid[-1][\"phase_count\"])"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c08-6e900f13",
   "metadata": {},
   "source": [
    "## Step 4: Version 3 renames a field\n",
    "The gateway team wants `energy_kwh` called `energy_wh` (in watt-hours). Predict first: will the\n",
    "registry accept it?"
   ]
  },
  {
   "cell_type": "code",
   "id": "c09-5e5d6fe3",
   "metadata": {},
   "source": [
    "V3 = changed(V2, lambda fields: [f.update(name=\"energy_wh\") for f in fields if f[\"name\"] == \"energy_kwh\"])\n",
    "print(\"compatible with version 2:\", registry.test_compatibility(SUBJECT, Schema(V3, \"AVRO\")))\n",
    "try:\n",
    "    registry.register_schema(SUBJECT, Schema(V3, \"AVRO\"))\n",
    "except Exception as e:\n",
    "    print(f\"register: HTTP {e.http_status_code}: {str(e)[:230]}\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c10-b7aaa36f",
   "metadata": {},
   "source": [
    "Refused with HTTP 409: a version 3 reader couldn't read the existing readings, because they have\n",
    "no `energy_wh` and the new field has no default. Nothing reached the topic.\n",
    "\n",
    "## Break it: switch the check off\n",
    "Someone sets this subject's compatibility to `NONE` \"just for this release\", registers version\n",
    "3 and writes 50 readings. Predict first: what does the grid report, still on version 2, see?"
   ]
  },
  {
   "cell_type": "code",
   "id": "c11-c5022897",
   "metadata": {},
   "source": [
    "registry.set_compatibility(SUBJECT, \"NONE\")\n",
    "write(V3, readings[100:150], phase_count=3, energy_wh=500.0)\n",
    "grid = read_all(V2)\n",
    "print(\"readable:\", sum(isinstance(r, dict) for r in grid), \" failed:\", sum(isinstance(r, str) for r in grid))\n",
    "print(next(r for r in grid if isinstance(r, str)))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c12-753012a5",
   "metadata": {},
   "source": [
    "Version 2 readers fail on every version 3 reading. Those readings are in the topic now; turning\n",
    "the check back on doesn't remove them.\n",
    "\n",
    "## Clean up\n",
    "Delete the topic and the subject, so the check is back to the default next time. If anything gets\n",
    "stuck, `docker compose down -v` in the `code/` folder resets the whole lab."
   ]
  },
  {
   "cell_type": "code",
   "id": "c13-77caf633",
   "metadata": {},
   "source": [
    "lab.delete_topics(admin, [TOPIC])\n",
    "registry.delete_config(SUBJECT)\n",
    "registry.delete_subject(SUBJECT)\n",
    "registry.delete_subject(SUBJECT, permanent=True)\n",
    "admin.delete_consumer_groups([\"m05l02-reader\"])\n",
    "print(\"cleaned up\")"
   ],
   "execution_count": null,
   "outputs": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
