Download 01-schemas-and-the-schema-registry.ipynb
Open it in JupyterLab or VS Code from the lab folder to run it. This page is a read-only view.
Schemas and the Schema Registry — practical
Meter readings get an Avro schema. You register it, write readings with it, look at the bytes that end up in Kafka, and read them back. Then a producer sends a reading with a renamed field, and another one sends plain JSON into the topic.
Lab services: kafka-1, kafka-2, kafka-3, schema-registry (docker compose --profile schemas up -d --wait)
· Lesson: modules/05-data-contracts/01-schemas-and-the-schema-registry.md
# Setup: find the lab's shared helpers and check the lab is running import sys, pathlib lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / "common").is_dir()) sys.path.insert(0, str(lab_root)) import json from common import settings, lab from common.scenario import Brightlane, parse_iso, to_json lab.require_kafka() lab.require_http("Schema Registry", settings.SCHEMA_REGISTRY_URL + "/subjects", start_hint="docker compose --profile schemas up -d --wait") lab.print_versions()
Step 1: The schema
The schema lives in the repository, next to the code: code/schemas/meter_reading.avsc.
schema_str = (lab_root / "schemas" / "meter_reading.avsc").read_text() for field in json.loads(schema_str)["fields"]: print(f"{field['name']:<11} {json.dumps(field['type']):<50} {field['doc']}")
Step 2: Write readings with the schema
AvroSerializer turns a dict into Avro bytes. The first time, it registers the schema under
the subject <topic>-value and gets the schema's id back. to_avro converts the reading's ISO
time text into a datetime, which the timestamp-millis field expects.
from confluent_kafka import Producer from confluent_kafka.admin import AdminClient from confluent_kafka.schema_registry import SchemaRegistryClient from confluent_kafka.schema_registry.avro import AvroSerializer from confluent_kafka.serialization import SerializationContext, MessageField TOPIC = "m05l01-telemetry" admin = AdminClient({"bootstrap.servers": settings.KAFKA_BOOTSTRAP}) lab.recreate_topic(admin, TOPIC, partitions=3) registry = SchemaRegistryClient({"url": settings.SCHEMA_REGISTRY_URL}) for subject in [s for s in registry.get_subjects() if s.startswith(TOPIC)]: registry.delete_subject(subject) registry.delete_subject(subject, permanent=True) to_avro = lambda reading, ctx: {**reading, "ts": parse_iso(reading["ts"])} serialize = AvroSerializer(registry, schema_str, to_avro) context = SerializationContext(TOPIC, MessageField.VALUE) readings = [e for _, e in Brightlane().stream(sessions=10) if "type" not in e][:200] producer = Producer(settings.PRODUCER_DEFAULTS) for reading in readings: producer.produce(TOPIC, key=reading["charger_id"], value=serialize(reading, context)) producer.flush(10) latest = registry.get_latest_version(f"{TOPIC}-value") print("subjects:", [s for s in registry.get_subjects() if s.startswith("m05l01")]) print(f"subject {latest.subject}: version {latest.version}, schema id {latest.schema_id}")
Step 3: The bytes in Kafka
A plain consumer, without a deserializer, sees the raw bytes of the first record.
from confluent_kafka import Consumer, TopicPartition raw_reader = Consumer({"bootstrap.servers": settings.KAFKA_BOOTSTRAP, "group.id": "m05l01-raw", "enable.auto.commit": False}) raw_reader.assign([TopicPartition(TOPIC, p, 0) for p in range(3)]) msg = raw_reader.poll(10) raw = msg.value() print("first 5 bytes:", raw[:5].hex(" ")) print("magic byte:", raw[0], " schema id:", int.from_bytes(raw[1:5], "big")) print("the rest:", raw[5:40], "...") avro_bytes = sum(len(serialize(r, context)) for r in readings) json_bytes = sum(len(to_json(r)) for r in readings) print(f"all 200 readings as Avro: {avro_bytes:,} bytes; as JSON: {json_bytes:,} bytes")
Step 4: Read them back
AvroDeserializer reads the schema id from each record, fetches that schema from the registry
once (then caches it), and decodes the rest.
from confluent_kafka.schema_registry.avro import AvroDeserializer deserialize = AvroDeserializer(registry) print(deserialize(raw, context))
Break it: a renamed field, and a producer without a schema
The gateway's next release sends power in watts, as power_w. Predict first: what happens when
it serializes such a reading with the registered schema?
renamed = {k: v for k, v in readings[0].items() if k != "power_kw"} | {"power_w": 21200} try: serialize(renamed, context) except Exception as e: print(f"producer side: {type(e).__name__}: {str(e)[:120]}")
The producer can't even create the record: the mistake shows up at the source, not as zeros in a report. Now a quick script writes one reading as plain JSON into the same topic, and a reader with the Avro deserializer meets it:
producer.produce(TOPIC, key="BL-N03-04", value=to_json(readings[1]), partition=0) producer.flush(10) end = raw_reader.get_watermark_offsets(TopicPartition(TOPIC, 0), timeout=10)[1] raw_reader.assign([TopicPartition(TOPIC, 0, end - 1)]) msg = raw_reader.poll(10) try: deserialize(msg.value(), context) except Exception as e: print(f"consumer side, offset {msg.offset()}: {type(e).__name__}: {e}") raw_reader.close()
The registry can't stop someone from writing other bytes into a topic: Kafka accepts any bytes. The reader finds out, record by record. Lesson 5.3 shows how to handle such a record.
Clean up
Delete the topic and the subject. If anything gets stuck, docker compose down -v in the code/
folder resets the whole lab.
lab.delete_topics(admin, [TOPIC]) registry.delete_subject(f"{TOPIC}-value") registry.delete_subject(f"{TOPIC}-value", permanent=True) admin.delete_consumer_groups(["m05l01-raw"]) print("cleaned up")