Notebook · code/module-05/01-schemas-and-the-schema-registry.ipynb

Download 01-schemas-and-the-schema-registry.ipynb

Open it in JupyterLab or VS Code from the lab folder to run it. This page is a read-only view.

Schemas and the Schema Registry — practical

Meter readings get an Avro schema. You register it, write readings with it, look at the bytes that end up in Kafka, and read them back. Then a producer sends a reading with a renamed field, and another one sends plain JSON into the topic.

Lab services: kafka-1, kafka-2, kafka-3, schema-registry (docker compose --profile schemas up -d --wait) · Lesson: modules/05-data-contracts/01-schemas-and-the-schema-registry.md

# Setup: find the lab's shared helpers and check the lab is running
import sys, pathlib
lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / "common").is_dir())
sys.path.insert(0, str(lab_root))

import json
from common import settings, lab
from common.scenario import Brightlane, parse_iso, to_json

lab.require_kafka()
lab.require_http("Schema Registry", settings.SCHEMA_REGISTRY_URL + "/subjects",
                 start_hint="docker compose --profile schemas up -d --wait")
lab.print_versions()

Step 1: The schema

The schema lives in the repository, next to the code: code/schemas/meter_reading.avsc.

schema_str = (lab_root / "schemas" / "meter_reading.avsc").read_text()
for field in json.loads(schema_str)["fields"]:
    print(f"{field['name']:<11} {json.dumps(field['type']):<50} {field['doc']}")

Step 2: Write readings with the schema

AvroSerializer turns a dict into Avro bytes. The first time, it registers the schema under the subject <topic>-value and gets the schema's id back. to_avro converts the reading's ISO time text into a datetime, which the timestamp-millis field expects.

from confluent_kafka import Producer
from confluent_kafka.admin import AdminClient
from confluent_kafka.schema_registry import SchemaRegistryClient
from confluent_kafka.schema_registry.avro import AvroSerializer
from confluent_kafka.serialization import SerializationContext, MessageField

TOPIC = "m05l01-telemetry"
admin = AdminClient({"bootstrap.servers": settings.KAFKA_BOOTSTRAP})
lab.recreate_topic(admin, TOPIC, partitions=3)
registry = SchemaRegistryClient({"url": settings.SCHEMA_REGISTRY_URL})
for subject in [s for s in registry.get_subjects() if s.startswith(TOPIC)]:
    registry.delete_subject(subject)
    registry.delete_subject(subject, permanent=True)

to_avro = lambda reading, ctx: {**reading, "ts": parse_iso(reading["ts"])}
serialize = AvroSerializer(registry, schema_str, to_avro)
context = SerializationContext(TOPIC, MessageField.VALUE)

readings = [e for _, e in Brightlane().stream(sessions=10) if "type" not in e][:200]
producer = Producer(settings.PRODUCER_DEFAULTS)
for reading in readings:
    producer.produce(TOPIC, key=reading["charger_id"], value=serialize(reading, context))
producer.flush(10)

latest = registry.get_latest_version(f"{TOPIC}-value")
print("subjects:", [s for s in registry.get_subjects() if s.startswith("m05l01")])
print(f"subject {latest.subject}: version {latest.version}, schema id {latest.schema_id}")

Step 3: The bytes in Kafka

A plain consumer, without a deserializer, sees the raw bytes of the first record.

from confluent_kafka import Consumer, TopicPartition

raw_reader = Consumer({"bootstrap.servers": settings.KAFKA_BOOTSTRAP, "group.id": "m05l01-raw",
                       "enable.auto.commit": False})
raw_reader.assign([TopicPartition(TOPIC, p, 0) for p in range(3)])
msg = raw_reader.poll(10)
raw = msg.value()
print("first 5 bytes:", raw[:5].hex(" "))
print("magic byte:", raw[0], "  schema id:", int.from_bytes(raw[1:5], "big"))
print("the rest:", raw[5:40], "...")
avro_bytes = sum(len(serialize(r, context)) for r in readings)
json_bytes = sum(len(to_json(r)) for r in readings)
print(f"all 200 readings as Avro: {avro_bytes:,} bytes; as JSON: {json_bytes:,} bytes")

Step 4: Read them back

AvroDeserializer reads the schema id from each record, fetches that schema from the registry once (then caches it), and decodes the rest.

from confluent_kafka.schema_registry.avro import AvroDeserializer

deserialize = AvroDeserializer(registry)
print(deserialize(raw, context))

Break it: a renamed field, and a producer without a schema

The gateway's next release sends power in watts, as power_w. Predict first: what happens when it serializes such a reading with the registered schema?

renamed = {k: v for k, v in readings[0].items() if k != "power_kw"} | {"power_w": 21200}
try:
    serialize(renamed, context)
except Exception as e:
    print(f"producer side: {type(e).__name__}: {str(e)[:120]}")

The producer can't even create the record: the mistake shows up at the source, not as zeros in a report. Now a quick script writes one reading as plain JSON into the same topic, and a reader with the Avro deserializer meets it:

producer.produce(TOPIC, key="BL-N03-04", value=to_json(readings[1]), partition=0)
producer.flush(10)
end = raw_reader.get_watermark_offsets(TopicPartition(TOPIC, 0), timeout=10)[1]
raw_reader.assign([TopicPartition(TOPIC, 0, end - 1)])
msg = raw_reader.poll(10)
try:
    deserialize(msg.value(), context)
except Exception as e:
    print(f"consumer side, offset {msg.offset()}: {type(e).__name__}: {e}")
raw_reader.close()

The registry can't stop someone from writing other bytes into a topic: Kafka accepts any bytes. The reader finds out, record by record. Lesson 5.3 shows how to handle such a record.

Clean up

Delete the topic and the subject. If anything gets stuck, docker compose down -v in the code/ folder resets the whole lab.

lab.delete_topics(admin, [TOPIC])
registry.delete_subject(f"{TOPIC}-value")
registry.delete_subject(f"{TOPIC}-value", permanent=True)
admin.delete_consumer_groups(["m05l01-raw"])
print("cleaned up")