28 min · theory + hands-on practical

Evolving a schema without breaking readers

A schema is a contract, but contracts change: the grid operator wants a new field, a team wants better names. Producers and consumers are deployed by different teams on different days, so for a while old and new versions run side by side. This lesson shows which changes they survive, and how the registry keeps the others out.

Theory

Example: a new field while billing runs old code

The grid operator now wants to know how many phases each charger uses, to estimate the load per phase. The gateway team can send phase_count in every reading from tomorrow. Billing reads the same readings, but only deploys every two weeks, and its next release is in nine days.

Can the gateway ship tomorrow? And next month, when the gateway team also wants to rename energy_kwh to energy_wh, what happens to every consumer that still expects energy_kwh?

Writer schema and reader schema

Every Avro record is written with one schema, the writer schema, whose id is in the record. A consumer can decode it with that schema and get exactly the writer's fields. It can also name its own reader schema: the version its code was written for. Avro then translates each record:

In Python, AvroDeserializer(registry, reader_schema) uses a reader schema. Without one, you get the writer's fields as they are, and your code has to cope with fields it doesn't expect, or missing ones.

Compatibility modes

Before the registry accepts a new version in a subject, it checks it against the earlier ones. The subject's compatibility mode says what "compatible" means:

Left: BACKWARD, the registry's default. Data written with v1 must be readable by a reader with v2, so upgrade consumers first. Right: FORWARD. Data written with v2 must be readable by a reader with v1, so upgrade producers first. FULL means both, so either side can go first; the transitive variants check against every older version. Adding a field with a default is FULL compatible, adding one without a default is FORWARD only, and renaming a field is neither.

What to notice: the mode decides the deploy order. With the default, consumers must be upgraded before producers.

Each mode also exists as _TRANSITIVE (for example BACKWARD_TRANSITIVE): the new version is checked against every earlier version, not only the latest. That matters when consumers read old data: a reader that replays a topic from the beginning meets every version ever written.

Which changes are safe

Checking before you deploy

You can ask the registry whether a schema would be accepted, without registering it: test_compatibility() in the Python client, or a POST to /compatibility/subjects/<subject>/versions/latest. An incompatible registration is refused with HTTP 409. Run that check in the build pipeline of every producer, so a breaking change fails the build, not the consumers (lesson 9.2).

In production

Practical

Note

Notebook: code/module-05/02-evolving-a-schema.ipynb · Lab services: kafka-1, kafka-2, kafka-3, schema-registry · Time: ~12 min

What you'll do: add phase_count compatibly, read old and new readings with old and new readers, have a rename refused, and switch the check off to see why it's there.

Step 1: Version 1, and 50 readings written with it

write(schema_str, batch) writes readings with a given writer schema, registering it if it's new:

versions: [1]  compatibility: BACKWARD

Step 2: Version 2 adds an optional field

code/module-05/02-evolving-a-schema.ipynb
V2 = changed(V1, lambda fields: fields.append({"name": "phase_count", "type": ["null", "int"], "default": None}))
print("compatible with version 1:", registry.test_compatibility(SUBJECT, Schema(V2, "AVRO")))
compatible with version 1: True
versions: [1, 2]

Step 3: Old and new readers

Billing reads with version 1, the grid report with version 2:

code/module-05/02-evolving-a-schema.ipynb
billing, grid = read_all(V1), read_all(V2)
billing (v1) reads 100 readings; fields of the newest: ['charger_id', 'energy_kwh', 'firmware', 'power_kw', 'session_id', 'ts']
grid (v2) on an old reading: phase_count = None
grid (v2) on a new reading:  phase_count = 3

Step 4: Version 3 renames a field

Predict first: will the registry accept energy_kwh renamed to energy_wh?

compatible with version 2: False
register: HTTP 409: Schema being registered is incompatible with an earlier schema for subject "m05l02-telemetry-value", details: [{errorType:'READER_FIELD_MISSING_DEFAULT_VALUE', description:'The field 'energy_wh' at path '/fields/4' in the new sche

Break it: switch the check off

Someone sets the subject to NONE "just for this release", registers version 3 and writes 50 readings. Predict first: what does the grid report, still on version 2, see?

code/module-05/02-evolving-a-schema.ipynb
registry.set_compatibility(SUBJECT, "NONE")
readable: 100  failed: 50
offset 100: SchemaResolutionError: No default value for field energy_kwh in com.brightlane.charging.MeterReading

What just happened

Version 2 only added a field with a default, so the registry accepted it under BACKWARD, and both readers coped with both kinds of data. Billing's version 1 reader skipped phase_count in new readings. The grid's version 2 reader filled in the default, None, for old readings. The gateway could ship before billing, because adding a field with a default is safe in both directions. The rename was different: a version 3 reader would find no energy_wh in the existing readings and have no default for it, so the registry refused it with HTTP 409. Its message names the field and the reason. With the check switched off, version 3 got in, and every version 3 reading broke the version 2 reader: it expects energy_kwh and has no default for it. Those 50 readings stay in the topic; switching the check back on doesn't remove them.

Clean up

The last cell deletes the topic and the subject. If anything gets stuck, docker compose down -v in the code/ folder resets the whole lab.

Key takeaways

Next: Poison pills and dead-letter topics