Evolving a schema without breaking readers
A schema is a contract, but contracts change: the grid operator wants a new field, a team wants better names. Producers and consumers are deployed by different teams on different days, so for a while old and new versions run side by side. This lesson shows which changes they survive, and how the registry keeps the others out.
Theory
Example: a new field while billing runs old code
The grid operator now wants to know how many phases each charger uses, to estimate the load per
phase. The gateway team can send phase_count in every reading from tomorrow. Billing reads the
same readings, but only deploys every two weeks, and its next release is in nine days.
Can the gateway ship tomorrow? And next month, when the gateway team also wants to rename
energy_kwh to energy_wh, what happens to every consumer that still expects energy_kwh?
Writer schema and reader schema
Every Avro record is written with one schema, the writer schema, whose id is in the record. A consumer can decode it with that schema and get exactly the writer's fields. It can also name its own reader schema: the version its code was written for. Avro then translates each record:
- a field the writer has but the reader doesn't know is skipped;
- a field the reader expects but the writer didn't have gets the reader's default;
- a field the reader expects, with no default, that the writer didn't have: the record can't be read.
In Python, AvroDeserializer(registry, reader_schema) uses a reader schema. Without one, you get
the writer's fields as they are, and your code has to cope with fields it doesn't expect, or
missing ones.
Compatibility modes
Before the registry accepts a new version in a subject, it checks it against the earlier ones. The subject's compatibility mode says what "compatible" means:
What to notice: the mode decides the deploy order. With the default, consumers must be upgraded before producers.
- BACKWARD (the default): the new version can read data written with the previous one. You may add fields with a default and remove fields. Upgrade consumers first.
- FORWARD: the previous version can read data written with the new one. You may add fields and remove fields that have a default. Upgrade producers first.
- FULL: both. You may add or remove fields that have defaults. Either side can go first.
- NONE: no check at all.
Each mode also exists as _TRANSITIVE (for example BACKWARD_TRANSITIVE): the new version is
checked against every earlier version, not only the latest. That matters when consumers read
old data: a reader that replays a topic from the beginning meets every version ever written.
Which changes are safe
- Add a field: give it a default, usually
nullwith the type["null", "int"]. Then it's FULL compatible. - Remove a field: only if it has a default, and only after no reader needs it.
- Rename a field: not compatible. If only the name changes, an Avro alias in the reader schema
can map the old name. If the meaning or the unit changes, as in
energy_kwhtoenergy_wh, it's a new field: add it, write both for a while, move the readers over, then stop writing the old one. - Change a type: only to a wider one (
inttolong,floattodouble). Anything else is a new field.
Checking before you deploy
You can ask the registry whether a schema would be accepted, without registering it:
test_compatibility() in the Python client, or a POST to /compatibility/subjects/<subject>/versions/latest.
An incompatible registration is refused with HTTP 409. Run that check in the build pipeline of every
producer, so a breaking change fails the build, not the consumers (lesson 9.2).
In production
- Keep BACKWARD for most topics. Use BACKWARD_TRANSITIVE for topics consumers replay from the beginning, such as compacted topics. Use FULL when you can't control who deploys first.
- Never switch a live subject to NONE. If a change truly breaks the contract, create a new topic for the new version and move consumers over.
- Give every field a
doc, including its unit. Most bad changes are changes of meaning, not of type. - Remember that a schema change also changes your producers' code: test both deploy orders before release (exercise 5.1).
Practical
Note
Notebook: code/module-05/02-evolving-a-schema.ipynb
· Lab services: kafka-1, kafka-2, kafka-3, schema-registry · Time: ~12 min
What you'll do: add phase_count compatibly, read old and new readings with old and new
readers, have a rename refused, and switch the check off to see why it's there.
Step 1: Version 1, and 50 readings written with it
write(schema_str, batch) writes readings with a given writer schema, registering it if it's new:
versions: [1] compatibility: BACKWARD
Step 2: Version 2 adds an optional field
V2 = changed(V1, lambda fields: fields.append({"name": "phase_count", "type": ["null", "int"], "default": None}))
print("compatible with version 1:", registry.test_compatibility(SUBJECT, Schema(V2, "AVRO")))
compatible with version 1: True
versions: [1, 2]
Step 3: Old and new readers
Billing reads with version 1, the grid report with version 2:
billing, grid = read_all(V1), read_all(V2)
billing (v1) reads 100 readings; fields of the newest: ['charger_id', 'energy_kwh', 'firmware', 'power_kw', 'session_id', 'ts']
grid (v2) on an old reading: phase_count = None
grid (v2) on a new reading: phase_count = 3
Step 4: Version 3 renames a field
Predict first: will the registry accept energy_kwh renamed to energy_wh?
compatible with version 2: False
register: HTTP 409: Schema being registered is incompatible with an earlier schema for subject "m05l02-telemetry-value", details: [{errorType:'READER_FIELD_MISSING_DEFAULT_VALUE', description:'The field 'energy_wh' at path '/fields/4' in the new sche
Break it: switch the check off
Someone sets the subject to NONE "just for this release", registers version 3 and writes 50
readings. Predict first: what does the grid report, still on version 2, see?
registry.set_compatibility(SUBJECT, "NONE")
readable: 100 failed: 50
offset 100: SchemaResolutionError: No default value for field energy_kwh in com.brightlane.charging.MeterReading
What just happened
Version 2 only added a field with a default, so the registry accepted it under BACKWARD, and both
readers coped with both kinds of data. Billing's version 1 reader skipped phase_count in new
readings. The grid's version 2 reader filled in the default, None, for old readings. The gateway
could ship before billing, because adding a field with a default is safe in both directions. The
rename was different: a version 3 reader would find no energy_wh in the existing readings and
have no default for it, so the registry refused it with HTTP 409. Its message names the field and
the reason. With the check switched off, version 3 got in, and every version 3 reading broke the
version 2 reader: it expects energy_kwh and has no default for it. Those 50 readings stay in the
topic; switching the check back on doesn't remove them.
Clean up
The last cell deletes the topic and the subject. If anything gets stuck, docker compose down -v
in the code/ folder resets the whole lab.
Key takeaways
- Avro translates from the writer schema to the reader's: unknown fields are skipped, missing ones get the reader's default.
- BACKWARD (the default): new readers read old data, so upgrade consumers first. FORWARD: producers first. FULL: either order.
- Add fields with defaults; never rename or change a field's meaning in place. Add a new field instead.
_TRANSITIVEmodes check every older version; use them for topics that are replayed from the start.- Check compatibility in the build pipeline; the registry refuses incompatible versions with HTTP 409.