Module 5 exercises

Start the lab with the Schema Registry first (docker compose --profile schemas up -d --wait in the code/ folder). Each exercise has a start notebook to work in and a solution notebook to compare with afterwards.

Exercise 1: Ship phase_count in both deploy orders

Goal: design a schema change that survives either deploy order. · Time: ~10 min · Starter: code/module-05/exercise-1/start/

The gateway team writes meter readings; the billing team reads them. They deploy on different days, and nobody can promise who goes first. So the subject m05e1-telemetry-value uses FULL compatibility. The start notebook registers version 1, writes 20 readings with it, and gives you PHASE_COUNT_FIELD, the new field for version 2.

Change PHASE_COUNT_FIELD so that three checks pass: the registry accepts version 2 under FULL; billing on version 1 can read readings the new gateway writes; and billing on version 2 can read the readings the old gateway wrote. Then, in a markdown cell, explain why your default is the right value, and whether "default": 3 would be correct.

You're done when: the notebook prints registry accepts v2 under FULL: True, gateway first works: True and billing first works: True, and your note explains the default.

Hints

Hint 1 Old readings don't have the field. What should a version 2 reader see for them?
Hint 2 In Avro, a field that may be empty has a union type, `["null", "int"]`, and its default must match the first type in the union.

Exercise 2: A replay tool with a dry run

Goal: bring parked records back safely, and only once. · Time: ~25 min · Starter: code/module-05/exercise-2/start/

After the legacy incident, m05e2-telemetry.dlq holds 45 parked records: 15 good Avro readings parked during a ledger outage, 20 complete legacy 3.9 payloads, and 10 payloads cut short. Each has the telemetry worker's headers (dlq.source.topic, dlq.source.offset and so on).

Finish module-05/exercise-2/start/replay_dlq.py:

The notebook runs your tool with --dry-run, then twice for real, then checks the telemetry topic.

You're done when: the notebook prints dry run sent nothing: True, replayed 35 readings: True, second run replayed nothing: True and repaired legacy readings: 20, with the right power: 20.

Hints

Hint 1 Try the cheap test first: if `deserialize(msg.value(), context)` works, the record is a good reading. A legacy payload raises `SerializationError`.
Hint 2 The legacy format: `session` instead of `session_id`, `power` in watts and `energy_wh` in watt-hours, both as text. A meter reading needs kW and kWh.
Hint 3 `producer.flush()` returns the number of records it couldn't send. Commit only if that's 0, with `consumer.commit(message=..., asynchronous=False)`.