Module 6 solutions
Solution 1: Add the sites table to CDC, with routing
The solution notebook is in code/module-06/exercise-1/solution/.
Approach. Capture both tables, keep only the new row state, and rename the topics:
CHANGES = {
"table.include.list": "public.sites,public.tariffs",
"transforms": "unwrap,route",
"transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState", # only the new row
"transforms.route.type": "org.apache.kafka.connect.transforms.RegexRouter",
"transforms.route.regex": r"m06e1_pricing\.public\.(.*)", # m06e1_pricing.public.sites ...
"transforms.route.replacement": "m06e1.$1", # ... becomes m06e1.sites
}
m06e1 topics: ['m06e1.sites', 'm06e1.tariffs']
sites records: 37 | tariffs records: 36
last sites record: ({'site_id': 'site-N03'}, {'site_id': 'site-N03', 'region': 'N', 'name': 'Brightlane North 03 (Station Square)', 'chargers': 8})
only the two topics: True
plain rows: True
rename arrived: True
With the start configuration, the connector wrote full change events to
m06e1_pricing.public.tariffs, and the two target topics stayed empty.
Why it works. Transforms run in the order listed in transforms. ExtractNewRecordState
replaces each change event with its after row, and turns a delete into a tombstone. RegexRouter
then rewrites the topic name: (.*) captures the table name and $1 puts it back after m06e1..
Because the router runs inside the connector, Debezium's default topics are never created. The
rename of site-N03 arrived as a 37th record with the same key, which is exactly what a compacted
topic keeps.
Two details matter. In the regex, the dots are escaped (\.), because an unescaped dot matches any
character. And the snapshot of both tables was taken when the connector started: adding a table to
an existing connector later needs an incremental snapshot (a Debezium signal), or the new
table's existing rows never arrive.
Solution 2: Break and recover the sink, no data lost
The solution notebook is in code/module-06/exercise-2/solution/.
Approach. Make the sink write a mirror, not a history, and make replays harmless:
"insert.mode": "upsert", # a replayed change overwrites the same row: harmless
"primary.key.mode": "record_key", # one row per tariff_id
"delete.enabled": "true", # a delete event removes the row
rows: pricing 33 | reporting 33
reporting matches pricing: True
With the start settings:
rows: pricing 33 | reporting 378
reporting matches pricing: False
Where the extra rows came from. With insert.mode=insert and no primary key, every change event
became a new row: 36 snapshot rows plus 200 updates make 236. The other 142 came from the crash. A
killed worker never saves its latest positions. After the restart, the source connector re-read part
of PostgreSQL's log from its last saved position and sent those changes again. The sink re-read part
of the topic from its last committed offset. Each repeated record became one more row. The 3 deletes
were ignored (delete.enabled=false).
Why upserts survive it. With the record key as the primary key, a repeated change overwrites the row with the same values it already has. Replaying a stretch of changes in order ends in the same state: the sink's writes are idempotent (lesson 3.2). Delete events remove rows, and a repeated delete finds nothing to delete. Connect gives at-least-once delivery on both sides, source and sink. A sink that is idempotent per key is the only way to end up with an exact copy anyway.