30 min · theory + hands-on practical

Sinks, errors and dead letters in Connect

Lesson 6.1 brought the tariffs into Kafka. This lesson takes them out again, into another database, with a sink connector. Then it does what production will do sooner or later: feeds the sink a record it can't read. And it shows how to look after connectors once they run.

Theory

Example: finance's weekly spreadsheet

Finance's monthly reports run on its own reporting database. To get tariffs into it, someone exports the pricing table every Friday and imports it there. The reports' prices are up to a week old, and twice this year the import was forgotten. Finance wants a copy that is always up to date, with no one in the loop.

The tariff changes are already in Kafka (lesson 6.1). A sink connector can write them into the reporting database as they arrive.

Sink connectors

A sink connector's tasks read topics as a consumer group, named connect-<connector name>, and write each record to the target system. The Debezium JDBC sink writes to relational databases. It understands Debezium's change events directly:

Like any consumer, a sink delivers at least once (lesson 3.2). After a crash, it continues from its last committed offset and writes some records again. Upserts by key make that harmless; plain inserts create duplicate rows. Exercise 6.2 shows how many.

When a record can't be processed

A record can fail in three places on its way through a sink task: in the converter (the bytes aren't what it expects), in a transform, or while writing to the target.

A sink task reads records from the tariffs topic, converts them from Avro, and upserts them into the reporting table. A record the converter can't read stops the task with errors.tolerance=none, the default, and every later record waits. With errors.tolerance=all, the record goes to a dead-letter topic with __connect.errors headers instead, and the task carries on. The task's position is the consumer group's committed offset.

What to notice: by default, one bad record stops everything behind it. That's lesson 5.3's poison pill, inside Connect.

errors.tolerance=all without a dead-letter topic drops bad records with nothing but a log line. Don't use it that way.

Operating connectors

Connectors and tasks have states: RUNNING, PAUSED, STOPPED, FAILED and UNASSIGNED. The REST API does the rest:

Command-line tools such as kcctl wrap the same API, and UIs like Kafbat UI can show connectors too.

Deleting a connector doesn't delete its position. A new connector with the same name picks it up: that's handy for a configuration rebuild, and a trap after a fresh start. While building this lab, a re-created Debezium connector tried to continue from an old position in a log that no longer existed, and failed. Reset the offsets before you delete a connector you want to start over.

In production

Practical

Note

Notebook: code/module-06/02-sinks-errors-and-dead-letters.ipynb · Lab services: kafka-1..3, schema-registry, postgres, connect · Time: ~14 min

What you'll do: copy the tariffs into the reporting database with a JDBC sink, time a change end to end, stop the sink with a bad record, recover with a dead-letter topic, and operate both connectors through the REST API.

Step 1: The source: tariff changes from the pricing database

The notebook registers the same CDC connector as lesson 6.1, as m06l02-pricing-cdc:

source: RUNNING

Step 2: The sink: a JDBC connector into the reporting database

code/module-06/02-sinks-errors-and-dead-letters.ipynb
    "insert.mode": "upsert",                # insert a new key, update an existing one
    "delete.enabled": "true",               # a delete event deletes the row
    "primary.key.mode": "record_key",       # the record key (tariff_id) is the primary key
    "schema.evolution": "basic",            # create the table, add new columns
    "collection.name.format": "tariffs",    # the table's name
reporting columns: [('tariff_id', 'text'), ('site_id', 'text'), ('price_per_kwh', 'numeric'), ('idle_fee_per_min', 'numeric'), ('valid_from', 'date')]
rows: pricing 36 | reporting 36

Step 3: A price change, end to end

price visible in reporting after 0.1 s
rows: pricing 35 | reporting 35

Break it: a record the sink can't read

A quick script writes one tariff as plain JSON into the topic. Predict first: what does the sink do, and what happens to the next real price change?

connector: RUNNING | task: FAILED
Caused by: org.apache.kafka.connect.errors.DataException: Failed to deserialize data for topic m06l02_pricing.public.tariffs to Avro:
Caused by: org.apache.kafka.common.errors.SerializationException: Error deserializing schema ID
Caused by: java.lang.IllegalArgumentException: Unknown magic byte!
tar-site-N01-1 in reporting: 0.4900 (pricing has 0.5000)

Step 4: Tolerate bad records, park them in a dead-letter topic

code/module-06/02-sinks-errors-and-dead-letters.ipynb
    "errors.tolerance": "all",                                  # don't stop on a bad record ...
    "errors.deadletterqueue.topic.name": DLQ,                   # ... park it here
    "errors.deadletterqueue.topic.replication.factor": "3",
    "errors.deadletterqueue.context.headers.enable": "true",    # with headers saying why
code/module-06/02-sinks-errors-and-dead-letters.ipynb
lab.connect_api("POST", "/connectors/m06l02-reporting-tariffs/restart?includeTasks=true&onlyFailed=true")
tar-site-N01-1 in reporting: 0.5000
value: b'{"tariff_id": "tar-site-N01-1", "price_per_kwh": 0.39}'
__connect.errors.topic                     m06l02_pricing.public.tariffs
__connect.errors.partition                 0
__connect.errors.offset                    39
__connect.errors.connector.name            m06l02-reporting-tariffs
__connect.errors.task.id                   0
__connect.errors.stage                     KEY_CONVERTER
__connect.errors.class.name                io.confluent.connect.avro.AvroConverter
__connect.errors.exception.class.name      org.apache.kafka.connect.errors.DataException
__connect.errors.exception.message         Failed to deserialize data for topic m06l02_pricing.public.tariffs to

Step 5: Operate the connectors

m06l02-pricing-cdc         source  RUNNING  tasks: ['RUNNING']
m06l02-reporting-tariffs   sink    RUNNING  tasks: ['RUNNING']
GET /connectors/m06l02-pricing-cdc/offsets
{"lsn_proc": 157326720, "lsn_events_processed": 1, "messageType": "UPDATE", "lsn_commit": 157117496, "lsn": 157326720, "txId": 931, "ts_usec": 1791308569798030}
GET /connectors/m06l02-reporting-tariffs/offsets
{"partition": {"kafka_partition": 0, "kafka_topic": "m06l02_pricing.public.tariffs"}, "offset": {"kafka_offset": 41}}

Then the notebook stops the sink, changes a price, and resumes it:

PUT /connectors/m06l02-reporting-tariffs/stop
while stopped: 0.5000
PUT /connectors/m06l02-reporting-tariffs/resume
after resume: 0.4800

What just happened

The JDBC sink created the reporting table from the Avro schema and kept it equal to the pricing table: a price change arrived within a tenth of a second, and the delete removed the row. Then the plain JSON record reached the sink's converter, which found no magic byte. With the default tolerance, the task failed and stopped there, and the next real price change waited behind it, while the connector itself still said RUNNING. With errors.tolerance=all and a dead-letter topic, the restarted task parked the bad record with headers naming the topic, partition, offset, stage and error, and the waiting change went through. Finally, the offsets showed the two kinds of position: a place in PostgreSQL's log for the source, a topic offset for the sink. Stop and resume kept the sink's position, so the change made while it was stopped arrived as soon as it ran again.

Clean up

The last cell removes both connectors (resetting their offsets first), the slot and publication, the topics, schemas and the reporting table, and resets the pricing database. If anything gets stuck, docker compose down -v in the code/ folder resets the whole lab.

Key takeaways

Next: Module 6 exercises