Moving data in and out: Kafka Connect and CDC

Not every event starts in a producer you wrote. Brightlane's tariffs live in a PostgreSQL table, and finance wants them in its own database. This module moves data in and out of Kafka with Kafka Connect: change data capture with Debezium on the way in, a JDBC sink on the way out, and what happens when something goes wrong in between.

By the end of this module you can: - explain Connect's workers, connectors, tasks, converters and transforms, and manage connectors through the REST API; - capture a PostgreSQL table with Debezium and read its snapshot and change events; - flatten change events into a compacted "latest row" topic with transforms; - spot and prevent a replication slot that fills the database's disk; - write a topic into a database with an idempotent sink, and handle bad records with a dead-letter topic.

Lessons

  1. Change data capture from PostgreSQL
  2. Sinks, errors and dead letters in Connect

Then practise with the exercises and check yourself with the module quiz.