Kafka, Hands-On
Apache Kafka keeps streams of events, such as payments, sensor readings and clicks, in durable logs that many systems can read at their own pace. This course teaches you to use it with confidence in real work: first the ideas, then the code, then the failures, the security and the operations that production brings.
What you'll build
You join Brightlane Charging, a fictional network of 240 public chargers for electric cars at 36 sites. Its charger readings, charging sessions and charger status flow through Kafka to billing, a report for the national grid operator, a live charger map and field technicians. Prices come from a PostgreSQL database. You build and run all of it on a three-broker Kafka 4.3 cluster in Docker on your Mac, with Python.
What you'll learn
- Decide whether Kafka fits a problem, and explain a cluster's brokers, KRaft controllers, topics, partitions and replicas well enough to reason about failures.
- Write Python producers that keep per-key order, survive broker failures without losing or duplicating data, and reach a measured throughput target.
- Write Python consumers that scale in groups, commit offsets correctly, rebalance quickly, and use share groups when work-queue semantics fit.
- Configure retention, compaction and replication for a topic from its business requirements.
- Govern data with schemas: evolve them safely and keep bad records out of the main path.
- Move data in and out with Kafka Connect and Debezium, and build exactly-once, stateful, windowed processing in Python.
- Secure, monitor, operate, test and recover a Kafka deployment, and review it for production.
How every lesson works
Each lesson has two parts. Theory explains one to three ideas in plain words, starting from a concrete situation at Brightlane, with diagrams. Practical has you run a notebook (or a few scripts) against the lab, watch the idea happen, then change or break one thing and see the effect. Each module ends with exercises (with solutions) and a short quiz.
Before you start
You need to be comfortable reading Python, and you need Docker Desktop, OrbStack or Colima with 6 GB of memory for the Docker VM (8 GB recommended). No Kafka experience is needed. Set up the lab once by following the lab setup; it takes about 15 minutes.
The course is written for Apache Kafka 4.3.1, the confluent-kafka Python client 2.15.1 and Python 3.13 (checked October 2026). Words in bold where they're first explained are collected in the glossary.
Modules
- First events at Brightlane: why an event log, the parts of a Kafka cluster, and reading any record by offset or time.
- Writing events: producers: keys and order, durable writes, batching and throughput.
- Reading events: consumers: consumer groups and lag, offset commits, rebalances, and share groups for work queues.
- Inside the cluster: replication and storage: failover and the controller quorum, retention and deleting data, compacted topics.
- Data contracts: schemas and bad data: Avro and the Schema Registry, safe schema evolution, poison pills and dead-letter topics.
- Moving data in and out: Kafka Connect and CDC: change data capture with Debezium, sinks and their errors.
- Stream processing in Python: exactly once with transactions, state with changelogs, event-time windows.
- Securing and operating the cluster: TLS and SCRAM logins, ACLs and quotas, metrics and alerts, changes without downtime.
- Production-ready Brightlane: disaster recovery with MirrorMaker 2, testing in layers, and a production-readiness review.