Lab setup
The lab is a three-broker Apache Kafka 4.3.1 cluster in Docker, plus the Python environment for the notebooks. Set it up once (about 15 minutes); after that, starting it takes about 20 seconds.
Caution
Lab only. The lab runs without encryption or passwords, with small memory settings and combined broker+controller nodes. Never run Kafka like this in production. Module 8 shows the production settings.
1. What you need
- macOS on Apple Silicon or Intel. Linux works the same. On Windows, use WSL2.
- A Docker runtime with Compose v2: Docker Desktop, OrbStack or Colima. Give its virtual machine 6 GB of memory (8 GB recommended):
- Docker Desktop: Settings → Resources → Memory.
- Colima:
colima start --memory 8 --cpu 4. - Python 3.13. The easiest way is uv, which downloads it for you.
Without uv:
brew install python@3.13. - About 8 GB of free disk space for the images. Most of it is for modules 6 and 7: the Kafka Connect image with the Debezium connectors is 2.4 GB on its own.
2. Set up once
From this code/ folder, create a virtual environment and install the pinned packages.
With uv:
uv venv --python 3.13 .venv
uv pip install -r requirements.txt
Or with plain Python:
python3.13 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
requirements.txt pins every package, so you get exactly what the course was tested with. Its
top-level packages are listed in requirements.in.
3. Start the lab
docker compose up -d --wait
The first start builds a small broker image (the official apache/kafka:4.3.1 image plus a
metrics agent used in module 8) and downloads the images, so it takes a few minutes. After that it
takes about 20 seconds. --wait returns when every service is healthy:
✔ Container kafka-hands-on-lab-kafka-1-1 Healthy
✔ Container kafka-hands-on-lab-kafka-2-1 Healthy
✔ Container kafka-hands-on-lab-kafka-3-1 Healthy
✔ Container kafka-hands-on-lab-kafka-ui-1 Healthy
Check it any time with:
docker compose ps
Later modules start extra services, grouped in Compose profiles. Each lesson names the command it needs, and most notebooks start what they need themselves.
| Profile | Starts | Used in |
|---|---|---|
schemas |
Schema Registry | modules 5, 7 and 9 |
connect |
PostgreSQL and Kafka Connect with Debezium (and Schema Registry) | modules 6 and 7 |
monitoring |
Prometheus, KMinion and Grafana | module 8 |
secure |
a separate one-node broker with TLS, SCRAM logins and ACLs | modules 8 and 9 |
dr |
a one-node disaster-recovery cluster and MirrorMaker 2 | module 9 |
For example: docker compose --profile monitoring up -d --wait. The monitoring and secure
profiles together add about 1 GB of memory, dr about 1.5 GB. Lesson 9.3 runs the core lab with
secure, monitoring and dr at once: about 5.3 GB, so give the Docker VM 8 GB for it.
Lesson 9.2's tests start their own Kafka container with Testcontainers, which also pulls its small
clean-up container (testcontainers/ryuk) the first time.
4. Open the notebooks
source .venv/bin/activate
jupyter lab
JupyterLab opens in your browser. The notebooks are in module-01/, module-02/ and so on.
VS Code works too: open this folder and pick the .venv interpreter as the kernel. Run each
notebook's first cell; it checks the lab and prints the versions in use.
Some lessons also run scripts in a terminal, always from this folder, for example
python module-03/session_feed.py.
5. Stop, reset, start over
| You want to… | Run |
|---|---|
| Stop the lab, keep its data | docker compose stop |
| Start it again | docker compose up -d --wait |
| Restart one broker | docker compose restart kafka-2 |
| Remove everything, data included | docker compose down -v |
| Stop a profile's services | docker compose --profile monitoring stop |
| Start the secure broker from scratch | secure.reset_broker() in a notebook, then rerun lesson 8.1 |
| Start mirroring from scratch | dr.reset_mirroring("m09") in a notebook (removes the lessons' copies too) |
If a notebook leaves the lab in a strange state, docker compose down -v followed by
docker compose up -d --wait gives you a clean cluster in under a minute.
6. Services and ports
All ports are published on 127.0.0.1 only, so nothing is reachable from your network.
| Service | What it is | Address from your Mac | Address inside Docker | Module |
|---|---|---|---|---|
| kafka-1 | broker + controller, node 1 | 127.0.0.1:9092 |
kafka-1:19092 |
1 |
| kafka-2 | broker + controller, node 2 | 127.0.0.1:9093 |
kafka-2:19092 |
1 |
| kafka-3 | broker + controller, node 3 | 127.0.0.1:9094 |
kafka-3:19092 |
1 |
| kafka-ui | Kafbat UI, read-only | http://127.0.0.1:8080 | kafka-ui:8080 |
1 |
| schema-registry | Confluent Schema Registry | http://127.0.0.1:8081 | schema-registry:8081 |
5 |
| postgres | PostgreSQL: pricing and reporting databases | 127.0.0.1:5432 |
postgres:5432 |
6 |
| connect | Kafka Connect with Debezium | http://127.0.0.1:8083 | connect:8083 |
6 |
| prometheus | Prometheus, with the lab's alert rules | http://127.0.0.1:9090 | prometheus:9090 |
8 |
| kminion | KMinion, consumer-lag exporter | — | kminion:8080 |
8 |
| grafana | Grafana, Brightlane dashboard (no login needed to view) | http://127.0.0.1:3000 | grafana:3000 |
8 |
| kafka-secure | one-node broker, SASL_SSL only |
127.0.0.1:9095 |
kafka-secure:9095 |
8 |
| kafka-dr | one-node disaster-recovery cluster | 127.0.0.1:9096 |
kafka-dr:19092 |
9 |
| mirror-maker | MirrorMaker 2, copying primary ↔ dr | — | — | 9 |
The controllers talk to each other on port 19093 inside Docker. Each broker also serves Prometheus metrics on port 7071 inside Docker (module 8).
The secure broker is a cluster of its own. On its first start it waits for the certificate that
lesson 8.1 makes; the lab's CA certificate is then kept in .lab-data/secure/ca.crt. Grafana's
admin user is admin with the password grafana-lab; viewing needs no login.
7. Troubleshooting
"Kafka isn't reachable on 127.0.0.1:9092…" The lab isn't running. Run
docker compose up -d --wait here and rerun the cell.
port is already allocated Another program uses 9092–9094 or 8080 (often another Kafka). Stop
it, or change the left-hand port numbers in compose.yaml. If you change a Kafka port, change the
matching HOST://127.0.0.1:… in KAFKA_ADVERTISED_LISTENERS too, and set
KAFKA_BOOTSTRAP before starting Jupyter, for example
export KAFKA_BOOTSTRAP=127.0.0.1:19092,127.0.0.1:19093,127.0.0.1:19094.
A broker stays unhealthy or keeps restarting. Look at its log:
docker compose logs --tail 50 kafka-1. "Out of memory" or a container that just stops means the
Docker VM has too little memory (section 1). A corrupted state after a hard crash is fixed with
docker compose down -v.
Apple Silicon. Every image in the lab has a native arm64 version; nothing runs under emulation.
Notebook cells hang. A cell that waits for Kafka has a time limit, at most about a minute. If
one hangs longer, check docker compose ps, then restart the kernel.
Logs and tools inside a broker. The Kafka command-line tools are in /opt/kafka/bin:
docker compose exec kafka-1 /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:19092 --list
8. Versions
| Component | Version |
|---|---|
Apache Kafka (image apache/kafka) |
4.3.1 |
| confluent-kafka (Python client) | 2.15.1 (librdkafka 2.15.1) |
| Python | 3.13 |
Kafbat UI (image kafbat/kafka-ui) |
v1.5.0 |
| Prometheus JMX exporter agent | 1.6.0 |
| Confluent Schema Registry (module 5) | 8.2.4 |
| Debezium Connect, PostgreSQL (module 6) | 3.7.0.Final, 18.6 |
| Prometheus, Grafana, KMinion (module 8) | 3.15.0, 13.2.3, 2.3.6 |
| MirrorMaker 2 (module 9) | from Apache Kafka 4.3.1 |
| pytest, testcontainers (module 9) | 9.1.1, 4.15.0 |
| JupyterLab | 4.6.4 |
Code licence: MIT (see LICENSE).