Lab setup

The lab is a three-broker Apache Kafka 4.3.1 cluster in Docker, plus the Python environment for the notebooks. Set it up once (about 15 minutes); after that, starting it takes about 20 seconds.

Caution

Lab only. The lab runs without encryption or passwords, with small memory settings and combined broker+controller nodes. Never run Kafka like this in production. Module 8 shows the production settings.

1. What you need

2. Set up once

From this code/ folder, create a virtual environment and install the pinned packages.

With uv:

uv venv --python 3.13 .venv
uv pip install -r requirements.txt

Or with plain Python:

python3.13 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

requirements.txt pins every package, so you get exactly what the course was tested with. Its top-level packages are listed in requirements.in.

3. Start the lab

docker compose up -d --wait

The first start builds a small broker image (the official apache/kafka:4.3.1 image plus a metrics agent used in module 8) and downloads the images, so it takes a few minutes. After that it takes about 20 seconds. --wait returns when every service is healthy:

 ✔ Container kafka-hands-on-lab-kafka-1-1   Healthy
 ✔ Container kafka-hands-on-lab-kafka-2-1   Healthy
 ✔ Container kafka-hands-on-lab-kafka-3-1   Healthy
 ✔ Container kafka-hands-on-lab-kafka-ui-1  Healthy

Check it any time with:

docker compose ps

Later modules start extra services, grouped in Compose profiles. Each lesson names the command it needs, and most notebooks start what they need themselves.

Profile Starts Used in
schemas Schema Registry modules 5, 7 and 9
connect PostgreSQL and Kafka Connect with Debezium (and Schema Registry) modules 6 and 7
monitoring Prometheus, KMinion and Grafana module 8
secure a separate one-node broker with TLS, SCRAM logins and ACLs modules 8 and 9
dr a one-node disaster-recovery cluster and MirrorMaker 2 module 9

For example: docker compose --profile monitoring up -d --wait. The monitoring and secure profiles together add about 1 GB of memory, dr about 1.5 GB. Lesson 9.3 runs the core lab with secure, monitoring and dr at once: about 5.3 GB, so give the Docker VM 8 GB for it.

Lesson 9.2's tests start their own Kafka container with Testcontainers, which also pulls its small clean-up container (testcontainers/ryuk) the first time.

4. Open the notebooks

source .venv/bin/activate
jupyter lab

JupyterLab opens in your browser. The notebooks are in module-01/, module-02/ and so on. VS Code works too: open this folder and pick the .venv interpreter as the kernel. Run each notebook's first cell; it checks the lab and prints the versions in use.

Some lessons also run scripts in a terminal, always from this folder, for example python module-03/session_feed.py.

5. Stop, reset, start over

You want to… Run
Stop the lab, keep its data docker compose stop
Start it again docker compose up -d --wait
Restart one broker docker compose restart kafka-2
Remove everything, data included docker compose down -v
Stop a profile's services docker compose --profile monitoring stop
Start the secure broker from scratch secure.reset_broker() in a notebook, then rerun lesson 8.1
Start mirroring from scratch dr.reset_mirroring("m09") in a notebook (removes the lessons' copies too)

If a notebook leaves the lab in a strange state, docker compose down -v followed by docker compose up -d --wait gives you a clean cluster in under a minute.

6. Services and ports

All ports are published on 127.0.0.1 only, so nothing is reachable from your network.

Service What it is Address from your Mac Address inside Docker Module
kafka-1 broker + controller, node 1 127.0.0.1:9092 kafka-1:19092 1
kafka-2 broker + controller, node 2 127.0.0.1:9093 kafka-2:19092 1
kafka-3 broker + controller, node 3 127.0.0.1:9094 kafka-3:19092 1
kafka-ui Kafbat UI, read-only http://127.0.0.1:8080 kafka-ui:8080 1
schema-registry Confluent Schema Registry http://127.0.0.1:8081 schema-registry:8081 5
postgres PostgreSQL: pricing and reporting databases 127.0.0.1:5432 postgres:5432 6
connect Kafka Connect with Debezium http://127.0.0.1:8083 connect:8083 6
prometheus Prometheus, with the lab's alert rules http://127.0.0.1:9090 prometheus:9090 8
kminion KMinion, consumer-lag exporter — kminion:8080 8
grafana Grafana, Brightlane dashboard (no login needed to view) http://127.0.0.1:3000 grafana:3000 8
kafka-secure one-node broker, SASL_SSL only 127.0.0.1:9095 kafka-secure:9095 8
kafka-dr one-node disaster-recovery cluster 127.0.0.1:9096 kafka-dr:19092 9
mirror-maker MirrorMaker 2, copying primary ↔ dr — — 9

The controllers talk to each other on port 19093 inside Docker. Each broker also serves Prometheus metrics on port 7071 inside Docker (module 8).

The secure broker is a cluster of its own. On its first start it waits for the certificate that lesson 8.1 makes; the lab's CA certificate is then kept in .lab-data/secure/ca.crt. Grafana's admin user is admin with the password grafana-lab; viewing needs no login.

7. Troubleshooting

"Kafka isn't reachable on 127.0.0.1:9092…" The lab isn't running. Run docker compose up -d --wait here and rerun the cell.

port is already allocated Another program uses 9092–9094 or 8080 (often another Kafka). Stop it, or change the left-hand port numbers in compose.yaml. If you change a Kafka port, change the matching HOST://127.0.0.1:… in KAFKA_ADVERTISED_LISTENERS too, and set KAFKA_BOOTSTRAP before starting Jupyter, for example export KAFKA_BOOTSTRAP=127.0.0.1:19092,127.0.0.1:19093,127.0.0.1:19094.

A broker stays unhealthy or keeps restarting. Look at its log: docker compose logs --tail 50 kafka-1. "Out of memory" or a container that just stops means the Docker VM has too little memory (section 1). A corrupted state after a hard crash is fixed with docker compose down -v.

Apple Silicon. Every image in the lab has a native arm64 version; nothing runs under emulation.

Notebook cells hang. A cell that waits for Kafka has a time limit, at most about a minute. If one hangs longer, check docker compose ps, then restart the kernel.

Logs and tools inside a broker. The Kafka command-line tools are in /opt/kafka/bin:

docker compose exec kafka-1 /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:19092 --list

8. Versions

Component Version
Apache Kafka (image apache/kafka) 4.3.1
confluent-kafka (Python client) 2.15.1 (librdkafka 2.15.1)
Python 3.13
Kafbat UI (image kafbat/kafka-ui) v1.5.0
Prometheus JMX exporter agent 1.6.0
Confluent Schema Registry (module 5) 8.2.4
Debezium Connect, PostgreSQL (module 6) 3.7.0.Final, 18.6
Prometheus, Grafana, KMinion (module 8) 3.15.0, 13.2.3, 2.3.6
MirrorMaker 2 (module 9) from Apache Kafka 4.3.1
pytest, testcontainers (module 9) 9.1.1, 4.15.0
JupyterLab 4.6.4

Code licence: MIT (see LICENSE).