34 min · theory + hands-on practical

The production-readiness review

Everything this course taught ends up as a list of things that must be true before real traffic arrives. This lesson writes that list, turns it into a check a pipeline can run, reviews Brightlane's next release with it, and fixes what it finds. It closes with the patterns to reach for, the ones to avoid, and the ways to run Kafka.

Theory

Example: is it ready?

Brightlane's next release connects the first 1,500 motorway chargers. Everything worked in staging: the gateway wrote, billing charged, the dashboard was green. The CTO asks a different question: is it ready? Ready means it keeps working when a broker restarts, a service crashes or someone makes a mistake, and that someone notices when it doesn't. Working on a good day is not the test.

The team writes down what "ready" means, as a list, and checks the release against it.

The production checklist

Each line is something an earlier lesson showed failing:

Area Must be true Lessons
Cluster 3+ brokers; no automatic topic creation; new and internal topics get 3 replicas and need 2 in sync; no unclean leader elections 1.2, 4.1
Topics every topic in a reviewed file; replication factor 3, min.insync.replicas 2; retention chosen for each; compaction where the latest value counts 2.2, 4.1–4.3, 8.4
Clients producers acks=all with idempotence; consumers store an offset only after processing; transactional services read read_committed; idempotent processing by business key 2.2, 3.2, 7.1
Data schemas with compatibility checks; a dead-letter path for bad records 5.1–5.3
Security TLS and a login per service; deny by default; no wildcard ACLs; quotas for heavy clients 8.1, 8.2
Observability alerts for broker down, offline and under-min-ISR partitions, controller count and consumer lag, each with a runbook 8.3
Operations rolling restarts that wait for replication; topics as code; upgrade notes read 8.4
Recovery a copy at another site; failover and failback runbooks, rehearsed 9.1
Tests the four layers, in CI 9.2

Turning the list into a check

A list in a wiki is read once. A readiness check is a program that verifies the list against the running system and the release's files, and exits with an error if anything fails. A CI pipeline then refuses to deploy a release that fails it, every time, without anyone remembering to look.

The course's readiness_check.py reads two files of the release: the topics (lesson 8.4's format) and each service's client settings. It looks at the clusters through the Admin API and Prometheus's API, and prints PASS, FAIL or SKIP per check, with the lesson that explains each failure.

The release under review in the middle, with the six areas the readiness check looks at around it: the cluster's brokers and defaults, the topics' replicas, min ISR and retention, the services' client settings, security, alerts with runbooks, and disaster recovery. Each area names the modules that teach it.

What to notice: every area is something a program can read. What it can't read stays for people.

Some of the list can't be automated: whether the runbooks were rehearsed, whether someone is on call, whether the cluster survives the peak load. A review meeting covers those, with the check's report on the table.

Patterns worth knowing

Anti-patterns

Ways to run Kafka

The costs that surprise people: data crossing availability zones (replication and consumers), storage for long retention, and the number of partitions, which managed services often bill.

What's coming

Kafka 4.4 is in its release-candidate phase in October 2026; read its upgrade notes before moving. Further out, diskless topics (KIP-1150, accepted as a direction in March 2026) would write topic data straight to object storage instead of broker disks, with no broker-to-broker replication. They trade latency for much lower storage and cross-zone costs. The details are still being designed, so none of it is in a release yet.

In production

Practical

Note

Script: code/module-09/readiness_check.py, with the release in code/module-09/release/ · Notebook: code/module-09/03-production-readiness-review.ipynb · Lab services: kafka-1..3, kafka-secure, prometheus, kminion, grafana, kafka-dr, mirror-maker (the notebook starts them; together about 5 GB of the Docker VM's memory) · Time: ~16 min

What you'll do: put the release's topics and ACLs in place, run the check, fix each finding the way the course taught, and run it again until it's green.

Step 1: The first review

The setup creates the release's topics on the primary cluster, exactly as the file says, and the secure broker's ACLs as they are today. Then:

area      check                              result  detail
--------  ---------------------------------  ------  --------------------------------------------------------------------------------
cluster   3 or more brokers                  PASS    3 brokers
...
topics    billing.session-charges            FAIL    min.insync.replicas=1: one broker's disk is the only copy (lesson 2.2)
topics    charging.charger-status            FAIL    unclean.leader.election.enable=true: a stale replica may become leader (lesson 4.1)
clients   gateway                            FAIL    producer acks=1: a leader failure can lose writes (lesson 2.2); producer without idempotence: retries can duplicate (lesson 2.2)
clients   billing                            FAIL    reads read_uncommitted next to a transactional producer (lesson 7.1)
clients   grid                               FAIL    auto-commit stores offsets before processing: a crash skips records (lesson 3.2)
security  deny by default                    PASS    StandardAuthorizer
security  no wildcard ACLs                   FAIL    User:svc-grid ALL on TOPIC * (lesson 8.2)
...
alerts    partitions under min ISR           FAIL    no alert rule (lesson 8.3)
...
dr        MirrorMaker 2 is copying           PASS    last heartbeat 1 s ago
dr        every topic has a copy             PASS    4 topics copied to the dr cluster

readiness: 15 passed, 7 failed, 0 skipped
exit code 1: a CI pipeline would stop the release

Every failure came with a reason someone once found convincing. The release file says min.insync.replicas: 1 # "billing must never stop writing during a broker restart".

Step 2: Fix the topics

The file first, then the cluster, as lesson 8.4's apply() does:

code/module-09/03-production-readiness-review.ipynb
release_topics["topics"]["billing.session-charges"]["config"]["min.insync.replicas"] = 2
del release_topics["topics"]["charging.charger-status"]["config"]["unclean.leader.election.enable"]

Step 3: Fix the client settings

code/module-09/03-production-readiness-review.ipynb
clients["services"]["gateway"]["producer"].update({"acks": "all", "enable.idempotence": True})
clients["services"]["billing"]["consumer"]["isolation.level"] = "read_committed"
clients["services"]["grid"]["consumer"]["enable.auto.offset.store"] = False

Step 4: Fix the ACLs

Grid reporting's wildcard is replaced by the three ACLs of exercise 8.1:

removed 1 ACL
added 3 ACLs

Step 5: Add the missing alert

code/module-09/03-production-readiness-review.ipynb
rule = {"alert": "KafkaUnderMinIsrPartitions",
        "expr": "sum(kafka_server_replicamanager_underminisrpartitioncount) > 0",

Step 6: Review again

topics    billing.session-charges            PASS    6 partitions, 3 replicas, min ISR 2
topics    charging.charger-status            PASS    3 partitions, 3 replicas, min ISR 2
clients   gateway                            PASS    settings as in modules 2, 3 and 8
clients   billing                            PASS    settings as in modules 2, 3 and 8
clients   grid                               PASS    settings as in modules 2, 3 and 8
security  no wildcard ACLs                   PASS
alerts    partitions under min ISR           PASS    KafkaUnderMinIsrPartitions
...
readiness: 22 passed, 0 failed, 0 skipped
exit code 0: the release may go ahead

What just happened

The check read the release's files and the running lab, and each finding mapped to a lesson. Each reason was plausible, and each was wrong:

The wildcard ACL was lesson 8.2's incident waiting to happen again, and without an under-min-ISR alert the gateway could stop writing with nobody paged.

The fixes changed files first, then the cluster: the files are the record of what's intended, so the next deployment doesn't undo the fix. The second review passed all 22 checks and exited with code 0, the signal a pipeline uses to let the release through. In the break-it step, the same check caught a broker that didn't come back: run nightly, it finds drift as well as bad releases. What the check can't see is still yours to review: rehearsed runbooks, on-call, and a load test at the growth plan's peak.

Break it: a bad night

The check isn't only for releases. Run every night against production, it notices what changed since. Predict first: kafka-3 doesn't come back after maintenance. What does the review say?

area      check                              result  detail
--------  ---------------------------------  ------  ---------------------------------------------------
cluster   3 or more brokers                  FAIL    2 brokers
...
readiness: 21 passed, 1 failed, 0 skipped
exit code 1: a CI pipeline would stop the release

Clean up

The last cell removes the release's topics and their copies, the ACLs and the extra alert rule. This is the course's last practical. To stop the lab: docker compose --profile secure --profile monitoring --profile dr stop in the code/ folder; to delete it with all its data, docker compose --profile secure --profile monitoring --profile dr --profile connect --profile schemas down -v.

Key takeaways

Next: Module 9 exercises