Download 01-encryption-and-authentication.ipynb
Open it in JupyterLab or VS Code from the lab folder to run it. This page is a read-only view.
Encryption and authentication — practical
The lab's secure broker only accepts TLS connections with a SCRAM login. It starts without a certificate and waits. You make a certificate authority and the broker's certificate, create users for Brightlane's services, connect, and then try three clients that get it wrong.
Lab services: kafka-secure (docker compose --profile secure up -d; the notebook starts it)
· Lesson: modules/08-security-and-operations/01-encryption-and-authentication.md
# Setup: find the lab's shared helpers import sys, pathlib lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / "common").is_dir()) sys.path.insert(0, str(lab_root)) import logging, time from common import settings, lab, secure
Step 1: A broker without a certificate
The notebook removes any earlier secure broker and starts a fresh one. Its start script waits for a certificate.
secure.reset_broker() lab.compose("--profile", "secure", "up", "-d", "kafka-secure") time.sleep(5) print(lab.compose("--profile", "secure", "logs", "--tail", "1", "kafka-secure", echo=False))
Step 2: A certificate authority and the broker's certificate
secure.make_certificates() creates a CA (a key pair and a self-signed certificate) and a
certificate for the broker, signed by the CA. The broker's certificate lists the names clients
may use to reach it.
from cryptography import x509 ca_pem, server_pem = secure.make_certificates() ca = x509.load_pem_x509_certificate(ca_pem) server = x509.load_pem_x509_certificates(server_pem)[0] print("CA: ", ca.subject.rfc4514_string()) print("broker: ", server.subject.rfc4514_string(), "| signed by:", server.issuer.rfc4514_string()) print("valid until:", server.not_valid_after_utc.date()) print("names: ", [str(n.value) for n in server.extensions.get_extension_for_class(x509.SubjectAlternativeName).value])
The broker gets its key and certificate; clients only get the CA certificate, to check the broker's. Then the broker starts.
secure.install_certificates(ca_pem, server_pem) started = time.monotonic() lab.compose("--profile", "secure", "up", "-d", "--wait", "kafka-secure") print(f"broker ready after {time.monotonic() - started:.0f} s") log = lab.compose("--profile", "secure", "logs", "--no-log-prefix", "kafka-secure", echo=False) shown = dict.fromkeys(line.strip() for line in log.splitlines() if line.strip().startswith(("listeners =", "listener.security.protocol.map ="))) print("\n".join(shown))
Step 3: Log in as admin, create the services' users
The admin's SCRAM credentials were added when the broker formatted its storage. Admin creates one user per service.
from confluent_kafka.admin import AdminClient, NewTopic admin = AdminClient(secure.client_config(*secure.ADMIN)) cluster = admin.describe_cluster(request_timeout=15).result() print("connected over SASL_SSL to cluster", cluster.cluster_id) secure.create_users(admin) for user, future in admin.describe_user_scram_credentials(list(secure.USERS)).items(): print(user, [(info.mechanism.name, info.iterations) for info in future.result().scram_credential_infos])
The broker stores a salted hash per user, never the password, and shows only the mechanism and the number of hash iterations.
Step 4: Write and read, as admin and as a service
from confluent_kafka import Producer admin.create_topics([NewTopic("charging.telemetry", 3, 1)])["charging.telemetry"].result() def write_one(config): report = [] producer = Producer({**config, "message.timeout.ms": 10000}) producer.produce("charging.telemetry", key=b"BL-N03-04", value=b'{"power_kw": 48.2}', on_delivery=lambda err, msg: report.append(err.str() if err else f"stored at offset {msg.offset()}")) producer.flush(15) return report[0] print("admin: ", write_one(secure.client_config(*secure.ADMIN))) print("svc-gateway:", write_one(secure.client_config("svc-gateway", secure.USERS["svc-gateway"])))
svc-gateway logged in, so the broker knows who it is: authentication worked. But no rule allows it to write anything yet. That's authorization, lesson 8.2.
Break it: three clients that get it wrong
why() tries to connect for a few seconds and returns what the client reports: its first log
line about the connection if there is one, otherwise its first error.
def why(config, seconds=6): logged, errors = [], [] class Keep(logging.Handler): def emit(self, record): logged.append(record.getMessage().split("/bootstrap: ")[-1]) log = logging.getLogger(f"why-{time.monotonic()}") log.setLevel(logging.INFO) # connection errors are logged at INFO log.addHandler(Keep()) log.propagate = False producer = Producer({**config, "logger": log, "error_cb": lambda err: errors.append(err.str())}) until = time.monotonic() + seconds while time.monotonic() < until and not logged: producer.poll(0.5) return (logged or errors or ["no problem"])[0] good = secure.client_config("svc-gateway", secure.USERS["svc-gateway"]) print("wrong password: ", why({**good, "sasl.password": "guess"})) print("plaintext client:", why({"bootstrap.servers": settings.SECURE_BOOTSTRAP})) print("CA not trusted: ", why({k: v for k, v in good.items() if k != "ssl.ca.location"}))
Clean up
Delete the topic. The secure broker keeps running: lesson 8.2 continues with it. To stop it:
docker compose --profile secure stop.
admin.delete_topics(["charging.telemetry"])["charging.telemetry"].result() print("cleaned up")