Notebook · code/module-08/01-encryption-and-authentication.ipynb

Download 01-encryption-and-authentication.ipynb

Open it in JupyterLab or VS Code from the lab folder to run it. This page is a read-only view.

Encryption and authentication — practical

The lab's secure broker only accepts TLS connections with a SCRAM login. It starts without a certificate and waits. You make a certificate authority and the broker's certificate, create users for Brightlane's services, connect, and then try three clients that get it wrong.

Lab services: kafka-secure (docker compose --profile secure up -d; the notebook starts it) · Lesson: modules/08-security-and-operations/01-encryption-and-authentication.md

# Setup: find the lab's shared helpers
import sys, pathlib
lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / "common").is_dir())
sys.path.insert(0, str(lab_root))

import logging, time
from common import settings, lab, secure

Step 1: A broker without a certificate

The notebook removes any earlier secure broker and starts a fresh one. Its start script waits for a certificate.

secure.reset_broker()
lab.compose("--profile", "secure", "up", "-d", "kafka-secure")
time.sleep(5)
print(lab.compose("--profile", "secure", "logs", "--tail", "1", "kafka-secure", echo=False))

Step 2: A certificate authority and the broker's certificate

secure.make_certificates() creates a CA (a key pair and a self-signed certificate) and a certificate for the broker, signed by the CA. The broker's certificate lists the names clients may use to reach it.

from cryptography import x509

ca_pem, server_pem = secure.make_certificates()
ca = x509.load_pem_x509_certificate(ca_pem)
server = x509.load_pem_x509_certificates(server_pem)[0]
print("CA:         ", ca.subject.rfc4514_string())
print("broker:     ", server.subject.rfc4514_string(), "| signed by:", server.issuer.rfc4514_string())
print("valid until:", server.not_valid_after_utc.date())
print("names:      ", [str(n.value) for n in server.extensions.get_extension_for_class(x509.SubjectAlternativeName).value])

The broker gets its key and certificate; clients only get the CA certificate, to check the broker's. Then the broker starts.

secure.install_certificates(ca_pem, server_pem)
started = time.monotonic()
lab.compose("--profile", "secure", "up", "-d", "--wait", "kafka-secure")
print(f"broker ready after {time.monotonic() - started:.0f} s")
log = lab.compose("--profile", "secure", "logs", "--no-log-prefix", "kafka-secure", echo=False)
shown = dict.fromkeys(line.strip() for line in log.splitlines()
                      if line.strip().startswith(("listeners =", "listener.security.protocol.map =")))
print("\n".join(shown))

Step 3: Log in as admin, create the services' users

The admin's SCRAM credentials were added when the broker formatted its storage. Admin creates one user per service.

from confluent_kafka.admin import AdminClient, NewTopic

admin = AdminClient(secure.client_config(*secure.ADMIN))
cluster = admin.describe_cluster(request_timeout=15).result()
print("connected over SASL_SSL to cluster", cluster.cluster_id)
secure.create_users(admin)
for user, future in admin.describe_user_scram_credentials(list(secure.USERS)).items():
    print(user, [(info.mechanism.name, info.iterations) for info in future.result().scram_credential_infos])

The broker stores a salted hash per user, never the password, and shows only the mechanism and the number of hash iterations.

Step 4: Write and read, as admin and as a service

from confluent_kafka import Producer

admin.create_topics([NewTopic("charging.telemetry", 3, 1)])["charging.telemetry"].result()

def write_one(config):
    report = []
    producer = Producer({**config, "message.timeout.ms": 10000})
    producer.produce("charging.telemetry", key=b"BL-N03-04", value=b'{"power_kw": 48.2}',
                     on_delivery=lambda err, msg: report.append(err.str() if err else f"stored at offset {msg.offset()}"))
    producer.flush(15)
    return report[0]

print("admin:      ", write_one(secure.client_config(*secure.ADMIN)))
print("svc-gateway:", write_one(secure.client_config("svc-gateway", secure.USERS["svc-gateway"])))

svc-gateway logged in, so the broker knows who it is: authentication worked. But no rule allows it to write anything yet. That's authorization, lesson 8.2.

Break it: three clients that get it wrong

why() tries to connect for a few seconds and returns what the client reports: its first log line about the connection if there is one, otherwise its first error.

def why(config, seconds=6):
    logged, errors = [], []
    class Keep(logging.Handler):
        def emit(self, record):
            logged.append(record.getMessage().split("/bootstrap: ")[-1])
    log = logging.getLogger(f"why-{time.monotonic()}")
    log.setLevel(logging.INFO)                      # connection errors are logged at INFO
    log.addHandler(Keep())
    log.propagate = False
    producer = Producer({**config, "logger": log, "error_cb": lambda err: errors.append(err.str())})
    until = time.monotonic() + seconds
    while time.monotonic() < until and not logged:
        producer.poll(0.5)
    return (logged or errors or ["no problem"])[0]

good = secure.client_config("svc-gateway", secure.USERS["svc-gateway"])
print("wrong password:  ", why({**good, "sasl.password": "guess"}))
print("plaintext client:", why({"bootstrap.servers": settings.SECURE_BOOTSTRAP}))
print("CA not trusted:  ", why({k: v for k, v in good.items() if k != "ssl.ca.location"}))

Clean up

Delete the topic. The secure broker keeps running: lesson 8.2 continues with it. To stop it: docker compose --profile secure stop.

admin.delete_topics(["charging.telemetry"])["charging.telemetry"].result()
print("cleaned up")