Download 02-authorization-and-quotas.ipynb
Open it in JupyterLab or VS Code from the lab folder to run it. This page is a read-only view.
Authorization and quotas — practical
Billing logs in to the secure broker as svc-billing and finds it may do nothing. You add its
permissions one failure at a time, check what it still can't do, then put a byte-rate quota on
the gateway while a runaway charger floods it. Last, you replay the grid incident.
Lab services: kafka-secure (docker compose --profile secure up -d; the notebook starts it
and sets it up as in lesson 8.1 if needed)
· Lesson: modules/08-security-and-operations/02-authorization-and-quotas.md
# Setup: find the lab's shared helpers import sys, pathlib lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / "common").is_dir()) sys.path.insert(0, str(lab_root)) import json, logging, time from confluent_kafka import Consumer, KafkaException, Producer from confluent_kafka.admin import (AclBinding, AclBindingFilter, AclOperation, AclPermissionType, AdminClient, NewTopic, ResourcePatternType, ResourceType) from common import lab, secure from common.scenario import Brightlane, to_json quiet = logging.getLogger("quiet") # the notebook prints the errors; mute the client's own log quiet.addHandler(logging.NullHandler()) quiet.propagate = False
Step 1: Topics and sessions, as admin
secure.ensure_ready() checks that the secure broker runs with its certificate and users, and
returns an admin client. Admin creates the topics and writes 20 charging sessions.
admin = secure.ensure_ready() topics = ["charging.sessions", "billing.session-charges", "charging.telemetry"] existing = admin.list_topics(timeout=10).topics for name in [t for t in topics if t in existing]: admin.delete_topics([name])[name].result() time.sleep(2) for name, future in admin.create_topics([NewTopic(t, 3, 1) for t in topics]).items(): future.result() events = list(Brightlane().stream(sessions=20, readings=False)) producer = Producer(secure.client_config(*secure.ADMIN)) for key, event in events: producer.produce("charging.sessions", key=key, value=to_json(event)) producer.flush(10) print(len(events), "session events in charging.sessions")
Step 2: Billing, one missing permission at a time
run_billing() does what lesson 7.1's billing service does, once: it reads the sessions as
svc-billing, writes a charge per finished session and commits the charges and its position in
one transaction. It returns the charge count, or the first error it meets.
def run_billing(price_eur_per_kwh=0.39): config = {**secure.client_config("svc-billing", secure.USERS["svc-billing"]), "logger": quiet} failed = [] producer = Producer({**config, "transactional.id": "billing-1"}) consumer = Consumer({**config, "group.id": "billing", "auto.offset.reset": "earliest", "enable.auto.commit": False, "isolation.level": "read_committed"}) try: producer.init_transactions(15) consumer.subscribe(["charging.sessions"]) records, until = [], time.monotonic() + 20 while len(records) < len(events) and time.monotonic() < until: for record in consumer.consume(50, 1.0): if record.error(): raise KafkaException(record.error()) records.append(record) producer.begin_transaction() charged = 0 for record in records: event = json.loads(record.value()) if event["type"] == "session_stopped": charge = {"session_id": event["session_id"], "amount_eur": round(event["energy_kwh"] * price_eur_per_kwh, 2)} producer.produce("billing.session-charges", key=record.key(), value=json.dumps(charge), on_delivery=lambda err, msg: failed.append(err.str()) if err else None) charged += 1 producer.send_offsets_to_transaction(consumer.position(consumer.assignment()), consumer.consumer_group_metadata()) producer.commit_transaction() return f"charged {charged} sessions" except KafkaException as e: return f"charges not written: {failed[0]}" if failed else e.args[0].str() finally: consumer.close() print(f" {'no ACLs':38} -> {run_billing()}")
Logged in, but allowed nothing: the broker denies everything that no ACL allows. Each ACL below
allows one operation on one resource for User:svc-billing. The notebook adds them one at a
time and runs billing after each.
BILLING = "User:svc-billing" def allow(principal, resource_type, name, operation, pattern=ResourcePatternType.LITERAL): return AclBinding(resource_type, name, pattern, principal, "*", operation, AclPermissionType.ALLOW) needed = [ ("WRITE transactional ids billing-*", allow(BILLING, ResourceType.TRANSACTIONAL_ID, "billing-", AclOperation.WRITE, ResourcePatternType.PREFIXED)), ("READ group billing", allow(BILLING, ResourceType.GROUP, "billing", AclOperation.READ)), ("READ topic charging.sessions", allow(BILLING, ResourceType.TOPIC, "charging.sessions", AclOperation.READ)), ("WRITE topic billing.session-charges", allow(BILLING, ResourceType.TOPIC, "billing.session-charges", AclOperation.WRITE)), ] for label, binding in needed: admin.create_acls([binding])[binding].result() time.sleep(1) print(f"+ {label:38} -> {run_billing()}")
The ACLs as the broker stores them:
def acls_of(principal): match = AclBindingFilter(ResourceType.ANY, None, ResourcePatternType.ANY, principal, None, AclOperation.ANY, AclPermissionType.ANY) return admin.describe_acls(match).result() lab.table([(b.restype.name, b.name, b.resource_pattern_type.name, b.operation.name, b.permission_type.name) for b in acls_of(BILLING)], ["resource", "name", "pattern", "operation", "permission"])
Step 3: What billing still can't do
Least privilege means billing can do its job and nothing else. Three things it has no ACL for:
config = {**secure.client_config("svc-billing", secure.USERS["svc-billing"]), "logger": quiet} billing_admin = AdminClient(config) report = [] p = Producer({**config, "message.timeout.ms": 8000}) p.produce("charging.sessions", value=b"{}", on_delivery=lambda err, msg: report.append(err.str() if err else "written")) p.flush(12) print("write to charging.sessions: ", report[0]) try: billing_admin.delete_topics(["charging.sessions"], operation_timeout=15)["charging.sessions"].result() print("delete charging.sessions: deleted") except KafkaException as e: print("delete charging.sessions: ", e.args[0].str()) print("topics billing can see: ", sorted(billing_admin.list_topics(timeout=10).topics))
Billing doesn't even see charging.telemetry in the topic list: listing needs the DESCRIBE
operation, which READ and WRITE include.
Step 4: A runaway charger and a quota
A charger on a debug firmware build sends 2,000 readings a second, each with a 320-character
trace attached. The gateway forwards them all as svc-gateway. runaway_burst() replays
16 seconds of that and shows, every 2 seconds, how many KB/s reached the broker and how many
records wait in the gateway's producer.
GATEWAY = "User:svc-gateway" gateway_write = allow(GATEWAY, ResourceType.TOPIC, "charging.telemetry", AclOperation.WRITE) admin.create_acls([gateway_write])[gateway_write].result() time.sleep(1) reading = next(event for key, event in Brightlane().stream(sessions=1) if "power_kw" in event) value = to_json({**reading, "charger_id": "BL-V02-01", "debug_trace": "0" * 320}) print(f"one reading: {len(value)} bytes; 2,000 a second = {2000 * len(value) / 1024:.0f} KB/s") def runaway_burst(seconds=16, rate=2000): delivered, throttles = [0], [] producer = Producer({**secure.client_config("svc-gateway", secure.USERS["svc-gateway"]), "linger.ms": 20, "throttle_cb": lambda event: throttles.append(event.throttle_time)}) def on_delivery(err, msg): if not err: delivered[0] += 1 started, sent, last, rows = time.monotonic(), 0, (0, 0), [] while len(rows) < seconds // 2: elapsed = time.monotonic() - started while sent < min(elapsed, seconds) * rate: producer.produce("charging.telemetry", key="BL-V02-01", value=value, on_delivery=on_delivery) sent += 1 producer.poll(0.01) if elapsed >= (len(rows) + 1) * 2: # every 2 seconds: one row rate_kb = (delivered[0] - last[1]) * len(value) / 1024 / (elapsed - last[0]) rows.append((f"{len(rows) * 2}-{len(rows) * 2 + 2} s", f"{rate_kb:.0f}", sent - delivered[0])) last = (elapsed, delivered[0]) average_kb = delivered[0] * len(value) / 1024 / last[0] flush_started = time.monotonic() producer.flush(120) lab.table(rows, ["time", "KB/s delivered", "waiting in the producer"]) print(f"average over {seconds} s: {average_kb:.0f} KB/s") print(f"{sent} readings; the last ones waited {time.monotonic() - flush_started:.1f} s more in the producer") print(f"throttled {len(throttles)} times, longest {max(throttles, default=0) * 1000:.0f} ms") runaway_burst()
Now a quota: at most 512 KB/s of produce traffic for svc-gateway, per broker. The Python client
has no API for quotas, so admin sets it with kafka-configs.sh inside the broker's container.
def kafka_configs(*args): return lab.kafka_cli("kafka-configs.sh", "--bootstrap-server", "127.0.0.1:9095", "--command-config", "/etc/kafka/secrets/admin.properties", "--entity-type", "users", "--entity-name", "svc-gateway", *args, node="kafka-secure", echo=False) kafka_configs("--alter", "--add-config", "producer_byte_rate=524288") print(kafka_configs("--describe")) time.sleep(2) runaway_burst()
Nothing failed. The broker accepted every request, told the producer how long to hold back, and stopped reading from its connection for that time. The backlog moved into the gateway's producer.
Break it: the grid incident
The grid team asked for "access to the topics" and got ALL on every topic (*). A bug in their
clean-up job then deleted the wrong topic. Replay it:
GRID = "User:svc-grid" everything = allow(GRID, ResourceType.TOPIC, "*", AclOperation.ALL) admin.create_acls([everything])[everything].result() time.sleep(1) grid_admin = AdminClient(secure.client_config("svc-grid", secure.USERS["svc-grid"])) def grid_deletes(topic): try: grid_admin.delete_topics([topic], operation_timeout=15)[topic].result() return "deleted" except KafkaException as e: return e.args[0].str() print("with ALL on *: grid deletes charging.telemetry ->", grid_deletes("charging.telemetry")) removed = admin.delete_acls([AclBindingFilter(ResourceType.ANY, None, ResourcePatternType.ANY, GRID, None, AclOperation.ANY, AclPermissionType.ANY)]) print("removed", sum(len(f.result()) for f in removed.values()), "ACL of svc-grid") time.sleep(1) print("without that ACL: grid deletes charging.sessions ->", grid_deletes("charging.sessions"))
One wildcard ACL turned a bug in one service into an outage of the gateway's main topic. Exercise
1 gives svc-grid the permissions it really needs.
Clean up
Remove the ACLs, the quota and the topics. The secure broker keeps running for the exercises; to
stop it: docker compose --profile secure stop.
for principal in (BILLING, GATEWAY): for future in admin.delete_acls([AclBindingFilter(ResourceType.ANY, None, ResourcePatternType.ANY, principal, None, AclOperation.ANY, AclPermissionType.ANY)]).values(): future.result() kafka_configs("--alter", "--delete-config", "producer_byte_rate") remaining = [t for t in topics if t in admin.list_topics(timeout=10).topics] for name, future in admin.delete_topics(remaining).items(): future.result() print("cleaned up")