Notebook · code/module-08/02-authorization-and-quotas.ipynb

Download 02-authorization-and-quotas.ipynb

Open it in JupyterLab or VS Code from the lab folder to run it. This page is a read-only view.

Authorization and quotas — practical

Billing logs in to the secure broker as svc-billing and finds it may do nothing. You add its permissions one failure at a time, check what it still can't do, then put a byte-rate quota on the gateway while a runaway charger floods it. Last, you replay the grid incident.

Lab services: kafka-secure (docker compose --profile secure up -d; the notebook starts it and sets it up as in lesson 8.1 if needed) · Lesson: modules/08-security-and-operations/02-authorization-and-quotas.md

# Setup: find the lab's shared helpers
import sys, pathlib
lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / "common").is_dir())
sys.path.insert(0, str(lab_root))

import json, logging, time
from confluent_kafka import Consumer, KafkaException, Producer
from confluent_kafka.admin import (AclBinding, AclBindingFilter, AclOperation, AclPermissionType,
                                   AdminClient, NewTopic, ResourcePatternType, ResourceType)
from common import lab, secure
from common.scenario import Brightlane, to_json

quiet = logging.getLogger("quiet")            # the notebook prints the errors; mute the client's own log
quiet.addHandler(logging.NullHandler())
quiet.propagate = False

Step 1: Topics and sessions, as admin

secure.ensure_ready() checks that the secure broker runs with its certificate and users, and returns an admin client. Admin creates the topics and writes 20 charging sessions.

admin = secure.ensure_ready()
topics = ["charging.sessions", "billing.session-charges", "charging.telemetry"]
existing = admin.list_topics(timeout=10).topics
for name in [t for t in topics if t in existing]:
    admin.delete_topics([name])[name].result()
time.sleep(2)
for name, future in admin.create_topics([NewTopic(t, 3, 1) for t in topics]).items():
    future.result()

events = list(Brightlane().stream(sessions=20, readings=False))
producer = Producer(secure.client_config(*secure.ADMIN))
for key, event in events:
    producer.produce("charging.sessions", key=key, value=to_json(event))
producer.flush(10)
print(len(events), "session events in charging.sessions")

Step 2: Billing, one missing permission at a time

run_billing() does what lesson 7.1's billing service does, once: it reads the sessions as svc-billing, writes a charge per finished session and commits the charges and its position in one transaction. It returns the charge count, or the first error it meets.

def run_billing(price_eur_per_kwh=0.39):
    config = {**secure.client_config("svc-billing", secure.USERS["svc-billing"]), "logger": quiet}
    failed = []
    producer = Producer({**config, "transactional.id": "billing-1"})
    consumer = Consumer({**config, "group.id": "billing", "auto.offset.reset": "earliest",
                         "enable.auto.commit": False, "isolation.level": "read_committed"})
    try:
        producer.init_transactions(15)
        consumer.subscribe(["charging.sessions"])
        records, until = [], time.monotonic() + 20
        while len(records) < len(events) and time.monotonic() < until:
            for record in consumer.consume(50, 1.0):
                if record.error():
                    raise KafkaException(record.error())
                records.append(record)
        producer.begin_transaction()
        charged = 0
        for record in records:
            event = json.loads(record.value())
            if event["type"] == "session_stopped":
                charge = {"session_id": event["session_id"],
                          "amount_eur": round(event["energy_kwh"] * price_eur_per_kwh, 2)}
                producer.produce("billing.session-charges", key=record.key(), value=json.dumps(charge),
                                 on_delivery=lambda err, msg: failed.append(err.str()) if err else None)
                charged += 1
        producer.send_offsets_to_transaction(consumer.position(consumer.assignment()),
                                             consumer.consumer_group_metadata())
        producer.commit_transaction()
        return f"charged {charged} sessions"
    except KafkaException as e:
        return f"charges not written: {failed[0]}" if failed else e.args[0].str()
    finally:
        consumer.close()

print(f"  {'no ACLs':38} -> {run_billing()}")

Logged in, but allowed nothing: the broker denies everything that no ACL allows. Each ACL below allows one operation on one resource for User:svc-billing. The notebook adds them one at a time and runs billing after each.

BILLING = "User:svc-billing"

def allow(principal, resource_type, name, operation, pattern=ResourcePatternType.LITERAL):
    return AclBinding(resource_type, name, pattern, principal, "*", operation, AclPermissionType.ALLOW)

needed = [
    ("WRITE transactional ids billing-*", allow(BILLING, ResourceType.TRANSACTIONAL_ID, "billing-",
                                                AclOperation.WRITE, ResourcePatternType.PREFIXED)),
    ("READ group billing", allow(BILLING, ResourceType.GROUP, "billing", AclOperation.READ)),
    ("READ topic charging.sessions", allow(BILLING, ResourceType.TOPIC, "charging.sessions", AclOperation.READ)),
    ("WRITE topic billing.session-charges", allow(BILLING, ResourceType.TOPIC, "billing.session-charges",
                                                  AclOperation.WRITE)),
]
for label, binding in needed:
    admin.create_acls([binding])[binding].result()
    time.sleep(1)
    print(f"+ {label:38} -> {run_billing()}")

The ACLs as the broker stores them:

def acls_of(principal):
    match = AclBindingFilter(ResourceType.ANY, None, ResourcePatternType.ANY, principal, None,
                             AclOperation.ANY, AclPermissionType.ANY)
    return admin.describe_acls(match).result()

lab.table([(b.restype.name, b.name, b.resource_pattern_type.name, b.operation.name, b.permission_type.name)
           for b in acls_of(BILLING)], ["resource", "name", "pattern", "operation", "permission"])

Step 3: What billing still can't do

Least privilege means billing can do its job and nothing else. Three things it has no ACL for:

config = {**secure.client_config("svc-billing", secure.USERS["svc-billing"]), "logger": quiet}
billing_admin = AdminClient(config)

report = []
p = Producer({**config, "message.timeout.ms": 8000})
p.produce("charging.sessions", value=b"{}", on_delivery=lambda err, msg: report.append(err.str() if err else "written"))
p.flush(12)
print("write to charging.sessions: ", report[0])
try:
    billing_admin.delete_topics(["charging.sessions"], operation_timeout=15)["charging.sessions"].result()
    print("delete charging.sessions:   deleted")
except KafkaException as e:
    print("delete charging.sessions:  ", e.args[0].str())
print("topics billing can see:     ", sorted(billing_admin.list_topics(timeout=10).topics))

Billing doesn't even see charging.telemetry in the topic list: listing needs the DESCRIBE operation, which READ and WRITE include.

Step 4: A runaway charger and a quota

A charger on a debug firmware build sends 2,000 readings a second, each with a 320-character trace attached. The gateway forwards them all as svc-gateway. runaway_burst() replays 16 seconds of that and shows, every 2 seconds, how many KB/s reached the broker and how many records wait in the gateway's producer.

GATEWAY = "User:svc-gateway"
gateway_write = allow(GATEWAY, ResourceType.TOPIC, "charging.telemetry", AclOperation.WRITE)
admin.create_acls([gateway_write])[gateway_write].result()
time.sleep(1)
reading = next(event for key, event in Brightlane().stream(sessions=1) if "power_kw" in event)
value = to_json({**reading, "charger_id": "BL-V02-01", "debug_trace": "0" * 320})
print(f"one reading: {len(value)} bytes; 2,000 a second = {2000 * len(value) / 1024:.0f} KB/s")

def runaway_burst(seconds=16, rate=2000):
    delivered, throttles = [0], []
    producer = Producer({**secure.client_config("svc-gateway", secure.USERS["svc-gateway"]),
                         "linger.ms": 20,
                         "throttle_cb": lambda event: throttles.append(event.throttle_time)})
    def on_delivery(err, msg):
        if not err:
            delivered[0] += 1
    started, sent, last, rows = time.monotonic(), 0, (0, 0), []
    while len(rows) < seconds // 2:
        elapsed = time.monotonic() - started
        while sent < min(elapsed, seconds) * rate:
            producer.produce("charging.telemetry", key="BL-V02-01", value=value, on_delivery=on_delivery)
            sent += 1
        producer.poll(0.01)
        if elapsed >= (len(rows) + 1) * 2:              # every 2 seconds: one row
            rate_kb = (delivered[0] - last[1]) * len(value) / 1024 / (elapsed - last[0])
            rows.append((f"{len(rows) * 2}-{len(rows) * 2 + 2} s", f"{rate_kb:.0f}", sent - delivered[0]))
            last = (elapsed, delivered[0])
    average_kb = delivered[0] * len(value) / 1024 / last[0]
    flush_started = time.monotonic()
    producer.flush(120)
    lab.table(rows, ["time", "KB/s delivered", "waiting in the producer"])
    print(f"average over {seconds} s: {average_kb:.0f} KB/s")
    print(f"{sent} readings; the last ones waited {time.monotonic() - flush_started:.1f} s more in the producer")
    print(f"throttled {len(throttles)} times, longest {max(throttles, default=0) * 1000:.0f} ms")

runaway_burst()

Now a quota: at most 512 KB/s of produce traffic for svc-gateway, per broker. The Python client has no API for quotas, so admin sets it with kafka-configs.sh inside the broker's container.

def kafka_configs(*args):
    return lab.kafka_cli("kafka-configs.sh", "--bootstrap-server", "127.0.0.1:9095",
                         "--command-config", "/etc/kafka/secrets/admin.properties",
                         "--entity-type", "users", "--entity-name", "svc-gateway", *args,
                         node="kafka-secure", echo=False)

kafka_configs("--alter", "--add-config", "producer_byte_rate=524288")
print(kafka_configs("--describe"))
time.sleep(2)
runaway_burst()

Nothing failed. The broker accepted every request, told the producer how long to hold back, and stopped reading from its connection for that time. The backlog moved into the gateway's producer.

Break it: the grid incident

The grid team asked for "access to the topics" and got ALL on every topic (*). A bug in their clean-up job then deleted the wrong topic. Replay it:

GRID = "User:svc-grid"
everything = allow(GRID, ResourceType.TOPIC, "*", AclOperation.ALL)
admin.create_acls([everything])[everything].result()
time.sleep(1)
grid_admin = AdminClient(secure.client_config("svc-grid", secure.USERS["svc-grid"]))

def grid_deletes(topic):
    try:
        grid_admin.delete_topics([topic], operation_timeout=15)[topic].result()
        return "deleted"
    except KafkaException as e:
        return e.args[0].str()

print("with ALL on *:      grid deletes charging.telemetry ->", grid_deletes("charging.telemetry"))
removed = admin.delete_acls([AclBindingFilter(ResourceType.ANY, None, ResourcePatternType.ANY, GRID, None,
                                              AclOperation.ANY, AclPermissionType.ANY)])
print("removed", sum(len(f.result()) for f in removed.values()), "ACL of svc-grid")
time.sleep(1)
print("without that ACL:   grid deletes charging.sessions  ->", grid_deletes("charging.sessions"))

One wildcard ACL turned a bug in one service into an outage of the gateway's main topic. Exercise 1 gives svc-grid the permissions it really needs.

Clean up

Remove the ACLs, the quota and the topics. The secure broker keeps running for the exercises; to stop it: docker compose --profile secure stop.

for principal in (BILLING, GATEWAY):
    for future in admin.delete_acls([AclBindingFilter(ResourceType.ANY, None, ResourcePatternType.ANY, principal,
                                                      None, AclOperation.ANY, AclPermissionType.ANY)]).values():
        future.result()
kafka_configs("--alter", "--delete-config", "producer_byte_rate")
remaining = [t for t in topics if t in admin.list_topics(timeout=10).topics]
for name, future in admin.delete_topics(remaining).items():
    future.result()
print("cleaned up")