{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c00-0691b92c",
   "metadata": {},
   "source": [
    "# Authorization and quotas — practical\n",
    "\n",
    "Billing logs in to the secure broker as `svc-billing` and finds it may do nothing. You add its\n",
    "permissions one failure at a time, check what it still can't do, then put a byte-rate quota on\n",
    "the gateway while a runaway charger floods it. Last, you replay the grid incident.\n",
    "\n",
    "**Lab services:** kafka-secure (`docker compose --profile secure up -d`; the notebook starts it\n",
    "and sets it up as in lesson 8.1 if needed)\n",
    "· **Lesson:** `modules/08-security-and-operations/02-authorization-and-quotas.md`"
   ]
  },
  {
   "cell_type": "code",
   "id": "c01-93d724a1",
   "metadata": {},
   "source": [
    "# Setup: find the lab's shared helpers\n",
    "import sys, pathlib\n",
    "lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / \"common\").is_dir())\n",
    "sys.path.insert(0, str(lab_root))\n",
    "\n",
    "import json, logging, time\n",
    "from confluent_kafka import Consumer, KafkaException, Producer\n",
    "from confluent_kafka.admin import (AclBinding, AclBindingFilter, AclOperation, AclPermissionType,\n",
    "                                   AdminClient, NewTopic, ResourcePatternType, ResourceType)\n",
    "from common import lab, secure\n",
    "from common.scenario import Brightlane, to_json\n",
    "\n",
    "quiet = logging.getLogger(\"quiet\")            # the notebook prints the errors; mute the client's own log\n",
    "quiet.addHandler(logging.NullHandler())\n",
    "quiet.propagate = False"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c02-9b396f45",
   "metadata": {},
   "source": [
    "## Step 1: Topics and sessions, as admin\n",
    "`secure.ensure_ready()` checks that the secure broker runs with its certificate and users, and\n",
    "returns an admin client. Admin creates the topics and writes 20 charging sessions."
   ]
  },
  {
   "cell_type": "code",
   "id": "c03-143c4f52",
   "metadata": {},
   "source": [
    "admin = secure.ensure_ready()\n",
    "topics = [\"charging.sessions\", \"billing.session-charges\", \"charging.telemetry\"]\n",
    "existing = admin.list_topics(timeout=10).topics\n",
    "for name in [t for t in topics if t in existing]:\n",
    "    admin.delete_topics([name])[name].result()\n",
    "time.sleep(2)\n",
    "for name, future in admin.create_topics([NewTopic(t, 3, 1) for t in topics]).items():\n",
    "    future.result()\n",
    "\n",
    "events = list(Brightlane().stream(sessions=20, readings=False))\n",
    "producer = Producer(secure.client_config(*secure.ADMIN))\n",
    "for key, event in events:\n",
    "    producer.produce(\"charging.sessions\", key=key, value=to_json(event))\n",
    "producer.flush(10)\n",
    "print(len(events), \"session events in charging.sessions\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c04-de019f67",
   "metadata": {},
   "source": [
    "## Step 2: Billing, one missing permission at a time\n",
    "`run_billing()` does what lesson 7.1's billing service does, once: it reads the sessions as\n",
    "`svc-billing`, writes a charge per finished session and commits the charges and its position in\n",
    "one transaction. It returns the charge count, or the first error it meets."
   ]
  },
  {
   "cell_type": "code",
   "id": "c05-3020bc96",
   "metadata": {},
   "source": [
    "def run_billing(price_eur_per_kwh=0.39):\n",
    "    config = {**secure.client_config(\"svc-billing\", secure.USERS[\"svc-billing\"]), \"logger\": quiet}\n",
    "    failed = []\n",
    "    producer = Producer({**config, \"transactional.id\": \"billing-1\"})\n",
    "    consumer = Consumer({**config, \"group.id\": \"billing\", \"auto.offset.reset\": \"earliest\",\n",
    "                         \"enable.auto.commit\": False, \"isolation.level\": \"read_committed\"})\n",
    "    try:\n",
    "        producer.init_transactions(15)\n",
    "        consumer.subscribe([\"charging.sessions\"])\n",
    "        records, until = [], time.monotonic() + 20\n",
    "        while len(records) < len(events) and time.monotonic() < until:\n",
    "            for record in consumer.consume(50, 1.0):\n",
    "                if record.error():\n",
    "                    raise KafkaException(record.error())\n",
    "                records.append(record)\n",
    "        producer.begin_transaction()\n",
    "        charged = 0\n",
    "        for record in records:\n",
    "            event = json.loads(record.value())\n",
    "            if event[\"type\"] == \"session_stopped\":\n",
    "                charge = {\"session_id\": event[\"session_id\"],\n",
    "                          \"amount_eur\": round(event[\"energy_kwh\"] * price_eur_per_kwh, 2)}\n",
    "                producer.produce(\"billing.session-charges\", key=record.key(), value=json.dumps(charge),\n",
    "                                 on_delivery=lambda err, msg: failed.append(err.str()) if err else None)\n",
    "                charged += 1\n",
    "        producer.send_offsets_to_transaction(consumer.position(consumer.assignment()),\n",
    "                                             consumer.consumer_group_metadata())\n",
    "        producer.commit_transaction()\n",
    "        return f\"charged {charged} sessions\"\n",
    "    except KafkaException as e:\n",
    "        return f\"charges not written: {failed[0]}\" if failed else e.args[0].str()\n",
    "    finally:\n",
    "        consumer.close()\n",
    "\n",
    "print(f\"  {'no ACLs':38} -> {run_billing()}\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c06-1c4ea9c5",
   "metadata": {},
   "source": [
    "Logged in, but allowed nothing: the broker denies everything that no ACL allows. Each ACL below\n",
    "allows one operation on one resource for `User:svc-billing`. The notebook adds them one at a\n",
    "time and runs billing after each."
   ]
  },
  {
   "cell_type": "code",
   "id": "c07-a6aeb881",
   "metadata": {},
   "source": [
    "BILLING = \"User:svc-billing\"\n",
    "\n",
    "def allow(principal, resource_type, name, operation, pattern=ResourcePatternType.LITERAL):\n",
    "    return AclBinding(resource_type, name, pattern, principal, \"*\", operation, AclPermissionType.ALLOW)\n",
    "\n",
    "needed = [\n",
    "    (\"WRITE transactional ids billing-*\", allow(BILLING, ResourceType.TRANSACTIONAL_ID, \"billing-\",\n",
    "                                                AclOperation.WRITE, ResourcePatternType.PREFIXED)),\n",
    "    (\"READ group billing\", allow(BILLING, ResourceType.GROUP, \"billing\", AclOperation.READ)),\n",
    "    (\"READ topic charging.sessions\", allow(BILLING, ResourceType.TOPIC, \"charging.sessions\", AclOperation.READ)),\n",
    "    (\"WRITE topic billing.session-charges\", allow(BILLING, ResourceType.TOPIC, \"billing.session-charges\",\n",
    "                                                  AclOperation.WRITE)),\n",
    "]\n",
    "for label, binding in needed:\n",
    "    admin.create_acls([binding])[binding].result()\n",
    "    time.sleep(1)\n",
    "    print(f\"+ {label:38} -> {run_billing()}\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c08-359a3c70",
   "metadata": {},
   "source": [
    "The ACLs as the broker stores them:"
   ]
  },
  {
   "cell_type": "code",
   "id": "c09-73cdec95",
   "metadata": {},
   "source": [
    "def acls_of(principal):\n",
    "    match = AclBindingFilter(ResourceType.ANY, None, ResourcePatternType.ANY, principal, None,\n",
    "                             AclOperation.ANY, AclPermissionType.ANY)\n",
    "    return admin.describe_acls(match).result()\n",
    "\n",
    "lab.table([(b.restype.name, b.name, b.resource_pattern_type.name, b.operation.name, b.permission_type.name)\n",
    "           for b in acls_of(BILLING)], [\"resource\", \"name\", \"pattern\", \"operation\", \"permission\"])"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c10-fae6b08e",
   "metadata": {},
   "source": [
    "## Step 3: What billing still can't do\n",
    "Least privilege means billing can do its job and nothing else. Three things it has no ACL for:"
   ]
  },
  {
   "cell_type": "code",
   "id": "c11-99f47ff9",
   "metadata": {},
   "source": [
    "config = {**secure.client_config(\"svc-billing\", secure.USERS[\"svc-billing\"]), \"logger\": quiet}\n",
    "billing_admin = AdminClient(config)\n",
    "\n",
    "report = []\n",
    "p = Producer({**config, \"message.timeout.ms\": 8000})\n",
    "p.produce(\"charging.sessions\", value=b\"{}\", on_delivery=lambda err, msg: report.append(err.str() if err else \"written\"))\n",
    "p.flush(12)\n",
    "print(\"write to charging.sessions: \", report[0])\n",
    "try:\n",
    "    billing_admin.delete_topics([\"charging.sessions\"], operation_timeout=15)[\"charging.sessions\"].result()\n",
    "    print(\"delete charging.sessions:   deleted\")\n",
    "except KafkaException as e:\n",
    "    print(\"delete charging.sessions:  \", e.args[0].str())\n",
    "print(\"topics billing can see:     \", sorted(billing_admin.list_topics(timeout=10).topics))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c12-8b679623",
   "metadata": {},
   "source": [
    "Billing doesn't even see `charging.telemetry` in the topic list: listing needs the `DESCRIBE`\n",
    "operation, which `READ` and `WRITE` include.\n",
    "\n",
    "## Step 4: A runaway charger and a quota\n",
    "A charger on a debug firmware build sends 2,000 readings a second, each with a 320-character\n",
    "trace attached. The gateway forwards them all as `svc-gateway`. `runaway_burst()` replays\n",
    "16 seconds of that and shows, every 2 seconds, how many KB/s reached the broker and how many\n",
    "records wait in the gateway's producer."
   ]
  },
  {
   "cell_type": "code",
   "id": "c13-ba251cd8",
   "metadata": {},
   "source": [
    "GATEWAY = \"User:svc-gateway\"\n",
    "gateway_write = allow(GATEWAY, ResourceType.TOPIC, \"charging.telemetry\", AclOperation.WRITE)\n",
    "admin.create_acls([gateway_write])[gateway_write].result()\n",
    "time.sleep(1)\n",
    "reading = next(event for key, event in Brightlane().stream(sessions=1) if \"power_kw\" in event)\n",
    "value = to_json({**reading, \"charger_id\": \"BL-V02-01\", \"debug_trace\": \"0\" * 320})\n",
    "print(f\"one reading: {len(value)} bytes; 2,000 a second = {2000 * len(value) / 1024:.0f} KB/s\")\n",
    "\n",
    "def runaway_burst(seconds=16, rate=2000):\n",
    "    delivered, throttles = [0], []\n",
    "    producer = Producer({**secure.client_config(\"svc-gateway\", secure.USERS[\"svc-gateway\"]),\n",
    "                         \"linger.ms\": 20,\n",
    "                         \"throttle_cb\": lambda event: throttles.append(event.throttle_time)})\n",
    "    def on_delivery(err, msg):\n",
    "        if not err:\n",
    "            delivered[0] += 1\n",
    "    started, sent, last, rows = time.monotonic(), 0, (0, 0), []\n",
    "    while len(rows) < seconds // 2:\n",
    "        elapsed = time.monotonic() - started\n",
    "        while sent < min(elapsed, seconds) * rate:\n",
    "            producer.produce(\"charging.telemetry\", key=\"BL-V02-01\", value=value, on_delivery=on_delivery)\n",
    "            sent += 1\n",
    "        producer.poll(0.01)\n",
    "        if elapsed >= (len(rows) + 1) * 2:              # every 2 seconds: one row\n",
    "            rate_kb = (delivered[0] - last[1]) * len(value) / 1024 / (elapsed - last[0])\n",
    "            rows.append((f\"{len(rows) * 2}-{len(rows) * 2 + 2} s\", f\"{rate_kb:.0f}\", sent - delivered[0]))\n",
    "            last = (elapsed, delivered[0])\n",
    "    average_kb = delivered[0] * len(value) / 1024 / last[0]\n",
    "    flush_started = time.monotonic()\n",
    "    producer.flush(120)\n",
    "    lab.table(rows, [\"time\", \"KB/s delivered\", \"waiting in the producer\"])\n",
    "    print(f\"average over {seconds} s: {average_kb:.0f} KB/s\")\n",
    "    print(f\"{sent} readings; the last ones waited {time.monotonic() - flush_started:.1f} s more in the producer\")\n",
    "    print(f\"throttled {len(throttles)} times, longest {max(throttles, default=0) * 1000:.0f} ms\")\n",
    "\n",
    "runaway_burst()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c14-ed276f1f",
   "metadata": {},
   "source": [
    "Now a quota: at most 512 KB/s of produce traffic for `svc-gateway`, per broker. The Python client\n",
    "has no API for quotas, so admin sets it with `kafka-configs.sh` inside the broker's container."
   ]
  },
  {
   "cell_type": "code",
   "id": "c15-42950718",
   "metadata": {},
   "source": [
    "def kafka_configs(*args):\n",
    "    return lab.kafka_cli(\"kafka-configs.sh\", \"--bootstrap-server\", \"127.0.0.1:9095\",\n",
    "                         \"--command-config\", \"/etc/kafka/secrets/admin.properties\",\n",
    "                         \"--entity-type\", \"users\", \"--entity-name\", \"svc-gateway\", *args,\n",
    "                         node=\"kafka-secure\", echo=False)\n",
    "\n",
    "kafka_configs(\"--alter\", \"--add-config\", \"producer_byte_rate=524288\")\n",
    "print(kafka_configs(\"--describe\"))\n",
    "time.sleep(2)\n",
    "runaway_burst()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c16-ae774bc1",
   "metadata": {},
   "source": [
    "Nothing failed. The broker accepted every request, told the producer how long to hold back, and\n",
    "stopped reading from its connection for that time. The backlog moved into the gateway's producer.\n",
    "\n",
    "## Break it: the grid incident\n",
    "The grid team asked for \"access to the topics\" and got `ALL` on every topic (`*`). A bug in their\n",
    "clean-up job then deleted the wrong topic. Replay it:"
   ]
  },
  {
   "cell_type": "code",
   "id": "c17-a316d5ff",
   "metadata": {},
   "source": [
    "GRID = \"User:svc-grid\"\n",
    "everything = allow(GRID, ResourceType.TOPIC, \"*\", AclOperation.ALL)\n",
    "admin.create_acls([everything])[everything].result()\n",
    "time.sleep(1)\n",
    "grid_admin = AdminClient(secure.client_config(\"svc-grid\", secure.USERS[\"svc-grid\"]))\n",
    "\n",
    "def grid_deletes(topic):\n",
    "    try:\n",
    "        grid_admin.delete_topics([topic], operation_timeout=15)[topic].result()\n",
    "        return \"deleted\"\n",
    "    except KafkaException as e:\n",
    "        return e.args[0].str()\n",
    "\n",
    "print(\"with ALL on *:      grid deletes charging.telemetry ->\", grid_deletes(\"charging.telemetry\"))\n",
    "removed = admin.delete_acls([AclBindingFilter(ResourceType.ANY, None, ResourcePatternType.ANY, GRID, None,\n",
    "                                              AclOperation.ANY, AclPermissionType.ANY)])\n",
    "print(\"removed\", sum(len(f.result()) for f in removed.values()), \"ACL of svc-grid\")\n",
    "time.sleep(1)\n",
    "print(\"without that ACL:   grid deletes charging.sessions  ->\", grid_deletes(\"charging.sessions\"))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c18-bf9632ae",
   "metadata": {},
   "source": [
    "One wildcard ACL turned a bug in one service into an outage of the gateway's main topic. Exercise\n",
    "1 gives `svc-grid` the permissions it really needs.\n",
    "\n",
    "## Clean up\n",
    "Remove the ACLs, the quota and the topics. The secure broker keeps running for the exercises; to\n",
    "stop it: `docker compose --profile secure stop`."
   ]
  },
  {
   "cell_type": "code",
   "id": "c19-a320d1f9",
   "metadata": {},
   "source": [
    "for principal in (BILLING, GATEWAY):\n",
    "    for future in admin.delete_acls([AclBindingFilter(ResourceType.ANY, None, ResourcePatternType.ANY, principal,\n",
    "                                                      None, AclOperation.ANY, AclPermissionType.ANY)]).values():\n",
    "        future.result()\n",
    "kafka_configs(\"--alter\", \"--delete-config\", \"producer_byte_rate\")\n",
    "remaining = [t for t in topics if t in admin.list_topics(timeout=10).topics]\n",
    "for name, future in admin.delete_topics(remaining).items():\n",
    "    future.result()\n",
    "print(\"cleaned up\")"
   ],
   "execution_count": null,
   "outputs": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
