{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c00-c78da22a",
   "metadata": {},
   "source": [
    "# The production-readiness review — practical\n",
    "\n",
    "Brightlane's next release is ready: its topics in `module-09/release/topics.yaml`, its services'\n",
    "client settings in `clients.yaml`. `readiness_check.py` reviews the release and the lab against\n",
    "the rules of this course. You run it, fix each finding the way the course taught, and run it\n",
    "again until it's green.\n",
    "\n",
    "**Lab services:** kafka-1..3, kafka-secure, prometheus, kminion, grafana, kafka-dr, mirror-maker\n",
    "(the notebook starts them; together they need about 5 GB of the Docker VM's memory)\n",
    "· **Lesson:** `modules/09-production-ready/03-production-readiness-review.md`"
   ]
  },
  {
   "cell_type": "code",
   "id": "c01-f369dd59",
   "metadata": {},
   "source": [
    "# Setup: find the lab's shared helpers\n",
    "import sys, pathlib\n",
    "lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / \"common\").is_dir())\n",
    "sys.path.insert(0, str(lab_root))\n",
    "\n",
    "import shutil, subprocess, yaml\n",
    "from confluent_kafka.admin import (AclBinding, AclBindingFilter, AclOperation, AclPermissionType, AlterConfigOpType,\n",
    "                                   ConfigEntry, ConfigResource, NewTopic, ResourcePatternType, ResourceType)\n",
    "from common import dr, lab, secure\n",
    "\n",
    "lab.compose(\"--profile\", \"secure\", \"--profile\", \"monitoring\", \"--profile\", \"dr\", \"up\", \"-d\", \"--wait\")\n",
    "secure_admin = secure.ensure_ready()\n",
    "RELEASE = lab_root / \".lab-data\" / \"release\"                    # a working copy of the release files\n",
    "shutil.rmtree(RELEASE, ignore_errors=True)\n",
    "shutil.copytree(lab_root / \"module-09\" / \"release\", RELEASE)\n",
    "print(\"release files copied to\", RELEASE.relative_to(lab_root))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c02-73aecb83",
   "metadata": {},
   "source": [
    "The setup puts the release's topics on the primary cluster, exactly as the file says, and the\n",
    "secure broker's ACLs as they are in production today."
   ]
  },
  {
   "cell_type": "code",
   "id": "c03-05ea2a54",
   "metadata": {},
   "source": [
    "topics = yaml.safe_load((RELEASE / \"topics.yaml\").read_text())[\"topics\"]\n",
    "dr.reset_mirroring(\"charging.\", \"billing.\")                     # no topics, copies or groups from earlier\n",
    "admin = lab.admin_client()\n",
    "for name, future in admin.create_topics([NewTopic(name, spec[\"partitions\"], spec[\"replication_factor\"],\n",
    "                                                  config={k: str(v) for k, v in spec[\"config\"].items()})\n",
    "                                         for name, spec in topics.items()]).items():\n",
    "    future.result()\n",
    "lab.wait_until(lambda: all(\"primary.\" + t in dr.admin(\"dr\").list_topics(timeout=10).topics for t in topics),\n",
    "               timeout=90, interval=3, what=\"MirrorMaker 2 to copy the topics\")\n",
    "\n",
    "def allow(user, resource_type, name, operation, pattern=ResourcePatternType.LITERAL):\n",
    "    return AclBinding(resource_type, name, pattern, f\"User:{user}\", \"*\", operation, AclPermissionType.ALLOW)\n",
    "\n",
    "everything = AclBindingFilter(ResourceType.ANY, None, ResourcePatternType.ANY, None, None, AclOperation.ANY,\n",
    "                              AclPermissionType.ANY)\n",
    "for future in secure_admin.delete_acls([everything]).values():\n",
    "    future.result()\n",
    "production_acls = [\n",
    "    allow(\"svc-billing\", ResourceType.TOPIC, \"charging.sessions\", AclOperation.READ),\n",
    "    allow(\"svc-billing\", ResourceType.GROUP, \"billing\", AclOperation.READ),\n",
    "    allow(\"svc-billing\", ResourceType.TOPIC, \"billing.session-charges\", AclOperation.WRITE),\n",
    "    allow(\"svc-billing\", ResourceType.TRANSACTIONAL_ID, \"billing-\", AclOperation.WRITE, ResourcePatternType.PREFIXED),\n",
    "    allow(\"svc-gateway\", ResourceType.TOPIC, \"charging.\", AclOperation.WRITE, ResourcePatternType.PREFIXED),\n",
    "    allow(\"svc-grid\", ResourceType.TOPIC, \"*\", AclOperation.ALL),     # \"temporary\", since the grid project\n",
    "]\n",
    "for future in secure_admin.create_acls(production_acls).values():\n",
    "    future.result()\n",
    "lab.remove_alert_rules(\"readiness\")\n",
    "print(f\"{len(topics)} topics, {len(production_acls)} ACLs\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c04-fef0967f",
   "metadata": {},
   "source": [
    "## Step 1: The first review"
   ]
  },
  {
   "cell_type": "code",
   "id": "c05-cba707c3",
   "metadata": {},
   "source": [
    "def review():\n",
    "    result = subprocess.run([sys.executable, \"module-09/readiness_check.py\", \"--release\", \".lab-data/release\"],\n",
    "                            cwd=lab_root, capture_output=True, text=True)\n",
    "    print(result.stdout.strip() or result.stderr[-2000:])\n",
    "    print(f\"exit code {result.returncode}:\", \"a CI pipeline would stop the release\" if result.returncode\n",
    "          else \"the release may go ahead\")\n",
    "\n",
    "review()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c06-c33295da",
   "metadata": {},
   "source": [
    "## Step 2: Fix the topics\n",
    "Change the file first, then apply the same change to the cluster, as lesson 8.4's `apply()` does.\n",
    "`billing.session-charges` needs two in-sync replicas like every other topic; the charger map\n",
    "goes back to the cluster's default, no unclean elections."
   ]
  },
  {
   "cell_type": "code",
   "id": "c07-96c5804d",
   "metadata": {},
   "source": [
    "release_topics = yaml.safe_load((RELEASE / \"topics.yaml\").read_text())\n",
    "release_topics[\"topics\"][\"billing.session-charges\"][\"config\"][\"min.insync.replicas\"] = 2\n",
    "del release_topics[\"topics\"][\"charging.charger-status\"][\"config\"][\"unclean.leader.election.enable\"]\n",
    "(RELEASE / \"topics.yaml\").write_text(yaml.safe_dump(release_topics, sort_keys=False))\n",
    "\n",
    "changes = [ConfigResource(ResourceType.TOPIC, \"billing.session-charges\", incremental_configs=[\n",
    "               ConfigEntry(\"min.insync.replicas\", \"2\", incremental_operation=AlterConfigOpType.SET)]),\n",
    "           ConfigResource(ResourceType.TOPIC, \"charging.charger-status\", incremental_configs=[\n",
    "               ConfigEntry(\"unclean.leader.election.enable\", None, incremental_operation=AlterConfigOpType.DELETE)])]\n",
    "for resource, future in admin.incremental_alter_configs(changes).items():\n",
    "    future.result()\n",
    "    print(\"changed\", resource.name)"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c08-a0e3ea58",
   "metadata": {},
   "source": [
    "## Step 3: Fix the client settings\n",
    "The gateway writes with `acks=all` and idempotence (lesson 2.2). Billing reads `read_committed`\n",
    "next to its transactional producer (lesson 7.1). Grid reporting stores an offset only after the\n",
    "record is processed (lesson 3.2)."
   ]
  },
  {
   "cell_type": "code",
   "id": "c09-ee766c1a",
   "metadata": {},
   "source": [
    "clients = yaml.safe_load((RELEASE / \"clients.yaml\").read_text())\n",
    "clients[\"services\"][\"gateway\"][\"producer\"].update({\"acks\": \"all\", \"enable.idempotence\": True})\n",
    "clients[\"services\"][\"billing\"][\"consumer\"][\"isolation.level\"] = \"read_committed\"\n",
    "clients[\"services\"][\"grid\"][\"consumer\"][\"enable.auto.offset.store\"] = False\n",
    "(RELEASE / \"clients.yaml\").write_text(yaml.safe_dump(clients, sort_keys=False))\n",
    "print(yaml.safe_dump(clients[\"services\"][\"grid\"], sort_keys=False))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c10-14f8f433",
   "metadata": {},
   "source": [
    "## Step 4: Fix the ACLs\n",
    "Replace grid reporting's wildcard with what its job needs (exercise 8.1)."
   ]
  },
  {
   "cell_type": "code",
   "id": "c11-7a041af5",
   "metadata": {},
   "source": [
    "wildcard = AclBindingFilter(ResourceType.TOPIC, \"*\", ResourcePatternType.LITERAL, \"User:svc-grid\", None,\n",
    "                            AclOperation.ALL, AclPermissionType.ALLOW)\n",
    "for future in secure_admin.delete_acls([wildcard]).values():\n",
    "    print(\"removed\", len(future.result()), \"ACL\")\n",
    "grid_acls = [allow(\"svc-grid\", ResourceType.TOPIC, \"charging.telemetry\", AclOperation.READ),\n",
    "             allow(\"svc-grid\", ResourceType.GROUP, \"grid-reporting\", AclOperation.READ),\n",
    "             allow(\"svc-grid\", ResourceType.TOPIC, \"grid.site-load-5m\", AclOperation.WRITE)]\n",
    "for future in secure_admin.create_acls(grid_acls).values():\n",
    "    future.result()\n",
    "print(\"added\", len(grid_acls), \"ACLs\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c12-2974429c",
   "metadata": {},
   "source": [
    "## Step 5: Add the missing alert\n",
    "Partitions under min ISR refuse `acks=all` writes: the gateway's telemetry stops. That must page."
   ]
  },
  {
   "cell_type": "code",
   "id": "c13-003db9db",
   "metadata": {},
   "source": [
    "rule = {\"alert\": \"KafkaUnderMinIsrPartitions\",\n",
    "        \"expr\": \"sum(kafka_server_replicamanager_underminisrpartitioncount) > 0\",\n",
    "        \"for\": \"30s\", \"labels\": {\"severity\": \"page\"},\n",
    "        \"annotations\": {\"summary\": \"{{ $value }} partitions have fewer in-sync replicas than min.insync.replicas: \"\n",
    "                                   \"acks=all writes to them fail\",\n",
    "                        \"runbook_url\": \"https://runbooks.brightlane.example/kafka/under-min-isr\"}}\n",
    "print(lab.add_alert_rules(\"readiness\", yaml.safe_dump({\"groups\": [{\"name\": \"readiness\", \"rules\": [rule]}]})))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c14-f490d3bc",
   "metadata": {},
   "source": [
    "## Step 6: Review again"
   ]
  },
  {
   "cell_type": "code",
   "id": "c15-c16ecff4",
   "metadata": {},
   "source": [
    "review()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c16-6b7f1b83",
   "metadata": {},
   "source": [
    "## Break it: a bad night\n",
    "The check isn't only for releases. Run every night against production, it notices what changed\n",
    "since: here, a broker that didn't come back after maintenance."
   ]
  },
  {
   "cell_type": "code",
   "id": "c17-71197a92",
   "metadata": {},
   "source": [
    "lab.compose(\"stop\", \"kafka-3\")\n",
    "review()\n",
    "lab.compose(\"up\", \"-d\", \"--wait\", \"kafka-3\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c18-b2c2b582",
   "metadata": {},
   "source": [
    "## Clean up\n",
    "Remove the release's topics and copies, the ACLs and the extra alert rule. This is the course's\n",
    "last practical: to stop everything, `docker compose --profile secure --profile monitoring --profile dr stop`;\n",
    "to delete the lab with all its data, `docker compose --profile secure --profile monitoring --profile dr --profile connect --profile schemas down -v`."
   ]
  },
  {
   "cell_type": "code",
   "id": "c19-c611b105",
   "metadata": {},
   "source": [
    "dr.reset_mirroring(\"charging.\", \"billing.\")\n",
    "for future in secure_admin.delete_acls([everything]).values():\n",
    "    future.result()\n",
    "lab.remove_alert_rules(\"readiness\")\n",
    "shutil.rmtree(RELEASE, ignore_errors=True)\n",
    "print(\"cleaned up\")"
   ],
   "execution_count": null,
   "outputs": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
