{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c00-dcd7a8fa",
   "metadata": {},
   "source": [
    "# Observability: metrics, lag and alerts — practical\n",
    "\n",
    "Prometheus collects the brokers' metrics, KMinion's view of consumer lag, and the billing\n",
    "worker's own metrics. You read the cluster's health in a few numbers, open the dashboard, stop a\n",
    "broker and watch the alerts, then stop billing and answer the 02:00 question: Kafka or billing?\n",
    "\n",
    "**Lab services:** kafka-1..3, prometheus, kminion, grafana (`docker compose --profile monitoring up -d --wait`;\n",
    "the notebook starts them)\n",
    "· **Lesson:** `modules/08-security-and-operations/03-metrics-lag-and-alerts.md`"
   ]
  },
  {
   "cell_type": "code",
   "id": "c01-400ee8cb",
   "metadata": {},
   "source": [
    "# Setup: find the lab's shared helpers\n",
    "import sys, pathlib\n",
    "lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / \"common\").is_dir())\n",
    "sys.path.insert(0, str(lab_root))\n",
    "\n",
    "import time\n",
    "import requests\n",
    "from common import lab, settings\n",
    "\n",
    "lab.compose(\"--profile\", \"monitoring\", \"up\", \"-d\", \"--wait\")\n",
    "admin = lab.admin_client()\n",
    "TOPIC, GROUP = \"m08l03-sessions\", \"m08l03-billing\"\n",
    "lab.recreate_topic(admin, TOPIC, partitions=3)\n",
    "lab.delete_groups(admin, [GROUP])              # in case an earlier run left it"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c02-0a568785",
   "metadata": {},
   "source": [
    "## Step 1: What Prometheus collects\n",
    "Every 5 seconds Prometheus asks each target for its metrics: the three brokers' JMX exporters,\n",
    "KMinion, and a port on your Mac where the billing worker will publish its own."
   ]
  },
  {
   "cell_type": "code",
   "id": "c03-ab580225",
   "metadata": {},
   "source": [
    "targets = requests.get(settings.PROMETHEUS_URL + \"/api/v1/targets\", timeout=10).json()[\"data\"][\"activeTargets\"]\n",
    "lab.table([(t[\"labels\"][\"job\"], t[\"labels\"][\"instance\"], t[\"health\"], t[\"lastError\"][:45])\n",
    "           for t in sorted(targets, key=lambda t: t[\"labels\"][\"instance\"])],\n",
    "          [\"job\", \"instance\", \"health\", \"last error\"])"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c04-d9adfd46",
   "metadata": {},
   "source": [
    "The `services` target is down: nothing runs on port 8001 yet.\n",
    "\n",
    "## Step 2: The cluster's health in six numbers"
   ]
  },
  {
   "cell_type": "code",
   "id": "c05-73d25b7f",
   "metadata": {},
   "source": [
    "health = {\n",
    "    \"brokers answering\":           'sum(up{job=\"kafka\"})',\n",
    "    \"active controllers\":          \"sum(kafka_controller_kafkacontroller_activecontrollercount)\",\n",
    "    \"under-replicated partitions\": \"sum(kafka_server_replicamanager_underreplicatedpartitions)\",\n",
    "    \"under min ISR\":               \"sum(kafka_server_replicamanager_underminisrpartitioncount)\",\n",
    "    \"offline partitions\":          \"max(kafka_controller_kafkacontroller_offlinepartitionscount)\",\n",
    "    \"request handlers idle (min)\": \"min(kafka_server_kafkarequesthandlerpool_requesthandleravgidlepercent)\",\n",
    "}\n",
    "\n",
    "def one(expr):\n",
    "    \"\"\"The query's value, added up over all the series it returns.\"\"\"\n",
    "    return sum(lab.prom(expr).values())\n",
    "\n",
    "def show_health():\n",
    "    lab.table([(name, f\"{one(expr):.2f}\".rstrip(\"0\").rstrip(\".\")) for name, expr in health.items()],\n",
    "              [\"signal\", \"now\"])\n",
    "\n",
    "show_health()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c06-79959717",
   "metadata": {},
   "source": [
    "## Step 3: Billing runs and publishes its own metrics\n",
    "The session feed writes 40 events a second. `metered_worker.py` bills them and serves its metrics\n",
    "on port 8001: its business counters, and numbers taken from the client's statistics."
   ]
  },
  {
   "cell_type": "code",
   "id": "c07-a3ff6d15",
   "metadata": {},
   "source": [
    "feed = lab.start_script(\"module-03/session_feed.py\", \"--topic\", TOPIC, \"--rate\", \"40\", name=\"m08l03-feed\")\n",
    "worker = lab.start_script(\"module-08/metered_worker.py\", \"--topic\", TOPIC, \"--group\", GROUP, name=\"m08l03-worker\")\n",
    "time.sleep(30)\n",
    "own = requests.get(\"http://127.0.0.1:8001/metrics\", timeout=5).text\n",
    "print(\"\\n\".join(line for line in own.splitlines() if line.startswith(\"billing_\") and \"_created\" not in line))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c08-f10cba2d",
   "metadata": {},
   "source": [
    "Two views of the same lag, a few seconds apart. KMinion asks the cluster for the group's committed\n",
    "offsets; the worker's client knows the last record it processed."
   ]
  },
  {
   "cell_type": "code",
   "id": "c09-fa62ecd0",
   "metadata": {},
   "source": [
    "KMINION_LAG = f'sum(kminion_kafka_consumer_group_topic_lag{{group_id=\"{GROUP}\"}})'\n",
    "WORKER_LAG = \"sum(billing_consumer_lag)\"\n",
    "WORKER_UP = 'up{job=\"services\"}'                 # 1 if the worker answered the last scrape\n",
    "ARRIVING = f'sum(rate(kafka_server_brokertopicmetrics_messagesin_total{{topic=\"{TOPIC}\"}}[30s]))'\n",
    "for _ in range(4):\n",
    "    print(f\"lag, KMinion (committed): {one(KMINION_LAG):4.0f}   worker (processed): {one(WORKER_LAG):4.0f}\"\n",
    "          f\"   arriving: {one(ARRIVING):.0f}/s\")\n",
    "    time.sleep(5)"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c10-7cb4493a",
   "metadata": {},
   "source": [
    "## Step 4: The dashboard\n",
    "Grafana starts with a Prometheus data source and the Brightlane dashboard already set up. Open\n",
    "the URL below in your browser; it doesn't need a login."
   ]
  },
  {
   "cell_type": "code",
   "id": "c11-2e665f8e",
   "metadata": {},
   "source": [
    "board = requests.get(settings.GRAFANA_URL + \"/api/dashboards/uid/brightlane-kafka\", timeout=10).json()\n",
    "print(settings.GRAFANA_URL + board[\"meta\"][\"url\"])\n",
    "for panel in board[\"dashboard\"][\"panels\"]:\n",
    "    print(f\"  {panel['title']:42} {panel['targets'][0]['expr']}\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c12-fc979789",
   "metadata": {},
   "source": [
    "## Step 5: A broker stops\n",
    "Stop kafka-2 and watch the numbers and the alerts every 5 seconds. An alert is `pending` while\n",
    "its condition holds but for less than its `for` time, then `firing`."
   ]
  },
  {
   "cell_type": "code",
   "id": "c13-7bb5ff9f",
   "metadata": {},
   "source": [
    "cluster_alerts = [\"KafkaBrokerDown\", \"KafkaUnderReplicatedPartitions\", \"KafkaOfflinePartitions\",\n",
    "                  \"KafkaActiveControllerCount\"]\n",
    "lab.compose(\"stop\", \"kafka-2\")\n",
    "stopped = time.monotonic()\n",
    "while time.monotonic() - stopped < 90:\n",
    "    time.sleep(5)\n",
    "    now = lab.alerts()\n",
    "    print(f\"+{time.monotonic() - stopped:3.0f} s  brokers up: {one(health['brokers answering']):.0f}\"\n",
    "          f\"  under-replicated: {one(health['under-replicated partitions']):3.0f}  \"\n",
    "          + \", \".join(f\"{name} {state}\" for name, state in now.items() if name in cluster_alerts))\n",
    "    if all(now.get(name) == \"firing\" for name in cluster_alerts[:2]):\n",
    "        break"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c14-f5d1cbb1",
   "metadata": {},
   "source": [
    "Bring kafka-2 back and wait until the alerts clear."
   ]
  },
  {
   "cell_type": "code",
   "id": "c15-f10113b5",
   "metadata": {},
   "source": [
    "lab.compose(\"up\", \"-d\", \"--wait\", \"kafka-2\")\n",
    "started = time.monotonic()\n",
    "lab.wait_until(lambda: not set(lab.alerts()) & set(cluster_alerts), timeout=120, interval=5, what=\"the cluster alerts to clear\")\n",
    "print(f\"cluster alerts cleared {time.monotonic() - started:.0f} s after kafka-2 was back\")\n",
    "show_health()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c16-07d8024f",
   "metadata": {},
   "source": [
    "## Break it: the 02:00 page\n",
    "Billing stops, as if its process hung. The page says billing is late. Is it Kafka or billing?\n",
    "Stop the worker and watch."
   ]
  },
  {
   "cell_type": "code",
   "id": "c17-5c8214f5",
   "metadata": {},
   "source": [
    "lab.stop_script(worker)\n",
    "stopped = time.monotonic()\n",
    "while time.monotonic() - stopped < 120:\n",
    "    time.sleep(5)\n",
    "    state = lab.alerts().get(\"BillingLagHigh\", \"inactive\")\n",
    "    print(f\"+{time.monotonic() - stopped:3.0f} s  lag (KMinion): {one(KMINION_LAG):5.0f}\"\n",
    "          f\"  worker answering: {one(WORKER_UP):.0f}  BillingLagHigh {state}\")\n",
    "    if state == \"firing\":\n",
    "        break"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c18-bb50fef5",
   "metadata": {},
   "source": [
    "The worker's own metrics disappeared with it. The answer comes from the other metrics:"
   ]
  },
  {
   "cell_type": "code",
   "id": "c19-617d8a06",
   "metadata": {},
   "source": [
    "show_health()\n",
    "print(f\"sessions arriving: {one(ARRIVING):.0f}/s   billing worker answering: {one(WORKER_UP):.0f}\")\n",
    "print(\"firing:\", [name for name, state in lab.alerts().items() if state == \"firing\"])"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c20-d225941d",
   "metadata": {},
   "source": [
    "## Clean up\n",
    "Stop the feed, delete the topic and the group. The monitoring services keep running for the\n",
    "exercises; to stop them: `docker compose --profile monitoring stop`."
   ]
  },
  {
   "cell_type": "code",
   "id": "c21-634b54fa",
   "metadata": {},
   "source": [
    "lab.stop_script(feed)\n",
    "time.sleep(2)\n",
    "lab.delete_groups(admin, [GROUP])\n",
    "lab.delete_topics(admin, [TOPIC])\n",
    "print(\"cleaned up\")"
   ],
   "execution_count": null,
   "outputs": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
