Download 03-metrics-lag-and-alerts.ipynb
Open it in JupyterLab or VS Code from the lab folder to run it. This page is a read-only view.
Observability: metrics, lag and alerts — practical
Prometheus collects the brokers' metrics, KMinion's view of consumer lag, and the billing worker's own metrics. You read the cluster's health in a few numbers, open the dashboard, stop a broker and watch the alerts, then stop billing and answer the 02:00 question: Kafka or billing?
Lab services: kafka-1..3, prometheus, kminion, grafana (docker compose --profile monitoring up -d --wait;
the notebook starts them)
· Lesson: modules/08-security-and-operations/03-metrics-lag-and-alerts.md
# Setup: find the lab's shared helpers import sys, pathlib lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / "common").is_dir()) sys.path.insert(0, str(lab_root)) import time import requests from common import lab, settings lab.compose("--profile", "monitoring", "up", "-d", "--wait") admin = lab.admin_client() TOPIC, GROUP = "m08l03-sessions", "m08l03-billing" lab.recreate_topic(admin, TOPIC, partitions=3) lab.delete_groups(admin, [GROUP]) # in case an earlier run left it
Step 1: What Prometheus collects
Every 5 seconds Prometheus asks each target for its metrics: the three brokers' JMX exporters, KMinion, and a port on your Mac where the billing worker will publish its own.
targets = requests.get(settings.PROMETHEUS_URL + "/api/v1/targets", timeout=10).json()["data"]["activeTargets"] lab.table([(t["labels"]["job"], t["labels"]["instance"], t["health"], t["lastError"][:45]) for t in sorted(targets, key=lambda t: t["labels"]["instance"])], ["job", "instance", "health", "last error"])
The services target is down: nothing runs on port 8001 yet.
Step 2: The cluster's health in six numbers
health = { "brokers answering": 'sum(up{job="kafka"})', "active controllers": "sum(kafka_controller_kafkacontroller_activecontrollercount)", "under-replicated partitions": "sum(kafka_server_replicamanager_underreplicatedpartitions)", "under min ISR": "sum(kafka_server_replicamanager_underminisrpartitioncount)", "offline partitions": "max(kafka_controller_kafkacontroller_offlinepartitionscount)", "request handlers idle (min)": "min(kafka_server_kafkarequesthandlerpool_requesthandleravgidlepercent)", } def one(expr): """The query's value, added up over all the series it returns.""" return sum(lab.prom(expr).values()) def show_health(): lab.table([(name, f"{one(expr):.2f}".rstrip("0").rstrip(".")) for name, expr in health.items()], ["signal", "now"]) show_health()
Step 3: Billing runs and publishes its own metrics
The session feed writes 40 events a second. metered_worker.py bills them and serves its metrics
on port 8001: its business counters, and numbers taken from the client's statistics.
feed = lab.start_script("module-03/session_feed.py", "--topic", TOPIC, "--rate", "40", name="m08l03-feed") worker = lab.start_script("module-08/metered_worker.py", "--topic", TOPIC, "--group", GROUP, name="m08l03-worker") time.sleep(30) own = requests.get("http://127.0.0.1:8001/metrics", timeout=5).text print("\n".join(line for line in own.splitlines() if line.startswith("billing_") and "_created" not in line))
Two views of the same lag, a few seconds apart. KMinion asks the cluster for the group's committed offsets; the worker's client knows the last record it processed.
KMINION_LAG = f'sum(kminion_kafka_consumer_group_topic_lag{{group_id="{GROUP}"}})' WORKER_LAG = "sum(billing_consumer_lag)" WORKER_UP = 'up{job="services"}' # 1 if the worker answered the last scrape ARRIVING = f'sum(rate(kafka_server_brokertopicmetrics_messagesin_total{{topic="{TOPIC}"}}[30s]))' for _ in range(4): print(f"lag, KMinion (committed): {one(KMINION_LAG):4.0f} worker (processed): {one(WORKER_LAG):4.0f}" f" arriving: {one(ARRIVING):.0f}/s") time.sleep(5)
Step 4: The dashboard
Grafana starts with a Prometheus data source and the Brightlane dashboard already set up. Open the URL below in your browser; it doesn't need a login.
board = requests.get(settings.GRAFANA_URL + "/api/dashboards/uid/brightlane-kafka", timeout=10).json() print(settings.GRAFANA_URL + board["meta"]["url"]) for panel in board["dashboard"]["panels"]: print(f" {panel['title']:42} {panel['targets'][0]['expr']}")
Step 5: A broker stops
Stop kafka-2 and watch the numbers and the alerts every 5 seconds. An alert is pending while
its condition holds but for less than its for time, then firing.
cluster_alerts = ["KafkaBrokerDown", "KafkaUnderReplicatedPartitions", "KafkaOfflinePartitions", "KafkaActiveControllerCount"] lab.compose("stop", "kafka-2") stopped = time.monotonic() while time.monotonic() - stopped < 90: time.sleep(5) now = lab.alerts() print(f"+{time.monotonic() - stopped:3.0f} s brokers up: {one(health['brokers answering']):.0f}" f" under-replicated: {one(health['under-replicated partitions']):3.0f} " + ", ".join(f"{name} {state}" for name, state in now.items() if name in cluster_alerts)) if all(now.get(name) == "firing" for name in cluster_alerts[:2]): break
Bring kafka-2 back and wait until the alerts clear.
lab.compose("up", "-d", "--wait", "kafka-2") started = time.monotonic() lab.wait_until(lambda: not set(lab.alerts()) & set(cluster_alerts), timeout=120, interval=5, what="the cluster alerts to clear") print(f"cluster alerts cleared {time.monotonic() - started:.0f} s after kafka-2 was back") show_health()
Break it: the 02:00 page
Billing stops, as if its process hung. The page says billing is late. Is it Kafka or billing? Stop the worker and watch.
lab.stop_script(worker) stopped = time.monotonic() while time.monotonic() - stopped < 120: time.sleep(5) state = lab.alerts().get("BillingLagHigh", "inactive") print(f"+{time.monotonic() - stopped:3.0f} s lag (KMinion): {one(KMINION_LAG):5.0f}" f" worker answering: {one(WORKER_UP):.0f} BillingLagHigh {state}") if state == "firing": break
The worker's own metrics disappeared with it. The answer comes from the other metrics:
show_health() print(f"sessions arriving: {one(ARRIVING):.0f}/s billing worker answering: {one(WORKER_UP):.0f}") print("firing:", [name for name, state in lab.alerts().items() if state == "firing"])
Clean up
Stop the feed, delete the topic and the group. The monitoring services keep running for the
exercises; to stop them: docker compose --profile monitoring stop.
lab.stop_script(feed) time.sleep(2) lab.delete_groups(admin, [GROUP]) lab.delete_topics(admin, [TOPIC]) print("cleaned up")