Notebook · code/module-08/03-metrics-lag-and-alerts.ipynb

Download 03-metrics-lag-and-alerts.ipynb

Open it in JupyterLab or VS Code from the lab folder to run it. This page is a read-only view.

Observability: metrics, lag and alerts — practical

Prometheus collects the brokers' metrics, KMinion's view of consumer lag, and the billing worker's own metrics. You read the cluster's health in a few numbers, open the dashboard, stop a broker and watch the alerts, then stop billing and answer the 02:00 question: Kafka or billing?

Lab services: kafka-1..3, prometheus, kminion, grafana (docker compose --profile monitoring up -d --wait; the notebook starts them) · Lesson: modules/08-security-and-operations/03-metrics-lag-and-alerts.md

# Setup: find the lab's shared helpers
import sys, pathlib
lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / "common").is_dir())
sys.path.insert(0, str(lab_root))

import time
import requests
from common import lab, settings

lab.compose("--profile", "monitoring", "up", "-d", "--wait")
admin = lab.admin_client()
TOPIC, GROUP = "m08l03-sessions", "m08l03-billing"
lab.recreate_topic(admin, TOPIC, partitions=3)
lab.delete_groups(admin, [GROUP])              # in case an earlier run left it

Step 1: What Prometheus collects

Every 5 seconds Prometheus asks each target for its metrics: the three brokers' JMX exporters, KMinion, and a port on your Mac where the billing worker will publish its own.

targets = requests.get(settings.PROMETHEUS_URL + "/api/v1/targets", timeout=10).json()["data"]["activeTargets"]
lab.table([(t["labels"]["job"], t["labels"]["instance"], t["health"], t["lastError"][:45])
           for t in sorted(targets, key=lambda t: t["labels"]["instance"])],
          ["job", "instance", "health", "last error"])

The services target is down: nothing runs on port 8001 yet.

Step 2: The cluster's health in six numbers

health = {
    "brokers answering":           'sum(up{job="kafka"})',
    "active controllers":          "sum(kafka_controller_kafkacontroller_activecontrollercount)",
    "under-replicated partitions": "sum(kafka_server_replicamanager_underreplicatedpartitions)",
    "under min ISR":               "sum(kafka_server_replicamanager_underminisrpartitioncount)",
    "offline partitions":          "max(kafka_controller_kafkacontroller_offlinepartitionscount)",
    "request handlers idle (min)": "min(kafka_server_kafkarequesthandlerpool_requesthandleravgidlepercent)",
}

def one(expr):
    """The query's value, added up over all the series it returns."""
    return sum(lab.prom(expr).values())

def show_health():
    lab.table([(name, f"{one(expr):.2f}".rstrip("0").rstrip(".")) for name, expr in health.items()],
              ["signal", "now"])

show_health()

Step 3: Billing runs and publishes its own metrics

The session feed writes 40 events a second. metered_worker.py bills them and serves its metrics on port 8001: its business counters, and numbers taken from the client's statistics.

feed = lab.start_script("module-03/session_feed.py", "--topic", TOPIC, "--rate", "40", name="m08l03-feed")
worker = lab.start_script("module-08/metered_worker.py", "--topic", TOPIC, "--group", GROUP, name="m08l03-worker")
time.sleep(30)
own = requests.get("http://127.0.0.1:8001/metrics", timeout=5).text
print("\n".join(line for line in own.splitlines() if line.startswith("billing_") and "_created" not in line))

Two views of the same lag, a few seconds apart. KMinion asks the cluster for the group's committed offsets; the worker's client knows the last record it processed.

KMINION_LAG = f'sum(kminion_kafka_consumer_group_topic_lag{{group_id="{GROUP}"}})'
WORKER_LAG = "sum(billing_consumer_lag)"
WORKER_UP = 'up{job="services"}'                 # 1 if the worker answered the last scrape
ARRIVING = f'sum(rate(kafka_server_brokertopicmetrics_messagesin_total{{topic="{TOPIC}"}}[30s]))'
for _ in range(4):
    print(f"lag, KMinion (committed): {one(KMINION_LAG):4.0f}   worker (processed): {one(WORKER_LAG):4.0f}"
          f"   arriving: {one(ARRIVING):.0f}/s")
    time.sleep(5)

Step 4: The dashboard

Grafana starts with a Prometheus data source and the Brightlane dashboard already set up. Open the URL below in your browser; it doesn't need a login.

board = requests.get(settings.GRAFANA_URL + "/api/dashboards/uid/brightlane-kafka", timeout=10).json()
print(settings.GRAFANA_URL + board["meta"]["url"])
for panel in board["dashboard"]["panels"]:
    print(f"  {panel['title']:42} {panel['targets'][0]['expr']}")

Step 5: A broker stops

Stop kafka-2 and watch the numbers and the alerts every 5 seconds. An alert is pending while its condition holds but for less than its for time, then firing.

cluster_alerts = ["KafkaBrokerDown", "KafkaUnderReplicatedPartitions", "KafkaOfflinePartitions",
                  "KafkaActiveControllerCount"]
lab.compose("stop", "kafka-2")
stopped = time.monotonic()
while time.monotonic() - stopped < 90:
    time.sleep(5)
    now = lab.alerts()
    print(f"+{time.monotonic() - stopped:3.0f} s  brokers up: {one(health['brokers answering']):.0f}"
          f"  under-replicated: {one(health['under-replicated partitions']):3.0f}  "
          + ", ".join(f"{name} {state}" for name, state in now.items() if name in cluster_alerts))
    if all(now.get(name) == "firing" for name in cluster_alerts[:2]):
        break

Bring kafka-2 back and wait until the alerts clear.

lab.compose("up", "-d", "--wait", "kafka-2")
started = time.monotonic()
lab.wait_until(lambda: not set(lab.alerts()) & set(cluster_alerts), timeout=120, interval=5, what="the cluster alerts to clear")
print(f"cluster alerts cleared {time.monotonic() - started:.0f} s after kafka-2 was back")
show_health()

Break it: the 02:00 page

Billing stops, as if its process hung. The page says billing is late. Is it Kafka or billing? Stop the worker and watch.

lab.stop_script(worker)
stopped = time.monotonic()
while time.monotonic() - stopped < 120:
    time.sleep(5)
    state = lab.alerts().get("BillingLagHigh", "inactive")
    print(f"+{time.monotonic() - stopped:3.0f} s  lag (KMinion): {one(KMINION_LAG):5.0f}"
          f"  worker answering: {one(WORKER_UP):.0f}  BillingLagHigh {state}")
    if state == "firing":
        break

The worker's own metrics disappeared with it. The answer comes from the other metrics:

show_health()
print(f"sessions arriving: {one(ARRIVING):.0f}/s   billing worker answering: {one(WORKER_UP):.0f}")
print("firing:", [name for name, state in lab.alerts().items() if state == "firing"])

Clean up

Stop the feed, delete the topic and the group. The monitoring services keep running for the exercises; to stop them: docker compose --profile monitoring stop.

lab.stop_script(feed)
time.sleep(2)
lab.delete_groups(admin, [GROUP])
lab.delete_topics(admin, [TOPIC])
print("cleaned up")