{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c00-7bf67f38",
   "metadata": {},
   "source": [
    "# Batching, compression and throughput — practical\n",
    "\n",
    "You measure two kinds of writing the gateway does: the live trickle of readings, and the replay\n",
    "of 100,000 readings buffered during an outage. You see what `linger.ms`, compression and\n",
    "`batch.size` change: requests, latency, throughput and bytes on disk.\n",
    "\n",
    "**Lab services:** kafka-1, kafka-2, kafka-3 · **Lesson:** `modules/02-producers/03-batching-compression-and-throughput.md`"
   ]
  },
  {
   "cell_type": "code",
   "id": "c01-11171afd",
   "metadata": {},
   "source": [
    "# Setup: find the lab's shared helpers and check the lab is running\n",
    "import sys, pathlib\n",
    "lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / \"common\").is_dir())\n",
    "sys.path.insert(0, str(lab_root))\n",
    "\n",
    "import json, time, statistics\n",
    "from common import settings, lab\n",
    "from common.scenario import Brightlane, to_json\n",
    "\n",
    "lab.require_kafka()\n",
    "lab.print_versions()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c02-b70c8b62",
   "metadata": {},
   "source": [
    "## Step 1: Prepare the readings\n",
    "100,000 meter readings, already serialized, keyed by charger."
   ]
  },
  {
   "cell_type": "code",
   "id": "c03-47eb7a98",
   "metadata": {},
   "source": [
    "from confluent_kafka import Producer\n",
    "from confluent_kafka.admin import AdminClient\n",
    "\n",
    "TOPIC = \"m02l03-readings\"\n",
    "admin = AdminClient({\"bootstrap.servers\": settings.KAFKA_BOOTSTRAP})\n",
    "backlog = [(e[\"charger_id\"], to_json(e)) for _, e in Brightlane().stream(sessions=460) if \"type\" not in e][:100_000]\n",
    "size_mb = sum(len(v) for _, v in backlog) / 1e6\n",
    "print(f\"{len(backlog):,} readings, {size_mb:.1f} MB, average {size_mb * 1e6 / len(backlog):.0f} bytes\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c04-ab003166",
   "metadata": {},
   "source": [
    "## Step 2: A function that times one write\n",
    "`send()` writes records with the given settings, either as fast as possible or at a steady\n",
    "`rate` per second, and reports (after one warm-up record that opens the connections):\n",
    "- **readings/s**: from the first `produce()` to the last delivery report;\n",
    "- **p50 / p99 ms**: each record's time from `produce()` to the broker's acknowledgement\n",
    "  (`msg.latency()`);\n",
    "- **records/request**: how many readings each produce request carried, from the client's statistics."
   ]
  },
  {
   "cell_type": "code",
   "id": "c05-3f0ca516",
   "metadata": {},
   "source": [
    "def send(label, records, rate=None, **config):\n",
    "    lab.recreate_topic(admin, TOPIC, partitions=6)\n",
    "    latencies, stats = [], []\n",
    "    p = Producer({**settings.PRODUCER_DEFAULTS, **config,\n",
    "                  \"statistics.interval.ms\": 1000, \"stats_cb\": stats.append})\n",
    "    p.produce(TOPIC, value=b\"warm-up\")            # connect and get a producer id before timing\n",
    "    p.flush(10)\n",
    "    start = time.monotonic()\n",
    "    for i, (key, value) in enumerate(records):\n",
    "        if rate:                                  # steady stream: wait for this record's send time\n",
    "            time.sleep(max(0.0, start + i / rate - time.monotonic()))\n",
    "        for _ in range(1000):                     # bounded retry if the local queue is full\n",
    "            try:\n",
    "                p.produce(TOPIC, key=key, value=value,\n",
    "                          on_delivery=lambda err, msg: latencies.append(msg.latency()))\n",
    "                break\n",
    "            except BufferError:\n",
    "                p.poll(0.05)\n",
    "        p.poll(0)\n",
    "    p.flush(60)\n",
    "    seconds = time.monotonic() - start\n",
    "    p.poll(1.2)                                   # one more statistics report after the last delivery\n",
    "    requests = sum(b[\"req\"].get(\"Produce\", 0) for b in json.loads(stats[-1])[\"brokers\"].values()) - 1\n",
    "    q = statistics.quantiles(latencies, n=100)\n",
    "    return {\"settings\": label, \"readings/s\": round(len(records) / seconds),\n",
    "            \"p50 ms\": round(q[49] * 1000, 1), \"p99 ms\": round(q[98] * 1000, 1),\n",
    "            \"records/request\": round(len(records) / requests, 1)}"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c06-9abc4434",
   "metadata": {},
   "source": [
    "## Step 3: A steady stream with linger.ms 0, 5 and 50\n",
    "Live, the gateway sends a steady stream, not a burst. `linger.ms` is how long the client waits\n",
    "for more records before it sends a batch: 0 sends at once, 5 is the default (in the Python\n",
    "client, and in Java since Kafka 4.0), 50 waits longer. Each run sends 5,000 readings at 2,500 a\n",
    "second, so it takes two seconds."
   ]
  },
  {
   "cell_type": "code",
   "id": "c07-6da4b7cc",
   "metadata": {},
   "source": [
    "steady = [send(f\"linger {ms}\", backlog[:5000], rate=2500, **{\"linger.ms\": ms}) for ms in (0, 5, 50)]\n",
    "lab.table([list(r.values()) for r in steady], list(steady[0].keys()))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c08-1dca31ac",
   "metadata": {},
   "source": [
    "You should see the same throughput in every row, 2,500 a second, because that's the rate we\n",
    "send at. The requests differ a lot: with `linger 0` a request carries one or two readings; with\n",
    "`linger 50` it carries about a hundred, and each reading waits about 50 ms longer. Your numbers\n",
    "will differ a little.\n",
    "\n",
    "## Step 4: The backlog, as fast as possible, with and without compression\n",
    "Now the replay: all 100,000 readings at once. Compression works on whole batches. Compare\n",
    "throughput, and the bytes the brokers store for the topic."
   ]
  },
  {
   "cell_type": "code",
   "id": "c09-ceef9634",
   "metadata": {},
   "source": [
    "def stored_bytes():\n",
    "    \"\"\"Total size of the topic's partitions on all brokers, from kafka-log-dirs.sh.\"\"\"\n",
    "    out = lab.kafka_cli(\"kafka-log-dirs.sh\", \"--bootstrap-server\", \"localhost:19092\",\n",
    "                        \"--describe\", \"--topic-list\", TOPIC, echo=False)\n",
    "    data = json.loads(out.splitlines()[-1])\n",
    "    return sum(p[\"size\"] for b in data[\"brokers\"] for d in b[\"logDirs\"] for p in d[\"partitions\"])\n",
    "\n",
    "plain = send(\"burst, none\", backlog)\n",
    "plain_bytes = stored_bytes()\n",
    "zstd = send(\"burst, zstd\", backlog, **{\"compression.type\": \"zstd\"})\n",
    "zstd_bytes = stored_bytes()\n",
    "lab.table([list(r.values()) for r in (plain, zstd)], list(plain.keys()))\n",
    "print(f\"\\nstored on disk (3 replicas): none {plain_bytes / 1e6:.1f} MB, zstd {zstd_bytes / 1e6:.1f} MB \"\n",
    "      f\"({plain_bytes / zstd_bytes:.1f}x smaller)\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c10-dbefca7d",
   "metadata": {},
   "source": [
    "You should see thousands of readings per request: in a burst, batches fill up to the client's\n",
    "1 MB `batch.size` long before `linger.ms` matters. Latency is several hundred milliseconds,\n",
    "because records wait behind each other in the client's queue. The zstd topic takes several times\n",
    "less disk.\n",
    "\n",
    "## Change it: a tiny batch.size\n",
    "Predict first: replay the burst again, but cap each batch at 1,000 bytes (`batch.size`), about\n",
    "seven readings. What happens to records per request and to throughput?"
   ]
  },
  {
   "cell_type": "code",
   "id": "c11-220fc369",
   "metadata": {},
   "source": [
    "tiny = send(\"burst, batch.size 1000\", backlog, **{\"batch.size\": 1000})\n",
    "lab.table([list(r.values()) for r in (plain, tiny)], list(plain.keys()))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "c12-aa1fe798",
   "metadata": {},
   "source": [
    "A batch is sent as soon as it is full, so each request carries only a handful of records and the\n",
    "per-request work dominates: throughput drops several times over. In a burst, `batch.size` limits\n",
    "you; in a steady stream, `linger.ms` decides how full batches get.\n",
    "\n",
    "## Clean up\n",
    "Delete the topic. If anything gets stuck, `docker compose down -v` in the `code/` folder resets\n",
    "the whole lab."
   ]
  },
  {
   "cell_type": "code",
   "id": "c13-8d94287e",
   "metadata": {},
   "source": [
    "lab.delete_topics(admin, [TOPIC])\n",
    "print(\"cleaned up\")"
   ],
   "execution_count": null,
   "outputs": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
