Notebook · code/module-02/03-batching-compression-and-throughput.ipynb

Download 03-batching-compression-and-throughput.ipynb

Open it in JupyterLab or VS Code from the lab folder to run it. This page is a read-only view.

Batching, compression and throughput — practical

You measure two kinds of writing the gateway does: the live trickle of readings, and the replay of 100,000 readings buffered during an outage. You see what linger.ms, compression and batch.size change: requests, latency, throughput and bytes on disk.

Lab services: kafka-1, kafka-2, kafka-3 · Lesson: modules/02-producers/03-batching-compression-and-throughput.md

# Setup: find the lab's shared helpers and check the lab is running
import sys, pathlib
lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / "common").is_dir())
sys.path.insert(0, str(lab_root))

import json, time, statistics
from common import settings, lab
from common.scenario import Brightlane, to_json

lab.require_kafka()
lab.print_versions()

Step 1: Prepare the readings

100,000 meter readings, already serialized, keyed by charger.

from confluent_kafka import Producer
from confluent_kafka.admin import AdminClient

TOPIC = "m02l03-readings"
admin = AdminClient({"bootstrap.servers": settings.KAFKA_BOOTSTRAP})
backlog = [(e["charger_id"], to_json(e)) for _, e in Brightlane().stream(sessions=460) if "type" not in e][:100_000]
size_mb = sum(len(v) for _, v in backlog) / 1e6
print(f"{len(backlog):,} readings, {size_mb:.1f} MB, average {size_mb * 1e6 / len(backlog):.0f} bytes")

Step 2: A function that times one write

send() writes records with the given settings, either as fast as possible or at a steady rate per second, and reports (after one warm-up record that opens the connections): - readings/s: from the first produce() to the last delivery report; - p50 / p99 ms: each record's time from produce() to the broker's acknowledgement (msg.latency()); - records/request: how many readings each produce request carried, from the client's statistics.

def send(label, records, rate=None, **config):
    lab.recreate_topic(admin, TOPIC, partitions=6)
    latencies, stats = [], []
    p = Producer({**settings.PRODUCER_DEFAULTS, **config,
                  "statistics.interval.ms": 1000, "stats_cb": stats.append})
    p.produce(TOPIC, value=b"warm-up")            # connect and get a producer id before timing
    p.flush(10)
    start = time.monotonic()
    for i, (key, value) in enumerate(records):
        if rate:                                  # steady stream: wait for this record's send time
            time.sleep(max(0.0, start + i / rate - time.monotonic()))
        for _ in range(1000):                     # bounded retry if the local queue is full
            try:
                p.produce(TOPIC, key=key, value=value,
                          on_delivery=lambda err, msg: latencies.append(msg.latency()))
                break
            except BufferError:
                p.poll(0.05)
        p.poll(0)
    p.flush(60)
    seconds = time.monotonic() - start
    p.poll(1.2)                                   # one more statistics report after the last delivery
    requests = sum(b["req"].get("Produce", 0) for b in json.loads(stats[-1])["brokers"].values()) - 1
    q = statistics.quantiles(latencies, n=100)
    return {"settings": label, "readings/s": round(len(records) / seconds),
            "p50 ms": round(q[49] * 1000, 1), "p99 ms": round(q[98] * 1000, 1),
            "records/request": round(len(records) / requests, 1)}

Step 3: A steady stream with linger.ms 0, 5 and 50

Live, the gateway sends a steady stream, not a burst. linger.ms is how long the client waits for more records before it sends a batch: 0 sends at once, 5 is the default (in the Python client, and in Java since Kafka 4.0), 50 waits longer. Each run sends 5,000 readings at 2,500 a second, so it takes two seconds.

steady = [send(f"linger {ms}", backlog[:5000], rate=2500, **{"linger.ms": ms}) for ms in (0, 5, 50)]
lab.table([list(r.values()) for r in steady], list(steady[0].keys()))

You should see the same throughput in every row, 2,500 a second, because that's the rate we send at. The requests differ a lot: with linger 0 a request carries one or two readings; with linger 50 it carries about a hundred, and each reading waits about 50 ms longer. Your numbers will differ a little.

Step 4: The backlog, as fast as possible, with and without compression

Now the replay: all 100,000 readings at once. Compression works on whole batches. Compare throughput, and the bytes the brokers store for the topic.

def stored_bytes():
    """Total size of the topic's partitions on all brokers, from kafka-log-dirs.sh."""
    out = lab.kafka_cli("kafka-log-dirs.sh", "--bootstrap-server", "localhost:19092",
                        "--describe", "--topic-list", TOPIC, echo=False)
    data = json.loads(out.splitlines()[-1])
    return sum(p["size"] for b in data["brokers"] for d in b["logDirs"] for p in d["partitions"])

plain = send("burst, none", backlog)
plain_bytes = stored_bytes()
zstd = send("burst, zstd", backlog, **{"compression.type": "zstd"})
zstd_bytes = stored_bytes()
lab.table([list(r.values()) for r in (plain, zstd)], list(plain.keys()))
print(f"\nstored on disk (3 replicas): none {plain_bytes / 1e6:.1f} MB, zstd {zstd_bytes / 1e6:.1f} MB "
      f"({plain_bytes / zstd_bytes:.1f}x smaller)")

You should see thousands of readings per request: in a burst, batches fill up to the client's 1 MB batch.size long before linger.ms matters. Latency is several hundred milliseconds, because records wait behind each other in the client's queue. The zstd topic takes several times less disk.

Change it: a tiny batch.size

Predict first: replay the burst again, but cap each batch at 1,000 bytes (batch.size), about seven readings. What happens to records per request and to throughput?

tiny = send("burst, batch.size 1000", backlog, **{"batch.size": 1000})
lab.table([list(r.values()) for r in (plain, tiny)], list(plain.keys()))

A batch is sent as soon as it is full, so each request carries only a handful of records and the per-request work dominates: throughput drops several times over. In a burst, batch.size limits you; in a steady stream, linger.ms decides how full batches get.

Clean up

Delete the topic. If anything gets stuck, docker compose down -v in the code/ folder resets the whole lab.

lab.delete_topics(admin, [TOPIC])
print("cleaned up")