Download 03-batching-compression-and-throughput.ipynb
Open it in JupyterLab or VS Code from the lab folder to run it. This page is a read-only view.
Batching, compression and throughput — practical
You measure two kinds of writing the gateway does: the live trickle of readings, and the replay
of 100,000 readings buffered during an outage. You see what linger.ms, compression and
batch.size change: requests, latency, throughput and bytes on disk.
Lab services: kafka-1, kafka-2, kafka-3 · Lesson: modules/02-producers/03-batching-compression-and-throughput.md
# Setup: find the lab's shared helpers and check the lab is running import sys, pathlib lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / "common").is_dir()) sys.path.insert(0, str(lab_root)) import json, time, statistics from common import settings, lab from common.scenario import Brightlane, to_json lab.require_kafka() lab.print_versions()
Step 1: Prepare the readings
100,000 meter readings, already serialized, keyed by charger.
from confluent_kafka import Producer from confluent_kafka.admin import AdminClient TOPIC = "m02l03-readings" admin = AdminClient({"bootstrap.servers": settings.KAFKA_BOOTSTRAP}) backlog = [(e["charger_id"], to_json(e)) for _, e in Brightlane().stream(sessions=460) if "type" not in e][:100_000] size_mb = sum(len(v) for _, v in backlog) / 1e6 print(f"{len(backlog):,} readings, {size_mb:.1f} MB, average {size_mb * 1e6 / len(backlog):.0f} bytes")
Step 2: A function that times one write
send() writes records with the given settings, either as fast as possible or at a steady
rate per second, and reports (after one warm-up record that opens the connections):
- readings/s: from the first produce() to the last delivery report;
- p50 / p99 ms: each record's time from produce() to the broker's acknowledgement
(msg.latency());
- records/request: how many readings each produce request carried, from the client's statistics.
def send(label, records, rate=None, **config): lab.recreate_topic(admin, TOPIC, partitions=6) latencies, stats = [], [] p = Producer({**settings.PRODUCER_DEFAULTS, **config, "statistics.interval.ms": 1000, "stats_cb": stats.append}) p.produce(TOPIC, value=b"warm-up") # connect and get a producer id before timing p.flush(10) start = time.monotonic() for i, (key, value) in enumerate(records): if rate: # steady stream: wait for this record's send time time.sleep(max(0.0, start + i / rate - time.monotonic())) for _ in range(1000): # bounded retry if the local queue is full try: p.produce(TOPIC, key=key, value=value, on_delivery=lambda err, msg: latencies.append(msg.latency())) break except BufferError: p.poll(0.05) p.poll(0) p.flush(60) seconds = time.monotonic() - start p.poll(1.2) # one more statistics report after the last delivery requests = sum(b["req"].get("Produce", 0) for b in json.loads(stats[-1])["brokers"].values()) - 1 q = statistics.quantiles(latencies, n=100) return {"settings": label, "readings/s": round(len(records) / seconds), "p50 ms": round(q[49] * 1000, 1), "p99 ms": round(q[98] * 1000, 1), "records/request": round(len(records) / requests, 1)}
Step 3: A steady stream with linger.ms 0, 5 and 50
Live, the gateway sends a steady stream, not a burst. linger.ms is how long the client waits
for more records before it sends a batch: 0 sends at once, 5 is the default (in the Python
client, and in Java since Kafka 4.0), 50 waits longer. Each run sends 5,000 readings at 2,500 a
second, so it takes two seconds.
steady = [send(f"linger {ms}", backlog[:5000], rate=2500, **{"linger.ms": ms}) for ms in (0, 5, 50)] lab.table([list(r.values()) for r in steady], list(steady[0].keys()))
You should see the same throughput in every row, 2,500 a second, because that's the rate we
send at. The requests differ a lot: with linger 0 a request carries one or two readings; with
linger 50 it carries about a hundred, and each reading waits about 50 ms longer. Your numbers
will differ a little.
Step 4: The backlog, as fast as possible, with and without compression
Now the replay: all 100,000 readings at once. Compression works on whole batches. Compare throughput, and the bytes the brokers store for the topic.
def stored_bytes(): """Total size of the topic's partitions on all brokers, from kafka-log-dirs.sh.""" out = lab.kafka_cli("kafka-log-dirs.sh", "--bootstrap-server", "localhost:19092", "--describe", "--topic-list", TOPIC, echo=False) data = json.loads(out.splitlines()[-1]) return sum(p["size"] for b in data["brokers"] for d in b["logDirs"] for p in d["partitions"]) plain = send("burst, none", backlog) plain_bytes = stored_bytes() zstd = send("burst, zstd", backlog, **{"compression.type": "zstd"}) zstd_bytes = stored_bytes() lab.table([list(r.values()) for r in (plain, zstd)], list(plain.keys())) print(f"\nstored on disk (3 replicas): none {plain_bytes / 1e6:.1f} MB, zstd {zstd_bytes / 1e6:.1f} MB " f"({plain_bytes / zstd_bytes:.1f}x smaller)")
You should see thousands of readings per request: in a burst, batches fill up to the client's
1 MB batch.size long before linger.ms matters. Latency is several hundred milliseconds,
because records wait behind each other in the client's queue. The zstd topic takes several times
less disk.
Change it: a tiny batch.size
Predict first: replay the burst again, but cap each batch at 1,000 bytes (batch.size), about
seven readings. What happens to records per request and to throughput?
tiny = send("burst, batch.size 1000", backlog, **{"batch.size": 1000}) lab.table([list(r.values()) for r in (plain, tiny)], list(plain.keys()))
A batch is sent as soon as it is full, so each request carries only a handful of records and the
per-request work dominates: throughput drops several times over. In a burst, batch.size limits
you; in a steady stream, linger.ms decides how full batches get.
Clean up
Delete the topic. If anything gets stuck, docker compose down -v in the code/ folder resets
the whole lab.
lab.delete_topics(admin, [TOPIC]) print("cleaned up")