Download 03-rebalances-and-the-new-group-protocol.ipynb
Open it in JupyterLab or VS Code from the lab folder to run it. This page is a read-only view.
Rebalances and the new group protocol — practical
Three billing workers share six partitions while a feed writes 40 events a second. You restart them one by one, like a deploy, under four set-ups, and measure how much each one disturbs the group. Then a worker gets stuck and is thrown out of the group.
Lab services: kafka-1, kafka-2, kafka-3 · Scripts: module-03/session_feed.py,
module-03/billing_worker.py · Lesson: modules/03-consumers/03-rebalances-and-the-new-group-protocol.md
# Setup: find the lab's shared helpers and check the lab is running import sys, pathlib lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / "common").is_dir()) sys.path.insert(0, str(lab_root)) import re, time from datetime import datetime from common import settings, lab lab.require_kafka() lab.print_versions()
Step 1: Start the feed
from confluent_kafka.admin import AdminClient TOPIC = "m03l03-sessions" admin = AdminClient({"bootstrap.servers": settings.KAFKA_BOOTSTRAP}) lab.recreate_topic(admin, TOPIC, partitions=6) feed = lab.start_script("module-03/session_feed.py", "--topic", TOPIC, "--rate", "40", name="m03l03-feed")
Step 2: A rolling restart, measured
rolling_restart() starts workers A, B and C, lets them settle, then restarts them one at a time:
stop A cleanly (Ctrl+C), start a new A, wait until it has partitions, then B, then C. Every worker
logs each ASSIGNED and REVOKED with a timestamp (its on_assign and on_revoke callbacks).
From those logs the function counts:
- partitions revoked: how many times a partition was taken from a worker;
- seconds without owner: summed over all partitions, how long each one had no worker at all.
EVENT = re.compile(r"^(\d\d:\d\d:\d\d\.\d{3}) \w+ (ASSIGNED|REVOKED|LOST) \[(.*)\]") def disruption(scripts, since): """Partitions revoked, and partition-seconds without an owner, after `since` (time of day).""" events = [] for s in scripts: for line in s.lines(): m = EVENT.match(line) if m: t = datetime.strptime(m.group(1), "%H:%M:%S.%f").time() events.append((t, m.group(2), [int(p) for p in m.group(3).split(",") if p.strip()])) owners, free_since, revoked, unowned = {p: 0 for p in range(6)}, {}, 0, 0.0 secs = lambda t: t.hour * 3600 + t.minute * 60 + t.second + t.microsecond / 1e6 for t, kind, parts in sorted(events): for p in parts: if kind == "ASSIGNED": if owners[p] == 0 and p in free_since and t >= since: unowned += secs(t) - max(secs(free_since[p]), secs(since)) owners[p] += 1 else: owners[p] = max(0, owners[p] - 1) revoked += t >= since if owners[p] == 0: free_since[p] = t return revoked, round(unowned, 1)
The restart itself:
def rolling_restart(label, *flags): group = f"m03l03-{label.replace(' ', '-')}" start = lambda n, tag: lab.start_script("module-03/billing_worker.py", "--name", n, "--topic", TOPIC, "--group", group, "--work-ms", "5", *flags, name=f"{group}-{tag}") running = {n: start(n, n) for n in "ABC"} every = list(running.values()) time.sleep(10) # let the group settle since = datetime.now().time() for n in "ABC": lab.stop_script(running[n]) # a clean stop, like a deploy running[n] = start(n, n + "2") every.append(running[n]) lab.wait_until(lambda: any(re.search(r"ASSIGNED \[\d", l) for l in running[n].lines()), timeout=60, interval=0.2, what=f"{n} to get partitions") time.sleep(2) for s in running.values(): lab.stop_script(s) revoked, unowned = disruption(every, since) return {"set-up": label, "partitions revoked": revoked, "seconds without owner": unowned} results = [rolling_restart("classic eager")] lab.table([list(r.values()) for r in results], list(results[0].keys()))
The worker's default is the classic protocol with the eager range,roundrobin assignors.
Look at the log of worker B, which was never restarted itself, during A's restart:
b_log = lab.Script("B", None, lab.LOG_DIR / "m03l03-classic-eager-B.log") for line in b_log.lines()[:12]: if "ASSIGNED" in line or "REVOKED" in line: print(line)
B gives up all its partitions each time anyone leaves or joins, then gets them back. That's a stop-the-world rebalance.
Step 3: Cooperative rebalancing
The same restart with partition.assignment.strategy=cooperative-sticky.
results.append(rolling_restart("classic cooperative", "--assignor", "cooperative-sticky")) lab.table([list(r.values()) for r in results], list(results[0].keys()))
Step 4: The new consumer protocol (KIP-848)
The same restart with group.protocol=consumer: the broker computes the assignment.
results.append(rolling_restart("consumer protocol", "--protocol", "consumer")) lab.table([list(r.values()) for r in results], list(results[0].keys()))
Step 5: Static membership
Back to the classic protocol, but each worker has a fixed group.instance.id (its name) and a
10-second session timeout. A worker that comes back with the same id within the timeout gets
its old partitions back without a rebalance.
results.append(rolling_restart("static members", "--static", "--session-timeout-ms", "10000")) lab.table([list(r.values()) for r in results], list(results[0].keys()))
Compare the four rows: the eager protocol revokes the most partitions; cooperative, the consumer protocol and static membership revoke only what has to move. Your numbers will vary a little.
Break it: a stuck worker
Predict first: workers A and B share the partitions. After 100 events, A's handler stalls for 15
seconds (a slow database call). A's max.poll.interval.ms is 8 seconds. What happens to A's
partitions, and to A when it wakes up?
group = "m03l03-stuck" flags = ["--topic", TOPIC, "--group", group, "--work-ms", "5", "--max-poll-ms", "8000", "--session-timeout-ms", "6000"] b = lab.start_script("module-03/billing_worker.py", "--name", "B", *flags, name="m03l03-stuck-B") lab.wait_until(lambda: b.lines("ASSIGNED [0"), timeout=30, what="B to get partitions") a = lab.start_script("module-03/billing_worker.py", "--name", "A", *flags, "--stall-at", "100", "--stall-ms", "15000", name="m03l03-stuck-A") for _ in range(8): time.sleep(3) print(time.strftime("%H:%M:%S"), lab.group_members(admin, group)) for s in (a, b): lab.stop_script(s) for s in (a, b): print(f"--- {s.name}") for line in s.lines(): if any(k in line for k in ("ASSIGNED", "REVOKED", "LOST", "STALLING", "ERROR")): print(line)
A stops calling consume() for 15 seconds. After 8 seconds its client gives up its membership,
and B takes all six partitions. When A wakes up, its next consume() returns
_MAX_POLL_EXCEEDED, it learns its partitions were lost, and it rejoins. The events A was
working on when it stalled were handed to B as well.
Clean up
Stop the feed and delete the topic and the groups. If anything gets stuck, docker compose down -v
in the code/ folder resets the whole lab.
lab.stop_script(feed) lab.delete_topics(admin, [TOPIC]) admin.delete_consumer_groups(["m03l03-classic-eager", "m03l03-classic-cooperative", "m03l03-consumer-protocol", "m03l03-static-members", "m03l03-stuck"]) print("cleaned up")