Notebook · code/module-03/03-rebalances-and-the-new-group-protocol.ipynb

Download 03-rebalances-and-the-new-group-protocol.ipynb

Open it in JupyterLab or VS Code from the lab folder to run it. This page is a read-only view.

Rebalances and the new group protocol — practical

Three billing workers share six partitions while a feed writes 40 events a second. You restart them one by one, like a deploy, under four set-ups, and measure how much each one disturbs the group. Then a worker gets stuck and is thrown out of the group.

Lab services: kafka-1, kafka-2, kafka-3 · Scripts: module-03/session_feed.py, module-03/billing_worker.py · Lesson: modules/03-consumers/03-rebalances-and-the-new-group-protocol.md

# Setup: find the lab's shared helpers and check the lab is running
import sys, pathlib
lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / "common").is_dir())
sys.path.insert(0, str(lab_root))

import re, time
from datetime import datetime
from common import settings, lab

lab.require_kafka()
lab.print_versions()

Step 1: Start the feed

from confluent_kafka.admin import AdminClient

TOPIC = "m03l03-sessions"
admin = AdminClient({"bootstrap.servers": settings.KAFKA_BOOTSTRAP})
lab.recreate_topic(admin, TOPIC, partitions=6)
feed = lab.start_script("module-03/session_feed.py", "--topic", TOPIC, "--rate", "40", name="m03l03-feed")

Step 2: A rolling restart, measured

rolling_restart() starts workers A, B and C, lets them settle, then restarts them one at a time: stop A cleanly (Ctrl+C), start a new A, wait until it has partitions, then B, then C. Every worker logs each ASSIGNED and REVOKED with a timestamp (its on_assign and on_revoke callbacks). From those logs the function counts: - partitions revoked: how many times a partition was taken from a worker; - seconds without owner: summed over all partitions, how long each one had no worker at all.

EVENT = re.compile(r"^(\d\d:\d\d:\d\d\.\d{3}) \w+ (ASSIGNED|REVOKED|LOST) \[(.*)\]")

def disruption(scripts, since):
    """Partitions revoked, and partition-seconds without an owner, after `since` (time of day)."""
    events = []
    for s in scripts:
        for line in s.lines():
            m = EVENT.match(line)
            if m:
                t = datetime.strptime(m.group(1), "%H:%M:%S.%f").time()
                events.append((t, m.group(2), [int(p) for p in m.group(3).split(",") if p.strip()]))
    owners, free_since, revoked, unowned = {p: 0 for p in range(6)}, {}, 0, 0.0
    secs = lambda t: t.hour * 3600 + t.minute * 60 + t.second + t.microsecond / 1e6
    for t, kind, parts in sorted(events):
        for p in parts:
            if kind == "ASSIGNED":
                if owners[p] == 0 and p in free_since and t >= since:
                    unowned += secs(t) - max(secs(free_since[p]), secs(since))
                owners[p] += 1
            else:
                owners[p] = max(0, owners[p] - 1)
                revoked += t >= since
                if owners[p] == 0:
                    free_since[p] = t
    return revoked, round(unowned, 1)

The restart itself:

def rolling_restart(label, *flags):
    group = f"m03l03-{label.replace(' ', '-')}"
    start = lambda n, tag: lab.start_script("module-03/billing_worker.py", "--name", n, "--topic", TOPIC,
                                            "--group", group, "--work-ms", "5", *flags, name=f"{group}-{tag}")
    running = {n: start(n, n) for n in "ABC"}
    every = list(running.values())
    time.sleep(10)                                         # let the group settle
    since = datetime.now().time()
    for n in "ABC":
        lab.stop_script(running[n])                        # a clean stop, like a deploy
        running[n] = start(n, n + "2")
        every.append(running[n])
        lab.wait_until(lambda: any(re.search(r"ASSIGNED \[\d", l) for l in running[n].lines()),
                       timeout=60, interval=0.2, what=f"{n} to get partitions")
        time.sleep(2)
    for s in running.values():
        lab.stop_script(s)
    revoked, unowned = disruption(every, since)
    return {"set-up": label, "partitions revoked": revoked, "seconds without owner": unowned}

results = [rolling_restart("classic eager")]
lab.table([list(r.values()) for r in results], list(results[0].keys()))

The worker's default is the classic protocol with the eager range,roundrobin assignors. Look at the log of worker B, which was never restarted itself, during A's restart:

b_log = lab.Script("B", None, lab.LOG_DIR / "m03l03-classic-eager-B.log")
for line in b_log.lines()[:12]:
    if "ASSIGNED" in line or "REVOKED" in line:
        print(line)

B gives up all its partitions each time anyone leaves or joins, then gets them back. That's a stop-the-world rebalance.

Step 3: Cooperative rebalancing

The same restart with partition.assignment.strategy=cooperative-sticky.

results.append(rolling_restart("classic cooperative", "--assignor", "cooperative-sticky"))
lab.table([list(r.values()) for r in results], list(results[0].keys()))

Step 4: The new consumer protocol (KIP-848)

The same restart with group.protocol=consumer: the broker computes the assignment.

results.append(rolling_restart("consumer protocol", "--protocol", "consumer"))
lab.table([list(r.values()) for r in results], list(results[0].keys()))

Step 5: Static membership

Back to the classic protocol, but each worker has a fixed group.instance.id (its name) and a 10-second session timeout. A worker that comes back with the same id within the timeout gets its old partitions back without a rebalance.

results.append(rolling_restart("static members", "--static", "--session-timeout-ms", "10000"))
lab.table([list(r.values()) for r in results], list(results[0].keys()))

Compare the four rows: the eager protocol revokes the most partitions; cooperative, the consumer protocol and static membership revoke only what has to move. Your numbers will vary a little.

Break it: a stuck worker

Predict first: workers A and B share the partitions. After 100 events, A's handler stalls for 15 seconds (a slow database call). A's max.poll.interval.ms is 8 seconds. What happens to A's partitions, and to A when it wakes up?

group = "m03l03-stuck"
flags = ["--topic", TOPIC, "--group", group, "--work-ms", "5", "--max-poll-ms", "8000", "--session-timeout-ms", "6000"]
b = lab.start_script("module-03/billing_worker.py", "--name", "B", *flags, name="m03l03-stuck-B")
lab.wait_until(lambda: b.lines("ASSIGNED [0"), timeout=30, what="B to get partitions")
a = lab.start_script("module-03/billing_worker.py", "--name", "A", *flags, "--stall-at", "100", "--stall-ms", "15000", name="m03l03-stuck-A")
for _ in range(8):
    time.sleep(3)
    print(time.strftime("%H:%M:%S"), lab.group_members(admin, group))
for s in (a, b):
    lab.stop_script(s)
for s in (a, b):
    print(f"--- {s.name}")
    for line in s.lines():
        if any(k in line for k in ("ASSIGNED", "REVOKED", "LOST", "STALLING", "ERROR")):
            print(line)

A stops calling consume() for 15 seconds. After 8 seconds its client gives up its membership, and B takes all six partitions. When A wakes up, its next consume() returns _MAX_POLL_EXCEEDED, it learns its partitions were lost, and it rejoins. The events A was working on when it stalled were handed to B as well.

Clean up

Stop the feed and delete the topic and the groups. If anything gets stuck, docker compose down -v in the code/ folder resets the whole lab.

lab.stop_script(feed)
lab.delete_topics(admin, [TOPIC])
admin.delete_consumer_groups(["m03l03-classic-eager", "m03l03-classic-cooperative",
                              "m03l03-consumer-protocol", "m03l03-static-members", "m03l03-stuck"])
print("cleaned up")