30 min · theory + hands-on practical

Rebalances and the new group protocol

Every time a member joins or leaves, a consumer group has to reshuffle its partitions. That reshuffle is a rebalance. Done badly, it pauses the whole group, sometimes for many seconds, several times per deploy. This lesson shows what triggers rebalances, how the old and the new group protocols handle them, and the two timeouts that decide when a member counts as gone.

Theory

Example: the pause at every deploy

Billing runs as three worker instances. Every deploy restarts them one after the other. The grid team noticed that, during every deploy, the billing lag jumped six times: once each time an instance stopped and once each time its replacement started. Every jump meant all three workers had paused, not just the one being restarted.

The same week, at 02:00, the tariff database answered one query after 40 seconds instead of 40 milliseconds. The worker waiting for it was thrown out of the group. Its partitions went to the other two, which re-processed the batch it was working on. When the stuck worker woke up, it found its partitions gone.

What triggers a rebalance

A group rebalances whenever its membership or its subscription changes:

The classic protocol: eager and cooperative

In the classic protocol, a broker acts as the group's coordinator, but the members work out the assignment themselves. One member, the group leader, runs an assignor, a function that maps partitions to members, and the coordinator passes its plan to everyone.

Assignors come in two styles:

Worker C leaves a group of A, B and C. With an eager rebalance, A and B also give up all their partitions until the new assignment arrives. With an incremental rebalance, A and B keep reading and only C's partitions move.

What to notice: in the eager picture, two healthy workers stop because a third one left.

The consumer protocol (KIP-848)

Kafka 4.0 made a new group protocol generally available, designed in KIP-848 (a KIP, Kafka Improvement Proposal, is the design document for a change). With it, the broker computes the assignment, and members learn their target assignment through their regular heartbeats. Changes are incremental by design: there is no moment where the whole group stops and waits.

The Python client supports it since confluent-kafka 2.12. You opt in per consumer:

consumer = Consumer({"bootstrap.servers": BOOTSTRAP, "group.id": "billing",
                     "group.protocol": "consumer"})

Three classic settings move to the broker, as settings of the group (consumer.session.timeout.ms, consumer.heartbeat.interval.ms) or a choice of server-side assignor (group.remote.assignor, uniform or range). The client then rejects session.timeout.ms, heartbeat.interval.ms and partition.assignment.strategy. Because members heartbeat every 5 seconds by default, a change can take a few seconds to settle, but members whose partitions don't move never stop reading.

Two timeouts, and static membership

A consumer has two separate ways to be declared gone:

Inside a consumer process, a background thread sends heartbeats to the group coordinator; if they stop for session.timeout.ms, the process is presumed dead. If your code doesn't call consume() for max.poll.interval.ms, the member is presumed stuck and leaves the group.

What to notice: heartbeats come from a background thread, so a busy handler keeps heartbeating. Only max.poll.interval.ms catches a handler that never comes back.

Static membership gives each instance a fixed identity, group.instance.id, for example the pod name in Kubernetes. When a static member restarts within its session timeout, it gets its old partitions back with no rebalance at all. The price: its partitions wait, unread, while it's away.

In production

Practical

Note

Scripts: code/module-03/session_feed.py and code/module-03/billing_worker.py (the notebook starts and restarts them) · Notebook: code/module-03/03-rebalances-and-the-new-group-protocol.ipynb · Lab services: kafka-1, kafka-2, kafka-3 · Time: ~16 min (about 2 minutes of waiting)

What you'll do: run the same rolling restart of three workers under four set-ups and measure the disruption from the workers' own logs. Then watch a stuck worker get thrown out.

Step 1: Start the feed

The feed writes 40 events a second to the 6-partition topic m03l03-sessions.

Step 2: A rolling restart, measured

The billing worker logs every assignment change from its on_assign and on_revoke callbacks:

code/module-03/billing_worker.py
    def on_assign(consumer, partitions):
        print(f"{now()} {args.name} ASSIGNED {sorted(p.partition for p in partitions)}", flush=True)

rolling_restart(label, *flags) starts A, B and C, waits 10 seconds, then restarts them one at a time: a clean stop, a new instance, a wait until it has partitions. disruption() reads the logs and counts two things: how many times a partition was revoked from a worker, and the total seconds without owner summed over all partitions.

set-up         partitions revoked  seconds without owner
-------------  ------------------  ---------------------
classic eager  24                  5.5

Here is worker B during A's restart. B itself was not being restarted:

16:23:08.011 B ASSIGNED [2, 3]
16:23:17.043 B REVOKED [2, 3]
16:23:17.049 B ASSIGNED [2, 3]
16:23:19.335 B REVOKED [2, 3]

B gave up its partitions and got the same ones back: a stop-the-world rebalance.

Step 3: Cooperative rebalancing

code/module-03/03-rebalances-and-the-new-group-protocol.ipynb
results.append(rolling_restart("classic cooperative", "--assignor", "cooperative-sticky"))

Step 4: The new consumer protocol (KIP-848)

code/module-03/03-rebalances-and-the-new-group-protocol.ipynb
results.append(rolling_restart("consumer protocol", "--protocol", "consumer"))

Step 5: Static membership

code/module-03/03-rebalances-and-the-new-group-protocol.ipynb
results.append(rolling_restart("static members", "--static", "--session-timeout-ms", "10000"))

All four runs (your numbers will vary):

set-up               partitions revoked  seconds without owner
-------------------  ------------------  ---------------------
classic eager        24                  5.5
classic cooperative  12                  7.7
consumer protocol    14                  6.0
static members       12                  0.9

The last column varies most for the consumer protocol: in three of our runs it was 1.0, 1.0 and 6.0 seconds. The "partitions revoked" column hardly varies.

Break it: a stuck worker

Predict first: A and B share the partitions. After 100 events, A's handler stalls for 15 seconds. A's max.poll.interval.ms is 8 seconds. What happens to A's partitions, and to A when it wakes?

code/module-03/03-rebalances-and-the-new-group-protocol.ipynb
a = lab.start_script("module-03/billing_worker.py", "--name", "A", *flags, "--stall-at", "100", "--stall-ms", "15000", name="m03l03-stuck-A")

The members every 3 seconds, then the two workers' logs:

16:24:35 {'A': [0, 1, 2], 'B': [3, 4, 5]}
16:24:38 {'B': [0, 1, 2, 3, 4, 5]}
16:24:41 {'B': [0, 1, 2, 3, 4, 5]}
16:24:44 {'B': [0, 1, 2, 3, 4, 5]}
16:24:47 {'A': [0, 1, 2], 'B': [3, 4, 5]}
--- m03l03-stuck-A
16:24:28.900 A ASSIGNED [0, 1, 2]
16:24:29.618 A STALLING for 15000 ms
16:24:44.623 A ERROR KafkaError{code=_MAX_POLL_EXCEEDED,val=-147,str="Application maximum poll interval (8000ms) exceeded by 334ms"}
16:24:44.623 A LOST [0, 1, 2]
16:24:47.038 A ASSIGNED [0, 1, 2]
--- m03l03-stuck-B
16:24:28.900 B ASSIGNED [3, 4, 5]
16:24:37.965 B REVOKED [3, 4, 5]
16:24:37.971 B ASSIGNED [0, 1, 2, 3, 4, 5]
16:24:47.033 B REVOKED [0, 1, 2, 3, 4, 5]
16:24:47.038 B ASSIGNED [3, 4, 5]

Note

This step uses the classic protocol. When we ran the same stall with group.protocol=consumer and confluent-kafka 2.15.1, the stuck member kept its partitions until it woke up. Check how your client version behaves before relying on max.poll.interval.ms with the new protocol.

What just happened

Each restart changed the group's membership twice, and how much that disturbed the others depended on the protocol. The eager classic run revoked twice as many partitions as the others, because healthy workers like B gave up everything at every change. The cooperative run revoked only the moving partitions, but those waited for two rebalance rounds, so its "seconds without owner" stayed high. The consumer protocol also revoked only what had to move, and B never stopped. Its moved partitions got their new owner at the next heartbeat, every 5 seconds, which is why its number ranged from 1 to 6 seconds. Static members got their own partitions straight back, at about one second each time. (With handlers this fast, an eager rebalance finishes quickly. With slow batches, every eager member must finish its batch before anyone continues, and the pause grows.) In the break-it step, heartbeats kept A "alive", but max.poll.interval.ms caught the stall: A's client left the group, and B took all six partitions. When A woke up it got _MAX_POLL_EXCEEDED, lost its partitions and rejoined. B had re-read the events A had been working on.

Clean up

The last cell stops the feed and deletes the topic and the five groups. If anything gets stuck, docker compose down -v in the code/ folder resets the whole lab.

Key takeaways

Next: Work queues with share groups