Rebalances and the new group protocol
Every time a member joins or leaves, a consumer group has to reshuffle its partitions. That reshuffle is a rebalance. Done badly, it pauses the whole group, sometimes for many seconds, several times per deploy. This lesson shows what triggers rebalances, how the old and the new group protocols handle them, and the two timeouts that decide when a member counts as gone.
Theory
Example: the pause at every deploy
Billing runs as three worker instances. Every deploy restarts them one after the other. The grid team noticed that, during every deploy, the billing lag jumped six times: once each time an instance stopped and once each time its replacement started. Every jump meant all three workers had paused, not just the one being restarted.
The same week, at 02:00, the tariff database answered one query after 40 seconds instead of 40 milliseconds. The worker waiting for it was thrown out of the group. Its partitions went to the other two, which re-processed the batch it was working on. When the stuck worker woke up, it found its partitions gone.
What triggers a rebalance
A group rebalances whenever its membership or its subscription changes:
- a member joins, for example a new instance starts;
- a member leaves cleanly, calling
close(); - a member dies: it stops sending heartbeats for
session.timeout.ms; - a member is stuck: it doesn't call
poll()orconsume()formax.poll.interval.ms; - a subscribed topic gets more partitions, or the subscription changes.
The classic protocol: eager and cooperative
In the classic protocol, a broker acts as the group's coordinator, but the members work out the assignment themselves. One member, the group leader, runs an assignor, a function that maps partitions to members, and the coordinator passes its plan to everyone.
Assignors come in two styles:
- Eager (
range,roundrobin): every member gives up all its partitions at the start of the rebalance, waits until everyone has rejoined, then gets a new set. Nothing is read in between. This is the Python client's default (range,roundrobin). - Cooperative (
cooperative-sticky): members keep the partitions that don't move. Only the partitions changing owner are revoked, in a first round, and handed out in a second.
What to notice: in the eager picture, two healthy workers stop because a third one left.
The consumer protocol (KIP-848)
Kafka 4.0 made a new group protocol generally available, designed in KIP-848 (a KIP, Kafka Improvement Proposal, is the design document for a change). With it, the broker computes the assignment, and members learn their target assignment through their regular heartbeats. Changes are incremental by design: there is no moment where the whole group stops and waits.
The Python client supports it since confluent-kafka 2.12. You opt in per consumer:
consumer = Consumer({"bootstrap.servers": BOOTSTRAP, "group.id": "billing",
"group.protocol": "consumer"})
Three classic settings move to the broker, as settings of the group (consumer.session.timeout.ms,
consumer.heartbeat.interval.ms) or a choice of server-side assignor (group.remote.assignor,
uniform or range). The client then rejects session.timeout.ms, heartbeat.interval.ms and
partition.assignment.strategy. Because members heartbeat every 5 seconds by default, a change can
take a few seconds to settle, but members whose partitions don't move never stop reading.
Two timeouts, and static membership
A consumer has two separate ways to be declared gone:
What to notice: heartbeats come from a background thread, so a busy handler keeps heartbeating.
Only max.poll.interval.ms catches a handler that never comes back.
session.timeout.ms(45 s by default) catches a dead process. Until it expires, the dead member's partitions are not read by anyone, as lesson 3.1 showed.max.poll.interval.ms(5 minutes by default) catches a stuck one. The client leaves the group on its own; when the code finally callsconsume()again, it gets_MAX_POLL_EXCEEDEDand learns its partitions are lost.
Static membership gives each instance a fixed identity, group.instance.id, for example the pod
name in Kubernetes. When a static member restarts within its session timeout, it gets its old
partitions back with no rebalance at all. The price: its partitions wait, unread, while it's away.
In production
- With Kafka 4.x brokers and confluent-kafka 2.12 or later, set
group.protocol=consumer. With older brokers, usepartition.assignment.strategy=cooperative-sticky. - Keep the time to process one
consume()batch far belowmax.poll.interval.ms. If a downstream call can hang, give it its own timeout, and fetch smaller batches. - Use static membership where instances have stable names and restart quickly (StatefulSets, VMs), with a session timeout longer than a normal restart.
- Make processing idempotent anyway (lesson 3.2): every rebalance can hand a half-done batch to another member.
- Watch the rebalance rate per group (lesson 8.3). Frequent rebalances point to crashing or stuck members.
Practical
Note
Scripts: code/module-03/session_feed.py and
code/module-03/billing_worker.py (the notebook starts and restarts them)
· Notebook: code/module-03/03-rebalances-and-the-new-group-protocol.ipynb
· Lab services: kafka-1, kafka-2, kafka-3 · Time: ~16 min (about 2 minutes of waiting)
What you'll do: run the same rolling restart of three workers under four set-ups and measure the disruption from the workers' own logs. Then watch a stuck worker get thrown out.
Step 1: Start the feed
The feed writes 40 events a second to the 6-partition topic m03l03-sessions.
Step 2: A rolling restart, measured
The billing worker logs every assignment change from its on_assign and on_revoke callbacks:
def on_assign(consumer, partitions):
print(f"{now()} {args.name} ASSIGNED {sorted(p.partition for p in partitions)}", flush=True)
rolling_restart(label, *flags) starts A, B and C, waits 10 seconds, then restarts them one at a
time: a clean stop, a new instance, a wait until it has partitions. disruption() reads the logs and
counts two things: how many times a partition was revoked from a worker, and the total
seconds without owner summed over all partitions.
set-up partitions revoked seconds without owner
------------- ------------------ ---------------------
classic eager 24 5.5
Here is worker B during A's restart. B itself was not being restarted:
16:23:08.011 B ASSIGNED [2, 3]
16:23:17.043 B REVOKED [2, 3]
16:23:17.049 B ASSIGNED [2, 3]
16:23:19.335 B REVOKED [2, 3]
B gave up its partitions and got the same ones back: a stop-the-world rebalance.
Step 3: Cooperative rebalancing
results.append(rolling_restart("classic cooperative", "--assignor", "cooperative-sticky"))
Step 4: The new consumer protocol (KIP-848)
results.append(rolling_restart("consumer protocol", "--protocol", "consumer"))
Step 5: Static membership
results.append(rolling_restart("static members", "--static", "--session-timeout-ms", "10000"))
All four runs (your numbers will vary):
set-up partitions revoked seconds without owner
------------------- ------------------ ---------------------
classic eager 24 5.5
classic cooperative 12 7.7
consumer protocol 14 6.0
static members 12 0.9
The last column varies most for the consumer protocol: in three of our runs it was 1.0, 1.0 and 6.0 seconds. The "partitions revoked" column hardly varies.
Break it: a stuck worker
Predict first: A and B share the partitions. After 100 events, A's handler stalls for 15 seconds.
A's max.poll.interval.ms is 8 seconds. What happens to A's partitions, and to A when it wakes?
a = lab.start_script("module-03/billing_worker.py", "--name", "A", *flags, "--stall-at", "100", "--stall-ms", "15000", name="m03l03-stuck-A")
The members every 3 seconds, then the two workers' logs:
16:24:35 {'A': [0, 1, 2], 'B': [3, 4, 5]}
16:24:38 {'B': [0, 1, 2, 3, 4, 5]}
16:24:41 {'B': [0, 1, 2, 3, 4, 5]}
16:24:44 {'B': [0, 1, 2, 3, 4, 5]}
16:24:47 {'A': [0, 1, 2], 'B': [3, 4, 5]}
--- m03l03-stuck-A
16:24:28.900 A ASSIGNED [0, 1, 2]
16:24:29.618 A STALLING for 15000 ms
16:24:44.623 A ERROR KafkaError{code=_MAX_POLL_EXCEEDED,val=-147,str="Application maximum poll interval (8000ms) exceeded by 334ms"}
16:24:44.623 A LOST [0, 1, 2]
16:24:47.038 A ASSIGNED [0, 1, 2]
--- m03l03-stuck-B
16:24:28.900 B ASSIGNED [3, 4, 5]
16:24:37.965 B REVOKED [3, 4, 5]
16:24:37.971 B ASSIGNED [0, 1, 2, 3, 4, 5]
16:24:47.033 B REVOKED [0, 1, 2, 3, 4, 5]
16:24:47.038 B ASSIGNED [3, 4, 5]
Note
This step uses the classic protocol. When we ran the same stall with group.protocol=consumer
and confluent-kafka 2.15.1, the stuck member kept its partitions until it woke up. Check how your
client version behaves before relying on max.poll.interval.ms with the new protocol.
What just happened
Each restart changed the group's membership twice, and how much that disturbed the others depended
on the protocol. The eager classic run revoked twice as many partitions as the others, because
healthy workers like B gave up everything at every change. The cooperative run revoked only the
moving partitions, but those waited for two rebalance rounds, so its "seconds without owner" stayed
high. The consumer protocol also revoked only what had to move, and B never stopped. Its moved
partitions got their new owner at the next heartbeat, every 5 seconds, which is why its number
ranged from 1 to 6 seconds. Static members got their own partitions straight back, at about one
second each time. (With handlers this fast, an eager rebalance finishes quickly. With slow batches,
every eager member must finish its batch before anyone continues, and the pause grows.) In the
break-it step, heartbeats kept A "alive", but max.poll.interval.ms caught the stall: A's client
left the group, and B took all six partitions. When A woke up it got _MAX_POLL_EXCEEDED, lost its
partitions and rejoined. B had re-read the events A had been working on.
Clean up
The last cell stops the feed and deletes the topic and the five groups. If anything gets stuck,
docker compose down -v in the code/ folder resets the whole lab.
Key takeaways
- A group rebalances when members join, leave, die, get stuck, or when partitions change.
- Eager assignors stop the whole group at every change; cooperative and KIP-848 move only what must move.
- Use
group.protocol=consumer(Kafka 4.x, confluent-kafka 2.12+); otherwisecooperative-sticky. session.timeout.mscatches dead processes;max.poll.interval.mscatches stuck handlers.- Static membership (
group.instance.id) lets a member restart without any rebalance.