Notebook · code/module-03/01-consumer-groups-share-the-work.ipynb

Download 01-consumer-groups-share-the-work.ipynb

Open it in JupyterLab or VS Code from the lab folder to run it. This page is a read-only view.

Consumer groups share the work — practical

A session feed writes 40 events a second. You start billing workers one by one, watch the group share the six partitions and the lag rise and fall, add one worker too many, and then crash one.

Lab services: kafka-1, kafka-2, kafka-3 · Scripts: module-03/session_feed.py, module-03/billing_worker.py · Lesson: modules/03-consumers/01-consumer-groups-share-the-work.md

# Setup: find the lab's shared helpers and check the lab is running
import sys, pathlib
lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / "common").is_dir())
sys.path.insert(0, str(lab_root))

import time
from common import settings, lab

lab.require_kafka()
lab.print_versions()

Step 1: Start the session feed

The topic has 6 partitions. The feed script writes session events at 40 a second until it's stopped. The notebook starts it in the background; its output goes to a log file. (You can run the same command in a terminal instead and watch it.)

from confluent_kafka.admin import AdminClient

TOPIC, GROUP = "m03l01-sessions", "m03l01-billing"
admin = AdminClient({"bootstrap.servers": settings.KAFKA_BOOTSTRAP})
lab.recreate_topic(admin, TOPIC, partitions=6)
admin.delete_consumer_groups([GROUP])

feed = lab.start_script("module-03/session_feed.py", "--topic", TOPIC, "--rate", "40", name="m03l01-feed")
time.sleep(3)
feed.tail(1)

You should see the feed's start line: session feed: 40,000 events -> m03l01-sessions at 40/s ....

Step 2: One billing worker

The worker handles one event in 50 ms, so it manages about 20 events a second: half the feed's rate. Lag is how far the group is behind: for each partition, the end offset minus the group's committed offset.

from confluent_kafka import ConsumerGroupTopicPartitions, TopicPartition
from confluent_kafka.admin import OffsetSpec

def lag_now():
    """Total lag of the group, from its committed offsets and the partitions' end offsets."""
    tps = [TopicPartition(TOPIC, p) for p in range(6)]
    committed = admin.list_consumer_group_offsets([ConsumerGroupTopicPartitions(GROUP, tps)])[GROUP].result()
    ends = admin.list_offsets({tp: OffsetSpec.latest() for tp in tps})
    return sum(ends[TopicPartition(TOPIC, tp.partition)].result().offset - max(tp.offset, 0)
               for tp in committed.topic_partitions)

def watch(seconds, every=5):
    for _ in range(int(seconds / every)):
        time.sleep(every)
        print(f"{time.strftime('%H:%M:%S')}  lag {lag_now():>5}  members {lab.group_members(admin, GROUP)}")

workers = {"A": lab.start_script("module-03/billing_worker.py", "--name", "A", "--topic", TOPIC,
                                 "--group", GROUP, "--work-ms", "50", name="m03l01-A")}
watch(20)

You should see worker A own all six partitions and the lag grow by roughly 100 every 5 seconds.

Step 3: Three workers

Start B and C with the same group id. The group splits the partitions between the members.

for name in ["B", "C"]:
    workers[name] = lab.start_script("module-03/billing_worker.py", "--name", name, "--topic", TOPIC,
                                     "--group", GROUP, "--work-ms", "50", name=f"m03l01-{name}")
watch(25)

You should see two partitions per worker and, after a short rise during the rebalance, the lag falling: three workers handle about 60 events a second, more than the feed's 40.

Change it: seven workers, six partitions

Predict first: you start D, E, F and G. What does each of the seven workers get?

for name in ["D", "E", "F", "G"]:
    workers[name] = lab.start_script("module-03/billing_worker.py", "--name", name, "--topic", TOPIC,
                                     "--group", GROUP, "--work-ms", "50", name=f"m03l01-{name}")
watch(10)

Six workers get one partition each and one gets none. A partition is read by one member of a group at a time, so members beyond the partition count sit idle.

Break it: a worker crashes

Stop the extra workers, then kill worker B the hard way (kill -9: no clean-up, like a power cut). Predict first: how long do B's partitions go unread?

for name in ["D", "E", "F", "G"]:
    lab.stop_script(workers.pop(name))
time.sleep(5)
print("members before the crash:", lab.group_members(admin, GROUP))
lab.kill_script(workers.pop("B"))
crashed = time.monotonic()
for _ in range(14):
    time.sleep(5)
    members = lab.group_members(admin, GROUP)
    print(f"+{time.monotonic() - crashed:4.0f}s  lag {lag_now():>5}  members {members}")
    if "B" not in members and sum(len(p) for p in members.values()) == 6:
        break                                     # B is gone and its partitions have new owners

For about 45 seconds the group still lists B as the owner of its partitions, and the lag grows. Only when B's session times out (session.timeout.ms, 45 s by default) does the group give its partitions to A and C.

Clean up

Stop the workers and the feed, then delete the topic and the group. If anything gets stuck, docker compose down -v in the code/ folder resets the whole lab.

for w in workers.values():
    lab.stop_script(w)
lab.stop_script(feed)
feed.tail(1)
lab.delete_topics(admin, [TOPIC])
admin.delete_consumer_groups([GROUP])
print("cleaned up")