Download 04-work-queues-with-share-groups.ipynb
Open it in JupyterLab or VS Code from the lab folder to run it. This page is a read-only view.
Work queues with share groups — practical
Forty fault tickets sit in a topic with a single partition. Four technicians in a share group work through them together. One technician crashes while holding a ticket, one works too slowly for the ticket's lock, and one ticket nobody can fix keeps bouncing.
Lab services: kafka-1, kafka-2, kafka-3 · Script: module-03/technician.py
· Lesson: modules/03-consumers/04-work-queues-with-share-groups.md
# Setup: find the lab's shared helpers and check the lab is running import sys, pathlib lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / "common").is_dir()) sys.path.insert(0, str(lab_root)) import time from collections import Counter from common import settings, lab from common.scenario import Brightlane, to_json lab.require_kafka() lab.print_versions()
Step 1: The tickets, and the share group's settings
The topic has 1 partition, so all four technicians share it. (With more partitions, the broker
decides which technicians share which partition, and the run would differ every time.) Share
groups are configured on the broker, per group: start at the earliest ticket (the default is
latest), and lock each handed-out ticket for 15 seconds (default 30; 15 is the smallest the
broker allows).
from confluent_kafka import Producer from confluent_kafka.admin import AdminClient, ConfigResource, ConfigEntry, AlterConfigOpType TOPIC, GROUP = "m03l04-tickets", "m03l04-technicians" admin = AdminClient({"bootstrap.servers": settings.KAFKA_BOOTSTRAP}) lab.recreate_topic(admin, TOPIC, partitions=1) admin.delete_consumer_groups([GROUP]) group_config = ConfigResource("group", GROUP, incremental_configs=[ ConfigEntry("share.auto.offset.reset", "earliest", incremental_operation=AlterConfigOpType.SET), ConfigEntry("share.record.lock.duration.ms", "15000", incremental_operation=AlterConfigOpType.SET)]) admin.incremental_alter_configs([group_config])[group_config].result()
Tickets come in one at a time over the day, so each is written (and flushed) on its own. One
extra ticket, tkt-09999, has an error code no technician knows.
tickets = Brightlane().fault_tickets(40) tickets.append({"ticket_id": "tkt-09999", "charger_id": "BL-V02-01", "site_id": "site-V02", "error_code": "E-UNKNOWN", "priority": "high", "opened_at": tickets[-1]["opened_at"]}) producer = Producer(settings.PRODUCER_DEFAULTS) for t in tickets: producer.produce(TOPIC, key=t["ticket_id"], value=to_json(t)) producer.flush(10) print(len(tickets), "tickets:", dict(Counter(t["error_code"] for t in tickets)))
Step 2: Four technicians, one share group
Each technician takes one ticket at a time. The script acknowledges each ticket explicitly:
ACCEPT when fixed, RELEASE when it can't fix it. Nobody knows E-UNKNOWN. Ben also can't fix
cable faults (E-CABLE). Cleo is slow: a ticket takes her 20 seconds. Dev is careful: 2 seconds.
Ana is quick: 0.3 seconds.
def technician(name, work_ms, skip="E-UNKNOWN"): return lab.start_script("module-03/technician.py", "--name", name, "--topic", TOPIC, "--group", GROUP, "--work-ms", str(work_ms), "--skip-codes", skip, name=f"m03l04-{name}") techs = {"ana": technician("ana", 300), "ben": technician("ben", 300, "E-UNKNOWN,E-CABLE"), "cleo": technician("cleo", 20000), "dev": technician("dev", 2000)} time.sleep(6) for name, t in techs.items(): print(f"{name:<5} has taken {len(t.lines(' GOT ')):>2} tickets, fixed {len(t.lines(' DONE ')):>2}")
You should see all four technicians with tickets from one partition: a consumer group would have kept three of them idle. Ana has fixed the most; Cleo is still on her first. The broker's view of the members shows each one assigned the same partition:
print(lab.kafka_cli("kafka-share-groups.sh", "--bootstrap-server", "localhost:19092", "--describe", "--group", GROUP, "--members"))
Step 3: A technician's tablet dies
Predict first: Dev's process is killed while he holds a ticket. What happens to that ticket?
holding = techs["dev"].lines(" GOT ")[-1] print("dev holds:", holding.split(" GOT ")[1]) lab.kill_script(techs["dev"]) held = holding.split(" GOT ")[1].split()[0] lab.wait_until(lambda: any(f"GOT {held} " in l and "delivery=2" in l for t in techs.values() for l in t.lines()), timeout=40, what=f"{held} to be handed out again") print([l for t in techs.values() for l in t.lines() if f"GOT {held} " in l and "delivery=2" in l])
You should see Dev's ticket go to another technician with delivery=2, within a second or two.
The broker released it as soon as Dev's connection dropped.
Step 4: Let the others finish
Wait 30 seconds, stop everyone, and count per technician.
time.sleep(30) for t in techs.values(): lab.stop_script(t) lab.table([(n, len(t.lines(" GOT ")), len(t.lines(" DONE ")), len(t.lines("RELEASED"))) for n, t in techs.items()], ["technician", "tickets taken", "fixed", "released"])
Break it: the slow technician and the bouncing ticket
Look at every ticket that was handed out more than once.
for t in techs.values(): for line in t.lines(): if " GOT " in line and "delivery=1 " not in line: print(line) print("Cleo:", [l for l in techs["cleo"].lines() if " GOT " in l or " DONE " in l])
Three things to find in this list:
- Cleo's first ticket was handed to someone else after 15 seconds, the lock duration, while she
was still working on it. It was fixed twice.
- tkt-09999 went round with delivery=2, 3, 4, 5 and then stopped: after 5 deliveries
(share.delivery.count.limit) the broker archives a record. Nobody will see it again.
- Ben's cable tickets were given straight back to Ben, because his client kept fetching while he
released them, until the delivery limit archived them unfixed. (The cable tickets Ana happened
to take first, she fixed.)
Finally, the share group's state on the broker:
print(lab.kafka_cli("kafka-share-groups.sh", "--bootstrap-server", "localhost:19092", "--describe", "--group", GROUP))
Clean up
Delete the topic and the share group. If anything gets stuck, docker compose down -v in the
code/ folder resets the whole lab.
lab.delete_topics(admin, [TOPIC]) admin.delete_consumer_groups([GROUP]) print("cleaned up")