Notebook · code/module-03/04-work-queues-with-share-groups.ipynb

Download 04-work-queues-with-share-groups.ipynb

Open it in JupyterLab or VS Code from the lab folder to run it. This page is a read-only view.

Work queues with share groups — practical

Forty fault tickets sit in a topic with a single partition. Four technicians in a share group work through them together. One technician crashes while holding a ticket, one works too slowly for the ticket's lock, and one ticket nobody can fix keeps bouncing.

Lab services: kafka-1, kafka-2, kafka-3 · Script: module-03/technician.py · Lesson: modules/03-consumers/04-work-queues-with-share-groups.md

# Setup: find the lab's shared helpers and check the lab is running
import sys, pathlib
lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / "common").is_dir())
sys.path.insert(0, str(lab_root))

import time
from collections import Counter
from common import settings, lab
from common.scenario import Brightlane, to_json

lab.require_kafka()
lab.print_versions()

Step 1: The tickets, and the share group's settings

The topic has 1 partition, so all four technicians share it. (With more partitions, the broker decides which technicians share which partition, and the run would differ every time.) Share groups are configured on the broker, per group: start at the earliest ticket (the default is latest), and lock each handed-out ticket for 15 seconds (default 30; 15 is the smallest the broker allows).

from confluent_kafka import Producer
from confluent_kafka.admin import AdminClient, ConfigResource, ConfigEntry, AlterConfigOpType

TOPIC, GROUP = "m03l04-tickets", "m03l04-technicians"
admin = AdminClient({"bootstrap.servers": settings.KAFKA_BOOTSTRAP})
lab.recreate_topic(admin, TOPIC, partitions=1)
admin.delete_consumer_groups([GROUP])
group_config = ConfigResource("group", GROUP, incremental_configs=[
    ConfigEntry("share.auto.offset.reset", "earliest", incremental_operation=AlterConfigOpType.SET),
    ConfigEntry("share.record.lock.duration.ms", "15000", incremental_operation=AlterConfigOpType.SET)])
admin.incremental_alter_configs([group_config])[group_config].result()

Tickets come in one at a time over the day, so each is written (and flushed) on its own. One extra ticket, tkt-09999, has an error code no technician knows.

tickets = Brightlane().fault_tickets(40)
tickets.append({"ticket_id": "tkt-09999", "charger_id": "BL-V02-01", "site_id": "site-V02",
                "error_code": "E-UNKNOWN", "priority": "high", "opened_at": tickets[-1]["opened_at"]})
producer = Producer(settings.PRODUCER_DEFAULTS)
for t in tickets:
    producer.produce(TOPIC, key=t["ticket_id"], value=to_json(t))
    producer.flush(10)
print(len(tickets), "tickets:", dict(Counter(t["error_code"] for t in tickets)))

Step 2: Four technicians, one share group

Each technician takes one ticket at a time. The script acknowledges each ticket explicitly: ACCEPT when fixed, RELEASE when it can't fix it. Nobody knows E-UNKNOWN. Ben also can't fix cable faults (E-CABLE). Cleo is slow: a ticket takes her 20 seconds. Dev is careful: 2 seconds. Ana is quick: 0.3 seconds.

def technician(name, work_ms, skip="E-UNKNOWN"):
    return lab.start_script("module-03/technician.py", "--name", name, "--topic", TOPIC, "--group", GROUP,
                            "--work-ms", str(work_ms), "--skip-codes", skip, name=f"m03l04-{name}")

techs = {"ana": technician("ana", 300), "ben": technician("ben", 300, "E-UNKNOWN,E-CABLE"),
         "cleo": technician("cleo", 20000), "dev": technician("dev", 2000)}
time.sleep(6)
for name, t in techs.items():
    print(f"{name:<5} has taken {len(t.lines(' GOT ')):>2} tickets, fixed {len(t.lines(' DONE ')):>2}")

You should see all four technicians with tickets from one partition: a consumer group would have kept three of them idle. Ana has fixed the most; Cleo is still on her first. The broker's view of the members shows each one assigned the same partition:

print(lab.kafka_cli("kafka-share-groups.sh", "--bootstrap-server", "localhost:19092",
                    "--describe", "--group", GROUP, "--members"))

Step 3: A technician's tablet dies

Predict first: Dev's process is killed while he holds a ticket. What happens to that ticket?

holding = techs["dev"].lines(" GOT ")[-1]
print("dev holds:", holding.split(" GOT ")[1])
lab.kill_script(techs["dev"])
held = holding.split(" GOT ")[1].split()[0]
lab.wait_until(lambda: any(f"GOT {held} " in l and "delivery=2" in l for t in techs.values() for l in t.lines()),
               timeout=40, what=f"{held} to be handed out again")
print([l for t in techs.values() for l in t.lines() if f"GOT {held} " in l and "delivery=2" in l])

You should see Dev's ticket go to another technician with delivery=2, within a second or two. The broker released it as soon as Dev's connection dropped.

Step 4: Let the others finish

Wait 30 seconds, stop everyone, and count per technician.

time.sleep(30)
for t in techs.values():
    lab.stop_script(t)
lab.table([(n, len(t.lines(" GOT ")), len(t.lines(" DONE ")), len(t.lines("RELEASED")))
           for n, t in techs.items()], ["technician", "tickets taken", "fixed", "released"])

Break it: the slow technician and the bouncing ticket

Look at every ticket that was handed out more than once.

for t in techs.values():
    for line in t.lines():
        if " GOT " in line and "delivery=1 " not in line:
            print(line)
print("Cleo:", [l for l in techs["cleo"].lines() if " GOT " in l or " DONE " in l])

Three things to find in this list: - Cleo's first ticket was handed to someone else after 15 seconds, the lock duration, while she was still working on it. It was fixed twice. - tkt-09999 went round with delivery=2, 3, 4, 5 and then stopped: after 5 deliveries (share.delivery.count.limit) the broker archives a record. Nobody will see it again. - Ben's cable tickets were given straight back to Ben, because his client kept fetching while he released them, until the delivery limit archived them unfixed. (The cable tickets Ana happened to take first, she fixed.)

Finally, the share group's state on the broker:

print(lab.kafka_cli("kafka-share-groups.sh", "--bootstrap-server", "localhost:19092",
                    "--describe", "--group", GROUP))

Clean up

Delete the topic and the share group. If anything gets stuck, docker compose down -v in the code/ folder resets the whole lab.

lab.delete_topics(admin, [TOPIC])
admin.delete_consumer_groups([GROUP])
print("cleaned up")