Notebook · code/module-01/01-why-brightlane-needs-an-event-log.ipynb

Download 01-why-brightlane-needs-an-event-log.ipynb

Open it in JupyterLab or VS Code from the lab folder to run it. This page is a read-only view.

Why Brightlane needs an event log — practical

You write five charging-session events once, then let three independent readers (billing, the grid report and a team that starts next week) each read all of them, at their own pace.

Lab services: kafka-1, kafka-2, kafka-3 · Lesson: modules/01-first-events/01-why-brightlane-needs-an-event-log.md

# Setup: find the lab's shared helpers and check the lab is running
import sys, pathlib
lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / "common").is_dir())
sys.path.insert(0, str(lab_root))

from common import settings, lab
from common.scenario import Brightlane, to_json, from_json

lab.require_kafka()
lab.print_versions()

Step 1: Create the topic

A topic is a named log. This one holds Brightlane's session events. One partition keeps the first lesson simple; lesson 1.2 explains partitions.

from confluent_kafka.admin import AdminClient, NewTopic

TOPIC = "m01l01-sessions"
admin = AdminClient({"bootstrap.servers": settings.KAFKA_BOOTSTRAP})
lab.delete_topics(admin, [TOPIC])                                   # start clean on reruns
admin.delete_consumer_groups(["m01l01-billing", "m01l01-grid", "m01l01-app", "m01l01-latecomer"])

futures = admin.create_topics([NewTopic(TOPIC, num_partitions=1, replication_factor=3)])
futures[TOPIC].result()                                             # raises if creation failed
print(f"created {TOPIC}")

You should see created m01l01-sessions.

Step 2: The gateway writes five events

The charger gateway is the producer. It appends each event to the end of the log. It doesn't know who will read the events.

from confluent_kafka import Producer

bl = Brightlane()
events = bl.stream(sessions=5, readings=False)[:5]     # the first five session events of the evening

producer = Producer({"bootstrap.servers": settings.KAFKA_BOOTSTRAP})
for charger_id, event in events:
    producer.produce(TOPIC, key=charger_id, value=to_json(event))
    print(f"{event['ts']}  {event['type']:<16} {event['session_id']}  {charger_id}")
print("not yet stored:", producer.flush(10))           # wait until Kafka has stored all of them

You should see five session_started events between 17:30:07 and 17:30:46.

Step 3: Billing reads them

Billing is a consumer. It polls the log for events after its own position. group.id names the reader: Kafka remembers each group's position separately. auto.offset.reset=earliest tells a brand-new group to start at the beginning of the log.

import time
from confluent_kafka import Consumer

def read_available(consumer, max_wait=10.0, quiet=2.0):
    """Poll until no new event arrives for `quiet` seconds (or `max_wait` passes)."""
    got, deadline, last = [], time.monotonic() + max_wait, time.monotonic()
    while time.monotonic() < deadline and time.monotonic() - last < quiet:
        msg = consumer.poll(0.5)
        if msg is None or msg.error():
            continue
        got.append(from_json(msg.value()))
        last = time.monotonic()
    return got

def reader(group_id, start="earliest"):
    c = Consumer({"bootstrap.servers": settings.KAFKA_BOOTSTRAP, "group.id": group_id,
                  "auto.offset.reset": start})
    c.subscribe([TOPIC])
    return c

billing = reader("m01l01-billing")
print("billing read:", [e["session_id"] for e in read_available(billing)])

You should see billing read all five session ids, oldest first.

Step 4: The grid report reads the same events

Reading didn't remove anything. A second reader with its own group.id gets all five too.

grid = reader("m01l01-grid")
print("grid read:   ", [e["session_id"] for e in read_available(grid)])

You should see the same five ids. Billing's read had no effect on what the grid report sees.

Step 5: Three more events arrive

Each reader continues from where it stopped. Nobody gets the first five again.

for charger_id, event in bl.stream(sessions=8, readings=False)[5:8]:
    producer.produce(TOPIC, key=charger_id, value=to_json(event))
producer.flush()

print("billing read:", [e["session_id"] for e in read_available(billing)])
print("grid read:   ", [e["session_id"] for e in read_available(grid)])

You should see both readers get the same three new ids, and only those.

Step 6: A team that starts next week

The driver-app team arrives later. The log still has every event, so they start from the beginning without asking anyone to resend data.

app = reader("m01l01-app")
history = read_available(app)
print(f"app read {len(history)} events, first {history[0]['ts']}, last {history[-1]['ts']}")

You should see app read 8 events. The log kept them; it didn't matter that two other readers had already processed them.

Break it: a newcomer that starts at "latest"

Predict first: a new group with auto.offset.reset=latest subscribes now. How many of the 8 stored events will it read?

latecomer = reader("m01l01-latecomer", start="latest")
print("latecomer read:", len(read_available(latecomer, max_wait=6)), "events")

It reads 0. A new group has no position yet, and latest places it at the end of the log: it only sees events written from now on. This is the most common reason for "my consumer shows nothing". The Python client's default is latest.

Clean up

Close the consumers (they leave their groups at once), then delete the topic and the groups. If anything gets stuck, docker compose down -v in the code/ folder resets the whole lab.

for c in (billing, grid, app, latecomer):
    c.close()
lab.delete_topics(admin, [TOPIC])
admin.delete_consumer_groups(["m01l01-billing", "m01l01-grid", "m01l01-app", "m01l01-latecomer"])
print("cleaned up")