Download 01-why-brightlane-needs-an-event-log.ipynb
Open it in JupyterLab or VS Code from the lab folder to run it. This page is a read-only view.
Why Brightlane needs an event log — practical
You write five charging-session events once, then let three independent readers (billing, the grid report and a team that starts next week) each read all of them, at their own pace.
Lab services: kafka-1, kafka-2, kafka-3 · Lesson: modules/01-first-events/01-why-brightlane-needs-an-event-log.md
# Setup: find the lab's shared helpers and check the lab is running import sys, pathlib lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / "common").is_dir()) sys.path.insert(0, str(lab_root)) from common import settings, lab from common.scenario import Brightlane, to_json, from_json lab.require_kafka() lab.print_versions()
Step 1: Create the topic
A topic is a named log. This one holds Brightlane's session events. One partition keeps the first lesson simple; lesson 1.2 explains partitions.
from confluent_kafka.admin import AdminClient, NewTopic TOPIC = "m01l01-sessions" admin = AdminClient({"bootstrap.servers": settings.KAFKA_BOOTSTRAP}) lab.delete_topics(admin, [TOPIC]) # start clean on reruns admin.delete_consumer_groups(["m01l01-billing", "m01l01-grid", "m01l01-app", "m01l01-latecomer"]) futures = admin.create_topics([NewTopic(TOPIC, num_partitions=1, replication_factor=3)]) futures[TOPIC].result() # raises if creation failed print(f"created {TOPIC}")
You should see created m01l01-sessions.
Step 2: The gateway writes five events
The charger gateway is the producer. It appends each event to the end of the log. It doesn't know who will read the events.
from confluent_kafka import Producer bl = Brightlane() events = bl.stream(sessions=5, readings=False)[:5] # the first five session events of the evening producer = Producer({"bootstrap.servers": settings.KAFKA_BOOTSTRAP}) for charger_id, event in events: producer.produce(TOPIC, key=charger_id, value=to_json(event)) print(f"{event['ts']} {event['type']:<16} {event['session_id']} {charger_id}") print("not yet stored:", producer.flush(10)) # wait until Kafka has stored all of them
You should see five session_started events between 17:30:07 and 17:30:46.
Step 3: Billing reads them
Billing is a consumer. It polls the log for events after its own position. group.id names
the reader: Kafka remembers each group's position separately.
auto.offset.reset=earliest tells a brand-new group to start at the beginning of the log.
import time from confluent_kafka import Consumer def read_available(consumer, max_wait=10.0, quiet=2.0): """Poll until no new event arrives for `quiet` seconds (or `max_wait` passes).""" got, deadline, last = [], time.monotonic() + max_wait, time.monotonic() while time.monotonic() < deadline and time.monotonic() - last < quiet: msg = consumer.poll(0.5) if msg is None or msg.error(): continue got.append(from_json(msg.value())) last = time.monotonic() return got def reader(group_id, start="earliest"): c = Consumer({"bootstrap.servers": settings.KAFKA_BOOTSTRAP, "group.id": group_id, "auto.offset.reset": start}) c.subscribe([TOPIC]) return c billing = reader("m01l01-billing") print("billing read:", [e["session_id"] for e in read_available(billing)])
You should see billing read all five session ids, oldest first.
Step 4: The grid report reads the same events
Reading didn't remove anything. A second reader with its own group.id gets all five too.
grid = reader("m01l01-grid") print("grid read: ", [e["session_id"] for e in read_available(grid)])
You should see the same five ids. Billing's read had no effect on what the grid report sees.
Step 5: Three more events arrive
Each reader continues from where it stopped. Nobody gets the first five again.
for charger_id, event in bl.stream(sessions=8, readings=False)[5:8]: producer.produce(TOPIC, key=charger_id, value=to_json(event)) producer.flush() print("billing read:", [e["session_id"] for e in read_available(billing)]) print("grid read: ", [e["session_id"] for e in read_available(grid)])
You should see both readers get the same three new ids, and only those.
Step 6: A team that starts next week
The driver-app team arrives later. The log still has every event, so they start from the beginning without asking anyone to resend data.
app = reader("m01l01-app") history = read_available(app) print(f"app read {len(history)} events, first {history[0]['ts']}, last {history[-1]['ts']}")
You should see app read 8 events. The log kept them; it didn't matter that two other readers
had already processed them.
Break it: a newcomer that starts at "latest"
Predict first: a new group with auto.offset.reset=latest subscribes now. How many of the 8
stored events will it read?
latecomer = reader("m01l01-latecomer", start="latest") print("latecomer read:", len(read_available(latecomer, max_wait=6)), "events")
It reads 0. A new group has no position yet, and latest places it at the end of the log:
it only sees events written from now on. This is the most common reason for "my consumer
shows nothing". The Python client's default is latest.
Clean up
Close the consumers (they leave their groups at once), then delete the topic and the groups.
If anything gets stuck, docker compose down -v in the code/ folder resets the whole lab.
for c in (billing, grid, app, latecomer): c.close() lab.delete_topics(admin, [TOPIC]) admin.delete_consumer_groups(["m01l01-billing", "m01l01-grid", "m01l01-app", "m01l01-latecomer"]) print("cleaned up")