27 min · theory + hands-on practical

Why Brightlane needs an event log

This is the first lesson of the course. It answers one question: what problem does Kafka solve that a database doesn't? You'll answer it with Brightlane Charging, the company you work for throughout the course, and then watch it happen in the lab.

Theory

Example: the database that everyone asks

Brightlane Charging runs 240 public chargers for electric cars at 36 sites. Every charger talks to Brightlane's charger gateway, a service that stores each charging session in a PostgreSQL database.

Three teams need that data:

On a weekday evening about 160 sessions run at once. At 17:45 the gateway database sits at 90% CPU, and most of that work is answering the same questions again and again. Last Tuesday billing missed three sessions: they ended in the same second as billing's previous check, and the "since my last check" query skipped them.

Then a fourth team arrives. Fleet sales wants every session from last week to build an offer for a taxi company. But the database keeps only each session's latest state. Nobody can say exactly what happened last week, in what order.

An event log

The fix is to stop asking the database "what is the state now?" and start recording what happened.

An event is a record of something that happened, written in the past tense and never changed afterwards. "Session ses-7f3a19 started on charger BL-N03-04 at 17:31" is an event. So is "it stopped at 18:09 after 31.4 kWh".

An event log is an ordered list of events. New events are only ever added at the end. Old events are kept for a set time, called retention (7 days by default in Kafka), and nobody edits them.

Apache Kafka is a system that stores event logs on several servers and lets many programs write and read them at the same time. In Kafka:

Top: billing, the grid report, the app and a new team all poll the gateway database, which runs at 90% CPU. Bottom: the gateway appends events to one log; billing, the grid report and a new team each read it from their own position.

What to notice: in the lower half the database is gone from the picture, and each reader points at a different place in the same log.

The gateway now does one thing: it appends each event to the topic, once. It doesn't know or care who reads it. Billing can't miss an event because it doesn't search by time: it reads the next event after the last one it handled.

Many readers, one copy

Reading an event from a Kafka topic doesn't remove it. Each reader keeps its own position: a number that says how far into the log it has read. Kafka calls these numbers offsets: the first event in a log is offset 0, the next is 1, and so on. Lesson 1.3 looks at offsets closely.

Because positions are per reader, three things become easy:

Kafka groups readers that work together under a shared name, the consumer group. Kafka remembers the position of each group. Module 3 covers groups in depth; for now, one group means one independent reader.

This is the difference from a queue, such as RabbitMQ or Amazon SQS. A queue hands each message to one receiver and deletes it once it has been handled. That's ideal for sharing out work, but a second team can't read the same messages later. (Kafka 4.2 added share groups, which give Kafka queue-like reading too. Lesson 3.4 shows them.)

When Kafka fits, and when it doesn't

Kafka is worth its cost when at least two of these are true:

Brightlane ticks all three. Kafka is the wrong tool when:

In production

Running Kafka means at least three servers, monitoring, upgrades and people who understand it. Managed services (Amazon MSK, Confluent Cloud, Aiven, Google Cloud's managed Kafka and others) take the servers off your hands but not the design decisions this course teaches. Lesson 9.3 compares the options.

Practical

Note

Notebook: code/module-01/01-why-brightlane-needs-an-event-log.ipynb · Lab services: kafka-1, kafka-2, kafka-3 · Time: ~17 min (plus ~15 min for the one-time lab setup)

What you'll do: write five session events once, then let billing, the grid report and a team that "arrives next week" each read them independently. Then you'll see the most common reason a new consumer shows nothing.

The lab

The lab runs Kafka in Docker on your Mac. If you haven't set it up yet, follow the lab setup once: it installs Python packages and explains the Docker settings. Then start the lab from the code/ folder:

docker compose up -d --wait

Output (the last lines; it takes about 20 seconds the first time):

 ✔ Container kafka-hands-on-lab-kafka-1-1   Healthy
 ✔ Container kafka-hands-on-lab-kafka-2-1   Healthy
 ✔ Container kafka-hands-on-lab-kafka-3-1   Healthy
 ✔ Container kafka-hands-on-lab-kafka-ui-1  Healthy

Your Mac runs the notebooks, scripts and a browser. Docker runs three Kafka nodes, kafka-1 to kafka-3, and the Kafbat UI. Python reaches the brokers on 127.0.0.1 ports 9092 to 9094 and the browser reaches the UI on port 8080.

What to notice: your Python code runs on the Mac, outside Docker, and reaches each Kafka node on its own port: 9092, 9093 and 9094.

Open the notebook in JupyterLab (jupyter lab from the code/ folder) and run the setup cell. It checks the lab and prints the versions:

Python           3.13.5
confluent-kafka  2.15.1  (librdkafka 2.15.1)
Kafka (brokers)  4.3.1

Step 1: Create the topic

A topic must exist before anyone writes to it. The lab is configured like production here: it doesn't create topics automatically when a client mentions a new name.

code/module-01/01-why-brightlane-needs-an-event-log.ipynb
TOPIC = "m01l01-sessions"
admin = AdminClient({"bootstrap.servers": settings.KAFKA_BOOTSTRAP})
# ...
futures = admin.create_topics([NewTopic(TOPIC, num_partitions=1, replication_factor=3)])
futures[TOPIC].result()                                             # raises if creation failed

AdminClient is the client for managing topics. settings.KAFKA_BOOTSTRAP holds the brokers' addresses (lesson 1.2 explains them). Ignore num_partitions and replication_factor for now.

created m01l01-sessions

Step 2: The gateway writes five events

code/module-01/01-why-brightlane-needs-an-event-log.ipynb
producer = Producer({"bootstrap.servers": settings.KAFKA_BOOTSTRAP})
for charger_id, event in events:
    producer.produce(TOPIC, key=charger_id, value=to_json(event))
    print(f"{event['ts']}  {event['type']:<16} {event['session_id']}  {charger_id}")
print("not yet stored:", producer.flush(10))           # wait until Kafka has stored all of them

produce() only queues the event inside your program; the client sends queued events in the background. flush() waits until Kafka has stored them all and returns how many are still waiting.

Output (your dates show yesterday's date: the generator always replays yesterday evening's peak):

2026-10-05T17:30:07Z  session_started  ses-a5db25  BL-N02-03
2026-10-05T17:30:22Z  session_started  ses-53ca2d  BL-N14-02
2026-10-05T17:30:30Z  session_started  ses-fa7469  BL-C04-01
2026-10-05T17:30:30Z  session_started  ses-9f1ca9  BL-N13-06
2026-10-05T17:30:46Z  session_started  ses-b9d6df  BL-C01-04
not yet stored: 0

Step 3: Billing reads them

The notebook defines two small helpers. reader() creates a consumer in its own group:

code/module-01/01-why-brightlane-needs-an-event-log.ipynb
def reader(group_id, start="earliest"):
    c = Consumer({"bootstrap.servers": settings.KAFKA_BOOTSTRAP, "group.id": group_id,
                  "auto.offset.reset": start})
    c.subscribe([TOPIC])
    return c

group.id names the reader. auto.offset.reset says where a group with no stored position starts: earliest means at offset 0. read_available() calls consumer.poll() in a loop and stops when no new event has arrived for 2 seconds.

code/module-01/01-why-brightlane-needs-an-event-log.ipynb
billing = reader("m01l01-billing")
print("billing read:", [e["session_id"] for e in read_available(billing)])
billing read: ['ses-a5db25', 'ses-53ca2d', 'ses-fa7469', 'ses-9f1ca9', 'ses-b9d6df']

Step 4: The grid report reads the same events

code/module-01/01-why-brightlane-needs-an-event-log.ipynb
grid = reader("m01l01-grid")
print("grid read:   ", [e["session_id"] for e in read_available(grid)])
grid read:    ['ses-a5db25', 'ses-53ca2d', 'ses-fa7469', 'ses-9f1ca9', 'ses-b9d6df']

Billing's read removed nothing.

Step 5: Three more events arrive

The gateway writes three more events. Both readers poll again.

billing read: ['ses-13ad04', 'ses-a2481d', 'ses-79f44e']
grid read:    ['ses-13ad04', 'ses-a2481d', 'ses-79f44e']

Each reader continued from its own position and got only the new events.

Step 6: A team that starts next week

code/module-01/01-why-brightlane-needs-an-event-log.ipynb
app = reader("m01l01-app")
history = read_available(app)
print(f"app read {len(history)} events, first {history[0]['ts']}, last {history[-1]['ts']}")
app read 8 events, first 2026-10-05T17:30:07Z, last 2026-10-05T17:30:59Z

The newcomer read the whole history without anyone resending it.

Break it: a newcomer that starts at "latest"

Predict first: a new group with auto.offset.reset=latest subscribes now. How many of the 8 stored events will it read?

code/module-01/01-why-brightlane-needs-an-event-log.ipynb
latecomer = reader("m01l01-latecomer", start="latest")
print("latecomer read:", len(read_available(latecomer, max_wait=6)), "events")
latecomer read: 0 events

What just happened

The gateway wrote each event once, to the end of the topic: that's the event log. Billing, the grid report and the app each kept their own position, so reading never removed anything and nobody slowed anybody else down. That's "many readers, one copy". The app's group started at offset 0 and replayed the full history, which is what fleet sales needed. The latecomer read nothing because a new group with latest starts at the end of the log and waits for new events. The Python client's default is latest, and it's the most common reason a new consumer "shows nothing".

Clean up

The last cell closes the consumers, deletes the topic and deletes the four groups. If anything gets stuck, run docker compose down -v in the code/ folder to reset the whole lab, then docker compose up -d --wait again.

Key takeaways

Next: Brokers, controllers and partitions