30 min · theory + hands-on practical

State: turning a stream into a table

Pricing a session needs two things the session's own events don't carry: the site's current tariff, and who the driver was. A stream processor keeps such things as local state. This lesson builds that state from Kafka topics, keeps it safe in Kafka, and rebuilds it after a crash.

Theory

Example: one database query per session

Brightlane's first pricing service looked up the tariff in PostgreSQL for every finished session. At the evening peak that was fine. During the growth-plan load test (25 times today's traffic), the pricing database spent most of its time answering the same 36 questions: "what's the tariff at site-N03?". The pricing team asked billing to stop. The tariffs change a few times a month, and billing asked for them thousands of times a minute.

The tariffs are already in Kafka, as a CDC topic (lesson 6.1). The service can keep its own copy.

Streams and tables

A table holds the current value for each key; a stream is a sequence of events. They are two views of the same data:

A compacted topic is both at once: the changelog of a table, kept small enough to rebuild the table from it.

Local state, backed by a changelog

A stream processor keeps two kinds of state:

The service keeps its own state in a state store, in memory or on local disk, and writes every change to a compacted changelog topic as well. Kafka Streams does exactly this, with RocksDB as the store.

The pricing service reads session events and the tariffs table topic. It keeps the tariffs by site, open sessions and driver totals in memory. In one transaction per batch it writes the charges, every state change to two compacted changelog topics, and the consumer offsets. On a clean stop it writes a checkpoint file; after a crash it restores its state by reading the changelogs from the start.

What to notice: the state changes go into the same transaction as the output and the offsets, so the restored state always matches the committed position.

After a clean stop, the service saves a checkpoint: its state, and how far the changelogs went. It then restarts from the checkpoint and reads only newer changelog records. After a crash there's no checkpoint, and it rebuilds the state from the changelogs. Compaction keeps that fast: one record per key, not every change ever made.

Joining a stream with a table

Pricing a session is an enrichment join: each event looks up the current value in a table, here the tariff for its site. Two ways to make the table available:

Co-partitioning also explains why this service can keep open sessions at all. A session's start and stop events are both keyed by charger id, so they're in the same partition, and the same instance sees both. If they were keyed differently, the service would first have to repartition: write the events again, keyed by session id, to an intermediate topic.

In production

Practical

Note

Script: code/module-07/pricing_service.py (the notebook starts, stops and crashes it) · Notebook: code/module-07/02-state-stream-to-table.ipynb · Lab services: kafka-1..3, schema-registry, postgres, connect · Time: ~14 min

What you'll do: run the pricing service on a CDC tariffs table, change a price while it runs, restart it cleanly and after a crash, check its state against its output, and then run it without a changelog.

This lesson needs the lab's connect profile (lesson 6.1):

docker compose --profile connect up -d --wait

Step 1: The tariffs table, from CDC

The notebook registers lesson 6.1's flattening connector for this lesson:

tariffs connector: RUNNING
tariffs in the table topic: 36

Step 2: Start the service on the first sessions

The service loads the whole tariffs table, keyed by site, then follows its changes:

code/module-07/pricing_service.py
    read_to_end(args.tariffs, apply_tariff)

On every session_started it remembers the session; on every session_stopped it prices it:

code/module-07/pricing_service.py
            tariff = tariffs[session["site_id"]]
            amount = round(event["energy_kwh"] * float(tariff["price_per_kwh"]), 2)

Each state change goes to the store and to its changelog topic:

code/module-07/pricing_service.py
        if not args.no_changelog:
            producer.produce(changelog[store], key=key, value=None if value is None else to_json(value))
20:24:55.317 tariffs table loaded: 36 sites
20:24:55.388 state restored in 0.07 s (changelogs only, no checkpoint, 0 changelog records read): 0 open sessions, 0 driver totals

Step 3: A price change while it runs

20:24:58.819 tariff changed: site-N03 now 0.6500 EUR/kWh

Step 4: A clean restart, then a crash

20:25:03.865 state restored in 0.13 s (checkpoint + changelog records written since, 0 changelog records read): 98 open sessions, 225 driver totals
20:25:05.846 CRASH after 20 priced sessions
20:25:07.847 state restored in 1.05 s (changelogs only, no checkpoint, 776 changelog records read): 98 open sessions, 225 driver totals
20:25:15.408 stopped: priced=174 unknown=0

Step 5: Is the state right?

charges: 400 for 400 sessions (400 finished)
driver totals: 399, different from their charges: 0, open sessions left: 0
prices charged at site-N03: [0.51, 0.65]

Break it: no changelog

The same service with --no-changelog keeps its state in memory only. It crashes after 150 priced sessions and starts again. Predict first: what happens to sessions that started before the crash and finish after it?

20:25:21.827 state restored in 0.00 s (no changelog: starting with empty state): 0 open sessions, 0 driver totals
20:25:28.030 stopped: priced=231 unknown=38
charges: 362 (400 sessions finished)

What just happened

The service built the tariffs table from the compacted CDC topic once, then applied its changes as they came: the price change at site-N03 reached it within a second, and sessions finishing after it were charged 0.65 instead of 0.51, without a single database query. Its own state lived in memory, with every change also written to a changelog in the same transaction as the charges and offsets. The clean restart loaded the checkpoint and read nothing more. After the crash, the service read 776 changelog records in about a second and had exactly the state of the last committed transaction: 98 open sessions and 225 driver totals. The crashed transaction, with its 20 charges, was aborted and done again. At the end, every driver total matched the sum of that driver's charges. Without a changelog, the restarted service had forgotten which driver started which session. For 38 sessions it saw a stop without a start, couldn't price them, and those drivers were never charged.

Clean up

The last cell deletes the connector and its slot, the topics, groups, checkpoint files and schemas, and resets the database. If anything gets stuck, docker compose down -v in the code/ folder resets the whole lab.

Key takeaways

Next: Event time and windows