State: turning a stream into a table
Pricing a session needs two things the session's own events don't carry: the site's current tariff, and who the driver was. A stream processor keeps such things as local state. This lesson builds that state from Kafka topics, keeps it safe in Kafka, and rebuilds it after a crash.
Theory
Example: one database query per session
Brightlane's first pricing service looked up the tariff in PostgreSQL for every finished session. At the evening peak that was fine. During the growth-plan load test (25 times today's traffic), the pricing database spent most of its time answering the same 36 questions: "what's the tariff at site-N03?". The pricing team asked billing to stop. The tariffs change a few times a month, and billing asked for them thousands of times a minute.
The tariffs are already in Kafka, as a CDC topic (lesson 6.1). The service can keep its own copy.
Streams and tables
A table holds the current value for each key; a stream is a sequence of events. They are two views of the same data:
- Apply a stream of changes in order, keeping the latest value per key, and you get a table. That's what the live map did with charger statuses in lesson 4.3.
- Record every change of a table, and you get a stream: its changelog. That's what Debezium produced from the tariffs table in lesson 6.1.
A compacted topic is both at once: the changelog of a table, kept small enough to rebuild the table from it.
Local state, backed by a changelog
A stream processor keeps two kinds of state:
- Tables it reads, such as the tariffs, built from a compacted topic at start-up and kept up to date from it. They need no backup: the topic is the source.
- State it creates: here the open sessions (who started which session at which site) and each driver's running total. If the process dies, this state is gone, unless it was saved.
The service keeps its own state in a state store, in memory or on local disk, and writes every change to a compacted changelog topic as well. Kafka Streams does exactly this, with RocksDB as the store.
What to notice: the state changes go into the same transaction as the output and the offsets, so the restored state always matches the committed position.
After a clean stop, the service saves a checkpoint: its state, and how far the changelogs went. It then restarts from the checkpoint and reads only newer changelog records. After a crash there's no checkpoint, and it rebuilds the state from the changelogs. Compaction keeps that fast: one record per key, not every change ever made.
Joining a stream with a table
Pricing a session is an enrichment join: each event looks up the current value in a table, here the tariff for its site. Two ways to make the table available:
- The whole table in every instance (a "global table"): each instance reads all partitions of the tariffs topic. Simple, and fine for small tables like 36 tariffs.
- A partitioned table: each instance keeps only the keys of its own partitions. That needs co-partitioning: the stream and the table keyed by the same thing, with the same number of partitions and the same partitioner, so key X is in partition p of both.
Co-partitioning also explains why this service can keep open sessions at all. A session's start and stop events are both keyed by charger id, so they're in the same partition, and the same instance sees both. If they were keyed differently, the service would first have to repartition: write the events again, keyed by session id, to an intermediate topic.
In production
- Choose keys for the joins and state you'll need, not only for ordering (lesson 2.1). Changing keys later means repartitioning.
- Back every piece of state you create with a compacted changelog topic, written in the same transaction as your output (lesson 7.1).
- Restoring a large store takes time. Kafka Streams keeps standby replicas (warm copies of the state on another instance) and checkpoints to make restarts quick.
- In Python there is no Kafka Streams. Quix Streams offers state stores with changelogs and windows as a library; Flink (PyFlink) as a cluster (lesson 7.3).
Practical
Note
Script: code/module-07/pricing_service.py (the notebook starts, stops and crashes it)
· Notebook: code/module-07/02-state-stream-to-table.ipynb
· Lab services: kafka-1..3, schema-registry, postgres, connect · Time: ~14 min
What you'll do: run the pricing service on a CDC tariffs table, change a price while it runs, restart it cleanly and after a crash, check its state against its output, and then run it without a changelog.
This lesson needs the lab's connect profile (lesson 6.1):
docker compose --profile connect up -d --wait
Step 1: The tariffs table, from CDC
The notebook registers lesson 6.1's flattening connector for this lesson:
tariffs connector: RUNNING
tariffs in the table topic: 36
Step 2: Start the service on the first sessions
The service loads the whole tariffs table, keyed by site, then follows its changes:
read_to_end(args.tariffs, apply_tariff)
On every session_started it remembers the session; on every session_stopped it prices it:
tariff = tariffs[session["site_id"]]
amount = round(event["energy_kwh"] * float(tariff["price_per_kwh"]), 2)
Each state change goes to the store and to its changelog topic:
if not args.no_changelog:
producer.produce(changelog[store], key=key, value=None if value is None else to_json(value))
20:24:55.317 tariffs table loaded: 36 sites
20:24:55.388 state restored in 0.07 s (changelogs only, no checkpoint, 0 changelog records read): 0 open sessions, 0 driver totals
Step 3: A price change while it runs
20:24:58.819 tariff changed: site-N03 now 0.6500 EUR/kWh
Step 4: A clean restart, then a crash
20:25:03.865 state restored in 0.13 s (checkpoint + changelog records written since, 0 changelog records read): 98 open sessions, 225 driver totals
20:25:05.846 CRASH after 20 priced sessions
20:25:07.847 state restored in 1.05 s (changelogs only, no checkpoint, 776 changelog records read): 98 open sessions, 225 driver totals
20:25:15.408 stopped: priced=174 unknown=0
Step 5: Is the state right?
charges: 400 for 400 sessions (400 finished)
driver totals: 399, different from their charges: 0, open sessions left: 0
prices charged at site-N03: [0.51, 0.65]
Break it: no changelog
The same service with --no-changelog keeps its state in memory only. It crashes after 150 priced
sessions and starts again. Predict first: what happens to sessions that started before the crash and
finish after it?
20:25:21.827 state restored in 0.00 s (no changelog: starting with empty state): 0 open sessions, 0 driver totals
20:25:28.030 stopped: priced=231 unknown=38
charges: 362 (400 sessions finished)
What just happened
The service built the tariffs table from the compacted CDC topic once, then applied its changes as they came: the price change at site-N03 reached it within a second, and sessions finishing after it were charged 0.65 instead of 0.51, without a single database query. Its own state lived in memory, with every change also written to a changelog in the same transaction as the charges and offsets. The clean restart loaded the checkpoint and read nothing more. After the crash, the service read 776 changelog records in about a second and had exactly the state of the last committed transaction: 98 open sessions and 225 driver totals. The crashed transaction, with its 20 charges, was aborted and done again. At the end, every driver total matched the sum of that driver's charges. Without a changelog, the restarted service had forgotten which driver started which session. For 38 sessions it saw a stop without a start, couldn't price them, and those drivers were never charged.
Clean up
The last cell deletes the connector and its slot, the topics, groups, checkpoint files and schemas,
and resets the database. If anything gets stuck, docker compose down -v in the code/ folder
resets the whole lab.
Key takeaways
- A table is a stream of changes reduced to the latest value per key; a compacted topic holds both views.
- Build lookup tables from their compacted topics instead of querying their database per event.
- Back the state you create with a compacted changelog topic, written in the same transaction as output and offsets.
- A checkpoint makes clean restarts instant; after a crash, state is rebuilt from the changelogs.
- Keyed state and partitioned joins need co-partitioning: the same key, partition count and partitioner.
Next: Event time and windows