Stream processing in Python

Billing, pricing and the grid report all read events, compute something and write results. This module builds them as stream processors in plain Python: exactly once with Kafka transactions, with state that survives crashes, and with time windows that cope with late data. Doing it by hand shows what the frameworks do for you.

By the end of this module you can: - write a consume-transform-produce service that is exactly once with Kafka transactions; - explain transaction markers, read_committed, transactional ids and fencing; - keep a table and the service's own state locally, backed by compacted changelog topics, and restore it; - join a stream with a table, and explain co-partitioning; - compute event-time windows with a grace period, handle late data, and choose a framework.

Lessons

  1. Exactly-once billing with transactions
  2. State: turning a stream into a table
  3. Event time and windows

Then practise with the exercises and check yourself with the module quiz.