Reading events: consumers

Producers write events once; consumers decide what happens to them. Most production incidents on the reading side come down to four questions: who reads which partition, where a restarted reader continues, what a membership change costs, and what to do when the work isn't a stream at all but a pile of jobs. This module answers them with billing workers you start, crash and restart.

By the end of this module you can: - run several consumers as one group, read its assignment and lag, and size the group for the load; - choose a commit strategy, predict what a crash repeats or skips, and make processing idempotent; - explain what triggers a rebalance, and cut its cost with the KIP-848 protocol or static membership; - tell a stuck consumer from a dead one, and set max.poll.interval.ms and session.timeout.ms; - use a share group for independent jobs, with acknowledgements, locks and a delivery limit.

Lessons

  1. Consumer groups share the work
  2. Committing offsets: where processing resumes
  3. Rebalances and the new group protocol
  4. Work queues with share groups

Then practise with the exercises and check yourself with the module quiz.