Writing events: producers

Every record in Kafka starts in a producer, and most data problems start there too: events in the wrong order, records lost in a broker restart, duplicates after a retry, or a client that floods the brokers with tiny requests. This module covers the three decisions every producer makes: where a record goes, when it counts as stored, and how records travel together.

By the end of this module you can: - choose a record key and a partition count, and explain what order Kafka does and doesn't keep; - configure a producer that neither loses nor duplicates data when brokers fail, and read its delivery reports; - explain acks, min.insync.replicas, the high watermark and the idempotent producer; - tune batching and compression, and measure throughput, latency and records per request.

Lessons

  1. Keys decide order
  2. Durable writes: acks, replicas and idempotence
  3. Batching, compression and throughput

Then practise with the exercises and check yourself with the module quiz.