33 min · theory + hands-on practical

Durable writes: acks, replicas and idempotence

A producer's job ends when Kafka has the record safely. This lesson is about what "safely" means: how many copies, who confirms them, and what happens when a broker disappears halfway. Get these settings wrong and you either lose data or store it twice, and nothing warns you.

Theory

Example: the restart at 18:05

On Thursday at 18:05, during the evening peak, the platform team restarted the brokers one by one to apply a patch. The next morning two problems turned up:

Both problems come from three settings: acks, min.insync.replicas and enable.idempotence.

acks: when the producer hears "stored"

acks is a producer setting. It decides when the partition's leader answers a produce request:

The producer sends a batch to the leader. With acks=0 it hears nothing. With acks=1 the leader answers once it has written the batch. With acks=all the leader answers only after the in-sync followers have copied it.

What to notice: acks=all waits for the followers' next fetch, which is a few milliseconds on a healthy cluster. That wait is what buys safety.

In-sync replicas and min.insync.replicas

Lesson 1.2 introduced the ISR, the replicas that are fully caught up. A follower that stops fetching for 30 seconds (replica.lag.time.max.ms) is dropped from the ISR.

acks=all waits for every replica currently in the ISR. If two brokers are down, the ISR may be just the leader, and then acks=all is no safer than acks=1. The topic setting min.insync.replicas closes that gap. It says how many in-sync replicas must exist before the leader accepts an acks=all write. With fewer, the leader refuses the write with NOT_ENOUGH_REPLICAS.

With 3 replicas, min.insync.replicas=2 is the standard choice: the topic keeps accepting writes with one broker down, and every acknowledged record exists on at least two brokers. Setting it to 3 means any single broker restart stops all writes.

There's a second effect, on the reading side. Consumers can only read up to the partition's high watermark: the offset up to which all in-sync replicas have the records. On clusters created with Kafka 4.1 or later (where the "eligible leader replicas" feature is on by default), the high watermark also stops moving while the ISR is smaller than min.insync.replicas. So records written with acks=1 during that time are stored, but nobody can read them yet.

A partition with three replicas; kafka-1 is down. With min.insync.replicas=2, acks=all writes succeed. With 3, acks=all writes are refused and acks=1 writes are stored but stay above the high watermark, unreadable.

What to notice: records 201 to 220 sit in the leader's log, but the high watermark stays at 201 until the ISR is big enough again.

Retries, timeouts and delivery reports

When a write fails for a temporary reason (a leader moving, too few in-sync replicas, a network blip), the client retries it on its own, again and again. It gives up only when the record has waited longer than the delivery timeout: message.timeout.ms in the Python client (5 minutes by default), delivery.timeout.ms in Java. Then it reports _MSG_TIMED_OUT, which hides the broker's real reason.

The only place your code learns the outcome is the delivery report: a callback you pass to produce(), called from poll() or flush(). It gets an error, or the partition and offset where the record was stored. Some errors are final at once, such as a record larger than the topic allows (MSG_SIZE_TOO_LARGE).

Don't add your own retry loop around the same record. The client already retries, and resending from your code creates duplicates and breaks order.

The idempotent producer

Retries create the duplicate problem: the leader stored a batch, the acknowledgement was lost, and the producer sent the batch again. An idempotent producer fixes it:

  1. When it starts, the producer gets a producer id from the cluster.
  2. It numbers every batch it sends to a partition: sequence 0, 1, 2…
  3. The leader remembers the last sequence numbers per producer id. A batch it has already stored is acknowledged again but not stored twice. A batch that arrives too early (out of order) is refused and retried.

The producer, id 4, sends batch sequence 7. The leader stores it, but the acknowledgement is lost. The producer retries the same batch with the same sequence number. The leader keeps one copy and acknowledges again.

What to notice: the retry carries the same producer id and sequence number, so the leader can tell it's a repeat.

Idempotence also keeps order during retries with up to 5 requests in flight. It needs acks=all. Java clients have it on by default since Kafka 3.0, but the Python client doesn't: enable.idempotence defaults to false in librdkafka. Turn it on.

It doesn't protect against everything. A producer that restarts gets a new producer id, and if your code calls produce() twice for the same reading, Kafka sees two different records. Module 3 deals with duplicates on the consumer side, and lesson 7.1 with transactions.

In production

Practical

Note

Notebook: code/module-02/02-durable-writes.ipynb · Lab services: kafka-1, kafka-2, kafka-3 · Time: ~17 min

What you'll do: write meter readings with delivery reports, then stop a broker and see what acks=all, min.insync.replicas, acks=1 and acks=0 each do.

Step 1: Write with acks=all and an idempotent producer

write(records, **config) sends records with the given settings and counts the outcomes from the delivery reports. It sets message.timeout.ms to 10 seconds, so the notebook doesn't wait 5 minutes for failures.

code/module-02/02-durable-writes.ipynb
    def report(err, msg):
        outcomes[err.name() if err else "delivered"] += 1
        if not err and msg.offset() is not None:
            offsets.append(msg.offset())
code/module-02/02-durable-writes.ipynb
safe = {"acks": "all", "enable.idempotence": True}
print(write(readings[:100], **safe))
({'delivered': 100}, (0, 99))

The client's statistics show the producer id the cluster assigned (the number varies):

idempotence state: Assigned | producer id: 4 | epoch: 0

On a freshly started lab you may also see one GETPID … Coordinator load in progress: retrying warning first. The client asked for a producer id while the cluster was still loading; it retries.

Step 2: Lose one broker

The notebook stops a broker that holds a follower copy (here kafka-1), then writes 100 more readings.

$ docker compose stop kafka-1
leader: 3 | in-sync replicas: [2, 3]
({'delivered': 100}, (101, 200))

From now on, every new client in the notebook also logs that 127.0.0.1:9092 refuses connections: it tries all its bootstrap addresses, and kafka-1 is down. That's expected.

Step 3: Demand three in-sync copies

The notebook raises the topic's min.insync.replicas from 2 to 3 with admin.incremental_alter_configs() and writes 20 readings:

({'_MSG_TIMED_OUT': 20}, None) after 10 s

Every write failed, but only after the 10-second delivery timeout, and the error doesn't say why. With retries off, the broker's real answer shows:

code/module-02/02-durable-writes.ipynb
print(write(readings[220:221], **{"acks": "all", "enable.idempotence": False, "retries": 0}))
({'NOT_ENOUGH_REPLICAS': 1}, None)

Step 4: acks=1 in the same situation

code/module-02/02-durable-writes.ipynb
print("readable up to (before):", c.get_watermark_offsets(tp, timeout=10)[1])
print(write(readings[221:241], acks="1"))
print("readable up to (after): ", c.get_watermark_offsets(tp, timeout=10)[1])

get_watermark_offsets() returns the partition's high watermark as its second value:

readable up to (before): 201
({'delivered': 20}, (201, 220))
readable up to (after):  201

The producer heard "stored" for offsets 201 to 220, yet consumers still can't read past 201. The notebook then starts kafka-1 again and waits until all three replicas are in sync:

$ docker compose up -d --wait kafka-1
readable up to (broker back): 221

Break it: acks=0 never hears "no"

Predict first: the topic's maximum record size drops to 1,000 bytes, and a producer with acks=0 sends a 5,000-byte record. What does its delivery report say?

acks=0:   ({'delivered': 1}, None) | readable end: 221 -> 221
acks=all: ({'MSG_SIZE_TOO_LARGE': 1}, None)

What just happened

Step 1 wrote with the safe settings: acks=all plus an idempotent producer with its own producer id. Step 2 lost one replica; two were still in sync, which met min.insync.replicas=2, so writes went on. Step 3 demanded three in-sync copies with only two available. The leader refused every write, the client retried until its delivery timeout, and the report only said _MSG_TIMED_OUT. With retries off you saw the real reason, NOT_ENOUGH_REPLICAS. Step 4 showed the other side: acks=1 writes were acknowledged, but the high watermark didn't move, so no consumer could read them until the ISR recovered. In the break-it step the broker rejected the oversized record. With acks=0 the producer never heard it: its report said "delivered", while the readable end stayed at 221. That record is lost without a trace.

Clean up

The notebook has already restarted kafka-1 and set min.insync.replicas back to 2. The last cell closes the consumer and deletes the topic and the group. If anything gets stuck, docker compose down -v in the code/ folder resets the whole lab.

Key takeaways

Next: Batching, compression and throughput