Durable writes: acks, replicas and idempotence
A producer's job ends when Kafka has the record safely. This lesson is about what "safely" means: how many copies, who confirms them, and what happens when a broker disappears halfway. Get these settings wrong and you either lose data or store it twice, and nothing warns you.
Theory
Example: the restart at 18:05
On Thursday at 18:05, during the evening peak, the platform team restarted the brokers one by one to apply a patch. The next morning two problems turned up:
- 312 meter readings were gone. Weeks earlier someone had set the gateway to
acks=1"for speed". kafka-2 had acknowledged those readings and then restarted before its followers copied them. The follower that took over as leader never had them. - 40 readings were stored twice. A test producer with the default Python settings resent a batch whose acknowledgement got lost in the restart. Billing counted that energy twice.
Both problems come from three settings: acks, min.insync.replicas and
enable.idempotence.
acks: when the producer hears "stored"
acks is a producer setting. It decides when the partition's leader answers a produce request:
acks=0: the leader never answers. The producer assumes success as soon as the request is on the network. Errors are invisible.acks=1: the leader answers once the batch is in its own log. If the leader dies before a follower copies the batch, the batch is lost.acks=all(also written-1): the leader answers once every in-sync replica has the batch. This is the default in both the Java and the Python client.
What to notice: acks=all waits for the followers' next fetch, which is a few milliseconds
on a healthy cluster. That wait is what buys safety.
In-sync replicas and min.insync.replicas
Lesson 1.2 introduced the ISR, the replicas that are fully caught up. A follower that stops
fetching for 30 seconds (replica.lag.time.max.ms) is dropped from the ISR.
acks=all waits for every replica currently in the ISR. If two brokers are down, the ISR may
be just the leader, and then acks=all is no safer than acks=1. The topic setting
min.insync.replicas closes that gap. It says how many in-sync replicas must exist before
the leader accepts an acks=all write. With fewer, the leader refuses the write with
NOT_ENOUGH_REPLICAS.
With 3 replicas, min.insync.replicas=2 is the standard choice: the topic keeps accepting writes
with one broker down, and every acknowledged record exists on at least two brokers. Setting it to
3 means any single broker restart stops all writes.
There's a second effect, on the reading side. Consumers can only read up to the partition's
high watermark: the offset up to which all in-sync replicas have the records. On clusters
created with Kafka 4.1 or later (where the "eligible leader replicas" feature is on by default),
the high watermark also stops moving while the ISR is smaller than min.insync.replicas. So
records written with acks=1 during that time are stored, but nobody can read them yet.
What to notice: records 201 to 220 sit in the leader's log, but the high watermark stays at 201 until the ISR is big enough again.
Retries, timeouts and delivery reports
When a write fails for a temporary reason (a leader moving, too few in-sync replicas, a network
blip), the client retries it on its own, again and again. It gives up only when the record has
waited longer than the delivery timeout: message.timeout.ms in the Python client (5 minutes
by default), delivery.timeout.ms in Java. Then it reports _MSG_TIMED_OUT, which hides the
broker's real reason.
The only place your code learns the outcome is the delivery report: a callback you pass to
produce(), called from poll() or flush(). It gets an error, or the partition and offset where
the record was stored. Some errors are final at once, such as a record larger than the topic
allows (MSG_SIZE_TOO_LARGE).
Don't add your own retry loop around the same record. The client already retries, and resending from your code creates duplicates and breaks order.
The idempotent producer
Retries create the duplicate problem: the leader stored a batch, the acknowledgement was lost, and the producer sent the batch again. An idempotent producer fixes it:
- When it starts, the producer gets a producer id from the cluster.
- It numbers every batch it sends to a partition: sequence 0, 1, 2…
- The leader remembers the last sequence numbers per producer id. A batch it has already stored is acknowledged again but not stored twice. A batch that arrives too early (out of order) is refused and retried.
What to notice: the retry carries the same producer id and sequence number, so the leader can tell it's a repeat.
Idempotence also keeps order during retries with up to 5 requests in flight. It needs
acks=all. Java clients have it on by default since Kafka 3.0, but the Python client doesn't:
enable.idempotence defaults to false in librdkafka. Turn it on.
It doesn't protect against everything. A producer that restarts gets a new producer id, and if your
code calls produce() twice for the same reading, Kafka sees two different records. Module 3
deals with duplicates on the consumer side, and lesson 7.1 with transactions.
In production
- Producers:
acks=allandenable.idempotence=true. In Python, set both explicitly. - Topics: replication factor 3 and
min.insync.replicas=2. Never make it equal to the replication factor. - Leave
unclean.leader.election.enableoff (the default). Turning it on lets an out-of-date replica become leader, which loses data. - Treat every error in a delivery report as an alert-worthy event: log it, count it, and decide
what the application does (stop, or park the record). Call
flush()before a producer exits. - Keep the delivery timeout long enough to ride out a broker restart (the 5-minute default does), and make sure your application's own timeouts don't cut it short.
Practical
Note
Notebook: code/module-02/02-durable-writes.ipynb
· Lab services: kafka-1, kafka-2, kafka-3 · Time: ~17 min
What you'll do: write meter readings with delivery reports, then stop a broker and see what
acks=all, min.insync.replicas, acks=1 and acks=0 each do.
Step 1: Write with acks=all and an idempotent producer
write(records, **config) sends records with the given settings and counts the outcomes from the
delivery reports. It sets message.timeout.ms to 10 seconds, so the notebook doesn't wait 5
minutes for failures.
def report(err, msg):
outcomes[err.name() if err else "delivered"] += 1
if not err and msg.offset() is not None:
offsets.append(msg.offset())
safe = {"acks": "all", "enable.idempotence": True}
print(write(readings[:100], **safe))
({'delivered': 100}, (0, 99))
The client's statistics show the producer id the cluster assigned (the number varies):
idempotence state: Assigned | producer id: 4 | epoch: 0
On a freshly started lab you may also see one GETPID … Coordinator load in progress: retrying
warning first. The client asked for a producer id while the cluster was still loading; it retries.
Step 2: Lose one broker
The notebook stops a broker that holds a follower copy (here kafka-1), then writes 100 more readings.
$ docker compose stop kafka-1
leader: 3 | in-sync replicas: [2, 3]
({'delivered': 100}, (101, 200))
From now on, every new client in the notebook also logs that 127.0.0.1:9092 refuses
connections: it tries all its bootstrap addresses, and kafka-1 is down. That's expected.
Step 3: Demand three in-sync copies
The notebook raises the topic's min.insync.replicas from 2 to 3 with
admin.incremental_alter_configs() and writes 20 readings:
({'_MSG_TIMED_OUT': 20}, None) after 10 s
Every write failed, but only after the 10-second delivery timeout, and the error doesn't say why. With retries off, the broker's real answer shows:
print(write(readings[220:221], **{"acks": "all", "enable.idempotence": False, "retries": 0}))
({'NOT_ENOUGH_REPLICAS': 1}, None)
Step 4: acks=1 in the same situation
print("readable up to (before):", c.get_watermark_offsets(tp, timeout=10)[1])
print(write(readings[221:241], acks="1"))
print("readable up to (after): ", c.get_watermark_offsets(tp, timeout=10)[1])
get_watermark_offsets() returns the partition's high watermark as its second value:
readable up to (before): 201
({'delivered': 20}, (201, 220))
readable up to (after): 201
The producer heard "stored" for offsets 201 to 220, yet consumers still can't read past 201. The notebook then starts kafka-1 again and waits until all three replicas are in sync:
$ docker compose up -d --wait kafka-1
readable up to (broker back): 221
Break it: acks=0 never hears "no"
Predict first: the topic's maximum record size drops to 1,000 bytes, and a producer with
acks=0 sends a 5,000-byte record. What does its delivery report say?
acks=0: ({'delivered': 1}, None) | readable end: 221 -> 221
acks=all: ({'MSG_SIZE_TOO_LARGE': 1}, None)
What just happened
Step 1 wrote with the safe settings: acks=all plus an idempotent producer with its own producer
id. Step 2 lost one replica; two were still in sync, which met min.insync.replicas=2, so writes
went on. Step 3 demanded three in-sync copies with only two available. The leader refused
every write, the client retried until its delivery timeout, and the report only said
_MSG_TIMED_OUT. With retries off you saw the real reason, NOT_ENOUGH_REPLICAS. Step 4 showed
the other side: acks=1 writes were acknowledged, but the high watermark didn't move, so no
consumer could read them until the ISR recovered. In the break-it step the broker rejected the
oversized record. With acks=0 the producer never heard it: its report said "delivered", while
the readable end stayed at 221. That record is lost without a trace.
Clean up
The notebook has already restarted kafka-1 and set min.insync.replicas back to 2. The last cell
closes the consumer and deletes the topic and the group. If anything gets stuck, docker compose
down -v in the code/ folder resets the whole lab.
Key takeaways
acks=allwaits for every in-sync replica;acks=1trusts the leader alone;acks=0hears nothing, ever.min.insync.replicas=2with 3 replicas keeps acknowledged data on two brokers and survives one broker down.- The client retries temporary errors until
message.timeout.ms; the delivery report is your only source of truth. - The idempotent producer removes duplicates from retries. It's off by default in Python: set
enable.idempotence=true. - Records are readable only up to the high watermark, which waits for enough in-sync replicas.