Disaster recovery with MirrorMaker 2
Replication inside a cluster (module 4) survives a broker. It doesn't survive the building the brokers stand in. This lesson keeps a copy of Brightlane's data on a second cluster at another site, moves billing there when the first site fails, and counts what that costs.
Theory
Example: the North site overheats
At 19:40 on a hot Friday, the cooling of Brightlane's North data centre fails. To save the hardware, the operators shut the room down: all three brokers, the billing servers and the gateway. The repair takes six hours. Chargers keep charging and buffer their readings for a few minutes, but nothing reaches Kafka. Billing stops, and the grid operator's 15-minute reports stop with it.
Replicas on three brokers didn't help: all three were in the same room. Brightlane needs a second copy somewhere else, and a plan for using it.
Failure domains, RPO and RTO
A failure domain is a set of things that fail together: a disk, a broker, a rack, a data centre,
a cloud region. Each protection covers some of them. Replication with min.insync.replicas=2
covers a broker. Rack awareness (lesson 8.4) covers a rack or an availability zone. A whole site
needs copies at another site.
Two numbers describe what a site failure may cost:
- RPO (recovery point objective): how much recent data you may lose, as time. "Up to 10 seconds of sessions."
- RTO (recovery time objective): how long until the service runs again. "Billing within 15 minutes."
There are two ways to spread Kafka over sites:
- A stretched cluster has brokers and controllers in three sites, and replicas spread over them.
With
acks=alla write is in two sites before it counts, so the RPO is zero and failover is automatic. The price: every write waits for another site, so the sites must be close (a few milliseconds apart), and you need three of them for the controllers' majority. - Mirrored clusters are independent clusters, one per site, with a tool copying topics from one to the other in the background. Writes stay fast and the sites can be far apart. The copy runs a little behind, so the RPO is that lag, and moving clients over is a step someone takes, so the RTO is minutes.
Brightlane has two sites: the North data centre and a smaller one in the cloud. Mirrored clusters it is, active-passive: one cluster serves the clients, the other only receives copies until it's needed. (In an active-active setup, both serve clients and copy to each other.)
MirrorMaker 2
MirrorMaker 2 is Apache Kafka's tool for copying between clusters. It's a set of Kafka Connect
connectors (lesson 6.1): one copies records and topic settings, one translates consumer offsets, one
writes heartbeats that show the link works. It runs on its own (connect-mirror-maker.sh, as in the
lab) or in a Connect cluster. The older MirrorMaker 1 was removed in Kafka 4.2.
clusters = primary, dr
primary.bootstrap.servers = kafka-1:19092,kafka-2:19092,kafka-3:19092
dr.bootstrap.servers = kafka-dr:19092
# Normal operation: copy Brightlane's topics and consumer-group offsets from primary to dr.
# Remote topics get the source's name as a prefix: charging.sessions -> primary.charging.sessions.
primary->dr.enabled = true
primary->dr.topics = charging\..*, billing\..*, m09.*
primary->dr.groups = .*
A copied topic is a remote topic. By default its name starts with the source cluster's name:
charging.sessions becomes primary.charging.sessions. The prefix shows where data came from, and
lets two clusters copy to each other without copying copies back. The alternative policy,
IdentityReplicationPolicy, keeps names unchanged; that's simpler for clients, but works only one way.
Two settings make the copy trustworthy. Without exactly-once support, a MirrorMaker 2 that is killed copies part of a topic again when it restarts; while building this lesson, a restart copied a whole topic twice. With it, each batch of copied records and the position it reached are written in one transaction (lesson 7.1):
dr.exactly.once.source.support = enabled
primary.exactly.once.source.support = enabled
The transactions' commit markers take offsets too. So a record's offset in the copy differs from its offset in the original. Never compare offsets across clusters; count records, or let MirrorMaker 2 translate.
What to notice: MirrorMaker 2 carries two things to the DR site: the records, and how far each consumer group had got. Billing needs both to continue there.
Failing over with translated offsets
Committed offsets belong to one cluster. For billing to continue on the DR cluster, it needs its
group's position in the copy. MirrorMaker 2 regularly writes offset syncs: pairs of "this offset
in the original is that offset in the copy". From them it translates each group's committed offsets
into offsets of the copy, and with sync.group.offsets.enabled = true it writes them into the same
group on the DR cluster.
The translation is careful: it goes to the last sync at or before the committed offset, never past it. So after a failover, billing re-reads some records it already handled on the primary. Add the records the copy never received, and an unplanned failover always has two costs:
What to notice: both costs come from the same thing, the copy running behind. You can make them small, never zero, so billing must tolerate them: charge by session id, idempotently (lesson 3.2).
After the failover, the gateway writes to a topic of the DR cluster's own, charging.sessions
there, and billing reads both it and the copy, with a pattern: ^(primary\.)?charging\.sessions$.
Going back to the primary later, the failback, is a planned move, done at a quiet moment
without losing or repeating anything. Exercise 1 does it.
In production
- Watch the copy's lag (MirrorMaker 2's
replication-latency-msandrecord-age-msmetrics, or the age of the newest heartbeat) and alert when it threatens your RPO. - Write the failover and failback runbooks, and rehearse them a few times a year on a quiet day. An untried runbook fails at step 3.
- Let a person decide to fail over a whole site. Automatic failover between sites risks both sides serving at once after a network split.
- The DR cluster must carry the full load after a failover: same partitions, enough brokers.
- MirrorMaker 2 doesn't copy users and passwords: create the SCRAM users on the DR cluster too. It
can copy topic ACLs (
sync.topic.acls.enabled). Services that use schemas need a Schema Registry at the DR site as well. - A copy isn't a backup: a topic deleted by mistake can be gone from both clusters minutes later, and tiered storage (lesson 4.2) isn't one either. If you need point-in-time recovery, copy to object storage with a sink connector.
- Managed services have their own copying: Confluent's Cluster Linking keeps offsets identical, and Amazon MSK Replicator copies between MSK clusters.
Practical
Note
Scripts: code/module-09/dr_billing.py and code/module-03/session_feed.py (the notebook starts, kills and moves them)
· Notebook: code/module-09/01-disaster-recovery-with-mirrormaker.ipynb
· Lab services: kafka-1..3, kafka-dr, mirror-maker (docker compose --profile dr up -d --wait; the notebook starts them) · Time: ~18 min
What you'll do: run the gateway and billing on the primary while MirrorMaker 2 copies, make the North site fail, move billing and the gateway to the DR cluster, count the sessions charged twice, and find what the DR cluster never received.
Step 1: Normal operation
The gateway writes 40 session events a second, and billing charges each finished session into a ledger file, one line per charge. Every 5 seconds the notebook counts the records on both sides:
copied = dr.count_records("dr", COPY) # counted, not taken from offsets: see the lesson
written = dr.count_records("primary", TOPIC)
records on primary: 216 copied to dr: 0 not copied yet: 216
records on primary: 461 copied to dr: 438 not copied yet: 23
...
records on primary: 1437 copied to dr: 1415 not copied yet: 22
MirrorMaker 2 created the copy and its own topics on the DR cluster:
['heartbeats', 'm09l01-sessions', 'mm2-configs.primary.internal', 'mm2-offset-syncs.primary.internal', 'mm2-offsets.primary.internal', 'mm2-status.primary.internal', 'primary.checkpoints.internal', 'primary.heartbeats', 'primary.m09l01-sessions']
Step 2: Billing's offsets, translated
committed on primary: {'m09l01-sessions': {0: 515, 1: 445, 2: 436}}
translated on dr: {'primary.m09l01-sessions': {0: 708, 1: 702, 2: 577}}
Step 3: The North site fails
First the link to the DR site gets congested: the notebook pauses MirrorMaker 2's container for 10 seconds. Then the three brokers, billing and the gateway die at once:
lab.compose("--profile", "dr", "pause", "mirror-maker")
time.sleep(10)
lab.compose("kill", "kafka-1", "kafka-2", "kafka-3")
lab.kill_script(billing)
lab.kill_script(gateway)
charged on primary before the failure: 886
records copied to dr: 1450
billing's offsets on dr: {'primary.m09l01-sessions': {0: 708, 1: 702, 2: 577}}
Step 4: Fail over
Billing starts on the DR cluster with the same group and a topic pattern; the gateway connects there with new sessions. After 20 seconds:
ledger: 1418 charges for 1238 sessions; charged twice: 180
Step 5: The primary comes back
The notebook keeps MirrorMaker 2 stopped while the brokers start, and compares:
records on primary: 1852 copied to dr: 1450
finished sessions never copied: 208, of which 208 were charged on the primary before the failure
Then MirrorMaker 2 continues, copies the missing records, and billing on the DR cluster finds them:
ledger: 2374 charges for 1986 sessions; charged twice: 388
Your numbers differ a little from run to run.
Break it: fail over without translated offsets
Predict first: billing starts on the DR cluster under a new group name, so it has no translated offsets, and reads the copy from the beginning.
the new group charged 886 sessions; 886 of them were charged before
What just happened
In the first seconds the copy was empty: MirrorMaker 2 looks for new topics every 5 seconds in the
lab. After that it ran about 23 records behind, about half a second: MirrorMaker 2 commits its copy
in transactions, and a read_committed reader sees each batch once it's committed. That half
second was the RPO on an ordinary day.
Billing's offsets arrived on the DR cluster as different numbers, because the copy also holds commit markers. They were translated to points MirrorMaker 2 had synced, never beyond what billing had committed. When billing continued on the DR cluster, it re-read the records between those points and what it had really done: 180 sessions charged twice. That's the at-least-once side of a failover.
The 10-second pause made the other side visible. 402 records reached the primary but never the copy, with 208 finished sessions. Billing happened to have charged them on the primary already. Had the primary's disks been lost, they would have been lost for good; the 10 seconds were the RPO that evening. When the primary came back, MirrorMaker 2 copied them, and billing on the DR cluster charged them a second time: 388 duplicates in all. In production, an idempotent ledger keyed by session id absorbs every one of them.
Without translated offsets, billing had no position on the DR cluster and started from the beginning: every session in the copy was charged again. The offset translation is what makes a failover a matter of seconds of repeated work rather than hours.
Clean up
The last cell stops billing and the gateway, and removes the topics, groups and copies on both
clusters. The DR cluster and MirrorMaker 2 keep running for exercise 1; to stop them:
docker compose --profile dr stop.
Key takeaways
- Replicas protect against a broker; a site failure needs a copy at another site: a stretched cluster (RPO 0, close sites) or mirrored clusters (RPO = the copy's lag).
- MirrorMaker 2 copies topics as
primary.<topic>and translates consumer offsets; turn on its exactly-once support so a restart doesn't copy records twice. - Offsets in a copy differ from the original's: count records, or use translated offsets.
- An unplanned failover repeats some work and may lose the last moments: make consumers idempotent and watch the copy's lag.
- Fail back as a planned move, at a quiet moment, with a rehearsed runbook.