34 min · theory + hands-on practical

Disaster recovery with MirrorMaker 2

Replication inside a cluster (module 4) survives a broker. It doesn't survive the building the brokers stand in. This lesson keeps a copy of Brightlane's data on a second cluster at another site, moves billing there when the first site fails, and counts what that costs.

Theory

Example: the North site overheats

At 19:40 on a hot Friday, the cooling of Brightlane's North data centre fails. To save the hardware, the operators shut the room down: all three brokers, the billing servers and the gateway. The repair takes six hours. Chargers keep charging and buffer their readings for a few minutes, but nothing reaches Kafka. Billing stops, and the grid operator's 15-minute reports stop with it.

Replicas on three brokers didn't help: all three were in the same room. Brightlane needs a second copy somewhere else, and a plan for using it.

Failure domains, RPO and RTO

A failure domain is a set of things that fail together: a disk, a broker, a rack, a data centre, a cloud region. Each protection covers some of them. Replication with min.insync.replicas=2 covers a broker. Rack awareness (lesson 8.4) covers a rack or an availability zone. A whole site needs copies at another site.

Two numbers describe what a site failure may cost:

There are two ways to spread Kafka over sites:

Brightlane has two sites: the North data centre and a smaller one in the cloud. Mirrored clusters it is, active-passive: one cluster serves the clients, the other only receives copies until it's needed. (In an active-active setup, both serve clients and copy to each other.)

MirrorMaker 2

MirrorMaker 2 is Apache Kafka's tool for copying between clusters. It's a set of Kafka Connect connectors (lesson 6.1): one copies records and topic settings, one translates consumer offsets, one writes heartbeats that show the link works. It runs on its own (connect-mirror-maker.sh, as in the lab) or in a Connect cluster. The older MirrorMaker 1 was removed in Kafka 4.2.

code/lab/mirror-maker/mm2.properties
clusters = primary, dr
primary.bootstrap.servers = kafka-1:19092,kafka-2:19092,kafka-3:19092
dr.bootstrap.servers = kafka-dr:19092

# Normal operation: copy Brightlane's topics and consumer-group offsets from primary to dr.
# Remote topics get the source's name as a prefix: charging.sessions -> primary.charging.sessions.
primary->dr.enabled = true
primary->dr.topics = charging\..*, billing\..*, m09.*
primary->dr.groups = .*

A copied topic is a remote topic. By default its name starts with the source cluster's name: charging.sessions becomes primary.charging.sessions. The prefix shows where data came from, and lets two clusters copy to each other without copying copies back. The alternative policy, IdentityReplicationPolicy, keeps names unchanged; that's simpler for clients, but works only one way.

Two settings make the copy trustworthy. Without exactly-once support, a MirrorMaker 2 that is killed copies part of a topic again when it restarts; while building this lesson, a restart copied a whole topic twice. With it, each batch of copied records and the position it reached are written in one transaction (lesson 7.1):

code/lab/mirror-maker/mm2.properties
dr.exactly.once.source.support = enabled
primary.exactly.once.source.support = enabled

The transactions' commit markers take offsets too. So a record's offset in the copy differs from its offset in the original. Never compare offsets across clusters; count records, or let MirrorMaker 2 translate.

Two sites. In the North site the primary cluster holds charging.sessions; the gateway writes to it and billing reads it and commits its offsets. MirrorMaker 2 copies the topic to the DR site's cluster as primary.charging.sessions, and translates billing's committed offsets into offsets of the copy, which it writes into billing's group on the DR cluster. After a failover, the gateway writes to the DR cluster's own charging.sessions, and billing reads both that topic and the copy, starting from the translated offsets.

What to notice: MirrorMaker 2 carries two things to the DR site: the records, and how far each consumer group had got. Billing needs both to continue there.

Failing over with translated offsets

Committed offsets belong to one cluster. For billing to continue on the DR cluster, it needs its group's position in the copy. MirrorMaker 2 regularly writes offset syncs: pairs of "this offset in the original is that offset in the copy". From them it translates each group's committed offsets into offsets of the copy, and with sync.group.offsets.enabled = true it writes them into the same group on the DR cluster.

The translation is careful: it goes to the last sync at or before the committed offset, never past it. So after a failover, billing re-reads some records it already handled on the primary. Add the records the copy never received, and an unplanned failover always has two costs:

One partition of charging.sessions on the primary when the North site fails, from old to new. The oldest records were charged and their offsets were translated to the DR cluster: nothing to do. The next records were charged on the primary, but MirrorMaker 2 hadn't translated those offsets yet: billing on the DR cluster charges them again. The newest records never reached the DR cluster: they are missing there until the primary comes back, and lost if it never does. That last part is the recovery point.

What to notice: both costs come from the same thing, the copy running behind. You can make them small, never zero, so billing must tolerate them: charge by session id, idempotently (lesson 3.2).

After the failover, the gateway writes to a topic of the DR cluster's own, charging.sessions there, and billing reads both it and the copy, with a pattern: ^(primary\.)?charging\.sessions$. Going back to the primary later, the failback, is a planned move, done at a quiet moment without losing or repeating anything. Exercise 1 does it.

In production

Practical

Note

Scripts: code/module-09/dr_billing.py and code/module-03/session_feed.py (the notebook starts, kills and moves them) · Notebook: code/module-09/01-disaster-recovery-with-mirrormaker.ipynb · Lab services: kafka-1..3, kafka-dr, mirror-maker (docker compose --profile dr up -d --wait; the notebook starts them) · Time: ~18 min

What you'll do: run the gateway and billing on the primary while MirrorMaker 2 copies, make the North site fail, move billing and the gateway to the DR cluster, count the sessions charged twice, and find what the DR cluster never received.

Step 1: Normal operation

The gateway writes 40 session events a second, and billing charges each finished session into a ledger file, one line per charge. Every 5 seconds the notebook counts the records on both sides:

code/module-09/01-disaster-recovery-with-mirrormaker.ipynb
    copied = dr.count_records("dr", COPY)            # counted, not taken from offsets: see the lesson
    written = dr.count_records("primary", TOPIC)
records on primary:   216   copied to dr:     0   not copied yet: 216
records on primary:   461   copied to dr:   438   not copied yet: 23
...
records on primary:  1437   copied to dr:  1415   not copied yet: 22

MirrorMaker 2 created the copy and its own topics on the DR cluster:

['heartbeats', 'm09l01-sessions', 'mm2-configs.primary.internal', 'mm2-offset-syncs.primary.internal', 'mm2-offsets.primary.internal', 'mm2-status.primary.internal', 'primary.checkpoints.internal', 'primary.heartbeats', 'primary.m09l01-sessions']

Step 2: Billing's offsets, translated

committed on primary: {'m09l01-sessions': {0: 515, 1: 445, 2: 436}}
translated on dr:     {'primary.m09l01-sessions': {0: 708, 1: 702, 2: 577}}

Step 3: The North site fails

First the link to the DR site gets congested: the notebook pauses MirrorMaker 2's container for 10 seconds. Then the three brokers, billing and the gateway die at once:

code/module-09/01-disaster-recovery-with-mirrormaker.ipynb
lab.compose("--profile", "dr", "pause", "mirror-maker")
time.sleep(10)
lab.compose("kill", "kafka-1", "kafka-2", "kafka-3")
lab.kill_script(billing)
lab.kill_script(gateway)
charged on primary before the failure: 886
records copied to dr:    1450
billing's offsets on dr: {'primary.m09l01-sessions': {0: 708, 1: 702, 2: 577}}

Step 4: Fail over

Billing starts on the DR cluster with the same group and a topic pattern; the gateway connects there with new sessions. After 20 seconds:

ledger: 1418 charges for 1238 sessions; charged twice: 180

Step 5: The primary comes back

The notebook keeps MirrorMaker 2 stopped while the brokers start, and compares:

records on primary: 1852   copied to dr: 1450
finished sessions never copied: 208, of which 208 were charged on the primary before the failure

Then MirrorMaker 2 continues, copies the missing records, and billing on the DR cluster finds them:

ledger: 2374 charges for 1986 sessions; charged twice: 388

Your numbers differ a little from run to run.

Break it: fail over without translated offsets

Predict first: billing starts on the DR cluster under a new group name, so it has no translated offsets, and reads the copy from the beginning.

the new group charged 886 sessions; 886 of them were charged before

What just happened

In the first seconds the copy was empty: MirrorMaker 2 looks for new topics every 5 seconds in the lab. After that it ran about 23 records behind, about half a second: MirrorMaker 2 commits its copy in transactions, and a read_committed reader sees each batch once it's committed. That half second was the RPO on an ordinary day.

Billing's offsets arrived on the DR cluster as different numbers, because the copy also holds commit markers. They were translated to points MirrorMaker 2 had synced, never beyond what billing had committed. When billing continued on the DR cluster, it re-read the records between those points and what it had really done: 180 sessions charged twice. That's the at-least-once side of a failover.

The 10-second pause made the other side visible. 402 records reached the primary but never the copy, with 208 finished sessions. Billing happened to have charged them on the primary already. Had the primary's disks been lost, they would have been lost for good; the 10 seconds were the RPO that evening. When the primary came back, MirrorMaker 2 copied them, and billing on the DR cluster charged them a second time: 388 duplicates in all. In production, an idempotent ledger keyed by session id absorbs every one of them.

Without translated offsets, billing had no position on the DR cluster and started from the beginning: every session in the copy was charged again. The offset translation is what makes a failover a matter of seconds of repeated work rather than hours.

Clean up

The last cell stops billing and the gateway, and removes the topics, groups and copies on both clusters. The DR cluster and MirrorMaker 2 keep running for exercise 1; to stop them: docker compose --profile dr stop.

Key takeaways

Next: Testing Kafka applications