Notebook · code/module-09/01-disaster-recovery-with-mirrormaker.ipynb

Download 01-disaster-recovery-with-mirrormaker.ipynb

Open it in JupyterLab or VS Code from the lab folder to run it. This page is a read-only view.

Disaster recovery with MirrorMaker 2 — practical

MirrorMaker 2 copies Brightlane's sessions from the primary cluster to a disaster-recovery (DR) cluster, with billing's offsets. Then the primary site fails. You move billing and the gateway to the DR cluster, count what was charged twice, bring the primary back, and see what the DR cluster never received.

Lab services: kafka-1..3, kafka-dr, mirror-maker (docker compose --profile dr up -d --wait; the notebook starts them) · Lesson: modules/09-production-ready/01-disaster-recovery-with-mirrormaker.md

# Setup: find the lab's shared helpers
import sys, pathlib
lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / "common").is_dir())
sys.path.insert(0, str(lab_root))

import time
from common import dr, lab, settings

lab.compose("--profile", "dr", "up", "-d", "--wait")
TOPIC, GROUP, LEDGER = "m09l01-sessions", "m09l01-billing", ".lab-data/m09l01-ledger.jsonl"
COPY = "primary." + TOPIC                 # the name MirrorMaker 2 gives the copy on the dr cluster
dr.reset_mirroring("m09l01")              # a clean start: no topics, groups or copies from earlier runs
lab.recreate_topic(dr.admin("primary"), TOPIC, partitions=3)
lab.recreate_topic(dr.admin("dr"), TOPIC, partitions=3, replication=1)   # for the gateway, after a failover
(settings.LAB_ROOT / LEDGER).unlink(missing_ok=True)
print("ready")

Step 1: Normal operation

The gateway writes 40 session events a second to the primary, and billing charges every finished session into its ledger file. MirrorMaker 2 copies the topic to the DR cluster.

gateway = lab.start_script("module-03/session_feed.py", "--topic", TOPIC, "--rate", "40", name="m09l01-gateway")
billing = lab.start_script("module-09/dr_billing.py", "--cluster", "primary", "--topics", f"^{TOPIC}$",
                           "--group", GROUP, "--ledger", LEDGER, name="m09l01-billing-primary")
for _ in range(6):
    time.sleep(5)
    copied = dr.count_records("dr", COPY)            # counted, not taken from offsets: see the lesson
    written = dr.count_records("primary", TOPIC)
    print(f"records on primary: {written:5}   copied to dr: {copied:5}   not copied yet: {written - copied}")

What MirrorMaker 2 created on the DR cluster:

print(sorted(t for t in dr.admin("dr").list_topics(timeout=10).topics if not t.startswith("__")))

Step 2: Billing's offsets, translated

Billing commits its offsets on the primary. MirrorMaker 2 translates them into offsets of the copy and writes them into the same consumer group on the DR cluster. The copy's offsets are different numbers: MirrorMaker 2 writes the copy in transactions, and every commit marker takes an offset.

print("committed on primary:", dr.committed("primary", GROUP))
print("translated on dr:    ", dr.committed("dr", GROUP))

Step 3: The North site fails

First the link to the DR site gets congested: MirrorMaker 2 stops copying for 10 seconds (the notebook pauses its container). Then the North site loses power, with the three brokers, billing and the gateway in it.

lab.compose("--profile", "dr", "pause", "mirror-maker")
time.sleep(10)
lab.compose("kill", "kafka-1", "kafka-2", "kafka-3")
lab.kill_script(billing)
lab.kill_script(gateway)
lab.compose("--profile", "dr", "unpause", "mirror-maker")
print("charged on primary before the failure:", dr.ledger_report(LEDGER)[0])
print("records copied to dr:   ", dr.count_records("dr", COPY))
print("billing's offsets on dr:", dr.committed("dr", GROUP))

Step 4: Fail over

Billing starts on the DR cluster with the same group, so it continues from the translated offsets. Its topic pattern matches the copy and the topic the gateway writes to from now on. The gateway connects to the DR cluster too, with new sessions.

billing_dr = lab.start_script("module-09/dr_billing.py", "--cluster", "dr", "--topics", f"^(primary\\.)?{TOPIC}$",
                              "--group", GROUP, "--ledger", LEDGER, name="m09l01-billing-dr")
gateway_dr = lab.start_script("module-03/session_feed.py", "--topic", TOPIC, "--rate", "40", "--seed-offset", "1",
                              name="m09l01-gateway-dr", env={"KAFKA_BOOTSTRAP": settings.DR_BOOTSTRAP})
time.sleep(20)
billing_dr.tail(3)
charges, sessions, twice = dr.ledger_report(LEDGER)
print(f"ledger: {charges} charges for {sessions} sessions; charged twice: {twice}")

Step 5: The primary comes back

Before MirrorMaker 2 can copy anything more, compare what the primary holds with the copy. The notebook keeps MirrorMaker 2 stopped while the brokers start.

lab.compose("--profile", "dr", "stop", "mirror-maker")
lab.compose("up", "-d", "--wait", "kafka-1", "kafka-2", "kafka-3")
print("records on primary:", dr.count_records("primary", TOPIC), "  copied to dr:", dr.count_records("dr", COPY))
copied = set(dr.stopped_sessions("dr", COPY))
missing = [s for s in dr.stopped_sessions("primary", TOPIC) if s not in copied]
charged = {entry["session_id"] for entry in dr.ledger(LEDGER)}
print(f"finished sessions never copied: {len(missing)}, of which {sum(s in charged for s in missing)}"
      f" were charged on the primary before the failure")

Now let MirrorMaker 2 continue. It copies the missing records, and billing on the DR cluster charges what it finds.

lab.compose("--profile", "dr", "up", "-d", "mirror-maker")
lab.wait_until(lambda: dr.count_records("dr", COPY) == dr.count_records("primary", TOPIC), timeout=120,
               interval=2, what="the copy to catch up")
time.sleep(8)
charges, sessions, twice = dr.ledger_report(LEDGER)
print(f"ledger: {charges} charges for {sessions} sessions; charged twice: {twice}")

Break it: fail over without translated offsets

What if billing on the DR cluster had no translated offsets, for example because the group name was different there? A new group, reading the copy from the beginning:

other = ".lab-data/m09l01-ledger-new.jsonl"
(settings.LAB_ROOT / other).unlink(missing_ok=True)
fresh = lab.start_script("module-09/dr_billing.py", "--cluster", "dr", "--topics", f"^primary\\.{TOPIC}$",
                         "--group", GROUP + "-new", "--ledger", other, name="m09l01-billing-new")
time.sleep(15)
lab.stop_script(fresh)
already = {entry["session_id"] for entry in dr.ledger(LEDGER)}
again = [entry["session_id"] for entry in dr.ledger(other)]
print(f"the new group charged {len(again)} sessions; {sum(s in already for s in again)} of them were charged before")

Clean up

Stop billing and the gateway, and remove the topics, groups and copies. The DR cluster and MirrorMaker 2 keep running for exercise 1; to stop them: docker compose --profile dr stop.

lab.stop_script(gateway_dr)
lab.stop_script(billing_dr)
dr.reset_mirroring("m09l01")
print("cleaned up")