Module 9 solutions

Solution 1: Fail back to the primary

The solution notebook is in code/module-09/exercise-1/solution/.

What the start version does wrong. It restarts billing on the primary at once. There, billing's group still has the offsets it committed before the North site failed, so it charges again every session it had already charged on the DR cluster from the copy. Depending on timing, it may also start the copy of the DR topic from the beginning, with no offsets there yet. In the build:

sessions in the history: 1627
charged again after failing back: 1134
never charged: 0
all right: False

Approach. A planned move in five steps, each waiting for a check:

code/module-09/exercise-1/solution/fail-back.ipynb
    # 1. Quiet moment: the gateway stops writing to the dr cluster (chargers buffer for a minute).
    lab.stop_script(gateway_dr)
code/module-09/exercise-1/solution/fail-back.ipynb
    lab.wait_until(lambda: charged_everything("dr", ["primary." + TOPIC, TOPIC], [BEFORE]), timeout=60,
                   interval=2, what="billing on dr to charge everything")
    lab.stop_script(billing_dr)
code/module-09/exercise-1/solution/fail-back.ipynb
    ends = [TopicPartition(t, p, end) for t in (TOPIC, "dr." + TOPIC)
            for p, end in dr.end_offsets("primary", t).items()]
    admin.alter_consumer_group_offsets([ConsumerGroupTopicPartitions(GROUP, ends)])[GROUP].result()
failback took 10 s
sessions in the history: 1633
charged again after failing back: 0
never charged: 0
all right: True

Why it works. Once the gateway stops writing to the DR cluster, nothing new arrives. Billing on the DR cluster then charges everything in both topics it reads: the copy of the primary's topic, complete once the primary is back, and the DR cluster's own topic. At that moment every record on the primary has been charged. That covers the primary's own topic, records copied before the failure and records copied after it came back, and the copy of the DR topic. So the right position on the primary is the end of both topics, and the solution sets it there. alter_consumer_group_offsets() only works on an empty group: the billing member that died with the North site has to time out first (45 seconds), so the solution waits for that too.

Offsets in the copies can't be compared with the originals (transaction markers, lesson 9.1), which is why the checks count records and session ids, never offsets.

Your runbook should say, in order: check the primary is healthy and MirrorMaker 2 copies both ways; stop the gateway's writes to the DR cluster; let billing finish there and check it; stop it; check the primary has everything written on the DR cluster; set billing's offsets on the primary; start billing, then the gateway; watch lag and charges per minute. And: until the offsets are set, you can stop and keep running on the DR cluster.

Solution 2: An integration test for the dead-letter path

The solution is in code/module-09/exercise-2/solution/.

Approach. Run the service against the container with a dead-letter topic, then check the four promises one by one:

code/module-09/exercise-2/solution/test_billing_dlq.py
    run_until_idle(BillingService(consumer, producer, "charges", "0.49", dlq_topic="charges.dlq"))
code/module-09/exercise-2/solution/test_billing_dlq.py
    charged = [json.loads(r.value())["session_id"] for r in read_all(kafka_bootstrap, "charges")]
    assert sorted(charged) == sorted(must_charge)                     # each finished session, once
code/module-09/exercise-2/solution/test_billing_dlq.py
        source = (int(headers["dlq.source.partition"]), int(headers["dlq.source.offset"]))
        assert sessions[source] == record.value()                     # points at the original
$ python -m pytest module-09/exercise-2/solution/test_billing_dlq.py -q --tb=short
.                                                                        [100%]
1 passed in 27.54s
...
your test passes: True
it fails against billing without a dead-letter path: True
done: True

Why it works. The charges and the parked records are read with read_committed, so the test sees what downstream services see: only what billing's transactions committed. Comparing the parked records with the originals, through their headers, proves the headers are right, not only present. The last assertion matters after a restart: billing's committed offsets are at the end of sessions, so it will never meet the broken events again.

Why the second check matters. Against the billing without a dead-letter path, run_batch() raises on the first broken event, aborts and rewinds: lesson 5.3's poison pill, and the test fails. A test that wrote only good sessions would pass against both versions and prove nothing about the dead-letter path. Breaking the code on purpose and watching the test fail is the quickest way to know a test tests something.