28 min · theory + hands-on practical

Testing Kafka applications

A Kafka application has two kinds of code: what it decides, and how it talks to Kafka. They fail differently and are tested differently. This lesson builds billing's tests in four layers, and replays the bug they would have caught.

Theory

Example: one cent, one session in forty

In spring, a developer tidied up billing's charge calculation: the decimal arithmetic became round(energy * price, 2). Shorter, and the tests passed: they checked 10 kWh at 0.49 €/kWh, 4.90 €. Nothing happened for months. At 0.49 €/kWh, only 7 sessions in 20,000 land exactly on half a cent.

Then the motorway sites opened with a 0.50 €/kWh tariff. 52.5 kWh at 0.49 is 25.725 €; 56.05 kWh at 0.50 is 28.025 €. The tariff terms say "rounded half up to the cent", 25.73 and 28.03. But in binary floating point 25.725 is stored as 25.724999…, and round() makes it 25.72. At 0.50 €/kWh, 1 session in 40 was charged a cent too little. Finance found it in the monthly reconciliation, not a test. (Lesson 7.1's billing service uses the same round(): fine for showing transactions, wrong for money.)

Four layers of tests

Each layer catches what the others can't, at a different cost:

  1. Unit tests check business logic alone: no Kafka, no network, milliseconds each. Here, charge_for(): its rounding, its edge cases, and all 20,000 scenario sessions compared with exact decimal arithmetic.
  2. Tests with fake clients check the service's steps. A fake is a small stand-in object that behaves like the real client for the test and records what it was asked to do. Did the batch begin a transaction, produce, send the offsets and commit, in that order? Did a failed write abort? Still milliseconds, but a fake only knows what you assumed about Kafka.
  3. Integration tests run the real flow against a real broker. Testcontainers is a library that starts a Docker container for a test run, hands the test its address, and removes it afterwards. Here it runs Kafka 4.3.1, and the test crashes billing in the middle of a batch and reads the charges as an auditor would. About 20 seconds, and it checks what no fake can: transactions, read_committed, the broker's own errors.
  4. Schema compatibility tests check that a proposed schema can still be read by today's consumers, and read today's data, before the change is merged (lesson 5.2).

Four layers of tests for billing, from fast and many to slow and few. Unit tests check charge_for() alone, 20,000 sessions in a fraction of a second. Tests with fake clients check the service's steps: begin, produce, offsets, commit, and abort on failure. An integration test runs the transactional flow against Kafka 4.3.1 in a container, about 20 seconds. Schema compatibility tests check the proposed schema against the published one before a merge.

What to notice: most tests sit in the fast layers. The slow one is there for what only a real broker can show.

Code that can be tested

Tests are easy when the code is shaped for them:

code/module-09/billing.py
def charge_for(stopped, price_eur_per_kwh):
    """The charge for one finished session: energy x price, rounded half up to the cent.

    Decimal, not float: 52.5 x 0.49 is 25.725 exactly, and the tariff terms say half up, 25.73.
    In binary floating point it's 25.724999..., which round() turns into 25.72.
    """
    amount = Decimal(str(stopped["energy_kwh"])) * Decimal(str(price_eur_per_kwh))

A Kafka container for tests

The container needs two listeners, for the same reason as the lab in lesson 1.2. Tests on your machine reach the broker at a mapped port, chosen before the start so the broker can advertise it. The broker talks to itself, for transactions, on an address inside the container:

code/module-09/tests/kafka_container.py
    env = {"KAFKA_NODE_ID": "1", "KAFKA_PROCESS_ROLES": "broker,controller",
           "KAFKA_LISTENERS": "HOST://:9092,BROKER://:19092,CONTROLLER://:9093",
           "KAFKA_ADVERTISED_LISTENERS": f"HOST://127.0.0.1:{port},BROKER://localhost:19092",
           "KAFKA_INTER_BROKER_LISTENER_NAME": "BROKER",

With one listener, the first version of this fixture hung: the broker tried to reach itself at 127.0.0.1:<mapped port>, which only exists outside the container. The Testcontainers package for Python has a ready-made KafkaContainer, but it's built for Confluent's images; the course starts the Apache image with the generic DockerContainer. librdkafka also has a built-in mock cluster (test.mock.num.brokers), handy for injecting errors, but it only imitates part of Kafka.

In production

Practical

Note

Tests: code/module-09/tests/ for code/module-09/billing.py · Notebook: code/module-09/02-testing-kafka-applications.ipynb · Lab services: none of the lab's; the integration test starts its own Kafka container (needs Docker) · Time: ~14 min

What you'll do: run each layer of billing's tests, try two proposed schema changes, then apply the refactor that under-charged drivers and watch the tests catch it. You can run every command in a terminal too, from the code/ folder.

Step 1: Unit tests of the charge

code/module-09/tests/test_charge.py
@pytest.mark.parametrize("energy_kwh, price, expected", [
    (10.0, "0.49", "4.90"),
    (13.25, "0.49", "6.49"),     # 6.4925: below the half cent, down
    (52.5, "0.49", "25.73"),     # 25.725: exactly half a cent, up (the tariff terms)
$ python -m pytest module-09/tests/test_charge.py -v
module-09/tests/test_charge.py::test_amount_is_energy_times_price_rounded_half_up[10.0-0.49-4.90] PASSED [ 11%]
...
module-09/tests/test_charge.py::test_every_scenario_session_matches_decimal_arithmetic PASSED [100%]

============================== 9 passed in 0.11s ===============================

Step 2: The service with fake clients

code/module-09/tests/test_billing_with_fakes.py
def test_one_charge_per_finished_session_then_offsets_then_commit():
    producer = FakeProducer()
    service = BillingService(FakeConsumer(session_events(10.0, 52.5)), producer, "charges", "0.49")
    assert service.run_batch() == (2, 0)
    assert producer.calls == ["begin", ("produce", "charges", "ses-0"), ("produce", "charges", "ses-1"),
                              "offsets", "commit"]
$ python -m pytest module-09/tests/test_billing_with_fakes.py -v
module-09/tests/test_billing_with_fakes.py::test_one_charge_per_finished_session_then_offsets_then_commit PASSED [ 33%]
module-09/tests/test_billing_with_fakes.py::test_a_failed_write_aborts_and_rewinds PASSED [ 66%]
module-09/tests/test_billing_with_fakes.py::test_an_unreadable_event_is_parked_and_the_batch_still_commits PASSED [100%]

============================== 3 passed in 0.02s ===============================

Step 3: The transactional flow against a real Kafka

The test writes 40 sessions, lets billing crash on its 25th charge, restarts it, and checks the charges an auditor sees:

code/module-09/tests/test_billing_integration.py
    with pytest.raises(RuntimeError):                     # the 25th charge "crashes" its batch
        run_until_idle(service)
    service.producer = billing                            # restarted, without the crash
    run_until_idle(service)
$ python -m pytest module-09/tests/test_billing_integration.py -v
module-09/tests/test_billing_integration.py::test_a_crashed_batch_is_charged_exactly_once PASSED [100%]

============================== 1 passed in 18.85s ==============================

Step 4: Schema compatibility

The tests pass for today's schema. Then two proposals add a field phase_count, first without a default:

--- proposed: phase_count with no default
$ python -m pytest module-09/tests/test_schema_compatibility.py -q --tb=line
F.......                                                                 [100%]
=================================== FAILURES ===================================
E   fastavro._read_common.SchemaResolutionError: No default value for field phase_count in com.brightlane.charging.MeterReading
...
FAILED module-09/tests/test_schema_compatibility.py::test_new_consumers_read_old_readings
1 failed, 7 passed in 0.03s

--- proposed: phase_count with default null
$ python -m pytest module-09/tests/test_schema_compatibility.py -q --tb=line
........                                                                 [100%]
8 passed in 0.03s

Break it: the refactor

Predict first: the notebook applies the one-line refactor to a copy of billing.py and runs the unit tests against it. Which tests fail?

code/module-09/02-testing-kafka-applications.ipynb
before = '"amount_eur": str(amount.quantize(CENT, rounding=ROUND_HALF_UP))}'
after = '"amount_eur": f"{round(stopped[\'energy_kwh\'] * float(price_eur_per_kwh), 2):.2f}"}'
..F.F...F                                                                [100%]
=================================== FAILURES ===================================
E   AssertionError: assert '25.72' == '25.73'
...
E   AssertionError: assert '28.02' == '28.03'
...
E   AssertionError: ses-575bd2
    assert Decimal('28.02') == Decimal('28.03')
...
FAILED module-09/tests/test_charge.py::test_amount_is_energy_times_price_rounded_half_up[52.5-0.49-25.73]
FAILED module-09/tests/test_charge.py::test_amount_is_energy_times_price_rounded_half_up[56.05-0.50-28.03]
FAILED module-09/tests/test_charge.py::test_every_scenario_session_matches_decimal_arithmetic
3 failed, 6 passed in 0.09s

Step 5: The whole suite, as CI runs it

$ python -m pytest module-09/tests -q
21 passed in 18.86s
$ python -m pytest module-09/tests -q -m not integration
20 passed, 1 deselected in 0.18s

What just happened

The unit tests checked the one thing finance cares about, the amount, in about a tenth of a second, including all 20,000 scenario sessions. With the refactor, two of the seven hand-picked cases failed, and the scenario test named the first session at 0.50 €/kWh that came out a cent short. One case, 10.5 kWh at 0.45, passed with floats too: 4.725 happens to be stored a hair above the half. Floats are right by luck, so a test with whole numbers proves nothing about rounding.

The fakes showed the service's order of steps: charges, then offsets, then commit, and an abort with no commit when a write failed. The integration test proved what those steps achieve against a real broker. A crash in the middle of a batch left no charge behind for read_committed readers, and after the restart each of the 40 sessions had exactly one charge.

The schema test rejected the field without a default: a consumer with the new schema can't read yesterday's readings, which lack it. With a default of null, both directions worked. Run that test before the merge, and the mistake never reaches the registry.

Clean up

The integration test removed its container at the end of the run. The notebook's last cell deletes the refactored copy and the proposed schema.

Key takeaways

Next: The production-readiness review