Testing Kafka applications
A Kafka application has two kinds of code: what it decides, and how it talks to Kafka. They fail differently and are tested differently. This lesson builds billing's tests in four layers, and replays the bug they would have caught.
Theory
Example: one cent, one session in forty
In spring, a developer tidied up billing's charge calculation: the decimal arithmetic became
round(energy * price, 2). Shorter, and the tests passed: they checked 10 kWh at 0.49 €/kWh, 4.90 €.
Nothing happened for months. At 0.49 €/kWh, only 7 sessions in 20,000 land exactly on half a cent.
Then the motorway sites opened with a 0.50 €/kWh tariff. 52.5 kWh at 0.49 is 25.725 €; 56.05 kWh at
0.50 is 28.025 €. The tariff terms say "rounded half up to the cent", 25.73 and 28.03. But in binary
floating point 25.725 is stored as 25.724999…, and round() makes it 25.72. At 0.50 €/kWh, 1 session
in 40 was charged a cent too little. Finance found it in the monthly reconciliation, not a test.
(Lesson 7.1's billing service uses the same round(): fine for showing transactions, wrong for money.)
Four layers of tests
Each layer catches what the others can't, at a different cost:
- Unit tests check business logic alone: no Kafka, no network, milliseconds each. Here,
charge_for(): its rounding, its edge cases, and all 20,000 scenario sessions compared with exact decimal arithmetic. - Tests with fake clients check the service's steps. A fake is a small stand-in object that behaves like the real client for the test and records what it was asked to do. Did the batch begin a transaction, produce, send the offsets and commit, in that order? Did a failed write abort? Still milliseconds, but a fake only knows what you assumed about Kafka.
- Integration tests run the real flow against a real broker. Testcontainers is a library
that starts a Docker container for a test run, hands the test its address, and removes it
afterwards. Here it runs Kafka 4.3.1, and the test crashes billing in the middle of a batch and
reads the charges as an auditor would. About 20 seconds, and it checks what no fake can:
transactions,
read_committed, the broker's own errors. - Schema compatibility tests check that a proposed schema can still be read by today's consumers, and read today's data, before the change is merged (lesson 5.2).
What to notice: most tests sit in the fast layers. The slow one is there for what only a real broker can show.
Code that can be tested
Tests are easy when the code is shaped for them:
- Keep the decision apart from the Kafka calls.
charge_for()takes a session event and a price and returns a charge; nothing in it knows about Kafka.
def charge_for(stopped, price_eur_per_kwh):
"""The charge for one finished session: energy x price, rounded half up to the cent.
Decimal, not float: 52.5 x 0.49 is 25.725 exactly, and the tariff terms say half up, 25.73.
In binary floating point it's 25.724999..., which round() turns into 25.72.
"""
amount = Decimal(str(stopped["energy_kwh"])) * Decimal(str(price_eur_per_kwh))
- Pass the clients in.
BillingService(consumer, producer, ...)doesn't create its own clients, so a test can hand it fakes, or real clients connected to a container. - Use repeatable data. The scenario's seeded generator gives every run the same sessions, so a failure names a session id you can look at again.
- Test the failure paths on purpose. The integration test wraps the producer in
CrashAfter, which fails on its 25th write, exactly where production would hurt. - Never sleep and hope. Wait for a condition: read until three empty polls, run until idle.
A Kafka container for tests
The container needs two listeners, for the same reason as the lab in lesson 1.2. Tests on your machine reach the broker at a mapped port, chosen before the start so the broker can advertise it. The broker talks to itself, for transactions, on an address inside the container:
env = {"KAFKA_NODE_ID": "1", "KAFKA_PROCESS_ROLES": "broker,controller",
"KAFKA_LISTENERS": "HOST://:9092,BROKER://:19092,CONTROLLER://:9093",
"KAFKA_ADVERTISED_LISTENERS": f"HOST://127.0.0.1:{port},BROKER://localhost:19092",
"KAFKA_INTER_BROKER_LISTENER_NAME": "BROKER",
With one listener, the first version of this fixture hung: the broker tried to reach itself at
127.0.0.1:<mapped port>, which only exists outside the container. The Testcontainers package for
Python has a ready-made KafkaContainer, but it's built for Confluent's images; the course starts
the Apache image with the generic DockerContainer. librdkafka also has a built-in mock cluster
(test.mock.num.brokers), handy for injecting errors, but it only imitates part of Kafka.
In production
- Run the fast tests on every save and every commit; they take under a second.
- Run integration tests in CI on every merge request. CI runners such as GitHub Actions' Linux machines have Docker. Pin the broker image to the version production runs, and run the suite against the next version before upgrading.
- Give each test its own topics, or its own container, so tests can't see each other's records.
- Check that a test can fail: break the code on purpose and watch the test go red. Exercise 2 does this for the dead-letter path.
- Schema checks belong in the pipeline that changes schemas, before the producer that uses them is
deployed. Against a real registry, its compatibility API answers the same question:
POST /compatibility/subjects/<subject>/versions/latest. - Load tests (lesson 2.3) and failure drills (lessons 4.1, 8.4, 9.1) need a full cluster: keep a staging environment for them.
Practical
Note
Tests: code/module-09/tests/ for code/module-09/billing.py
· Notebook: code/module-09/02-testing-kafka-applications.ipynb
· Lab services: none of the lab's; the integration test starts its own Kafka container (needs Docker) · Time: ~14 min
What you'll do: run each layer of billing's tests, try two proposed schema changes, then apply
the refactor that under-charged drivers and watch the tests catch it. You can run every command in a
terminal too, from the code/ folder.
Step 1: Unit tests of the charge
@pytest.mark.parametrize("energy_kwh, price, expected", [
(10.0, "0.49", "4.90"),
(13.25, "0.49", "6.49"), # 6.4925: below the half cent, down
(52.5, "0.49", "25.73"), # 25.725: exactly half a cent, up (the tariff terms)
$ python -m pytest module-09/tests/test_charge.py -v
module-09/tests/test_charge.py::test_amount_is_energy_times_price_rounded_half_up[10.0-0.49-4.90] PASSED [ 11%]
...
module-09/tests/test_charge.py::test_every_scenario_session_matches_decimal_arithmetic PASSED [100%]
============================== 9 passed in 0.11s ===============================
Step 2: The service with fake clients
def test_one_charge_per_finished_session_then_offsets_then_commit():
producer = FakeProducer()
service = BillingService(FakeConsumer(session_events(10.0, 52.5)), producer, "charges", "0.49")
assert service.run_batch() == (2, 0)
assert producer.calls == ["begin", ("produce", "charges", "ses-0"), ("produce", "charges", "ses-1"),
"offsets", "commit"]
$ python -m pytest module-09/tests/test_billing_with_fakes.py -v
module-09/tests/test_billing_with_fakes.py::test_one_charge_per_finished_session_then_offsets_then_commit PASSED [ 33%]
module-09/tests/test_billing_with_fakes.py::test_a_failed_write_aborts_and_rewinds PASSED [ 66%]
module-09/tests/test_billing_with_fakes.py::test_an_unreadable_event_is_parked_and_the_batch_still_commits PASSED [100%]
============================== 3 passed in 0.02s ===============================
Step 3: The transactional flow against a real Kafka
The test writes 40 sessions, lets billing crash on its 25th charge, restarts it, and checks the charges an auditor sees:
with pytest.raises(RuntimeError): # the 25th charge "crashes" its batch
run_until_idle(service)
service.producer = billing # restarted, without the crash
run_until_idle(service)
$ python -m pytest module-09/tests/test_billing_integration.py -v
module-09/tests/test_billing_integration.py::test_a_crashed_batch_is_charged_exactly_once PASSED [100%]
============================== 1 passed in 18.85s ==============================
Step 4: Schema compatibility
The tests pass for today's schema. Then two proposals add a field phase_count, first without a
default:
--- proposed: phase_count with no default
$ python -m pytest module-09/tests/test_schema_compatibility.py -q --tb=line
F....... [100%]
=================================== FAILURES ===================================
E fastavro._read_common.SchemaResolutionError: No default value for field phase_count in com.brightlane.charging.MeterReading
...
FAILED module-09/tests/test_schema_compatibility.py::test_new_consumers_read_old_readings
1 failed, 7 passed in 0.03s
--- proposed: phase_count with default null
$ python -m pytest module-09/tests/test_schema_compatibility.py -q --tb=line
........ [100%]
8 passed in 0.03s
Break it: the refactor
Predict first: the notebook applies the one-line refactor to a copy of billing.py and runs the unit
tests against it. Which tests fail?
before = '"amount_eur": str(amount.quantize(CENT, rounding=ROUND_HALF_UP))}'
after = '"amount_eur": f"{round(stopped[\'energy_kwh\'] * float(price_eur_per_kwh), 2):.2f}"}'
..F.F...F [100%]
=================================== FAILURES ===================================
E AssertionError: assert '25.72' == '25.73'
...
E AssertionError: assert '28.02' == '28.03'
...
E AssertionError: ses-575bd2
assert Decimal('28.02') == Decimal('28.03')
...
FAILED module-09/tests/test_charge.py::test_amount_is_energy_times_price_rounded_half_up[52.5-0.49-25.73]
FAILED module-09/tests/test_charge.py::test_amount_is_energy_times_price_rounded_half_up[56.05-0.50-28.03]
FAILED module-09/tests/test_charge.py::test_every_scenario_session_matches_decimal_arithmetic
3 failed, 6 passed in 0.09s
Step 5: The whole suite, as CI runs it
$ python -m pytest module-09/tests -q
21 passed in 18.86s
$ python -m pytest module-09/tests -q -m not integration
20 passed, 1 deselected in 0.18s
What just happened
The unit tests checked the one thing finance cares about, the amount, in about a tenth of a second, including all 20,000 scenario sessions. With the refactor, two of the seven hand-picked cases failed, and the scenario test named the first session at 0.50 €/kWh that came out a cent short. One case, 10.5 kWh at 0.45, passed with floats too: 4.725 happens to be stored a hair above the half. Floats are right by luck, so a test with whole numbers proves nothing about rounding.
The fakes showed the service's order of steps: charges, then offsets, then commit, and an abort
with no commit when a write failed. The integration test proved what those steps achieve against a
real broker. A crash in the middle of a batch left no charge behind for read_committed readers, and
after the restart each of the 40 sessions had exactly one charge.
The schema test rejected the field without a default: a consumer with the new schema can't read yesterday's readings, which lack it. With a default of null, both directions worked. Run that test before the merge, and the mistake never reaches the registry.
Clean up
The integration test removed its container at the end of the run. The notebook's last cell deletes the refactored copy and the proposed schema.
Key takeaways
- Test in layers: business logic alone, the service's steps with fakes, the real flow against Kafka in a container, schemas before merging.
- Keep decisions apart from Kafka calls, and pass clients in: that's what makes the fast layers possible.
- Testcontainers starts a throwaway Kafka for a test run; give the container a host listener and an internal one.
- Test the failure paths on purpose, and check that a test can fail.
- Money is computed with decimals and an explicit rounding rule, never with
round()on floats.