Module 8 solutions

Solution 1: Least privilege for grid reporting

The solution notebook is in code/module-08/exercise-1/solution/.

Approach. One ACL for each thing the job does:

code/module-08/exercise-1/solution/grid-acls.ipynb
grid_acls = [
    allow(ResourceType.TOPIC, TELEMETRY, AclOperation.READ),        # fetch telemetry (includes DESCRIBE)
    allow(ResourceType.GROUP, GROUP, AclOperation.READ),            # join grid-reporting, commit its offsets
    allow(ResourceType.TOPIC, SITE_LOAD, AclOperation.WRITE),       # write the site-load records
]
grid job: read 1932 readings, wrote 50 site-load records
write charging.telemetry       denied: Broker: Topic authorization failed
read charging.sessions         denied: Subscribed topic not available: charging.sessions: Broker: Topic authorization failed
join group billing             denied: FindCoordinator response error: Group authorization failed.
delete charging.telemetry      denied: Broker: Topic authorization failed
create topic grid.scratch      denied: Authorization failed.
change telemetry's retention   denied: Topic authorization failed.
grid job runs: True
forbidden actions all denied: True
no ALL, no wildcard: True
least privilege: True

With an empty list, the job stops at its first step, finding its group's coordinator: FindCoordinator response error: Group authorization failed. Adding the ACLs one at a time leads to the same three as lesson 8.2's billing, minus the transactional id.

Why it works. The broker denies everything no ACL allows (allow.everyone.if.no.acl.found=false), so the list only has to name what the job does. READ on the topic lets it fetch, READ on the group lets it join and commit offsets, and WRITE on the output topic lets it produce. READ and WRITE include DESCRIBE, which the client needs for the topics' metadata. Every action in the audit uses an operation or a resource the list doesn't name, so each is refused.

Literal, not prefixed. A PREFIXED write rule for grid. would also pass this audit. It's fine when a team owns a namespace of topics, but it covers every topic anyone creates under that prefix later. Grant it on purpose, not by default.

The solution notebook is in code/module-08/exercise-2/solution/.

Approach. Divide the records behind by the records arriving per second:

code/module-08/exercise-2/solution/billing-behind-alert.ipynb
seconds_behind = (f'sum(kminion_kafka_consumer_group_topic_lag{{group_id="{GROUP}", topic_name="{TOPIC}"}})'
                  f' / sum(rate(kminion_kafka_topic_high_water_mark_sum{{topic_name="{TOPIC}"}}[1m]))')
runbook_url = "https://runbooks.brightlane.example/kafka/billing-behind"
estimate: 3.6   BillingBehind inactive
...
+  5 s  estimate: 2.8   BillingBehind inactive
+ 10 s  estimate: 7.8   BillingBehind inactive
...
+ 35 s  estimate: 32.8   BillingBehind pending
...
+ 50 s  estimate: 47.8   BillingBehind firing
summary:     Billing is 47.8s behind
runbook_url: https://runbooks.brightlane.example/kafka/billing-behind
quiet while billing keeps up: True
fired after billing stopped:  True (after 50 s)
value is in seconds:          True (48)
runbook link:                 True
all checks passed: True

The start version, which counts records, fails: its value when firing was 818, records, not seconds.

The runbook's first steps.

  1. Is Kafka healthy? The dashboard's top row: all brokers up, one active controller, no offline or under-min-ISR partitions. If not, follow the Kafka runbook first; billing waits for Kafka.
  2. Is billing running? up{job="services"} for the billing workers. 0 means the process is gone or hung: restart it, then read its last log lines for the cause.
  3. Running but slow? billing_consumer_lag per partition. One partition growing: a stuck or poison record, or a hot key. All growing: billing is too slow for the arrival rate; add workers, up to the partition count. A rising billing_rebalances means workers keep leaving the group.
  4. More sessions than usual? Compare the arrival rate with the same hour last week. A burst catches up by itself if billing is faster than the arrival rate.

Why it works. While billing keeps up, it is about 140 records behind (committed offsets trail by up to 5 seconds of auto-commit), which at 40 sessions a second is 3.5 seconds. When billing stops, the lag grows by 40 a second while the arrival rate stays 40, so the estimate grows by one second per second, and it passes 30 at the right moment, whatever the traffic. The same rule works at night and at the growth plan's peak.

Limits of the estimate. It assumes sessions keep arriving at the last minute's rate. If they stop while billing is behind, the rate falls to 0 and the estimate becomes infinite, so the alert fires; with no lag and no arrivals the result is "not a number", and it stays quiet. The worker's own billing_last_event_age_seconds is exact while it runs, but disappears when it stops, which is when the alert matters most. KMinion's numbers don't depend on billing being alive.