Module 8 exercises

Each exercise has a start notebook to work in and a solution notebook to compare with afterwards. The notebooks start the lab services they need.

Exercise 1: Least privilege for grid reporting

Goal: write a service's ACLs from what its job does. · Time: ~15 min · Starter: code/module-08/exercise-1/start/

After the incident of lesson 8.2, svc-grid has no ACLs, and the grid report can't run. The job reads charging.telemetry as a member of the consumer group grid-reporting, and writes one record per site and 5-minute window to grid.site-load-5m. Fill grid_acls in the start notebook with the permissions this job needs, and nothing more. The notebook then runs the job as svc-grid, and tries six things grid reporting must not do: write telemetry, read sessions, join billing's group, delete telemetry, create a topic, and change telemetry's retention.

You're done when: the notebook prints least privilege: True. The job writes its site-load records, all six forbidden actions are denied, and no ACL uses ALL or the name *.

Hints

Hint 1 List what the job does, one resource at a time: which topic it reads, which group it joins, which topic it writes. Run the notebook with an empty list first; the job's error names the first missing permission.
Hint 2 Three ACLs are enough. `READ` and `WRITE` include `DESCRIBE`, so the job can see the topics it uses.

Goal: express consumer lag as time, and write an alert the on-call person can act on. · Time: ~20 min · Starter: code/module-08/exercise-2/start/

Billing's alert counts records: more than 500 behind. After the growth plan that's half a second of sessions at peak, and minutes at night. In the start notebook, change seconds_behind into a PromQL expression for how many seconds billing (group m08e2-billing on topic m08e2-sessions) is behind, and set runbook_url. The notebook loads the rule BillingBehind (more than 30 seconds, held for 15 seconds) into Prometheus, checks it stays quiet while billing keeps up, then stops billing and waits for it to fire. Last, replace the runbook cell with the first steps of your runbook.

You're done when: the notebook prints all checks passed: True: quiet while billing keeps up, firing after billing stops with a value in seconds, and a runbook link. Your runbook cell says what to check first and what each answer means.

Hints

Hint 1 KMinion's `kminion_kafka_topic_high_water_mark_sum` grows by one for every record written to a topic. `rate(...[1m])` turns it into records per second.
Hint 2 Records behind ÷ records arriving per second = seconds of sessions billing hasn't handled yet.