32 min · theory + hands-on practical

Authorization and quotas

Lesson 8.1 made every service log in. Now the broker knows who is asking; this lesson decides what each one may do, and how much of the brokers it may use.

Theory

Example: one bug and one flood

Two incidents in one month. First, grid reporting's clean-up job was meant to delete its own temporary topics. A bug in its name filter also matched charging.telemetry, and the job deleted it: three days of meter readings gone, and the gateway failed every write for 20 minutes until someone recreated the topic. Grid reporting could delete it because, when it asked for "access to the topics", it got permission for everything on every topic.

Second, a charger at site-V02 received a debug firmware build. It sent 2,000 readings a second, each with a 320-character debug trace. The gateway forwarded them all, about 900 KB/s. In ten minutes that one charger wrote more than all 240 chargers write in a day, and nothing stopped it.

Authorization with ACLs

Authorization decides what an authenticated principal may do. A broker plug-in, the authorizer, checks every request. In KRaft clusters it's the StandardAuthorizer, which keeps its rules in the metadata log:

code/lab/kafka-secure/server.properties
authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer
allow.everyone.if.no.acl.found=false

The rules are ACLs (access control lists). One ACL entry, an ACL binding, says:

READ, WRITE, DELETE and ALTER include DESCRIBE, the right to see a resource's metadata.

The broker decides in a fixed order. Super users, listed in super.users, may do everything. For everyone else, a matching DENY wins over any ALLOW. If nothing matches, allow.everyone.if.no.acl.found decides; with false, as in the lab, the answer is no. This is deny by default.

A request arrives: svc-billing wants to WRITE to the topic charging.sessions. If the principal is a super user, it is allowed. Otherwise, if any DENY binding matches the principal, operation and resource, it is denied. Otherwise, if an ALLOW binding matches, it is allowed. If nothing matches, it is denied, because the lab sets allow.everyone.if.no.acl.found to false.

What to notice: with deny by default, you never need a rule that forbids something. You list what each service may do, and everything else is refused.

What each kind of client needs

Client ACLs
Producer WRITE on its topics. Idempotence needs nothing more since Kafka 3.0.
Transactional producer the above, plus WRITE on its TRANSACTIONAL_IDs
Consumer in a group READ on its topics, and READ on its group
Topic administration CREATE, DELETE or ALTER_CONFIGS on the topics, or on the cluster

Kafka Connect and Kafka Streams also need permissions on their internal topics and groups; their documentation lists them. In Python, create_acls(), describe_acls() and delete_acls() manage ACLs; the command line has kafka-acls.sh.

Least privilege

Least privilege means each service gets exactly the permissions its job needs. In practice:

You don't have to guess the list. Run the service with no ACLs, and each refusal names the missing permission: Transactional Id authorization failed, Group authorization failed, and so on. The practical does exactly that with billing.

Client quotas

A client quota limits how much of the brokers one principal may use. Kafka has four kinds:

Quota Limits
producer_byte_rate bytes per second written, per broker
consumer_byte_rate bytes per second read, per broker
request_percentage share of the broker's request threads' time
controller_mutation_rate topic and partition creations and deletions per second

You set them per user, per client id, or as a default for everyone, with kafka-configs.sh. The Java Admin API can set them too; the Python client can't yet.

A quota doesn't reject anything. The broker measures each principal's rate over a sliding window of eleven one-second samples (quota.window.num, quota.window.size.seconds). When the rate is above the quota, the broker works out how long the client must pause to bring the average back down. It sends that throttle time with its answer, and stops reading from the connection for that long. The client holds back too, and its records wait in the producer. This is throttling.

The gateway's producer sends 920 KB/s as svc-gateway. The broker answers normally until its average over about 11 one-second windows passes the 512 KB/s quota. Then it answers with a throttle time, for example 1.9 seconds, and stops reading from that connection for that long. The producer holds back too, and records wait in its buffer. After the pause it sends again. Nothing fails, and every broker enforces the quota on its own.

What to notice: the quota is an average, not a speed limit per second. It allows a short burst and then long pauses.

Two details bite. The window counts as at least 10 seconds long, even for a client that just started, so a new client's burst passes at full speed for a few seconds. And each broker enforces the quota on its own: a user with 512 KB/s, writing to partitions led by three brokers, can write up to 3 × 512 KB/s.

Also be clear about what a quota on svc-gateway protects. It protects the brokers, and everyone else using them, from the gateway. It slows every reading the gateway sends, the good chargers' too. Dropping the runaway charger's readings is the gateway's own job, with a limit per charger.

In production

Practical

Note

Notebook: code/module-08/02-authorization-and-quotas.ipynb · Lab services: kafka-secure (docker compose --profile secure up -d; the notebook starts it and sets it up as in lesson 8.1 if needed) · Time: ~16 min

What you'll do: give billing its permissions one failure at a time, check what it still can't do, flood the gateway with a runaway charger with and without a quota, and replay the grid incident.

Step 1: Topics and sessions, as admin

secure.ensure_ready() checks that the secure broker runs with its certificate and users, and returns an admin client. Admin creates charging.sessions, billing.session-charges and charging.telemetry, and writes 20 charging sessions:

40 session events in charging.sessions

Step 2: Billing, one missing permission at a time

run_billing() does what lesson 7.1's billing service does, once, logged in as svc-billing: read the sessions, write a charge per finished session, commit charges and position in one transaction. It returns the charge count, or the first error. The notebook adds four ACLs one by one:

code/module-08/02-authorization-and-quotas.ipynb
needed = [
    ("WRITE transactional ids billing-*", allow(BILLING, ResourceType.TRANSACTIONAL_ID, "billing-",
                                                AclOperation.WRITE, ResourcePatternType.PREFIXED)),
    ("READ group billing", allow(BILLING, ResourceType.GROUP, "billing", AclOperation.READ)),
  no ACLs                                -> Failed to find transaction coordinator: sasl_ssl://127.0.0.1:9095/1: Broker: Transactional Id authorization failed: Transactional Id authorization failed.
+ WRITE transactional ids billing-*      -> FindCoordinator response error: Group authorization failed.
+ READ group billing                     -> Subscribed topic not available: charging.sessions: Broker: Topic authorization failed
+ READ topic charging.sessions           -> charges not written: Broker: Topic authorization failed
+ WRITE topic billing.session-charges    -> charged 20 sessions

describe_acls() shows them as the broker stores them:

resource          name                     pattern   operation  permission
----------------  -----------------------  --------  ---------  ----------
GROUP             billing                  LITERAL   READ       ALLOW
TOPIC             charging.sessions        LITERAL   READ       ALLOW
TRANSACTIONAL_ID  billing-                 PREFIXED  WRITE      ALLOW
TOPIC             billing.session-charges  LITERAL   WRITE      ALLOW

Step 3: What billing still can't do

write to charging.sessions:  Broker: Topic authorization failed
delete charging.sessions:   Broker: Topic authorization failed
topics billing can see:      ['billing.session-charges', 'charging.sessions']

Step 4: A runaway charger and a quota

runaway_burst() sends 16 seconds of the runaway charger's readings as svc-gateway, 2,000 a second, and prints every 2 seconds what reached the broker and what waits in the producer. It counts throttling through the client's throttle_cb:

code/module-08/02-authorization-and-quotas.ipynb
    producer = Producer({**secure.client_config("svc-gateway", secure.USERS["svc-gateway"]),
                         "linger.ms": 20,
                         "throttle_cb": lambda event: throttles.append(event.throttle_time)})

Without a quota:

one reading: 471 bytes; 2,000 a second = 920 KB/s
time     KB/s delivered  waiting in the producer
-------  --------------  -----------------------
0-2 s    914             26
2-4 s    926             0
...
14-16 s  921             0
average over 16 s: 919 KB/s
32000 readings; the last ones waited 0.0 s more in the producer
throttled 0 times, longest 0 ms

Then admin sets a quota of 512 KB/s (524,288 bytes) for svc-gateway, and the burst runs again:

code/module-08/02-authorization-and-quotas.ipynb
kafka_configs("--alter", "--add-config", "producer_byte_rate=524288")
Quota configs for user-principal 'svc-gateway' are producer_byte_rate=524288.0
SCRAM credential configs for user-principal 'svc-gateway' are SCRAM-SHA-512=iterations=8192
time     KB/s delivered  waiting in the producer
-------  --------------  -----------------------
0-2 s    915             22
2-4 s    920             21
4-6 s    851             324
6-8 s    32              4179
8-10 s   22              8094
10-12 s  192             11249
12-14 s  802             11762
14-16 s  935             11687
average over 16 s: 584 KB/s
32000 readings; the last ones waited 10.7 s more in the producer
throttled 50 times, longest 4895 ms

Your numbers differ a little; the shape is the same.

Break it: the grid incident

Grid reporting gets ALL on every topic, the permission from the example. Its admin client then deletes charging.telemetry. After that ACL is removed, it tries to delete charging.sessions:

code/module-08/02-authorization-and-quotas.ipynb
everything = allow(GRID, ResourceType.TOPIC, "*", AclOperation.ALL)
with ALL on *:      grid deletes charging.telemetry -> deleted
removed 1 ACL of svc-grid
without that ACL:   grid deletes charging.sessions  -> Broker: Topic authorization failed

What just happened

With no ACLs, billing could do nothing, although it had logged in: deny by default. Each run stopped at the first thing billing does without permission: register its transactional id, find its group's coordinator, subscribe to the topic, write the charges. Each error named the missing permission, and four ACLs were enough. Billing still couldn't write to its input or delete it, and it didn't even see charging.telemetry: listing topics needs DESCRIBE, which only its READ and WRITE include.

Without a quota the gateway delivered 920 KB/s, everything the charger sent. With the quota, the first 5 seconds still passed at full speed: the broker divides by at least 10 seconds, so the average stayed under 512 KB/s. Then the broker answered with throttle times of up to 4.9 seconds, and deliveries dropped to about 25 KB/s while 11,000 readings piled up in the producer. Nothing failed. Over 16 seconds the average was 584 KB/s; the longer the burst, the closer it gets to 512. The gateway's other chargers would have waited in the same queue.

In the break-it step, one wildcard ACL let a bug in one service delete another team's topic. Without it, the same call was refused. Exercise 1 gives svc-grid what it really needs.

Clean up

The last cell removes the ACLs, the quota and the topics. The secure broker keeps running for the exercises; to stop it: docker compose --profile secure stop.

Key takeaways

Next: Observability: metrics, lag and alerts