Authorization and quotas
Lesson 8.1 made every service log in. Now the broker knows who is asking; this lesson decides what each one may do, and how much of the brokers it may use.
Theory
Example: one bug and one flood
Two incidents in one month. First, grid reporting's clean-up job was meant to delete its own
temporary topics. A bug in its name filter also matched charging.telemetry, and the job deleted
it: three days of meter readings gone, and the gateway failed every write for 20 minutes until
someone recreated the topic. Grid reporting could delete it because, when it asked for "access to
the topics", it got permission for everything on every topic.
Second, a charger at site-V02 received a debug firmware build. It sent 2,000 readings a second,
each with a 320-character debug trace. The gateway forwarded them all, about 900 KB/s. In ten
minutes that one charger wrote more than all 240 chargers write in a day, and nothing stopped it.
Authorization with ACLs
Authorization decides what an authenticated principal may do. A broker plug-in, the
authorizer, checks every request. In KRaft clusters it's the StandardAuthorizer, which keeps
its rules in the metadata log:
authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer
allow.everyone.if.no.acl.found=false
The rules are ACLs (access control lists). One ACL entry, an ACL binding, says:
- who: a principal, such as
User:svc-billing, and optionally the host it connects from; - may or may not:
ALLOWorDENY; - do what: an operation, such as
READ,WRITE,CREATE,DELETE,DESCRIBEorALTER_CONFIGS; - on what: a resource: its type (
TOPIC,GROUP,TRANSACTIONAL_ID,CLUSTER) and its name, matchedLITERAL(exactly this name) orPREFIXED(every name that starts with it).
READ, WRITE, DELETE and ALTER include DESCRIBE, the right to see a resource's metadata.
The broker decides in a fixed order. Super users, listed in super.users, may do everything.
For everyone else, a matching DENY wins over any ALLOW. If nothing matches,
allow.everyone.if.no.acl.found decides; with false, as in the lab, the answer is no. This is
deny by default.
What to notice: with deny by default, you never need a rule that forbids something. You list what each service may do, and everything else is refused.
What each kind of client needs
| Client | ACLs |
|---|---|
| Producer | WRITE on its topics. Idempotence needs nothing more since Kafka 3.0. |
| Transactional producer | the above, plus WRITE on its TRANSACTIONAL_IDs |
| Consumer in a group | READ on its topics, and READ on its group |
| Topic administration | CREATE, DELETE or ALTER_CONFIGS on the topics, or on the cluster |
Kafka Connect and Kafka Streams also need permissions on their internal topics and groups; their
documentation lists them. In Python, create_acls(), describe_acls() and delete_acls() manage
ACLs; the command line has kafka-acls.sh.
Least privilege
Least privilege means each service gets exactly the permissions its job needs. In practice:
- one principal per service, as in lesson 8.1;
- literal topic and group names, so a new topic doesn't silently become readable;
PREFIXEDnames only for a namespace a team really owns, such asbilling-for billing's transactional ids, one per instance;- never
ALLoperations, and never the name*(every resource of that type), outside admin tools.
You don't have to guess the list. Run the service with no ACLs, and each refusal names the missing
permission: Transactional Id authorization failed, Group authorization failed, and so on. The
practical does exactly that with billing.
Client quotas
A client quota limits how much of the brokers one principal may use. Kafka has four kinds:
| Quota | Limits |
|---|---|
producer_byte_rate |
bytes per second written, per broker |
consumer_byte_rate |
bytes per second read, per broker |
request_percentage |
share of the broker's request threads' time |
controller_mutation_rate |
topic and partition creations and deletions per second |
You set them per user, per client id, or as a default for everyone, with kafka-configs.sh. The
Java Admin API can set them too; the Python client can't yet.
A quota doesn't reject anything. The broker measures each principal's rate over a sliding window of
eleven one-second samples (quota.window.num, quota.window.size.seconds). When the rate is above
the quota, the broker works out how long the client must pause to bring the average back down. It
sends that throttle time with its answer, and stops reading from the connection for that long.
The client holds back too, and its records wait in the producer. This is throttling.
What to notice: the quota is an average, not a speed limit per second. It allows a short burst and then long pauses.
Two details bite. The window counts as at least 10 seconds long, even for a client that just started, so a new client's burst passes at full speed for a few seconds. And each broker enforces the quota on its own: a user with 512 KB/s, writing to partitions led by three brokers, can write up to 3 × 512 KB/s.
Also be clear about what a quota on svc-gateway protects. It protects the brokers, and everyone
else using them, from the gateway. It slows every reading the gateway sends, the good chargers'
too. Dropping the runaway charger's readings is the gateway's own job, with a limit per charger.
In production
- Run with
allow.everyone.if.no.acl.found=falsefrom day one; adding it later means finding every client's permissions under pressure. - Keep ACLs as code, reviewed like any change, and applied by a tool (lesson 8.4 does this for topics).
- The broker logs refusals through the
kafka.authorizer.loggerlogger: a first stop when a new service fails to start. - Set a default user quota (
--entity-type users --entity-default) as a safety net, and higher ones for services known to be heavy, such as a replay tool. - Alert on sustained throttling. A throttled producer's records wait; if they wait longer than its
delivery timeout (
message.timeout.ms, 300 seconds by default in the Python client), they fail. - Managed services have their own layer on top: Confluent Cloud has role-based access, Amazon MSK can use IAM policies instead of ACLs.
Practical
Note
Notebook: code/module-08/02-authorization-and-quotas.ipynb
· Lab services: kafka-secure (docker compose --profile secure up -d; the notebook starts it and sets it up as in lesson 8.1 if needed) · Time: ~16 min
What you'll do: give billing its permissions one failure at a time, check what it still can't do, flood the gateway with a runaway charger with and without a quota, and replay the grid incident.
Step 1: Topics and sessions, as admin
secure.ensure_ready() checks that the secure broker runs with its certificate and users, and
returns an admin client. Admin creates charging.sessions, billing.session-charges and
charging.telemetry, and writes 20 charging sessions:
40 session events in charging.sessions
Step 2: Billing, one missing permission at a time
run_billing() does what lesson 7.1's billing service does, once, logged in as svc-billing: read
the sessions, write a charge per finished session, commit charges and position in one transaction.
It returns the charge count, or the first error. The notebook adds four ACLs one by one:
needed = [
("WRITE transactional ids billing-*", allow(BILLING, ResourceType.TRANSACTIONAL_ID, "billing-",
AclOperation.WRITE, ResourcePatternType.PREFIXED)),
("READ group billing", allow(BILLING, ResourceType.GROUP, "billing", AclOperation.READ)),
no ACLs -> Failed to find transaction coordinator: sasl_ssl://127.0.0.1:9095/1: Broker: Transactional Id authorization failed: Transactional Id authorization failed.
+ WRITE transactional ids billing-* -> FindCoordinator response error: Group authorization failed.
+ READ group billing -> Subscribed topic not available: charging.sessions: Broker: Topic authorization failed
+ READ topic charging.sessions -> charges not written: Broker: Topic authorization failed
+ WRITE topic billing.session-charges -> charged 20 sessions
describe_acls() shows them as the broker stores them:
resource name pattern operation permission
---------------- ----------------------- -------- --------- ----------
GROUP billing LITERAL READ ALLOW
TOPIC charging.sessions LITERAL READ ALLOW
TRANSACTIONAL_ID billing- PREFIXED WRITE ALLOW
TOPIC billing.session-charges LITERAL WRITE ALLOW
Step 3: What billing still can't do
write to charging.sessions: Broker: Topic authorization failed
delete charging.sessions: Broker: Topic authorization failed
topics billing can see: ['billing.session-charges', 'charging.sessions']
Step 4: A runaway charger and a quota
runaway_burst() sends 16 seconds of the runaway charger's readings as svc-gateway, 2,000 a
second, and prints every 2 seconds what reached the broker and what waits in the producer. It counts
throttling through the client's throttle_cb:
producer = Producer({**secure.client_config("svc-gateway", secure.USERS["svc-gateway"]),
"linger.ms": 20,
"throttle_cb": lambda event: throttles.append(event.throttle_time)})
Without a quota:
one reading: 471 bytes; 2,000 a second = 920 KB/s
time KB/s delivered waiting in the producer
------- -------------- -----------------------
0-2 s 914 26
2-4 s 926 0
...
14-16 s 921 0
average over 16 s: 919 KB/s
32000 readings; the last ones waited 0.0 s more in the producer
throttled 0 times, longest 0 ms
Then admin sets a quota of 512 KB/s (524,288 bytes) for svc-gateway, and the burst runs again:
kafka_configs("--alter", "--add-config", "producer_byte_rate=524288")
Quota configs for user-principal 'svc-gateway' are producer_byte_rate=524288.0
SCRAM credential configs for user-principal 'svc-gateway' are SCRAM-SHA-512=iterations=8192
time KB/s delivered waiting in the producer
------- -------------- -----------------------
0-2 s 915 22
2-4 s 920 21
4-6 s 851 324
6-8 s 32 4179
8-10 s 22 8094
10-12 s 192 11249
12-14 s 802 11762
14-16 s 935 11687
average over 16 s: 584 KB/s
32000 readings; the last ones waited 10.7 s more in the producer
throttled 50 times, longest 4895 ms
Your numbers differ a little; the shape is the same.
Break it: the grid incident
Grid reporting gets ALL on every topic, the permission from the example. Its admin client then
deletes charging.telemetry. After that ACL is removed, it tries to delete charging.sessions:
everything = allow(GRID, ResourceType.TOPIC, "*", AclOperation.ALL)
with ALL on *: grid deletes charging.telemetry -> deleted
removed 1 ACL of svc-grid
without that ACL: grid deletes charging.sessions -> Broker: Topic authorization failed
What just happened
With no ACLs, billing could do nothing, although it had logged in: deny by default. Each run stopped
at the first thing billing does without permission: register its transactional id, find its group's
coordinator, subscribe to the topic, write the charges. Each error named the missing permission, and
four ACLs were enough. Billing still couldn't write to its input or delete it, and it didn't even see
charging.telemetry: listing topics needs DESCRIBE, which only its READ and WRITE include.
Without a quota the gateway delivered 920 KB/s, everything the charger sent. With the quota, the first 5 seconds still passed at full speed: the broker divides by at least 10 seconds, so the average stayed under 512 KB/s. Then the broker answered with throttle times of up to 4.9 seconds, and deliveries dropped to about 25 KB/s while 11,000 readings piled up in the producer. Nothing failed. Over 16 seconds the average was 584 KB/s; the longer the burst, the closer it gets to 512. The gateway's other chargers would have waited in the same queue.
In the break-it step, one wildcard ACL let a bug in one service delete another team's topic. Without
it, the same call was refused. Exercise 1 gives svc-grid what it really needs.
Clean up
The last cell removes the ACLs, the quota and the topics. The secure broker keeps running for the
exercises; to stop it: docker compose --profile secure stop.
Key takeaways
- The authorizer checks every request: super users pass, a matching
DENYrefuses, a matchingALLOWpermits, and withallow.everyone.if.no.acl.found=falseeverything else is refused. - An ACL binding is principal +
ALLOW/DENY+ operation + resource (type and a literal or prefixed name). - Find a service's least privilege by running it without ACLs: each error names the missing permission.
- A client quota throttles instead of rejecting: the broker averages over about 10 seconds and makes the client pause, so records wait in the producer.
- Quotas are enforced per broker and per principal; they protect the cluster from a client, not a client from its own sources.