Why Kafka ACL design decides who can read your data
A shared Kafka cluster often grows its access rules one request at a time: a team asks for write access to a topic, someone adds an ACL, and months later nobody can say which principal can read or write which topic. This page explains how to build a deny-by-default ACL model with the StandardAuthorizer that scales, using prefixed rules per team or application, deliberate exceptions and a regular review. The model matters because authorization is the layer that decides what an authenticated client may actually do with your data.
In KRaft mode, the StandardAuthorizer stores ACLs in the cluster metadata log, so every broker applies the same rules once they replicate. Kafka's documented default is restrictive: a resource with no matching ACL is limited to super users unless allow.everyone.if.no.acl.found is set to true. Teams that rely on that setting to keep an existing cluster working should treat it as a migration bridge, not a design choice. Authorization is pluggable and can integrate with external authorization services, but the built-in authorizer is what most self-hosted clusters run, so confirm the authorizer class and the setting in your own broker configuration before anything else. The Kafka authorization documentation is the reference for the exact behavior.
You need a deliberate model rather than per-request grants once more than a handful of applications share a cluster, once topics carry data from different sensitivity classes, or once you need to answer an audit question about who can read a topic. Per-request grants create a list that no one owns and no one reviews. A deliberate model names the owners of each topic prefix, decides which principals may create or delete topics, and sets the rules for super users and administrative access. Before you write any rule, inventory your principals, topics and consumer groups, and confirm which authentication mechanism (TLS, SCRAM, Kerberos or OAUTHBEARER) produces the principal names your ACLs will match.
Choosing between prefixed, literal and convenience ACLs
A literal ACL names one exact resource, such as a single topic or consumer group. It is the most precise form because it grants access to one object and nothing else. It fits fixed, well-known resources like a shared audit topic, a control topic that a single service owns, or a consumer group that must keep a stable name. The main cost is volume: every new topic needs its own ACL entry, so literal rules multiply quickly across many teams. Teams that rely on literal ACLs usually automate creation through a provisioning pipeline so that entries are not added by hand.
- Narrowest match: Access applies to exactly one resource name, so a typo grants nothing extra.
- Easy to explain in review: Each entry maps to one principal and one named object.
- High entry count: Large platforms can end up with thousands of entries, which makes drift harder to spot.
- Fragile for new resources: A new topic is unreachable until someone adds its entry, which pushes teams toward workarounds.
- Good default for shared systems: Resources that many teams must not reach are often best handled as literal entries.
Check that every literal entry has an owner and a named ticket or service, and confirm the exact flags for your Kafka version in the authorization documentation.
A prefixed ACL matches every resource whose name starts with a given string, such as payments. for all topics owned by a payments team. It is the usual way to scale least privilege across many teams because one entry covers a family of resources that follow a naming convention. The risk is over-broad matching: a short or careless prefix can grant access to topics that belong to another team, and a prefix that is a parent of another prefix can reach resources you never intended. Naming standards are therefore part of the security design, not only a convention.
- Scales with naming: One entry per team or application covers new topics without a ticket for each one.
- Over-broad risk: A prefix like
appmay matchapp-adminor other teams' resources, so prefixes should end with a delimiter. - Depends on enforced naming: Topic creation policies must block names outside the approved pattern, or the prefix grants nothing predictable.
- Harder to review: Reviewers must reason about every topic a prefix could match, not only the ones that exist today.
- Pairs with ownership records: Keep a register that maps each prefix to an owning team and an application.
Confirm how your Kafka version evaluates prefixed rules, and test each prefix against the real topic list before approving it.
The authorization documentation describes convenience options that bundle common permissions for producers and consumers, so an operator does not have to list each operation by hand. They reduce errors when a client needs the usual set of rights to write to or read from a topic. The convenience is also a source of excess: a bundle may grant more operations than a specific application uses, and the reviewer may not see the full set without checking the documentation. Treat these options as a starting point and then narrow them to the operations each application actually needs.
- Fewer mistakes at setup: A producer or consumer bundle covers the standard operations in one step.
- Hidden breadth: A bundle can include operations, such as topic metadata reads, that the application never calls.
- Consumer group rights: Consumers also need access to their group, so check that the group resource is scoped as tightly as the topic.
- Idempotent writes need care: Producer bundles that enable idempotent or transactional writes require their own resource rules, which should be listed explicitly.
- Review by expanded rights: Record the expanded operation list in the review record, not just the shortcut name.
Verify the exact convenience flags and which operations each one grants for your Kafka version before using them in an approved template.
Most platforms combine the three forms rather than picking one. A practical pattern is to use prefixed entries for each team's own topics and consumer groups, literal entries for shared or sensitive resources, and convenience options only where they map cleanly to what an application does. The decision depends on how strictly your organization enforces topic names, how many teams share the cluster, and how often ACLs are reviewed. Write the rule for each form in your standard so that every new entry follows the same logic.
- Team-owned resources: Use prefixed entries with a delimiter, tied to a naming policy that creation controls enforce.
- Shared or sensitive resources: Use literal entries with named owners and a justification.
- Single-purpose applications: Use literal or narrow prefixed entries, and avoid broad patterns that cover unrelated systems.
- Mixed rules on one resource: Check how overlapping allow and deny rules interact in your version before relying on them.
- Written decision rules: Record which form applies to which case so reviewers can test new requests against the same standard.
Before approving a design, have a second person try to name a topic that a prefix would match but the owner should not reach, and confirm the result on a test cluster.
Designing topic and group naming so prefixes can carry access
Prefixed ACLs only stay manageable when the names they match are predictable and owned. The cards below cover the naming and ownership decisions that determine whether one prefix grants exactly the access a team needs, and what goes wrong when any of them is skipped.
Topic naming convention
Choose one fixed pattern for topic names, such as environment, owning team, domain and purpose, and write it down before the first topic is created. A prefix rule like payments. grants access to every topic that starts with those characters, so a free-form name such as payments_v2 or paymentsarchive can silently fall under a rule written for another team. Check that the separator and character set are unambiguous, and that no team can create a name that another team's prefix already matches.
Overlapping prefix risk
Before granting any prefixed ACL, list every existing topic and group name that the prefix would match, not just the ones the team plans to use. Two prefixes where one is a leading substring of the other, such as orders and orders-eu, can give a team read access to a neighbor's data. Review prefix ACLs for this overlap during onboarding and whenever a new team or domain is added.
Consumer group naming
Consumer groups carry their own ACLs, and a group name that matches a team prefix lets that team commit offsets and join the group. Adopt a convention that encodes the owning application and its environment, for example team-application-env, and keep group names separate from topic names so a group rule cannot be mistaken for a topic rule. Confirm that no two applications can pick the same group name, because shared names let one service consume another service's partitions.
Transactional ID naming
Producers that use transactions need a transactional.id and a matching TRANSACTIONAL_ID ACL. If transactional IDs are not namespaced by team, a producer can take over another producer's transaction and fence it. Give each application a stable transactional ID prefix and grant the matching ACL only to that application's principal. Confirm that the prefix used for transactional IDs does not overlap with the prefixes used for topics or groups.
Ownership of shared topics
A shared topic read by several teams has no single obvious owner, and that gap is where stale ACLs accumulate. Record an accountable owner for each topic in a catalog or in the topic configuration, and make that owner approve any new read or write grant. Without this step, the reader list grows over time and nobody can say which application still needs access. Ask who signs off a new consumer, and how a grant is removed when the consumer is retired.
Dedicated prefixes for platform identities
Connect workers, schema tooling, monitoring agents and administrators need access that is separate from application traffic. Give each platform identity its own namespace, for example an internal topic prefix or a dedicated group prefix, so that a broad application rule cannot reach platform topics. Check that no application team has a prefix that matches the internal namespace, and that platform identities are listed in the access review alongside application principals.
ACL configuration options and when each fits
This table compares the broker and ACL settings that most affect least privilege in a KRaft cluster, with the situations where each fits and the risk of leaving it permissive. Read each row as a decision: keep the restrictive choice unless you have a named reason to relax it, and confirm defaults against the release you run.
| Setting or option | When it fits | Risk of the permissive or default choice |
|---|---|---|
| authorizer.class.name set to StandardAuthorizer | Any KRaft cluster that needs ACL enforcement, so ACLs live in the cluster metadata log | Without an authorizer enforcing ACLs, any authenticated or connecting client may reach topics and groups; confirm the setting for your version |
| allow.everyone.if.no.acl.found set to false | Deny-by-default clusters where every resource has an explicit grant for named principals | Setting it to true lets any principal use a resource that has no ACL, so a newly created topic is open to every client |
| allow.everyone.if.no.acl.found set to true | Temporary migration windows while you inventory which clients use which resources, with a dated plan to remove it | Unlisted resources become readable and writable by anyone who can authenticate, and the gap often persists after the migration ends |
| super.users | Only the platform admin identities and broker identities that must bypass ACL checks | Every listed super user bypasses all ACLs; a shared or long-lived account in this list defeats the rest of the model |
| Literal resource pattern type | Single named topics or groups with a stable, known owner | Exact names are easy to audit but must be granted one by one, which pushes teams toward broad grants to keep up |
| Prefixed resource pattern type | Per-team or per-application namespaces such as a shared topic name prefix owned by one team | A broad prefix such as an empty or very short one matches far more resources than intended; keep prefixes specific |
| Wildcard principal (User:*) | Rarely justified; reserve for narrow, documented cases such as an internal health check with read-only scope | Grants every authenticated principal the permission, which is equivalent to opening the resource to all clients |
| Producer and consumer convenience options on the admin tool | Quick setup of the write or read permission set for one principal, followed by a review of the generated entries | Convenience sets bundle several operations, so they can grant more than the client uses; review the resulting ACLs before release |
| Periodic ACL review against an inventory of principals and owners | Every cluster after the initial build, on a schedule your change process sets | ACLs for departed teams, retired applications and old service accounts remain active and continue to grant access |
Setting up deny-by-default ACLs step by step
This sequence turns on the StandardAuthorizer with deny-by-default behavior, grants access through narrow prefixed ACLs, and confirms with test clients that unlisted principals are refused. The result is a cluster where every permitted operation traces back to a named principal and a reviewed rule.
Confirm authentication and principal names first
Before any ACL is written, the team confirms how each client and broker authenticates, whether by SSL certificate, SASL/SCRAM, or Kerberos, and what principal name each identity produces. The platform owner documents the mapping for every team, including the broker's inter-broker principal. The check is a short table of expected principals that someone has verified by connecting a test client and reading the principal the broker logs for it. Authentication design is covered in the Kafka security overview, and the mapping must be settled before ACLs reference it.
Set the StandardAuthorizer on every broker
The platform team configures the brokers to use the StandardAuthorizer, the authorizer that stores ACLs in the KRaft metadata log. The change goes into the broker configuration and is applied through a controlled rolling restart so that the cluster is not left in a mixed state. The check is that each broker reports the authorizer at startup and that the cluster-level configuration matches across all nodes. The authorization documentation describes the required settings for this authorizer.
Keep allow.everyone.if.no.acl.found disabled
The team confirms that allow.everyone.if.no.acl.found is absent or set to false in every broker's configuration. With this setting off, resources that have no ACL are restricted to super users, so a new topic that nobody has granted access to stays closed. The check is to read the effective broker configuration on each node and confirm the value, then create a test topic with no ACLs and confirm that a non-super-user client cannot produce to it. Any exception that the business needs must be written down with an owner and a review date.
Define super users narrowly
Super users bypass ACL checks, so the list in super.users should contain only the broker identities and a very small set of named administrator accounts. Application teams and automation accounts are not added to it. The check is to list the configured super users, match each one to a documented owner, and confirm that no application principal appears. Keeping this list short is the single largest factor in whether the rest of the model means anything, so the review is recorded with the cluster design.
Create prefixed ACLs for each team
For each team or application, the platform owner creates ACLs that use a prefixed resource pattern matching the team's topic naming convention, such as a shared prefix for all topics that team owns. Each team's producer principal receives write access to its prefix, and its consumer principal receives read access to the prefix plus the consumer group prefix it uses. Grants stay scoped to the minimum operations required. The check is to list ACLs by principal and confirm that every entry maps to a documented team, a prefix, and an operation set, with no wildcard grants to all principals.
Test that unlisted principals are denied
The team runs test clients under principals that have no grants, and under a team principal attempting to write to another team's prefix. A produce from an unlisted principal should fail with an authorization error, as should a consume from a topic outside the principal's prefix. The check is that each denied operation fails for the expected reason and that the broker audit output records the denial. Each test is written down with the principal, the topic, the operation, and the observed result, so the evidence can be reused in later reviews.
Verify allowed operations and record the baseline
The same test clients then confirm that each team's intended producer and consumer operations succeed against their own prefix. This catches grants that are too narrow, which otherwise get widened in a hurry under deadline pressure. The team exports the final ACL listing and the test results and stores them as the baseline for the first access review. The check is that every permitted operation passes, every denied one fails, and the exported ACL list matches the approved design document.
Security risks in Kafka ACL models and how to contain them
Most Kafka authorization failures come from ACLs that were correct when written and drifted afterward. The cards below cover the risks that recur in self-hosted clusters, the control that contains each one, and what to check during a review.
Wildcard ACLs granting broad access
A wildcard resource or principal, such as User:* or a literal * resource name, grants every client access to every matching topic or group. Wildcards often appear while a team is still onboarding and are then never narrowed. Replace them with prefixed ACLs scoped to a team or application name. During review, list every ACL with a wildcard and confirm each one has a documented reason and an owner.
Shared service principals
When several applications connect as one principal, such as User:app-svc, an ACL change for one workload changes access for all of them, and the audit log cannot tell which application read or wrote a topic. Issue one principal per application or team and bind each to its own prefixed ACLs. Verify that no two deployment configs share credentials and that consumer group names are also owned by a single principal.
Excessive super users
Super users bypass ACL checks entirely, so each one is a principal that can read, alter or delete any resource. Keep the super user list short and limited to platform administrators, and use separate accounts for routine work. Because a resource without ACLs is restricted to super users by default in KRaft clusters, the list also determines who can reach resources nobody has granted yet. Confirm the list in the broker configuration matches the approved administrator roster.
Stale ACLs from departed teams
ACLs outlive the applications they were written for. When a team leaves or a service is retired, its topics, consumer groups and principal often keep their grants, and a later team may inherit a name that still carries access. Tie every ACL to a ticket or owner record, and run a periodic review that compares active principals with the ACL list. Remove grants for principals that have no recent authentication activity once the owning team confirms retirement.
ACLs that mask a plaintext listener
An ACL check protects only the listener it applies to. A cluster that still exposes a PLAINTEXT listener alongside a secured one can accept unauthenticated connections, and clients on that port may bypass the identity that the ACLs depend on. Kafka security is optional, so mixed listeners are possible by default. Confirm that every advertised listener uses an authenticated security protocol, and test the plaintext port from outside the broker network to confirm it refuses connections.
Unreviewed allow.everyone.if.no.acl.found
Setting allow.everyone.if.no.acl.found to true turns the default-deny model into default-allow for any resource that has no ACL. The setting is easy to add during troubleshooting and easy to forget. Keep it at its default unless there is a documented migration reason, and record a removal date. During review, check the effective broker configuration on every node, since a value set on one broker still exposes requests routed to it.
Reviewing Kafka ACLs on a schedule without breaking producers
Start the review by exporting the full ACL set from the running cluster and treating that export as the record you audit. In KRaft mode the StandardAuthorizer keeps ACLs in the cluster metadata log, so the export should come from the cluster itself and not from the configuration files your team wrote. Match every principal to a named owner, such as the team that owns the service account or the application that uses it. Any principal you cannot tie to an owner is the first candidate for removal, but confirm with the platform team before revoking, because an orphaned grant may still back a batch job that runs quarterly. Then look for prefixed ACLs that cover more topics or groups than their owner actually uses, since a PREFIXED grant on a short prefix quietly gives one team access to another team's data. Confirm how your environment exports ACLs and whether it keeps an audit log of who changed them, and document those answers in the review record.
Treat every ACL edit as a change with a written request, an approver, a rollback command and a record of the before and after state. Keep the change small enough that one owner can verify it, and avoid bundling a removal with a new grant. Test the change first in a non-production cluster that uses the same listener security, authorizer settings and principal names as production, because a grant that works with a test principal can still fail when the real service account authenticates through a different mechanism. Run the producer and consumer paths the service depends on, including the group and transactional ID permissions that are easy to forget, and check that the application still starts, produces and commits offsets. Only then apply the change in production during a window the owner agrees to. The most common cause of an outage after a tightened ACL is a missing permission on a consumer group or transactional ID, not the topic itself. Confirm with your security and operations teams which review tooling, approval workflow and audit log retention apply in your environment before you set the review cadence.
Limits to confirm before you commit to an ACL model
An ACL model is only as sound as the authorizer, metadata and identities beneath it. The cards below cover the version, external authorization, non-human principals and metadata scale questions to settle before rules are written.
Kafka version and authorizer behavior
Confirm which Kafka release you run and that your cluster uses the StandardAuthorizer on KRaft. Authorizer defaults and handling of resources with no ACL have shaped access decisions across releases, so write your deny-by-default test against the exact version in production. Check the upgrade notes for your target release before you assume the behavior carries over.
Whether external authorization is required
Kafka authorization is pluggable and can integrate with external authorization services. Decide early whether your policy must live in a central engine that auditors or security teams already use, or whether built-in ACLs are enough. If an external engine is required, confirm which authorizer plugin supports it and how it stays in sync with broker configuration.
Connect worker principals
Kafka Connect workers, connectors and their internal topics act as principals that need their own ACLs. Confirm which principal each worker authenticates as, which topics it creates and writes, and whether connector tasks inherit that identity. A shared worker principal can quietly widen access for every connector it runs.
Streams application principals
Kafka Streams applications need read access to inputs, write access to outputs and full control of their application ID prefixed internal topics and state. Confirm how many applications share one principal, how changelog and repartition topics are named, and whether a team can create them without a platform ticket. Shared principals make prefixed rules broader than intended.
Delegation token principals
Delegation tokens carry the principal that created them, so ACLs written for a human or service account also govern the framework workers that use the token. Confirm which principal owns token creation, how long tokens live in your environment, and whether workers in different teams can receive tokens that reach each other's topics.
ACL count and metadata growth
ACLs are stored in the cluster metadata log, so every prefixed or literal rule adds to the state brokers must hold and replicate. Ask your platform team how the cluster behaves as rule counts grow, how often ACLs are listed or changed, and how a large rule set affects controller and broker restarts. Test the expected team count against your own cluster rather than assuming headroom.
Questions teams ask about Kafka ACL design
How much effort does a Kafka ACL model take to design and roll out?
The effort depends mostly on how many teams, topics and applications need access, and on how much of the current access is undocumented. A small cluster with one owner can be scoped quickly, while a shared platform with many teams needs a naming scheme, a request process and a migration plan for existing clients. No duration is supplied for this work, so ask any provider to break the effort into discovery of current clients and ACLs, model design, client-by-client cutover, and recurring review. Each phase should appear as a line item in the written scope before work starts.
What makes the effort go up in an ACL project?
Four factors usually raise the effort:
- The number of teams and applications that each need their own principals and prefixed rules.
- Shared topics and consumer groups that several teams read or write, which need agreed ownership.
- External identity, such as Active Directory, LDAP or an OIDC provider, that must map to Kafka principals.
- Clients that use wildcard access or super user credentials today, because each one must be tested before its access is narrowed.
Each factor adds discovery, testing and coordination work, so ask how the quote handles each one.
Is the ACL model a one-time cost or does it recur?
Designing the model and migrating existing clients is a one-time piece of work, but the model needs ongoing upkeep. New teams, new topics and staff changes all create access requests, and stale grants accumulate if nobody removes them. Ask whether a quote covers only the initial build or also a recurring access review, and who will run that review after handover.
What does the work need from our platform and security teams?
Platform engineers need to provide the current broker configuration, the list of clients and their principals, and a test cluster where changes can be validated before production. Security teams need to define who approves access, how principals map to people or services, and what evidence of review they must retain. Both teams should agree on the ownership of each topic and consumer group before any deny rule is enabled, because unclear ownership is the most common cause of delayed cutover.
Can our own team design and run Kafka ACLs without outside help?
Yes, if the team has someone who understands the StandardAuthorizer, the KRaft metadata log and how the client principals authenticate, and enough time to inventory existing access. Apache Kafka is Apache-2.0 licensed, so there is no license fee for the authorizer itself. The main question is capacity: the inventory, the cutover and the first review cycle are where internal teams usually fall behind. Outside help is worth considering when the same people also need to run the cluster, or when audit evidence must be produced on a set schedule. Confirm the authorization behavior against the Kafka authorization documentation for your Kafka version before you start.
Which skills and inputs should we ask a provider to confirm?
Ask the provider to show how they will inventory existing principals and grants, how they will test each client before a deny rule takes effect, and how they will handle super users and break-glass access. Ask which Kafka version the design targets and whether the upgrade path affects the authorizer settings. Also ask what they need from your team, what they will hand over in writing, and whether a retainer for periodic review is part of the offer. A clear written scope protects both sides when the cluster has undocumented clients.
Does a Kafka ACL design by itself satisfy an audit?
No. An ACL design reduces access, but auditors look for evidence that access is approved, reviewed and removed when it is no longer needed. Only the auditor or certification body can issue an audit report or certification, and DeployOpen prepares the systems, controls and evidence for that assessment. Ask your compliance lead which control expectations apply to your organization before defining the review cadence and retained records.
How DeployOpen scopes and delivers Kafka ACL work
The sequence below moves from a written scope to a documented, tested ACL model running in your own Kafka cluster, with each step closing on a concrete output. It is built so your team can review every principal and grant before it reaches production.
Agree the written scope first
Before any change, DeployOpen and your team agree in writing which clusters, listeners, teams and applications the ACL work covers, what is out of scope, and what the deliverables are. Your team confirms the authentication mechanisms already in use, such as TLS client certificates, SCRAM or Kerberos, because ACLs only match the principals those mechanisms produce. The step is done when the signed scope lists every cluster, deliverable and owner.
Inventory principals, topics and groups
DeployOpen collects the list of principals, topics, consumer groups, transactional IDs and cluster-level operations in use, and records which application or team owns each one. Where a principal has no clear owner, the gap is flagged for your team to resolve before a grant is written for it. The output is an inventory spreadsheet or file that maps each principal to the resources it reads and writes, and each unowned or unused item is listed separately.
Confirm the deny-by-default baseline
DeployOpen checks how the cluster handles resources with no ACL, since the StandardAuthorizer restricts them to super users unless allow.everyone.if.no.acl.found is set to true. Your team confirms that setting, the list of super users, and whether any listener allows anonymous access. The step closes with a recorded baseline showing the current authorizer configuration and any exceptions your team has approved.
Design the prefixed ACL model
DeployOpen drafts the ACL model in your cluster's terms, usually grouping grants by team or application with prefixed resource patterns and separating producer, consumer and administrative permissions. Convenience options for producers and consumers are used where they match the inventory, and wildcard grants are reviewed individually. The output is a design document listing each principal, the resource pattern it receives, the operations it may perform, and the reason for each grant.
Verify with test clients
Before anything is applied to production, DeployOpen runs test clients under each planned principal in a non-production cluster you control. The tests confirm that allowed produce, consume and admin operations succeed and that out-of-scope operations, such as writing to another team's topic or reading an unlisted group, are denied. Any failure leads to a corrected grant and a repeated test. The step closes with a test log showing each expected allow and deny result.
Apply changes in controlled batches
Approved ACLs are applied in batches, starting with the teams whose applications are best understood, and the team checks each application's logs for authorization errors after every batch. Your change owners approve each batch against the design document, and rollback steps are written beforehand for each grant set. The step is complete when every planned principal has its approved grants and no unexpected authorization failures appear in the monitored applications.
Hand over documentation and ownership
DeployOpen delivers the final ACL model, the principal inventory, the test log, the rollback steps and a short runbook for requesting, changing and removing ACLs. Your team receives the list of named owners who approve future access changes and the schedule on which ACL reviews will run. The handover is complete when your team has walked through the runbook and confirmed ownership of every principal.
Review ACLs on a recurring basis
ACLs drift as applications, teams and super user lists change, so reviews are needed after the initial rollout. Your team can keep this in-house, or agree a retainer with DeployOpen for periodic review of the inventory against the design, with findings and proposed changes written up for your approval. The output of each review is a dated list of added, changed and removed grants, with the owner who approved each one.
