Why teams enforce Kafka topic rules at the broker
On a shared Kafka cluster, the first topic created by one team often sets a pattern that others copy, and within a year the cluster holds topics named in three styles, with replication factors of one and retention set to whatever the producer defaulted to. Conventions and review tickets rarely stop this on their own, because topics can be created by any principal that holds the right ACLs, and automatic creation can bypass the ticket entirely. The Kafka broker offers two plugin hooks for this: CreateTopicPolicy, which validates a topic when it is created, and AlterConfigPolicy, which validates configuration changes to existing topics. Both run on the broker, so a rejected request never reaches the log.
Reach for these hooks when a rule must hold regardless of who calls the API or which tool they use, such as a minimum replication factor, a required naming prefix, or a ceiling on retention for topics that hold regulated data. Conventions suit guidance that can be waived, and a ticket queue suits decisions that need a person. A broker policy suits a check that is objective, fast and the same for every client. Note that Kafka does not ship these policies as configurable rules you can switch on. You or your platform team must write the Java class, package it as a plugin, and deploy it to every broker, then configure the broker to load it.
Because the policy runs inside the broker, a bug in it affects every producer and administrative tool at once, so treat it like broker code. Confirm the exact interface signatures and configuration keys for your Kafka version against the official documentation before you write anything, and check whether your cluster's management layer already offers equivalent validation. Plan how you will roll a policy out to brokers without an outage, and decide who owns the policy source, its tests and its release cycle.
Which policy hook fits which rule
This table compares the broker-side controls that shape topic configuration: CreateTopicPolicy, AlterConfigPolicy, built-in broker settings and ACLs. Read each row as a rule you want enforced, then check which mechanism covers it at creation, after creation, or only as a default. The main takeaway is that no single hook covers every rule, so most platforms combine a policy with ACLs and broker defaults.
| Rule | Mechanism | What it covers | What to watch for |
|---|---|---|---|
| Naming pattern for new topics | CreateTopicPolicy | Checks the requested topic name before the topic exists, for any client or tool | Only applies to creation. Confirm how names are validated in your own test cluster before relying on it. |
| Minimum replication factor on new topics | CreateTopicPolicy | Reads the requested replication factor and rejects values below your floor | Clients that omit the value get the broker default, so check the default separately. |
| Partition count and retention ceiling on new topics | CreateTopicPolicy | Validates requested partition counts and retention values at creation time | Topics created before the policy existed are not checked. Inventory them. |
| Config changes to existing topics | AlterConfigPolicy | Validates later changes to topic, and where applicable broker, configuration | Policy checks values only. Pair it with ACLs on alter operations to control who can request changes. |
| Fallback values when clients omit settings | Broker defaults such as default replication and auto-creation settings | Sets the value used when a client does not specify one, with no per-team logic | Defaults cannot reject an explicit value. Confirm current key names in the Kafka 4.3 broker configuration reference. |
| Who may create, alter or read topics | ACLs through the StandardAuthorizer | Grants or denies principals operations on resources, stored in the cluster metadata log | ACLs do not inspect values. Resources without ACLs are limited to super users unless allow.everyone.if.no.acl.found is set to true. |
| Safe rollout of a new rule | Staged testing with test principals on a non-production cluster | Shows which requests a policy rejects before production clients are affected | Confirm in the 4.3 documentation how policy classes are loaded and restarted, and plan a rollback path. |
Rules teams commonly enforce with policies
Most shared Kafka clusters enforce a short list of topic rules at creation or alteration time. Each rule below inspects one part of a topic request, fails in a specific way when it is missing, and needs a plan for producers that already create or alter topics today.
Naming pattern checks
A CreateTopicPolicy implementation can reject any topic whose name does not match an agreed pattern, such as a team or domain prefix followed by a product and purpose. Without this check, names drift, ownership becomes guesswork, and ACL prefix rules become hard to write. Before enforcing, inventory existing topic names and decide whether legacy names are grandfathered or renamed, since topics cannot be renamed in place. Confirm which clients create topics outside the policy path.
Minimum replication factor
The policy reads the requested replication factor and rejects topics below a floor, so a single-broker copy never backs production data. Missing this check, a team can create a topic with replication factor 1 and lose its only copy on one broker failure. Coordinate with producers that pass an explicit replication factor, and confirm that the floor fits your broker count and failure domains. Check what happens to existing topics that already fall below the floor.
Partition count ceilings
The policy compares requested partition counts with an agreed maximum, which limits metadata load and per-broker file handles. Without a ceiling, one team can create thousands of partitions on a shared cluster and slow controller operations for everyone. Ask producers and framework templates how many partitions they assume, because a ceiling set too low can break an application that expects to scale out later. Partition counts can increase later but cannot be reduced, so set the ceiling with that asymmetry in mind.
Retention limits
The policy inspects retention.ms and retention.bytes on creation and on AlterConfigPolicy changes, rejecting values above an agreed bound or unlimited retention. Without it, a single topic set to keep data indefinitely can consume disk shared by every tenant. Coordinate with teams that need long retention for replay or audit, and agree on an exception path rather than a silent override. Confirm that the limit matches the storage you have sized for the cluster.
Cleanup policy checks
The policy validates cleanup.policy so that topics meant for event history use delete and topics meant for latest state per key use compact, and it blocks unexpected changes between the two. Missing this check, a compacted topic can be switched to deletion and lose state that consumers rebuild from, or a change log can silently grow without bounds. Ask the owning teams which behavior each topic depends on, and test policy changes against a staging cluster before the rule reaches production.
Building and rolling out a topic policy safely
This sequence takes a CreateTopicPolicy or AlterConfigPolicy class from a compatibility check through unit tests, broker packaging, a non-production trial and a communicated enablement, so each change is proven before it can reject topic requests on a shared cluster.
Confirm Kafka version and Java runtime
The platform engineer confirms which Kafka release each broker runs and that the broker JVM meets that release's requirements. Kafka 4.x brokers require Java 17, according to the Kafka upgrade notes, so the policy class must be compiled against the same Kafka client and server API version the brokers run. Check the build target against the broker version and record both in the change ticket. Done when every broker reports the same version and the build targets that version.
Write and unit test the policy class
A developer implements the policy interface for the rule set agreed with the governance group, such as a naming pattern, a minimum replication factor or a retention ceiling. Each rule should produce a clear error message that tells the requesting team what to change. Unit tests cover one accepted and one rejected request per rule, including the boundary values. Done when the test suite passes locally and the rejected cases return the expected messages.
Package the policy onto every broker
The platform team builds the policy into a jar and places it on the classpath of every broker, using the same artifact and checksum on each node. Mismatched jars across brokers cause inconsistent behavior depending on which broker handles a request, so the team should confirm the file is identical on all nodes after each rolling restart. Done when a check of each broker's classpath shows the same artifact version.
Configure the policy class name
The operator sets the create-topic or alter-config policy class name in each broker's server configuration, using the fully qualified class name of the implementation. Brokers read this setting at startup, so the change is applied through a rolling restart in the same order used for other configuration changes. Confirm the broker logs show the policy class loaded without errors. Done when the broker logs show a successful load on each node and no startup exceptions appear.
Test the policy in a non-production cluster
The team runs the same policy build and configuration on a non-production cluster that mirrors the production security setup, including the authorization model. They send a set of deliberately non-compliant requests through the admin client and expect each to be rejected with the agreed message. They then send compliant requests and expect them to succeed, and they repeat the checks for alter-config changes. Done when a rejected test request returns the policy error and an accepted compliant request creates or updates the topic as expected.
Review existing topics before enforcement
Before the policy takes effect in production, the team lists the current topics and their configurations and compares them against the rules. Topics that already break a rule will not be re-validated by a create policy, but any later alter request on them may be checked, so the owners of those topics need to know in advance. Done when the team has a list of noncompliant existing topics and each owner has agreed on a remediation or exception path.
Enable the policy with a communicated change window
The platform owner announces the change window to all teams that create or alter topics, along with the rule set, the error messages they will see and the contact for exceptions. During the agreed window, the team performs the production rollout using the tested artifact and configuration. Afterward, they run the same rejected and accepted request checks against production with a test topic name reserved for this purpose. Done when the production checks match the non-production results and the teams have confirmed receipt of the change notice.
How to test a policy before it blocks production traffic
Before any policy reaches a production broker, build a test matrix that pairs every rule with requests that must pass and requests that must fail. Each rule needs at least one compliant and one non-compliant case for every request type it can touch. Cover topic creation through the admin client, topic creation requested by provisioning tooling, and configuration changes made through AlterConfigs. For the policy itself, the most important cases are the edge values: a name that sits exactly on the pattern boundary, a replication factor equal to the minimum, and a retention value equal to the limit. Run the matrix against a non-production cluster that uses the same Kafka release and the same policy class as production, because a policy written for one version of the request handling can behave differently on another. Confirm the release you run against in the Kafka downloads page before you build the cluster.
The larger risk is not the requests you wrote down but the callers you forgot. Policies run for every request that creates a topic or changes its configuration, including requests that the platform makes on its own. Kafka Connect creates internal topics for offsets, configs and status when a worker starts, and Kafka Streams creates changelog and repartition topics for stateful applications. If those internal names or replication settings fall outside your rules, a Connect worker or Streams application can fail on startup while the policy appears to work for human users. Inventory these callers before rollout by reading the topic creation and configuration audit trail for a representative period, listing the client IDs and principals involved, and checking each Connect worker's internal topic settings and each Streams application's configuration. Ask every team that owns automation to name the topics their tools create.
Roll the policy out in stages rather than as a single change. The safest first step is a period of observation: run the policy logic in a mode that logs violations and allows the request, review those logs with the owners of each client, and fix the callers or the rule before enforcement begins. Keep a documented rollback that removes the policy class from the broker configuration and performs a rolling restart, and rehearse it in the test cluster. For each caller, write down the expected outcome, so that a failure in production can be checked against a known result. Plan these steps with your change board, and confirm with your platform team how policy configuration reaches each broker in your environment before the first change window.
Security and operational considerations for policy plugins
A policy plugin runs inside the broker, so its code, its dependencies and its failure behavior affect the whole cluster. The cards below cover the risks to design for before a policy reaches production, from code review to audit trails.
Policies run inside the broker
A CreateTopicPolicy or AlterConfigPolicy class is loaded into the broker JVM and executes on the request path for every matching topic operation. Any library it pulls in shares that process and its classpath, so a dependency upgrade can change broker behavior. Treat the policy as broker code: pin its dependencies, review changes like any other broker change, and confirm the build artifact you deploy is the one you tested. Ask your Kafka distribution or platform team how plugin JARs are delivered to brokers in your environment.
Policy errors can block topic creation
If a policy throws an exception or responds slowly, the request that triggered it fails, and automated pipelines that create topics will stop with it. Keep the policy logic small and deterministic, avoid calls to external services on the request path, and make sure exception messages explain the rule that failed so teams can fix their request. Test failure paths explicitly, including malformed input and missing configuration values, before a rollout reaches shared environments.
Policies do not replace authorization
A policy checks the shape of a request, such as a name pattern or a replication factor, but it does not decide who may make the request. Access still comes from ACLs. In KRaft clusters the StandardAuthorizer stores ACLs in the metadata log, and resources without a matching ACL are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Confirm that the principals who create topics hold the expected ACLs and that no broad grant bypasses the naming rules your policy enforces.
Review policy changes like access changes
A change to the policy class, its configuration or the broker settings that load it can silently relax or tighten rules across every team. Keep the policy source, its build and its broker configuration under version control, require review for each change, and record which version runs on which brokers. Ask who approves a change and how a rollback is performed. Do not let policy edits go through the same informal channel as ordinary application config.
Audit rejected requests
Rejected topic and config requests are the main signal that teams are hitting the rules, and that signal disappears if rejections are not logged in a reviewable form. Decide which fields the policy logs for each denial, such as principal, topic name and rule, and where those logs go. Route them to your central logging system with retention that matches your internal audit needs. Confirm with your security team which events they need to correlate with authorization and configuration change records.
Test policies before enforcement
A policy that rejects valid requests is an outage for the teams that depend on it. Run the policy against a staging cluster with representative requests from automation and from manual creation, including requests that should pass. Introduce the rule with a period of logged but unenforced results where your design allows it, then enforce it. Record which test requests were used so the evidence can be reproduced when the policy changes later.
Questions teams ask before writing topic policies
How much effort does a topic policy take to write and roll out?
The effort depends on how many rules you enforce, how many clients and tools create or alter topics today, and how much of that behavior you need to keep working during the change. A policy class with a few checks is a small Java task. The larger part is inventory: finding every producer, admin script, Connect worker, Strimzi resource or GitOps pipeline that creates topics, then testing each one against the new rules. Ask any partner to split its estimate into policy code, test harness, inventory, rollout across environments and handover, so you can see which part is driving the total.
Who does the work: our team, a partner, or both?
Most teams split the work. Platform engineers own the broker configuration, the policy deployment and the rollback path, because the policy runs inside their brokers. Application teams own the inventory of their topic-creating tools and fix their scripts. A partner can write and test the policy classes, build the inventory and test suite, and document the rollout, under a fixed scope agreed in writing before work starts. Decide upfront who signs off on each rule, since that decision matters more than the code.
Does a topic policy replace naming conventions or self-service provisioning?
No. A policy is an enforcement point, not a process. It rejects topic creation or configuration changes that break rules you have already decided, such as a name pattern or a minimum replication factor. Naming conventions still need to be designed, documented and agreed with teams, and self-service provisioning still needs a way to request topics, review them and record ownership. The policy makes those decisions binding at the broker, so they cannot be skipped by a script that talks to the cluster directly.
What Java skills does the team need to maintain a policy?
The policy classes are small Java classes that implement the Kafka admin policy interfaces and are packaged into a jar placed on each broker's classpath. Your team needs someone comfortable with Java build tooling, unit testing, and reading Kafka's admin API and configuration keys. They do not need to modify Kafka itself. Plan for the build pipeline to produce a versioned artifact tied to the Kafka release you run, because a broker upgrade can change the surrounding APIs and must be tested with the policy loaded.
Can our own team build and run this without outside help?
Yes, if you have Java and Kafka operations skills and enough time to test. Many teams write a first policy in-house. The usual gaps are the inventory of topic-creating clients, a test environment that mirrors production behavior, and a rollout plan that avoids blocking a deployment at a bad moment. Decide whether your team will own the policy after handover, and who will review changes to it. An outside partner adds most value on the inventory and rollout work, where an experienced reviewer can spot missed paths.
What should a quote for topic policy work include?
A quote should follow from a written scope that lists the rules to enforce, the Kafka version and deployment type, the number of environments, and the topic-creating tools in use. Ask for deliverables by name: policy source and build, unit and integration tests, an inventory of affected clients, a staged rollout plan, a rollback procedure and documentation for operators. Check what is one-time (design, build, first rollout) and what recurs (rule changes, Kafka upgrades that require retesting). Ask how changes after sign-off are priced.
What goes wrong when a policy rolls out without testing?
The usual failure is a policy that blocks legitimate work. A deployment script that creates topics with a default retention value, or a Connect connector that writes to a topic with a non-matching name, can start failing the moment the policy loads. Policies also apply at different points: create-time checks do not catch every config change unless the alter policy is also in place. Test the policy in a non-production cluster with the real tools, log rejections in a warning mode where your Kafka version supports it, and keep a documented way to remove the policy quickly.
How DeployOpen scopes and delivers topic governance
DeployOpen approaches broker-side topic rules as a written, testable part of a self-hosted Kafka platform in your own infrastructure. We start from the naming, replication, retention and ownership rules your teams already agree on, then encode them as CreateTopicPolicy and AlterConfigPolicy classes, test them against your cluster, and hand over the code and runbooks with the platform.
How we scope
- The written scope lists each rule to enforce, such as name patterns, minimum replication factor and retention ceilings.
- We record which requests are blocked, which are only reported, and which exceptions need an owner's approval.
- The scope names the Kafka version, the cluster topology and the identity provider that policy decisions depend on.
- Engagements are fixed-scope, so deliverables and timelines are agreed in writing before the build starts.
How we build
- We implement the policy classes in Java against the Kafka version your cluster runs, and configure them on the brokers.
- Each rule gets unit tests and a broker-level test run that shows which topic requests pass and which fail.
- We confirm how auto-created topics and admin client calls interact with the policies before any rule reaches production.
- Rollout steps run first in a non-production cluster, with a documented path for tightening rules later.
How we hand over
- You receive the policy source, build artifacts, broker configuration and the test suite, all kept in your own repository.
- Runbooks explain how to change a rule, roll out a new broker version and read rejection messages in logs.
- Documentation records which rules sit in policy code and which are enforced by ACLs, quotas or provisioning workflows.
- A handover session walks your platform team through the tests, the rollout and the rollback steps.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer, covering policy changes after Kafka upgrades.
- Retainer work can include re-running the policy test suite against each new supported Kafka release before you upgrade.
- Ask any provider to confirm, in writing, how policy changes are reviewed, tested and released into your environment.
