Self-service Kafka topic provisioning

Why self-service Kafka topic provisioning needs rules before it needs speed

Self-service provisioning on a shared Kafka cluster means each team can create, change and retire its own topics through a reviewed, automated path, without filing a ticket with a central platform team and without the cluster filling up with topics nobody owns. The reader's usual starting point is one of two failures: teams queue for a central group to create every topic, or topics appear ad hoc because a producer's first write quietly created one with default settings. Both are solved by the same building blocks, and the rules have to come first because speed without them just multiplies the second failure. Kafka's documented authorization model gives you a safe default to build on. In KRaft clusters, the StandardAuthorizer stores ACLs in the cluster metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Confirm that this setting is false (or unset) in your cluster before you open provisioning to teams, because a permissive default undermines every later control. The usual building blocks are disabling broker-side topic auto-creation so that a write to an unknown topic fails, and describing topics and their ACLs as declarative resources that live in version control. The Apache Kafka authorization documentation is the reference for how the authorizer evaluates these rules. This is an open-source platform build, not a vendor product that switches on self-service. Apache Kafka is licensed under Apache-2.0 with no paid edition from the project, so the provisioning workflow, its review rules and its ownership metadata are components your team or a partner designs, runs and maintains inside your own infrastructure. Decide early who approves a request, which partition and retention ranges are allowed without review, and who is accountable when a topic is abandoned. Those decisions determine the design more than the tooling does.

Which provisioning model fits your teams

This table compares four ways to create Kafka topics and users: ticket-based requests, Git pull requests reviewed by platform staff, Kubernetes-native self-service with an admission check, and a portal form that writes Git commits. Read each row against your team count, change volume and audit needs, then check the final row before you accept any default that lets clients create resources on their own.

Model	When it fits	Who approves	Audit trail and effort to run
Ticket-based creation	Small platform with few teams and infrequent topic changes	Platform staff approve each request by hand	Trail lives in the ticket system, often weak on exact settings; low tooling effort but slow and error-prone at volume
Git pull request reviewed by platform staff	Several teams that already work in Git and want reviewed, repeatable changes	Platform reviewers merge after checking partitions, retention and ownership	Strong history in Git commits and pull requests; moderate effort to write review rules and keep manifests current
Kubernetes-native self-service with an admission check	Many teams on Strimzi or similar operators who need fast, policy-checked creation	Admission policy approves automatically within agreed limits; staff handle exceptions	Resource events and policy results are recorded; higher effort to write, test and version the admission rules
Portal form that writes Git commits	Large organizations that want a guided form without teams editing YAML	Form validation plus platform review of the generated pull request	Commits carry the audit trail; highest build effort because the portal, validation and Git integration must be maintained
Auto-created topics or open ACLs (keep off)	Only for disposable development clusters with no shared data	No one approves the creation	Weakest trail; set allow.everyone.if.no.acl.found to false so resources without ACLs stay restricted to super users

Setting up self-service topic creation in order

This sequence turns a shared Kafka cluster into one where teams request topics through reviewed Git changes, with auto-creation off, ownership recorded on every resource, and access enforced by the broker rather than by convention.

Confirm the Kafka version and KRaft mode
Before changing any topic behavior, confirm the cluster runs a supported Apache Kafka release in KRaft mode, not ZooKeeper. The downloads page lists 4.3.1, 4.2.2 and 4.1.2 as supported, and Kafka 4.0 and later support only KRaft. Check the broker version on each node and confirm the controller quorum is healthy with your admin tooling. Done means every broker reports a supported version, no ZooKeeper configuration remains, and the team has written down the exact version they will target. If a cluster is older, confirm the upgrade path in the upgrade notes first, since clusters older than 3.3.x are advised to move to 3.9.x before 4.x.
Disable automatic topic creation on brokers
Set auto.create.topics.enable to false in the broker configuration, then restart or roll the brokers according to your change process. Without this setting, any client that produces to or consumes from a missing topic creates it with broker defaults, which bypasses every review rule you write later. Confirm the change on every broker, not only on one node. Verification: produce a test message to a topic name that does not exist, using a client that has write access, and confirm the broker rejects it and the topic does not appear in the topic list. Keep this test as a repeatable check for later changes.
Define the naming and ownership metadata schema
Write a short schema that every topic and principal must carry: the naming pattern, the owning team, the data classification, the contact channel and the retention rationale. Store these as fields in the KafkaTopic and KafkaUser custom resources, typically as labels and annotations, so reviewers and tooling read them the same way. Agree the schema with platform, security and the first two or three product teams. Done means a reviewer can answer who owns a topic, who to call, and what data it holds, by reading the resource alone. Naming rules belong in the topic naming pages, and this schema should reference them rather than restate them.
Create the Git repository layout for topic resources
Create a repository where each team has a directory, containing KafkaTopic and KafkaUser manifests for the topics and principals it owns. Keep platform defaults and cluster-wide policy in a separate directory that only the platform group can change through code owners. Use one Strimzi-managed cluster reference per environment so the same manifests can be promoted from test to production. Protect the main branch so changes arrive only through reviewed merge requests. Verification: a sample team directory with one topic and one user applies cleanly to a test cluster, and the Strimzi Topic Operator reports the topic as ready. Confirm the exact resource API versions against the Strimzi release you deploy.
Write review rules for partitions, replication and retention
Encode the limits a reviewer cares about as rules that a machine can check: maximum partitions per topic, minimum replication factor, allowed retention and cleanup policies, and required ownership fields. Keep the numbers in a platform-owned file that you set after sizing the cluster, because appropriate values depend on your hardware, traffic and recovery targets. Put a comment in each rule explaining why it exists. Verification: write two deliberately non-compliant test manifests, one with excess partitions and one with a missing owner, and confirm the rules flag both before any cluster is touched. The partition and retention sizing pages cover how to choose the values.
Wire CI checks into merge requests
Add a pipeline stage that runs on every merge request and performs three checks: schema validation of the manifests, the review rules from the previous step, and a server-side dry run against a test cluster where the platform supports it. Make the stage a required status check so no change merges while it fails. Verification: open a merge request containing a failing manifest and confirm the pipeline blocks merging, then open one with a compliant manifest and confirm it passes. Record the pipeline output so auditors and reviewers can see what was checked for each change.
Restrict access so unauthorized producers are refused
Create KafkaUser resources with authorization enabled and grant each principal access only to topics it owns, using the StandardAuthorizer for KRaft clusters. Confirm that resources without ACLs are restricted to super users, and do not set allow.everyone.if.no.acl.found to true. Verification: from a client authenticated as one team's principal, attempt to produce to another team's topic and confirm the broker returns an authorization error, then confirm the owning team can produce successfully. Repeat the test for consume and for a topic with no ACLs. Keep these negative tests in the pipeline or a scheduled job so access regressions are caught.
Verify the full path with test topics
Run one end-to-end test that follows the real workflow: a team opens a merge request for a new topic, checks pass, the change merges, the operator creates the topic, and the owning principal can produce and consume while others are refused. Record the topic metadata, the ACL state and the pipeline run as evidence. Then remove the test topic through a merge request to confirm deletion follows the same path. Done means the sequence works without any manual topic creation on the brokers, and the platform team can show the evidence for each control. Schedule this test after each upgrade of Kafka or Strimzi.

Review rules that stop partition and retention sprawl

Self-service works only when the request form carries rules that a reviewer or a pipeline can check. The cards below cover the parameters that most often grow without limit on a shared cluster, plus the ownership record that makes cleanup possible later.

Partition count caps and justification

Partitions set the parallelism ceiling for a topic, but they also add replication work, open file handles and metadata load on every broker. Require each request to state the expected throughput and consumer count, and cap the partition count at a default that platform staff can raise through review. A reviewer should be able to see why a topic needs 48 partitions instead of 6. Confirm the cap against your brokers' sizing, not a generic number.

Minimum replication factor

A topic created with a replication factor of one loses data when a single broker fails, and self-service requests are the most likely place for that setting to slip through. Set a platform minimum, typically matching the replication factor you use for the cluster's default durability, and reject requests below it. Confirm how your Strimzi or admin tooling applies the minimum to existing topics, since a check at creation time does not change topics created earlier.

Retention limits and storage ceilings

Retention is the setting that quietly consumes disk. Define a maximum retention time and a maximum size per topic, then require teams with larger needs to justify them in the request. Compacted topics need separate rules, because their size depends on key cardinality rather than time. Ask each team to estimate its storage per topic, and check that the sum fits the brokers' disk with headroom for rebalancing.

Naming pattern checks

A naming pattern is only useful if the request form validates it before a topic exists, because topics cannot be renamed. Check that each name starts with the owning team or domain prefix and uses the agreed separators and environment markers. Reject names that embed personal names or ticket numbers. Pair this check with ACLs that limit each team to its own prefix, so a misnamed topic cannot be created through another route.

Required owner and contact labels

A topic without an owner becomes a liability once its producer stops running. Make a named team, a contact channel and a service tier required fields in the topic definition, and store them as labels or metadata in the Strimzi KafkaTopic resource so they travel with the topic in Git. Confirm that the platform can report topics missing these fields, and schedule that report as part of routine review.

Deletion and ownership transfer

Teams reorganize, and topics outlive the people who created them. Define how ownership moves when a team is renamed or merged, who may approve a transfer, and how a topic is retired: first a deprecation period with the consumer group listed, then removal through the same Git change that created it. Until these steps are written down, unused topics will keep consuming partitions and retention budget. Broker-side enforcement of many of these rules is a separate topic, covered in the sibling page on CreateTopicPolicy and AlterConfigPolicy, and the results depend on how you configure it.

What ownership metadata to record and where it lives

Record ownership as part of the same change that creates the topic, not in a spreadsheet kept beside the cluster. For each topic, store the owning team, the service that produces or consumes it, the data classification (the handling level your security policy assigns to the data), an on-call contact, a short retention justification, and a cost center. The first four decide who gets paged and who approves a change, the classification decides which ACLs and encryption expectations apply, and the cost center feeds any later showback report. Put the values that define the resource in the Git manifest, the KafkaTopic resource itself, using metadata.labels and metadata.annotations for owner, service, classification and cost center, and spec.config for retention settings such as retention.ms, so the reason for each retention value sits in the same review as the value. Keep the contact details and escalation paths in the team catalog your organization already runs, and reference the team identifier from the manifest so contact details change in one place. Confirm with your Strimzi version that the topic operator passes labels and annotations through as you expect, because metadata that never reaches the cluster cannot be queried later.

Topic names carry ownership forward whether you intend them to or not, and Kafka topics cannot be renamed. A topic that was named after a team that later split, or after a product that was retired, keeps that name for its whole life, and the only way out is a new topic plus a migration of producers, consumers and data. For that reason, settle the naming pattern before the first topic is approved and make the owner field the authority for who is responsible, never the name. If a team is reorganized, update the owner label and the catalog entry, and leave the topic name alone. Also decide what happens to a topic when its owner leaves: require a named successor before the old contact is removed, and have the review process reject manifests with empty owner or classification fields.

Before you rely on this model, confirm three things with your own environment rather than assuming them. First, check how your chosen tooling reads and reports these fields, because labels are simple to add and easy to drift. Second, decide who may edit the ownership fields, since a team that can change its own classification can also lower it. Third, agree how often the records are reviewed against the live cluster, so a topic that exists in Kafka but not in Git is found and either adopted or removed. Ownership that is checked on a schedule stays accurate, and ownership that is only written at creation time starts to drift within the first reorganization.

Security and operational considerations for self-service access

Self-service only works if every team can change topics within limits that the platform enforces. The cards below cover the controls that keep those limits real: who is allowed to do what, which credentials automation uses, how the Git review path is protected, and how changes are traced afterward.

Least-privilege ACLs per principal

Each team's principal should receive ACLs only for the topics, consumer groups and transactional IDs it owns, such as a prefix it controls. Kafka's StandardAuthorizer stores these ACLs in the cluster metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Check that this setting is false in your cluster, and confirm that a team cannot grant itself access to another team's prefix.

Separate GitOps and team credentials

The automation that applies topic and user manifests should authenticate as its own principal, distinct from the principals that produce and consume data. Kafka supports SASL mechanisms including SCRAM-SHA-512 and OAUTHBEARER, so the automation can use a separate credential with ACLs limited to administrative operations. If that credential is compromised, data-plane access for application teams is not exposed, and the blast radius of a bad commit stays contained.

Protect the repository and review path

In a GitOps model, the Git repository is the control point for who can create topics and which partition counts and retention values are accepted. Require reviews from the platform team for changes to shared settings, restrict who can merge to the main branch, and enforce the same rules in CI before a manifest reaches the cluster. Confirm that branch protection and review requirements are applied to the repository the operator watches, not only to a copy.

Encrypt listeners with TLS

Kafka encrypts traffic between clients, brokers and tools with SSL/TLS, but security is optional, so a cluster can run unsecured or mix secured and unsecured listeners. For self-service access, each provisioned principal should connect only through a secured listener, and the platform should not expose a plaintext listener to tenant networks. Verify listener configuration in every environment, since a single plaintext listener can bypass the access model for everything behind it.

Keep secrets out of manifests

A KafkaUser resource can request credentials, and the resulting secret must not be written into the Git repository, a values file or a pipeline log. Let the operator generate credentials into Kubernetes Secrets, restrict who can read those secrets with RBAC, and deliver them to application workloads through a secret store. Check that the manifests in version control contain names and settings only, and that a scan blocks commits containing key material.

Trace who changed what

Ownership metadata tells you who a topic belongs to, but an audit trail tells you who approved and applied each change. Combine Git history for intent with Kubernetes audit logs for the applied state, and record which principal made each ACL change in the cluster. Ask your team how long these records are kept and who can read them, since answering a question about a retention or access change depends on both sources agreeing.

Questions teams ask before building self-service Kafka topics

What actually drives the cost of a self-service Kafka topic platform?

The main cost drivers are how many identity systems and teams must be connected, how many topic and quota rules the platform has to enforce, and how much Kubernetes operator setup the cluster needs before any team can use it. Identity integration usually takes the most discussion, because each team's group membership must map to Kafka principals and ACLs. Policy design adds cost in proportion to the number of rules, and each rule needs testing against real requests. Costs are one-time for design, build and rollout, and recurring for changes to teams, policies and operator versions. Ask any provider for a quote that separates these components and states which ones are fixed and which are priced per team or per rule.

What does the client need to provide for this work?

You need three kinds of people available. Platform owners decide the naming scheme, partition and retention limits, and which teams may create topics. Identity administrators control the groups that will map to Kafka principals and tell us how group changes flow from your directory. Git process owners define the repository layout, review rules and who approves merges into the desired-state repository. You also need access to a non-production Kafka cluster with Strimzi or an equivalent operator installed, and a named owner for the metadata fields that record team and contact details. Confirm these roles and their time commitment in the written scope before work starts.

Can our internal platform team build self-service Kafka provisioning alone?

Often yes, if the team already runs Kubernetes, understands Kafka ACLs and has someone who owns the Git review process. The hard parts are less about the YAML and more about agreeing on policy: which partition counts and retention periods are allowed, who may request an exception, and how ownership is recorded and audited. Teams without recent Kafka operations experience usually benefit from an outside review of the ACL model and the reconciliation loop before it reaches production. Assess your team against those skills honestly, and ask a partner to scope only the parts you cannot cover in-house, such as an independent review of the design.

How long does a pilot take?

A pilot's length depends on scope rather than on a fixed template. The number of teams onboarded, whether identity integration must be built or only configured, the number of policy rules, and whether the operator and the Git workflow already exist all change the timeline. A pilot for one or two teams on an existing cluster with a working Git process is a much smaller effort than a pilot that also builds group mapping and a new cluster. Ask for the timeline in the written scope, with milestones tied to the teams, environments and approvals your side must supply, rather than a single date.

How do we ask for a fixed-scope quote?

Start by writing down the teams that will use the platform, the kinds of topics they need, the identity source, the Kafka and Kubernetes versions in use, and the policy rules you already know you want. Include the Git hosting platform, the review model and any compliance requirements that touch topic metadata. Share the list of environments and whether production access is needed during the engagement. With that information, a provider can propose deliverables, a timeline and responsibilities in writing. Ask the quote to list what is excluded, such as ongoing support, operator upgrades and additional teams added later.

Does disabling topic auto-creation break existing applications?

It can, if applications rely on Kafka creating topics the first time they produce or consume. Before you turn auto-creation off on a running cluster, inventory the clients, their topic names and their failure behavior. Then move the required topics into declared resources and test each application in a non-production environment. Applications that publish to missing topics will fail loudly once auto-creation is disabled, which is the intended outcome, but teams need warning and a clear path to request topics. Confirm with the Kafka version and configuration you run how the setting is applied, and check the Apache Kafka 4.3 upgrade notes for any changes that affect your version.

Do we need a paid Kafka edition or license to do this?

No. Apache Kafka is released under the Apache License 2.0, and the project itself offers no paid edition. Strimzi and Argo CD are also open source under the Apache License 2.0, so the core stack carries no license fee. Costs come from the infrastructure you run, the people who operate it, and any external services you choose, such as a managed identity provider. Check the licensing of any additional component you add, and confirm with each vendor whether a commercial support plan is offered if your organization requires one.

Where to read the upstream documentation

Before you write a provisioning policy, read the upstream Kafka documentation for the exact release you run. The Kafka security overview explains which client authentication mechanisms exist (SSL, or SASL with GSSAPI, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 or OAUTHBEARER) and that security is optional, so a cluster can carry unsecured listeners next to secured ones. Use it to decide which principal a self-service pipeline or team identity will authenticate as, and to confirm that the listeners the provisioning tooling talks to are the ones you intend to enforce.

The authorization and ACL documentation answers the question that matters most for self-service: who may create, alter and describe topics. In KRaft clusters the StandardAuthorizer stores ACLs in the cluster metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Read it alongside your own configuration to confirm that no topic is open by default, and to check how the authorizer handles prefixed ACLs, which naming-based ownership relies on. Confirm the authorizer class and the super user list in your environment, not from memory.

The Kafka 4.3 upgrade notes cover version-specific behavior, including changes to auto-creation, configuration defaults and tooling that can alter how provisioning requests are handled. Check them for every release you plan to run, and rerun your policy tests after each upgrade. The Kafka downloads page lists the currently supported releases, so confirm which line your platform is on before relying on any note.

How DeployOpen scopes and delivers self-service Kafka provisioning

DeployOpen plans, implements, documents and supports self-service topic provisioning on Kafka clusters you run in your own infrastructure. We start from your team model, naming and ownership rules, and the GitOps tooling you already use, then build the request path, policies and review gates together. Engagements are fixed-scope, with the scope agreed in writing before work begins.

How we scope

Written scope lists the Kafka version, KRaft cluster layout and the teams that will request topics.
We agree which resources are self-service, such as KafkaTopic and KafkaUser objects, and which stay with the platform team.
The scope records partition, retention and replication defaults, review rules and the ownership metadata every topic must carry.

How we build

We disable broker auto-creation of topics and configure the Git repository layout for topic and principal definitions.
We wire Strimzi KafkaTopic and KafkaUser resources into your GitOps pipeline with review rules for partitions and retention.
We test rejected requests, policy failures and ACL grants in a non-production cluster before any team uses the path.

How we hand over

You receive the repository layout, pipeline configuration, review rule documentation and ownership metadata standard.
Runbooks describe how a team requests a topic, how the platform team approves exceptions and how ownership changes are recorded.
Handover sessions walk your platform and security engineers through changes, rollback and the evidence each change leaves behind.

How we support

Ongoing support, Kafka upgrades and continuous testing are available as retainers after handover.
Support can cover Kafka 4.x upgrades, Strimzi version changes and review rule updates as your teams grow.
Confirm with us during scoping which support level, response arrangements and testing cadence your organization needs.