Shared multi-tenant Kafka vs one cluster per team: which model fits your organization
For most organizations, the answer is one shared Kafka cluster per trust and compliance boundary, with tenant isolation enforced by authentication, ACLs and quotas rather than by separate clusters. Separate clusters per team make sense when a team needs a different blast radius, a different compliance scope, or a different upgrade schedule that a shared cluster cannot accommodate. The decision is less about how many teams exist and more about which failures and obligations must be contained. In Apache Kafka 4.x running in KRaft mode, a shared cluster is one set of controllers and brokers serving many principals, each limited by the resources it is allowed to touch. A dedicated cluster is a separate set of controllers and brokers with its own metadata log, its own configuration and its own operating cost.
The isolation primitives on a shared cluster are documented in the Kafka security overview and the authorization documentation. Clients authenticate with SSL or one of the SASL mechanisms, including GSSAPI, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER. In KRaft clusters, the StandardAuthorizer keeps ACLs in the cluster metadata log, and resources without an ACL are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Quotas are a platform control: they limit produce and consume rates and request share per principal, but they do not stop one tenant from affecting shared controllers or shared disks in ways a quota alone cannot prevent. The Kafka security overview is the starting point for confirming which of these controls you will rely on.
The Apache Kafka project is licensed under Apache-2.0 and has no paid edition from the project itself, so the choice between shared and dedicated clusters is not a licensing decision. The cost difference is operational: each additional cluster adds controllers, brokers, monitoring, upgrades, backups and on-call coverage. Before choosing, confirm with your team which tenants share regulated data, which need independent maintenance windows, and whether your infrastructure can support the number of clusters you are considering. A short written list of these requirements will tell you quickly whether a shared cluster with strict ACLs and quotas is enough.
Shared, dedicated and hybrid cluster models compared
This table compares a shared cluster, one dedicated cluster per team and a hybrid of the two across the dimensions that usually decide the model. Read each row against your own tenant count, compliance boundaries and operating headcount. The main takeaway is that isolation and quota controls reduce sharing risk but do not remove the need for separate clusters where blast radius or compliance scope demands it.
| Dimension | Shared cluster | One cluster per team | Hybrid (tiered clusters) |
|---|---|---|---|
| Isolation and blast radius | A broker, controller or misconfigured topic can affect every tenant on the cluster | A failure or bad change stays inside one team's cluster | Shared tier carries low-risk workloads; critical teams get isolated clusters |
| Quota enforcement | Byte-rate and request quotas per principal are required to protect neighbors | Quotas are still useful but matter less because noisy neighbors are limited to one team | Quotas apply inside the shared tier only |
| Operating cost per cluster | One set of brokers, controllers, monitoring and upgrades to fund and staff | Each cluster repeats the fixed cost of brokers, controllers, monitoring and on-call work | Two or more cluster profiles to size, monitor and patch |
| Compliance separation | Separation relies on ACLs, encryption and quotas inside one trust domain | Clear boundary per regulated workload, which simplifies scope and evidence | Regulated data gets its own cluster; other data stays shared |
| Upgrade coordination | One upgrade window affects all tenants, so teams must agree on timing | Each team controls its own upgrade timing but must track releases on every cluster | Shared tier is upgraded centrally; isolated clusters follow their own schedules |
| Cross-cluster sharing with MirrorMaker 2 | Usually not needed; consumers join the same cluster directly | MirrorMaker 2 replicates topics, configuration, consumer group offsets and ACLs between clusters | Replication is used for selected topics only, with clear ownership of each copy |
| Best fit | Many small or internal teams with similar risk and a platform team to run quotas and policies | Few teams with strict isolation, separate regulatory scope or very different throughput profiles | Mixed portfolio where most workloads are low risk and a few need dedicated capacity |
| What to confirm before choosing | Quota and policy tooling you will enforce, and who approves exceptions | Staffing for each cluster's upgrades, monitoring and recovery testing | Criteria for placing a new workload in each tier and how moves between tiers happen |
How isolation works on a shared Kafka cluster
On a shared Kafka cluster, the broker can separate who is allowed to do what, but it cannot separate the hardware underneath. Authentication comes first: clients, brokers and tools authenticate with SSL or SASL, and the documented SASL mechanisms are GSSAPI (Kerberos), PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER. Each team should have its own principal, because authorization decisions are made per principal. In KRaft clusters, the StandardAuthorizer stores ACLs in the cluster metadata log, so every team's permissions live in the same metadata that the controllers replicate. Confirm which authentication mechanism your identity provider can support before you assign principals, since changing it later touches every client.
The rule that matters most during setup is that a resource with no ACL is restricted to super users unless allow.everyone.if.no.acl.found is set to true. Leave that setting at its default and verify it on every cluster. If it is enabled, a topic that a team forgot to lock down becomes readable and writable by anyone with valid credentials, which removes the isolation the rest of the design depends on. Test this with a non-super-user principal against a topic it should not reach, and record the result as part of the onboarding evidence.
Quotas and ACLs limit access and throughput, but they do not give each tenant its own disks, network paths or controller capacity. Partitions from different teams land on the same brokers, share the same log directories and compete for the same network and request handlers. The controllers also hold every tenant's metadata in one log, so a burst of topic or ACL churn from one team slows metadata operations for all. Quotas reduce the damage a single client can do, but they do not remove it. Ask any partner or internal team to show, for your workload mix, how a noisy tenant would be detected and throttled before it affects others.
Six factors that decide the cluster count
These six factors pull in different directions, so the cluster count is a trade between them rather than a single answer. Each one changes how much a failure, a upgrade or a compliance audit touches, and each has a cost that recurs every month.
Blast radius
A shared cluster means one broker failure, bad configuration change or runaway producer can affect every team on it. Ask which teams can tolerate a shared outage, how many brokers and partitions a single incident would touch, and whether critical workloads can sit on their own cluster while lower-risk teams share. Map the largest consumer group and the most sensitive topic before deciding.
Quota limits
Client quotas can cap produce and consume byte rates and request share per principal, which keeps one tenant from starving others on a shared cluster. Quotas do not isolate storage, partition metadata or controller load, so a cluster with many small tenants can still hit limits that quotas do not cover. Confirm which limits your workloads need and test them under peak load.
Operating cost per cluster
Each cluster carries recurring work: monitoring, backups of configuration, access reviews, patching and on-call coverage, and the controller quorum needed for KRaft. Costs also come from idle capacity, since a small team may need the same minimum footprint as a large one. Ask for the platform team's estimate of hours per cluster per month and the infrastructure cost of the smallest supported footprint.
Compliance separation
Some regulations or contracts require that particular data be logically or physically separated, controlled by named administrators and covered by its own evidence. Decide whether separation needs its own cluster, its own encryption keys and authentication, or only separate ACLs and topics. Confirm the requirement with your compliance lead and your assessor, since the answer changes both the design and the audit scope.
Upgrade and maintenance coordination
Kafka supports rolling upgrades from the last three releases to the latest version, but every cluster still needs its own upgrade window, testing and rollback plan. More clusters mean more upgrade cycles, more client compatibility checks and more places where a team may pin an old version. Agree on a shared release calendar and decide who approves each cluster's upgrade before you split the platform.
Cross-cluster sharing
When teams on separate clusters need the same data, MirrorMaker 2 can replicate topics with their configuration, consumer group offsets and ACLs between clusters while preserving partitioning. This adds replication lag, a second set of brokers to run and a failover procedure to test. Ask which datasets cross cluster boundaries, how often they change and who owns the replication job.
Setting up the chosen cluster layout
This sequence turns the cluster decision into a working layout by mapping tenants to clusters, locking down identity and access, setting quota and policy baselines, and proving each control works before any team onboards. Each step ends with a check you can run and record.
Inventory tenants and their data classes
The platform owner lists every team that will produce or consume events, the topics each expects, the data classification of each flow, and any regulatory regime that applies. Security and compliance leads confirm the classes, and business owners confirm which flows must keep running if another flow fails. The step is done when every tenant has a named owner, a data class and a list of cross-team dependencies. This inventory is the input to the boundary decision, so gaps here carry into every later step.
Fix the cluster boundary per group
Using the inventory, the architect assigns each tenant group to a shared cluster or a dedicated cluster and records the reason against isolation needs, blast radius, compliance separation and operating capacity. Dedicated clusters are justified where a regulator, contract or failure domain requires separation, and every cluster needs a named operator. The output is a written cluster map showing which tenants share each cluster and which require a separate deployment. Confirm the map with the security and finance owners before provisioning anything.
Deploy KRaft clusters on a supported release
The platform team installs each cluster in KRaft mode on one of the supported releases, 4.3.1, 4.2.2 or 4.1.2, and records the Java runtime required by the upgrade notes. Confirm the chosen release against the Kafka downloads page at the time of deployment, since support status changes. The check is a healthy controller quorum and broker registration on every cluster, plus a test produce and consume round trip from a single admin client.
Set authentication and the ACL model
The security engineer enables TLS on the client listeners and selects a SASL mechanism that matches the identity system, such as SCRAM-SHA-256 or SCRAM-SHA-512, GSSAPI, or OAUTHBEARER, following the security overview. The StandardAuthorizer is configured so ACLs live in the cluster metadata log, and allow.everyone.if.no.acl.found is left at its restrictive setting unless there is a documented reason to change it. Each tenant receives a principal and prefix-based ACLs. The check is a test principal that can read and write its own prefix, and is denied on another tenant's topic and consumer group.
Define quota and policy baselines
The platform owner sets default client quotas for produce and consume rates, request percentage and connection limits, then records per-principal overrides for tenants with agreed higher needs. Topic creation and configuration rules, such as minimum replication factor, naming patterns and retention limits, are written as a policy using the CreateTopicPolicy and AlterConfigPolicy extension points, and tested on a non-production cluster first. The check is a deliberate over-limit client that gets throttled and a rejected topic that violates the naming rule, both captured in logs.
Configure MirrorMaker 2 for cross-cluster flows
For each flow that must cross clusters, the platform team configures MirrorMaker 2 on Kafka Connect, defining source and target aliases, topic allow lists and replication of consumer group offsets and ACLs as the geo-replication documentation describes. Replicated topics keep the source partitioning, so partition counts and key routing are reviewed on both sides. The step is done when the mirror is running and lag is measured. Replication of ACLs should be scoped with care, since a copied ACL can grant access on the target that the owners did not intend.
Validate ACLs, offsets and replication
Before onboarding teams, the platform owner runs a validation pass: a test principal confirms ACL enforcement on both clusters, a mirrored topic is checked for matching configuration and records, and a consumer group's committed offsets are confirmed on the target cluster after a controlled failover test. Results are written to a sign-off record that names the tester, the date and each failed check with its remediation. Only a clean pass on every cluster in the map moves the layout to production onboarding.
Security and operational risks of shared clusters
A shared Kafka cluster concentrates security and operational risk in one place. The cards below cover the controls most likely to fail as more teams, topics and principals share the same brokers and controllers. Each one names a decision or check to make before the platform grows.
Mixed secured and unsecured listeners
Kafka security is optional, so a cluster can run unsecured or mix secured and unsecured listeners. On a shared cluster, one forgotten plaintext listener exposes every tenant's traffic, regardless of how carefully the other teams configured their clients. Check each listener in the broker configuration, confirm that only TLS-backed listeners are advertised to clients, and test from outside the trusted network that plaintext ports do not answer.
ACL drift across teams
In KRaft clusters, the StandardAuthorizer stores ACLs in the cluster metadata log. Over time, grants added for one project remain after its service is retired, and wildcard or prefix ACLs drift away from the naming convention. Schedule periodic exports of ACLs, compare them with the owners recorded for each topic and group, and remove grants that no longer map to a running principal.
Super user sprawl
Resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Each super user can read and change every tenant's data, so every added super user widens the blast radius of a leaked credential. Keep the list short, assign administrative work to named roles where possible, and confirm that the allow.everyone.if.no.acl.found setting is false in every environment.
Shared control plane
Brokers, KRaft controllers and the admin credentials form one control plane for every tenant. A misconfigured broker setting, a bad partition reassignment or an overloaded controller affects all teams at once. Ask who can change cluster-wide configuration, how changes are reviewed and tested, and which runbook covers a controller outage before the first tenant is onboarded.
Coordinated KRaft upgrades
Every tenant shares the same broker version, so an upgrade is a platform-wide change that needs agreement from all teams. Rolling upgrades are supported from each release in the past year (the last three releases) to the latest version, and brokers require Java 17 since Kafka 4.0. Plan the upgrade window, confirm each tenant's client compatibility, and rehearse the upgrade on a non-production copy of the cluster.
Questions buyers ask before choosing a cluster model
Does running one Kafka cluster per team cost much more than a shared cluster?
It usually costs more, and the gap comes from repeated fixed work rather than from the software, which is free under the Apache License 2.0. Each cluster needs its own brokers, controller capacity, storage headroom, monitoring, certificates, upgrades and on-call coverage. Brokers are often sized for peak load and failure headroom, so many small clusters leave more idle capacity than one larger cluster shared with good quotas. Ask any partner to separate one-time build effort (design, automation, first deployment) from recurring work (upgrades, patching, capacity, access reviews) and to show how each scales with cluster count.
What does a shared Kafka platform require from our internal team?
It requires a small platform group that owns the cluster as a product, with people who can run it day to day. The usual roles are a platform owner who sets tenant rules, engineers who handle broker operations and upgrades, a security owner for TLS, SASL and ACL policy, and a named contact in each team who answers for their topics and quotas. A shared cluster also needs intake rules for new tenants and a change process, because one careless topic or client can affect everyone. Confirm with your own engineers whether that staffing exists before choosing shared operation.
How long does it take to decide on a cluster model and build it?
The duration depends on how many teams, regulatory boundaries and existing systems are involved, and no fixed figure applies to every organization. Decision time is driven by agreeing isolation requirements, data classification and who pays for capacity. Build time is driven by the number of clusters, identity integration, network paths, and whether cross-site replication is needed. Ask for a written scope with phases and deliverables before work starts, since most DeployOpen engagements are fixed-scope and agreed in writing.
Do Kafka quotas alone isolate teams from each other?
No. Quotas limit produce and consume byte rates and request processing share, which protects brokers from a single noisy client, but they do not separate data, control who can read a topic, or stop one team from affecting shared controller and metadata operations. Isolation of data depends on authentication, ACLs that restrict topics and consumer groups by prefix, and separate clusters where compliance or blast radius requires it. Treat quotas as one layer alongside authorization and naming rules, and test them with a deliberately heavy producer before relying on them.
How does MirrorMaker 2 handle ACLs and consumer offsets between clusters?
MirrorMaker 2 is built on Kafka Connect and replicates topics with their configuration, consumer groups with their offsets, and ACLs from one cluster to another, while preserving partitioning. That lets a second cluster inherit the access model, but it does not make the target cluster automatically safe to share. Engineers should confirm how replicated ACLs and principals map across security domains, how offset translation is used during failover, and whether replicated data is subject to the same retention and encryption rules as the source. The Kafka geo-replication documentation describes the configuration options.
Can our own team set up a multi-tenant Kafka platform without outside help?
A capable platform team can, provided it already runs Linux, Java services and TLS, and has time to learn KRaft operations, quotas and authorization in depth. Kafka is open source, so the software itself carries no license cost. The harder parts are the design decisions and the recurring discipline: tenant onboarding, quota tuning, upgrade sequencing across supported 4.x releases and evidence collection for reviews. Outside help is most useful for the design review, the initial build and knowledge transfer, and it can be scoped separately from any ongoing support retainer.
What does a fixed-scope DeployOpen engagement for this decision cover?
A fixed-scope engagement covers the agreed deliverables written down before work starts, such as a tenancy and isolation assessment, a recommended cluster layout, the KRaft deployment design, the authorization and quota model, and the handover documentation. Scope changes, extra clusters or additional sites are handled by written change. DeployOpen plans, implements, hardens and documents self-hosted platforms inside your own infrastructure, but it does not issue audit reports or certifications. DeployOpen does not publish a price list, so a quote follows from the written scope.
Sharing data between clusters with MirrorMaker 2
MirrorMaker 2 is the tool you add when a dedicated-cluster model still needs data to move between clusters. It is built on Kafka Connect and replicates topics along with their configuration, consumer groups with their offsets, and ACLs, while preserving partitioning. That means a team that owns a cluster can receive a copy of a topic from another cluster without the producers changing how they write. Before you commit to separate clusters, list every cross-cluster feed you expect, because each one becomes a replication flow you must deploy, monitor and secure.
Three situations usually justify it: regional copies for disaster recovery or local reads, team-to-team feeds where one team consumes another team's events from a different cluster, and migrations where traffic moves from an old cluster to a new one. Each has a different direction and failure mode. A regional copy is one-way and sized to the full data rate. A team feed may be one-way but filtered to a few topics. A migration is usually temporary and must be cut over with consumer offsets intact. Confirm which pattern you need per flow, and check the replication settings in the geo-replication documentation against your topic naming and ACL design before you copy anything.
The operational cost is the part that tends to be underestimated. Every replication flow consumes broker capacity on both ends, adds lag that consumers on the target must watch, and needs its own credentials, ACLs and TLS configuration. Replicated ACLs also mean that a permission change on the source can change access on the target, so ownership of those rules must be clear. Ask your team to estimate the number of flows, the throughput on each, and who on-call is for a stalled replicator. Those answers, not the broker count, decide whether separate clusters are cheaper to run than one shared platform with quotas.
How DeployOpen scopes and delivers a Kafka cluster layout
DeployOpen works through the cluster count question before any broker is installed, because the answer shapes tenancy controls, upgrade windows and recovery. We plan, build, document and hand over the layout inside your own infrastructure, and we can support it afterward under a retainer.
How we scope
- Map every team, data domain and compliance boundary that would share or separate clusters.
- Record the isolation, quota and recovery requirements each team needs in writing before design starts.
- Agree a fixed scope covering the cluster count, environments, deliverables and timelines in writing.
How we build
- Build the layout on KRaft with planned partition, retention and quota settings for each tenant.
- Configure authentication, ACLs and quotas, and test that one team cannot affect another's topics.
- Set up MirrorMaker 2 replication only where the scoped cross-cluster sharing requires it, and test failover.
How we hand over
- Document the cluster inventory, tenant boundaries, naming rules and the reasons behind each decision.
- Provide runbooks for adding a tenant, moving a team to a dedicated cluster and rotating credentials.
- Walk your platform and security teams through the test results and the evidence they can reuse.
How we support
- Offer ongoing support, upgrades and continuous testing as a retainer after handover.
- Review quota metrics and cluster growth with your team when the tenant mix changes.
- Plan Kafka upgrades within the supported release lines published by the project, scoped separately.
