Why Apache Kafka is the event backbone for SaaS teams running their own infrastructure
Apache Kafka fits a SaaS platform that must run in infrastructure the company controls, because it gives every product, analytics and audit consumer the same ordered event stream while the data stays in your cloud accounts or data centers. The engineering leader's problem is usually familiar: many tenants generate events that drive product features and analytics, and enterprise customers ask in security questionnaires where that data lives and who can reach it. Self-hosting Kafka answers the location question, but it moves responsibility for tenant isolation, quotas and evidence onto your team. Kafka is an Apache Software Foundation project released under the Apache License 2.0 (SPDX: Apache-2.0), and the project itself offers no paid edition. Licensing cost is therefore zero, and the recurring cost is the people, hardware or cloud capacity, and support contracts you choose to buy. Kafka 4.x runs only in KRaft mode, so there is no ZooKeeper ensemble to design or secure. Confirm with your team whether any existing cluster still depends on ZooKeeper, because the upgrade notes require migrating those clusters to KRaft before moving to 4.x. The Kafka downloads page lists the currently supported releases and is the right place to check the version you plan to standardize on. What changes for a SaaS company is the work around the broker. Tenant-aware topic naming, per-customer quotas, change data capture into analytics and retained access records all have to be designed deliberately, because Kafka supplies the mechanisms but not your tenancy model. Plan these decisions before the first customer onboards, since renaming topics and rebuilding ACLs later is expensive. Ask your platform team to write down which controls each one maps to, so that when a SOC 2 examination begins, the evidence already exists.
Requirements SaaS buyers and auditors place on the event platform
This table maps each requirement that SaaS buyers, auditors and regulators place on a shared event platform to the source fact behind it and what it means for topic design, access control or contracts. Kafka supplies mechanisms such as ACLs, quotas and encryption, but the evidence and contract obligations sit with the operator. Read each row as a design input to settle before the platform is built.
| Requirement | Source fact | What it means for Kafka design |
|---|---|---|
| Tenant isolation | Kafka ACLs in KRaft clusters are stored in the metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. | Default-deny ACLs per tenant prefix or principal. Confirm that no resource falls back to open access and that super user membership is tightly held. |
| Per-customer quotas | Kafka supports quotas as a broker feature, but the supplied facts do not specify quota types or limits; confirm exact quota options against the 4.3 documentation. | Set produce, fetch and request quotas per client or tenant so one customer cannot starve others. Document the quota model as part of the tenant contract. |
| SOC 2 access and change control (CC6 to CC9) | Security common criteria CC6 to CC9 cover logical access, system operations, change management and risk mitigation for every SOC 2 examination. | Kafka ACL changes, topic configuration changes and broker upgrades need recorded approvals and logs. The platform team, not Kafka, must produce this evidence. |
| GDPR Article 28(3) processor terms | Processor contracts must bind the processor to process personal data only on documented instructions, including transfers, and to meet Article 32 security obligations. | If the platform carries customer personal data, the processing terms must cover topics, retention, replicas, backups and any cross-region mirroring of those topics. |
| EU Data Act switching rights | The Data Act has applied since 12 September 2025 and requires providers to allow switching to another provider or on-premises infrastructure, with a two-month maximum notice period and a 30-day transitional period. | Keep topic schemas, retention and export paths documented so customers can move data. Confirm with counsel whether the Data Act applies to your service model. |
| No switching charges from 12 January 2027 | Providers may not impose switching charges from 12 January 2027; until then, reduced charges may not exceed costs directly linked to switching. | Price any data export or migration support in a way that fits this rule. Review the commercial model before that date. |
| NIS2 treatment of cloud and managed providers | NIS2 Annex I treats cloud computing and managed service providers as digital infrastructure and ICT service management, both of high criticality. Commission Regulation (EU) 2024/2690 sets technical requirements for these providers. | If you operate a multi-tenant platform for customers in scope, expect NIS2 measures to reach the event platform. Confirm your classification with counsel before scoping controls. |
Which Kafka settings and modules matter for multi-tenant SaaS
A multi-tenant Kafka platform stands or falls on a few early choices: the release line, how clients authenticate, how traffic is encrypted, how access is granted by default, how topics are named and partitioned, and how data is replicated to other regions. Each card below covers one of these decisions and what to check before the platform carries customer data.
KRaft cluster and release choice
Build the cluster in KRaft mode from the start, since Kafka 4.0 supports only KRaft and removed ZooKeeper mode. Choose from the supported lines 4.3.1, 4.2.2 and 4.1.2, and confirm the current support window on the downloads page before fixing a version. Plan for rolling upgrades, which the release plan supports from the last three releases to the latest version, and record the Java 17 requirement for brokers before you size hosts.
SASL mechanisms per client type
Pick SASL mechanisms by the kind of client rather than one mechanism for all. SCRAM-SHA-512 suits service accounts that each tenant's application uses, OAUTHBEARER lets you tie tokens to the identity provider your customers' workforce and services already use, and GSSAPI fits only where a Kerberos realm already exists. Check that each mechanism is configured on every client path, including Kafka Connect and tooling.
TLS on every listener
Encrypt traffic between clients, brokers and tools on every listener. Kafka's security is optional, so a cluster can run with unsecured listeners or mix secured and unsecured ones, and a single plaintext listener exposes tenant payloads and credentials. Audit the listener configuration on each broker, including internal and controller listeners, and verify it after every change rather than assuming the default.
StandardAuthorizer ACLs with a closed default
Use the StandardAuthorizer, which stores ACLs in the cluster metadata log, and keep allow.everyone.if.no.acl.found set to false. With that setting, a resource without an ACL is open only to super users, so a missing rule fails closed. Grant each tenant's principal access to its own topics and consumer groups only, and review super user membership as part of access reviews, since those reviews are often the evidence auditors ask for.
Tenant topic naming and partitioning
Set a topic naming convention that encodes tenant, environment and data class, such as a fixed prefix per tenant, so that ACL patterns and quota rules can match it. Decide up front whether each tenant gets dedicated topics or shares topics with a tenant key, because that choice drives partition counts, ordering guarantees and the number of ACL entries. Document the partition plan per topic so growth does not force an uncontrolled rebalance later.
MirrorMaker 2 for regional replication
Use MirrorMaker 2 to replicate topics with their configuration, consumer groups with offsets and ACLs to a second cluster. Because it preserves partitioning, consumers can resume on the target cluster from replicated offsets after a failover. Test failover and failback as scripted runbooks, and confirm which tenant data may leave a region under your contracts before enabling replication. Quotas and change data capture connectors are covered on the sibling pages linked from this section.
How tenant isolation changes topic, quota and ACL design
The first design decision for a multi-tenant Kafka platform is whether tenants share topics or own them. A shared topic keyed by tenant keeps the partition count and the number of topics small, which suits platforms with many small tenants. Each tenant's records are distinguished by a key or header, so every consumer must filter or be scoped by convention, and access control cannot be enforced per record by Kafka ACLs alone. A per-tenant topic gives each customer its own resource for ACLs, retention, quotas and offset tracking, but the partition total grows with tenants times topics per tenant, and the cluster's metadata and partition load grow with it. Consumer groups follow the same choice: shared topics often mean one consumer group per service reading all tenants, while per-tenant topics mean one group per tenant or service pair. Confirm the partition ceiling your chosen KRaft cluster size and hardware can carry before committing to the per-tenant path.
Noisy-neighbor risk comes from one tenant's producers or consumers taking more broker bandwidth, request handling or disk than their share. Kafka supports client quotas for producer and consumer byte rates, request rates, and per-user, per-client-ID or combined principals, but a quota limits throughput, not the partitions a tenant lands on. A large tenant writing to a few partitions can still saturate the brokers that lead them, so quotas must be planned with the partition layout: the number of partitions per tenant, the key strategy that spreads load, and the brokers that host the leaders. Test the quota values against the largest expected tenant before onboarding customers, and record which quota entity (user, client ID or both) each tenant's service uses, because a quota only applies to the principal or client ID it is bound to.
ACLs map tenants to principals in the StandardAuthorizer, which stores ACLs in the cluster metadata log. The usual pattern gives each tenant or tenant-facing service its own authentication identity, such as a SCRAM-SHA-512 user or an OAUTHBEARER subject, and grants that principal READ, WRITE or DESCRIBE on prefixed topic and consumer group names. Prefixed ACLs keep grant counts manageable, but a shared-topic design cannot isolate tenants with topic ACLs at all, so it needs application-level enforcement and clear ownership of the producer identity. Keep allow.everyone.if.no.acl.found set to false so that new topics are not open by default, and verify ACL behavior with a test tenant before you rely on it. Whether your SOC 2 evidence depends on this design is covered in the compliance section.
Rolling out a multi-tenant Kafka platform in phases
This sequence moves a SaaS team from tenant discovery to a tested, self-hosted Kafka platform that onboards new customers through a standard process. Each phase closes with a written output that the next phase depends on. The durations of each phase depend on scope and are set in the written plan before work starts.
Discover tenant counts and data classes
DeployOpen works with your product, data and security leads to record how many tenants you serve now, how fast that number is growing, which event types each product area produces, and which data classes those events carry, such as personal data, customer content or billing records. The output is an inventory of tenants, topics and data classes with an owner for each. You know this step is done when the inventory has been reviewed and signed off by the product and security owners.
Design the cluster and identity model
The team chooses the cluster layout, including whether tenants share one cluster or are grouped by data class, the KRaft controller and broker counts, and the regions or data centers that will hold the cluster. It also decides how clients authenticate, whether through SASL mechanisms such as SCRAM or OAUTHBEARER or through mutual TLS, and how identities map to tenants. The output is an architecture decision record. This step is done when the record is approved and each open decision has a named owner.
Set the topic and ACL standard
DeployOpen drafts the naming convention, partition and retention defaults, quota classes and the authorization pattern, using the StandardAuthorizer model in which ACLs live in the cluster metadata log and resources without ACLs are denied to non-super users by default. The standard states who may create topics, how each tenant's principals are scoped and how exceptions are requested. The output is a written standard and a set of reviewed templates. This step is done when the standard is accepted by platform, security and product leads.
Pilot the standard with one tenant
The team onboards one representative tenant using the standard exactly as written, including its topics, principals, ACLs and quotas. The pilot exposes gaps in the templates, the provisioning steps and the tenant-facing documentation. Your team records each change and updates the standard before wider rollout. The pilot is complete when the tenant produces and consumes its events under its own credentials and no ACL exception was needed outside the documented process.
Build change data capture into analytics
Where analytics needs data from operational databases, DeployOpen configures change data capture into a Kafka Connect pipeline and into your analytics store. The work covers connector configuration, schema handling, the topics that carry changes, and the access rules that limit which analysts can read which tenant data. The output is a working pipeline with documented lag and failure behavior. This step is done when changes flow end to end and a test deletion or update appears correctly in the analytics store.
Test load and failure scenarios
The team runs load tests that reflect the tenant mix from discovery, then injects broker, disk and network failures to confirm that replication, leadership changes and client retries behave as designed. Quotas are exercised so that one noisy tenant cannot starve others. Results are recorded against the targets set in the design. The output is a test report with findings and fixes. This step is done when the platform meets the agreed targets and the open findings are either closed or accepted in writing.
Onboard tenants and hand over runbooks
DeployOpen delivers operational runbooks for provisioning tenants, rotating credentials, adjusting quotas, moving partitions, handling broker removal and responding to alerts, along with the tenant onboarding guide. Your operations team runs the first onboardings with DeployOpen support available. Upgrades follow the rolling path from the last three releases to the latest version, as set in the project's time-based release plan, and each upgrade is scheduled in its own written plan. The step is done when your team has completed a tenant onboarding without assistance and the runbooks have been reviewed.
Change data capture into analytics: options and trade-offs
A source connector reads the source database's transaction log and publishes each row change to a topic without changes to application code. The connector is configured in Kafka Connect, so the work sits in the source database, the Connect workers and the topics. Confirm the license, maintenance status and supported database versions of any connector before you choose it, and see the Apache Kafka integrations pages on this site for connector detail.
- Effort drivers: Setup depends on enabling log-based replication on the source database, creating a replication user with the privileges it needs and sizing the Connect workers. Effort grows with the number of source databases and tables.
- Consistency: Log-based capture follows commit order for each table, but initial snapshots and restarts can produce duplicate events, so downstream tables must handle repeats by key.
- Schema changes: Column additions and renames flow into events as the source emits them. Set a compatibility rule in the schema registry and test how the sink handles changed columns before a migration runs.
- Operations: The team runs the workers, watches connector status and lag, rotates credentials and manages the source's log retention so that unread changes do not fill the database.
Before committing, check how the connector handles deletes, large transactions and failover of the source database.
The application writes an event row to an outbox table in the same database transaction as its business change. A relay, often a CDC connector reading that table, then publishes the event to Kafka. Because the business write and the event commit together, the service does not lose an event when the broker is briefly unavailable. The event is a deliberate contract that the service team designs for analytics, so it can differ from the internal table layout.
- Effort drivers: Each service that emits events needs an outbox table, a write path inside its transactions and a relay. Effort scales with the number of services and the number of event types each one publishes.
- Consistency: Consumers must still tolerate duplicates, and ordering holds per key only when the key is chosen deliberately.
- Schema changes: Event payloads are owned by the producing team. Version them explicitly and keep old fields until consumers have moved, since analytics jobs often lag product releases.
- Operations: The team maintains outbox cleanup, a retention rule for published rows, relay monitoring and tests that fail when a code path changes state without writing an event.
Confirm which team owns each event contract and how outbox rows are removed after publication.
Scheduled jobs query the production database or export files on a timer and load the results into the analytics store. This option needs no streaming components on the source side, and teams often build it first. It captures periodic snapshots, however, so it misses changes that happen between runs and adds load to the source at each run.
- Effort drivers: Most of the work is writing and testing extract queries, incremental watermark logic and load jobs. Heavy queries against production need a read replica or a window agreed with the owning team.
- Consistency: Each run reflects the data at its query time. Deletes and late updates need explicit handling, such as a full reload or a soft-delete column, or the warehouse drifts from the source.
- Schema changes: Export queries can fail quietly when a column is renamed. Pin the column list in each query and add a check that fails the job when the schema differs.
- Operations: The team owns scheduler reliability, retries, load windows and the cost of repeated scans. Kafka plays a limited role unless the exports are also published to topics for other consumers.
Before accepting this option, ask what freshness the analytics users need and whether periodic snapshots meet it.
What drives Kafka cost as event volume grows
Kafka cost rarely grows in a straight line with message count. It grows with the partitions, retention, replicas, consumers and network paths that each new customer or feed adds, so the design choices made early shape the bill for years. The cards below cover what to model, which costs are one-time and which recur, and what to request from the vendor or your own environment before committing to a size.
Partition count and broker sizing
Partitions set how many brokers can share the load and how many consumers can read in parallel, so they drive the broker count, memory and file handles your cluster needs. Adding partitions later is possible, but it changes key-to-partition mapping for keyed topics and raises per-broker overhead. Ask your team or partner to model partitions per tenant tier and the broker count that results, and confirm the limits in the KRaft and broker documentation for your target release.
Retention period per topic
Retention multiplies stored bytes by the number of days each topic keeps data, so a single topic set to long retention can outweigh the rest of the platform. SaaS teams often keep raw product events longer than they need for replay, which turns disk into a recurring cost. Set retention per topic class rather than cluster-wide, and record which topics are kept for replay, audit or analytics so each retention period has a stated purpose.
Replication factor and cross-region traffic
Each additional replica stores another full copy of the data and sends it across the network, so replication factor multiplies both disk and inter-broker traffic. Cross-region replication with MirrorMaker 2 adds a second copy of topics, offsets and ACLs, plus the bandwidth to ship them. Ask for the replication factor, minimum in-sync replica setting and the number of regions each tier requires, then price the replicated bytes rather than the produced bytes.
Consumer and connector fleet size
Consumers, Kafka Connect workers and stream processing jobs run as separate compute, and their count usually grows with the number of tenants and downstream systems rather than with message volume. Every new change data capture pipeline or analytics sink adds a connector task that needs CPU, memory and monitoring. Ask for a count of consumer groups and connectors per tier, the worker sizing for each, and who owns the upgrades of that fleet.
Storage growth on compacted topics
Compacted topics keep the latest value per key rather than a time window, so their size depends on the number of distinct keys and how often tombstones are removed. Growth can look flat for months and then rise sharply when customer or entity counts increase. Ask for the key cardinality, the expected size of each compacted topic and the compaction settings in use, then review those figures against actual disk trends each quarter.
Cloud egress and operating hours
When brokers, consumers or analytics warehouses sit in different cloud zones, regions or providers, data transfer charges can rise with every cross-boundary read and replica. Recurring operations work also costs money: monitoring, certificate rotation, patching and the quarterly feature upgrades each take engineer time. Ask your provider for a model that separates one-time migration and design work from recurring infrastructure, transfer and support hours, and confirm which of those your own cloud agreement bills.
What SaaS teams ask before hiring help for Kafka
What does a self-hosted Kafka implementation for a SaaS product cost?
The cost is set by the written scope, not by a list price, and DeployOpen does not publish one. The main drivers are the number of tenant models to design, how many customer-facing and internal topics need quotas and access rules, how many change data capture pipelines feed analytics, and how much of the surrounding platform (identity, secrets, monitoring, backups) must be built in the same engagement. Kafka itself is licensed under the Apache License 2.0, so there is no project license fee. Ask for a quote that separates one-time design and build work from recurring items such as upgrades, testing and on-call support, and that names the infrastructure you pay for directly in your own cloud or data center accounts.
How long does a multi-tenant Kafka build take?
The timeline depends on how many decisions are still open when work begins. A build moves faster when the tenant model, topic naming, quota tiers and access policy are agreed before implementation, and slower when each new customer type raises a new isolation question. Other drivers include how many source databases feed change data capture, whether the platform must run in more than one site, and how much integration with your identity provider and monitoring stack is needed. Because most engagements are fixed-scope, the agreed scope document should state the phases and their dependencies so you can see what sets the end date.
What must our team provide before the engagement starts?
Four inputs shape most of the design. First, a tenant model: whether each customer gets its own topic, a shared topic with keys, or a dedicated cluster, and how that maps to your billing and data residency rules. Second, a data classification for every event class, so retention, encryption and access can be set per class. Third, the identity provider that will issue client credentials, with the SASL mechanism your team can operate. Fourth, an on-call rota with named owners for broker, consumer lag and connector alerts. Missing any of these usually adds a design round after build has started.
Can our internal platform team run this without outside help?
Often yes, if the team already operates Linux, Kubernetes or VM infrastructure and has someone who understands distributed log systems. The work that tends to be hardest for internal teams is not the first install but the sequence of decisions that follow: quota design across tenants, ACL models that stay manageable as the number of teams grows, change data capture error handling, and a tested upgrade path. Compare your team's time against those items, not just against installation. A partner can cover some of them under a written scope, and a retainer can cover continuous upgrades and testing if your team prefers to keep day-to-day operation in-house.
How is SOC 2 evidence produced from a Kafka platform?
Kafka does not produce a SOC 2 report, and an examination is performed by a CPA firm, not by DeployOpen or the Kafka project. What Kafka can supply are configuration and access records: authentication settings using SSL or SASL, ACLs stored in the cluster metadata log under the StandardAuthorizer, TLS settings on listeners, and change history for topic and ACL definitions. The common criteria CC6 (logical access) and CC8 (change management) are where these records usually matter most. The platform around Kafka must add centralized log collection, access reviews, change tickets and evidence that controls operated over the examination period, which a Type 2 report covers. Confirm the exact criteria in scope with your auditor before designing the evidence set.
How are upgrades within Kafka 4.x handled?
Upgrades within 4.x are rolling upgrades, and the Kafka time-based release plan states that rolling upgrades are supported from each release in the past year, the last three releases, to the latest version. The effort depends on which release you start from, whether the cluster runs KRaft (ZooKeeper mode was removed in 4.0), and the Java version your brokers, Connect workers and clients run. Since 4.0, brokers, Connect and tools require Java 17, while clients and Kafka Streams applications need at least Java 11. The supported releases on the downloads page are 4.3.1, 4.2.2 and 4.1.2, so plan your target against that list and test connectors and client libraries before each step.
What changes in cost and effort as message volume grows?
Growth affects storage, network and operator time more than license cost, which is zero for Kafka itself. Storage grows with retention period, replication factor and throughput, so retention per topic should be set from the consumers that need replay rather than from a single global default. Partition count affects broker load and recovery time, and each change data capture connector adds offset tracking and error handling work. Ask any partner to model these drivers against your projected volume and to state which costs are fixed infrastructure and which scale with usage.
How DeployOpen scopes and delivers Kafka for SaaS platforms
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside the client's own cloud accounts, data centers or air-gapped networks. For SaaS and technology companies, that means a Kafka platform designed around your tenants, quotas, data pipelines and audit evidence, delivered under a written scope.
How we scope
- Written scope listing tenant model, topic naming, quota policy and the clusters in scope before work starts
- Inventory of producers, consumers, CDC sources and analytics sinks that the platform must serve
- Agreed deliverables, timelines and acceptance criteria, with the SOC 2 control areas the platform must support identified
How we build
- KRaft cluster deployment, topic and ACL configuration, and per-customer quotas built and tested in your environment
- Change data capture and analytics connectors configured, with schema and retention rules applied to each topic
- Security testing of the cluster and its client access paths, with findings tracked to closure before handover
How we hand over
- Runbooks for provisioning tenants, rotating credentials, scaling partitions and restoring from failure
- Documentation of configuration, access rules and change records, arranged so your auditor can trace each control
- Knowledge transfer sessions with your platform and security engineers, ending with a documented handover of ownership
How we support
- Ongoing support, upgrades and continuous testing available as a retainer after handover
- Planned upgrades between supported Kafka 4.x releases, with compatibility checks for connectors and clients
- Quotes derived from the written scope, so each support item and its frequency is visible before you commit
