Kafka across data centers: which multi-site layout fits your organization
If you run or plan Kafka in more than one data center, the main choice is between local clusters with mirroring and a stretched cluster, and the right answer depends on latency between sites, which failure domains you need to survive, and who consumes the data. A *local cluster* is an independent Kafka cluster in each site that serves its own producers and consumers. An *aggregate cluster* is a central cluster that receives copies of topics from several local clusters so that one place holds a global view. A *stretch cluster* is a single Kafka cluster whose brokers and controllers are spread across sites and act as one system. Kafka is open source under the Apache License 2.0 (Apache-2.0) with no paid edition from the project, so the cost is the infrastructure, the network between sites and the people who operate it.
Local clusters with mirroring keep each site's writes and reads inside that site, so a WAN outage or a slow link does not stall local producers. Cross-site copies are handled by MirrorMaker 2, which is built on Kafka Connect and replicates topics with their configuration, consumer groups with offsets and ACLs between clusters while preserving partitioning. A stretched cluster, by contrast, makes every produce and replication path depend on inter-site latency and on the controller quorum staying reachable across sites. Before choosing, ask your network team for measured round-trip time and packet loss between sites, define which site failures you must tolerate, and list the consumers that need data from other sites and how fresh it must be. Confirm these requirements in your own environment with your Kafka operators and network owners, because the answers usually decide the layout.
How the three multi-site layouts differ
Each site runs its own independent Kafka cluster, and MirrorMaker 2 copies selected topics to the other sites. MirrorMaker 2 is built on Kafka Connect. It replicates topics with their configuration, consumer groups with their offsets and ACLs between clusters, and it preserves partitioning, so each record keeps its partition in the target.
- Data movement: Mirror connectors read from the source cluster and write to the target cluster asynchronously, so each site's writes never wait on another site.
- Site outage: Producers and consumers at surviving sites keep working on their local clusters. Mirrored copies at other sites stop at the last record that replicated before the outage.
- Consumers: Each site reads its local topics without cross-site dependency. Consumers that need remote data read the mirrored topics and see replication lag.
- Operations: Effort lands on running and monitoring the mirror connectors, watching replication lag, and planning how consumer offsets fail over and back.
Confirm how your replication policy maps topic names in the target cluster, and who owns the mirror connectors during an incident.
Each site keeps a local cluster for its own workloads, and a central aggregate cluster receives mirrored copies of selected topics from every site. The central cluster becomes the single place where a global view of the data exists. Data moves through MirrorMaker 2 in one direction per source site, and the mechanics match the local layout, but the central cluster is where cross-site analytics and consumers run.
- Data movement: Each site's selected topics flow into the aggregate cluster, and the aggregate usually does not feed data back to the sites. Confirm this direction for your own design.
- Site outage: A failed site stops adding new data to the aggregate but keeps what it already replicated. The other sites continue to feed the central cluster.
- Consumers: Global consumers read the aggregate and see a delay equal to replication lag. Site-local consumers read their own cluster and have no dependency on the central one.
- Operations: Effort concentrates on the central cluster's capacity, access control over merged data, and the number of mirror connectors, which grows with each site added.
Confirm which team owns the central cluster and how it will handle a site that stops replicating for an extended period.
A single Kafka cluster spans two or more sites, with brokers and controllers placed in each site. Replication between sites happens inside the cluster through partition replicas, so there is no separate mirroring layer. The design depends on inter-site network quality, because replication and controller traffic cross the WAN on every write path.
- Data movement: Partition leaders and followers sit in different sites, and followers replicate across the WAN continuously.
- Site outage: Whether the cluster stays available depends on where the controller quorum and replicas sit. Test the outcome of losing each site before you commit to this layout.
- Consumers: Consumers see one cluster and one set of offsets, with no mirror lag to reconcile. Cross-site latency still affects produce and fetch timing.
- Operations: Effort lands on network capacity and latency monitoring, socket buffer tuning for WAN links, and rehearsing site failure against the live cluster.
Check the latency and bandwidth the inter-site link can sustain, and where each controller and replica will sit, before accepting this layout.
Which configuration option fits which site pattern
This table compares three multi-site layouts (local clusters with mirroring, an aggregate cluster for a global view, and a stretched cluster) against the factors that usually decide between them. Read each row against your own WAN round-trip time, replication throughput, and data and staffing constraints, and treat the cells as starting points for discussion rather than a verdict.
| Decision factor | Local clusters with mirroring | Stretched cluster |
|---|---|---|
| Inter-site latency tolerance | Tolerates higher and variable WAN latency because each site's cluster is independent of the link | Requires low, stable latency between sites, since controllers and replicas depend on it for every commit |
| Consumer locality | Consumers read from their local cluster, with mirrored copies available elsewhere | Consumers at a remote site may read across the WAN unless fetch-from-follower is configured and verified |
| Ordering needs | Ordering holds within each cluster and partition; cross-cluster order is not guaranteed by mirroring | Ordering is maintained across sites within a single partition leader history, which suits global ordering needs |
| Data residency | Easier to keep data at the site where it is produced, with only approved topics mirrored | Replicas span sites, so confirm with legal and compliance whether copies may leave each jurisdiction |
| Failure isolation | A site outage or cluster fault stays contained; the other site keeps operating | A link or quorum problem can affect both sites at once, so failure domains must be designed carefully |
| ACL and configuration replication | MirrorMaker 2 can replicate ACLs and consumer offsets, but scope and sync behavior must be tested | ACLs live in one metadata log shared by all brokers, so access policy is applied everywhere without a sync step |
| Operations staffing | Requires running and upgrading several clusters plus mirroring components and their monitoring | One cluster to operate, but a sensitive quorum that needs careful maintenance across sites |
How to choose and stand up a multi-site layout
This sequence turns a multi-site requirement into a documented layout, sized WAN links, replication flows and a rehearsed failover, so each site's role and its recovery behavior are known before production traffic depends on them.
Document sites, data classes and recovery goals
Platform owners, application teams and the business owner of each data class list every site, the workloads that run there, the topics they produce and consume, and the recovery point and recovery time each team needs after a site loss. Compliance or data residency constraints that limit where copies may live are recorded here too. The step is done when every topic is assigned to a data class and every class has written recovery goals that the business has signed off, even if the numbers are still provisional.
Decide per topic between local and global copies
For each topic, the team decides whether consumers need a copy in every site (a local copy, mirrored between clusters) or a single combined view across sites (an aggregate cluster fed by mirroring). Topics that must stay within one jurisdiction are marked as local only. The output is a topic register with a placement decision and a reason for each row. Ask consumers to confirm the decision for their topics, because a wrong guess here is expensive to reverse once data has flowed.
Pick the layout and confirm stretch cluster fit
The team selects one of three shapes: one cluster per site connected by MirrorMaker 2, aggregate clusters for a global view, or a single stretched cluster across sites. A stretched cluster is only acceptable when the inter-site latency and the failure behavior of the controller quorum and partition leadership have been reviewed against the recovery goals from step one. Document the rejected options and why. The step closes with a signed layout decision and an architecture diagram that names every cluster, controller set and mirror direction.
Size WAN links and tune socket buffers
Network engineers measure the bandwidth and round-trip time of each inter-site link, then estimate replication traffic from the production rates of mirrored topics, including the traffic for every additional mirror hop. Socket buffer sizes for replication are set to match the bandwidth-delay product of each link, and the same values are applied in both directions. Check the result with a sustained replication test at expected peak load, watching for throughput that stalls below link capacity. Confirm the link headroom with the network provider before go-live.
Deploy each cluster from the bootstrapping and hardware guides
Each site's cluster is built using the KRaft bootstrapping procedure, with controller and broker hardware, OS settings and JVM settings taken from the sizing and tuning guides. Clusters are not joined to each other at this stage. The check for this step is that every cluster reports a healthy quorum, all brokers are registered, and a test topic can be produced to and consumed from inside each site without any cross-site traffic.
Configure MirrorMaker 2 flows, topic and offset mapping
The team defines each MirrorMaker 2 flow with its source and target aliases, topic filters and consumer group filters, and sets the replication factor and naming behavior for replicated topics. Offset synchronization is enabled so that consumer groups can translate offsets when they move to another cluster. Topic configuration and ACL syncing are included in the flow only where the layout requires them. The check is that a test message produced at the source appears at the target under the expected name, and that a test consumer group's offsets are translated to the target cluster.
Verify lag, offset translation and ACL sync
Operators measure replication lag under normal and peak load and compare it to the recovery point goals from step one. They confirm that translated offsets resume a consumer group at the expected position and that ACLs created at the source appear at the target, using the authorization model of each cluster. Any gap between the ACL state at the two clusters is documented and assigned an owner. Lag thresholds and alerts are set and tested before the rehearsal begins.
Rehearse failover and record the results
The team runs a planned failover: producers are redirected to the surviving cluster, consumers resume from translated offsets, and the observed data loss and recovery time are recorded against the goals. The rehearsal then includes failback to the original site. The step is complete when the measured results, the runbook steps that were missing or wrong, and the owners of each fix are written down, and the runbook is approved for use. Repeat the rehearsal after material changes to topics, ACLs, network links or Kafka versions.
When a stretched cluster is acceptable
A stretched cluster spans two or more sites with one set of brokers and controllers, so it is only worth considering when the network between the sites is low-latency, stable and well understood. Inter-site latency that is low and steady keeps replication and controller traffic predictable. Measure round-trip time and packet loss over several weeks, including peak hours and backup windows, rather than relying on a single test. If the link is occasionally slow or congested, a stretched layout will convert network problems into producer timeouts and under-replicated partitions, and those are hard to diagnose from the application side.
The layout also needs enough sites to keep a quorum after losing one. Kafka's KRaft controllers use a Raft majority, so with two sites the usual design puts a majority on one side and leaves the other site unable to elect a leader. Plan a third site, or a third failure domain such as a small witness location, to host controllers. Confirm controller placement, the number of voters and how a site loss changes the quorum against the Kafka 4.3 documentation and your own failover tests, not against assumptions carried over from older releases.
Replication settings must match the failure you accept. Set the replication factor and min.insync.replicas for topics so that losing one site still leaves enough in-sync replicas to accept writes, and decide explicitly whether producers should favor availability or durability during a partition. Write down the site partition plan: which site keeps serving, how producers and consumers are redirected, when an operator intervenes and how the other site rejoins and catches up. Run that plan as a rehearsal with the controller quorum, min.insync.replicas and client retry settings in place. Ask your vendor or implementer for the test results in writing before the stretched design goes into production.
Security and operations across sites
Every link between sites widens the set of places where traffic, credentials and access rules can fail. The cards below cover the controls to decide before a multi-site layout goes live, from encrypting inter-site traffic to watching replication lag and keeping mirrored data inside approved regions.
TLS on inter-site links
Replication traffic crosses networks you may not own end to end, so it needs the same encryption as client traffic. Kafka supports SSL/TLS between clients, brokers and tools, but security is optional, so a listener can run unsecured without anyone noticing. Confirm that every inter-site listener, including the MirrorMaker 2 connection to the remote cluster, uses TLS, and decide how certificates are issued and rotated across both sites.
Choosing a SASL mechanism
The documented SASL mechanisms are GSSAPI (Kerberos), PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER. Each one ties to a different identity source, so the choice affects where credentials live and how they are revoked. Pick one per site based on the identity systems you already run, and confirm that both sites accept the mechanisms your replication connectors will use. PLAIN sends credentials that must be protected by TLS.
ACL replication and default deny
In KRaft clusters, the StandardAuthorizer stores ACLs in the cluster metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. A mirrored topic can arrive at the remote site without the access rules its consumers expect. Check that MirrorMaker 2 replicates the ACLs you need, and test that a consumer in the target cluster is denied until its rule exists.
MirrorMaker 2 credentials
MirrorMaker 2 runs as Kafka Connect connectors that hold credentials for the source and target clusters. Those credentials usually have broad read access, so a leaked connector configuration exposes data on both sites. Give each replication flow its own principal with the narrowest topic and group permissions, store its secrets in a managed vault rather than plain configuration files, and plan rotation as a routine change.
Monitoring replication lag
Replication lag is the gap between data written at the source and data confirmed at the target, and it defines how much data you would lose in a site failure. Track lag per topic and partition, alert on sustained growth instead of brief spikes, and check connector health separately from broker health. Agree with the business owners what lag is acceptable before go-live, because the number determines the recovery point you can honestly promise.
Data residency for mirrored topics
Mirroring copies every record, key and header to the remote site, which can move regulated personal data across a border even when the producing application stayed local. Map which topics may leave each jurisdiction, filter or exclude the rest in the replication configuration, and confirm that compacted or retained copies at the target follow the same retention rules. Document these decisions with the teams that own the data.
Questions teams ask before choosing a multi-site layout
How much does a multi-site Kafka layout cost and what drives it?
Kafka itself is free under the Apache License 2.0, so the cost is the infrastructure, the people who build and run it, and the testing that proves it works. Cost rises with the number of sites, the number of clusters (a local-cluster design runs one per site plus mirroring, while a stretched design runs one larger cluster), the WAN capacity between sites, and the number of environments you must replicate for testing. Recurring costs include WAN circuits, hosts in each site, storage growth, and staff time for upgrades and monitoring. DeployOpen does not publish prices. Most engagements are fixed-scope, so ask for a quote built from a written scope that lists sites, cluster count, replication flows, failover tests and handover items.
How long does a design and build take, and what changes the duration?
Duration depends on the number of sites, how many topics and consumer groups must be mirrored, how many existing producers and consumers must be re-pointed, and whether network links, firewall rules and certificates are ready when work starts. Failover and recovery testing often takes longer than the build itself. Your quote should state the phases, the dependencies you own, and the acceptance tests that close each phase. Confirm the timeline against your own change calendar and any freeze periods before agreeing to a start date.
What does the work need from our team?
Expect to provide named owners for network design between sites, DNS and certificate authorities, identity and access groups, and the business owners who decide acceptable recovery objectives for each data flow. Your platform team should also expect to review the topic inventory, retention needs and consumer behavior. Ask the partner to list these inputs in the written scope so that gaps show up before build work starts rather than during cutover.
Can our own engineers design and build this themselves?
Yes, if they already run Kafka in production and have experience with KRaft controllers, TLS, ACLs and MirrorMaker 2 or a similar replication tool. The official geo-replication documentation covers how MirrorMaker 2 replicates topics, consumer group offsets and ACLs between clusters, and it is a good starting point for an internal design review. The harder parts are usually WAN tuning, failure testing and operating the layout over several upgrades. Compare your team's capacity and experience against the scope, and consider a partner for the parts you have not run before.
Is a stretched cluster safe for disaster recovery?
A stretched cluster can keep one logical cluster across sites, but it is not a backup. Replication is synchronous within the cluster, so a slow or partitioned WAN link affects producers and consumers, and a bad configuration change or deleted topic can propagate to every site. Disaster recovery needs a separate copy of data and metadata, tested failover, and a clear decision on what data loss is acceptable. Many teams therefore choose separate clusters with mirroring for recovery. Confirm the controller quorum placement, the minimum number of sites that must stay reachable, and the recovery test results before you accept a stretched design for production.
What does a socket buffer tuning step involve for WAN replication?
Replication over a high-latency link is limited by the TCP window, so socket buffer settings such as the send and receive buffer sizes on brokers and the OS must match the bandwidth-delay product of the link. Tune these values with measured throughput and latency, not guesses, and keep the same settings documented per site. The Linux and filesystem tuning page covers the OS side in more detail. Ask the partner to show a test plan that measures replication lag before and after tuning, and confirm that the chosen values are supported by your network team.
When should we hire a partner for a multi-site Kafka layout?
Consider a partner when the layout crosses regulated data boundaries, when recovery objectives are strict, when your team has not operated multi-cluster replication before, or when the migration must finish inside a fixed window. A partner is also useful when you need an independent review of the design. DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks. It does not issue audit reports or certifications. Start by sharing your site list, data classes and recovery goals so the written scope can be drafted.
How DeployOpen scopes and delivers a multi-site Kafka design
DeployOpen approaches Kafka across data centers as a written design decision first, then a build in your own infrastructure. We plan the layout around your sites, failure scenarios and access model before any cluster is provisioned, and we document what is replicated, what is not, and who operates each part.
How we scope
- Engagements are fixed-scope, with sites, data flows, latency and failure requirements agreed in writing before work starts.
- The written scope names the layout chosen (local clusters with mirroring, an aggregate cluster, or a stretched cluster) and the reasons for it.
- Scope lists which topics are mirrored, which consumer groups and ACLs follow, and which stay local to a site.
How we build
- We build the chosen layout in your cloud accounts, data centers or restricted networks, using the Kafka 4.x release you select.
- MirrorMaker 2 configuration, replication settings and WAN socket buffer tuning are tested against the links you actually have.
- Failover and failback steps for consumers, producers and offsets are rehearsed in a test environment before any cutover.
How we hand over
- Handover includes architecture diagrams, topic and replication inventories, and the runbooks your operators will use.
- We document the monitoring signals for replication lag and link health, with the thresholds agreed with your team.
- Your team receives configuration in version control, with a list of decisions that must be revisited at each Kafka upgrade.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers after handover.
- Retainers can cover link changes, new sites, Kafka release upgrades and periodic failover exercises, as agreed in writing.
- Confirm with us during scoping which parts of support sit inside a retainer and which are billed as separate scoped work.
