Why multi-datacenter streaming recovery is a separate design decision
When a site fails, the streams that feed payments, orders or device telemetry must either keep flowing from another site or stop in a way you can explain. Neither outcome is automatic. A replicated topic is not a backed-up topic: a produce error, a bad consumer deploy or a misconfigured retention setting is copied to the standby site just as faithfully as the original data. Recovery design therefore has to answer three questions for each stream separately: how much data may be lost if the primary site disappears, how long consumers may be paused, and whether the standby site can be trusted to contain the same records in the same order. A stream carrying idempotent status updates and a stream carrying ledger-relevant events rarely share the same answer, so the design should not assume one pattern for the whole estate.
On self-hosted Apache Kafka, the platform answer for cross-site replication is MirrorMaker 2, which runs on Kafka Connect and copies topics with their configuration, consumer groups with offsets and ACLs between clusters while preserving partitioning. Where Pulsar is the chosen platform, the equivalent starting point is Pulsar geo-replication. Neither product decides your recovery objectives for you. Before choosing an active-passive or active-active pattern, confirm with your platform team which replication lag you can accept, which ACL and offset state must survive a switch, and which topics must never be replicated across the boundary you have to respect.
The practical next step is to classify streams by business impact and write the recovery objectives for each class before any cluster is sized. Ask the implementation partner how replication lag will be measured and alerted on, how consumer offsets will be translated for the standby cluster, and what evidence will show that a failover was rehearsed rather than assumed. Those answers, not the replication feature list, determine whether a site loss stays an operational event or becomes a data incident.
Which requirements shape the recovery design
This table maps each requirement that affects a multi-site streaming design to its source and to what it means for the build. Read it before choosing between active-passive and active-active, because several rows (offset continuity, ordering and residency) constrain that choice more than throughput does.
| Requirement | Where it comes from | What it means for the build |
|---|---|---|
| Recovery point objective per stream | Internal business impact analysis and regulator expectations for the function that owns the stream | Set the objective per topic, not per cluster. Replication lag under normal load determines the achievable point, so measure it in your own environment before committing. |
| Recovery time objective per stream | Internal business continuity policy and, for regulated firms, operational resilience rules | Decide whether failover is manual or automated. Automated switchover needs health signals and runbooks that are tested, not just scripted. |
| Consumer offset continuity | Platform behavior: MirrorMaker 2 replicates consumer groups with offsets between clusters, per the Kafka geo-replication documentation | Consumers must resume from translated offsets on the target site. Confirm translation accuracy and the handling of gaps and duplicates in each consumer before relying on it. |
| Ordering across sites | Application requirement, such as per-account or per-order sequencing, plus Kafka partition semantics | Ordering holds within a partition on one cluster. Replication can interleave writes from two active sites, so define which site owns each key or accept reconciliation logic in consumers. |
| Data residency of replicated topics | Data protection law, contractual terms and sovereignty policy that bind where personal data may be stored | Every replica is a copy. Map which topics carry personal data and confirm which regions and facilities may hold them before enabling cross-region mirroring. |
| ACL and configuration replication | Platform behavior: MirrorMaker 2 replicates topic configuration and ACLs, and KRaft clusters store ACLs in the metadata log | Replicated ACLs can fail to match the target's identity setup. Test authentication and authorization on the standby site, including service accounts and certificates, not only data flow. |
| Evidence of tested failover | Assessor expectations under resilience and continuity controls, and internal audit requirements | Keep dated records of switchover tests, measured lag, offset translation results and rollback. Assessors sample these records, so the test plan must produce them by design. |
| Upgrade parity between sites | Platform behavior: Kafka rolling upgrades support moves from the last three releases to the latest version, per the Kafka time-based release plan | Run the same supported release at both sites and schedule upgrades so a failover target never runs an unsupported or untested version. |
Which streaming platforms fit multi-site recovery, and why
Multi-site recovery depends on how a platform replicates topics, consumer state and metadata between clusters, and on how much operating work each option leaves to your team. The cards below compare the two platforms with native cross-cluster replication, then explain where NATS and RabbitMQ fit only if your topic scope requires them. Confirm each platform's cross-site behavior against the exact version you plan to run.
Apache Kafka with MirrorMaker 2
MirrorMaker 2 is built on Kafka Connect and replicates topics with their configuration, consumer groups with offsets and ACLs between clusters, while preserving partitioning. It is licensed under Apache-2.0, and the Apache Software Foundation project offers no paid edition, so replication tooling, support and operating effort sit with you or your partner. Check how offsets are translated on failover, which ACLs replicate, and whether your current Kafka release is one of the supported lines before you design the topology.
Apache Pulsar geo-replication
Pulsar provides built-in geo-replication as a core feature of the platform, licensed under Apache-2.0. Its architecture separates stateless brokers from BookKeeper bookies that store data, with a metadata store that can be Oxia, recommended for new clusters, ZooKeeper or RocksDB. Pulsar's LTS line gives you a longer support window to plan around, but confirm which metadata store and replication settings your version supports before committing to a cross-site design.
Pulsar support windows
Pulsar ships an LTS release every 18 months with 24 months of active support and 36 months of security support, and feature releases every three months with six months of support. Pulsar 4.0 LTS reaches the end of active support in October 2026, so a site failover design built on it needs a planned upgrade path. Ask your partner to map each cluster's version against these windows and to schedule upgrades so both sites stay on supported lines.
Kafka version and KRaft prerequisites
Apache Kafka 4.0 supports only KRaft mode, so ZooKeeper-based clusters must migrate before upgrading to 4.x. Rolling upgrades are supported from each release in the past year to the latest version, which affects how you sequence upgrades across two sites. Clusters older than 3.3.x should move to 3.9.x before 4.x. Confirm the upgrade path for each cluster and whether MirrorMaker 2 and Kafka Connect on both sides run compatible releases.
Metadata and storage at each site
In Pulsar, storage capacity grows by adding BookKeeper bookies independently of brokers, which changes how you size each site and how fast a standby can take over. In Kafka, the KRaft controller quorum and the StandardAuthorizer's ACLs live in each cluster's metadata log, so a failed site's access rules must be present on the surviving cluster. Document which metadata each platform keeps locally and test whether a standby site can authorize clients without manual repair.
NATS and RabbitMQ where scope requires
Choose NATS or RabbitMQ only when your topic scope calls for protocols or messaging patterns that Kafka or Pulsar do not serve well, such as AMQP queues or lightweight subject-based messaging. Their cross-site patterns differ from Kafka's and Pulsar's, and this page does not supply verified replication details for them. Ask your partner to document each platform's cross-site behavior, failover steps and tested recovery objectives before adding a second streaming system to the estate.
Active-passive or active-active: choosing the replication pattern
In an active-passive design, one cluster serves production traffic and a second cluster in another site receives a copy of the data and stays on standby. Producers and consumers connect only to the primary. When the primary site fails, operators or automation redirect clients to the secondary, which is the failover step. The model is easier to reason about because there is one source of truth for writes at any moment, so ordering within a topic is not split across sites. The cost is that the standby carries capacity that serves no traffic until something goes wrong, and the switch is only as quick as your runbook and client configuration allow.
- Producers: applications must point at a logical endpoint, such as a DNS name or configuration key, so the switch does not require code changes.
- Consumers: consumer groups on the standby resume from translated offsets, so you must test that replay does not exceed what your downstream systems can absorb.
- Replication lag: the recovery point is the last replicated message, so the lag you measure in normal operation sets the data you may lose.
- Standby capacity: size the secondary to carry production load at failover, not only to hold replicated data.
- Failback: returning traffic to the original site is a separate, planned procedure, covered in its own tab.
Ask the partner to show the failover runbook and the last failover test, including the measured replication lag and the client endpoints that changed.
In an active-active design, both sites accept writes and replicate to each other. Each site serves local producers and consumers, which can reduce latency for distant users and avoids keeping idle capacity. The trade-off is that the same logical stream can receive events in two places, so ordering is no longer guaranteed across sites, and the system must decide how to handle duplicates and conflicting updates. Replication tools such as MirrorMaker 2 and Pulsar geo-replication copy data in each direction, but they do not resolve business conflicts on their own. Active-active therefore usually requires topic naming rules that prevent two sites from writing to the same topic, or consumers that tolerate reordering and duplicates.
- Topic ownership: a common pattern is one writer per topic or per key range, with replicated copies on the other site.
- Ordering: events for one entity must stay on one site, or consumers must reconcile order using timestamps or sequence numbers from the producer.
- Duplicates: replication and retries can deliver a message more than once, so consumers need idempotent handling.
- Offset translation: consumers that read from both sites must track which offsets map to which source.
- Failover: a site loss shifts only its owned topics, which requires a documented ownership transfer, not an automatic switch.
Ask the partner which topics are written from both sites, how conflicts are resolved in application code, and what the consumer tests showed for duplicate and out-of-order events.
Whatever pattern you choose, a consumer group on the secondary cluster needs a position that corresponds to where it stopped on the primary. MirrorMaker 2 replicates consumer groups with their offsets and includes checkpoint data to translate offsets between clusters, so a consumer can resume near the point of failure. Translation is approximate in practice because replicated offsets lag the source, which means consumers may reprocess some messages after a switch. Pulsar geo-replication handles subscriptions differently, so the translation approach depends on the messaging system and its version. Plan for at-least-once processing unless your downstream systems enforce idempotency.
- Checkpoints: confirm that the checkpoint topic is replicated and monitored, since translation depends on it.
- Reprocessing window: measure how many messages a group may reprocess after a switch, and whether downstream writes can absorb them.
- Consumer groups: list every group that must run on the secondary, including groups owned by other teams.
- Connectors: Kafka Connect source and sink tasks keep their own offsets, so connector state must be included in the failover plan.
- Testing: use a controlled failover to check resumed positions against expected counts.
Ask the partner to show offset translation results from a test, with the reprocessing count and the groups covered.
Failback is the return of production traffic to the original site after it has been repaired. It is a separate step because the data state is different after failover: the secondary has accepted writes that the primary never saw, and the primary may hold messages the secondary has not yet received. Reversing replication direction before the two sites are reconciled can duplicate or lose data. In an active-passive design, failback usually means rebuilding the primary as a standby, letting it catch up, confirming the replication lag is within your objective, and then scheduling a planned switch. In active-active, failback is mainly ownership transfer back to the original writer for affected topics.
- Reconciliation: decide which site's data is authoritative for each topic and document the rule before failback starts.
- Catch-up: wait until replication has drained and the lag is inside the agreed objective before moving traffic.
- Change window: plan the switch with producers and consumers paused or drained, since each restart may trigger reprocessing.
- Rollback path: keep a defined way to return to the secondary if the primary fails again during failback.
- Evidence: record timestamps, offsets and sign-off for each step, since auditors and resilience reviews may ask for it.
Ask the partner whether failback has been exercised separately from failover, and how long the previous test kept the estate in a degraded state.
How consumer offset translation works during failover
MirrorMaker 2 replicates consumer group offsets along with the topics, so a consumer group that was reading from the primary cluster can be pointed at the secondary cluster and resume from a translated position rather than from the beginning or the latest message. The documented behavior covers replicating offsets for consumer groups between clusters. It does not mean the secondary holds an identical offset value for every message, because the source and target clusters assign offsets independently. Translation is therefore a mapping that the replicated checkpoints provide, and it is only as current as the last checkpoint that reached the secondary. Before you commit to any failover design, confirm in your Kafka version's geo-replication documentation how checkpoints are emitted and how the offset translation is read by consumers, and document the exact procedure your team will run.
The main risk is a stale checkpoint. If replication lagged when the primary failed, the translated offset points earlier than the last message the consumer actually processed, so the group reprocesses a window and produces duplicates unless consumers are idempotent. If the mapping points later than what was processed, messages are skipped. Plan for both. Make consumers idempotent on a business key, record the source cluster and offset alongside each side effect where your design allows it, and set an explicit rule for how far back a switchover may rewind. Decide who owns the cutover decision for each consumer group, because a group switched without its producers being redirected will sit idle or read a diverging stream.
Test with real consumer groups, not a synthetic topic. Use production group IDs, realistic lag and the same client libraries and configuration, then run a switchover and measure the duplicate and gap counts against your recovery objectives. Repeat the test after each Kafka upgrade, because the offset and checkpoint behavior is part of what you must re-verify. Ask your implementation partner to show the test evidence for each group and to document the procedure for returning traffic to the primary. Confirm with the vendor or your own Kafka version which MirrorMaker 2 settings control checkpoint frequency and whether your client libraries support the translation step without custom code.
How a recovery rollout runs, phase by phase
The sequence turns a single-site streaming platform into one that can keep serving when a site fails, and each phase closes with a document or test result the organization can review and keep. Ownership is shared between the DeployOpen team, the platform owners and the business owners of each stream.
Scope stream classes and recovery objectives
The DeployOpen team inventories topics, consumer groups and connectors, then groups them into stream classes by business criticality, data sensitivity and acceptable data loss and downtime. Business owners of each class sign the recovery time and recovery point objectives, and the platform team confirms they are achievable with the chosen pattern. The output is a signed objectives register that names an owner for every class and states which pattern (active-passive or active-active) applies to it. The step is done when no topic is unassigned and every owner has signed.
Build the secondary site and replication topology
The team provisions the secondary cluster in the organization's second data center or cloud region, sized for the replicated load, and documents the replication topology. For Kafka this usually means MirrorMaker 2 running on Kafka Connect, with replication flows defined per stream class. For Pulsar it means geo-replication between clusters. Network paths, latency and bandwidth are measured against the objectives, and the topology is checked for replication loops. The output is a replication topology diagram and a configuration baseline in version control. The step is done when test messages arrive at the secondary with preserved partitioning.
Configure ACLs and TLS on both sides
Security engineers apply TLS on every listener and authentication through SSL or SASL on both clusters, then write ACLs for replication principals, consumers and producers. Kafka's authorizer behavior is checked explicitly, because resources without ACLs are restricted to super users unless the allow-everyone setting is changed, and that setting should be a documented decision. Replicated ACLs and consumer group settings are verified on the secondary before any traffic depends on them. The output is an access matrix per principal and a configuration review signed by the security owner. The step is done when a test principal without rights is denied on both sites.
Write offset translation and switchover runbooks
The team documents how consumer offsets are translated between clusters so that consumers resume near the point where they stopped, and how producers and consumers are redirected to the secondary. Each runbook lists the commands or API calls, the decision owner, the order of steps, the checks after each step and the rollback path. Consumers that cannot tolerate duplicates or gaps are identified, and their handling is written down. The output is a reviewed runbook per stream class, stored with the objectives register. The step is done when an operator who did not write the runbook can follow it in a test environment.
Run failover and failback drills
The team runs controlled failover drills against the non-production copy first, then against production stream classes agreed with business owners. Each drill measures the actual time to redirect clients, the data gap and the offset position consumers resumed from, and compares these to the signed objectives. Failback is tested with the same rigor, because returning traffic to the primary site is where data divergence is most likely. The output is a drill report per stream class with measured results, defects and corrective actions. The step is done when every class has a drill that meets its objectives or has a logged, owner-approved gap.
Hand over runbooks, monitoring and operating duties
DeployOpen walks the platform, security and operations teams through the topology, runbooks, monitoring of replication lag and the drill reports. The handover covers who decides on failover, how that decision is recorded, and which alerts trigger it. Operators run one supervised drill themselves before handover is accepted. The output is a handover pack that includes the objectives register, topology, access matrix, runbooks and drill reports, along with the schedule for future drills, which the organization sets. The step is done when the client's named operators sign acceptance of the pack.
Evidence auditors and regulators ask for from a failover
This table lists the records an assessor typically asks to see after a site failure, the component that produces each one, and what to check before accepting it. Read each row as a question to answer with a file, export or log, not as a claim to be taken on trust. DeployOpen prepares this evidence inside your environment, while independent auditors and assessors judge whether it is sufficient.
| Evidence item | Produced by | What the assessor checks |
|---|---|---|
| Recorded recovery objectives per stream | Approved service register and runbook owned by the business and platform team | Each stream maps to a stated recovery point and recovery time target, with an approver and review date |
| Replication configuration export | MirrorMaker 2 configuration (Kafka Connect) or Pulsar geo-replication cluster settings | Source and target clusters, replicated topics, consumer groups and ACL sync are explicitly listed and match the register |
| Replication lag monitoring output | MirrorMaker 2 metrics exported to the monitoring stack, or Pulsar broker and cluster metrics | Lag history covers the audit period, with alert thresholds and records of alerts acted on |
| ACL and authorization export | Kafka StandardAuthorizer metadata or Pulsar authorization policies, exported from the cluster | Mirrored ACLs exist on the standby site and no resource depends on unintended default access |
| TLS and authentication configuration | Broker, controller and client listener configuration, plus certificate inventory | Encryption and SASL mechanisms are enforced on every replication path, including inter-site links |
| Consumer offset translation records | Offset translation output or checkpoint data from MirrorMaker 2, plus consumer group restore logs | Consumers resumed at the expected position after failover, with any duplicate or skipped messages documented |
| Failover and failback test records | Test runbook execution log, timestamps from the orchestration tool and cluster event logs | Timestamps show measured switchover and return times against the recorded objectives, with defects and follow-up tickets |
| Upgrade change records for Kafka or Pulsar | Change management system, release notes and upgrade logs from each cluster | Versions tested on the standby site before production, with rollback steps and approvals for each change |
What buyers ask before hiring help with multi-site streaming recovery
What drives the cost of multi-site streaming recovery?
The cost is driven by how many streams need recovery, how fast they must come back, how many sites are involved, which platform you run, and how far the failover tests go. Each stream with a different recovery objective adds design and validation work, and a second or third site adds replication links, capacity and network paths to build and monitor. Costs are one-time for design, build and the first test cycle, and recurring for replication infrastructure, cross-site traffic, testing and on-call coverage. Ask any partner for a quote that lists these components separately, with the stream count, target recovery objectives, site count and test scope written out so you can compare proposals on the same basis.
How long does a multi-site streaming rollout take?
The timeline depends on the number of streams, the platform (Kafka with MirrorMaker 2, or Pulsar with geo-replication), the readiness of the target site, and how many failover scenarios must be proven before go-live. Work usually moves in stages: design of recovery objectives and topology, a build on the secondary site, replication and offset translation tests, then one or more failover exercises. Delays most often come from network links, firewall approvals and waiting for application owners to confirm which consumers can switch sites. Fixed-scope engagements agree deliverables and dates in writing before work starts, so ask for a dated plan with each dependency named.
What does the work need from our own team?
Your team must supply decisions and access that no partner can make for you. These include the recovery time and recovery point objectives for each business function, the list of producers and consumer groups that depend on each topic, network routes and firewall changes between sites, and the names of people who approve a failover. Your platform or SRE team also needs to own the cluster after handover, including Kafka or Pulsar upgrades, certificate rotation and capacity changes. Ask the partner to name these inputs in the scope so gaps show up before the schedule slips.
Can our own team build multi-site streaming recovery in-house?
Yes, if your team already runs the chosen streaming platform in production and has people who understand replication, ACLs and TLS across sites. In-house delivery is harder when the team has not yet tested offset translation for consumers, has no written recovery objectives, or lacks a repeatable failover runbook. A partner can be useful for the first design, an independent review of the topology, or a supervised first failover, while your team keeps operating the result. Compare the options by the skills that must be present on day one and after handover, not only by the initial build effort.
How often should failover tests run, and who runs them?
Failover tests should run on a schedule your risk owners approve, and after any significant change to topology, platform version or consumer design. The test scope should cover switching producers and consumers, verifying consumer group offsets after translation, checking ACLs on the secondary cluster, and confirming that recovery objectives are met. Your operations team should run routine tests so the runbook stays current, while an independent party can review or run scenario tests. If you need a formal continuity schedule for a regulated function, confirm the required test frequency with your regulator or assessor, since the answer depends on your obligations.
Is cross-site replication enough without backups?
No. Replication copies data to a second site, so a bad deletion, a corrupted producer, or a misconfigured topic can be replicated too. MirrorMaker 2 replicates topics with their configuration, consumer group offsets and ACLs, but it does not give you point-in-time restore of a topic to an earlier state. A backup approach that restores topics and metadata from a separate copy, stored outside the replication path, addresses that gap. Ask any partner how they will protect against logical errors as well as site loss, and how backups are tested for restoration.
Which platform should we choose for geo-replication, Kafka or Pulsar?
Kafka with MirrorMaker 2 is the common choice where teams already run Kafka and Kafka Connect, and it replicates topics, consumer groups and ACLs between clusters. Pulsar offers geo-replication as a built-in feature and separates brokers from BookKeeper storage, which changes how capacity and failover are designed. Your choice should follow existing skills, the Kafka or Pulsar client ecosystem you depend on, and the support window of the release you plan to run. Ask the partner which versions they will deploy and confirm they are inside a supported release window before you commit.
What should we ask for in a multi-site recovery proposal?
Ask for a written scope listing the streams, sites, recovery objectives, replication method, offset translation approach and failover tests included. Request the deliverables for each phase, the runbooks and their owners, the handover plan, and the acceptance criteria for a failover test. Check how security controls such as TLS and ACLs are applied on both sites, and how upgrades will be handled across clusters. A proposal that cannot explain these items in plain terms is a signal to ask more questions before signing.
Questions to ask an implementation partner about failover
Start with the objectives, because every other answer depends on them. Ask the partner to show written recovery time and recovery point objectives for each stream or topic group, and to explain how those objectives map to the replication pattern chosen. If the answer is that objectives will be agreed later, or that replication lag is the only target, treat that as a warning. Ask how they test offset translation: which tool or script maps consumer group offsets between clusters when using MirrorMaker 2 (the checkpoints topic is the usual starting point to confirm), what happens to consumers that resume on the secondary, and whether they have measured the duplicate or skipped message window in a real switchover. Ask for the last test record, not a description of the design.
Next, ask how client switchover works and who runs it. A strong answer names the mechanism that moves producers and consumers (DNS, a service registry, or configuration with bootstrap servers), the decision owner, and the rollback path. Ask how failback is handled, since replication direction must be reversed and offsets reconciled before traffic returns. Ask what happens to ACLs and TLS on the secondary site: whether ACLs in the KRaft metadata log replicate with the topics, whether certificates and SASL credentials are provisioned independently, and how the secondary is kept from becoming an unprotected path into the data. Ask how upgrades are coordinated across both sites, including the Kafka or Pulsar version order and the rolling upgrade limits in the vendor's release plan.
Finally, ask what handover looks like. Expect a runbook with failover and failback steps, named roles, a test calendar, and documented evidence from each exercise. Be concerned if the partner cannot show a completed failover test, if the runbook has never been run by someone other than its author, or if monitoring of replication lag and consumer lag on the secondary is missing. Ask for a written scope that states which tests are included, how many, and what the client must provide, such as a second site and a maintenance window.
Ask the partner which failure modes the test plan covers. A full site loss, a broker loss, a network partition between sites, and a corrupted topic each need different steps, and a plan that tests only the first is incomplete. Ask whether they will run a switchover under load with realistic consumer groups, and how they verify that no event is lost on the primary before the cutover. Ask how they handle split-brain risk when both sites are reachable but the link between them is down, and who has the authority to declare a failover. These questions show whether the partner has thought through the decision, not only the tooling. For each answer, ask for evidence you can check: test reports with dates and measured results, the exact replication configuration and its change control, and the list of settings that differ between sites. Confirm the versions the partner intends to use against the current supported releases on the Kafka downloads page and the Pulsar release policy before you agree on a design, because support windows affect how long a site can run an older release. Ask what the partner will hand over at the end of the engagement and what they will support afterward, and whether the failover test is repeated after each major upgrade. An illustrative red flag: a proposal that promises zero data loss without describing how it is measured.
Integration points and deployment patterns that decide recovery
A site failure exposes the components that replicate, route and store data between sites, not only the brokers themselves. The cards below cover the replication layer, the metadata and storage layers, the network between sites and how cloud accounts or data centers are separated, since each one decides whether a failover is clean or partial.
Kafka Connect as the replication runtime
MirrorMaker 2 runs on Kafka Connect, so the replication jobs are connectors with their own tasks, worker group and offsets. If the Connect cluster sits in the failed site, replication stops even when both brokers are healthy. Check where the workers run, how the REST interface is secured and whether a standby worker group can take over the connector configuration. Confirm how your team will restart connectors after a site loss.
REST administration as a control point
Kafka Connect is administered through a REST interface and does not provide a graphical console, so connector configuration, status checks and restarts run through the API or third-party tools. During an incident, the team that operates failover depends on that API being reachable from the surviving site. Ask who holds the credentials, where the configuration is stored in version control and whether a runbook covers API access without the primary site.
Pulsar metadata store placement
A Pulsar cluster depends on a metadata store, which can be Oxia (recommended for new clusters), ZooKeeper or RocksDB. Brokers route messages, but they cannot locate topics without that store. Its quorum layout across sites decides whether a surviving site can keep serving or must wait for the failed site to return. Confirm which store you will run, how it is placed across sites and who owns its backup.
BookKeeper bookies and storage recovery
Pulsar keeps persistent data on Apache BookKeeper bookies, which act as a distributed write-ahead log, while brokers stay stateless. Storage grows by adding bookies independently of brokers, so capacity at the surviving site must be planned for the full write load, not the share it carried before. Check how ensemble and write quorum settings map to sites, how many bookies a surviving site needs to keep writes accepted and how bookie replacement is rehearsed.
Network paths and inter-site bandwidth
Replication is only as current as the link between sites. A saturated or intermittent WAN lets lag grow silently, and the recovery point becomes older than the number on the wiki. Measure sustained replication throughput against peak produce rates, confirm the path is private and encrypted, and check whether the link has an independent route. Ask your network team to document the failure behavior of each path, not only its nominal capacity.
Separated cloud accounts and data centers
Placing the primary and standby clusters in the same cloud account, identity tenant or certificate authority means one administrative mistake or credential compromise can affect both. Separate accounts, networks and key material make the standby site a true alternative. Confirm which control plane, DNS zone and secrets store each site depends on, and whether the standby can authenticate without the primary's identity provider.
How DeployOpen scopes and tests multi-site streaming recovery
DeployOpen plans, implements, hardens, documents and supports self-hosted streaming platforms inside your own data centers or cloud accounts. For multi-datacenter event streaming and disaster recovery, we start from the recovery objectives your business has agreed, then design replication, failover and testing around them. Engagements are fixed-scope, with the scope agreed in writing before work starts.
How we scope
- Agree recovery objectives per stream or business function, and confirm which topics must survive a site loss
- Map the sites, network links, identity and certificate dependencies that replication and failover must cross
- Write the scope covering replication pattern, consumer offset handling, failover runbooks and test plan before work starts
How we build
- Deploy the chosen replication path, such as MirrorMaker 2 for Kafka or geo-replication for Pulsar, in your environment
- Configure TLS, SASL and ACLs on both sites so replicated topics keep the same access controls as the source
- Build and verify consumer offset translation so applications can resume on the secondary site without reprocessing unexpectedly
How we hand over
- Document the architecture, replication topology, failover and failback procedures, and the monitoring for replication lag
- Run failover tests against the agreed objectives and record measured results, gaps and remediation actions
- Train your operators on runbooks, alerts and the upgrade path for both clusters, with written handover acceptance
How we support
- Offer ongoing support, upgrades and periodic failover testing as a retainer after handover
- Repeat recovery tests after significant changes such as new topics, version upgrades or network changes
- Confirm with your team the support hours, escalation paths and test frequency in the written scope
