Ceph RBD mirroring for disaster recovery

When Ceph RBD mirroring for disaster recovery fits your recovery plan

Choose rbd-mirror when your virtual machine disks or database volumes live on one Ceph cluster and you need a second copy of those RBD images at another site. The daemon replicates images asynchronously to a peer cluster. It does not decide when to fail over, move your virtual machines or databases, reconfigure clients, or keep dated restore points the way a backup system does. Those remain your job, and the recovery plan has to name who performs each step and in what order.

Backups alone are often enough when a multi-day restore is acceptable and you mainly need protection against deletion, corruption or ransomware. Replication copies every change, including accidental deletes and corrupted writes, so it protects against site loss but not against bad data reaching both clusters. Pair it with snapshot retention or an independent backup target. Stretch mode is the other option. It runs one cluster across two sites with synchronous writes, which suits a two-site design with low latency between them. Mirroring is the better fit when the sites are farther apart, when the two clusters must be operated independently, or when a site outage must not stall writes on the primary.

Your recovery point depends on replication lag, not on a fixed schedule. Asynchronous replication means that writes acknowledged at the primary may not yet exist at the secondary when the primary site fails, so the data you lose is the lag at the moment of failure. Before you commit to a recovery point objective, measure lag under your real write load with the mirroring status tools and confirm with your Ceph release which mirroring modes and status fields apply to the version you plan to run. Ask your team to document the lag threshold that triggers an alert and the procedure for a planned failover, since that lag figure is what your business must accept.

Journal mode versus snapshot mode for RBD images

This table compares journal-based and snapshot-based rbd-mirror replication across capture method, recovery point, primary overhead and day-to-day operations. Read it per image: the mode you pick sets your recovery point objective and how much orchestration your team owns during a failover.

Aspect	Journal mode	Snapshot mode
How changes are captured	Every write to the image is recorded in the RBD journal before it is applied, and the peer replays that journal in order.	The primary creates mirror snapshots on a schedule or on demand, and only the differences between snapshots are sent to the peer.
Recovery point on the target	Crash-consistent up to the last replayed journal entry, so the target can usually be brought up near the moment of failure.	Point-in-time as of the last completed mirror snapshot, so anything written after that snapshot is lost.
Write overhead on the primary	Higher, because each write also goes through the journal, which adds I/O and capacity on the primary pool.	Lower per write, with cost concentrated in snapshot creation and the data transferred between snapshots.
Suits which images	Images that need the smallest achievable data loss window, such as databases and VM disks with strict recovery targets.	Images where periodic copies meet the recovery objective, such as large, slowly changing datasets or secondary backups.
Operational steps to set up	Enable journaling on each image, size the journal, add peers, then enable mirroring for the pool or image.	Enable snapshot-based mirroring on each image, set a mirror snapshot schedule, add peers, then monitor schedule completion.
Failover and promotion	Promote the image on the peer after confirming replay is caught up, then point clients at the promoted copy.	Promote from the latest completed mirror snapshot, accepting that writes after it are not on the peer.
Monitoring to watch	Replay lag and journal growth on the primary, since a stalled peer can hold journal data.	Snapshot schedule success, snapshot count and the age of the last completed mirror snapshot.
Confirm for your release	Check the RBD mirroring documentation for your Ceph release for journal feature requirements, supported image features and failover commands.	Check the same documentation for schedule syntax, snapshot retention behavior and any feature limits on your release.

How rbd-mirror moves image changes between clusters

Replication between two Ceph clusters depends on a small set of components that you must place, authenticate and watch. The rbd-mirror daemon runs on the peer side and pulls image changes from the primary cluster, applying them to the image on the secondary. Mirroring is enabled per pool or per image, and the mode you pick determines which images the daemon handles. Before any data moves, the two clusters must be bootstrapped as peers, which means exchanging credentials for a mirroring user and registering each cluster with the other. Confirm the exact peer bootstrap commands, the required cephx capabilities and the daemon placement for your Ceph release in the RBD mirroring section of the Ceph documentation, because these steps have changed between releases.

The data path is a network path, not a local operation. The rbd-mirror daemon connects from the secondary site to the primary's monitors and OSDs, so the firewall rules, routing and bandwidth between sites decide how far behind the copy can fall. Traffic between clusters uses the Ceph messenger, and whether it is encrypted depends entirely on configuration. The msgr2 protocol offers a secure mode that encrypts all post-authentication traffic with AES-GCM and adds integrity checks, as described in the Ceph messenger v2 documentation. Set and verify that mode on both clusters and on the network path, and do not assume encryption is on because the protocol supports it.

Replication health is reported by the cluster, and you should read it instead of assuming the secondary is current. Use the mirror status commands to check each image's state, its description and the time of its last update, and treat a stalled daemon or a growing gap as an incident. Replication is asynchronous, so the secondary always trails the primary by some amount, and that gap is your recovery point exposure. Build alerts on mirror state and lag, run periodic test restores against the secondary, and document the lag you consider acceptable so that a failover decision is based on measured data.

One-way or two-way mirroring by recovery goal

In a one-way design, every mirrored image has one primary copy on the production cluster and a non-primary copy on the disaster recovery cluster. Applications write only to the primary. The recovery copy receives changes from the primary and stays read-only to clients until someone promotes it, so write authority stays in one place and split-brain is unlikely by design.

Write location: Hypervisors, Kubernetes workloads and applications write only to the production cluster. The recovery cluster is a standby target, and clients should not be attached to its images.
Split-brain risk: Low by design, because only one side holds primary for an image. The remaining risk is procedural, such as promoting the recovery image while production is still running.
Failover: Promote the recovery image after confirming the production site is lost or intentionally stopped. If the primary cannot be reached, promotion may need to be forced, and the recovery point is whatever had replicated before the loss.
Failback: Demote the recovered site, let it resync from the new primary, then promote it back in a planned window. Writes made on the recovery site while it was primary must be carried back or accepted as lost.
Idle capacity: The recovery cluster carries hardware that sits mostly idle, so its sizing and refresh cycle are a budget decision.

Confirm with the business owner how long a site can be down before service must return, how much recent data may be lost, and who has authority to declare a failover.

Setting up RBD mirroring between two clusters

This sequence takes a primary and a secondary Ceph cluster from supported releases and a deployed rbd-mirror daemon to a working peer relationship, with chosen pools or images replicating and their status confirmed on both sides. Confirm exact commands and capability settings against the release in use before running any of them.

Confirm both clusters run a supported release
Check that the primary and secondary clusters both run an actively maintained Ceph release, such as Tentacle (20.2.x) or Squid (19.2.x), and that each reports a healthy state before you change anything. Record the exact versions, the health output and any open warnings for each cluster. Read the release notes for your target version to confirm whether the two sites may run different releases during the setup. Done when both versions and health states are written in the change record and no unresolved warnings affect the pools you plan to mirror.
Deploy the rbd-mirror daemon with cephadm
Deploy the rbd-mirror daemon as a cephadm-managed service on the cluster that will receive mirrored images, and on the primary too if you want replication in both directions. Place the daemons on hosts that can reach the peer cluster's monitors and OSDs over the network you intend to use for replication. Running more than one daemon can add resilience, so confirm how your release distributes work across them. Use the orchestrator placement command from the cephadm documentation for your release. Done when ceph orch ps lists the rbd-mirror daemons as running and the cluster health report does not flag them.
Plan pools and images for mirroring
Decide which pools hold the block images that need a second copy, and whether each pool will mirror in pool mode (all images with mirroring enabled) or image mode (selected images only). Confirm that the pool names match on both clusters, because the peer must find the same pool by name. Choose journal or snapshot mode for each image based on the recovery point and write-load trade-offs described above, and list any images that are excluded. Done when you have a written table of pools, images, mode and owner, and the team responsible for the peer side has signed off on it.
Bootstrap the peer with a dedicated mirror user
Create a dedicated Ceph user for mirroring on each side, granted only the capabilities the mirror daemon needs, and keep administrator keys out of the peer configuration. Generate a bootstrap token on the source cluster and import it on the destination to establish the peer relationship for each mirrored pool. Confirm the capability strings and the bootstrap command syntax in the RBD mirroring documentation for your release before you run them. Done when rbd mirror pool info on each side lists the other cluster as a peer with a site name and the expected direction.
Enable mirroring on pools or images
Enable mirroring on the chosen pools in the mode you selected, then enable it on individual images if you chose image mode. Enable journaling on any image that uses journal mode before mirroring starts, since journal mode depends on it. Confirm that each image to be protected exists on the primary and that the peer is reachable before you enable it. Done when rbd mirror image status on the primary lists the planned images in an expected state and the secondary begins to show them as they replicate.
Verify mirror status on both sides
Run the pool and image status commands on the primary and on the secondary, then compare the reported state, description, last update and peer connection for each image. Confirm that the rbd-mirror daemons are connected to the peer and that no image shows an error or a sync that has stopped progressing. Save this output as your baseline so you can tell later whether a lag is new. Done when both sides report the expected pool mode, every planned image appears, and the status shows no errors.
Test a write and confirm arrival
Create a dedicated test image on the primary, write a known data pattern to it, and wait for the image to appear on the secondary. Compare a checksum of the data on the primary with the copy on the secondary using a read method your release supports. Note how long the change took to arrive and record it as a baseline. Remove the test image from both sides when the check is complete. Done when the test data matches on both clusters and the status output shows the image as replicated.

Security controls for cross-site block replication

Replicating block images to a second cluster creates a second path into your data and a second place where failover can go wrong. The controls below cover who or what can read and write across that link, how the traffic is protected, which network carries it, and how failover is recorded. Check each one against your own threat model before the peer is bootstrapped.

Dedicated peer cephx identity

Create a separate cephx user for the rbd-mirror peer on each cluster instead of reusing an administrative key or an application client. Grant it only the monitor and OSD capabilities that rbd-mirror needs for the pools being mirrored, and confirm the exact caps against the current RBD mirroring documentation for your release. Keep it apart from application users so that revoking one identity does not interrupt the other.

Encrypted replication traffic with msgr2

Replication data crosses a link you usually do not own end to end. Ceph's messenger v2 protocol on port 3300 has a secure mode that encrypts post-authentication traffic with AES-GCM and adds integrity checks. Confirm that secure mode is enabled on both clusters and that the peers negotiate it. Plan for the CPU cost of encryption on busy links and for the fact that inter-site throughput, not the cipher, is usually the limit.

Separate replication network path

Keep rbd-mirror traffic off the client and cluster networks where possible, using a dedicated routed path or VLAN between sites. Ask your network team to confirm firewall rules, MTU and bandwidth reservations for the replication path, and to bond links across two switches as the Ceph hardware guidance advises. A congested replication link slows recovery points without any error, so monitor it as a separate service.

Rotating peer credentials

Set a rotation schedule for the peer cephx key and the bootstrap token used to pair the clusters, and record who owns each step. Rotation means generating a new key, updating the peer on the other site, and then removing the old key, so write the sequence down and rehearse it in a lab. Confirm how your cluster stores these keys and who can read the configuration that holds them.

Guarding secondary images

Mirrored secondary images are not meant to be written to during normal operation, but a client that maps the wrong image can still change data. Restrict client capabilities on the secondary pool so that only the failover runbook and approved operators can promote an image or write to it. Confirm with your hypervisor or orchestration team which clients can attach to secondary images, and test that a write attempt is refused.

Audit trail for failover actions

Promotion, demotion, resync and peer changes are the actions most likely to be questioned after an incident. Record who ran each command, when, from which host, and why, and keep those records outside the cluster being failed over. Ask your auditors which evidence they expect for change and access control, and confirm that your logging captures the mon and rbd CLI activity you rely on.

Failover and failback steps that RBD leaves to your runbook

rbd-mirror replicates images, but it does not decide when to fail over. It does not promote images on your behalf, fence a failed primary, remap clients, or restart the virtual machines and databases that sit on top of the images. Those steps belong to your runbook, and most of the effort in a Ceph RBD disaster recovery design lands there. Before you write anything, confirm with the Ceph documentation for your release how mirrored images are promoted and demoted, and which commands your team will run, because the exact procedure is a version-dependent detail you should verify rather than assume.

A workable runbook covers five stages in order: stop or fence the primary so that no client keeps writing to it, promote the secondary images, redirect clients to the secondary site, verify application consistency, and later demote the images and resync for failback. Each stage needs a named owner, a written check, and a rollback point. Consistency checks are where teams often underestimate the work, since a crash-consistent block image still needs database recovery, filesystem checks and service restarts. Rehearse the full sequence on a non-production pool at least once per planning cycle, and record how long each stage took on your hardware.

Replication lag sets how much recent data you can lose. Changes reach the secondary only after they are replicated, so whatever was written on the primary but not yet mirrored at the moment of failure is at risk. Measure lag continuously with the mirror status tooling, set an alert threshold your business owners have signed off on, and treat a lag breach as an incident before any disaster happens. Confirm with your storage team whether your link capacity and journal settings can keep up with peak write rates, because a secondary that trails during busy periods gives you a recovery point that is weaker than your documented objective.

Questions teams ask before they hire help with RBD mirroring

How much effort and cost does an RBD mirroring project involve?

The cost depends on how much work the scope contains, and most of that work is planning and testing rather than typing commands. The main drivers are the number of images and pools to replicate, the number of clusters involved, the network link between sites and how much of it is already measured, the application runbooks that must be written or updated for failover, and the number of failover and failback tests the business requires. Each of these is a line item in a fixed-scope proposal. Ask for a quote that separates design, build, runbook writing, testing and any ongoing support, and that states which items are one-time and which recur, such as monitoring and periodic drills. DeployOpen does not publish prices, so the figure comes from the written scope.

How long does a scoped RBD mirroring engagement take?

Duration is set by the scope, not by the mirroring feature itself, which is quick to enable on a single image. The length of an engagement depends on how many pools and images need replication, whether the second cluster already exists, how long the network assessment and change approvals take, how many application teams must validate their recovery steps, and how many failover tests are planned. Ask the proposal to list each phase, the client input each phase needs, and the dependencies between them, so that delays in approvals or access show up in the schedule rather than as surprises.

What does the work need from our team?

The work needs named people from several groups, and the engagement moves at the speed of their availability. Expect to provide:

Storage owners who can grant access to both clusters, approve pool and image changes, and schedule maintenance windows
Network owners who can confirm bandwidth, latency and firewall rules between the sites
Application owners who can describe their databases and virtual machines, confirm what consistency they need, and sign off on recovery steps
Change approvers who can authorize failover drills in production or in an agreed test environment

Ask for a written responsibilities matrix in the scope so each dependency has an owner and a date.

Can our internal team set up RBD mirroring without a partner?

Yes, a team with Ceph operations experience can enable and test rbd-mirror, and the Ceph documentation covers the commands. The harder part is the orchestration that RBD leaves to you: deciding between journal and snapshot mode for each workload, defining the promotion and demotion order, scripting or documenting failover and failback, handling split-brain risk, and proving that applications recover correctly. Assess your team honestly on those points and on whether it can run drills without disrupting production. Many organizations split the work, keeping cluster operations in-house and buying design review, runbook development and test support.

How does RBD mirroring differ from backups and from stretch mode?

RBD mirroring replicates block images to a second cluster so that a site failure can be recovered from a secondary copy with a short switchover. It is not a backup: deletions, corruption and ransomware writes replicate too, and a mirror does not give you point-in-time history on its own. Backups keep independent, versioned copies that can be restored after logical damage, and they should still exist alongside mirroring. Stretch mode runs one cluster across two sites with a tiebreaker monitor and handles site loss inside a single cluster, which has different failure domains, distance limits and operational rules. Choose based on your recovery objectives, the failure you are protecting against, and the distance between sites.

How do we verify that replication works after go-live?

Verification combines the status that rbd-mirror reports with application-level checks. Confirm that each mirrored image shows a healthy replication state on both clusters, that the replay lag stays within the recovery point you agreed with the business, and that alerts fire when mirroring stops or falls behind. Then run scheduled failover and failback drills on representative workloads, and check that applications start, data is consistent and the reverse replication works afterward. Ask for the monitoring metrics, alert thresholds and drill schedule to be written into the handover documents, and confirm the exact status fields and commands for your Ceph release in the official documentation before you build alerts on them.

What should we ask for in a written scope for this work?

A useful scope names the clusters, pools and image counts, the replication mode chosen for each workload and the reason for it, and the recovery point and recovery time targets the business has agreed. It should list the network assessment, the failover and failback procedures to be written, the drills to be run and who attends them, the handover material, and any ongoing support or retainer. It should also state what is excluded, such as application changes or backup redesign, and which client roles must be filled before work starts. Fixed-scope proposals are agreed in writing before work begins, so use the scope to test whether the plan covers the outcomes you need.

Limits to confirm in your environment before you rely on it

Mirroring limits depend on the Ceph release you run, the image features you enable, the hardware between sites and the platform that consumes the images. None of these can be settled from a design document alone, so each card below names what to check and where to check it.

Supported features and image types

Confirm which image features, image types and mirroring modes your running release supports before you design around them. Support for journal-based and snapshot-based mirroring, and for particular image features, has changed across releases, so the release on your cluster is the reference, not an older tutorial. Read the release-specific RBD mirroring section of the Ceph documentation for the version you run, then reproduce your image configuration in a lab cluster.

Image count and mirror daemon capacity

Measure how many images the rbd-mirror daemon can replicate within your recovery objective, not how many it can hold. Image count, change rate per image and the number of daemons all affect how quickly the queue drains after a link outage. Run a load test with production-like image counts and write patterns, watch daemon status and queue depth, and record the time to fully catch up after a simulated outage.

Encrypted images and clones

Test mirroring of encrypted images and of clones before relying on them for recovery. Encryption keys must be available on the secondary cluster for a failed-over image to open, and clone parent and child relationships must be preserved or rebuilt correctly at the peer. Check the Ceph RBD encryption and clone documentation for your release, then perform a failover drill on an encrypted clone and confirm the image mounts and reads correctly.

Replication lag under your write rate

Calculate replication lag from your actual write rate and inter-site link capacity, then verify it under load. Journal-based mirroring can add write overhead on the primary, and sustained write bursts can grow the backlog beyond your recovery point objective. Measure the mirror status lag over a representative peak period, and document the lag you can accept during a site-wide event on the link you have, not the link you planned.

Snapshot schedules and retention

Confirm how scheduled snapshots and retention rules interact with mirror snapshots on both clusters. Snapshot-based mirroring creates its own snapshots, and retention pruning that removes a snapshot the mirror still needs can stall replication or force a full resync. Review the RBD snapshot schedule documentation for your release, then run a test that prunes aggressively and confirm replication continues without a full resync.

Hypervisor and Kubernetes driver behavior

Confirm whether your hypervisor or Kubernetes storage driver can reattach a failed-over image without manual steps. Mirroring replicates the image, but it does not start virtual machines, update VM configuration, or rebind Persistent Volumes on the secondary site. Read the vendor documentation for your platform and the Ceph-CSI or Rook documentation for your release, then rehearse a full failover that includes the consuming workload, not just the image promotion.

How DeployOpen scopes and delivers RBD mirroring

DeployOpen plans and builds RBD mirroring inside infrastructure you control, starting from the recovery objectives your business has agreed. The work covers the mirroring design, the failover and failback runbooks, and the testing that shows whether the second cluster can actually take over.

How we scope

Confirm which pools and images need mirroring, and which applications depend on them.
Record recovery point and recovery time targets agreed with your business owners before design starts.
Define whether the setup is one-way or two-way, and whether it uses journal or snapshot mode.

How we build

Configure rbd-mirror daemons, peer bootstrap and pool mirroring on both clusters, under a written plan.
Test failover and failback against a non-production image set before any production image is enrolled.
Build the orchestration scripts or runbooks that RBD does not provide, such as promotion order and client reattachment.

How we hand over

Document the mirroring topology, replication lag monitoring, and the steps for promotion and demotion.
Run a recorded failover drill with your operations staff and capture the results against the agreed targets.
Deliver the runbooks, configuration files and monitoring alerts so your team can operate the setup without us.

How we support

Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts.
Ongoing support, upgrades and repeat failover testing are available as a retainer.
Ask any partner to confirm how mirroring behaves across the Ceph release you plan to run.