How CephFS snapshot mirroring protects a file system at a second site
CephFS snapshot mirroring replicates snapshots of selected directories to a remote CephFS file system on a schedule, asynchronously and push-based, through the cephfs-mirror daemon. Teams need it when a CephFS file system holds data that must survive the loss of a site, but the business can accept a recovery point that lags the primary by the replication interval. The source cluster pushes snapshot deltas outward, so the primary keeps serving clients at full speed and the remote site does not have to acknowledge writes before they are committed locally.
The model differs from the other Ceph options for two sites. Stretch mode runs one Ceph cluster across two data centers with a tiebreaker monitor, so writes are synchronous across both sites and a site loss is handled inside the cluster, but it needs low-latency links and a different design. RBD mirroring replicates block images, and it works at the block layer without knowing what files exist, which makes it a poor fit when you need to restore individual directories from a consistent file-level snapshot. CephFS mirroring works at directory granularity, so you choose which trees go to the second site and you recover from the remote snapshots you have already taken and replicated.
The recovery point you can promise is set by snapshot frequency and replication lag, not by the mirror itself. Before you commit to a target, confirm with your own testing how long a full synchronization and a delta cycle take for your directory sizes and link capacity, and decide who owns the snapshot schedule that feeds the mirror. The mirror also needs both clusters on Pacific or later, a peer added with the recommended peer_bootstrap method, and a user with the right capabilities on each side, so confirm those prerequisites against the CephFS mirroring documentation before you design the recovery procedure.
Mirroring configuration options and when each fits
This table compares the main CephFS mirroring choices: how directories are selected, how peers are added, how many mirror daemons run, and where snapshot schedules come from. Read each row against your recovery point objective and your operations capacity, then check the trade-off column before settling on a configuration.
| Decision | Option | When it fits | Trade-off for recovery planning |
|---|---|---|---|
| Peer setup | peer_bootstrap (recommended) | Most new deployments where both clusters are under your control and an operator can run the bootstrap steps in sequence. | Token handling is part of the procedure. Plan who holds the bootstrap token and how it is rotated or revoked after setup. |
| Peer setup | Manual peer addition | Environments where the bootstrap flow does not fit your change process or you need explicit control over each peer specification. | More steps to get wrong. Verify the peer on both sides and record the exact settings so a failover runbook can reproduce them. |
| Mirror daemons | Single cephfs-mirror daemon | Smaller file systems or modest change rates, where the project's recommendation of one daemon keeps operations simple. | One daemon is a single point of mirroring failure. Monitor its health and confirm how quickly you can restart or redeploy it. |
| Mirror daemons | Multiple cephfs-mirror daemons | Larger directory sets or higher snapshot volume that a single daemon cannot keep current, with capacity to operate more processes. | Confirm the supported configuration for your release before adding daemons. Each extra daemon adds placement and monitoring work. |
| Directory scope | Mirror selected directories | Recovery targets only cover critical trees such as project data or shared application storage, and the rest can be rebuilt. | Directories outside the list are not replicated. Keep the list under change control so new critical paths are added. |
| Directory scope | Mirror the whole file system | Recovery must cover everything in the file system with a single rule and the team wants fewer per-directory actions. | Replicates data you may not need to recover quickly, which adds load and remote capacity use. Confirm the remote sizing. |
| Snapshot source | Snapshots created by snap_schedule | Your recovery points should follow a fixed interval with automatic retention, using the snapshot schedule module. | Mirroring replicates snapshots, so the recovery point is only as recent as the last snapshot that reached the remote cluster. Align schedule and retention with mirroring lag. |
| Snapshot source | Snapshots created by scripts or applications | Application-consistent points are required, such as after a database flush, and an external process controls timing. | You must ensure the script runs on schedule and that snapshot names and retention are tracked outside Ceph. Test restores from these points. |
Setting up CephFS mirroring between two clusters
This sequence sets up asynchronous snapshot replication of selected CephFS directories from a primary cluster to a secondary cluster. When it finishes, snapshots taken on the primary reach the secondary, and the storage team can state which snapshot is the latest recovery point.
Confirm both clusters run Pacific or later
The storage team runs
ceph versionson the primary and the secondary cluster and checks that every daemon reports Pacific or a later release. Mirroring depends on this baseline, so the check comes before any configuration change. Record both cluster names, the CephFS file system names to replicate, and the directories in scope so that later steps refer to the same objects. The step is complete when both clusters report a supported version and the plan lists each file system and directory.Enable mirroring on the source file system
On the primary cluster, the storage team enables the mirroring manager module with
ceph mgr module enable mirroring, then enables mirroring for the file system withceph fs snapshot mirror enable <fs_name>. Mirroring is switched on per file system, so each file system to be replicated needs this command. Check the module list to confirm mirroring is active, then confirm the file system reports mirroring as enabled. Review the CephFS snapshot mirroring documentation for your release if the command output differs from what this sequence describes.Deploy the cephfs-mirror daemon
The team deploys one cephfs-mirror daemon with cephadm, for example through
ceph orch apply cephfs-mirror, and the docs recommend a single daemon. Confirm in the mirroring documentation for your release which cluster should host the daemon, then place it there. Useceph orch psto confirm the daemon is running on the expected host. The step is complete when the daemon shows as running and its host is recorded in the runbook.Create a peer bootstrap token on the primary
On the primary cluster, the storage team runs
ceph fs snapshot mirror peer_bootstrap createfor the file system, naming the remote client user and a site name. The command creates the user with the capabilities mirroring needs and returns a bootstrap token. Review the capabilities granted to that user against your access policy before continuing. Treat the token as a secret and move it only through your approved channel. The step is complete when the command returns a token and the user's capabilities have been reviewed.Import the token to create the peer
On the secondary cluster, the team runs
ceph fs snapshot mirror peer_bootstrap importwith the target file system and the token from the previous step. The import registers the primary as a peer so the two clusters can exchange snapshot data. Once the import finishes, runceph fs snapshot mirror peer_liston the primary to confirm the peer is listed and points to the secondary. The step is complete when the peer appears in the list with the expected site name.Add directories to mirroring one at a time
The team adds each directory that needs disaster recovery with
ceph fs snapshot mirror add <fs_name> <path>, one command per directory. Directories that are not added are not replicated, so the list should match the recovery scope agreed earlier. After each addition, check the daemon status to confirm the directory is registered and not in an error state. The step is complete when every directory in scope appears in the status output with no errors.Verify snapshots arrive on the secondary
On the primary, the team creates a test snapshot in a mirrored directory through the
.snapdirectory, then waits for the sync interval to pass. On the secondary, they check that the same snapshot appears in the mirrored path. Replication is asynchronous, so the snapshot may appear after a delay, and the latest synced snapshot is the recovery point. Record that point and the peer and daemon status output as evidence. The step is complete when the test snapshot is confirmed on the secondary and the recovery point is documented.
Which user capabilities the mirror and peer accounts need
CephFS mirroring uses several cephx identities, one on each side of the replication link and one for the peer relationship itself. Each identity should hold only the capabilities its role requires, because a single over-broad key can read or change data on both clusters.
The mirror daemon's local identity
The cephfs-mirror daemon authenticates to the cluster it runs in and reads snapshots from the primary file system. Confirm the exact capability strings for this identity in the mirroring documentation before you create it, since the daemon needs read access to the file system and cluster status, not write access to user data. Check which cluster hosts the daemon, because that decides which side's keyring it needs.
The primary side read path
On the primary cluster, the identity used for mirroring must be able to read the directories being replicated and the snapshots taken on them. Grant access to the specific file system rather than to the whole cluster. Ask your team to verify that snapshot creation stays with the owner of the snapshot schedule, so the mirror identity cannot create, delete or rename snapshots by accident.
The secondary side write path
On the secondary cluster, the identity that receives mirrored data needs enough access to create and update the replicated directories and snapshots in the target file system. Keep this grant scoped to the target file system and, where practical, to the directories being replicated. Review it whenever the set of mirrored directories changes, because each new path widens the reach of the same key.
The peer bootstrap token
The peer_bootstrap workflow creates a token that lets one cluster register the other as a mirroring peer. Treat that token as a credential: it grants a path into the remote cluster, and anyone who holds it can attempt to register a peer. Transfer it over a channel you control, use it once for its intended purpose, and confirm the expiry and revocation steps with the mirroring documentation.
Scoping and rotating keys
Mirror credentials are long-lived service keys, so plan their rotation before go-live. Record which identity belongs to which cluster and which file system, and document who can issue or read the keyring. Over-broad grants such as full administrative caps on either side turn a replication fault into a risk to every pool and file system, so audit the capabilities after each change.
Confirming capability strings
Capability syntax changes between Ceph releases, so exact cap strings should come from the documentation for the release you run, not from older blog posts or this summary. Before rollout, create the identities in a lab cluster, run a full sync, and check that the daemon can replicate without extra grants. Keep the verified strings in your change record so later reviews can compare them.
What asynchronous snapshot replication means for recovery points
The most important thing to understand about CephFS mirroring is that the secondary cluster only holds data that has already been replicated as a snapshot. Your recovery point is therefore the last mirrored snapshot, not the last write made on the primary file system. Anything written after that snapshot was taken and transferred is absent from the secondary, and a failover will roll the directory back to that earlier state. Because mirroring is asynchronous and push-based, a recovery plan should assume a window of unreplicated change at all times, and the size of that window is a business decision you need to document before go-live, not after an incident.
Snapshot schedules set the granularity of that window. The cephfs-mirror daemon replicates snapshots that exist on the source directory, so a schedule that takes snapshots every hour can never give you a recovery point finer than an hour, regardless of how fast the network is. Check which snapshot schedule and retention rules apply to each mirrored directory, confirm that the snapshot names and cadence match what your recovery objectives say, and make sure the retention on the primary keeps at least the snapshots the mirror has yet to transfer. A failed or lagging mirror widens the gap in two ways: the newest snapshot on the secondary falls further behind the primary, and a backlog of snapshots can be pruned on the source before it reaches the remote side. Monitor the mirror status for each directory, alert on stalled or failed synchronization, and treat a daemon that is down as a recovery point incident.
A recovery test should measure what you would actually lose, not whether the job ran. For each mirrored directory, record the timestamp of the newest snapshot present on the secondary, compare it with the time of the simulated failure, and confirm that the data in that snapshot opens and passes your application checks. Ask your team or your partner to show the mirror status output, the last successful synchronization time and any warnings, and to demonstrate how a directory is promoted and written on the secondary. Confirm the Ceph release behavior for your version with the CephFS mirroring documentation and test restores at the cadence your recovery objectives require, since a mirror that was healthy last month says little about today.
Security and operational considerations for mirrored directories
Mirroring copies data between two clusters, so each link in the chain needs an owner: the trust between clusters, the credentials that let the mirror daemon act, the network path it uses, and the monitoring and upgrades that keep it working. The cards below cover the decisions and checks that connect those areas, and the points to confirm with your own team before go-live.
Trust between the two clusters
A peer relationship lets one cluster's mirror daemon read snapshots from the other, so treat the remote cluster as a trusted party. Bootstrap peers with peer_bootstrap and keep the bootstrap token out of shared tickets and chat logs. Confirm which administrators on each side can add, remove or change peers, and record who approves a change. Because the source cluster pushes data out, a compromise of the remote cluster should not grant write access to primary data, so review what the peer account can change.
Cephx credentials for the mirror
The mirror daemon authenticates with cephx, and its user should hold only the capabilities needed to read the mirrored file system and its snapshots. Avoid reusing an administrator keyring for mirroring. Set up a dedicated key per direction, store it through your normal secret handling, and rotate it on a schedule you can audit. Ask your team to confirm that the capability set matches the current CephFS mirroring documentation for the release you run, since capability requirements change between versions.
Network path between sites
Mirroring traffic crosses the link between data centers, so the path needs to be encrypted, firewalled and sized for the change rate of the directories you protect. The messenger v2 secure mode encrypts post-authentication traffic, and enabling it should be verified on both clusters rather than assumed. Restrict the mirror daemon's source addresses at the firewall, and document which ports and addresses are open between the two sites so that a network change does not silently pause replication.
Deleted and unmirrored directories
Mirroring is configured per directory, so a directory that is never added is never replicated, and a directory removed from the mirror set stops receiving new snapshots. Decide in advance what happens when a project directory is deleted on the primary: the remote copy keeps its last snapshots until someone removes them, and that retention is a recovery asset and a cost. Write down the approval step for removing a directory from mirroring and for deleting remote snapshots.
Monitoring mirror status
Check mirror status regularly rather than waiting for a restore to reveal a gap. Alert when a directory stops syncing, when the daemon restarts repeatedly, or when the most recent replicated snapshot falls behind the schedule you promised to the business. The recovery point you can actually restore is the last snapshot that finished on the remote side, so alerts should measure that age. Confirm which status fields your Ceph release exposes before building dashboards on them.
Supported releases on both clusters
Both clusters must run Pacific or later, and each side should stay on an actively maintained release line so that fixes reach the mirror path. Plan upgrades so that the remote cluster is not left behind on an older release while the primary moves forward. Check the release end-of-life dates on the Ceph releases page before choosing a version, and schedule a mirror health check before and after each upgrade window.
Questions buyers ask about CephFS mirroring cost and effort
What drives the implementation effort for CephFS snapshot mirroring?
The main drivers are how many file systems and directories must be mirrored, how many sites are involved, and how much of the surrounding setup is already in place. Each mirrored directory needs its own decision about scope, path and access, and each peer needs a bootstrap token exchanged between clusters. Effort also grows with the number of user capabilities you must define, the monitoring you want around snapshot lag, and the failover runbook your operations team will rely on. Ask any partner to show how these items are counted in a written scope before accepting a quote, and confirm whether the target cluster is new or already in service.
Can our own team set up CephFS mirroring without a partner?
Yes, if your team already runs Ceph and has access to both clusters. The setup uses documented steps: enable mirroring on the file system through the mirroring manager module, deploy the cephfs-mirror daemon, bootstrap the peer, and add directories one at a time. The harder part is usually not the commands but the design: which directories to protect, how recovery points will be explained to application owners, and how failover and failback will be tested. Teams that lack recent Ceph operations experience often use a partner for the design and test plan, then run the routine work themselves.
What does the client need to provide before mirroring work starts?
Expect to provide access to both clusters, a dedicated network path between them that the mirror daemon can use, and confirmation of the Ceph release running on each side. The documentation states that both clusters should run Pacific or later. You also need a list of directories and their owners, the recovery point objective each owner accepts, a change window for enabling mirroring, and a test window for failover exercises. Ask whether any directories contain data that cannot leave a specific site, because that changes the design before any configuration starts.
How long does setup take when the directory count changes?
Duration depends mainly on the number of directories, their size and churn, and the time needed to validate each one on the remote side. Adding a directory is a small step, but the first sync of a large tree can take far longer than the configuration itself, and it competes with production traffic on both clusters. Ask a prospective partner to state the assumptions behind any timeline, including the directory count, initial data volume, bandwidth between sites and whether test windows are included. No duration is given here because the supplied facts do not provide one.
Can DeployOpen run failover tests for our CephFS mirror?
DeployOpen plans and implements failover and failback tests as part of the scope when they are written into the agreed deliverables. A test plan usually covers promoting the secondary file system, confirming that expected directories and snapshots are present, checking the age of the last replicated snapshot against your recovery objective, and documenting the steps to return service to the primary site. Tests should be run in a defined window with the application owners present, and the results recorded as evidence. Confirm the exact test cases and who signs off before work begins.
How is recurring support for CephFS mirroring scoped?
Ongoing support is available as a retainer, and its scope should be written down before it starts. Typical recurring items include watching mirror daemon health, reviewing snapshot lag, adjusting directory lists as owners request changes, applying Ceph point releases and planning upgrades between supported major releases. Each item has a different frequency and a different owner, so ask which tasks are included, how incidents are escalated, and what happens when you upgrade either cluster. Only the Ceph releases listed in the supplied facts should be assumed as current, and you should confirm the support model for your release level.
What does asynchronous replication mean for our recovery points?
Mirroring copies snapshots, not live writes, so the remote copy is only as recent as the last snapshot that finished replicating. Any change made after that snapshot will be missing after a failover, and the gap depends on snapshot frequency, the amount of data changed and the bandwidth between sites. This is the number your application owners must accept as the recovery point. Ask for a documented way to measure replication lag and for the expected loss window to be recorded in the runbook, since the mirror cannot guarantee a zero-loss recovery point.
Planning failover and failback around mirrored snapshots
Mirroring copies snapshots to the second cluster, but it does not fail over anything. Promoting the secondary file system, pointing clients at it, and later returning service to the primary are operator procedures that you must write, assign and rehearse before an incident. The mirror daemon keeps the remote copy current, while the decision to cut over belongs to the people who own the service, the storage team and the application owner. Treat the runbook as a deliverable of the project, not an afterthought.
A useful CephFS recovery runbook names the mirrored directories, the peer relationship and the snapshot the business accepts as its recovery point. It should record the mirroring status checks to run before any cutover, the steps to make the secondary writable and mount it on client hosts, and the credentials and cephx capabilities the failover path needs, which are not the same as those used for steady-state replication. It also needs a failback sequence that reverses the direction of replication without overwriting data written during the outage. Confirm the exact commands and their behavior for your release against the CephFS mirroring documentation and your own lab test.
Validation happens on the secondary before clients rely on it. Compare directory trees, file counts and sizes against recorded expectations for the chosen snapshot, and run application-level checks such as database consistency queries or file-open tests for the workloads that use the share. Assign ownership explicitly: the storage operator runs the cluster and mirroring steps, the application owner signs off on data validity, and a named incident commander approves the cutover and the return to the primary. Rehearse the full cycle at least once in a non-production environment, and record how long each step took so the recovery time you promise is based on evidence.
How DeployOpen scopes and delivers CephFS mirroring
DeployOpen plans and implements CephFS snapshot mirroring inside your own clusters, in your data centers or cloud accounts, and agrees the scope in writing before any work starts. The written scope names the source and target file systems, the exact directories to mirror, the recovery point objective you expect the snapshot schedule to support, the cephfs-mirror host placement, and the Ceph user capabilities each side will use. Because mirroring is set per file system and per directory, the scope should list every directory individually. A directory that is missing from the list is not protected, and the document is where that gap gets caught before go-live. Ask for the scope to state which Ceph releases run on each side, since the CephFS mirroring documentation says both clusters should run Pacific or later.
Delivery follows the sequence in the scope: enable mirroring on both file systems, bootstrap the peer with peer_bootstrap, add the agreed directories, and confirm that the mirror daemon reports healthy status for each one. The work includes configuring the snapshot schedule that produces the recovery points being replicated, and documenting what a recovery point means for each directory. A written runbook covers how to promote the secondary file system, which clients to repoint and who approves the failover. Recovery tests are part of the agreed work when you ask for them: a planned test restores a sample directory from the remote cluster and compares it with the source, and the results go into the record your auditors or risk owners will read. Confirm with DeployOpen which tests are included in the fixed scope and which are offered as a separate engagement.
After handover, DeployOpen can support the mirror through a retainer. Retainers cover monitoring of mirror status, reviewing snapshot retention against the agreed recovery targets, rerunning recovery tests after upgrades, and applying Ceph point releases on the schedule you approve. Ask for the retainer terms in writing, including response expectations and what is excluded, because a quote for ongoing support should be tied to the directories and clusters in the original scope. Your own team should confirm the remaining decisions before the quote is final: who owns the second cluster, how much capacity the replicated data needs, and whether the network path between sites is sized for the change rate you expect.
