Proxmox VE ZFS storage replication

Scheduled ZFS replication in Proxmox VE copies a guest's ZFS-backed virtual disks from the node where the guest runs to one or more other cluster nodes at a set schedule. Each run sends the disk's ZFS snapshot to the target, which keeps the last transferred state as a local copy. The copy is not shared storage: each node holds its own dataset, and the target only becomes useful when a guest is started there. Understanding that model is the starting point for checking whether replication fits a given recovery goal. The first transfer is a full copy of the disk, which can take a long time for large volumes and places heavy load on the source and the network. Later runs are incremental: they send only the blocks changed since the last common snapshot. Before relying on this, confirm in the Proxmox VE administration guide how the replication engine chooses between full and incremental transfers, what happens if the common snapshot is lost, and how a failed run is retried. Ask your team to test an initial sync on representative disk sizes in your own hardware, because transfer time depends on disk size, change rate and network path, not on a fixed figure. A replication job defines the guest it protects, the target node, and the schedule on which transfers run. Several jobs can exist per guest, each with its own target. Check in the documentation which schedule syntax is supported, the minimum interval, and any limits on concurrent jobs or bandwidth. Those values are configuration limits that can change between releases, so record the version you tested against and confirm them before you write a recovery design around a specific frequency. Replication does not start a guest on its own. Failover to a target is a deliberate action: the guest configuration is available on the target and the administrator, or HA manager if combined with HA, starts the guest there using the most recent replicated state. Confirm in the guide what happens to the guest configuration, which disks are attached, and whether a guest that is still running on the source can be started on the target. Document a runbook that covers this step, including how to confirm the source is down before starting the copy elsewhere, so two instances do not write to disks at the same time. The main risk to plan for is the gap between the last successful transfer and the failure. Any writes made after that point are lost on failover, so the replication schedule defines your recovery point. Test a failover in a non-production cluster, measure how much data is missing, and compare that result with what the business accepts.

After a node fails, a guest recovered from ZFS storage replication returns to the most recent replicated copy, not to the moment of failure. Any write made after the last successful replication run is lost. The data loss window is therefore the time between that last successful run and the failure. Schedule interval alone does not define it. Before you choose an interval in Proxmox VE, agree with the application owner how much recent data the workload can afford to lose, and write that number down as the target the design must meet.

Three factors determine how large the window gets in practice. The first is the replication interval, which sets how often a new snapshot is sent. The second is the transfer duration, which is the time each incremental send takes to finish on the link between nodes. A transfer that runs longer than the interval causes runs to queue or overlap, so the copy on the target can fall further behind than the schedule suggests. The third is a failed or delayed job. A failed run leaves the target at its previous state until a later run succeeds, so the window grows for as long as the job stays broken, and a failure that goes unnoticed can stretch the window without anyone seeing it.

Illustrative example only, with no measured values: suppose a replication job runs every interval I, and each transfer takes T to complete. If a node fails just before a run finishes, the most recent good copy can be roughly I plus T older than the failure moment. If one run also fails and the next success happens a full interval later, the exposure grows to about two intervals plus T. Work the calculation with your own schedule and measured transfer times, and check the result against the agreed target. If the answer exceeds the target, shorten the interval, reduce the transfer size or fix the link before go-live.

Measure the window in a test cluster that matches production as closely as you can. Create a guest that writes a timestamped marker to disk at a known rate, let at least one full replication cycle complete, then power off the source node without a clean shutdown. After recovery on the target, compare the last marker present with the time of failure. Repeat the test while a job is deliberately failed or throttled, and record the replication job status and the last successful sync time that Proxmox VE reports. Use these results to validate the window, not the schedule you intended.

Before you rely on the recovered guest, confirm with the Proxmox VE documentation how a guest recovered from a replicated copy reports its data state, including whether the last sync time or any consistency indication is shown and where. Your runbook should state who checks that state after a failover, which timestamp they compare against the agreed window, and how they document any loss that occurred. Plan that verification step as part of the failover procedure rather than leaving it to be discovered during an incident.

Move from node-local ZFS replication to Ceph when a guest must survive the loss of a host or disk without restoring from the most recent replicated snapshot. ZFS replication copies data between nodes on a schedule, so a failover starts from the last completed sync and any writes after it are lost. Ceph distributes each object across failure domains that you define in CRUSH rules, so a failed host or drive leads to data recovery across the remaining hardware rather than a switch to an older copy. The decision turns on workload recovery needs: how much data loss the application can tolerate, how quickly guests must restart after a failure, and how much capacity and write load you expect to grow into.

Ceph is a separate storage system, and your team must staff and patch it as one. It runs its own monitors, managers and OSDs, has its own network design and its own upgrade sequence, and it sits beneath Proxmox VE rather than inside it. The Ceph hardware recommendations call for a dedicated drive per OSD, moving WAL and DB onto shared SSDs for HDD-based OSDs, enterprise-class SSDs for metadata, bonded network links across two switches, and memory sized for peak use rather than quiet periods. They also advise benchmarking before you buy, because no two clusters behave alike. Confirm the minimum node count and network layout for your replication settings with the Ceph documentation before you size hardware.

The Ceph source is licensed under LGPL-2.1 or LGPL-3, and the project itself offers no paid edition, so the software carries no license fee. The cost of running Ceph sits elsewhere: servers, drives, switches and cabling; staff time for design, monitoring, drive replacement, capacity expansion and patching; and any vendor or partner support contract you choose. Design, benchmarking and the migration of guests off ZFS are largely one-time costs, while patching, monitoring and hardware refresh recur. When you request a quote, ask for each of these components separately, state which party carries each one, and list what the scope excludes, such as the Proxmox VE subscription that remains a separate purchase.

Plan the release lifecycle before you choose a version. The Ceph releases page lists Squid (19.2.x, latest 19.2.7) with an estimated end of life of October 31, 2026, and Tentacle (20.2.x, latest 20.2.4) with an estimated end of life of June 1, 2027. Actively maintained releases receive periodic backports and security fixes, so an estimated end-of-life date is the point at which you must have upgraded. Cephadm, the deployment method that runs Ceph as containers, was introduced in v15.2.0 (Octopus) and does not support older releases. Confirm that the Ceph release you select is supported by cephadm and that your upgrade plan reaches a supported release before the end-of-life date. If you run Ceph under Kubernetes instead, the cephadm documentation points to Rook, a CNCF Graduated project, as the alternative deployment path.

A practical rule is to stay with ZFS replication when your guests tolerate a bounded loss window and your team cannot yet operate a second storage platform. Choose Ceph when guests need storage that stays available through a node or drive failure, when you can dedicate hardware and staff to it, and when you expect the cluster to grow. Before you commit, run a short proof of concept on hardware close to your planned build, measure recovery time after a deliberate node failure, and record the staff hours each month for patching and monitoring. These measurements give you the figures that a quote and an upgrade plan need.