Which Proxmox VE storage model fits your cluster
If your VM disks must keep running after a node fails and you cannot accept losing any acknowledged writes, choose hyper-converged Ceph, provided you can meet its node and network minimums. If you have fewer nodes, local ZFS storage, and can tolerate losing up to one replication interval of writes, Proxmox VE storage replication is the lower-cost path. The deciding question is how much write loss the workload can absorb, because that answer drives the hardware bill more than any feature list. Ceph fits when you can start with at least three servers, ideally identical, and dedicate at least 10 Gbps of network bandwidth to Ceph traffic. The Proxmox VE Ceph guide recommends 25 Gbps for high-performance NVMe setups and a separate 1 Gbps network for corosync. Three to five nodes without 10 Gbps switches can use a full mesh instead. Budget at least 8 GiB of memory per OSD and SSDs rather than HDDs in small setups. Ceph is the only option here where shared storage makes Proxmox HA recovery work without accepting data loss, which is the requirement behind most HA decisions. ZFS replication fits when you have fewer nodes, local ZFS pools, and a workload where a restart from a recent copy is acceptable. The default schedule runs every 15 minutes, so the Proxmox VE storage replication wiki's warning applies: data written after the last sync can be lost when a node fails. Proxmox HA is allowed in combination with storage replication, but only on that basis. Before you commit, confirm the replication interval against your application's recovery tolerance, test a forced node failure in a non-production cluster, and ask your hardware vendor to confirm the disk and RAM sizing for your ZFS pools.
Choosing by cluster size, budget and tolerance for loss
Small sites with two or three nodes usually need to decide first whether shared storage is worth the hardware it demands. Proxmox VE's hyper-converged Ceph guide describes Ceph as suited to small to medium-sized deployments and requires at least three servers, preferably identical. A two-node site cannot run Ceph in the recommended way, so local ZFS with storage replication is the realistic starting point for that size.
- Node count: Ceph needs at least three servers, so a two-node cluster rules it out under Proxmox's guidance.
- Replication model: ZFS storage replication runs every 15 minutes by default, so a node failure can lose changes made since the last sync.
- Network: Ceph calls for at least 10 Gbps used only for Ceph traffic, plus a separate 1 Gbps network for corosync.
- Memory and drives: Ceph needs at least 8 GiB of memory per OSD, and SSDs are preferred over HDDs in small setups.
- High availability: Proxmox HA needs at least three cluster nodes for reliable quorum, so a two-node cluster needs a third vote or witness planned before HA is enabled.
Before committing, confirm with your hardware vendor and Proxmox support how many nodes and what quorum design your HA requirement needs.
Three to five nodes is the range where Ceph becomes a real option, but the network decides whether it is practical. Proxmox's guide says three to five nodes without 10 Gbps switches can use a full mesh network, where each server links directly to the others. This removes the switch cost, but it adds cabling, bond and routing work that the team must maintain.
- Full mesh path: Direct server-to-server links can carry Ceph traffic when there are no 10 Gbps switches, which suits a small, fixed footprint.
- Switched path: With 10 Gbps switches, bond links across two switches so one switch failure does not isolate the storage network.
- Drive layout: Ceph's hardware guidance favors a dedicated drive per OSD and host bus adapters over RAID controllers.
- Peak memory: Size memory for peak use rather than quiet periods, and benchmark before buying because no two clusters are alike.
- Replication choice: If the team wants live migration without shared storage risk, Ceph fits; if it accepts a replication window, ZFS with storage replication stays simpler.
Ask your hardware provider to confirm the full mesh layout and failover behavior for your exact NIC and server models.
Larger clusters that need shared storage for live migration and for guests to restart on any node usually point toward Ceph. Ceph provides block storage through RBD across the cluster, so every node reaches the same VM disks. Proxmox's storage replication cannot provide that, because it supports only local ZFS storage and copies volumes between nodes on a schedule.
- Shared access: Ceph RBD gives every node access to VM disks, which removes the replication window for planned moves and HA restarts.
- Network scale: Proxmox recommends at least 25 Gbps for high-performance NVMe setups, separate from corosync traffic.
- Operating effort: Ceph requires monitoring of OSDs, placement and capacity, and ongoing upgrades to actively maintained releases.
- Release timing: Confirm which Ceph release line your Proxmox version supports and its estimated end of life before you standardize.
- Hardware cost drivers: Drive count, network speed, memory and enterprise SSDs drive the total, and these are typically recurring when you add nodes.
Ask for a written hardware bill of materials and a capacity plan that matches your VM growth before you sign off on the design.
Tolerance for data loss is the question that separates these sites more than node count alone. Proxmox's storage replication wiki warns that data written between the last sync and a node failure may be lost. With the default 15-minute schedule, that window is the most a team should expect to lose for replicated guests, unless the schedule is changed and the change is tested.
- Replication window: Data written since the last ZFS sync can be lost on node failure, so write-heavy databases need a different answer.
- HA with replication: Proxmox allows HA with storage replication, but the same loss window still applies.
- Shared storage alternative: Ceph keeps one copy of each VM disk available across nodes, avoiding a sync-based loss window, but it needs the node and network investment above.
- Application-level options: Some workloads can handle loss through their own replication, which can be a valid reason to keep ZFS.
- Recovery testing: Test a forced node failure and measure the actual data loss before relying on any window.
Confirm your recovery point objective in writing with the business owner, then match it to the storage model and test the result.
How Ceph shared storage supports Proxmox HA
Hyper-converged Ceph changes the failure story for Proxmox HA because every node reads the same guest disks. When a node fails, Proxmox VE can restart its guests on another node that already sees the RBD images, so there is no replication interval to lose data across. The Proxmox guide sets a floor of at least three servers, and it prefers identical hardware so that OSD, memory and network behavior stay predictable across the cluster. Plan for three nodes as the minimum, and treat four or five as the practical starting point if you want room to lose a host during maintenance.
Memory and disk choices drive most of the risk. The Proxmox Ceph guide recommends at least 8 GiB of memory per OSD, so a node with six OSDs needs at least 48 GiB before Ceph, the hypervisor and guests are counted. The guide favors SSDs over HDDs in small setups, and it recommends host bus adapters rather than RAID controllers so Ceph sees each drive directly. Ask your hardware vendor to confirm the controller is in HBA or passthrough mode, and benchmark the layout before purchase, since the Ceph hardware recommendations note that no two clusters behave alike.
HA also depends on quorum and fencing, which Ceph does not replace. Proxmox HA needs at least three cluster nodes for reliable quorum and watchdog-based fencing, so a node that loses contact is reset before its guests start elsewhere. Without that reset, the same disk could be written by two hosts. Confirm the watchdog behavior on each node during acceptance testing. Also check the Ceph lifecycle before you pick a release: Squid and Tentacle have different estimated end-of-life dates, so verify both on the Ceph releases page and plan the upgrade path before go-live.
Ceph and ZFS replication side by side
This table compares hyper-converged Ceph and local ZFS with storage replication across the factors that decide the design: nodes, network, data loss, HA, memory, cost and licensing. Read each row against your own node count and failure tolerance, since the right choice depends on which trade-off your workloads can accept.
| Factor | Proxmox VE with Ceph | Proxmox VE with ZFS replication |
|---|---|---|
| Minimum nodes | At least three servers, preferably identical, per the Proxmox VE Ceph guide | Two or more nodes can replicate, but HA needs at least three cluster nodes for reliable quorum |
| Network requirements | At least 10 Gbps used exclusively for Ceph, 25 Gbps for high-performance NVMe, plus a separate 1 Gbps network for corosync | No shared storage network required; replication traffic still needs enough bandwidth to finish each sync cycle, so confirm it with your own workload |
| Storage type | Shared, distributed storage across nodes through RBD, with CephFS available for file workloads | Local ZFS (zfspool) on each node; replication copies guest volumes to another node |
| Data loss window | Writes are replicated across OSDs before acknowledgement, so no replication lag window applies to the pool | Data written since the last sync can be lost if a node fails; the default schedule is every 15 minutes |
| HA behavior | Guests can restart on any surviving node because all nodes see the same shared storage | HA is allowed with replication, but guests restart from the last replicated state, so the loss window applies |
| Memory needs | At least 8 GiB per OSD, sized for peak use rather than calm periods | Driven by guest workloads and ZFS ARC; no per-disk Ceph memory floor applies |
| Hardware cost drivers | Enterprise SSDs, HBAs instead of RAID controllers, 10 to 25 Gbps switches, and extra capacity for replica overhead | Local disks per node, ECC memory and enough capacity to hold replicas of other nodes' guests |
| Operational complexity | Ceph health, OSD, monitor and network tuning are an ongoing operational skill set | Schedules, replication status and sync intervals; fewer moving parts but more manual recovery planning |
| Licensing | Ceph is LGPL-2.1 or LGPL-3.0 with no paid edition from the project; Proxmox VE is AGPL-3.0 with per-socket subscriptions for the enterprise repository and support | Proxmox VE is AGPL-3.0; the same per-socket subscription applies to the enterprise repository, and ZFS replication adds no separate license |
What drives the hardware cost of each option
Both options spend money on servers, network and disks, but the bill lands in different places. Ceph concentrates cost in identical nodes, a dedicated storage network and enterprise drives, while ZFS replication spreads it across local disks with spare capacity. Read each card as a line item to price in a written quote.
Server count and identical hardware
Ceph hyper-converged clusters need at least three servers, and Proxmox recommends that they be identical. Each node carries compute, OSDs and Ceph services, so every extra node adds CPU, memory and drives to the bill. ZFS replication also needs at least two nodes with matching local pools, but the node count often follows workload capacity rather than storage design. Ask the quote to show per-node cost and the number of nodes needed for your failure tolerance.
Ceph network switches and NICs
Proxmox recommends at least 10 Gbps used exclusively for Ceph traffic, and at least 25 Gbps for high-performance NVMe setups, with a separate 1 Gbps network for corosync. That means dual-port NICs, transceivers or cabling, and switches that carry Ceph traffic separately from guest traffic. For three to five nodes without 10 Gbps switches, a full mesh network is an option. Price the switch ports and optics along with the servers, since they are one-time purchases that the storage design cannot avoid.
Enterprise SSDs and a drive per OSD
Ceph's hardware guidance calls for a dedicated drive per OSD, enterprise-class SSDs for metadata, and moving WAL and DB to shared SSDs when HDDs hold the data. Proxmox advises SSDs over HDDs in small setups. Drive endurance and capacity set the largest storage line item, and every failed drive is a replacement cost. ZFS replication uses local disks that also need to be sized and redundant, so compare total usable capacity per dollar, not raw drive counts.
Memory sized for peak use
Proxmox recommends at least 8 GiB of memory per OSD, on top of what the guests need. Ceph's hardware guidance says to size memory for peak use rather than calm periods, which means recovery and rebalancing after a node failure. Memory is one-time cost per node, but it drives server model choice and DIMM count. Ask the vendor to show the memory configuration under a rebuild scenario, not only steady-state load.
Host bus adapters instead of RAID
Proxmox recommends host bus adapters rather than RAID controllers for Ceph, so each drive is presented directly to the OS and Ceph manages redundancy itself. HBAs are usually cheaper than RAID cards with cache and battery backup, but they change the server bill of materials and the spare parts list. Confirm the HBA model with the hardware vendor and with the Ceph hardware guidance before ordering, because RAID-based builds are a common source of rework.
ZFS disks and replication headroom
ZFS replication stores guest volumes on local ZFS pools and sends snapshots to another node on a schedule, which is 15 minutes by default. The receiving node needs free capacity for each replicated volume and enough disk throughput to absorb the sends, so local disks are sized larger than the guests alone require. The cost is one-time disk and spare capacity, with no per-node storage network. Ask the quote to show usable capacity after ZFS redundancy and replicated copies.
Recurring Proxmox subscriptions
Proxmox subscriptions are priced per CPU socket per year, and the same software features apply at every tier. Tiers differ in support entitlements and access to the enterprise repository, which is the recommended repository for production. Socket count is the main driver, so a Ceph cluster with more nodes recurs more subscription cost than a smaller ZFS pair. Confirm whether the quote covers every node, whether Ceph packages come from the enterprise repository, and what renews each year.
What the ZFS replication data loss window means in practice
Proxmox storage replication copies a guest's ZFS volumes to another node on a schedule, and the default interval is every 15 minutes. Anything written after the last completed sync is lost if the source node fails, so the data loss window is up to one replication interval of writes plus any sync still in progress. Proxmox's own storage replication wiki states this plainly and notes that high availability can be combined with replication on that understanding. Replication also works only with local ZFS (zfspool) storage, so every guest you want to protect must sit on a ZFS-backed storage definition on each node that could receive it.
Combining replication with Proxmox HA is acceptable for workloads where losing the last few minutes of changes is an operational nuisance rather than a regulatory or financial event. Typical candidates are stateless application tiers, build agents, internal tools with an external system of record, and test or lab guests. It is a poor fit for transactional databases, payment or ledger systems, and anything where a reconciliation process would be needed after a node failure. Pair replication with regular backups, ideally to a separate target such as Proxmox Backup Server, so that a corrupted or deleted volume can still be restored, since replicated copies follow the source's state.
Before you commit, confirm the replication interval you need, measure how long a sync takes at your change rate on your network, and test a forced node failure in a non-production cluster to see what the recovered guest actually contains. Ask your vendor or implementation partner to document the recovery point for each protected guest and who signs off on it. If your recovery point objective is shorter than the replication interval, replication alone will not meet it, and you should compare the shared-storage option described in this page's other sections.
How to move from one model to the other
This sequence lets a team change its Proxmox VE storage model later with evidence in hand: a confirmed node and network baseline, a tested failover path, a measured loss window and a planned disk migration.
Confirm node count and network layout
The infrastructure lead and the network engineer check how many cluster nodes exist, whether each node has identical hardware, and whether a dedicated 10 Gbps network can carry only Ceph traffic. A separate 1 Gbps network must stay in place for corosync. If three or more nodes cannot be reached on a suitable network, a switch to Ceph is not ready. Done means a signed-off inventory of nodes, NICs, switches and corosync paths that matches the Proxmox VE Ceph guidance.
Check the current storage model against HA needs
The platform owner records which storage type holds each guest disk today, whether it is local ZFS with storage replication or shared storage, and which VMs are expected to restart on another node. Proxmox HA also needs watchdog-based fencing and redundant hardware, so the owner confirms those are present. Done means a table of guests, storage backends and HA status that shows any gap before any change is made.
Test failover with a non-critical VM
The operations team creates a non-critical test VM on the target storage model, enables HA for it, and then simulates a node failure in a maintenance window agreed with the business owner. They watch whether fencing resets the failed node before recovery starts elsewhere and how long the guest takes to come back. Done means a written test record with the observed recovery behavior, the storage path used and any errors found.
Measure the loss window against business tolerance
For ZFS replication, the team reads the configured replication schedule, which runs every 15 minutes by default, and compares it with the data loss tolerance the application owners have set. For Ceph, they check whether the replication model meets the same tolerance and confirm the write behavior with the application owners. Done means a documented loss window per workload and a sign-off from each owner that the number is acceptable.
Plan the guest disk migration
The team lists every disk to move, the order of moves, the expected downtime each guest can accept and the rollback steps for each one. They confirm current backups and restore a sample disk before moving anything, because the migration changes where the disk data lives. The effort grows with the number of disks and the downtime allowed, so the plan should show both. Done means an approved change plan with a rollback step for every guest.
Migrate disks in agreed waves
The operators move guest disks in the order from the plan, starting with the least critical workloads. After each wave they start the guests, check application health and confirm the disk is on the new storage type. A failed check triggers the rollback step recorded for that guest. Done means each guest reports healthy on the new storage and the change log matches the moves made.
Retest HA on the new storage model
Once all disks are moved, the team repeats the failover test on at least one VM from each storage group and re-runs the loss window check against the new configuration. Storage migration alone does not prove HA works, so this retest is required. Done means a fresh test report that replaces the earlier one, and the runbook updated with the new storage and HA settings.
Record the decision and update operations
The platform owner files the final configuration, loss window, node and network inventory and test results with the change record, and updates monitoring, backup jobs and the on-call runbook. Vendor support tiers and repository choices are recorded alongside. Done means the operations team can answer, from the documentation alone, where each disk lives, how a node failure is handled and how much data could be lost.
Questions buyers ask before choosing Ceph or ZFS
Are three nodes enough to run Ceph in Proxmox VE?
Three nodes is the minimum Proxmox VE's hyper-converged Ceph guide requires, and it recommends three servers that are preferably identical. Three is the floor, not the comfortable target for every workload. Ceph keeps replicas across hosts, so a three-node cluster has little room to lose a node and still rebuild data quickly, and the guide describes hyper-converged Ceph as suited to small to medium deployments. Confirm with your own capacity model whether you need four or five nodes for your replica count, recovery time and growth. For three to five nodes without 10 Gbps switches, the guide notes that a full mesh network is an option.
What network speed does Ceph need in a Proxmox cluster?
Proxmox's Ceph guide recommends at least 10 Gbps of network bandwidth used exclusively for Ceph traffic, and at least 25 Gbps for high-performance NVMe setups. It also calls for a separate 1 Gbps network for corosync, the cluster communication layer. Ceph's hardware recommendations add that bonding network links across two switches reduces the risk of a single switch failure taking storage down. Ask your network team to confirm that the switch fabric, cabling and NICs meet these figures before you size the hardware, because a slow Ceph network shows up as poor VM disk performance long after purchase.
How much data can a ZFS replication sync lose?
With Proxmox VE storage replication, the window is the time since the last successful sync, and the default schedule runs every 15 minutes. Proxmox's storage replication wiki warns that data written after the last sync can be lost if a node fails. The loss window is therefore set by your replication schedule and by how much your application can tolerate losing. Shorten the interval only after confirming that the replication traffic and snapshot load fit your storage and network. Ask your team to document the recovery point objective each workload requires and to test a failover against it.
Does Proxmox HA work with ZFS replication?
Yes, Proxmox VE allows high availability in combination with storage replication, but the Proxmox wiki states that data written since the last sync can be lost when a node fails. HA also needs at least three cluster nodes for reliable quorum, redundant hardware and watchdog-based fencing, so a failed node is reset before its guests start elsewhere. Ceph is the option where shared storage is already present for HA, since guests do not depend on the last replicated snapshot. Confirm fencing and watchdog behavior in a test cluster before relying on either approach for production workloads.
What does a Proxmox VE subscription cost per socket, and which tiers differ only in support?
Proxmox VE subscriptions are priced per CPU socket per year. The Proxmox VE pricing page lists Community at 120 euros (no support tickets), Basic at 370 euros (3 tickets, 1 business day response), Standard at 550 euros (10 tickets, 4-hour response) and Premium at 1,100 euros (unlimited tickets, 2-hour response). All tiers include the same software features and differ in support entitlements and enterprise repository access, and the software can run without a subscription. Your yearly total is the tier price multiplied by sockets across every node. Check current pricing with Proxmox before budgeting, since the price list may change.
What skills does the team need to run Ceph or ZFS replication?
Ceph needs people who can operate distributed storage: monitors and managers, OSD replacement, placement and capacity planning, network tuning and upgrades across Ceph releases. The Ceph hardware guidance also calls for benchmarking before purchase and sizing memory for peak use. ZFS replication is a smaller operational surface, centered on pool health, snapshot schedules, replication job monitoring and failover testing. Both need a team that owns backups and can run restore tests. Write down which engineers own storage, who is on call for a failed disk or node, and how often the team will test recovery.
Can our in-house team deploy this, or should we scope it with DeployOpen?
Your team can deploy either option if it has the network, storage and Proxmox skills described above and time to validate the design. DeployOpen plans, implements, integrates, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure, and most engagements are fixed-scope with scope, deliverables and timelines agreed in writing before work starts. A written scope would typically cover the hardware and network review, the storage design, HA and fencing tests, backup and restore validation and runbooks. Ask any implementation partner for a written scope that names those deliverables, the time it covers and what your team must provide.
What running Ceph or ZFS replication asks of your team
The main difference in daily work is where the failure surface sits. With hyper-converged Ceph, your team operates a distributed storage cluster that runs alongside the hypervisors, so the routine work is cluster health, OSD lifecycle and network care. Health checks mean reviewing cluster status, placement group state and OSD up and in counts on a schedule, not only when a VM misbehaves. OSD replacement is a recurring task: you need a tested procedure for pulling a failed drive, rebuilding its OSD and waiting for recovery to finish before the next failure lands. Release lifecycle planning matters too. Cephadm manages Ceph as containers and the Ceph releases page lists Squid with an estimated end of life of October 31, 2026 and Tentacle with an estimated end of life of June 1, 2027, so you need an upgrade calendar that fits those dates. Network tuning, including keeping Ceph traffic on its own 10 Gbps or faster network and bonding links across two switches, is an ongoing engineering responsibility, not a one-time setup.
With ZFS replication the operational load is narrower but still real. Your team watches ZFS pool health, including scrub results, capacity and disk errors on each node, and monitors replication jobs, since replication runs every 15 minutes by default and a job that silently stops widens the data loss window without any VM showing a fault. The Proxmox storage replication documentation warns that data written after the last sync can be lost when a node fails, so the team must know the actual replication interval in production and communicate that number to application owners. Failover testing is the discipline that makes either model trustworthy. Schedule planned node failures in a maintenance window, confirm that HA recovers guests on a surviving node, and confirm watchdog fencing behaves as designed. Ask your Proxmox partner or account team which support entitlements cover the incidents you expect. Plan support and package access before go-live. Proxmox VE subscriptions are priced per CPU socket and differ in support level and enterprise repository access. The enterprise repository requires a valid subscription key and is the recommended choice for production, while the no-subscription repository suits testing and non-production work. Use the no-subscription repository in a lab to rehearse OSD replacement, node failure and upgrades, and confirm that your production nodes have the subscription packages your runbooks assume.
How DeployOpen scopes and delivers Proxmox VE storage
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure, including Proxmox VE clusters. For Proxmox VE with Ceph vs ZFS replication, the first step is a written scope that settles the storage design against your node count, network, data loss tolerance and hardware budget before any build work starts.
How we scope
- Confirm node count, hardware models and whether each node can take a dedicated Ceph drive per OSD
- Confirm the Ceph network (10 Gbps or faster) and separate 1 Gbps corosync network, or a full mesh layout
- Record the replication interval and the data loss window your workloads can accept for ZFS replication
- Agree in writing whether high availability is in scope, including watchdog-based fencing and quorum design
How we build
- Deploy the Proxmox VE cluster with Ceph or local ZFS storage on the hardware and networks agreed in scope
- Configure storage replication jobs or the Ceph pools and Proxmox HA resources according to the written design
- Test node failure, network loss and fencing behavior against the recovery expectations in the scope
- Document each storage design decision, configuration key and failure runbook for your operations team
How we hand over
- Deliver the as-built configuration, network diagram and storage layout with version and update records
- Walk your operators through failure drills, replication lag checks and Ceph health monitoring steps
- Provide a hardware and licensing checklist so your team can confirm subscription and repository choices
- Confirm the handover acceptance criteria listed in the written scope before the engagement closes
How we support
- Ongoing support, upgrades and continuous testing for the cluster are available as a retainer
- Retainer scope for Ceph and Proxmox upgrades is agreed in writing against the release windows you choose
- Periodic failover tests can be added to the retainer to confirm HA behavior after hardware changes
- Support requests and their response expectations are defined in the retainer agreement before work begins
