Can a three-node Ceph cluster run in production?
Yes, a three-node Ceph cluster can run in production, but only as a layout that accepts no spare capacity for rebuilding data after a host loss. With three servers you can place one monitor on each host, so the monitors keep quorum if one host fails, and you can set a replicated pool to three copies with the host as the failure domain, so each copy lands on a different server. The cluster stays readable and writable with one host down, but Ceph cannot re-create the lost copies on a fourth server, so the data sits in a degraded state until the failed host returns or you add hardware. That trade-off, not the software, is the main decision for a team starting with three nodes.
This page covers what that layout can and cannot do for block, file and object workloads: how the three monitors and the host failure domain behave when one node is down, why erasure coding is constrained at this size, and which signals should push you toward more nodes. It also explains how recovery time depends on the capacity you keep free. Teams usually need this page when budget or rack space limits them to three servers and they must decide whether that is enough for a production service, or when they are choosing between a three-node start and a larger first purchase.
Before you commit, confirm the release you plan to run. The Ceph releases page lists Tentacle (20.2.x) and Squid (19.2.x) as the actively maintained releases, and the end-of-life dates for each are published there, so check them against your support window. Ask your hardware vendor and your DeployOpen scope owner to confirm that each server has enough drives, memory and network ports for a bonded, two-switch design, because those constraints decide how much usable capacity a three-node cluster really gives you. Write down the failure you plan to survive and the time you accept to restore full redundancy, then test both on the bench before you go live.
Choosing a node count and data protection layout
This table compares three-host and four-or-more-host layouts for replicated and erasure-coded pools, focusing on failure domains, room to recover after one host fails, and the workloads each suits. Read it as a planning guide, then confirm exact placement and erasure code profile rules for your release in the Ceph documentation. The main takeaway: three hosts can survive a host loss, but they leave no spare host to rebuild onto.
| Factor | Three hosts | Four or more hosts |
|---|---|---|
| Replicated pool with host failure domain | Three replicas can each sit on a separate host, but no spare host exists to rebuild onto | Replicas can sit on distinct hosts, with spare hosts left for rebuilding after a failure |
| Erasure coding options | Only narrow profiles fit distinct hosts, and each leaves no spare host to rebuild onto | More profile widths fit distinct hosts, so overhead and failure tolerance can be balanced |
| Recovery after one host fails | Pools run degraded until the host returns or new hardware is added | Surviving hosts can take rebuilt data, provided they have free capacity |
| Headroom for one lost host | A larger share of raw capacity must be kept free to absorb a lost host | A smaller share of raw capacity must be kept free to absorb a lost host |
| Maintenance on one host | Taking a host down leaves pools degraded until it is back online | Rolling maintenance is possible with spare hosts, subject to capacity and recovery settings |
| Workloads that fit | Labs, test pools and non-critical tiers that can accept degraded periods | Production block storage for virtual machines, object tiers and workloads that cannot run degraded |
| Growth path | Adding a fourth host later changes placement and usually triggers data movement | Further hosts extend recovery room and profile choices as capacity grows |
What a three-node Ceph cluster does when one node fails
A three-node cluster keeps serving data after one host fails, but it cannot restore full redundancy on the two hosts that remain. With three monitors, the monitor quorum needs two of three to agree, so losing one monitor host leaves the cluster able to make decisions. With a host failure domain and three replicas, each object has one copy on each host. When one host goes down, the placement groups on it become degraded: the surviving two copies still serve reads and writes, provided the pool's min_size setting allows I/O with two copies. The CRUSH rule still asks for three copies on three separate hosts, and only two hosts are available, so Ceph has nowhere compliant to place the missing copy.
This is the central trade-off of the design. Data stays available, but protection is reduced until the host returns. Confirm the size and min_size values on each pool before go-live, and confirm which pools would stop accepting writes if a second host failed. Also check how the cluster reports this state, since degraded placement groups raise health warnings that your monitoring should catch. Recovery speed is not something this page can quote: it depends on drive throughput, network bandwidth, CPU and the amount of data on the failed host, so benchmark recovery on your own hardware before relying on any timeline.
When the failed host is down, you have two choices. If the host can return within a window you define in advance, repair it and let Ceph backfill the returned OSDs, which avoids buying capacity. If the outage is longer or the hardware is not repairable, add a fourth host so CRUSH can place the missing copy again, and plan the hardware order and installation before the outage happens. Either way, plan for the load of backfill on the remaining hosts, because client traffic and recovery share the same drives and network until recovery finishes.
Where erasure coding stops fitting a three-host cluster
Erasure coding saves usable capacity compared with replication, but it asks for more failure domains and more free room than a three-host cluster usually has. The cards below cover the constraints that tend to decide the question at this scale, and what to confirm before choosing a profile.
Host count needed for host-level placement
An erasure code profile splits each object into data and coding chunks, and with a host failure domain each chunk must land on a different host. A profile needs at least as many hosts as chunks it writes, so a profile with more chunks than your three hosts cannot place every chunk on its own host. Check the chunk count of any candidate profile against the number of hosts you actually have, and confirm the placement rule the profile creates in the Ceph documentation before you create a pool.
Little room left for recovery
When a host fails, the cluster must rebuild lost chunks on the hosts that remain. With only three hosts, the surviving hosts have little spare capacity to receive rebuilt data, and the pool may not reach a healthy state until the failed host returns or is replaced. Ask how much free space each surviving host would need to absorb a rebuild, and compare that figure with the full ratios you plan to run at.
Recovery traffic on a small network
Rebuilding chunks after a failure moves large volumes of data across the cluster network, and client reads and writes compete for the same links. On a network with only a pair of bonded links per host, recovery can slow client I/O for the duration of the rebuild. Measure recovery throughput on your candidate hardware during a planned test, and set recovery limits so that application latency stays within what your workloads need.
Supported profiles and plugins
Not every erasure code profile, plugin or technique is equally suited to every release, and support can change between Ceph versions. Before you commit to a profile, confirm which plugins and techniques your chosen release supports, and whether the profile you want is listed as supported for the failure domain you plan to use. Record the release and profile in your design document so that a later upgrade does not invalidate the pool layout.
When to start with more nodes than three
A lab or proof of concept is the one situation where three nodes is usually enough on its own terms. Three monitors give quorum, one host per failure domain lets you test host-level replication, and you can rehearse drive replacement and a node reboot without buying more hardware. The goal is to learn how the cluster behaves, not to hold data you cannot lose.
- Monitor placement: Run one monitor on each host so that losing one node still leaves two of three monitors for quorum.
- Failure domain: Keep the default host failure domain and test a replicated pool with three replicas, so each object lands on three different hosts.
- Recovery: Practice taking one host offline and watch the cluster report degraded placement groups, then return to healthy. Expect a single-host outage to slow I/O while data is rebuilt.
- Limits: Do not treat lab results as a capacity or performance baseline, since drive count, network and workload will differ in production.
- Hardware: Ask the vendor whether the lab hardware matches the intended production media and network, because a mismatch makes benchmarks misleading.
Before you reuse lab hosts for production, confirm the monitor, network and drive layout matches the design you intend to run.
A production cluster with a fixed, known capacity can run on three nodes if the workload fits within what remains after one host is lost. The risk is not the initial build but the recovery window: when a host fails, the cluster must rebuild its replicas onto the two surviving hosts, and those hosts absorb the full write and recovery load. Headroom has to be planned for that event before go-live.
- Monitor placement: Keep three monitors, one per host, and confirm each has enough disk space for its store.
- Failure domain: Use host as the failure domain with three replicas so the pool can survive one host loss.
- Recovery: Size usable capacity so the two remaining hosts can hold the rebuilt data without reaching the full ratios.
- Operations: Plan maintenance so one host is down at a time, and never schedule a planned reboot while another host is already degraded.
- Questions to ask: Ask the partner to show the usable capacity calculation with one host failed, and what happens if the cluster reaches the nearfull threshold during rebuild.
Confirm that the capacity model, not raw disk totals, is the figure the business signs off on.
A cluster expected to grow should start on three nodes only if the growth path is designed in advance. Adding a fourth or fifth host later changes the placement math, and moving to more failure domains can trigger large data movement. The design question is whether new hosts will be added in the same failure domain layout, and whether the first three hosts are placed so that expansion does not force a rebuild of the rules.
- Monitor placement: Plan for five monitors across at least three failure domains once the cluster grows beyond a small footprint, and confirm the monitor count with the vendor documentation for your release.
- Failure domain: Set CRUSH locations for hosts now, using rack or room labels if you expect those to become separate domains later.
- Data movement: Expect rebalancing when hosts join, and schedule it with recovery limits so client I/O is not starved.
- Hardware: Buy hosts with the same drive layout and network so that new nodes do not create uneven placement.
- Questions to ask: Ask how the placement rules and PG counts change when the fourth and fifth hosts arrive, and how long rebalancing is expected to take in your environment.
Confirm the growth plan in writing, including the host count at which you will add a second failure domain.
A cluster that must survive a full host loss without degraded service needs more than three hosts in most designs. With three hosts and three replicas, a host failure leaves the pool degraded until data is rebuilt, and client I/O can slow during that window. Service continuity depends on having enough surviving hosts to hold every replica and enough spare capacity to rebuild while serving clients, so the host count must follow from the availability requirement rather than from the monitor count.
- Monitor placement: Spread monitors across separate hosts, and where possible separate racks or rooms, so a single failure does not break quorum.
- Failure domain: Use enough hosts that the replica count or erasure code profile still has spare failure domains after one host is lost.
- Recovery: Model recovery with real drive and network speeds, and confirm the client impact during rebuild before accepting the design.
- Network: Bond links across two switches so a switch failure does not look like a host failure.
- Questions to ask: Ask the partner to state the host count, replica or erasure profile and the expected behavior during one host loss, in writing, with the assumptions behind any performance claim.
Confirm that the availability target is defined in terms of client-visible service, not only in terms of cluster health.
How to build a three-host layout with cephadm
This sequence takes three prepared hosts to a healthy Ceph cluster with three monitors, a host-level failure domain and a three-replica pool ready for handover. Each step ends with a check that confirms it worked before the next step begins.
Confirm host prerequisites on all three nodes
The engineer verifies that each host has the same container runtime, time synchronization, a consistent hostname resolution scheme, SSH access from the bootstrap host, and the disks reserved for OSDs with no existing partitions or filesystems. Confirm the exact host requirements for your chosen release in the cephadm documentation before starting. The step is done when each host passes the preflight checks and the disks you plan to use show as available in the inventory output.
Bootstrap the first node with cephadm
On the first host, the engineer runs cephadm bootstrap, which creates a single-node cluster with one Monitor and one Manager running as containers. Set the admin network and dashboard options according to your design, and record the generated admin credentials in your secrets store. The step is done when the command completes and the cluster status reports one monitor and one manager as up.
Add the second and third hosts over SSH
The engineer copies the cluster SSH public key to the other two hosts, then adds each host to the orchestrator inventory with its address and labels. Labels such as an admin label let cephadm place monitors and managers on those hosts later. The step is done when the host list shows all three nodes as online and cephadm can reach each one without errors.
Place three monitors across the three hosts
The engineer sets the monitor placement so that exactly one monitor runs on each host, which gives an odd count of three for quorum and places a monitor in each host failure domain. Confirm the placement against the cephadm documentation for your release. The step is done when the monitor map lists three monitors in quorum and each monitor is on a different host.
Add OSDs on each host with a service specification
The engineer describes the data drives in a cephadm OSD service specification, runs a dry run to confirm which devices will be consumed, and then applies it. Each host should contribute drives of similar size and class so that the three hosts carry comparable capacity. The step is done when every host reports its expected number of OSDs as up and in, and the OSD tree shows the drives under the correct host buckets.
Create a replicated pool with a host failure domain
The engineer creates a replicated pool with three replicas and a CRUSH rule whose failure domain is host, so each object copy lands on a different node. Check the rule and pool settings after creation, and confirm that the pool's minimum replica count is set according to your availability needs. The step is done when the pool reports a size of three, the CRUSH rule shows host as the chooseleaf type, and placement groups move to active and clean.
Run health checks and test a node outage
The engineer checks the cluster health, confirms the monitor quorum, verifies that all placement groups are active and clean, and reviews the Dashboard or logs for warnings. In a maintenance window, the engineer can stop one host to observe degraded but available I/O and then confirm recovery after it returns. Record the results and the commands used. The step is done when health reports OK, the outage test produced the expected behavior, and the findings are documented for handover.
Securing and operating a small cluster with few hosts
With three hosts, every security and network choice carries more weight, because there is no spare capacity to absorb a mistake. Start by deciding whether to split traffic into a public network for clients and a cluster network for replication and recovery. The public_network and cluster_network settings are chosen at bootstrap, so confirm them before the first host joins. On three nodes, recovery traffic from a failed host can saturate a shared link and slow client I/O, so a separate cluster network is worth planning even though it adds switch ports. The Ceph hardware recommendations advise bonding network links across two switches, so confirm that each host has two bonded NICs landing on different switches and that the bond mode is tested with a link pull before go-live.
Next, protect the management and monitor paths. The Ceph Dashboard supports single sign-on through SAML 2.0, and through OAuth2 when cephadm runs the Management Gateway and OAuth2 Proxy services. Confirm which of these your identity provider supports, and confirm that the dashboard is reachable only from your administration network. For monitor traffic, the msgr2 protocol listens on port 3300, and its secure mode encrypts all post-authentication traffic with AES-GCM and adds integrity checks. Check the ms_cluster_mode and ms_service_mode settings in your release's documentation, and confirm that every client and daemon on your version supports secure mode before you enforce it. A mismatch can block clients, so test with one client first.
Finally, plan host access and patching around having fewer people and no spare nodes. Decide who holds SSH access to the hosts that run cephadm, how that access is granted and revoked, and whether admin labels are limited to the hosts that need them. Patching a three-node cluster means taking one host down at a time, which leaves the cluster with no redundancy until the host returns. Schedule each host reboot only when the cluster reports healthy, and confirm the maintenance procedure with your release's upgrade and health documentation. Write down who approves a patch window, who watches recovery, and what you will do if a second host fails during that window.
Limits to confirm before you commit to three nodes
A three-node cluster is workable only if the limits around it hold in your environment. The cards below cover what replication does not protect, how much memory and drive capacity you need at peak, how to test hardware before buying, which release you will run and for how long, and what a single rack or room can take down with it.
Replication is not backup
Three replicas protect data when a host or drive fails, but they also copy accidental deletes, ransomware encryption and bad application writes to every copy. A three-node cluster has no second site to restore from by default. Ask your backup owner which copy lives outside the cluster, how often it is taken, and how long a restore takes at your data volume. Test a restore before you rely on it. The backup copy must sit outside the failure domain of the cluster.
Memory sized for peak use
The Ceph hardware recommendations advise sizing memory for peak use rather than calm periods. On three nodes, a recovery after a host failure concentrates load on the two survivors, so their memory and CPU must absorb rebuild traffic alongside client I/O. Ask your hardware vendor or platform team to confirm the per-OSD memory target for your chosen release, and to size each node so that two surviving nodes can run the workload during recovery.
Drive capacity for failure headroom
With three replicas across three hosts, each host holds a full copy share of your data, and losing one host forces the cluster to re-create those copies on the remaining two. Usable capacity must leave room for that rebuild and for the full ratios that stop writes before a disk fills. Confirm with your storage owner how much free space you must keep, and whether a rebuild on your drive count finishes inside the window you accept.
Benchmark candidate hardware first
The Ceph hardware recommendations say to benchmark before buying because no two clusters are alike. Test your candidate drives, controllers, NICs and switches with a workload that resembles your block, file or object traffic, including a run with one node taken offline. Ask the vendor or your performance engineer for the test plan and the acceptance numbers, and keep the results with the purchase record so the design can be checked against them later.
Active release and support window
Ceph's releases page lists Tentacle and Squid as actively maintained, with Squid's estimated end of life set for October 31, 2026. A cluster you deploy now will need an upgrade within its service life, so confirm which release your organization will standardize on, who approves upgrades, and how long you can run a version without fixes. Check the current end-of-life estimates before you sign off the design.
Single rack and power failure modes
Three nodes in one rack share a top-of-rack switch, a power feed and often a cooling zone. A failed switch, PDU or room event can take two of three hosts offline at once, which drops the cluster below the replica count needed for writes. Ask your facilities and network teams whether the nodes have separate power and switch paths, and whether a rack-level outage is an accepted risk. If not, plan for more hosts or failure domains before the purchase is final.
Questions buyers ask about a three-node Ceph design
Is a three-node Ceph cluster enough for production?
A three-node cluster can be a production design when it is sized and configured for three failure domains, but it has little room to absorb failures. With a host failure domain and three replicas, each object keeps one copy on each host, so the cluster can keep serving data with one host down while it recovers. It cannot re-create the third copy of data anywhere else until that host returns or new hardware joins. Three monitors give quorum with one monitor lost. Before you commit, confirm the replica count, the headroom you hold for a failed host, and whether your workload tolerates reduced redundancy during recovery.
What does a three-node Ceph design cost in hardware and effort?
Hardware cost comes from the drives, memory, CPU, and networking in each host, multiplied by three, plus the switches and cabling. Effort comes from design, bench testing candidate hardware, deployment, and handover. The Ceph hardware recommendations advise a dedicated drive per OSD, enterprise-class SSDs for metadata, bonded network links across two switches, and memory sized for peak use, so those choices drive most of the bill. Ask any quote to break out hardware you buy, hardware you already own, services for design and build, and any post-handover support. Recurring costs include support, patching and capacity growth, which are separate from the one-time build.
How long do the design and build take, and what sets that duration?
Duration depends on how much of the design is settled before work starts, how fast hardware arrives, and how much testing the client requires. The main drivers are the number of drives per host, whether bonded networks and two switches are already in place, whether the site has internet access for image pulls, and how long benchmarking takes on candidate hardware. Your quote should state the timeline in writing against a fixed scope, with the client actions and hardware delivery dates listed as dependencies so that slips are visible early.
What does the client team need to provide for this work?
Expect to provide hardware or approval to purchase it, network access and switch configuration, DNS and time sources, named owners for the storage and network teams, and a decision on the workloads the cluster will serve. For air-gapped sites you also need a local image registry and a way to move packages in. Your engineers should be ready to review the monitor placement, failure domain layout, and the capacity headroom, because those choices are hard to change after data is written. Confirm the exact list in the written scope.
Can our own engineers build a three-node Ceph cluster?
Yes, a team with Linux, networking and storage experience can build a small cluster from the documentation, and cephadm is designed to manage hosts, services and containers without hand-built systemd units. The harder parts are the decisions that are costly to reverse: CRUSH failure domains, placement group settings, media layout, and the network design. Also plan for the day-two work of drive replacement, health warnings and upgrades. If your team cannot cover these in its normal rotation, a fixed-scope design and build with handover documentation may be the better fit.
When should we move from three nodes to more?
Move to more nodes when you need the cluster to stay fully redundant during a host failure, when capacity growth would push you past safe fill levels, or when erasure coding needs more failure domains than three hosts provide. Erasure coding profiles need enough hosts to place each chunk on a separate failure domain, so a three-host cluster limits your choices. Planning for growth from the start also means choosing a failure domain layout that can grow without rebuilding the pools. Review these thresholds with your capacity model before you buy.
How DeployOpen scopes and delivers a Ceph cluster design
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure. For a three-node Ceph design, we start with a written scope that states the failure domain, replica or erasure coding choice, monitor placement and the recovery behavior you accept, then deliver the design as documents you can review and keep.
How we scope
- Confirm how many hosts, drives and network links are available before any design choice is made.
- Record the replica count, failure domain and whether erasure coding is in or out of scope.
- Agree what a single node outage must preserve, such as client I/O, quorum and data availability.
- Define the growth plan, including when the cluster moves to more nodes and how that changes the design.
- Fixed-scope engagements set scope, deliverables and timelines in writing before work starts.
How we build
- Produce a bill of materials and a host layout that matches the agreed replica and failure domain model.
- Lay out CRUSH placement and monitor placement, then check both against a one-node-down scenario.
- Test recovery behavior on a lab or staging build when your environment allows it, and record the results.
- Document the design decisions, trade-offs and the questions that still need vendor or environment confirmation.
- Review the design with your storage, network and security owners before any production purchase.
How we hand over
- Deliver the design record with diagrams, host inventory, CRUSH map rules and the chosen pool settings.
- Provide a runbook for a host failure, including health checks, expected recovery steps and escalation points.
- Walk your team through the capacity headroom and the trigger points for adding nodes or drives.
- Hand over the design in your own repositories and environment, so your team can maintain it.
- List open items with owners so nothing relies on undocumented assumptions.
How we support
- Ongoing support, upgrades and periodic review of the design are available as a retainer.
- Re-check the design when hardware, Ceph release or workload changes affect the replica or recovery model.
- Reassess the cluster before growth, so the move from three nodes is planned rather than reactive.
- Confirm the supported Ceph release and its end-of-life date with the project before selecting a version.
