Ceph failure domains across racks and rooms

Designing Ceph failure domains across racks and rooms for your site

A failure domain in Ceph is a group of hardware, such as a host, rack, row or room, that can fail together, and the failure domain you choose determines how many copies or erasure-coded chunks of each object land in separate groups. You make this choice before creating pools, because the CRUSH rules that enforce it are attached to pools and changing the layout later means moving data across the cluster. For a new cluster spread over racks or rooms, the question is simple to state: which single event must the cluster survive without losing data or availability, such as a failed host, a tripped rack power feed or a room outage?

CRUSH is the placement algorithm that decides where each object goes, and it works from a hierarchy of buckets that you describe in the CRUSH map. Each OSD is placed under its host, hosts sit in racks, racks sit in rows or rooms, and a CRUSH rule tells Ceph to choose a number of distinct buckets at a chosen level for each replica or chunk. Your site's layout is recorded through CRUSH locations, which cephadm and the OSD tooling can set, and you should confirm them with ceph osd tree before any pool is created. A host-level domain is usually enough when a single host is the largest unit you can lose and you have enough hosts for the replica count. Rack or room levels become necessary when a rack power feed or a room is a realistic loss and you need data to survive it intact.

Which failure domain fits which site layout

This table compares host, rack, row and room as CRUSH failure domains, showing what each one protects against, the site layout it needs and its main trade-off. Read it by matching your physical layout to the row that fits, then check the replica or erasure code requirement for that level before you set CRUSH locations.

Failure domain	What it protects against	Site layout it needs	Main trade-off
Host	A single server failure, such as a node, boot device or OSD host crash	At least three hosts with separate power supplies, ideally spread over more than one rack	Does not survive loss of a rack, PDU, top-of-rack switch or room
Rack	Loss of a rack's power feed, top-of-rack switch or cabling	At least three racks, each with independent power and network paths	Needs enough racks to hold each replica or chunk, so small sites may not qualify
Row	Failure of a row's shared power distribution or cooling zone	At least three rows, with distinct upstream feeds for each row	Rows are often mapped to rooms or power zones, so the layout must match the real electrical design
Room	Loss of a whole data hall, fire suppression event or building-level outage	Two or more rooms with separate power, cooling and network paths, usually with a third site or tiebreaker for monitor quorum	Each replica or chunk sits in a different room, which raises inter-room latency and cost of links
Mixed hierarchy (rooms and racks)	Room loss plus rack loss, using a rule that picks one replica per room and per rack within it	Enough rooms and racks in each room to satisfy the rule, with one rack per replica in each room	Usable capacity is set by the smallest room, and any room loss leaves less redundancy until repair finishes
Mixed hierarchy (rooms and hosts)	Room loss with host-level spread inside each room when racks are limited	Two rooms with at least two hosts per room, plus a monitor tiebreaker in a third location	Losing a host in a room reduces redundancy quickly, so recovery capacity must be sized for that case

How replica count and erasure code profiles need failure domains

Every replicated pool and erasure-coded profile asks CRUSH to place each copy or chunk in a different failure domain. The number of racks or rooms you can name determines which layouts are possible, and the layout determines how much of your raw disk you can actually use.

Replicated pool size

A replicated pool with size set to 3 stores three full copies of each object, and CRUSH must place each copy in a separate failure domain, such as a host, rack or room. If the CRUSH rule selects fewer distinct domains than the replica count, placement stays degraded. Usable capacity is roughly raw capacity divided by the replica count, before headroom. Confirm the rule's failure domain against your replica count before ordering hardware.

The min_size floor

The min_size setting is the number of copies that must be available before a pool accepts writes. Setting it below the safe level lets writes succeed with fewer copies in place, which raises the risk of data loss if another domain fails. With size 3, many operators keep min_size at 2 so a single failed domain does not stop I/O. Ask your team to document the trade-off between availability and durability for each pool.

Erasure code k and m

An erasure code profile splits each object into k data chunks and adds m coding chunks, and it can tolerate the loss of up to m chunks. Every one of the k + m chunks needs its own failure domain. For example, a 4+2 profile needs six distinct domains, and a rack-level failure domain with only five racks cannot place it. This is illustrative; confirm the profile against your topology.

Usable capacity by profile

Erasure coding stores k out of every k + m chunks as usable data, so its overhead is lower than replication for the same failure tolerance. Replication with 3 copies yields about one third usable capacity, while higher-k profiles yield more usable space but need more domains. Work out the ratio for your chosen profile, then reserve headroom for a failed domain and the full ratios. The sizing guide covers the arithmetic in detail.

Domain count before rack count

Count the failure domains you can physically support before choosing a profile. A three-room site can host a replica count of 3 at room level, but a 4+2 profile needs six rooms or six racks at a lower level. Rack and row counts also change when hardware is retired or added. Ask your facility team for the number of racks per room and how power and network feeds map to each room before you commit.

Recovery after a domain loss

When a domain fails, CRUSH cannot re-place data into the missing domain, so the pool runs degraded until the domain returns or capacity is added elsewhere. Erasure coded pools with few spare domains recover into fewer places and can take longer to return to full redundancy. Confirm recovery behavior with a test in your own environment, and plan the spare capacity needed to absorb one lost domain.

Failure domain choices for two-room and three-room sites

Two-room sites are the arrangement most likely to cause trouble. Ceph monitors need a majority to form quorum, so whichever room holds more monitors effectively controls the cluster. If that room is lost, the cluster stops accepting changes until monitors are restored, and the surviving room cannot decide alone. A tiebreaker location is needed to break the tie.

Monitors: Run an odd number of monitors and place a tiebreaker monitor in a third location, such as a small separate site. Confirm its network latency and power independence before relying on it.
Failure domain: Using the room as the CRUSH failure domain means copies are split across two rooms. Illustrative: four replicas, two per room. Losing one room leaves two copies, which may fall below the pool's min_size.
Room loss: Quorum survives only if the remaining room and the tiebreaker together hold a majority. Pools can keep serving I/O in a degraded state only if they stay above min_size.
Capacity: Each room must hold enough headroom to absorb data after the other room fails. Plan against the full ratios, not the average fill level.
Operator decisions: Decide in advance which room holds the monitor majority and what happens to writes during a room outage.

Before go-live, power off one room in a test environment and confirm that quorum, placement and client I/O behave as your design states.

Setting CRUSH locations and rules before the first pool

This sequence builds the physical hierarchy in CRUSH, places every host and OSD inside it, and creates a rule that spreads replicas across the chosen failure domain before any data is written. Each step ends with a check so that placement problems are found while the cluster is still empty.

Map the physical hierarchy on paper
The architect and the facilities team list every room, row and rack, the hosts in each rack, and the power and network feeds each rack depends on. Record which racks share a power feed, a top-of-rack switch or a cooling zone, because those shared parts decide what a single failure can take down. The output is a written inventory with one line per host, showing its room, row and rack. Done when every host in the bill of materials appears exactly once and the team agrees on which failure domains are meant to survive an outage.
Create the CRUSH buckets for each level
The Ceph administrator creates the room, row and rack buckets in the CRUSH map, usually with ceph osd crush add-bucket and a bucket type that matches the level. Buckets are then linked under the default root so that the hierarchy is room, row, rack, host and OSD. Confirm the bucket type names and any extra levels against the Ceph release in use before running commands. Done when the empty buckets appear in the CRUSH tree under the intended root, with no stray buckets left at the top level.
Assign each host and its OSDs a location
The administrator moves every host bucket into its rack with ceph osd crush move, then confirms that each OSD sits under its host. In a cephadm-managed cluster, check whether OSD placement is inherited from the host or set explicitly, since this affects what happens when a drive is replaced. Verify the result with ceph osd tree and compare it line by line with the inventory from the first step. Done when every OSD is listed under the expected host, rack and room, and no host is attached to the wrong bucket.
Check the tree and device classes
Before any rule depends on the map, the administrator reviews the full CRUSH tree and the device class of each OSD, which Ceph assigns automatically (for example hdd, ssd or nvme). Mixed media that is placed in one failure domain tree can distort later capacity calculations, so flag any OSD with an unexpected class. Review the output against the expected count of drives per host. Done when the tree shows the planned count of OSDs in every rack and every class matches the media it is meant to represent.
Create a rule with the failure domain and device class
The administrator creates a replicated or erasure-coded CRUSH rule that names the chosen failure domain (rack or room, for example) and, if needed, a device class, using commands such as ceph osd crush rule create-replicated. Confirm the exact syntax and any erasure code profile options against the Ceph release in use. Without a sufficient number of racks or rooms for the replica count or code profile, the rule will not be able to satisfy placement, so the team should check that count now. Done when ceph osd crush rule dump shows the intended failure type and device class for the new rule.
Create a test pool and confirm placement
The administrator creates a small test pool that uses the new rule, sets its size, and checks where its placement groups land. Using ceph pg ls-by-pool or the PG mapping commands, confirm that each PG's replicas sit in different racks or rooms, as the rule requires. Repeat the check with a deliberately impossible rule in a scratch test if the team wants proof that placement fails safely. Done when every sampled PG shows replicas in distinct failure domains and the pool reports healthy placement with no undersized PGs.
Create production pools and record the design
Once the test pool passes, the administrator creates the production pools with the same rule and records the CRUSH map, rule names, replica or code settings and failure domain in the cluster design document. Remove the test pool so it does not hold capacity or confuse later audits. The facilities team signs off that the recorded failure domains match the physical layout. Done when every production pool is listed with its rule, the design document matches ceph osd tree and the rule dump, and the test pool has been deleted.

Operational risks when rack and room domains change over time

Failure domains are set during design, but racks, rooms and hardware keep changing for years afterward. The risks below share one pattern: a physical change that the CRUSH map, power path or monitor layout no longer reflects. Each card names a risk and the check that catches it early.

Host moves trigger data movement

Moving a host to another rack or room changes its CRUSH location, so Ceph relocates the placement groups that now map elsewhere. That movement competes with client I/O while it runs. The CRUSH location change is what starts the data movement. Before any move, confirm the host's drives, network and power path, schedule the work, and wait for the cluster to report healthy before the next change.

Mislabeled CRUSH locations

A host labeled with the wrong rack or room places its replicas in a domain the physical layout does not describe. Ceph has no way to detect that the label is wrong. Compare the output of ceph osd tree with your physical inventory after every change, and record each host's location in your change log. Ask the team who signs off on label changes before they are applied.

Shared power and network paths

A failure domain holds only if the hardware inside it fails independently. Two switches in one room, or a single power feed serving hosts in two racks, make the shared element the real domain, whatever the CRUSH tree says. Ceph trusts the CRUSH tree, but the building decides what actually fails. Check that each host's bonded links land on separate switches and that each rack's power follows the layout you designed. The hardware recommendations advise bonding network links across two switches.

Monitor placement drift

Monitors must hold a majority to form quorum, so their placement has to follow the same domains as the data. If new hardware puts two monitors in one room and that room loses power, the cluster can lose quorum and stop serving I/O. Monitors that share a room share its outage. Review monitor and manager locations whenever a room or rack is added or retired, and confirm the placement with your cephadm service specifications before handing over.

Capacity skew between domains

When racks receive different hosts over time, each domain holds a different amount of raw capacity, and the smallest domain limits how much data can be placed safely. Cluster totals hide per-domain skew. Recheck capacity for each rack or room after every expansion. Ask the team to confirm the headroom it keeps for a failed host inside the largest domain, and set alerts on the nearfull and backfillfull ratios so warnings arrive before writes stop.

Recurring checks and alerts

Set alerts on the cluster health state, on OSDs that are marked down or out, on recovery that stays active after a planned move, and on changes in monitor quorum. Every hardware change needs a post-change check. Run a CRUSH tree review, a power and network path check and a monitor placement check at each change and on a review schedule you define. Keep the results in your runbook so the next team can compare them against earlier reviews.

Questions buyers ask about Ceph failure domains

How much effort does a Ceph failure domain design take?

The design effort depends on how many rooms, racks and hosts are in scope, how many pools need different protection, and how much of the existing facility data is available. The main work is collecting the physical layout, agreeing on the replica count or erasure code profile for each pool, mapping every OSD host to its rack and room, and testing the CRUSH rules before any data is written. Effort rises with each additional room, each pool with different durability needs, and each undocumented power or network path that must be verified. Ask any provider for a written scope that lists the sites, pools, hosts and verification tests included, and for the effort assumptions behind the quote.

Can a two-room Ceph cluster survive the loss of one room?

Usually not with full data availability and no further action, and this is the point buyers most often misjudge. With two rooms, the usual replica placement puts copies in both rooms, so losing one room can leave fewer copies than the pool needs, and the pool may stop accepting writes until the room returns. Monitors also need a majority to form quorum, so a two-room split typically needs a third site or a tiebreaker placement to keep quorum when one room fails. A three-room layout, or a third location holding monitors, is the common way to tolerate a full room loss while keeping the replica count. Confirm the exact behavior for your replica count and monitor placement in a test before relying on it.

What does the design need from our facilities and network teams?

The design needs accurate physical records and some confirmations from facilities and network teams. Facilities should confirm which racks share power feeds, which rooms share cooling and fire zones, and where each rack sits relative to the others. Network teams should confirm which top-of-rack switches sit in which rooms, how links cross between rooms, the latency and bandwidth between rooms, and whether bonded links can land on separate switches. Both teams should agree on a naming scheme for rooms and racks, because those names become CRUSH bucket names. Ask your team to produce a site map and a power and network path diagram before the design workshop starts.

Can we change the failure domain layout after the cluster is in production?

Yes, but changes move data, so they must be planned. Ceph can add or rename CRUSH buckets, move hosts between racks or rooms, and change CRUSH rules, but each change makes the cluster rebalance data across the affected OSDs. That rebalancing consumes network and disk bandwidth and can slow client traffic while it runs. Replica count and erasure code profile changes on existing pools also cause data movement and may need a new pool and a data migration rather than an in-place edit. Plan changes in a maintenance window, reduce recovery impact during the move, and treat a room or rack move as a significant operation with its own rollback plan.

What does a failure domain choice do to usable capacity?

Usable capacity falls as the protection level rises, and wider failure domains usually need more raw disk for the same usable space. With replication, usable capacity is roughly raw capacity divided by the replica count before headroom is applied. With erasure coding, usable capacity depends on the data and coding chunk counts, and each failure domain must hold chunks that can be rebuilt. Spreading across more rooms or racks can require more hosts before the profile fits, which affects the bill of materials. Keep headroom for a failed host or rack so the cluster can recover without filling. Ask any provider to show the usable capacity arithmetic for each pool with the chosen replica count or profile.

Can our own team design the failure domains without outside help?

Yes, if your team has Ceph operations experience, accurate site records and time to test. The work involves CRUSH map concepts, replica and erasure code arithmetic, monitor placement, and tests that deliberately take down a host, rack or room in a non-production cluster. Teams that have run Ceph before often handle the design internally and bring in a partner for review, for a second opinion on room-loss behavior, or to build the test plan. Consider the cost of an outage caused by a wrong layout, the staff time for testing, and the skills needed to operate the rules after handover. Ask a partner to review your design in writing before you commit to a model.

How do we check that the failure domains work before go-live?

Test each failure you plan to survive, in a staging or pre-production cluster, and record what Ceph does. Check the CRUSH tree to confirm that each host sits under the intended rack and room, then review the rules to confirm that each pool places replicas or chunks across the required domains. Simulate a host outage, a rack outage and a room outage, watching for degraded or incomplete placement groups, blocked writes and monitor quorum. Confirm that recovery finishes and that free space stays above the full ratios during rebuild. Keep the results as evidence and repeat the tests after any hardware or CRUSH change.

How DeployOpen scopes and delivers failure domain design

DeployOpen approaches failure domain design as a written, fixed-scope piece of work tied to your hardware, rooms and replication targets. We plan the CRUSH layout, build it in your environment, test that a lost rack or room behaves as the design predicts, and hand over the documentation your operations team needs to run it.

How we scope

We record the physical layout: hosts, racks, rows and rooms, and which of them share power or cooling.
We agree in writing which failure domain each pool must survive, such as host, rack or room.
We state the replica count or erasure code profile, the usable capacity result and the headroom you need for a failed domain.

How we build

We set CRUSH locations for every host and define rules that spread replicas or chunks across the agreed domains.
We check that the monitor placement matches the same domains so quorum survives the failure you planned for.
We test placement on the cluster before data arrives, using the placement output to confirm each copy lands in a separate domain.

How we hand over

You receive the CRUSH map, rule definitions and a record of the failure scenarios we tested and their results.
We document the usable capacity figures, the full and nearfull ratios set for your layout and the steps to add a new rack.
We walk your operations team through the checks they run after a host, rack or room outage.

How we support

Ongoing support, upgrades and continuous testing are available as a retainer, so layout changes stay checked over time.
We can re-run placement and failure tests when you add racks, rooms or new pools with different replication needs.
Scope changes are agreed in writing before work starts, and any follow-on effort is quoted from that revised scope.

Designing Ceph failure domains across racks and rooms