How to design Ceph high availability so a failed host does not stop your workloads
Ceph keeps workloads running through a failed host only when the design was built for that failure before the first pool was created. High availability in Ceph is a property of the design: monitor placement, pool redundancy, CRUSH failure domains, network paths and recovery settings. None of these is a feature you switch on later, and the default layout of a small test cluster rarely matches the losses your business expects. Start by writing down the failures you must survive, such as a single disk, a host, a rack, a switch or a whole site. Each one maps to a different layer of the design, so this list becomes the acceptance criteria for everything that follows.
What makes this harder than it looks is that a healthy status and a resilient layout are different things. A pool can report healthy while its CRUSH rule places every replica under one host or one rack, so the next loss of that domain stops I/O or loses data. Review each pool's replication size and min_size, the failure domain named in the CRUSH rule it uses, and whether the CRUSH hierarchy contains the racks or sites you plan to lose. Then confirm the design with a test. In a maintenance window, take a non-critical host out of service and observe which placement groups become degraded, inactive or stuck, and how long they take to recover.
The second trap is recovery. When a failed OSD or host is replaced, Ceph rebuilds the lost copies across the cluster, and that traffic can slow client I/O long after the hardware is back in service. Recovery and backfill limits, the cluster network and the number of OSDs per host all change how long clients stay exposed to that slowdown. Decide in advance how much client performance you will trade for a faster return to full redundancy, and write that decision into the operations runbook. The hardware recommendations advise bonding network links across two switches, which removes a single switch as a shared point of failure, but only if the bonds are cabled that way. Confirm the cabling in the rack, not only in the plan.
This page covers the availability and protection design: monitors, pools, failure domains, network paths, recovery behavior, and the backup and multisite choices that protect data beyond a single cluster. It does not size the cluster, plan a move from SAN, NAS or cloud storage, define RGW realms and zonegroups, run Ceph under Rook in Kubernetes, or harden the cluster against attackers. Those topics have their own pages, and the decisions made here feed into them. Before design starts, ask for or prepare a written list of failure scenarios with the expected impact of each, the recovery behavior you accept, and a named owner for every setting after handover.
Which Ceph components decide whether a failure costs data or uptime
Each Ceph component fails differently, and the cost of a failure depends on which part is affected and how the layout places it. The cards cover the control plane, the data path and the network, with what each design choice does and does not protect against.
Monitors and quorum
Monitors hold the cluster map and agree on its state through a quorum, so the cluster keeps serving I/O only while a majority of them can reach each other. Losing a minority of monitors is survivable by design. Losing the majority, isolating monitors behind a split network, or losing monitor data stores on every monitor host is not. Confirm the monitor count, host spread and disk placement in the release documentation, and decide how many monitor failures your layout must tolerate before you commit to it.
Managers and cephadm placement
Managers run orchestration, metrics and the Dashboard, with one active manager and the others on standby. cephadm places them using service specifications, so placement rules decide which hosts can take over. Managers do not hold object or block data, so losing all of them pauses management and monitoring but does not by itself stop client I/O. Confirm placement labels, host spread and failover behavior for your release, and make sure alerting does not depend on a single manager host.
OSDs and per-drive layout
OSDs store the data, and each one should own a dedicated drive. The Ceph hardware recommendations advise moving WAL and DB onto shared enterprise-class SSDs for HDD-based OSDs, which improves metadata and write behavior but creates a shared dependency. A failed shared SSD takes down every OSD whose WAL or DB it holds, which is a larger event than one failed drive. Check how many OSDs share each SSD, what recovery looks like after an SSD failure, and whether replacement procedures are documented for your layout.
CRUSH map and failure domains
The CRUSH map describes hosts, racks, rows or rooms as failure domains and sets the rules that place each copy or shard in a separate domain. It protects against losing a host, rack or room when the rule separates copies at that level. It does not protect against a rule that places several copies under one switch, power feed or hardware batch, because CRUSH only knows the hierarchy you give it. Confirm the failure-domain levels with your facilities team, label hosts accurately, and test a simulated domain loss before production use.
Replicated and erasure-coded pools
Replicated pools keep full copies and usually recover faster from a single loss. Erasure-coded pools store data and parity shards to use less raw capacity, with more CPU work and different recovery behavior. Both spread objects across OSDs through placement groups, and both protect only against the failure domains their CRUSH rule separates. Neither protects against deletion, ransomware or application-level corruption, which need snapshots and backups. Confirm pool type, size and minimum-available settings in the release documentation for each workload.
Cluster and public networks
The public network carries client and monitor traffic, and the cluster network carries replication, recovery and backfill between OSDs. Separating them keeps recovery traffic from competing with client I/O, but separation alone adds no redundancy. Bonded links spread across two switches protect against a single NIC, cable or switch failure. They do not protect against a misconfigured bond, mismatched MTU across hosts and switches, or a shared upstream router. Check bond mode and MTU end to end, and test a switch failure under load before go-live.
Decisions for Ceph high availability: replication, erasure coding and failure domains
This table compares the design choices that set how a Ceph cluster tolerates failures, with the option, when it fits and the operational cost each one carries. Read each row as a decision to settle before hardware is ordered, since most of these choices are costly to change later. Ceph's CRUSH rule authoring and RGW multisite configuration are covered on sibling pages and are not repeated here.
| Decision | When it fits | Operational cost |
|---|---|---|
| Replicated pools | Latency-sensitive block workloads where write speed and fast recovery matter more than capacity efficiency | Raw capacity equals usable capacity divided by the replica count, so overhead is high; recovery copies whole objects, adding network and disk load |
| Erasure-coded pools | Large, cold or archival data such as backups and object stores where capacity efficiency matters more than write speed | Lower capacity overhead than replication, but higher write latency and more CPU use; partial writes and recovery need more reads across hosts; some RBD and CephFS uses need confirmation with the target release |
| Host-level failure domain | Small clusters where each host is an independent failure unit and rack or site layout is not yet defined | A single rack power or switch failure can take out several hosts at once, so data can be lost or unavailable if placement is not spread further |
| Rack or row failure domain | Clusters spanning several racks with independent power and top-of-rack switching | Requires enough racks to hold the full set of replicas or chunks; uneven rack sizes limit usable capacity and can complicate rebalancing |
| Site-level failure domain or second site | Business requirements that tolerate loss of a whole data center, usually with a stretched or separate cluster | Stretched designs add inter-site latency to writes; a separate second cluster with asynchronous replication adds hardware, a second operations footprint and a recovery point that lags the primary |
| Monitor placement across failure domains | Any production cluster, with monitors spread so quorum survives loss of one rack or host group | Quorum needs a majority of monitors; an odd number spread across three failure domains is the usual pattern, and a two-site layout needs a tiebreaker placed elsewhere |
| Dedicated metadata SSDs for OSD WAL and DB | HDD-based OSDs where small random writes and recovery speed suffer without fast metadata devices | Adds SSD hardware and a failure dependency per shared device; losing a shared DB device takes down every OSD that uses it, so size and split it deliberately |
| Collocated OSD metadata on the data drives | Clusters built on all-flash OSDs or small pilots where extra devices are not justified | Lower hardware spend, but metadata and data compete for the same device, which can slow recovery and random I/O under load |
| Bonded network links across two switches | Any cluster where a single switch failure must not isolate storage nodes | Requires switch support for link aggregation or an equivalent bonding mode and careful mode selection; switch configuration errors are a common source of intermittent faults |
| Separate public and cluster networks | Clusters with heavy recovery traffic that should not compete with client I/O | Doubles the number of network ports and adds addressing and routing to design; confirm the layout against the Ceph hardware recommendations for the chosen release |
Build order for Ceph high availability and data protection
This sequence moves from written failure scenarios to a cluster whose CRUSH placement, monitor quorum, network paths and recovery behavior have each been checked. It ends with a restore test that proves the recovery targets can be met. Each step closes with a check the engineer can repeat later.
Confirm failure scenarios and recovery targets
Before hardware is racked, the client and the delivery team agree in writing on the failures the cluster must survive: a single disk, a host, a rack, a row, a switch, or a whole site. For each one, the document records the acceptable outage and the recovery point the business expects. These answers decide replica count, erasure coding profile, monitor placement and whether a second site is needed. The step closes when the data owner signs the scenario table and every failure maps to a stated behavior.
Map racks, rows and sites into CRUSH
The engineer creates CRUSH buckets for site, row and rack, then places each OSD host under the rack it physically occupies. The hierarchy should match the physical layout closely enough that a failed rack maps to one bucket in the map. The check is the OSD tree view, which lists the cluster hierarchy. Every host should appear under its rack, and no host or OSD should sit directly at the root or in a default bucket.
Place monitors across failure domains
Monitors form quorum by majority, so the engineer spreads an odd number of them across failure domains so that losing any single domain still leaves a majority. Placement is set through the orchestrator, using host labels or explicit host lists. The check is the quorum status view, which should list every expected monitor in quorum. The team should rehearse the loss of one failure domain in a staging cluster before production, so the result is known rather than assumed.
Deploy the cluster with cephadm
The engineer bootstraps the cluster with cephadm, which starts one monitor and one manager, then adds the remaining hosts and applies service specifications for monitors, managers, OSDs and gateways. Each specification names the hosts or labels where daemons may run. The check is the service listing, which should show every daemon running on the host the plan assigned. The cluster health summary should report no warnings before any data is written.
Create pools and verify rule separation
The engineer creates replicated or erasure-coded pools that match the scenario table, and assigns each pool to a CRUSH rule whose failure domain is at least the rack. The rule is then dumped and read to confirm that it selects distinct racks or sites for each copy or chunk. The step closes when the placement groups for every pool reach an active and clean state, and each pool's rule resolves to distinct failure domains.
Bond network links across two switches
Public and cluster networks each get bonded interfaces with member links split across two switches, which the Ceph hardware recommendations advise. The engineer checks the bond state on each host and, if LACP is used, the matching configuration on both switches. Then each path is tested by taking one link down at a time during a maintenance window while watching bond status and cluster health. The step closes when traffic moves over the surviving path and the cluster stays healthy with any single link down.
Set recovery limits and record a baseline
The engineer sets recovery and backfill concurrency so that rebuilding after a failure does not starve client I/O. Conservative values are applied first, then a controlled test marks one OSD out and back in while recovery time, client latency and throughput are measured. The Ceph hardware recommendations advise benchmarking before purchase, so these measurements should be compared with that earlier data. The check is a written baseline with the settings, measured results and sign-off from the operations owner.
Configure a backup target and test restore
The engineer configures the backup or replication target the recovery targets require, such as RBD mirroring, RGW multisite or an external backup tool. A restore is then run from that target into an isolated location, using a representative volume, bucket or file set. The restored data is compared against the source, and the application or service is started to confirm it works. The step closes when the restore completes, the content matches, and the measured restore time is recorded in the scenario table.
What drives the effort and cost of a Ceph high availability design
This table lists the design choices that most change the effort and cost of a Ceph high availability build, with how to reduce each one and whether it is paid once or recurs. Use it to see which decisions to settle early and which requirements to put in a written scope before asking for a quote.
| Driver | Why it changes effort or cost | How to keep it down | One-time or recurring |
|---|---|---|---|
| Failure domains | Mapping CRUSH hosts, racks or sites to physical layout drives rack diagrams, power and network planning, and how many failure scenarios must be tested. | Start from a standard host and rack hierarchy and add sites only when a site outage is in scope. | One-time design, revisited when hardware changes |
| Data volume and growth | More raw capacity means more OSDs, longer recovery windows after a failure and larger benchmarking and backup targets. | Confirm usable capacity and growth expectations in writing, and size the first phase to the committed workload. | One-time design, recurring capacity planning |
| Replication or erasure-coding profile | Replicated pools need more raw capacity, while erasure-coded pools change recovery behavior and placement rules that must be tested. | Limit the number of profiles to those the workloads actually need and document the reason for each. | One-time design, revisited on profile changes |
| Number of pools and client types | Block, file and object clients each need their own pools, access rules and validation, so each added type adds design and test work. | Group workloads with similar durability needs into shared pool profiles and add client types in later phases. | One-time build, recurring for new pools |
| Network topology and switch redundancy | Bonded links across two switches, separated cluster and public networks and failure tests of switch loss add cabling, configuration and verification work. | Use a single standard network template for all hosts and confirm switch redundancy with the network team early. | One-time build, recurring if hardware is added |
| Number of environments | Lab, staging, production and disaster recovery each need matching configuration, data handling rules and separate failure tests. | Keep non-production environments small and reuse the same deployment and configuration code across them. | One-time setup, recurring for maintenance and refresh |
| Backup and recovery targets | Defined recovery point and recovery time goals need backup tooling, restore drills and evidence of the results, which adds design and operating effort. | Agree target recovery goals per data class and test restores on a fixed schedule rather than for every data class at once. | One-time design, recurring restore testing and backup operation |
| Compliance evidence needs | Assessors may ask for configuration records, access logs, change history and test results, which must be collected and kept in a consistent format. | Confirm the control set and evidence list with the assessor before build, then automate collection where possible. | One-time setup, recurring evidence collection |
| Failure testing volume | Each failure scenario, such as a monitor loss, OSD loss, host loss or site loss, needs a plan, execution window, results record and rerun after changes. | Use a staged test plan that starts with the most likely failures and adds scenarios as the design matures. | One-time for initial tests, recurring for retests after changes |
How the high availability design changes by deployment model
A single-node cephadm cluster is built to learn the tooling, not to protect data. Bootstrapping creates one Monitor and one Manager on the same host, so losing that host stops the cluster and any data on it may be unrecoverable. Use it to practice day-two tasks such as adding services, reading health output and running upgrades in a disposable environment.
- Availability guarantee: none. No monitor quorum survives a failure, and no second copy of data exists on another host.
- Failure domains: only the host exists, and it is a single point of failure. No rack, power, network or site separation applies.
- Decisions that change: whether the lab shares hardware or networks with production, and whether learned settings are carried forward or the production cluster is designed from scratch.
- Version floor: cephadm does not support releases older than v15.2.0, so use a supported release for any environment that will be copied into production.
- Migration path: plan the production build as a new multi-host cluster instead of extending this one.
Confirm with your own team that no lab data is treated as production, and record the lab's release so the production build starts on a supported version.
A multi-host cephadm cluster places Monitors, Managers and OSDs across several hosts as containers. Availability depends on three things: a Monitor majority that stays up, replicas placed in separate CRUSH failure domains, and enough surviving capacity for recovery to finish. Design these together, because a correct replica count on a map that puts every copy under one switch still fails together.
- Availability guarantee: the cluster keeps serving while a Monitor majority is up and enough replicas survive in other domains.
- Failure domains: host is the minimum. Rack, power feed and top-of-rack switch domains must match real physical separations, not only labels in the CRUSH map.
- Monitor placement: spread Monitors across hosts in different domains and keep the count odd so a majority is always possible.
- Hardware and network: use a dedicated drive per OSD, move WAL and DB to shared SSDs on HDD-based OSDs, and bond network links across two switches, as the hardware recommendations advise.
- Decisions that change: replicated or erasure-coded pools, memory sized for peak recovery load, and the benchmark results that justify the purchase.
Before go-live, simulate the loss of one failure domain in a test cluster and confirm that data stays available and recovery completes within the window your team accepts.
The cephadm documentation points to Rook for Kubernetes deployments. Rook is a CNCF Graduated project, a status it has held since October 7, 2020. The Rook operator runs Ceph daemons as Kubernetes workloads, so availability now depends on both Ceph's replication rules and how Kubernetes schedules, drains and restarts pods on nodes.
- Availability guarantee: the same Monitor quorum and replication rules apply, but pod rescheduling and node drains can move daemons, so placement must be controlled.
- Failure domains: map Kubernetes nodes to Ceph hosts and label zones or racks to match real physical separations. Labels that do not match the hardware give false assurance.
- Disk planning: dedicate devices to OSDs and avoid sharing those disks with other workloads, so a noisy neighbor cannot slow recovery.
- Shared dependencies: decide whether the Kubernetes control plane and Ceph share failure domains. If one control plane outage takes out both, the storage design has a hidden single point of failure.
- Decisions that change: whether to run Rook inside the same cluster as applications or use an external cephadm cluster for storage.
Before scheduling node maintenance, confirm the Rook version and Ceph release combination against Rook's compatibility documentation, and test a node drain in a non-production cluster.
A two-site design uses asynchronous replication so that a whole site can be lost without losing all copies of data. Each site runs its own Ceph cluster with its own Monitors and quorum, so a site outage does not stop the surviving site. The trade-off is that replication is asynchronous, so writes acknowledged at the primary but not yet shipped can be lost during failover.
- Availability guarantee: service continues only at the surviving site after an explicit failover. Recovery point depends on replication lag at the time of the outage.
- Failure domains: each site is a separate domain with independent power, network and cooling. Shared links, shared DNS or a shared identity service are hidden dependencies that can fail both sites together.
- Replication mechanism: RBD or CephFS mirroring for block and file, and RGW multisite for object storage with realm, zonegroup and zone design. Confirm which mechanism is supported for your release.
- Decisions that change: the lag your business accepts, who authorizes failover, and the failback procedure once the primary site returns.
- Drills: test failover and failback under realistic change rates, because lag grows with write volume and link capacity.
Ask for a documented failover and failback test, and compare the measured replication lag with the recovery point the business has agreed to accept.
What replication protects against and what backup still has to cover
Replication keeps data available when a drive, host or rack fails, but it does not protect you from data that is wrong on every copy. Before you choose any recovery setting, decide what you are protecting against and how much you can afford to lose. Those two answers set the rest of this section, and they belong in writing before anyone edits a configuration value.
Recovery tuning is a trade-off, not a free improvement. When a drive or host fails, Ceph rebuilds the lost copies through backfill and recovery operations, and those operations use the same network links and disks that client I/O needs. Set recovery limits so that rebuilds finish within the window you agreed, and decide in advance which traffic takes priority when the two compete. Then measure client latency during a rebuild in a test cluster, using a load that matches your workloads. The recovery option names and their defaults change between releases, so confirm the current names in the Ceph release documentation for the version you run before you copy any value from older guidance.
Replication answers a narrow question: can the cluster keep serving data after a component fails? It does not answer whether the data is still correct. A deleted bucket, an application bug that writes bad records, a ransomware encryption run or a defect in the storage software is replicated to every copy within seconds. A separate protection layer is therefore required, and the design question is which layer covers which failure. Pool snapshots cover recent logical mistakes on the same cluster. RBD or CephFS mirroring to a second cluster covers loss of a site, but it also carries deletions and bad writes forward, so it needs a snapshot history behind it. RGW multisite replication can serve both availability and recovery for object data, yet it has the same propagation risk for deletes and overwrites unless versioning or retention is configured.
An external backup target gives you a copy that the Ceph cluster and its administrative credentials cannot change or delete directly. Keep it separate from the cluster's administrative access and restrict who can alter its retention settings. The part most teams skip is the restore test. Restore a sample of volumes, file trees and buckets to an isolated location on a schedule, measure how long the restore takes, and compare the result with the data that was written. The restore time you measure is the number you should compare with the recovery target you set earlier, not the time the backup job reports as complete.
Ask your implementation partner or your own team to write three things down before the design is final: the maximum acceptable rebuild duration after a host failure, the maximum acceptable data loss measured in time since the last recoverable copy, and the restore test cadence with its owner. Those targets decide whether you need snapshots, mirroring, multisite replication, an external backup target or a combination. They also decide the recovery settings, which should be tuned against those targets rather than against default values.
Access controls that protect a Ceph cluster during failure and recovery
Availability depends on who can change pools, failure domains and recovery state, and on how quickly those actions can be undone or traced. The cards below cover the access decisions that most often turn a routine fault into data unavailability. Transport encryption, encryption at rest and cephx key scope are covered on the Ceph security hardening page.
Who can run destructive commands
Pool deletion, changes to CRUSH failure domain rules and OSD removal can take data offline or move it across the wrong failure boundary in one command. Limit these actions to a small named group of administrators, usually through separate administrative keys rather than a shared admin credential. Check which accounts can run pool and CRUSH commands on each monitor, and confirm that change tickets are required before those commands run in production.
Safety flags against accidental pool deletion
Ceph requires the monitor setting that allows pool deletion to be enabled before a pool can be removed, and the pool name must be repeated on the command line. Keep that setting disabled in normal operation and enable it only during a planned change window. Confirm the setting's current value on every cluster in your change tooling, and document who may toggle it.
Credentials for backup and mirroring targets
Backup and mirroring jobs hold credentials that can read or overwrite data on a second cluster or object store, which makes them a recovery path and an attack path. Use dedicated, least-privilege identities for each replication peer, store them outside shared configuration files, and record their owners. Ask the team to show how each credential is rotated, how rotation is tested against the peer, and how a failed rotation is detected before the next scheduled sync.
Authorizing and logging recovery actions
During an incident, staff often need elevated access to mark OSDs out, adjust recovery throttles or force placement group operations. Define in advance who may approve each recovery action, who may execute it, and where the command history is recorded. Confirm that cluster and audit logs are shipped off the monitor hosts so that a compromised or overloaded node cannot erase the record of what was changed.
Separating monitoring access from administration
Dashboards, exporters and alerting systems need to read cluster health, but a read-only monitoring identity should never be able to change pools, OSDs or users. Create a monitoring identity with read capabilities only, and confirm it cannot reach administrative endpoints. Review which dashboard roles map to which people, and make sure alert routing does not depend on an administrator's personal account.
How to test that Ceph high availability works and fails safely
The fastest way to learn whether a Ceph high availability design works is to break it on purpose, in a controlled order, and compare the result with the design. Start with the monitors. Stop one monitor host and confirm that the remaining monitors keep quorum and that clients continue to read and write. Then stop enough monitors to exceed the number the design tolerates, and confirm that quorum is lost in the way the design predicts: I/O pauses, the cluster reports the condition, and the monitors agree on state once the hosts return. Record monitor membership and quorum output before and after each step. A test that shows exactly which monitors left and rejoined proves far more than one that only shows the cluster came back.
Next, power off or isolate one host in each failure domain defined in the CRUSH map, one domain at a time. For each one, confirm that client I/O continues, that the cluster reports a degraded state while pools stay available, and that the health output names the affected OSDs and placement groups. Then restore the host and watch data return to full redundancy. Time the rebuild and compare it with the recovery target agreed before the test. If the rebuild misses the target, the cause is usually recovery tuning, network capacity or drive performance, and the design should be revisited before anyone relies on it for a recovery commitment. Confirm the exact health states your release reports, because wording can change between versions.
Cut one of the two switches that the bonded links span and confirm that traffic moves to the surviving path without client timeouts beyond what the applications tolerate, monitor election churn or OSD flapping. Measure that failover while I/O runs, then restore the switch and confirm the links return cleanly. Next, test exposure. From each intended network, confirm that the msgr2 port 3300 answers, and from every other segment confirm that it does not. Scan from the segments that should be blocked, because firewall rules that look correct on paper often miss an alternate route. Finally, time a restore from the backup or mirror target into an isolated location and verify integrity by comparing checksums or sampled object hashes against the source.
Each test needs a pass condition written before it runs. A pass condition names the measurable outcome, such as quorum held, I/O continued, a rebuild finished within the agreed window, or a restored dataset matched its source, and it names the person who signs off. Each test also needs a rollback: the exact commands or hardware actions that return the cluster to its prior state, the person authorized to run them, and a stop rule for aborting the test. Run the full plan first in a staging copy built from the same release, configuration and network layout, then schedule production tests in a maintenance window. Keep the health output, timestamps, command logs and restore reports from every run. Those records become evidence for the operations record and for any independent assessment of your controls.
Questions teams ask before designing Ceph high availability
What drives the cost of a Ceph high availability design?
Scope and the number of failure domains drive most of the cost. The cost comes from people hours for design, review of hardware and network, lab or staging work, test execution and written documentation. One-time work includes the design, the build review and the first series of failure tests. Recurring costs appear only if you buy a retainer for upgrades, monitoring or continuous testing. Costs rise with the number of OSD hosts, racks, power feeds and switch domains to model, multisite requirements, the number of failure scenarios and retests, and how much access you can give during testing. Ask for an itemized quote that lists each deliverable, each test and the assumptions about your hardware. DeployOpen works to a fixed scope agreed in writing before work starts.
How long do Ceph high availability design and testing take?
No duration applies until the scope is written, so treat any single number with caution. The schedule depends mainly on how quickly you provide the hardware inventory, network diagrams and failure domain map, how many failure scenarios the test plan covers, whether tests can run only in change windows, and how many sites are involved. Each failed test adds a fix and a rerun, so the schedule also depends on how stable the first build is. Ask for a timeline tied to named milestones in the written scope, with dependencies on your team listed separately. Use the scope and the test results as the planning basis, and ask how the plan adjusts if a dependency slips.
What does our team need to provide for a Ceph design?
Your team supplies the inputs the design depends on. Prepare these before kickoff:
- Hardware inventory: host models, drive types for each OSD, memory, and any shared SSDs for WAL and DB devices.
- Network diagrams: switches, bonded links across two switches, VLANs, and routing between public and cluster networks.
- Failure domain map: which hosts share racks, power feeds, switches and sites.
- Recovery targets: the recovery point and recovery time your business has agreed for each data class.
- Change windows: when tests may take nodes or links offline, and who approves and monitors them.
Name an owner for each item. Gaps in these inputs usually appear as extra design review cycles, so the earlier they are supplied, the fewer surprises appear in the written scope.
Can one engineer design and test Ceph high availability alone?
One experienced engineer can design a small cluster alone, but testing is hard to do safely by one person. A failure test removes a host, a rack or a network path, so it needs a second person to watch health output, stop the test if needed, and record results. The design also benefits from a second reviewer who checks failure domains, monitor placement and recovery settings against your recovery targets. Single-person work creates a bottleneck on approvals and knowledge, and leaves no one to run tests when the designer is unavailable. Ask which of your roles cover storage, networking, change approval and on-call. The written scope should name who on your side attends each test, so the test results reflect how your operations team actually responds.
Does Ceph replication alone count as a backup strategy?
Replication alone does not count as a backup. Replicas protect against losing a disk or a host, but they also copy deletions, corrupted writes, ransomware encryption and operator mistakes to every replica. A backup needs at least one copy that is isolated by time, location or access, so an error on the live cluster cannot reach it. Plan snapshots or exports for each data class, keep at least one copy in a separate failure domain, and set retention to match your recovery targets. Confirm which snapshot, export and mirroring features apply to your Ceph release before you choose a method. Schedule restore tests, because a backup that has never been restored is unproven. DeployOpen plans and tests backups for self-hosted platforms as part of the same written scope.
Is a second data center required for Ceph high availability?
No, a second site is not required for host-level availability. A single site with separate failure domains for racks, power and switches can survive the loss of individual hosts or devices when it is designed and tested for that. A second site becomes necessary when the business must survive the loss of a whole location. Monitor quorum needs a majority of monitors, so a two-site layout needs a third location or tiebreaker to keep quorum when one site is lost. Replication across sites also changes write latency and bandwidth needs, so benchmark the links before you commit. Confirm your site-loss requirement against the recovery targets first, then choose the layout that meets it.
What happens when a Ceph failure test does not pass?
A failed test becomes a recorded finding that drives a fix. The written scope should state the pass criteria for each test before it runs, so every result is judged against an agreed standard. The team then classifies the finding, identifies the root cause, changes the design or configuration, and reruns the same scenario along with any affected tests. Each change goes through your change control, and the final report lists every failure, fix and rerun. Some failures reveal design limits that need a decision on hardware, recovery targets or failure domains, and those decisions stay with your team. Test results are the basis for planning, so a failed test shows where the design needs work and what to change.
Does DeployOpen issue an audit report for a Ceph design?
No. DeployOpen prepares systems, controls and evidence for compliance assessments, but it does not issue audit reports, attestations or certifications. Independent auditors and certification bodies issue those. For a Ceph high availability design, DeployOpen can document the failure domains, recovery settings, backup approach, change records and failure test results that an assessor may request. Ask your assessor early which evidence they expect, and agree on the format in the written scope. Keep the design documentation under your own version control so it stays current after handover. Confirm which compliance requirements apply to your organization with your assessor, since that responsibility remains with you.
What to monitor, back up, upgrade and document after a Ceph high availability build
Ongoing operations on a Ceph high availability build depend on a short set of signals that show redundancy slipping before users notice. Watch the health state reported by ceph status and ceph health detail, and alert on any move away from HEALTH_OK, including HEALTH_WARN. Degraded placement groups have fewer copies or shards than their pool requires, and undersized placement groups have fewer replicas than the target. Both need alerts that name the pool. Track monitor quorum changes as well, because losing quorum halts cluster map updates. Track OSD up and down events too, since a flapping OSD often points to a failing drive, a bad network path or memory pressure on the host.
Capacity headroom decides whether a host failure is an incident or an outage. After a host fails, the surviving OSDs must absorb the copies or shards that were on it, so the cluster needs free space to rebuild without pushing any OSD past its full threshold. Derive the headroom from your failure domain map: estimate the data held by the largest failure domain, add the space the rebuild needs on the remaining devices, and check the result against current utilization in ceph osd df. Record recovery duration after every failure or maintenance event. A lengthening trend means backfill throttles, network capacity or device health has changed, and that trend should trigger a capacity review before the next failure tests the cluster.
Cluster state is only part of what you need to rebuild. Back up the control plane on a schedule and store copies outside the cluster. Include the output of ceph config dump, the cephadm service specifications, exported CRUSH and OSD maps, and cephx keys exported with ceph auth export. Treat these exports as secrets, because anyone holding a keyring can act as that client. Replication and erasure coding protect the data on OSDs, and these exports only rebuild the configuration and identities around that data. Run restore tests on a separate lab cluster at a fixed interval. Record how long each step took, whether restored keys authenticate, and whether restored CRUSH rules place test objects as the map describes.
The release schedule sets how often you must upgrade. The Ceph releases page lists Squid with an estimated end of life of October 31, 2026, and Tentacle with an estimated end of life of June 1, 2027. The project states that actively maintained releases receive periodic backports and security fixes, so a cluster on an end-of-life release stops receiving them. Confirm these dates on the releases page at planning time, because estimates can change. Plan the next major upgrade window before the estimated end-of-life date, leaving time to rehearse the upgrade on a lab cluster that mirrors your CRUSH layout and pool types.
Upgrade in an order that keeps quorum and data availability intact at every step. With cephadm, the orchestrator upgrades managers first, then monitors, then OSDs, then the remaining daemons such as metadata servers and Object Gateways. Before moving to the next host or failure domain, confirm the cluster has returned to HEALTH_OK with placement groups active and clean. Upgrade one failure domain at a time so every pool keeps its replicas or shards during the change. Keep five records current: the failure domain map, recovery targets, test results, runbooks and change history. Incident responders and auditors will ask for these first, and an out-of-date runbook can be worse than none.
How DeployOpen works on Ceph high availability and data protection
DeployOpen plans, builds, documents and supports Ceph high availability and data protection inside your own infrastructure, whether on your cloud accounts, data centers or air-gapped networks. The work starts from written requirements for failure tolerance, recovery expectations and data protection, then turns them into a tested design your team can operate.
How we scope
- Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
- The written scope lists the failure domains, monitor placement, replication or erasure coding choices and the pools they apply to.
- Scope also names the multisite, backup and recovery tests to be run, and who is responsible for each one.
How we build
- Monitor quorum, manager placement and OSD layout are built and checked against the agreed failure domains before data is loaded.
- Network design covers bonded links across two switches and separation of client and cluster traffic, as the Ceph hardware recommendations describe.
- Recovery and backfill settings are tuned and then tested with simulated host, rack and disk failures, with results recorded.
How we hand over
- Handover includes the as-built design, failure domain map, pool inventory and documented recovery runbooks for each failure type.
- Backup and restore procedures are demonstrated with your team, including restore timing observed during testing.
- Your operators receive a list of health checks and alert conditions to monitor, with the cephadm procedures used to act on them.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers, scoped separately from the build.
- Support can cover periodic recovery drills, review of cluster health after hardware changes and planning for maintained Ceph releases.
- Confirm with us in the quote which releases, response expectations and test cadence a retainer includes before signing.
