Ceph capacity planning: what usable capacity means for your cluster
If you are deciding whether to buy or expand a Ceph cluster, the question to answer first is how much data you can store, and the answer is less than the raw capacity printed on the drive labels. Usable capacity is the raw capacity divided by the overhead your data protection scheme adds, minus the headroom you keep free so the cluster can heal. Those are three different numbers: raw capacity, usable capacity and safe capacity. Raw capacity is the sum of every OSD. Usable capacity is what a pool can hold after replication or erasure coding writes extra copies or parity. Safe capacity is the usable figure reduced further so that the cluster can recover when a host fails and so that writes do not stop at the full ratio.
This applies equally to block (RBD), file (CephFS) and object (RGW) pools, because all three store their data as RADOS objects placed by CRUSH. The overhead rules are set per pool, so a cluster can carry a three-way replicated block pool next to an erasure-coded object pool, and each one consumes raw space differently. Confirm the exact behavior of the release you will deploy, Tentacle (20.2.x) or Squid (19.2.x), against the Ceph active releases page before you commit to a sizing spreadsheet, and record which release your bill of materials assumes.
Your next step is to write down, for each planned pool, its protection scheme, its expected share of the data and the failure domain it spans. Then calculate usable capacity per pool, subtract the space needed to rebuild after your largest expected failure, and compare the result to the fill levels Ceph enforces. The ratios that control writes are covered in the sections that follow, so use this section to set the starting figures. Ask your hardware vendor and your team to confirm the drive count, drive size and host count before any of these numbers are treated as final.
Replication versus erasure coding: which overhead fits which workload
This table compares replicated and erasure-coded Ceph pools on the factors that change usable capacity: raw overhead, failure tolerance, recovery cost, CPU use and typical workloads. Read it as a starting point, then confirm minimum host counts and profile options for your chosen layout in the Ceph documentation.
| Factor | Replicated pool | Erasure-coded pool |
|---|---|---|
| Capacity overhead | Usable share is roughly 1 divided by the replica count, so three replicas leave about a third of raw space for data | Usable share depends on the k and m values in the profile, so a wider profile leaves more usable space but needs more failure domains |
| Failure tolerance | Data survives as long as enough replicas remain in different failure domains, which makes the failure domain choice central | Data survives up to m lost chunks, and the chunk layout must spread across enough hosts or racks to tolerate a failed domain |
| Recovery behavior | Recovery copies whole objects from surviving replicas, which is usually simpler and faster to start after a host loss | Recovery rebuilds missing chunks from several survivors, which costs more network and disk reads per lost object |
| CPU cost | Low, since the write path copies data to each replica without encoding | Higher, since encoding and decoding use CPU on writes and during degraded reads or recovery |
| Latency profile | Generally better suited to small random writes because each write has a simple fan-out | Small writes can be slower, so confirm performance for your I/O size before choosing it for latency-sensitive data |
| Typical workload | VM block storage and databases on RBD, where low latency and fast recovery matter most | Cold or large object data on RGW, such as backups, archives and media, where capacity efficiency matters more than write latency |
| Capacity planning impact | Plan usable space at the replica count, then reserve headroom for a failed host before setting full ratios | Plan usable space from the profile, but confirm that the number of hosts can absorb a failure without overfilling survivors |
| Minimum hosts and profile options | Confirm the minimum host count for your replica count and failure domain in the Ceph documentation | Confirm the minimum host count and any profile constraints for your k and m values in the Ceph documentation before ordering hardware |
How much headroom a failed host needs before data is safe
A Ceph cluster stays safe only if the surviving hardware can absorb the data from a lost host or failure domain. The cards below link the decisions that set that headroom: how much space to hold back, how the failure domain changes the math, where recovered data lands, how fill becomes uneven, and when to test again.
Reserve space for recovery
Decide how much free space to hold back so the cluster can re-create copies after losing one host or failure domain. Work from the usable capacity you plan to sell or allocate, then subtract the raw space that would be needed to rebuild the largest failed unit. Ask your team to state in writing which failure and how many copies must be restored before the cluster is considered healthy again. Confirm the exact recovery behavior with the Ceph documentation for your release.
Match headroom to the failure domain
The failure domain you set in CRUSH determines how many places a lost copy can be rebuilt. A host failure domain in a small cluster leaves fewer surviving hosts to absorb data than a rack or room failure domain spread across many hosts. Before you size headroom, list each failure domain, count its raw capacity, and check that enough separate domains exist for your replica count or erasure code profile to be placed after a loss.
Plan for writes onto surviving OSDs
When a host drops out, its data is re-created on the OSDs that remain, and those OSDs take on extra writes at the same time as client traffic. A cluster that looks half full can still have OSDs near their limits. Check per-OSD utilization, not only the cluster total, and confirm that the surviving drives can receive the recovered data without crossing the backfillfull or full ratios.
Watch uneven fill after removal
Removing a host changes the CRUSH weights of the remaining hosts, so some OSDs fill faster than others. Drives that were already fuller or smaller can reach nearfull first, even while the average looks safe. Compare the utilization spread across OSDs and hosts, and decide whether to add drives or hosts to even it out before the next expansion or failure.
Re-test headroom after hardware changes
Headroom calculated for one hardware generation does not hold after you add, retire or replace drives, hosts or racks. Each change alters raw capacity, failure domain counts and fill balance. Repeat the failure calculation after every change, record the result with the cluster configuration, and confirm the recovery time with a test in a non-production environment where you can.
Setting nearfull, backfillfull and full ratios without surprises
The nearfull, backfillfull and full ratios are the cluster-wide thresholds at which Ceph warns, stops accepting certain data movement, and stops accepting client writes. Each ratio is a fraction of raw OSD capacity. The nearfull ratio raises a health warning so someone can add capacity. The backfillfull ratio blocks backfill onto an OSD that is too full to receive data, which keeps recovery from pushing a device over the edge. The full ratio stops client writes entirely. Backfillfull sits between the other two because recovery must be able to proceed after a failure, while clients must still be stopped before any OSD runs out of space. If backfillfull were set at or above full, a rebuild could fill an OSD to the point where writes stop without any earlier signal.
Confirm the default values and the exact commands to read and set these ratios for Tentacle and Squid in the Ceph documentation, because defaults and the mechanism for changing them have changed across releases. On a running cluster, read the current values and the per-OSD utilization before you touch anything, and check the health output for nearfull or backfillfull warnings. The ratios are set cluster-wide, so a change affects every OSD. On a cephadm-managed cluster, verify the setting took effect, confirm health returns to its prior state, and recheck after any host or drive is added, since the thresholds are applied against each device's fill level.
Changing the ratios is a last resort, not a capacity plan. Raising full to buy time hides the problem and leaves no margin for a host failure, because the space a failed host held must be rebuilt onto the survivors. Plan capacity from the headroom calculation earlier in this guide, order hardware when nearfull first appears under normal load, and treat any ratio change as a temporary measure with a written reason, a rollback value and a date for the hardware to arrive. The decision should be documented alongside the operations runbook so the next on-call engineer does not inherit a threshold nobody remembers changing.
How to build a capacity model before you order hardware
This sequence turns pool inventory, redundancy choices and measured hardware performance into a usable capacity figure, with headroom for a failed host and alert thresholds. The output is a written model that shows when the cluster needs more capacity.
Inventory pools and their growth
The storage owner lists every planned pool, its workload (RBD block, CephFS file or RGW object), its current data and its expected growth over the planning period. Each pool needs a named owner and a growth estimate written down. Read current usage from the existing cluster with
ceph dfwhere one exists. The check is that no pool is missing an owner or a growth figure, and the list matches what applications will actually store.Choose replication or erasure coding per pool
For each pool, pick a replicated profile or an erasure-coded profile. Replication stores a set number of full copies, so usable space is raw space divided by the copy count. Erasure coding stores data and parity shards, so usable space is the data shard share of the total. Confirm the chosen profile against the host count in your planned failure domain, because a profile needing more shards than available hosts cannot place data. The output is one profile per pool and a check that each one fits the hardware.
Define the failure domain for each pool
Decide whether each pool spreads copies or shards across hosts, racks, rows or rooms. The CRUSH map must contain enough buckets at that level for the profile to place every copy or shard in a separate domain. Ask the storage team to show the current CRUSH tree, then compare its bucket counts with the profile requirements. Choosing a wider failure domain usually reduces what the hardware can safely hold, so record that effect in the model. The check is that each pool maps to a domain the topology can satisfy.
Compute usable capacity after overhead
Add the raw capacity of all planned OSDs, then apply each pool's redundancy overhead to get its usable share. Raw drive capacity is larger than usable capacity because of redundancy and internal metadata, so do not treat the raw total as available space. Confirm the per-OSD usable figure with a test cluster or benchmark rather than assuming the marketed drive size. The output is usable capacity per pool and in total. Cross-check the total against the MAX AVAIL values reported by
ceph dfon an existing cluster.Subtract failed-host headroom and ratio margins
Estimate how much data must be re-protected if the largest failure domain fails, and make sure the remaining hosts can absorb it without pushing any OSD past its backfillfull threshold. Also subtract the margin needed to stay below the nearfull and full ratios during normal operation. Confirm the current ratio values with the Ceph documentation for your release, since the defaults can change between versions. The check is that projected utilization after a single domain failure stays below the backfillfull ratio.
Set nearfull, backfillfull and full alert thresholds
Configure the three ratios so the cluster warns the team well before writes are blocked. Nearfull should trigger operational review, backfillfull should stop data movement that would fill OSDs, and full should be the last line of defense. Read the active values with
ceph osd dumpand set any changes through the supported configuration interface. Confirm that monitoring raises alerts on the nearfull condition and that someone receives them. The check is that a test or simulated warning reaches the on-call channel.Benchmark candidate hardware before ordering
Run a benchmark on the candidate drives, hosts and network before committing to a purchase, since no two clusters perform the same way. Measure per-OSD usable size, throughput and recovery behavior under load, then compare the results with the assumptions in the model. Adjust the capacity and headroom figures if measured values differ from the vendor specifications. The Ceph hardware recommendations advise this step explicitly. The output is a signed-off model that reflects measured hardware.
Set the reorder trigger and review cadence
Convert the model into a trigger: the utilization level and projected growth rate at which the team starts procurement. Include the vendor lead time and installation time, which the procurement team should confirm for the chosen hardware, because they are not fixed by Ceph. Schedule a review of pool growth against actual usage and repeat the model whenever a pool, profile or host count changes. The check is that the written trigger, the owner and the review date are recorded with the capacity model.
Operational risks that capacity plans must cover
A capacity plan is only as good as the controls that keep it true after handover. The cards below cover the operational risks that most often turn a sound usable-capacity estimate into a cluster that stops accepting writes. Each one names what to check and who should own it.
Monitor fill levels per OSD
Usable capacity depends on the fullest OSDs, not the cluster average. Uneven weights, mixed drive sizes or a recent failure can leave a few OSDs far fuller than the rest. Track utilization per OSD and per pool, and confirm that your monitoring reports both. Review the numbers against the headroom you planned for a failed host, not only against today's total.
Alert before nearfull
The nearfull, backfillfull and full ratios are the cluster's own warning and stop points, so alerts should fire well before them. Set your alert thresholds in your monitoring system and agree who responds at each level. Test that the alert reaches an on-call person, because a warning nobody sees gives the same result as no warning.
Full cluster effects on writes and recovery
When OSDs reach the full ratio, Ceph stops client writes to protect data, and recovery can stall on OSDs that have no room to receive data. A failed host then creates a double problem: fewer places to store data and less room to rebuild it. Document what your applications do when writes fail, and rehearse a recovery in a test cluster before you depend on the plan.
Control who changes ratios and pools
Raising a full ratio or adding a pool changes the capacity model without changing the hardware. Limit administrative access to the cluster and the dashboard, keep a record of changes to ratios, pools and replication settings, and review that record during capacity reviews. Treat any ratio change as a planning event, not a routine fix.
Include encryption and backup overhead
Encryption at rest, backup copies and snapshots all consume space that the raw-to-usable calculation may not show. Ask whether each pool will hold encrypted OSDs, whether backups land on the same cluster, and how snapshot retention grows over time. Put these in the same model as replication or erasure coding overhead so the forecast reflects what the cluster will really store.
Plan the order before the threshold
Hardware lead times, racking and installation take time, so the decision to buy must happen while the cluster still has room. Agree a trigger based on projected growth and the headroom for a failed host, then record which team owns the purchase. Confirm the lead time with your supplier and your data center operations team before you set the trigger.
Questions buyers ask before they plan Ceph capacity
How much does a Ceph capacity review cost?
DeployOpen does not publish a price list, and a quote follows from a written scope agreed before work starts. The main cost drivers are the number of pools and their replication or erasure coding profiles, how many failure domains the design must cover, whether the review includes a model for growth over several years, and whether candidate hardware needs to be benchmarked before purchase. A narrow review of one cluster with a few pools costs less than a review that tests several hardware options against a multi-site failure design. Ask for the deliverables, the assumptions, and whether the review recurs when you add pools or hardware. Most engagements are fixed-scope, so the quote should list exactly what is included.
How long does a Ceph capacity review take?
The duration depends mainly on how many clusters and pools are in scope, whether the cluster already exists or is still being specified, and how much data about growth and workloads is available at the start. A review of a planned cluster with a clear pool list usually moves faster than one that must reconstruct current usage from a running system. Benchmarking candidate hardware adds time because it depends on lab access and test runs. Your team should confirm the timeline in the written scope, including who must supply data and by when.
What must we provide before a capacity review starts?
Prepare four inputs so the model reflects your environment:
- A pool list with the intended use of each pool, such as RBD for virtual machine disks, CephFS, or RGW buckets
- The replication or erasure coding profile you expect for each pool, and any compression you plan to enable
- A growth forecast in terabytes per pool over the period your budget covers, with peak write patterns where you know them
- The hardware options under consideration, including drive sizes and types, host count, memory per host, and network design
If you already run Ceph, add current raw and used capacity per pool and recent growth history. Gaps in these inputs usually become assumptions in the model, so state them early.
Can our own team build the capacity model?
Yes, many teams can build a basic model with a spreadsheet once they understand the arithmetic: raw capacity, the overhead of replication or erasure coding, the number of failure domains you must survive, and the ratio thresholds you set. The harder parts are the failure-domain assumptions, the peak memory and CPU behavior during recovery, and checking the model against benchmarks. A review from DeployOpen is useful when you want an independent check, need the design documented for an audit or change board, or lack the time to test the model under realistic failure scenarios. Whichever route you take, confirm the ratio settings and recovery assumptions against the Ceph documentation for your release, and keep the model under change control.
When should we order more hardware for a Ceph cluster?
Order when the model shows that the cluster cannot absorb the loss of a host or planned growth while staying below the nearfull, backfillfull and full ratios with margin. Lead time for hardware is the key constraint, so trigger the order well before the projected crossing point, not when a warning appears. Watch for growth that outpaces the forecast, recovery that would fill the cluster after a failure, and pools that will need a new failure domain. Ask your vendor for lead times and confirm that the model still holds with the new hosts in place. The exact thresholds depend on your Ceph release and configuration, so confirm them in the release documentation before you set alarms.
How does the failure domain change usable capacity?
The failure domain sets how many copies or chunks must land on separate hosts, racks or rooms, and that limit caps how much raw capacity you can use. For example, a three-replica pool with a host failure domain stores three copies on three hosts, so usable capacity is roughly one third of raw capacity before headroom is reserved. An erasure coded pool with more chunks needs enough failure domains to place every chunk, and a small cluster may not have them. Illustrative only: a cluster with too few hosts for the chosen profile may refuse to place data correctly, which is why the failure-domain design belongs in the same review as the capacity model. The model should show what happens when one host, and then one rack, is lost.
Does DeployOpen issue a compliance attestation for storage capacity?
No. DeployOpen does not issue audit reports, attestations or certifications. A capacity review produces a documented model, assumptions, ratio settings and evidence of testing that your team can give to an auditor or assessor. Independent auditors and certification bodies decide whether your controls meet a requirement. Ask your assessor early which evidence they expect for capacity planning, such as change records, monitoring thresholds and recovery tests, so the review output fits their request.
What does a capacity review cover in the Ceph documentation?
The review draws on the Ceph hardware recommendations, which advise sizing memory for peak use rather than calm periods and benchmarking before buying because no two clusters are alike. Your release also matters, so check the active release list before you set thresholds. Tentacle (20.2.x) and Squid (19.2.x) are the actively maintained releases on the Ceph releases page, and that page notes they receive periodic backports and security fixes. Confirm which release you will deploy and whether its ratio defaults and behavior match the model. Confirm these details with your Ceph documentation for the release you choose.
How DeployOpen scopes Ceph capacity work
DeployOpen plans and deploys open-source platforms inside the client's own infrastructure, including the cloud accounts, data centers and air-gapped networks where a Ceph cluster will run. For capacity work, that means the usable-capacity model, the headroom policy and the full-ratio thresholds are designed against your real hardware, your failure domains and your growth expectations, and the resulting cluster stays under your control. DeployOpen does not hold the data or the administrative keys after handover unless a support agreement says otherwise.
Most engagements are fixed-scope, with the scope, deliverables and timelines agreed in writing before work starts. For a capacity review, the written scope should name the raw capacity per host and per device class, the replication or erasure coding profile for each pool, the number of failure domains in use, the host count you intend to tolerate losing, and the thresholds you want to set for nearfull, backfillfull and full. It should also list the outputs you will receive, such as a capacity model, configuration changes and an order-timing recommendation. Ask whether the scope covers verifying the settings on the live cluster or only producing the model.
A quote follows from that written scope, so the figures you receive can be compared line by line with your own team's estimate. Ask the quote to separate one-time design and validation work from any recurring support, and to state which items depend on information you have not yet provided, such as current utilization history or planned tenant growth. Ongoing support, upgrades and monitoring of capacity trends are available as a retainer, which suits clusters where the free space must be re-checked each quarter as data grows.
