Recovering from nearfull and full OSDs in Ceph

Recovering from nearfull and full OSDs in Ceph starts with knowing which ratio tripped

When writes stop or are about to stop on a Ceph cluster, the first task is to find out which threshold has been crossed, because each state calls for a different response. Nearfull is a warning: an OSD (object storage daemon) has passed the nearfull ratio, so the cluster is still accepting writes but is running out of room. Backfillfull means an OSD has crossed the backfillfull ratio, which blocks data from being moved onto it during recovery or rebalancing, even though client writes may still succeed. Full means the OSD has reached the full ratio and the cluster stops accepting client writes to affected placement groups. These ratios are set per cluster and per OSD, and the exact values in your cluster should be confirmed with ceph osd dump rather than assumed from defaults. Because Ceph spreads each pool's placement groups across many OSDs, a single full OSD can block writes to every pool that has data placed on it, which is why one drive can stall an entire application even when the cluster as a whole still reports free space.

This page is for the moment after that state has been identified. It covers how to read the per-OSD utilization with ceph osd df, how to decide whether a temporary ratio change is safe, how to rebalance data away from the hot OSD, how to delete data without causing a second incident, and how to add capacity so the cluster does not return to the same state. It does not replace triage of the health code that Ceph reports, so a team seeing OSD_NEARFULL, OSD_BACKFILLFULL or OSD_FULL should first confirm the code's meaning in the health checks guidance. It also does not cover why requests are slow, why placement groups are inconsistent, or how to swap a failed drive. Those are handled on their own pages, and a full OSD is sometimes a symptom of a failed or misweighted drive that needs replacing rather than more space. Check the status of the cluster's drives before choosing a fix, since adding data headroom to a failing device only delays the next failure.

Recovery options and when each fits

Raising the full ratios is the fastest way to let writes resume when an OSD has crossed the full threshold and client I/O has stopped. It buys time for recovery or deletion work to run, but it does not free any space. The cluster is simply allowed to fill further, which narrows the margin for the recovery traffic it still needs to complete.

Fits when: writes are blocked, the cluster has enough headroom to absorb a short increase, and the team has a concrete plan to free or add space within a known window.
Risk: an OSD that is already near its hard limit can become completely full, and recovery operations may then stall or fail on that device.
Confirm first: the current ratio settings and the exact OSD utilization, which the ceph osd df output shows, along with the release-specific commands for the Ceph version you run.
Record: the original values, the time of change, and the owner, so the ratios are restored once the cluster is healthy.
Avoid: using this as a permanent setting. Treat it as a bridge that must be closed with real capacity or data changes.

Raising ratios is a control-plane change, so it should be confirmed against the Ceph documentation for your release before anyone runs it under pressure.

Recovery sequence for a full cluster

This sequence brings a Ceph cluster that has stopped accepting writes back to normal operation without losing data. It orders the work so that growth stops first, space is freed or ratios are raised under a written plan, and the original ratios are restored only after backfill has finished.

Confirm the health state and the blocking flags
Run ceph health detail and ceph df to see which health codes are raised, such as OSD_FULL, OSD_NEARFULL or OSD_BACKFILLFULL, and whether the cluster has set the full or pause flags that stop client writes. Record the output in the incident log before changing anything, so the starting state can be compared later. The step is done when you can name each raised code and state whether client I/O is currently blocked or only warned. Confirm the current cluster-wide ratios with ceph osd dump, because the values in your environment may differ from defaults.
Identify the fullest OSDs and their pools
Run ceph osd df tree to rank OSDs by utilization and group them by host and CRUSH bucket, then use ceph osd pool ls detail and ceph df detail to map each pool to the OSDs that hold its data. A single overfull OSD often points to uneven weights or a skewed failure domain rather than a cluster that is simply out of space. The step is done when you have a short list of the fullest OSDs, the pools that drive their usage, and whether the imbalance is across the whole cluster or confined to one host or device class.
Stop the source of growth
Find and pause whatever is writing the most data: a bulk load, a backup job, a snapshot schedule, or a client application that has entered a retry loop. Coordinate with the owners of those workloads, because stopping them is usually a decision for the application team, not the storage operator alone. Do not add capacity or delete data until growth has stopped, or the cluster will keep moving toward full while you work. The step is done when ceph df shows usage flat across two or three samples taken at a short interval you choose for your environment.
Free space or raise ratios under a written plan
Choose one of two routes: delete or move data that is confirmed safe to remove, or temporarily raise the full ratio with ceph osd set-full-ratio so that recovery and client writes can proceed. Raising ratios only buys room to recover and must be recorded with the old value, the new value, the owner and the condition for reverting. Deletion of pool data should be approved by the data owner and checked against snapshot and retention rules first. The step is done when ceph health detail no longer lists full OSDs and the change record names the exact revert value.
Let backfill and recovery complete
Watch placement group states with ceph -s and ceph pg stat while data moves off the fullest OSDs, and keep recovery pressure at the level your team has agreed for client traffic. Watch for OSDs that cross the backfillfull threshold, since they can stall backfill even when clients can still write. If backfill stops making progress, check the OSD logs and the state of the drives before increasing any limits. The step is done when all placement groups report active and clean, with no degraded or misplaced objects remaining in ceph -s.
Restore the original ratios
Only after the cluster is healthy, set the full, backfillfull and nearfull ratios back to the values recorded in step one, using ceph osd set-full-ratio, ceph osd set-backfillfull-ratio and ceph osd set-nearfull-ratio. Restoring early can re-trigger the full state if utilization is still high, so check ceph osd df first. The step is done when ceph osd dump shows the original values and ceph health detail is clear of OSD capacity codes. Keep the change record open until the next step is complete.
Add capacity and escalate if the trend continues
Plan the capacity fix before the next incident: add hosts or OSDs, let the balancer distribute data in phases, and confirm the expansion timing with your team before utilization climbs again. Escalate to the vendor or to a specialist partner if the cluster cannot leave the full state after space is freed, if placement groups stay inactive, if the monitors report quorum problems, or if the cause of the imbalance is not clear. The step is done when a capacity plan with an owner and a review date exists and every escalation trigger is written into the runbook.

Risks of deleting data and raising ratios under pressure

When a cluster is nearfull or full, the fastest-looking fixes often create the next incident. The cards below cover the mistakes teams make under pressure, from deleting data that something still references to raising thresholds and never putting them back.

Deleting data that snapshots still reference

RBD and CephFS snapshots and clones share underlying objects with their parents, so deleting a source image or file may free far less space than expected, or none at all. Before removing anything, list the snapshots and clones that depend on it and confirm which of them the owners still need. Deleting the base image while clones exist can also leave the pool in a state that is harder to recover than the original full condition.

Raising ratios and forgetting to revert

Temporarily raising the full, backfillfull or nearfull ratios buys time to finish recovery, but a raised threshold also hides how close the cluster is to real exhaustion. Record the original values before changing them, set a named owner for the revert, and keep a dated ticket open until the ratios are back to their defaults or agreed values. Clusters that run with raised ratios for months tend to fail at the moment writes are needed most.

Backfill load slowing client I/O

Data moved to rebalance or re-replicate a full OSD competes with client reads and writes for disk, network and CPU. Recovery that looks harmless on an empty cluster can push application latency past what virtual machines or databases tolerate. Confirm the recovery and backfill limits before starting, watch client latency during the move, and agree in advance who can pause backfill if production slows.

Uneven CRUSH weights filling one OSD first

An OSD with a low CRUSH weight relative to its real capacity, or a host with fewer or smaller drives than its peers, fills before the rest of the cluster. Overall usage can look safe while one device is already at the backfillfull or full threshold, which blocks writes to the placement groups it serves. Compare per-OSD utilization against the weights, fix mismatched weights deliberately, and check the balancer state before adding any new capacity.

Deleting from RGW or RBD without checking consumers

An S3 bucket, object prefix or RBD image may be read by an application, backup job or lifecycle rule that nobody on the storage team owns. Deleting it to free space can break a workload that fails silently for days. Identify the bucket owners or image consumers, check lifecycle and versioning settings that may already be expiring data, and confirm retention obligations with the data owner before any deletion goes ahead.

Questions teams ask before fixing full OSDs

How long does recovery from full OSDs take?

There is no fixed duration, because the time depends on how much data must move and how fast the cluster can move it. The first phase is usually fast: clearing space, removing or moving data and lifting ratios back to safe values. The slower phase is rebalancing, where placement groups migrate to OSDs with free space. Duration is driven by the amount of data to move, the number of OSDs and hosts, drive speed, network bandwidth between hosts, and how much client I/O must continue during recovery. Recovery throttling settings trade speed for client latency. Ask your storage engineers to measure the current used capacity per OSD and the recovery rate observed during a previous rebalance, then build the estimate from those numbers. The Ceph hardware recommendations also advise benchmarking before buying, so a measured baseline on your own hardware is more reliable than any general figure.

Can our own team fix full OSDs without outside help?

Yes, many teams can, provided they have someone who understands CRUSH placement, OSD utilization, and the cephadm workflow for their release. The work is a sequence of checks: confirm which ratio tripped, identify the fullest OSDs with the cluster status and OSD utilization commands, decide what can be deleted or moved safely, change ratios only as a temporary measure, and watch recovery until the cluster returns to normal. The risk is in the decisions made under pressure, such as deleting the wrong data, raising ratios too far, or starting a large rebalance during business hours. Teams with no one who has done this before, or with data that cannot be lost, often choose to bring in a partner for the first recovery and keep the runbook for the next one.

What does an implementation partner add to a full OSD recovery?

A partner brings a second set of eyes on the decisions that carry the most risk, plus a written runbook your team can reuse. Typical work includes reviewing the CRUSH map and failure domains, checking whether placement groups are balanced, planning the order of ratio changes and data removal, confirming backups of anything being deleted, and documenting the steps with rollback points. DeployOpen plans, hardens, documents and supports open-source platforms inside your own infrastructure, so the recovery and the follow-up work stay in your environment. Most engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts. Ongoing support and monitoring reviews are available as retainers.

What do we need to provide before a recovery engagement starts?

The engagement needs read access to the cluster status, OSD utilization, pool and CRUSH configuration, and the cephadm service specifications, along with the Ceph release in use. It also needs the owners of the data in each pool, because deletion decisions depend on business rules, retention obligations and legal holds. Expect to provide a maintenance window or agreed change windows, contacts for storage, network and application teams, and access to the monitoring and alerting history for the period before the incident. Confirm with your change board whether ratio changes and data deletion need approval. These inputs should be listed in the written scope so both sides know what is ready on day one.

How do we avoid running into full OSDs again?

Prevention is mostly a capacity and monitoring discipline. Set alerts on OSD utilization well below the nearfull ratio so there is time to act, and track growth per pool rather than only total cluster usage. Keep enough free capacity to absorb the loss of a host or drive, because recovery from a failure consumes space on the remaining OSDs. Use the balancer and review placement group distribution regularly. Plan capacity additions before the nearfull threshold is reached, and test the expansion procedure in a non-production environment first. Monitoring and alert routing are covered in the Ceph monitoring guidance, and a written capacity review each quarter keeps the numbers current.

What drives the cost of a full OSD recovery engagement?

Cost follows the scope you agree in writing, so the main drivers are the size and number of clusters, the number of pools and failure domains to review, how much data must be verified or protected before deletion, and whether the work includes only recovery or also the capacity and runbook work that prevents a repeat. Costs that recur are retainers for monitoring, upgrades and support, along with any hardware you buy to add capacity. One-time costs include the assessment, the recovery work and the runbook. Ask for a quote that lists each deliverable, the environments covered, the assumptions about access and change windows, what is excluded, and the rate or fixed price for any follow-up work. DeployOpen does not publish a price list, so the written scope is the basis for any quote.

What should we confirm with the vendor before deleting data to free space?

Confirm with the data owner first, then with your Ceph release documentation and your backup team. Deleting objects, images or snapshots is permanent unless a verified backup or snapshot exists, so check retention and legal hold requirements before any deletion. Confirm that the Ceph release in use supports the commands and ratio settings you plan to change, and check the release notes for known issues in your version. The Ceph releases page lists which releases are actively maintained and their estimated end of life dates, which matters if the fix requires an upgrade. Record each decision and its approver in the change ticket so the sequence can be reviewed later.

Security and access controls during recovery

A full cluster is the moment when people are most tempted to make fast, broad changes. The cards below cover who should be allowed to change ratios, delete pools or images, protect backups, and use the dashboard, and how each action is recorded so the incident can be reviewed afterward.

Limit who can change ratios

Raising the full, backfillfull or nearfull ratios is a change that lets OSDs fill past their normal safety margin, so it should have a named approver. Restrict the Ceph admin keyring and cephadm shell access to the on-call storage engineers and record who holds them. Before anyone runs ceph osd set-full-ratio, confirm the change is written down with the current value, the new value and a revert time.

Gate pool and image deletion

Deleting a pool or RBD image is the one recovery action that cannot be undone, and it is often proposed as the quickest way to free space. Require a second person to confirm the pool name, its owner and its last successful backup before deletion. Pool deletion also depends on the monitor settings that allow it, so confirm those are not enabled as a routine default in your environment.

Protect backups before deleting data

Snapshots and backups on the same cluster share the capacity that is already full, so they may be the first data to be removed to recover space. Before deleting anything, copy the restore points you need to a separate target or confirm they are already off-cluster. Check the restore process itself, not only the backup job status, because a backup that cannot be restored does not protect the data.

Scope dashboard access during incidents

The Ceph Dashboard can display and change cluster state, so it needs the same controls as the command line. Give incident responders a read-only role by default and grant write access only for the people approved to change ratios or remove data. If your organization uses single sign-on for the dashboard, confirm that group membership is reviewed before the incident and removed after it.

Keep an audit trail of admin actions

During a full-cluster event, many commands are run under pressure, and the record of what changed is needed for the post-incident review and for any compliance evidence. Capture the cluster log and the commands issued from shared shells, and make sure timestamps are consistent across hosts. Ask whether your ceph log and daemon logs are shipped to a store that the storage team cannot edit.

Use a change record for each step

Each recovery step, such as setting a ratio, marking an OSD out, or removing a pool, should have one entry with the time, the operator, the reason and the reversal command. Keep the list short and ordered so the next shift can see what is still in effect. Remove temporary settings in the same record that created them, so a raised ratio does not outlive the incident.

Adding capacity before the next nearfull warning

The most reliable way to stop a nearfull warning from becoming a write outage is to add raw capacity while the cluster still has headroom. Expansion is a prevention step, not a recovery step, and it works best when it starts weeks before the ratios are reached. Adding hosts or OSDs with cephadm is the usual path: new hosts are enrolled into the cluster, and OSD services in the service specification deploy the new devices. Confirm before you start that the new drives match the media and sizing of the existing OSDs in their device class, because mixed hardware produces uneven utilization that looks like a capacity problem later.

Once new OSDs join, CRUSH begins moving placement groups onto them, and that data movement competes with client I/O. The Ceph balancer can be used to spread data in phases so the cluster does not swing from one imbalance to another all at once. Decide in advance how many phases you will allow, which pools may be moved first, and how recovery pressure will be limited during business hours. Watch ceph osd df and the per-OSD utilization over several days, not only at the end of the change, because the cluster is expected to settle to a stable spread before you consider the expansion finished.

Timing matters more than speed. Work out the growth rate of each pool from its recent history, then schedule expansion so the new capacity is online well before the nearfull ratio would be crossed at that rate. Ask your team to confirm the alerting thresholds that will warn them early, such as utilization percentages per OSD and projected days to full if your monitoring stack supports forecasting. For the step-by-step expansion procedure, including adding hosts and letting the balancer work, follow the related page on adding hosts and OSDs without hurting clients, since this section only covers how expansion fits into prevention.

Limits to confirm in your environment

Ratio changes and data deletion only buy time when the release, the pool layout and the hardware behind them can absorb the work. Each card below names one limit to confirm before you act on the steps in this guide, so the fix does not create the next incident.

Release and maintenance status

Confirm which Ceph release your cluster runs and whether it still receives backports and security fixes. Ratio defaults, the commands that set them and the way backfill is throttled can differ between releases, so the steps here should be checked against your version's documentation. Run the version check on every daemon type before you plan a change, and note any release that is near its end of life so an upgrade can be scheduled alongside the recovery.

Exact ratio behavior in your release

Confirm the nearfull, backfillfull and full ratios your cluster actually enforces before you touch them. Read the current values from the cluster rather than from memory or an old runbook, and check whether they are set on the OSD map or overridden per device class. Raising a ratio moves the point where writes stop, but it does not add space, so record the original value so you can restore it once capacity is added.

Pool replication or erasure coding layout

The raw space a pool consumes depends on its replica count or erasure-code profile, and a failed or rebalanced OSD can push several pools toward full at once. Confirm which pools share the same OSDs and which failure domain each one uses, because a host or rack with less free space can trigger a full condition even when the cluster total looks healthy. Ask your team to document the layout in writing before recovery begins.

Backup and snapshot retention

Snapshots and backup copies hold blocks that deleted data still references, so removing files from a client does not always free space. Confirm how many snapshots exist per image or pool, which retention policy your backup tool applies, and who may approve deletion. Deleting the wrong snapshot can remove the only recovery point, so verify that a restorable copy exists elsewhere before you free any space this way.

RGW and RBD client behavior

Object gateway and block clients react differently when a pool stops accepting writes. RGW requests may queue, retry or fail with server errors depending on the client, while RBD volumes can freeze or report I/O errors inside virtual machines. Confirm with each application owner what timeouts and retry logic they use, so you know whether a pause in writes will corrupt state or just delay jobs.

Hardware headroom for backfill

Recovery and backfill consume network bandwidth, drive IOPS and memory on the hosts that receive the data. Confirm the spare capacity on the remaining OSDs, the speed of your links and whether the planned new drives or hosts will arrive in time. Benchmark a representative host before you assume rebalancing will finish within your maintenance window, since no two clusters behave alike.

When to bring in an outside team for a full cluster

Bring in an outside team when the incident has outgrown the runbook, which usually shows up as more than one OSD at full at the same time, PGs stuck in backfill_toofull or recovery_toofull, or a cluster where clients are blocked and no one can say how much headroom remains. A single nearfull OSD on an otherwise balanced map is an internal task. Several full OSDs across different hosts, especially after a failure domain has already been lost, is a different class of problem, because each action (raising a ratio, removing data, marking an OSD out) changes where the remaining data can land. Treat the situation as escalated if the monitors are unhealthy, if recovery is competing with client I/O in a way you cannot throttle, or if the team has already made two or three changes without restoring writes.

The second group of signals concerns data at risk. Before any deletion or pool change, confirm that you have a recent, tested backup of the affected pools, and that you know whether RBD snapshots, RGW object versioning or lifecycle rules would free space or hold it. If nobody can state which pools hold the most data, which snapshots are retained, or whether any deletion would be irreversible, stop and get a second set of eyes. A partner can review the ceph df and ceph osd df output, the CRUSH map and failure domains, the ratio settings in effect, pool replication or erasure coding parameters, and the backup and snapshot inventory, then document options and their risks. Ask any partner to state in writing what they will and will not change in your production cluster, and who makes the final decision on deletions. No review can guarantee a particular recovery time or outcome.

The third signal is repetition. If the cluster has reached nearfull more than once in a short period, the root cause is capacity planning, not the incident. Confirm with your own team whether growth was tracked against the usable capacity after replication or erasure coding overhead, and whether the expansion procedure has been rehearsed. An outside review of the capacity model, alert thresholds and expansion plan is most useful before the next incident, while the cluster is still writable. Ask the reviewer to show how they would size headroom for your failure domains and how long a host loss would take to recover at your current recovery settings, so that the decision to expand is made on evidence rather than on the last outage.

How DeployOpen scopes and supports Ceph capacity recovery

DeployOpen treats a full or nearfull Ceph cluster as a capacity and change problem first, so recovery starts with a written view of which OSDs are filling, why, and what must change before writes resume. We plan the ratio changes, data movement and capacity additions as a fixed-scope piece of work, and we document how the cluster should be kept out of this state next time.

How we scope

Review OSD utilization, pool layout and CRUSH failure domains to find which devices are driving the nearfull or full state
Agree in writing which temporary ratio changes, data deletions and expansion steps fall inside the engagement
Confirm the change window, rollback steps and who approves each write-affecting action before work starts
Ask for a quote built from this written scope, since DeployOpen does not publish a price list

How we build

Work through recovery in a planned order: stabilize client writes, rebalance, then add capacity, with checkpoints at each stage
Verify each ratio change against the cluster's actual health output and recovery progress before moving on
Test any data deletion plan on non-production data or a snapshot-backed copy where the environment allows it
Build an expansion plan with the balancer and recovery settings agreed in advance so client I/O stays within limits

How we hand over

Deliver a runbook covering the ratio values, their defaults, the steps taken and how to reverse each change
Document the utilization thresholds, alert routes and owners the client team should monitor after recovery
Record the capacity forecast and the decision points for adding hosts or OSDs before the next nearfull warning
Walk the client operations team through the runbook and confirm they can repeat the procedure without us

How we support

Offer ongoing support, upgrades and continuous testing as a retainer if the client wants help beyond the fixed scope
Review capacity trends and expansion timing with the client team on a schedule agreed in the retainer terms
Confirm which Ceph release line the cluster runs and plan any upgrade separately so recovery work is not mixed with version changes
Ask your DeployOpen contact to confirm support hours, response expectations and escalation paths in the written agreement