Fixing BLUESTORE_SPILLOVER warnings before OSD metadata slows the cluster
A BLUESTORE_SPILLOVER warning means one or more OSDs are storing part of their BlueStore metadata on the slow device, usually the HDD, because the faster DB device that was meant to hold it has run out of room. The first question is whether it is urgent. In most clusters it is a performance and planning problem rather than an immediate outage: the OSD keeps working, but lookups that would have hit the SSD now touch the spinning disk, and latency rises under load. Confirm the scope before you act. Check which OSDs are listed, whether they share one DB device, and whether client latency or slow requests have changed since the warning appeared. A warning on a single OSD in a healthy cluster can wait for planned work. A warning spreading across many OSDs that also show slow ops needs a faster response.
Spillover happens when the BlueStore DB partition, which holds RocksDB metadata such as object maps and allocation information, fills up. The overflow is written to the main data device, so every affected OSD pays a penalty on metadata access. This is the expected failure mode for HDD-based OSDs that share SSDs for WAL and DB, as the Ceph hardware recommendations advise. One shared SSD may serve several OSDs, so one undersized DB device can spill several drives at once. Before you choose a fix, record the DB and WAL sizes in use, the device layout, and the Ceph release in the cluster, and ask your vendor or the release documentation whether compaction or DB resizing behaves as you expect on that version.
The fix is planned work, not a single command. Options include manual compaction to reclaim space, expanding or migrating the DB device to a larger partition, and sizing DB devices correctly for any new OSDs you add. Each option touches live OSDs, so plan the order, the recovery impact and a rollback point before you start, and test the procedure on one OSD first. Confirm with your own environment which steps your cephadm deployment supports, how long each OSD takes to return to service, and what you will monitor during the change. Your team can do this work in-house, or a partner can scope it as a fixed-scope engagement with written deliverables for the check, the change and the verification.
How to confirm spillover on a specific OSD
Start with the cluster-wide view. Run ceph health detail to list each OSD that has spilled metadata. The line for each OSD names the daemon and the amount of BlueFS data on the slow device. Read the OSD IDs first, then check whether they share a DB device, because one failing or filling SSD affects every OSD it serves.
- Affected OSD IDs: Note every ID listed. A pattern across several OSDs usually points to a shared DB device or to a sizing decision made when the OSDs were deployed, not to one drive.
- Reported spilled amount: Record the figure so you can compare it with later checks and see whether it is growing.
- Health code and severity: Confirm the code is BLUESTORE_SPILLOVER and note the severity, since your runbook should set different response times for each.
- Muted alerts: Check whether the warning was muted with
ceph health mute. A muted warning still exists, but it stops appearing in routine checks.
Save the full output with a timestamp before you change anything, so you can show the state at the start of the incident.
The health output tells you that spillover exists, while the BlueFS counters show where the bytes sit. Run ceph tell osd.<id> perf dump on the affected OSD and read the bluefs section, which reports the DB and slow device capacity and usage side by side. Read both devices from the same daemon so the figures are comparable.
- db_total_bytes and db_used_bytes: The capacity of the DB partition and how much of it BlueFS is using.
- slow_total_bytes and slow_used_bytes: The capacity of the main data device and how much BlueFS metadata it holds. A non-zero
slow_used_bytesis the spillover signal. - Change over time: Record the counters at each check. A steady rise matters more than any single reading.
- Access from the host: If
ceph tellcannot reach the daemon, open a shell withcephadm shellon the host that runs the OSD and query the daemon from there.
Record the used and total values for both devices, and confirm the field names against the release your cluster runs, since counter names can change between releases.
A DB device can be nearly full while no metadata has reached the slow device. That state needs action before the warning appears, and it calls for a different response from active spillover. Whether BlueFS has started writing to the slow device is the deciding question, so confirm it before you pick a fix.
- Full but not spilled:
db_used_bytesapproachesdb_total_byteswhileslow_used_bytesstays at zero. No BLUESTORE_SPILLOVER warning is expected yet, but the next growth will push metadata onto the HDD. - Spilled:
slow_used_bytesis above zero and the health output lists the OSD. Metadata is already on the slow device, and latency on that OSD will show it. - Conflicting evidence: If the health output and the counters disagree, confirm you queried the right OSD and that the counters are fresh. A restarted daemon resets some values.
- Full data device: A full main data device produces different warnings. Check that the DB device is the source before you change anything.
Write down which state you have confirmed, because the fix path in the later sections depends on it.
Once one OSD is confirmed, check whether the pattern repeats. Use ceph osd df to compare usage across OSDs, ceph device ls to map physical devices to OSDs, and ceph osd metadata <id> to find the metadata fields that record each OSD's DB device. Grouping OSDs by DB device shows whether one SSD is behind several spilled OSDs.
- Shared DB devices: Group the spilled OSDs by their DB device. Several OSDs on one SSD point to a device-level fix rather than per-OSD work.
- Host grouping: If spillover clusters on one host, compare that host's OSD and DB layout with the others.
- Deployment specification: Review the cephadm OSD service specification that created the OSDs to see how DB devices were assigned.
- Hardware guidance: Compare the DB device choice with the Ceph hardware recommendations, which advise moving WAL and DB to shared SSDs for HDD-based OSDs and using enterprise-class SSDs for metadata.
Build a list of affected OSDs and their DB devices before you schedule compaction or device work.
Compaction, DB expansion or DB migration: which fix fits
BLUESTORE_SPILLOVER has four common fixes, and they differ in how much they change and how much recovery load they add. Start with the least invasive option that matches your device layout, then move down the list only when the earlier fixes do not clear the warning.
Online compaction
Online compaction asks a running OSD to compact its RocksDB database without stopping it. It reclaims space on the DB device and can clear spillover when the DB is only slightly over its share, but it does not add capacity. Check whether the OSD's DB usage drops after compaction, and watch client latency while it runs, since compaction competes with client I/O on the same host.
Offline compaction
Offline compaction stops the OSD, runs the compaction tool against its database, and restarts it. It can reclaim space that online compaction leaves behind, at the cost of a short outage for that OSD. Plan it as a maintenance action in a window where losing one OSD is acceptable, and confirm the exact procedure for your Ceph release before running it on production data.
Expanding the DB logical volume
If the DB sits on a logical volume with free space in the same volume group, you can grow that volume and then tell BlueFS to use the new size. This keeps the OSD ID and data in place and avoids moving data, but it only works when the layout has room. Confirm the expansion command and the BlueStore resize behavior for your release, and check that the device is not already at its limit.
Migrating DB to a new device
The BlueFS migrate approach moves the DB contents from the current device to a larger one without rebuilding the OSD. It suits a fuller DB SSD that needs a bigger home, and it avoids the full recovery that a rebuild triggers. Stop the OSD for the move, confirm the target device is enterprise-class and at least as fast, and verify the OSD starts cleanly and spillover clears before you restart the next one.
Redeploying the OSD
Redeploying removes the OSD, recreates it with a DB device sized for the workload, and lets Ceph backfill its data. It is the most disruptive option because every object on that OSD must be rebuilt across the cluster, so it adds recovery load for a long period. Use it when the layout cannot be fixed in place, and schedule it with recovery settings reviewed and capacity headroom confirmed.
Fix options and when each fits
This table compares four ways to clear BLUESTORE_SPILLOVER: compaction, in-place DB expansion, migrating the DB to a new device and redeploying the OSD. Read each row against your hardware and your tolerance for downtime, and start with the least disruptive option that the warning's cause allows.
| Option | When it fits | What it touches | Downtime and data movement | What to verify afterward |
|---|---|---|---|---|
| Compaction | Spillover is small and the DB has free space on its device to reclaim | RocksDB on the OSD's DB device, run against one OSD at a time | OSD must be stopped or compacted online per your Ceph release; data movement is minimal | Confirm the spillover warning clears and check that BlueFS usage on the DB device drops |
| In-place DB expansion | The DB partition or logical volume can grow because the underlying device has unallocated space | The DB logical volume and BlueFS size on the OSD | OSD restart needed; little or no data movement | Confirm the new DB size is visible to the OSD and that spillover stops growing |
| DB migration to a new device | The DB device is undersized but the host has a larger SSD or NVMe to move metadata onto | DB volume, BlueFS data and the OSD's device links | OSD stopped during migration; metadata copied, not object data | Check OSD starts cleanly, health returns to OK and spillover no longer reports |
| Full OSD redeploy | The DB layout is wrong at the design level, or in-place steps are not supported for the release in use | The whole OSD: data, DB and WAL are recreated and refilled from replicas or erasure-coded peers | Heavy data movement and recovery load while the OSD rebuilds; plan for backfill time | Watch recovery finish, confirm PG states return to active and clean, and update the service spec for new OSDs |
Sequence for a planned spillover fix
This sequence moves an affected cluster from a BLUESTORE_SPILLOVER warning to a verified, documented state with one OSD or host at a time changed. Each step ends with a check, so you know whether to continue, escalate or roll back before touching the next device.
Confirm which OSDs spill and by how much
Run ceph health detail and record every OSD named in the BLUESTORE_SPILLOVER entry, along with its host and the amount of metadata reported on the slow device. On each affected OSD, check BlueFS usage with the bluefs stats tool available on your release, and confirm the command name in your version's documentation first. Save the list of OSD IDs, hosts and spilled sizes in your change record. The check: every OSD in the health output appears in your list, and the sizes match what the BlueFS output reports.
Check cluster health and recovery state
Run ceph -s and ceph health detail to confirm that the spillover warning is the only active issue. Run ceph osd df to check how much free space each OSD has, since a cluster near its nearfull ratio leaves little margin for rebalancing during the fix. If any OSD is down, or any placement groups are degraded, misplaced or backfilling, pause and resolve that first. The check: the cluster reports only the spillover warning, all OSDs are up and in, and placement groups are active and clean.
Plan one OSD or host at a time
Choose the fix path for each OSD: compaction first, then DB expansion or migration if compaction does not clear the spill. Before any OSD is stopped, run ceph osd ok-to-stop against its ID and set the noout flag so the cluster does not start rebalancing during the work. Work through one OSD or one host at a time, and confirm your failure domain can tolerate that device being out. Write the rollback step for each device in the change record. The check: ok-to-stop succeeds for the selected OSD, and the plan lists the device, the owner and the rollback action.
Run compaction on the affected OSD
Compaction rewrites the RocksDB data on the DB device and can release space held by obsolete entries, so it is the least invasive option to try first. Run ceph tell osd.<id> compact against one affected OSD and wait until the command returns and the OSD is back in service. Then repeat the BlueFS check from the first step. Compaction does not add capacity to a DB device that is simply too small, so a spill that persists after compaction moves you to the next step. The check: the BlueFS usage on the slow device drops, or the spillover entry for that OSD clears.
Expand or migrate the DB device
If compaction does not resolve the spill, either extend the existing DB logical volume and grow BlueFS into the new space, or attach a larger DB device and migrate the metadata onto it. Both paths use ceph-bluestore-tool against an OSD that is stopped, so run them in the window you planned, with the OSD down and the noout flag set. Keep the original DB device intact until the OSD starts cleanly and the cluster has returned to active and clean. Confirm the exact subcommands and flags for your release before running them. The check: the device size or BlueFS layout reflects the change, and the OSD starts and joins the cluster without errors.
Confirm the warning clears and BlueFS returns
Bring the OSD back into service, unset noout, and wait for placement groups to return to active and clean before touching the next OSD or host. Run ceph health detail and confirm that BLUESTORE_SPILLOVER no longer appears for the device you changed. Recheck the BlueFS usage on the slow device, which should show that metadata is back on the DB device or that the slow usage is reduced to what you expected. The check: the spillover warning is gone for the device, and the slow device counters show no spilled metadata or a clearly lower amount.
Document the change and update DB sizing
Record each OSD ID, the fix path used, the commands and their output, the device sizes before and after, and the time each change was made. Store this record where your operations team keeps change history, and link it to the original health alert. Then update the OSD service specification or your deployment runbook so that new OSDs use a DB device sized for your expected metadata growth, and confirm that sizing with your hardware vendor and the Ceph hardware recommendations for your release. The check: every changed OSD appears in the record, and the sizing rule for new OSDs is written down and reviewed.
Sizing DB devices for new OSDs so spillover does not return
The fix for recurring spillover is to size the DB device correctly before the next OSD is created, because a DB partition cannot be grown in place without rebuilding the OSD. DB size depends on how many OSDs share the device, the capacity of each data drive, and the workload the cluster will carry, including object counts, small-file patterns and metadata-heavy operations. No single ratio of DB space to data space should be treated as a universal rule. The Ceph sizing guidance and a benchmark on your own hardware are the inputs to use, and the numbers you record should be the basis for the purchase order.
The Ceph hardware recommendations set the baseline design: a dedicated drive per OSD, and moving WAL and DB off HDDs onto shared SSDs when OSDs are HDD-based. The same guidance calls for enterprise-class SSDs for metadata, because a consumer device that stalls under sustained writes will slow every OSD that depends on it. Confirm with your drive vendor that the endurance and write performance of the candidate devices suit a shared DB role. Also plan the failure domain: if one shared SSD serves many OSDs, losing it takes all of them down together, so the number of OSDs per DB device should be a recorded design decision.
Build headroom into the capacity plan rather than waiting for the warning. Spillover typically appears after a cluster has been filling for some time, so the sizing review should check how the DB devices will behave at the nearfull threshold of the planned capacity, not only on day one. Ask your cluster design team to document the DB size per OSD, the number of OSDs per shared device, the expansion path for adding hosts, and the monitoring that will alert before DB usage reaches its limit. This sizing decision belongs to the broader cluster design work covering hardware, media, failure domains and capacity, so it should be reviewed there and not settled as a one-off fix during an incident.
Risks during a DB device change
A DB device change touches OSD metadata, recovery traffic and cluster state at the same time. The risks below compound each other, so confirm health, versions and recovery before the first change and keep the rollback path in view.
Losing a shared SSD
One DB SSD often serves several HDD-backed OSDs. If that device fails during a migration or expansion, every OSD that depends on it goes down together, and the cluster must rebuild each affected OSD's data across the remaining hosts. Before you start, map which OSDs share each DB device with ceph osd metadata and confirm the failure domain can absorb losing all of them. Treat the SSD as the highest-risk component in the change.
Recovery load slowing clients
Moving or recreating a DB device forces OSDs to rebuild or re-read metadata, and that work competes with client I/O. Applications can see higher latency or slow requests while recovery runs. Check current recovery settings and client latency baselines before you begin, and agree on a window with application owners. Changing recovery priority is a tuning decision, so confirm the effect on a non-production copy first.
Repeated repair loops
Repeating a repair or redeployment on the same OSD without finding the cause can turn one warning into a cycle of rebuilds. Each pass adds recovery load and can hide a failing drive. If an OSD returns to the same spillover or error after a change, stop and check the underlying device's health and logs before running the procedure again. Record each attempt so the team can see the pattern.
Mismatched versions across tooling
Commands, cephadm and the cluster itself must be on compatible releases, or output and behavior may differ from the documentation you followed. A host running a different package set than the cluster can produce misleading status or fail mid-procedure. Confirm the running release on every daemon, the cephadm version on each admin host, and the procedure's release notes before you run anything that changes OSD metadata.
Skipping backups and health checks
A DB device change rewrites OSD metadata, and a mistake can make an OSD unusable. Without a recent configuration export, a verified backup of data you cannot rebuild, and a clean health baseline, you cannot tell whether a new warning came from the change or was already present. Capture ceph health detail, the OSD tree and device layout before starting, and confirm that your backup restore has been tested for the data it protects.
Questions teams ask before fixing spillover
Is BLUESTORE_SPILLOVER an emergency?
Usually not on its own. The warning means some BlueStore metadata has moved from the faster DB device onto the slower main device, so reads and writes that touch that metadata can slow down. Data is not lost and the OSD keeps running. The urgency depends on how much spillover exists, how fast it is growing, and whether the cluster is already under load or near full. Check whether slow ops or nearfull warnings appear alongside it, because those combinations need attention sooner. Confirm the current state with ceph health detail and the per-OSD BlueFS counters before deciding how quickly to act.
How long does a spillover fix take?
The duration depends on the fix you choose. Compaction on a single OSD is usually the fastest step to try, while expanding or migrating a DB device means draining and rebuilding OSDs, which takes longer because recovery traffic moves across the cluster. The main drivers are the number of affected OSDs, the amount of data that must be rebalanced, the recovery settings your team allows during business hours, and how many change windows you can use. Ask for a written plan that lists the steps per OSD and the expected recovery load, since no fixed duration is safe to assume without measuring your own cluster.
Can our own team fix spillover without outside help?
Often yes, if your team already runs cephadm clusters, understands OSD and BlueStore layout, and has a tested rollback. The steps are documented and a single OSD can be checked and compacted without stopping the cluster. The risk rises when many OSDs are affected, when the DB device is shared across several OSDs, or when recovery competes with client traffic. Outside help is most useful for planning the sequence, validating the procedure on one OSD, and reviewing the capacity math for new DB devices. Be honest about the time your staff can spare during the work and whether they have seen a recovery of this kind before.
What does DeployOpen need from us to start?
A fixed-scope engagement starts with access and information. Typically you should be ready to provide:
- Read access to the cluster through cephadm and the Ceph CLI, with the health output and OSD inventory
- Host details for the OSD nodes, including drive models, DB device layout and network topology
- Benchmark or performance data from the cluster, and recent recovery and client latency history
- Written change windows, approval contacts and a rollback owner
- Confirmation of how backups and snapshots are handled before any device work starts
The more complete the inventory, the more accurate the written scope and the fewer surprises during execution.
What does a spillover quote depend on?
A quote follows from the written scope, so it depends on what is in that scope. Main cost drivers are the number of hosts and OSDs affected, whether the work is limited to compaction and checks or includes DB device expansion or migration, the amount of testing and validation required in your environment, and the number of change windows. Recurring items may include follow-up monitoring review or a retainer for ongoing support, and these are separate from the one-time work. DeployOpen does not publish a price list, so ask for a quote that lists each deliverable, the assumptions behind it, and what would change the scope.
Can a spillover fix be reversed if something goes wrong?
Partly. Compaction is generally low risk, and an OSD that fails a check can be stopped and handled as a normal OSD problem. Expanding or migrating a DB device changes the on-disk layout, so the safe path is to drain or remove one OSD at a time, rebuild it with the new layout, and wait for recovery to finish before touching the next one. Reverting means redeploying the OSD with the old layout, which again moves data. Ask for the rollback procedure in writing, confirm that you have a verified backup or recovery path for the affected pools, and agree a stop condition before work begins.
How do we size DB devices for new OSDs so spillover does not return?
Start from the expected object count and workload, not from a fixed ratio alone. The Ceph hardware recommendations advise moving WAL and DB to shared SSDs for HDD-based OSDs and using enterprise-class SSDs for metadata, so the DB device should be an enterprise SSD with headroom for growth. Benchmark a representative workload before buying, because no two clusters behave alike. Check the current DB usage on existing OSDs, plan for recovery and expansion, and keep spare capacity on the DB device. Confirm the sizing rules for your Ceph release with the official documentation and your hardware vendor before ordering.
How DeployOpen scopes and delivers spillover fixes
DeployOpen handles a BLUESTORE_SPILLOVER fix as a written, fixed-scope engagement. Before any change, the scope names the affected OSDs, the hosts and drive models involved, the current ceph daemon osd.<id> perf dump and ceph health detail output, the chosen remedy (compaction, DB device expansion, or DB migration to a new device), the rollback path, and the acceptance checks. Deliverables and a timeline are agreed in the same document, so you know what you are paying for and what you must provide, such as a maintenance window, spare capacity in the cluster, and access to the hosts through cephadm.
During delivery, the work happens inside your cloud accounts, data centers or air-gapped network. Engineers confirm BlueFS spillover per OSD, check the DB device fill level and the balance of recovery against client I/O, and run the chosen procedure one OSD at a time so that PG states stay healthy between steps. Verification is evidence-based: the BlueFS counters are rechecked after each OSD, the health checks return to a clean state, and the cluster is observed through at least one full recovery cycle. Where your cluster runs the Ceph monitoring stack, the alert rules that fired are confirmed to clear.
The handover includes a change record: what was done on each host, the command output before and after, the sizing assumptions used for the DB device, and a runbook for the next expansion of new OSDs. This is the document your change board and auditors will ask for. DeployOpen can also operate the cluster under a retainer, covering health review, capacity forecasting against DB sizing, and upgrades to supported Ceph releases, if your team wants that support after the fix. Ask any partner to confirm which Ceph release your cluster runs and which support window applies to it before you sign.
