Replacing a failed OSD drive in a cephadm cluster

How to replace a failed OSD drive in a cephadm cluster without rebuilding the map

Start with the question that decides the procedure: has the drive failed outright, leaving the OSD down and its data unreadable, or is it still serving I/O while throwing errors? When a drive has died, replacing the hardware and reusing the same OSD ID is usually the cleanest path. Keeping the ID means the CRUSH map keeps the OSD in its original place, so the cluster does not have to re-map placement around a new identifier. The mechanism is ceph orch osd rm <osd-id> --replace, which holds the OSD's ID and CRUSH position for a replacement device. Before you touch the hardware, confirm with ceph orch osd rm status and ceph osd tree that the OSD is in the destroyed state.

Use this path when the failed drive is the only fault and the host, its controller and its other OSDs are healthy. If several OSDs on one host fail together, or the failed device is a shared WAL or DB SSD, stop and diagnose the host first, because one drive swap will not restore those OSDs. Once the drive is physically replaced, check that the host sees the new device and that it does not carry old partitions. Clear it with ceph orch device zap <host> <device> --force if needed, then rely on your existing OSD service specification to deploy a new OSD on it. Confirm that the spec's device filters match the new drive, because a mismatch may leave the disk unused while the cluster stays degraded. Record the drive model and slot so the next failure follows the same checks.

Replacing the drive does not fix the cause of the failure. If the drive came from a bad batch, sits behind a faulty controller, or runs hot, a replacement may fail the same way, so review SMART data and host logs before closing the ticket. The procedure also does not restore data by itself. Once the new OSD joins, backfill runs, and the cluster returns to HEALTH_OK only after its placement groups finish moving. Track that progress with ceph -s and set recovery load so client I/O is not starved. Placement groups already flagged as inconsistent need their own repair workflow, and a failed shared DB device needs a separate plan. The cephadm documentation covers the exact commands for your release, so check them against your version before running anything in production.

Choosing between replace, remove and redeploy paths

This is the standard case. The drive has failed completely, but the host, its other OSDs and the network are working. Cephadm can mark the OSD as destroyed and keep its ID, so the replacement drive comes back under the same identity.

Confirm the device is gone: check ceph health detail and ceph orch device ls on the host, and confirm in the kernel logs that the drive has dropped off the bus.
Confirm the failure is limited to one OSD: ceph osd tree should show only the failed OSD as down, and the other OSDs on the host should be up.
Check that removal is safe: ceph osd safe-to-destroy <id> reports whether the data is fully replicated elsewhere.
Start removal with replacement: run ceph orch osd rm <id> --replace, add --zap only if you want cephadm to wipe the device, and track progress with ceph orch osd rm status.
Let the specification redeploy: once the drive is physically replaced, the OSD service specification picks up the new device and reuses the preserved ID.

Before you close the ticket, check that the new OSD is up and in and that recovery is progressing.

A drive that returns read errors or SMART warnings but still serves data should be drained while it is still readable. Waiting for it to fail outright leaves you with a sudden loss of copies and a rebuild under pressure. Your first task is to confirm that stopping the OSD will not make placement groups unavailable.

Collect the evidence: review the drive's SMART data and the kernel logs, and use ceph health detail to see which OSD and placement groups are affected.
Check placement group state before acting: if PGs are already degraded, resolve that first, because extra backfill on top of an existing deficit increases risk.
Test stop safety: ceph osd ok-to-stop <id> reports whether stopping this OSD would leave any PGs without enough copies.
Replace like for like: ceph orch osd rm <id> --replace keeps the OSD ID, so CRUSH placement stays stable for the new drive. Use plain removal without --replace only when you intend to shrink capacity.
Plan the recovery load: a readable failing drive still holds data that must be backfilled elsewhere, so schedule the work with your recovery settings in mind.

Confirm that the OSD's data is fully backfilled and that the cluster reports no degraded PGs before you wipe or retire the failing drive.

When a whole host or a JBOD enclosure is lost, many OSDs fail together. This is a failure-domain event, so the priority is to confirm that the surviving copies still meet your replication or erasure-coding rules. Removing OSDs in this state can start data movement you do not need, especially if the host is only unreachable.

Map the affected failure domain: ceph osd tree shows which hosts or buckets are down, and you should compare that list with the CRUSH rules that protect your pools.
Check quorum and remaining copies: ceph -s shows whether monitors have quorum and whether PGs are degraded, undersized or inactive.
Hold removals while the host may return: if the host could come back, leave its OSDs in place so the cluster does not rebalance data twice.
Plan restoration or replacement: after the host is rebuilt and rejoined with cephadm, the existing OSD specification can bring back devices that are detected again.
Escalate inactive PGs: when PGs are inactive, the cluster cannot serve those objects until enough copies return, so involve your Ceph support path early.

Record which failure domain was lost and confirm that every affected PG has returned to active and clean before you retire any drive.

On HDD-based OSDs, the BlueStore DB or WAL often sits on a shared SSD. The failed device may be the HDD, the SSD, or both. Losing the shared SSD takes down every OSD that uses it, so the first step is to map which partitions belong to which OSD before any device is touched.

Identify the shared device: ceph osd metadata <id> and ceph-volume lvm list on the host show which DB or WAL device backs each OSD.
Determine what failed: if only the HDD failed, its DB partition on the SSD may remain usable, and if the SSD failed, every OSD sharing it is affected.
Replace the HDD carefully: ceph orch osd rm <id> --replace --zap wipes the HDD, so confirm that the db_devices section of your OSD specification still selects the shared SSD.
Watch for undersized DB partitions: a replacement that receives a smaller DB partition can later produce BLUESTORE_SPILLOVER warnings, so check the partition size against your original layout.
Zap only the intended device: zapping the shared SSD removes the DB partitions for every OSD on it, so limit zap actions to the device you plan to reuse.

Compare the device listing before and after redeployment to confirm that each OSD still maps to the DB or WAL device you expect.

Step-by-step replacement sequence with cephadm

This sequence takes a dead OSD out of service, keeps its OSD ID for the replacement drive, and lets the cluster rebuild the lost copies while clients keep working. Each step ends with a check so you know the cluster is ready for the next one.

Confirm the failed OSD and its host
Run ceph health detail to see which health check names the OSD, then run ceph osd tree to find its ID, host and device class. Record the OSD ID and the host it lives on before touching hardware. Also check ceph orch ps on that host to confirm the OSD daemon is down or in error. The step is done when you can name one OSD ID, one host and the device path that backs it, and when the other OSDs on that host show as up. If several OSDs are reporting failures on one host, stop and investigate the host or controller before replacing anything.
Mark the OSD out
Run ceph osd out <id> so the cluster stops placing new data on the failed OSD and starts moving its placement groups elsewhere. If the OSD is already down and out in ceph osd tree, this step confirms that state rather than changing it. Check ceph -s afterward to see the cluster begin to report recovery or backfill activity. Confirm that the cluster has enough free capacity on the remaining OSDs to absorb the data, since a near-full cluster can stall recovery. Ask your storage lead to confirm the replica or erasure-coding profile before this step, because it sets how much redundancy you have during rebuild.
Schedule removal with the replace flag
Run ceph orch osd rm <id> --replace through cephadm. The flag tells the orchestrator to remove the daemon and keep the OSD ID reserved, so the replacement drive can take the same ID later. Check the result with ceph orch osd rm status, which shows the removal progress and whether the OSD is draining or waiting. Wait until removal completes before pulling the drive. Confirm with ceph osd tree that the ID is still listed as destroyed, not absent. If the ID disappears, the replacement will get a new number and any automation that depends on the ID will need changes.
Physically swap the drive
Your data center or hands-on team removes the failed drive and installs the replacement in the same slot, with the same bay mapping recorded in your asset inventory. Record the serial number of the new drive and its slot. Check that the host's operating system and the cephadm host inventory show the new device after a rescan. Run ceph orch device ls <host> and confirm that the new device appears and is marked as available. If the device does not appear, check the controller, cabling and whether the drive is in a RAID or JBOD mode that hides it from the OS. This step is done when the new device is listed as available on the right host.
Zap the new device
Run ceph orch device zap <host> <device> --force to wipe any old partitions, LVM metadata or BlueStore labels from the new drive. This step destroys data on that device, so confirm the device path against the serial number you recorded before running it. Check the output for a success message, then rerun ceph orch device ls <host> and confirm the device is available with no leftover LVM or OSD metadata. A device that still reports as in use usually has a previous OSD label, and you should resolve that before continuing rather than forcing a second zap without understanding the cause.
Let the service spec redeploy the OSD
Check that your OSD service specification covers the new device, either by a device selector such as all available drives on the host or by an explicit path. Cephadm should then create an OSD on the zapped drive and reuse the reserved ID from the removed OSD. Watch ceph orch ps <host> and ceph osd tree until the new OSD appears as up and in with the same ID. If the spec does not match the device, cephadm will not deploy it, so review the spec with ceph orch ls osd --export before assuming a fault. The step is done when the ID matches the one you recorded in the first step.
Watch recovery until placement groups are clean
Monitor ceph -s and ceph health detail while the cluster backfills data onto the new OSD and the placement groups return to active and clean. Track the count of degraded and misplaced objects, and check that client I/O stays within your latency targets. Expect the cluster to remain in a warning state until the recovery finishes, and do not clear health warnings early. Confirm the final state with ceph health reporting OK and with no placement groups in degraded or backfilling states. Record the OSD ID, serial number, and the times of each step in your change log.

How the OSD service spec decides which drives get redeployed

In a cephadm cluster, the OSD service specification decides whether a replacement drive is picked up automatically. cephadm applies an OSD spec of type osd to hosts matched by placement, then uses its device selectors, such as data_devices and db_devices, to decide which unused drives become OSDs. When you swap a dead drive and the old OSD has been removed with --replace, cephadm keeps the OSD ID reserved for that position. If the new drive matches the spec's selectors and is clean (no partitions or leftover LVM metadata from another cluster), the orchestrator can create the OSD under the same ID without a manual ceph orch daemon add osd command. If the new drive does not match, nothing happens, and the cluster stays degraded until someone deploys it by hand. Confirm the exact selector fields, their supported forms and the behavior for reserved OSD IDs against the cephadm documentation for your release, and check the spec on a test host before relying on it during an incident.

Drives that were not created through a spec need different handling. OSDs deployed manually with ceph-volume or left over from an earlier release may not be tracked by the orchestrator as managed services, so a spec can ignore them or, worse, try to claim disks they already use. Before replacing a drive, list the current OSD and spec state with the orchestrator commands for OSDs and service specs, and confirm that the failed device belongs to a managed service. If it does not, plan the replacement as a manual ceph-volume task and document the difference for the next person on call. The same check matters after an upgrade or after a host was reinstalled outside cephadm.

A broad spec carries a real risk. A selector such as one that matches all rotational or all unused devices will also match a drive you intended to keep out of Ceph, such as an operating system disk, a drive reserved for another workload, or a newly attached disk that has not been assigned yet. Once cephadm creates an OSD on that device, the data on it is lost, and the rebalancing that follows adds load to the rest of the cluster. Before you rely on automatic redeployment, run a dry run or preview of the spec if your release supports it, review which devices would be selected on each host, and narrow the selectors by model, size, path or host where the spec allows it. Record the approved device list for each host so that the replacement drive can be checked against it during the change window.

Operational risks during a drive swap and how to reduce them

A drive swap touches the OSD map, the cluster's authentication keys, shared metadata devices and live client traffic, so a small mistake can turn one dead disk into a larger outage. The controls below address the failure points that most often appear during a replacement in a cephadm cluster.

Confirm the device before zapping

The most damaging mistake is running a zap command against the wrong disk, which erases a healthy OSD's data and forces a second recovery. Before any destructive step, match the OSD ID to its host and device path using the cluster's own metadata, and compare the serial number on the label with the drive in the chassis. Record the device path and serial in the change ticket, and have a second engineer confirm them before running zap.

Watch for a second failure

Recovery reads from surviving copies or erasure-coded shards, which puts sustained load on the drives that still hold data. A marginal drive in the same host or pool can fail during that window, and the cluster then has less redundancy than the original failure left it with. Check the health output and the SMART data of neighboring drives before you start, and pause the swap if another OSD starts reporting errors.

Limit recovery impact on clients

Backfill and recovery compete with client reads and writes for the same disks and network links. Without a plan, applications see slow requests while the cluster rebuilds. Decide the recovery priority before the swap, adjust the recovery and backfill settings your Ceph release exposes, and set a check interval for client latency. Confirm the exact option names against the documentation for your installed release before changing them.

Check shared DB and WAL devices

When an HDD-based OSD keeps its DB and WAL on a shared SSD, the replacement must get a clean partition on that SSD, and the old partition must be removed from the device. A stale partition or a misidentified LV can leave the new OSD pointing at the wrong metadata or block the deployment. Inventory the logical volumes on the shared device, confirm which one belonged to the failed OSD, and remove only that entry.

Clear leftover keys and host state

A failed OSD can leave behind an authentication key, a CRUSH entry or a stale host directory, and a new OSD created with the same ID can then fail to authenticate or inherit confusing state. After removal, confirm the old key is gone and that the OSD no longer appears in the CRUSH map under its former host. Verify these with the cluster's status commands rather than assuming that the removal command cleaned everything up.

Record who ran each command

A drive swap often involves several commands run over hours, sometimes by different people on shift. Without a record, nobody can later tell which drive was zapped, when recovery started or which spec change was applied. Keep a shell session log or ticket trail with timestamps, the commands run, the service spec revision in your version control system and the names of the approvers. This record also supports post-incident review and change audits.

Questions teams ask before replacing an OSD drive

How long does recovery take after replacing a failed OSD drive?

No fixed duration applies to every cluster, because recovery time depends on how much data the failed drive held, the drive's capacity, the number of OSDs sharing the work, and the network and media speed. Recovery starts after the replacement OSD joins and backfills its placement groups, and it competes with client I/O unless you set recovery limits. Before the work starts, ask the engineer to measure the data on the failed OSD with ceph osd df and to estimate throughput from a test on a comparable drive. Then watch ceph -s and the PG states during the event. Recovery finishes when all placement groups report active and clean.

Can our own team replace an OSD drive without a partner?

Yes, if your team operates cephadm and has tested the procedure in a non-production cluster. The core steps are to confirm the cluster health, mark the OSD for replacement with ceph orch osd rm and the --replace option, physically swap the drive, zap the new device, and let the service specification redeploy the OSD. Your team needs clear rules for when to stop and who approves each step. A partner is most useful when the team has not done this recently, when the cluster is already degraded, or when a change window is short. Confirm your team's skills against the cephadm documentation before deciding.

What does a partner add to an OSD replacement, and what does it cost?

A partner adds a written runbook for your cluster, a pre-check of health and capacity, a second pair of hands during the swap, verification of recovery, and a record of what changed. Cost comes from the engineer time to plan, perform and verify the work, plus any travel or on-site support, and it is usually fixed by a written scope. One-time costs cover the runbook and the first replacement. Recurring costs appear only if you buy a retainer for support. Ask any provider for the deliverables, the number of drives covered, the on-site or remote split, the maintenance window assumptions, and what happens if recovery is slower than expected. DeployOpen does not publish prices, and a quote follows from the written scope.

Does the OSD ID stay the same after replacing the drive?

Yes, when you remove the failed OSD with the --replace option, Ceph keeps the OSD ID reserved so the new drive can take over the same identity. This avoids changing CRUSH entries and reduces data movement compared with removing the OSD and creating a new one with a new ID. Confirm the ID with ceph osd tree after the removal, then check that the new OSD is assigned the same ID once it deploys. If the ID is not reused, stop and check the orchestrator logs before continuing, because a mismatch can point to a spec or device problem.

What must we provide before a partner replaces an OSD drive?

You should provide four things before the work starts:

Access to the cluster through cephadm, with an account that can run orchestrator and health commands, and access to the affected hosts.
A maintenance window or an agreed change window, along with rules for pausing recovery if client latency rises.
An inventory of hosts, OSDs, drive models, firmware versions, device paths, and the service specification used for OSD deployment.
A spare drive that matches your standard model, plus the physical access and escalation contacts needed for the swap.

Ask the provider to confirm which of these they need in writing, and check them against your change control process.

When should we call for help instead of retrying OSD commands?

Stop and escalate when a command fails in a way you do not understand, when the new OSD does not appear after the expected deployment, when ceph -s shows more than the affected OSD degraded, or when recovery slows client I/O beyond your agreed limits. Repeated retries can remove or recreate state you need for diagnosis. Also escalate if the failed OSD was part of a placement group marked inconsistent, if other drives in the same host show errors, or if the cluster is near full. Keep the command output and logs before any retry, and record the time of each step.

Which drive failures should we treat as more urgent than a routine swap?

Treat a failure as urgent when the cluster is already degraded, when several OSDs fail on the same host or in the same rack, or when the cluster is close to its full ratios. A failure in a host that carries metadata or WAL and DB devices shared across several OSDs also raises the stakes. In these cases, the replacement affects more placement groups and recovery takes longer. Check the failure domain layout in your CRUSH map before acting, and confirm the health state with your team before starting a swap.

How to confirm recovery finished and the cluster is healthy

Recovery is finished only when the cluster reports HEALTH_OK and every placement group is active+clean, so start by watching ceph -s after the new drive is in place. The output shows the monitor quorum, OSD counts as up and in, and the data section where degraded and misplaced object counts should fall steadily to zero. Run ceph health detail whenever ceph -s shows a warning, because it names the specific placement groups, OSDs or daemons behind each code. Expect the degraded count to drop over time rather than in a straight line, since backfill runs in stages and client load changes the pace. A flat or rising count after the rebuild has started is a reason to stop and investigate, not to wait.

Track placement group states rather than the overall percentage. Use ceph pg stat for a quick summary, and ceph pg ls degraded or ceph pg ls undersized to list any group still short of full redundancy. Groups moving through backfilling, backfill_wait or remapped are normal during the rebuild, but any group stuck in incomplete, down or peering for a long time needs attention. Confirm the replacement OSD is up and in with ceph osd tree, checking that its ID matches the one you kept and that it sits under the correct host bucket in the CRUSH map. Then confirm with ceph orch ps and ceph orch device ls that the orchestrator reports the new device as in use by the expected OSD service. A wrong host or a stray device class is a placement error to fix before you close the job.

Before you call the job complete, let at least one full set of scrubs run on the affected placement groups and check that they finish without new errors. Look for PG_DAMAGED or scrub error entries in ceph health detail, and confirm that the new drive shows no read or write errors in its SMART data and in the host kernel log. If the health codes persist after recovery reaches active+clean, do not keep retrying commands. Use the sibling pages on health warnings and their urgency only as a reference for the code names, and move to the dedicated troubleshooting pages for slow requests, inconsistent placement groups or stalled recovery. Record the final ceph -s output and the OSD tree in your change ticket so the next on-call engineer has a baseline.

How DeployOpen works on this

DeployOpen plans and carries out OSD drive replacement in a cephadm cluster that you own and run in your own infrastructure. We agree the scope in writing first, then rebuild your replacement procedure, test it against your own cluster and document the result so your team can repeat it.

How we scope

Inventory your hosts, drive models, OSD IDs and the service spec that currently places OSDs.
Agree in writing which hosts, drive types and recovery checks the engagement covers.
Confirm which maintenance windows, change approvals and client I/O limits apply before any drive is pulled.

How we build

Document the removal path with the --replace flag and confirm the OSD ID is kept for the new device.
Test zapping and redeployment through the service spec on a non-production host or lab cluster first.
Set up a recovery watch with health checks and a rollback note for each replacement step.

How we hand over

Deliver a written runbook covering each command, expected output and the stop conditions for the procedure.
Walk your operations team through one supervised replacement so they can run the next one alone.
Record the cluster state before and after the change so auditors and your own reviewers can follow it.

How we support

Engagements are fixed-scope, with the deliverables and timelines agreed in writing before work starts.
Ongoing support, upgrades and continuous testing are available as a retainer if your team wants help on later drive failures.
Ask any partner to confirm how they handle drives they did not install and which Ceph release they support.

Replacing a failed OSD drive in a cephadm cluster

How to replace a failed OSD drive in a cephadm cluster without rebuilding the map

Choosing between replace, remove and redeploy paths

Step-by-step replacement sequence with cephadm

Confirm the failed OSD and its host

Mark the OSD out

Schedule removal with the replace flag

Physically swap the drive

Zap the new device

Let the service spec redeploy the OSD

Watch recovery until placement groups are clean