Repairing inconsistent placement groups and scrub errors in Ceph
When a PG_DAMAGED or scrub error warning appears, the first decision is whether to repair immediately or to investigate first. Scrubbing is Ceph's periodic background check: each placement group (PG) is compared against its replicas or erasure-coded shards, and the scrub looks for objects whose data or metadata disagree. A PG marked inconsistent means the copies no longer match, so Ceph cannot be sure which one is correct. Before running any repair, confirm which PGs are affected, which OSDs hold them, and whether the warning is new or has been present for some time. The Ceph documentation on active releases is the place to confirm which release line your cluster runs, since repair behavior and health output can differ between versions.
A repair asks Ceph to overwrite the bad copy with a copy it considers authoritative, so the choice of source matters. If the inconsistency comes from a single OSD that is throwing read errors, a repair can copy good data over the damage and is usually the right step. If several OSDs disagree, or the inconsistency coincides with a recent failure, power event or upgrade, the team should first inspect the objects with the rados list-inconsistent-obj and rados list-inconsistent-pg commands and the scrub error details in the cluster log. A planned response means recording the affected objects, checking the drive's SMART data and kernel logs, confirming there is enough free capacity for recovery traffic, and agreeing who approves the repair. Confirm the exact output and options for your release in your own environment before relying on them.
Teams should treat a repeating inconsistency as a hardware or software signal, not a routine repair. If the same PG or OSD is flagged again after a repair, stop repairing on a schedule and investigate the drive, controller, cabling and BlueStore logs first. Keep a record of each repair, including the PG, the OSDs involved and the outcome, so that patterns across weeks are visible. Scrub scheduling and recovery priority are tuned separately, so confirm your current settings before changing them during an incident. Where the drive is failing, the replacement procedure is covered in the OSD replacement guidance, and the broader meaning of health codes is covered in the health warnings guidance.
Finding the inconsistent objects and their OSDs
This sequence moves from a damaged-PG health code to the exact objects and OSDs holding the disagreeing copies, so the repair decision rests on evidence rather than on the warning alone. Each step ends with a check that tells the operator whether to continue.
Confirm the health code and its scope
Start from the cluster status and the detailed health output to confirm which code is raised, typically PG_DAMAGED or an OSD_SCRUB_ERRORS style warning, and whether it is HEALTH_WARN or HEALTH_ERR. Note how many placement groups are affected and whether recovery or backfill is already running, since that changes how much data movement the cluster can absorb. The operator, usually the on-call storage engineer, records this in the incident log before touching anything. Check: the health detail names at least one placement group and the cluster status shows no unexplained new codes.
List the inconsistent placement groups
Query the cluster for placement groups flagged as inconsistent, using the listing command for inconsistent PGs in the pool or across all pools. Record each PG ID, its pool, and the last scrub and deep-scrub timestamps so you can tell whether the error was found by a routine scrub or a manual one. Confirm the exact subcommand and flags against the Ceph release in use, because output format has changed between releases. Check: the list is non-empty and matches the PG IDs shown in the health detail; any mismatch means the state is changing and should be re-queried.
Inspect the scrub error details for each object
For each inconsistent PG, request the inconsistency detail, which reports the affected object names, the shard or copy in each OSD, and the type of mismatch, such as a read error, a digest mismatch, or a size or omap difference. Keep the object names exactly as reported, since they are the input for every later decision. Confirm the detail command and its output fields against your release, as the schema differs across versions. Check: each object is listed with the shards that disagree, and you can say whether the error is a read error, a checksum mismatch, or a metadata difference.
Map the disagreeing copies to OSDs
Translate the shard information into OSD IDs using the PG mapping for each affected placement group, which shows the up and acting sets. Compare the OSD holding the bad copy against the other members of the acting set so you know which copy the scrub considers wrong. Record the host and the physical device behind each OSD ID with the OSD metadata and device listing, because the drive, not the OSD process, is what you will check later. Check: every disagreeing shard resolves to one OSD ID, one host, and one device, and no affected PG points at an OSD you cannot identify.
Check whether one OSD explains most errors
Group the affected PGs by OSD to see whether a single OSD or host appears in most or all of the inconsistencies. A concentration on one device points toward failing media, while errors spread across many OSDs point toward a wider cause such as a bug, a bad memory module, or a problem during a previous upgrade or recovery. Review the OSD logs from that daemon for read errors or checksum messages, and check the kernel log on its host for storage faults. Check: you have a written view of whether the errors cluster on one OSD, one host, or are spread out, and that view is recorded before any repair is requested.
Confirm the error is still present
Before choosing a repair path, re-check the inconsistent list and the health detail, because a routine scrub or recovery may already have cleared some errors. Note any PG that has returned to a clean state and remove it from the working list. Keep the object names and OSD mapping for the rest so the repair decision in the next stage refers to current data. Confirm this step against the release-specific documentation for scrub state and repair behavior. Check: the set of inconsistent PGs and objects is confirmed current, with a timestamp in the incident record.
When to repair now and when to hold
A single inconsistent object on an otherwise quiet OSD is the most common case, and usually the one where a repair is the right move. Before you run it, confirm that the error is isolated and that the drive is behaving normally. Pull the inconsistent objects and the shard or replica details first, so you know which copy the scrub flagged and whether the other copies agree. Check that the cluster is otherwise healthy, because a repair adds read and write load to the affected placement group.
- Confirm the object list: Identify the inconsistent objects in the affected placement group before you touch anything, and record them for the change ticket.
- Check the drive's own signals: Look at SMART attributes and kernel logs for the OSD's device. A clean device makes a one-off checksum mismatch more plausibly a transient fault.
- Compare the replicas or shards: Confirm that a majority of copies agree on the correct content, since the repair copies the authoritative version over the bad one.
- Check the cluster state: Make sure no other recovery, backfill or rebalance is running that would compete with the repair.
- Plan a follow-up scrub: Schedule a deep scrub of the placement group after the repair so you can confirm the error has cleared.
If the drive looks clean and the copies agree, a repair is usually low risk. Record what you checked so the next error on this drive has a baseline to compare against.
When the same OSD keeps producing scrub errors, the problem is more likely the device, its controller or its cabling than random bit rot. Repairing each error as it appears will keep the placement groups looking healthy while the drive keeps damaging data. The question to answer first is whether the OSD should be taken out of service, not whether the next repair will succeed. Treat repeated errors as a hardware signal and gather evidence before deciding.
- Count the recurrences: Note the OSD ID, the dates of each error and the placement groups affected, so you can see whether the errors cluster in time or spread out.
- Read the device health data: Check SMART counters for reallocated, pending or uncorrectable sectors, and check the kernel log for read errors, resets or timeouts on that device.
- Compare with peers: See whether other OSDs on the same host or in the same batch show similar warnings. A single failing drive looks different from a wider fault.
- Decide on removal: If the errors continue after repair, plan to drain and replace the device rather than repairing it again.
- Avoid a repair loop: Repeated repairs of the same objects on a failing drive can produce new errors on each pass, so stop and replace the device instead of cycling through repairs.
The risk of a hasty repair here is that it writes data back to media that cannot hold it reliably. Ask your hardware vendor which diagnostics they need before they accept an RMA, and keep the logs.
Scrub errors spread across several OSDs on one host point at something shared by those devices: the host controller, a backplane, memory, a power or cabling problem, or a kernel issue. Repairs on individual OSDs will not address a shared cause, and they will keep adding write and read load to the host while the fault is still active. The first step is to look at the host as a whole before you look at any single drive.
- Check the host controller and backplane: Review logs for HBA resets, link errors and enclosure events that affect several disks at once.
- Check memory and ECC events: Correctable and uncorrectable memory errors can corrupt data in flight, so review the host's memory error counters.
- Look at the timing: If the errors began after a kernel update, firmware change or power event, the change is a more likely trigger than the drives.
- Hold repairs until the host is stable: Pause repairs on that host until the shared cause is understood, and prioritize placement groups that are at risk.
- Verify the failure domain: Confirm that the CRUSH rules keep replicas on separate hosts, so one host fault does not remove the copies needed for a clean repair.
The risk of repairing first is that a repair can read from or write to a path that is still faulty, which can make the inconsistency harder to trace. Involve whoever owns the host hardware before running repairs at scale.
Scrub errors that appear while a cluster is recovering from a failed drive or backfilling after expansion are the hardest to read. Recovery already moves large amounts of data, and placement groups may be in a degraded state where some copies are missing or in transit. A repair started in that window can compete with recovery, slow client I/O and may act on a copy that is about to be replaced. Hold the repair until the cluster has a stable set of up-to-date copies, unless the data is at immediate risk.
- Check the placement group state: Confirm whether the affected placement group is degraded, backfilling or peering, since a repair on a moving group can act on stale data.
- Review the recovery settings: Check how much recovery and backfill is allowed to run alongside client traffic, and whether a repair will push it past your agreed limits.
- Confirm the source copy: Make sure the copy the repair will treat as authoritative is current and not one that is being rebuilt.
- Wait for recovery to finish when possible: If the data has enough healthy copies, let recovery complete, then scrub and repair the remaining errors.
- Escalate only if data is at risk: If the placement group has fewer healthy copies than the pool requires, treat it as an urgent incident and get a second engineer to review the repair plan.
A hasty repair in this state can overwrite a good copy with a bad one or add load that stalls recovery. Document the state of the cluster before and after, and rerun a deep scrub once recovery is complete.
Repair options and when each fits
This table compares four responses to inconsistent placement groups and scrub errors: waiting for the next scrub, running a targeted repair, replacing the OSD and letting recovery rebuild data, and escalating to outside help. Read each row for the trigger, the data risk and what to confirm before acting, and use the last row as the default when the cause is unclear.
| Option | Trigger | Data risk | What to confirm first |
|---|---|---|---|
| Wait for the next scheduled scrub | A single inconsistency on a PG that shows no read errors, no drive alerts and no recent host or network events | Low for the short term, but the bad copy stays in place and other replicas must remain healthy | Which replicas are clean, whether client I/O has hit the object, and whether the scrub schedule will run before the next change |
| Targeted repair of one placement group | PG_DAMAGED or scrub errors on a small number of PGs, with the damaged shard identified and the underlying drive showing no growing errors | Moderate: repair overwrites the inconsistent copy from the authoritative one, so confirm the chosen copy is the good one | The inconsistent objects listed in the PG report, the drive's SMART and kernel log history, and that the PG has enough healthy replicas to repair from |
| Replace the OSD and let recovery rebuild data | Repeated scrub errors on the same OSD, media errors in logs or SMART, or a drive that is failing | Higher during recovery because redundancy is reduced and recovery adds load, so confirm the cluster has spare capacity | The drive's health evidence, that replacement is planned through the service spec, and the recovery impact on client traffic |
| Escalate to a support or consulting engagement | Repairs that do not clear, repeated repair loops, errors across many hosts, suspected bugs, or data that must be recovered with care | Highest if actions are taken without a diagnosis, so get analysis before further writes or deletions | Whether your vendor support or a partner covers the release, the scope in writing, and which logs and cluster state to collect first |
| Default for an unclear cause | Errors with no obvious drive or host fault, or any situation where the team has not confirmed the damaged replica | Depends on the path chosen, so avoid repeating repairs until the cause is known | Current cluster health, the affected objects, recent changes, and the Ceph release in use, confirmed against the official docs |
Checking the underlying drive before blaming the data
When scrub errors come back on the same placement group or the same OSD, the drive or its controller is the first suspect, not the data. Ceph's scrubbing compares object copies and checksums, so an inconsistency tells you that one copy no longer reads back as it was written. A failing device, a marginal cable or a flaky HBA can produce exactly that pattern, and repairing the object without touching the hardware usually means the errors return within days. Before you run any repair, check whether the affected PGs share an OSD, a host or a controller.
Start with the kernel log on the host that carries the suspect OSD. Look for block layer I/O errors, medium errors, reset or timeout messages from the storage driver, and link resets from the HBA. Then read the SMART attributes for that device, focusing on reallocated, pending and uncorrectable sector counts and on the self-test log. BlueStore also logs read errors and checksum failures against the OSD in its own daemon log, which you can read through ceph log last or the cephadm logs on the host. Confirm which of these signals agree before you decide the drive is the cause, because a single transient event is not the same as a steadily worsening device.
In a cephadm cluster, map the OSD to its physical drive with ceph osd metadata <osd-id> and ceph device ls-by-daemon osd.<id>, then confirm the device path and serial number on the host. The hardware recommendations call for a dedicated drive per OSD, which makes this mapping direct: one OSD ID points at one serial number. If you have placed WAL and DB on shared SSDs, check those devices too, since a failing shared device can affect several OSDs at once. Confirm your own layout with your hardware vendor's tooling, because the mapping only helps if your inventory records match the drives in the chassis.
Why repair loops happen and how to break them
A placement group that goes inconsistent again soon after repair usually has a cause that repair does not touch. The cards below cover the causes we see most often, each with the symptom that points to it and the check that separates it from the others. Work through them in order before running another repair.
A failing drive
The most common cause is an OSD whose drive is still producing read or write errors after a repair. The same OSD appears in the damaged PG list again, often with shard read errors in the OSD log. Check the kernel log and the drive's SMART attributes for reallocated, pending or uncorrectable sectors on that OSD's device. If the errors return on the same device, stop repairing and replace the drive.
A bad controller or cable
Errors that move between drives on the same host point to the path between the drive and the CPU, not the media. Look for inconsistencies that appear on several OSDs behind one HBA, backplane slot or SAS cable, along with bus resets or link errors in the kernel log. Swapping the cable, reseating the controller or moving a drive to another slot will confirm the cause before you replace any hardware.
Memory faults
Bad RAM can corrupt data in flight or during checksum calculation, so a PG can look inconsistent even when every drive is healthy. The pattern is errors that appear on different hosts with no shared drive or controller. Check the machine check logs and run a memory test during a maintenance window. Ask your hardware vendor how to read the ECC error counters on your platform before you rule memory out.
Overloaded hosts during scrubbing
Scrubs read every object and compare replicas, so a host that is already short on memory, CPU or disk I/O can time out or fall behind, and the resulting mismatch looks like corruption. Check the OSD latency and the host load during the scrub window, and compare the failing OSDs against the busiest hosts. Reducing concurrent scrubs or moving scrub windows to quieter hours often clears the symptom.
Scrub and recovery scheduling conflicts
When a scrub starts on a PG that is still backfilling after a drive replacement or expansion, the PG may be read while its data is moving, and the comparison can report false inconsistencies. Look at the timing: does the error appear while recovery is still running on the same pool? Let recovery finish, confirm the PG is active and clean, and then scrub it again before deciding it is a real fault.
Running a safe repair and verifying the result
This sequence turns a damaged placement group back into a consistent one while keeping the evidence needed to show the fix held. Each stage ends with a check, so a repair is never run on a placement group whose cause is still active.
Capture the cluster state before any change
The on-call engineer records the output of ceph health detail, the list of inconsistent placement groups, the affected pool and the OSDs in each acting set. Save the output with a timestamp and the Ceph release version reported by the cluster. Keep this snapshot so you can compare it after the repair and show what was inconsistent. The step is done when the snapshot is stored outside the cluster and the object names, shards and OSD IDs are written down.
Confirm the underlying drive is healthy
Check the SMART data and kernel logs for each OSD in the acting set, and look for read errors, reallocated sectors, or BlueStore I/O errors in the OSD logs. Confirm that the host can read the device without errors. If the drive shows media errors or the same placement group keeps failing, schedule a drive replacement before repairing, because a repair on failing media will likely recur. The step is done when you have a written decision: the drive is healthy, or it is marked for replacement and the procedure in the OSD replacement guide is started.
Confirm the inconsistent objects and cause
Run the inspection commands for the placement group to list the objects and the shard or replica that disagree, and check whether the errors are data digest, omap or size mismatches. Note whether the errors appeared during a scheduled deep scrub or after a client error. Confirm that no other alerts are present, such as a recovering or backfilling placement group that could make the comparison stale. The step is done when each inconsistent object is named and you know which copy the cluster considers authoritative.
Run the repair on the placement group
Start the repair for the single placement group, using the ID recorded in the snapshot, and limit the change to that placement group. Watch the cluster log for the repair start and finish messages, and avoid running repairs on many placement groups at once while client load is high. Confirm which copy Ceph uses as the source of truth for each object, and review that the result matches your inspection notes. The step is done when the repair completes without new errors in the log.
Wait for a clean deep scrub
Repair is only proven when a later deep scrub of the same placement group finishes without errors. Use the scrub status for that placement group to confirm the deep scrub ran after the repair, and do not rely on the repair message alone. Watch recovery and client latency during the scrub, and pause further repairs if latency rises. The step is done when the placement group shows a deep scrub timestamp later than the repair and reports zero errors.
Confirm health returns to OK
Run the health command and check that the placement group is no longer listed as inconsistent and that the cluster reports HEALTH_OK, or that any remaining warnings are the ones you expected from your snapshot. Compare the current output with the saved snapshot to confirm nothing else changed. If the health code persists, read the detail before running more repairs. The step is done when the health output and the placement group state both match your acceptance criteria and the result is recorded.
Monitor for recurrence and record the outcome
Keep the placement group and its OSDs on watch through the next scheduled deep scrubs, and alert on any new scrub error or SMART warning for the same drive. Record the root cause, the drive decision, the commands and their output in your incident log. If the same placement group or drive produces errors again, treat it as a hardware or fault pattern rather than repeating repairs. Confirm which alerting and log tools your deployment uses, since the setup for these depends on your release, which you should confirm against the Ceph documentation for your version before relying on these commands.
Questions teams ask before repairing inconsistent placement groups
Can our own team repair an inconsistent placement group without outside help?
Yes, many teams can handle a single inconsistent PG in-house if they already operate the cluster and understand its failure domains. The work is mostly reading health output, identifying the affected OSDs and objects, checking the drive, and deciding whether a repair is safe. It becomes harder when the damage spans several PGs, the cluster is already degraded, or nobody on the team has run a repair before. A useful test is whether your engineers can explain what the scrub found and what they will do next before they run any command. Confirm your own runbooks and on-call coverage before deciding, because a repair can run for a long time and needs someone watching it.
What drives the effort of a scrub error repair?
The effort depends on how many PGs are affected, whether the damage comes from one failing drive or a wider pattern, and how well the cluster is running. Each inconsistent object needs to be traced to the OSDs that hold its copies, compared, and classified before a repair is chosen. A single drive with media errors is usually quicker to resolve than several hosts reporting mismatches, because the second case raises questions about network, memory, or firmware. Clusters with many objects, large OSDs, or heavy client traffic take longer to scrub and repair. Ask any partner to scope the review of logs, drive health records, and affected pools as separate line items so you can see where the time goes.
What access, logs, and maintenance windows does a repair need?
You need administrative access to the cluster through the Ceph command line or the cephadm orchestrator, read access to the logs of the hosts holding the affected OSDs, and access to drive health data such as SMART output. Someone must be able to run the commands during the repair, watch recovery progress, and stop work if client latency climbs. A maintenance window is useful when the repair competes with client I/O, though many teams schedule repairs for low-traffic periods rather than taking a full outage. Before starting, confirm who can approve pausing scrubs or adjusting recovery settings, and who owns the hardware if a drive needs replacement.
How long does a repair take for a cluster of our size?
Duration is driven by the amount of data to scrub and repair, the recovery load the cluster can absorb, and the time needed to diagnose the cause. A repair on a small number of objects can finish quickly once the cause is known, while a cluster with large OSDs or many affected PGs can take considerably longer because each PG must be verified after repair. Recovery and client traffic compete for the same disks and network links, so a busy cluster will slow the work. No duration can be stated without a look at your cluster's capacity, object count, and current health. Ask for a written estimate that lists the diagnostic phase, the repair phase, and the verification scrub separately.
Can a repair lose data?
A repair can make data loss worse if it is run before the cause is understood. Ceph repair generally copies the authoritative version of an object over a damaged replica, so the choice of source matters. If the drive holding the good copy is also failing, or if corruption exists on more than one replica, a blind repair may propagate the problem or remove the only good data. Taking a backup or confirming snapshots before repair, checking that enough healthy replicas exist, and reading the scrub output for each object reduce this risk. Confirm your backup and restore process with your team before any repair, since no repair procedure replaces a tested backup.
When should we bring in outside help for scrub errors?
Outside help is worth considering when scrub errors recur after repair, when several hosts or pools report inconsistencies at once, or when the cluster is already degraded. It also makes sense when the business impact is high and your team lacks someone who has repaired Ceph clusters before. A partner can review the health output, drive records, and logs, then write down a plan with clear checkpoints so your team can follow it. Ask for a fixed scope that covers diagnosis, repair steps, verification, and handover notes, and confirm in writing what your team must provide, including access and change approvals.
Limits to confirm in your environment
A scrub error repair is only as safe as the assumptions behind it. The release, the pool type, the hardware, the observability stack and the network access model each change which repair steps are safe and who can run them.
Ceph release and support window
Confirm the exact release running on every daemon, including the MONs, MGRs, OSDs and RGW hosts, before using any repair command. The actively maintained releases are Squid (19.2.x, first released September 26, 2024, estimated end of life October 31, 2026) and Tentacle (20.2.x, first released November 18, 2025, estimated end of life June 1, 2027). A cluster on a release nearing end of life should plan an upgrade before a long repair program, because fixes for scrub behavior arrive through backports on maintained branches.
Replicated or erasure-coded pool
Find out whether the affected pool is replicated or erasure coded, and its size and min_size. On a replicated pool, a scrub error usually means one copy disagrees with the majority, so the repair can copy a good replica back. On an erasure-coded pool, the reconstruction depends on the remaining chunks, so you need the profile (k and m) and the count of healthy shards before you run anything. Check that enough shards remain before deciding on a repair.
Drive and controller models
Record the drive make, model, firmware and interface for every OSD in the affected placement groups, along with the HBA or RAID controller and its cache settings. Some RAID controllers hide SMART data or cache writes in ways that make repairs unreliable. Confirm whether the drives are HDD with SSD-backed WAL and DB devices, since a fault in a shared DB device can affect several OSDs at once.
Existing monitoring and logging
Confirm what already collects health and kernel data: the Prometheus and Alertmanager stack, the log platform, and whether SMART and kernel messages from each host reach it. Without that history, you cannot tell whether a scrub error is a single bad sector, a failing drive or a controller problem that began weeks earlier. Ask whether alert routing for PG_DAMAGED already exists and who receives it.
Air-gapped or connected access
Establish whether the cluster can reach the internet, vendor repositories and container registries. In an air-gapped site, package and container images for any fix release must be staged through an approved transfer path, which lengthens the time to a patched daemon. Also confirm who can reach the hosts with cephadm access, because repair commands run on the cluster and any change to hosts is logged under your own access controls.
How DeployOpen investigates and repairs Ceph inconsistencies
When scrub errors appear, we start with a written investigation scope rather than a repair command. The scope names the affected pools and placement groups, the scrub and deep-scrub history, the OSDs that hold each inconsistent copy, and the evidence we need from your team: ceph health detail output, ceph pg query results, OSD logs and kernel messages from the hosts involved. You approve the scope, the decision points where we stop for your sign-off, and what is out of bounds, such as any repair on the primary during business hours. Investigation work runs inside your own cloud accounts, data centers or air-gapped network, using your change process and access controls.
Findings are documented as we go. For each inconsistent object we record the error type, the replicas that disagree, the drive SMART and BlueStore evidence, and whether the same OSD appears in earlier events. That record is what separates a one-off bad sector from a failing drive or a recurring fault that will return after repair. We hand over a runbook covering the checks we ran, the repair decisions and their reasons, the thresholds that should trigger drive replacement, and the rollback steps. Your team should confirm that the runbook matches your cephadm version, host access model and ticketing workflow before it goes into production use.
Many clients want help after the first incident, so we also offer ongoing support as a retainer. A retainer typically covers scrub-error triage, review of monitoring alerts, planned repairs during maintenance windows and verification after drive swaps. Retainer scope, response expectations and reporting cadence are written into the agreement, so you can compare them against your own on-call capacity. Most engagements are fixed-scope, and a quote follows from the scope you approve. To understand what drives effort, ask for a breakdown covering cluster size, number of affected placement groups, access constraints and whether the investigation includes a drive replacement.
