Troubleshooting Ceph slow ops

Slow ops in Ceph mean requests are waiting on one or more OSDs, and the cause is usually a drive, a network path, recovery load or a software defect. This page explains how to trace each one to its source.

Book a callTalk to us
On this page

Why Ceph slow ops happen and how to trace them to a cause

When clients see stalled I/O and the cluster reports SLOW_OPS or slow requests, the first thing to understand is that a slow op is a symptom, not a cause. In RADOS terms, an operation sent to an OSD has been in flight longer than the warning threshold without completing. The OSD, the network between it and its peers or clients, the drive underneath it, recovery traffic competing for the same resources, or a software defect can each produce that signature. The health message tells you which OSDs are holding requests and for how long, but not why. Your job is to follow the wait to the component that is actually slow, and to resist changing settings before you know which component that is. If the stall is intermittent, affects a whole pool, or persists after you have checked the obvious hardware, plan for outside help early, because each extra hour of guessing keeps client I/O degraded.

Before you tune anything, confirm which release you are running, because the bugs you will hit and the fixes you can apply depend on it. Squid (19.2.x) was first released September 26, 2024, and its estimated end of life is October 31, 2026. Tentacle (20.2.x) was first released November 18, 2025. The Ceph releases page states that actively maintained releases receive periodic backports and security fixes, so a cluster on an older point release may already have a fix you are about to diagnose from scratch. Check the running versions across daemons with ceph versions, and compare them with the release notes before you assume a defect is new. Confirm the current end-of-life dates against the releases page at the time you plan your work, since those dates are estimates.

Tracing a slow request to the OSDs behind it

This sequence takes a slow-ops health warning and narrows it to specific OSD daemons, the operations they are stuck on, and the hardware, host or network path they share. The output is a short list of suspects that the team can act on.

  1. Read the health detail for slow operations

    Start with ceph health detail, which lists the SLOW_OPS warning with the number of blocked operations, the oldest operation age, and the OSD IDs that report them. Note every OSD named, the age of the oldest request, and whether the count is growing between runs. Repeat the command a few minutes apart to see whether the problem is persistent or intermittent. The step is done when you have a written list of OSD IDs and their reported ages. Confirm exact output format and field names against the Ceph documentation for your installed release.

  2. Map OSD IDs to hosts and devices

    Run ceph osd tree to see which host and CRUSH location each reported OSD belongs to, and ceph osd df to see utilization and weight per OSD. Pair this with the drive model and device path from your inventory or from ceph device ls. The output shows whether the slow OSDs cluster on one host, one rack or one drive model. The step is done when every suspect OSD has a host, a location and a device recorded.

  3. Query the OSD for in-flight operations

    On the host that carries a suspect OSD, use ceph daemon osd.<id> dump_ops_in_flight through the admin socket, or the equivalent ceph tell osd.<id> dump_ops_in_flight if your release supports it from the cluster side. The output lists each operation still running, its age, its type, and the event timeline showing where it waits, such as queued for the device, waiting on replicas, or waiting for recovery. The step is done when you can name the stage where the oldest operations are stalled.

  4. Review recent slow operations in history

    Run ceph daemon osd.<id> dump_historic_ops or the matching historic-ops command for the release to see completed operations that exceeded the complaint threshold. Compare their durations and stages with the in-flight output. Repeated long waits at the same stage point to a consistent cause, while a few isolated long operations suggest a transient event such as a scrub or a recovery burst. The step is done when you can say whether the delay is constant or came in bursts.

  5. Check the drive and host under the OSD

    On the suspect host, inspect the device with smartctl -a on the underlying drive and check kernel logs for I/O errors, resets or timeouts. Look at device latency with your OS tools (for example iostat -x) during the slow window. Recurring media errors, reallocated sectors or high await times on one device point to a failing or overloaded drive. The step is done when the device has either been cleared or flagged as a likely cause.

  6. Check whether suspects share a network path

    If the slow OSDs span several hosts, look for a shared switch, rack, bond or uplink. Check interface errors and drops on the host network interfaces, and confirm bonded links are healthy on both switches. Compare the failing OSDs' host locations from the CRUSH tree with your network topology. The step is done when you can say whether the slow operations follow a host, a rack, a switch or a network path, or whether they do not share one.

  7. Record the suspects and recovery activity

    Check ceph status and ceph -s for active backfill or recovery, and note whether scrubs were running in the same window. Write down the OSD IDs, their shared attributes, the stage where operations stalled, and the recovery state at the time. Hand this to whoever decides between replacing hardware, reducing recovery pressure or escalating a suspected bug. The step is done when the findings are documented with timestamps and the next action is assigned.

Failing drives, network faults, recovery load or bugs

Slow requests that stay pinned to the same OSDs usually point to a drive. A failing device often slows down before it reports errors, so latency is the first signal. Start by finding the OSDs with high commit and apply latency, then check whether those OSDs share a physical device with a DB or WAL. The hardware recommendations advise moving WAL and DB to shared SSDs on HDD-based OSDs, so one saturated or failing SSD can slow several OSDs at once.

  • SMART errors: Run smartctl -a on the device behind the suspect OSD and look for reallocated, pending or uncorrectable sectors and command timeouts. A rising count between checks matters more than one reading.
  • Media wear: On SSDs, check the wear and percentage-used attributes. Metadata drives are expected to carry heavy write load, so wear there can explain sudden latency.
  • BlueStore latency: Compare per-OSD latency with ceph osd perf. One OSD far above its host peers points at that device. Several OSDs on one host climbing together points at a shared controller, cable or DB device.
  • Shared DB or WAL devices: Use ceph osd metadata and your host layout to see which OSDs depend on one SSD. A single slow shared device can look like a dozen slow OSDs.
  • Kernel device logs: Check the kernel log for I/O errors, resets and timeouts on the block device behind each OSD.

Confirm each suspect drive against its SMART history and its peers before you mark it out, because removing a healthy OSD adds recovery load to the rest of the cluster.

Which signals point to which cause

Slow requests in Ceph rarely announce their cause. The pattern you see, whether one OSD, one host, a recovery window, many OSDs at once or a change in version, narrows the search before you touch any hardware. Use the cards below to choose a first check, then confirm each cause in your own cluster before acting on it.

One OSD or one host slow

When slow requests name a single OSD, or every named OSD sits on one host, the likeliest causes are a failing drive, a saturated disk or a host-level problem such as a bad network port. Start with the OSD's own latency counters and the drive's SMART data on that host. Then compare the same metrics across peer OSDs in the same placement group. If one device is the outlier, treat the drive as suspect before changing any Ceph setting. Confirm the pattern in your environment with ceph health detail and ceph osd perf.

Slowness during backfill or recovery

If latency rises right after a drive replacement, a host addition or a rebalance, recovery traffic is competing with client I/O. This is often expected load, but it can become an outage-level problem when recovery settings are aggressive for your hardware. Check how many PGs are backfilling or recovering and whether the slow requests track that window. Review your recovery and backfill limits against your network and media. Confirm the recovery profile your team intends to run before the next expansion.

Slowness across many OSDs at once

When slow requests appear on many OSDs at the same time, and no single device stands out, suspect a shared dependency. Common examples include a monitor or manager under pressure, a network path that drops or delays packets, a clock problem, or a workload that suddenly floods the cluster. Check whether the affected OSDs share a switch, a bond, a rack or a time source. Compare network error counters on hosts that span the affected set. Confirm which shared path is involved before you tune any daemon.

Slowness after an upgrade

If slow requests begin after a Ceph upgrade or a change to configuration, compare the running versions across all daemons first, since mixed versions during an upgrade can produce unexpected behavior. Then review the release notes for the version you moved to and check for changed defaults or known issues that match your workload. Look for a timing match between the change and the first slow request. Confirm your upgrade path and rollback plan with your team before you change the cluster further.

Remediation options and when each fits

This table compares the main responses to slow ops in a Ceph cluster, from draining a suspect drive to upgrading to a maintained release. Read each row against the evidence you have collected: the option that fits a failing device is often the wrong one for a saturated network or a known bug. Throttling recovery is usually the safest first step when the cause is still unclear.

OptionWhen it fitsWhat it risks
Drain or replace a suspect OSD drivePer-OSD latency stands out from its peers, the drive shows media errors in kernel logs or SMART data, and the same host shows the pattern repeatedly.Removing an OSD triggers backfill across the cluster, which adds load while the drive is already degraded. Confirm capacity and recovery headroom before you start.
Throttle recovery and backfillSlow ops rise during rebalancing, drive replacement or a host returning to the cluster, and client latency recovers when recovery is paused or slowed.Recovery takes longer, so PGs stay degraded for longer and redundancy is reduced until work resumes. Lower limits gradually and keep an eye on at-risk data.
Correct a network pathOps cluster on hosts sharing one link or switch, packet errors or drops appear on specific interfaces, or latency between OSD hosts is uneven.Changes to bonding, MTU or switch configuration can cause wider outages if applied without a rollback plan. Test MTU end to end and change one host at a time.
Stop or reduce a heavy client workloadOne pool or client drives most of the I/O, and the saturated resource is the cluster rather than individual drives.Throttling a client affects its application, so agree the change with the owner first. Limits may need to be lifted once the root cause is fixed.
Schedule maintenance for scrubs and snapshot trimmingSlow ops appear at predictable times that match scrub windows or large snapshot deletions.Shifting scrubs to off-peak hours delays detection of silent corruption. Keep scrub coverage and review the schedule after the change.
Upgrade to a maintained releaseLogs or the release notes point to a defect fixed in a later point release, or the cluster runs a release that is no longer maintained.Upgrades carry their own risk and require a tested path, backups and a rollback plan. Check the release notes and confirm the path with the Ceph releases index before you start.
Add capacity or hostsSustained load is near what the current OSDs can serve, and the cluster has no headroom during recovery.Expansion itself moves data and adds recovery load. Plan it in phases so the change does not create the next incident.

How the messenger and monitors affect slow ops

Before blaming a drive, confirm how the network layer is configured, because the messenger shapes every OSD-to-client and OSD-to-OSD exchange. Ceph's msgr2 protocol listens on port 3300, and its secure mode encrypts all post-authentication traffic with AES-GCM and adds integrity checks. Encryption adds CPU work on each connection, so a host with a saturated core or an undersized processor can show added latency that looks like a slow disk. Check the ms_cluster_mode, ms_service_mode and ms_client_mode settings on both the monitors and the OSDs, confirm that all daemons agree on the mode, and compare CPU usage on the affected hosts during the slow period. Also confirm that port 3300 is open between every pair of hosts that carry replication traffic, since a firewall rule that blocks only one path can produce intermittent retries that surface as slow requests.

Monitors and managers do not serve client I/O, but their health determines how quickly you can see the problem and how accurate the reported state is. If the monitor quorum is unstable, or a manager has stopped publishing statistics, slow request warnings may lag behind reality or disappear from the cluster view while the underlying latency continues. Run ceph quorum_status and ceph mgr stat during an incident, and check the clock agreement between monitor hosts, because drift can disturb quorum. Confirm with your team which monitoring path you rely on for alerts, whether the dashboard, the Prometheus exporter or the logs, and test that each path still reports when a manager fails over. A quiet cluster view during an incident is not evidence that the OSDs are healthy.

Security and operational considerations during an incident

Slow requests usually arrive during an outage, when the pressure to change something is highest. The four areas below cover who can run triage commands, how changes get recorded, what evidence to keep before a daemon restarts, and how encrypted transport affects what you can see and fix.

Limit who can run triage commands

Read-only commands such as ceph health detail, ceph osd perf and ceph daemon queries show which OSDs are behind, but they also expose cluster topology, pool names and client addresses. Use named operator accounts with the narrowest Ceph capabilities that allow reads, and avoid sharing the admin keyring on jump hosts. Check who holds admin access on each host running cephadm before the incident, because a shared admin key makes later review of triage actions almost impossible.

Record every change made during triage

Restarts, OSD marks out or in, recovery throttle changes and reweights all alter data placement or client I/O while the incident is still open. Log the time, the operator, the command and the reason in the incident channel before running it, and note the value you changed from so it can be reverted. Ask your team to confirm whether a second engineer must approve changes to production pools during an active incident.

Preserve logs before restarting daemons

A restarted OSD or monitor can lose the in-memory state that explains a slow request, and rotated logs can disappear on busy hosts. Before restarting, copy the daemon logs, the output of ceph health detail and the ceph osd perf history from each affected host to storage outside the cluster. Confirm your log retention on each host, since evidence that is not copied off the host before a restart is usually gone when the review starts.

Account for encrypted transport in triage

Messenger v2 can run in a secure mode that encrypts all traffic after authentication with AES-GCM and adds integrity checks, which protects client and cluster traffic from eavesdropping. The same encryption means packet captures show only ciphertext, so network diagnosis has to rely on daemon-side counters, latency and retransmission metrics. Confirm with your team whether secure mode is enabled on each cluster, and whether the ports used by the monitors and OSDs are open between the hosts you are inspecting.

Questions teams ask before hiring help with slow ops

How much effort does a Ceph slow ops triage take?

Effort depends on how many hosts and OSDs are involved, how much history is available, and whether the cause is already suspected. A first pass usually means reading cluster health output, identifying which OSDs and hosts show the longest operation times, and checking whether the pattern follows one drive, one host, one network path or a recovery window. Effort rises when slow requests come and go, when logs are not retained, or when several causes overlap. Ask for the scope to state which hosts, daemons and time windows are covered, and whether the output is a written root cause or a set of next steps.

Can our own team diagnose slow ops internally?

Many teams can run the first checks themselves if they know Ceph well enough to read OSD operation history and interpret the health detail. The harder part is separating a failing drive from a saturated network, a recovery storm or a software defect, which needs per-OSD latency, kernel logs, switch counters and a record of recent changes side by side. Internal teams often lack one of these views during an incident. An honest self-assessment asks whether someone can name the slowest OSDs, check their drive health and explain the recovery settings in use within the first hour.

What access and data do we need to provide?

Expect to provide read access to the cluster's status and health output, OSD and host performance data, daemon logs for the affected period, and kernel or drive logs from the hosts involved. Metrics history matters because slow requests are often transient, so retained Prometheus data or equivalent is far more useful than a snapshot. You will also need a record of recent changes such as upgrades, recovery or balancing activity, new hardware and network work. Confirm with your security team how remote access is granted, what is logged, and whether any hosts sit in a restricted network segment.

Does the investigation need change windows or downtime?

Read-only investigation usually does not require downtime, but some fixes do. Draining or replacing a drive, changing recovery priorities, restarting an OSD or updating a network configuration each affects client I/O while it runs. Plan change windows with your storage owners before work starts, and agree which actions need approval at each step. Ask the provider to separate diagnostic steps from changes and to list which changes would require a window, a rollback plan or a maintenance notice.

How long does slow ops work take for different cluster sizes?

Duration is driven by cluster size, the number of suspect components, the quality of available telemetry and the fault type. A single failing drive in a small cluster is generally quicker to isolate than intermittent network loss across many hosts, and a recovery-driven slowdown depends on how long recovery runs and how it is tuned. A fixed-scope engagement should state the timeline in the written scope rather than as a general estimate. Ask what happens if the first findings point to a different cause, and how that change is handled in scope and schedule.

Is the fix a one-time cost or a recurring one?

Triage and root cause work are typically one-time, while the things that prevent recurrence can be either. Replacing hardware, correcting network design or upgrading to a supported Ceph release are one-time changes with their own ongoing support. Monitoring, alert tuning, periodic capacity reviews and upgrades create recurring work, which can be handled in-house or through an ongoing retainer. Ask the provider to split the quote into one-time work, recurring services and hardware or licensing costs, since Ceph itself is free to use under LGPL terms.

What should we ask for in a written quote for slow ops work?

A useful quote lists the hosts, OSDs and time windows in scope, the data sources you will provide, the deliverables (for example, a findings report, a prioritized action list and any runbook changes) and the acceptance criteria for each. It should state what is excluded, such as hardware purchases, network redesign or application-side tuning, and how changes are approved. Ask how work is staged, who performs changes, what rollback looks like, and how follow-up support is priced. Quotes follow from the written scope, so a clear scope produces a clearer quote.

How DeployOpen scopes and supports Ceph troubleshooting

DeployOpen approaches Ceph slow ops as a bounded investigation: we agree the scope in writing, collect evidence from your own cluster, and document the cause and the fix. Work runs inside your infrastructure, and ongoing support is available as a retainer once the cluster is stable.

How we scope

  • We confirm which clusters, hosts and time windows the written scope covers before any investigation starts.
  • We agree which Ceph release, deployment method and monitoring data you can share for the slow request review.
  • We list the access we need, such as read-only cluster commands and log exports, and what stays out of scope.
  • Engagements are fixed-scope, so deliverables and timelines are written down before work begins.

How we build

  • We collect slow request warnings, OSD latency counters and drive health data, then map each slow OSD to its host.
  • We separate drive, network, recovery and software causes by testing one hypothesis at a time on your cluster.
  • We check whether recovery, scrubbing or balancing activity lines up with the slow periods in your monitoring data.
  • Changes we recommend are tested in a planned window with a rollback step written into the change plan.

How we hand over

  • You receive a written findings report that names the affected OSDs, hosts and evidence behind each conclusion.
  • The report includes the exact commands we used, so your operators can repeat the checks later without us.
  • We document the remediation steps, open risks and the monitoring alerts we suggest adding for early warning.
  • A walkthrough with your storage and network teams covers how to read the output and escalate a future incident.

How we support

  • Ongoing support, upgrades and continuous testing are available as retainers after the initial investigation.
  • A retainer can cover repeat slow ops reviews, release upgrades and changes to hardware or network configuration.
  • Ask for the support hours, response expectations and escalation path in the written quote before you sign.

Sources and further reading

Related

Talk to our team.

Tell us what you're working on, whether it's a deployment, an audit, a security test or a cyber range. You'll speak with an engineer who can help you scope it.

  • 30-minute call: free, with no obligation.
  • NDA on request: we can sign before you share details.
  • Clear next steps: a scope and plan after the call.
Book a callhello@deployopen.com