Ceph mClock recovery vs client I/O tuning

How mClock balances recovery and client I/O in Ceph

The mClock scheduler decides how much of each OSD's capacity goes to client I/O and how much goes to background work such as recovery and backfill. Teams usually reach for it after a drive fails and one of two things happens: application latency climbs because recovery traffic competes with reads and writes, or recovery runs so slowly that the cluster stays degraded longer than the business accepts. Both symptoms come from the same split. When the split is wrong for the workload, adjusting it is a tuning decision, not a hardware purchase, and it should be made on evidence from the cluster you actually run.

This page covers the built-in profiles, how to measure what each OSD can really deliver through OSD capacity benchmarks, when custom settings are justified, and how to verify the result under load. The exact behavior, profile names and tunable options depend on the Ceph release in use, so confirm them in the Ceph documentation for your version before you change anything. Start by recording current recovery times, client latency and OSD utilization during a controlled test, so you can tell afterward whether a change helped or only moved the bottleneck.

Plan for the trade-off explicitly. Favoring recovery shortens the time data spends at reduced redundancy, but it can increase client latency during that window. Favoring client traffic protects applications, but a long recovery leaves less margin if a second failure occurs. Decide in advance which outcome matters more for each pool, record that decision with your change controls, and confirm with your team which profile the cluster is running before a drive failure forces the question.

Which mClock profile fits which goal

This table compares the three built-in mClock profiles and the custom option by what each favors, the cost it accepts and the situation where it fits. Read it against your own goal: protect applications, shorten the period of reduced redundancy, or keep both in reasonable balance. Confirm profile names against your Ceph release before you change anything.

Profile	What it favors	Trade-off it accepts	When it fits
balanced	A middle split of capacity between client I/O and background work such as recovery and backfill	Recovery runs slower than under high_recovery_ops, and client I/O is lower than under high_client_ops during heavy rebuilds	Default starting point for mixed workloads where applications must stay responsive and recovery should still finish in a reasonable window
high_client_ops	Client read and write operations, so applications see the least disruption during recovery	Recovery and backfill take longer, which extends the period in which data is less redundant	Latency-sensitive workloads such as virtual machine disks or databases on RBD that must keep serving users during a drive replacement
high_recovery_ops	Recovery and backfill, so placement groups return to full redundancy sooner	Client I/O slows during recovery, and applications may see higher latency or timeouts	Maintenance windows, or clusters where restoring redundancy quickly matters more than application latency for a limited period
custom	Specific mClock reservation, weight and limit values that you set for each OSD class or workload	Requires a benchmarked capacity baseline per OSD, and incorrect values can starve either clients or recovery	Clusters with measured hardware limits, mixed media or known workload patterns that the built-in profiles do not handle well
Profile names by release	Confirm that the profile name you plan to set is accepted by your installed Ceph release before you apply it	Names and availability can differ between releases, so a name copied from another cluster may be rejected	Any cluster, checked against your Ceph version in the release documentation and on a test cluster first

Choosing a profile for the situation you are in

When applications slow down while the cluster is rebuilding, recovery and backfill compete with client reads and writes for the same OSD queues. Confirm that link before changing anything. Client latency should rise when recovery traffic rises and ease when recovery pauses or finishes.

Client latency: Compare application latency and Ceph client operation latency during the recovery window with the period before the failure. A rise that tracks recovery activity points to scheduler contention.
Recovery rate: Record recovery throughput and the number of objects recovering per interval. Giving clients more share means accepting a slower rebuild.
Degraded object counts: Check that degraded and misplaced counts keep falling. Protection stays reduced while they remain above zero, so avoid holding recovery back for long periods.
Profile to try first: Set high_client_ops through osd_mclock_profile so client operations receive a larger share of the scheduler, then watch latency and recovery rate on the same timeline.

Before keeping the change, confirm that the latency improvement holds through a full peak period and that degraded counts still fall.

When a drive fails and the cluster stays degraded longer than expected, recovery share is usually the bottleneck. Protection is the priority in this situation, so the signals focus on how quickly degraded objects are clearing and whether the hardware is being used.

Degraded object counts: Track the count over several intervals. A flat or slowly falling line means recovery is being throttled, which is the condition this tab addresses.
Recovery rate: Compare throughput with the drive and network baselines you recorded. If recovery is far below what the hardware sustained during benchmarking, the scheduler is likely limiting it.
Client latency: Confirm that applications tolerate current latency before raising recovery priority. If latency is already high, favoring recovery will make it worse.
Profile to try first: Set high_recovery_ops for a bounded window, then return to balanced once degraded counts reach zero or the window you agreed with application owners ends.

Before keeping the change, record when the degraded count reached zero and confirm that client latency returned to its pre-failure baseline.

If you cannot tell which workload dominates, or the cluster serves many applications with different peaks, any single assumption can fail in either direction. Measure first, then start from the balanced profile so that neither clients nor recovery are given a large advantage without evidence.

Workload attribution: Identify which pools and clients are active during the event using pool and client I/O statistics. Without that, choosing a profile is a guess.
Latency and recovery together: Read client latency and recovery rate on one timeline. Each change should move one measure without destroying the other.
Degraded object counts: Confirm that protection improves under balanced before deciding whether a high-priority profile is needed.
Profile to try first: Start with balanced. Move to high_client_ops or high_recovery_ops only when the signals in the other two situations are clear.
Capacity baseline: If OSD capacity benchmarks were never recorded, measure them before testing custom settings, because the scheduler's capacity estimates depend on them.

Before keeping any change, note the observation window and the setting you changed so the result can be compared with the next event.

How to change the mClock profile safely

This sequence turns a profile change into a measured, reversible decision: you capture a baseline, confirm what the cluster runs, benchmark what each OSD can deliver, change one setting, and compare under the same load before you keep it.

Record the baseline before touching anything
Capture client latency, throughput and recovery progress while the cluster is in its current state, ideally at the same time of day as the workload you care about. Record the numbers from your application or monitoring, the output of the cluster health command, and how many placement groups remain degraded, misplaced or backfilling. Write the baseline down with the current scheduler settings. You are done when someone else could read the record and reproduce the measurement.
Confirm the release and active scheduler
Check which Ceph release every daemon runs and confirm that the OSDs use the mClock scheduler, not the older weighted priority queue. Also check whether the profile is already set cluster-wide or overridden per OSD. Option names and commands differ between releases, so take them from the Ceph documentation for your release, which is linked from the active releases page. You are done when the release, scheduler type and current profile are written next to the baseline.
Benchmark OSD capacity on representative hardware
Run the OSD capacity benchmark on drives that match production, and on a clean OSD where possible, so the scheduler gets realistic IOPS figures for its cost model. Do this on each media class separately, because HDD and SSD OSDs behave very differently. Keep the benchmark results with the baseline. If a drive class has no measured value, the scheduler falls back to defaults that may not match your hardware. You are done when each OSD class has a recorded capacity figure you can explain.
Change one profile or custom setting
Change a single variable: either the profile, or one custom option. Do not combine a profile switch with recovery throttles or a capacity override, because you will not know which change caused the result. Apply the change through the configuration interface your release supports and confirm the value was accepted by the daemons, not only stored in the configuration. You are done when the running value on the OSDs matches the intended change and the change is written in your change log.
Run the same load and compare results
Repeat the workload from the baseline with the same client load pattern, the same pool and the same failure or backfill state where possible. Measure client latency percentiles, client throughput and time to clear the degraded or misplaced count. Compare the two runs side by side. Treat a result as valid only if the background conditions were comparable; if the cluster state differed, repeat the run. You are done when you can state which metric improved, which got worse, and by how much, using your own measurements.
Keep the change or roll it back
If the change meets the goal you wrote down at the start, keep it and record the setting in your configuration management so it survives daemon restarts and cluster rebuilds. If it does not, or if client latency rises beyond what the application tolerates, return the setting to the recorded previous value and confirm the running value again. Rollback should be a single documented step that a second engineer can perform. You are done when the cluster runs the chosen setting and the rollback path has been tested or written down.
Monitor and reassess after the next event
Watch client latency and recovery time through the next real recovery, such as a drive replacement or a backfill after capacity is added. A profile that suits a quiet test may not suit a larger rebuild, so keep the baseline and the test method for reuse. Ask your Ceph support provider or the upstream documentation to confirm any behavior you did not expect. You are done when the next recovery event is compared against the same measurements and the decision is confirmed or revised.

When custom mClock settings are worth the effort

Custom mClock values are worth the effort only when a built-in profile cannot meet a requirement you can measure. The usual triggers are mixed hardware generations in one pool, OSDs with uneven capacity, client latency targets that must hold during recovery, or a recovery window that has to finish by a set time. If the balanced profile already keeps client latency and recovery time inside your targets during a drive failure in a test or maintenance window, a custom profile adds maintenance cost without a clear benefit. Confirm the targets first: write down the latency percentile your applications need and the longest acceptable time to restore redundancy, then test the profiles against them before touching any option.

The main risk of custom values is that they can hide a capacity error. When an operator raises recovery priority or sets per-OSD IOPS by hand, the cluster may appear healthy while individual OSDs are overloaded, especially after a host is added with faster or slower drives than its peers. Custom settings also fix assumptions about what each OSD can deliver. Those assumptions go stale after drive replacements, a change in the number of OSDs per host, or a move from HDD to SSD media. Whenever hardware or pool layout changes, repeat the OSD capacity benchmark and the pool-level tests from your baseline, then confirm the custom values still match what the drives measured.

Record a before and after result for every change so the decision can be audited and reversed. At minimum, capture the active profile and any overridden options in the cluster configuration, the client latency and throughput under your workload, the time to recovery after a test failure, and the Ceph health output during the run. Keep the previous configuration so you can restore it in one step if client latency worsens. Ask your partner or internal team to confirm which options your Ceph release exposes for override and how they are named in your version's documentation, because option names and defaults change between releases.

Security and operational risks of changing scheduler settings

Changing mClock settings alters how every OSD in the cluster shares its capacity between client traffic and background work. The risks below are operational more than technical: they come from changes made without evidence, settings that differ across daemons, and access that lets one person change production behavior without a record. Each card names the control that keeps the change reversible and explainable.

Changing settings without a baseline

Without recorded client latency, recovery rate and OSD capacity results taken before a change, nobody can tell whether a new profile helped or simply moved the slowdown elsewhere. Run a benchmark under representative load, save the output with the cluster version and the active profile, and repeat the same test after each change. Ask your operators to state which metric they expected to improve before they touch the setting.

Recovery held back for too long

Favoring client I/O can keep recovery slow enough that placement groups stay degraded for longer, which extends the window in which a second failure could cause data loss. Before you lower recovery priority during business hours, check how many PGs are degraded and how long the current recovery is projected to take. Set a clear point at which the profile is reverted, and assign someone to watch the degraded count until it is back in range.

Latency worsened for other tenants

A profile that favors recovery can raise latency for applications that share the same pools, including virtual machines and gateway workloads you do not own. Identify which pools and clients share the affected OSDs before a change, and confirm with each owner what latency they can tolerate. Measure latency from the client side as well as from the cluster, because OSD counters alone can hide a slow tenant.

Settings drifting between daemons

A custom value applied to some OSDs but not others, or a host that was missed during a rollout, produces an uneven cluster that is hard to diagnose. Set options centrally in the cluster configuration database rather than editing individual daemons, and then check the effective values on sampled OSDs after each change. Include mClock options in your routine configuration audit so that drift is caught before it affects performance.

Changes missing from change control

Scheduler changes made by hand during an incident often never reach the change ticket, so the next responder cannot see why performance shifted. Record every change with the time, the operator, the old and new values, the reason and the rollback command. Keep the same record for emergency changes, and review them the next business day so that temporary settings do not become permanent by accident.

Broad access to cluster configuration

Anyone who can change cluster configuration can alter recovery behavior for every workload, so access should match the job each person does. Limit administrative rights to a small named group, use separate roles for monitoring and changes, and review who holds those rights on a set schedule. Require a second person to approve changes to recovery or client priority during production hours.

Questions buyers ask about mClock tuning and outside help

What does an mClock tuning engagement cost, and what drives the price?

Cost depends on the written scope, which DeployOpen agrees with you before work starts. The main drivers are the number of clusters and OSD hosts, the mix of hardware and Ceph releases, how many client workloads need protecting during recovery, and whether OSD capacity benchmarks and pool-level tests are included. DeployOpen does not publish a price list, so ask for a quote built from a written scope that names those inputs. Ask which parts are one-time (assessment, benchmarking, profile selection, documentation) and which recur (re-tuning after hardware or release changes).

How long does mClock tuning take?

Duration follows the scope, not a fixed template. A single cluster with clear workload owners and existing metrics usually moves faster than a multi-cluster estate where benchmarks must be repeated on several hardware generations. Time is typically spent on baselining current recovery and client latency, running OSD capacity measurements, testing profiles in a maintenance window, and recording results. Ask for a timeline in the written scope with milestones tied to your change windows.

What does our team need to provide before tuning starts?

Three things make the work faster and safer. First, read access to cluster metrics and logs, including OSD latency, recovery progress and client I/O graphs for the period around a past drive failure. Second, maintenance windows where profile changes and controlled recovery tests can run. Third, named workload owners who can describe latency targets and tell the team which applications must stay responsive during recovery. Confirm access paths and change approval rules with your own change board before the engagement begins.

Can an internal team tune mClock without outside help?

Yes, a capable internal team can tune mClock, provided it can measure before and after each change. The Ceph documentation describes the profiles and OSD capacity settings, so the work is learnable. The harder parts are running repeatable benchmarks, interpreting recovery behavior under real load, and avoiding changes that move a bottleneck elsewhere. Teams that lack a test cluster, metrics history or someone who owns Ceph day to day often benefit from an outside review of the profile choice and the recorded results.

Which mClock profile should we start with?

Start with the balanced profile unless you have measured evidence that client I/O or recovery must be favored. Choose high_client_ops when applications suffer during recovery and recovery can safely run longer. Choose high_recovery_ops when the priority is restoring redundancy quickly and short-term client latency is acceptable. Confirm the behavior of each profile against the Ceph release you run, because defaults and options change between releases. Record the reasoning so later operators know why a profile was selected.

What can go wrong when recovery priority is changed?

The most common problem is trading one symptom for another. Favoring client I/O can extend time at reduced redundancy, which raises exposure if a second failure occurs. Favoring recovery can push client latency past application timeouts. Incorrect OSD capacity values can also make the scheduler allocate too little or too much to background work. Guard against these risks by testing during a maintenance window, watching placement group states and client latency together, and keeping a documented rollback to the previous settings.

When does a retainer make more sense than a one-time tuning engagement?

A retainer fits when the cluster changes often enough that tuning becomes a recurring task: frequent hardware additions, regular Ceph release upgrades, several workloads with shifting latency needs, or a team that wants a partner available after drive failures. A one-time engagement fits a stable cluster that needs a documented profile, benchmark baseline and runbook. DeployOpen offers ongoing support and upgrades as retainers, and the terms are set in writing. Ask what response and review cadence the retainer includes before comparing it with a project.

Limits to confirm in your environment before tuning

Scheduler profiles only help when the cluster underneath them is supported, measured and allowed to change. The six checks below cover the release, the option names, the hardware behind any capacity figure, the workload pattern, other maintenance work and change approval.

Is your release actively maintained?

Start with the Ceph release your cluster runs and check it against the releases index. Actively maintained releases receive periodic backports and security fixes, so a tuning change made on an end-of-life release may not be supported later. Confirm the exact version reported by your cluster, not the version you intended to deploy, because upgrades and partial rollouts often leave mixed versions in place.

Do the scheduler and option names match?

Confirm which scheduler is active on your OSDs and which profile names and option keys your release accepts before you write any setting. Option names and defaults can change between releases, so copy them from the documentation for your version. A key that is silently ignored gives you a false sense that recovery has been throttled.

What hardware produced the capacity figures?

OSD capacity values only describe the drives and CPUs that were measured. Record the drive model and class, whether WAL and DB sit on separate SSDs, the CPU cores per OSD, and the network speed used during the test. A capacity figure copied from a faster host or from a different media type will push recovery and client limits in the wrong direction.

When does your workload peak?

Identify the client I/O mix, including read and write ratio, block sizes and the hours when latency matters most. Compare that with the windows when recovery, backfill and scrubbing run. A profile that suits a batch workload can slow a transactional database during business hours, so tie each profile choice to a named peak window.

Which maintenance tasks share the budget?

List every background task that competes with recovery for the same OSD time: scrubbing, deep scrubs, snapshot trimming, balancer runs, cephadm upgrades and planned drive replacements. Ask whether any of them will start during the tuning window. Changing recovery priority without accounting for these tasks can make a slow recovery look like a scheduler fault.

Does policy allow production configuration changes?

Find out who approves changes to cluster configuration, whether a change window or a second reviewer is required, and how changes are recorded and rolled back. Confirm whether your support contract or internal change policy covers custom settings. Write down the previous values before you change them so you can restore them quickly if client latency rises.

How DeployOpen scopes and verifies mClock tuning

DeployOpen plans and tunes the mClock scheduler on Ceph clusters that run in your own data centers or cloud accounts. We start from the workloads your applications actually run, then test each profile change against measured client latency and recovery time before it reaches production.

How we scope

We review your cluster's current mClock profile, Ceph release and OSD media before agreeing what to change.
The written scope names the client workloads, recovery targets and maintenance windows the tuning must respect.
Engagements are fixed-scope, with deliverables and timelines agreed in writing before any change is made.

How we build

We run OSD capacity benchmarks on representative drives so custom mClock settings start from measured values.
We test balanced, high_client_ops and high_recovery_ops profiles against a staging copy of your workload pattern.
We record client latency and recovery duration before and after each change so the results can be compared later.

How we hand over

You receive the final configuration, the profile rationale and the benchmark results used to justify each setting.
We document how to switch profiles during a drive failure, including which settings to change and which to leave alone.
Your team walks through the runbook and the rollback steps before the engagement closes.

How we support

Ongoing support, upgrades and re-testing after Ceph release changes are available as a retainer.
We can re-run benchmarks when you add OSDs, change drive models or move to a newer Ceph release.
Confirm with us during scoping which support response expectations the retainer would cover in writing.

Balancing recovery and client I/O with the Ceph mClock scheduler