Ceph OSD balancing with upmap and read balancer

Why uneven OSD use happens and when to run the Ceph balancer

Run the Ceph balancer when one OSD climbs toward its full ratio while others sit far below it. The cluster can only accept writes as fast as its fullest OSD allows, so the spare capacity on the emptier drives is effectively lost. Uneven use usually comes from CRUSH placing placement groups (PGs) by weight and hash, from drives of different sizes or weights, from a host added or replaced, or from pools with uneven PG counts. Before you change anything, run ceph osd df tree to see per-OSD utilization and the spread between the fullest and emptiest OSDs. If the whole cluster is close to full, rebalancing will not create the capacity you need, and you should plan to add drives instead.

The upmap balancer, run by the manager's balancer module in upmap mode, adds per-PG upmap exceptions on top of CRUSH. Each exception tells one PG to place a replica on a specific OSD instead of the one CRUSH would choose. Because the changes are per PG, the balancer can correct distribution by moving a limited set of PGs, without reweighting whole OSDs. Two settings control how aggressive a run is: the misplaced ratio limit, which caps how much data may be in motion at once, and the deviation target, which sets how close each OSD must come to its fair share of PGs. Confirm the defaults and the minimum client release for upmap in the Ceph documentation for your version, and check the balancer state with ceph balancer status before and after a run.

How upmap and read balancing differ

Ceph places data with CRUSH, then offers two ways to correct uneven use: upmap exceptions that move data at rest, and the read balancer that changes which copy serves reads. The cards below separate the placement rules, the module that computes changes, the modes it runs in, and the read-side option, so you can tell which one addresses your imbalance before you change anything.

CRUSH placement rules

CRUSH computes where each placement group (PG) lives from the cluster map, failure domains and pool rules, without a central lookup table. It is the baseline that every balancing change starts from. Before you touch balancing, check the CRUSH hierarchy, device weights and failure domains, because uneven weights or mismatched host sizes cause imbalance that upmap cannot fully fix. Confirm the weights in your own cluster with the Ceph documentation for your release.

Upmap exceptions in the OSD map

Upmap stores explicit exceptions that remap individual PGs from the OSDs CRUSH chose to other OSDs. The change is recorded in the OSD map, so it persists across restarts and is visible to every client, and the data then moves through normal backfill. Upmap changes what the cluster stores on disk, but it does not change CRUSH rules themselves. Review the exception count and the resulting data movement before accepting a large batch.

The balancer manager module

The balancer is a manager module that computes upmap changes or CRUSH weight adjustments and applies them. It runs in a mode you select, and it works toward a deviation target that you can set. Check which mode is active and what the module plans to do before letting it run unattended, since it decides which PGs move and when.

Balancer modes and limits

The module supports modes such as upmap and crush-compat, and its misplaced ratio limit caps how much data may be misplaced at once during a balancing pass. The limit controls recovery load on client I/O, so set it with your workload in mind. Confirm the mode names and defaults for your release in the Ceph documentation rather than assuming them from older guides.

Read balancer for primary PGs

The read balancer changes which OSD serves as primary for each PG, so read load spreads across OSDs that hold replicas. It does not move stored data, so it adds no backfill for the data itself. Clients must be on Reef or later for the read balancing behavior to apply, so check every client version before relying on it, including older libraries embedded in hypervisors.

Balancer settings and when each fits

This table compares the four choices that control how the balancer moves data: mode, misplaced ratio limit, deviation target and read balancing. Read each row as a decision: keep the default on a stable cluster and change a setting only when measured imbalance or a client I/O constraint points to it. Tighter limits protect client traffic but slow convergence, so test values in a quiet window first.

Setting	Keep the default when	Change it when and what it trades off
Balancer mode (`upmap`)	Clients support upmap and the cluster is stable with a steady PG count per OSD.	Use `crush-compat` only if older clients block upmap. It adjusts weights, is less precise and can leave more variance between OSDs.
Misplaced ratio limit (`target_max_misplaced_ratio`)	Client I/O is moderate and no tight recovery deadline applies.	Lower it when client latency suffers during balancing. Raise it when a new capacity addition must settle quickly. Lower values extend balancing time, higher values increase movement load.
Deviation target (`upmap_max_deviation`)	Usage across OSDs is already within the spread your capacity planning accepts.	Tighten it for near-full pools or uneven drive sizes where the fullest OSD sets usable capacity. Loosen it to cut PG churn when the gain is small. Tighter targets mean more moves.
Read balancer (`upmap-read` mode)	Read load is spread well, or you have not confirmed that all clients meet the Reef or later requirement.	Enable it when primary PGs concentrate reads on a few OSDs. It changes primary placement, so expect extra peering and watch read latency during the pass.
Automatic balancing (on or off)	The cluster runs steady with routine capacity changes only.	Pause it during recovery, drive replacement or maintenance windows, because balancing and recovery compete for the same network and disk capacity.

Enabling and verifying the balancer on a live cluster

This sequence moves the balancer from a planned change to a verified result: it confirms that clients can accept upmap, previews the moves before any data shifts, applies them within a misplaced limit, and checks that utilization narrows without health regressions. Each step ends with a check you can record, so you can stop or roll back at any point.

Confirm client compatibility before enabling upmap
Before touching the balancer, list every connected client and its feature release using the cluster's session and feature reporting, and confirm that the minimum client release you plan to require matches what those clients run. Upmap entries are stored in the OSD map, so a client that cannot understand them can block the change. Ask your storage lead to confirm the minimum release for your Ceph version in the upmap documentation, and record the result in your change ticket. The step is done when every client reports a release at or above the requirement, or when you have a documented plan for the exceptions.
Confirm the balancer module is available
Check that the balancer manager module is enabled and reports its status through the orchestrator, since cephadm runs Ceph daemons as containers and module state is part of the cluster configuration. Run the balancer status query and confirm that it reports the mode currently set and that no plan is already queued from an earlier attempt. If the module is disabled, enable it through the manager module commands and recheck the status. The step is done when the status output shows the module active and the cluster is in the health state you recorded before the change.
Set the balancer mode to upmap
Set the balancer mode to upmap, which moves individual placement groups with explicit mappings rather than reweighting OSDs, and set the minimum client release as the cluster requires. Keep the balancer turned off for automatic runs until the plan has been reviewed, so no changes start on their own. Record the previous mode and the setting values so you can restore them exactly. The step is done when the status query reports the upmap mode and the automatic run remains off.
Generate and review a dry-run plan
Create a plan without executing it, then score it to see the projected change in OSD utilization spread. Read the list of proposed PG moves and check that they touch the pools and failure domains you expect, and that no move breaks the CRUSH rules for a pool with a stricter placement requirement. Compare the projected score against the baseline you captured earlier. The step is done when the plan is saved, its projected result is recorded, and the storage owner has approved the scope of the moves.
Apply the plan within the misplaced limit
Execute the approved plan while the misplaced ratio limit is set to the value your team agreed for this cluster, so the balancer does not move more data than the cluster can absorb alongside client traffic. Watch recovery and backfill progress and the client latency graphs during the run. Ask your storage team to confirm the limit value for your release in the balancer documentation. The step is done when the plan has completed, backfill has returned to idle, and no client-facing alerts were raised during the window.
Verify utilization spread and cluster health
Check per-OSD utilization against the baseline and confirm that the spread between the fullest and emptiest OSDs has narrowed to the deviation target you set. Run the cluster health check and confirm there are no new warnings, misplaced objects are zero, and placement groups are active and clean. Review the balancer status once more to confirm the score improved. The step is done when the spread meets the target and health matches or beats the pre-change state.
Document the result and keep a rollback path
Record the before and after utilization, the plan that was applied, the client release report, and the settings used. Keep the saved previous mode and settings and the upmap exception entries so they can be reverted. If health degrades or a client fails, stop the balancer, clear the upmap entries for the affected placement groups, and restore the previous mode, then rerun the health check. The step is done when the change record is complete and the rollback procedure has been reviewed by the operations team.

Measuring spread before and after balancing

Record the per-OSD utilization and PG count for every OSD before you change anything, because a balancer result can only be judged against a snapshot taken on the same cluster state. Capture the output of ceph osd df with its utilization and variance columns, and save the PG distribution per OSD alongside it. Note the balancer mode, the upmap_max_deviation value and any target_max_misplaced_ratio you have set, so the run can be reproduced or compared later. Store these files with the benchmark baseline you captured before production, since that baseline tells you what the drives and network can sustain when data moves. If you have no baseline, stop here and build one first, because without it you cannot tell whether slow requests come from the balancer or from hardware that was already near its limit. A good result is relative, not a fixed number. Success means the spread between the fullest and emptiest OSDs in each device class has narrowed to the deviation target you chose, the PG counts per OSD match their CRUSH weights within that same tolerance, and no OSD has moved into a near-full state during the run. Compare the before and after files for each failure domain, not only the cluster average, because an imbalance inside one rack or host often hides behind a healthy total. Confirm the current figures for balancer behavior and deviation handling in the Ceph balancer documentation for your release, since the defaults and option names are the ones to check against your own version. Watch the cluster while data moves, not only at the end. Track ceph status for recovery and backfill progress, and watch the slow request warnings and client latency on your monitoring dashboards. If slow requests climb, lower the misplaced ratio before you stop the balancer, since in-flight movement is what competes with client I/O. Recovery pressure is governed by the scheduler and its profile, which is covered in the mClock section, so adjust there rather than stopping mid-move. Pause and reassess if any OSD approaches its full ratio, or if backfill stalls for a long period. Once the run finishes, repeat the capture, compare it with the saved baseline, and keep both files with the change record.

Balancing choices for HDD, SSD and mixed clusters

HDD-based OSDs with WAL and DB on shared SSDs are the most common mixed layout in capacity clusters. The data sits on spinning media, so each PG move takes longer and competes with client reads, which makes the misplaced ratio the first setting to review. The shared SSD also links several OSDs together, so a single device failure can change the balance of many OSDs at once.

Misplaced ratio first: Lower the target misplaced ratio on HDD pools so backfill does not starve client reads. Confirm the current value with ceph config get mgr target_max_misplaced_ratio before you change it.
Deviation target: Keep the upmap deviation target realistic for the PG count per OSD, because tight targets on HDD clusters create long backfill queues.
DB sharing: Group OSDs by the SSD that holds their DB and note the groups, since losing a shared device affects all of them together.
Device class: Confirm the hdd class with ceph osd tree so the balancer compares only like with like.

Check the misplaced ratio and backfill rate before you touch the deviation target, and confirm the layout against the hardware recommendations.

On all-flash clusters, data movement completes quickly, so the balancer can run more often with less disruption. The limiting factors shift to manager CPU during plan computation, to uniform drive capacity within a class, and to the number of PGs each pool holds. Imbalance here usually comes from layout decisions rather than from slow movement.

Shorter backfill: The misplaced ratio matters mainly for client latency during a run, so review it alongside latency graphs rather than as the only brake.
Drive capacity mix: Drives of different sizes inside one class skew CRUSH weights. Check weights with ceph osd df before tuning the balancer.
Pool PG counts: Pools with too few PGs cannot balance evenly, whatever the upmap settings. Review pg_num with the autoscaler output first.
Read balancing: On Reef and later clients, review read balancing for primary PGs, since reads often dominate latency on flash.
Manager load: Watch manager CPU while plans are computed and schedule large runs in quiet windows.

Compare ceph osd df output before and after each run, and confirm the minimum client release the cluster requires before you enable read balancing.

Pools that map to OSDs of different classes or sizes produce the most surprising balancing results. The CRUSH rule decides which OSDs a pool can use, so the balancer can only move data within the set of OSDs that rule allows. Review the rule and device classes before adjusting any balancer setting, because a correct balancer cannot fix a wrong placement rule.

Class-specific rules: Confirm each pool's CRUSH rule targets the intended device class with ceph osd crush rule ls and ceph osd crush rule dump.
Weight accuracy: Mixed capacities in one class need correct CRUSH weights. The balancer cannot correct a weight error.
Spanning pools: A pool that spans two classes holds the smaller class to its limits, so balance each class separately where the layout allows.
Client compatibility: Confirm the minimum client release before you enable upmap or read balancing on a fleet with mixed client versions.
Offline preview: Use osdmaptool on an exported osdmap to preview plans before making live changes.

Map each pool to its rule and class first, then change balancer settings only for the pools whose layout you have confirmed.

Operational risks when rebalancing a production cluster

Balancing moves placement groups between OSDs, so it competes with client traffic, recovery and any other change already in flight. The cards below cover the risks that most often turn a routine balancing run into an incident, and the controls that keep each one bounded.

Data movement saturating network links

Every PG the balancer remaps is copied across the cluster network, and a run can move far more data than the balance it achieves. Check the cluster and public network utilization before starting, and confirm that bonded links across two switches are healthy so one failed link does not concentrate the load. Keep the balancer's misplaced ratio limit low enough that backfill traffic leaves headroom for client I/O. Record the link throughput observed during the first batch so later runs can be compared against it.

Misplaced ratio set too high

The misplaced ratio caps how much of the data may be in a misplaced state at once while the balancer works. A high value lets the balancer queue a large amount of movement in one pass, which is hard to pause safely and hard to explain to application owners. Start with a conservative value on a pool you understand, then raise it only after watching a full cycle. Write the chosen value into the change record so the next operator knows which limit was in force.

Client latency during balancing

Backfill competes with client reads and writes on the same OSD queues, so latency for virtual machines and databases can rise even when the cluster stays healthy. Watch slow request counts and client latency on the pools that carry the most sensitive workloads, not only cluster-wide averages. If latency climbs, pause the balancer rather than tuning recovery priorities mid-run. Agree in advance with application owners which latency thresholds will trigger a pause.

Old clients blocking upmap

Upmap entries require every connected client to understand the upmap feature, and a client that is too old can prevent the balancer from applying changes. Check the minimum client release the cluster reports before enabling upmap mode, and identify any older kernel clients or legacy hosts still mounting storage. Confirm the actual client population with the cluster's own feature reporting rather than from inventory alone. Plan the cut-over with the owners of those clients so the requirement is not discovered during a maintenance window.

Changes made during unrelated recovery

Balancing started while a drive is failed or PGs are recovering adds a second wave of movement on top of repair traffic, and the two can be hard to tell apart in graphs. Check cluster health before every run and defer balancing until recovery has finished unless the situation is an emergency. Record which recovery events were active when balancing began, so an unexplained slowdown can be traced back to its cause. Keep one change owner per cluster so balancer settings are not altered by two people at once.

Missing change records and monitoring

Balancer runs change placement across many OSDs, and without a record it is difficult to tell whether a later capacity or latency problem came from the balancer. Keep a change record for each run that names the mode, the deviation target, the misplaced ratio limit, the window and the person approving it. Monitor the balancer status and the health output throughout, and store the before and after OSD utilization spread. Ask your partner or internal team to confirm how these records will be retained and reviewed.

Questions teams ask before balancing OSDs

Can our own team balance OSDs without a partner?

Yes, a team that already runs its Ceph cluster can usually balance OSDs in-house, provided it can read cluster health output, plan change windows and watch client latency while data moves. The work is mostly careful observation: check the utilization spread per OSD, review the balancer status, adjust the limits, and verify the result over time. The harder part is judgment about how much movement the cluster can absorb during business hours, which depends on your workload. Teams with no one who has run recovery or rebalancing before often benefit from a short review of the plan before the first change.

What makes balancing effort larger or smaller?

Three factors drive most of the effort:

Cluster size and OSD count, since more OSDs and pools mean more placement groups to review and more variation to explain.
Client mix, because block, file and object workloads with different latency needs make it harder to pick safe movement limits.
Change control, since approvals, maintenance windows and rollback steps add review time even when the technical change is small.

A small, single-pool cluster with a clear change process is far less work than a multi-pool cluster shared by several applications.

Does balancing Ceph data require downtime?

Balancing should not require downtime, because the upmap and read balancers move placement gradually while the cluster keeps serving clients. Confirm this in your own environment before you start, since client impact depends on hardware, network capacity and how aggressive your limits are. Check cluster health and client latency during a trial run, and confirm with the Ceph documentation for your release that the balancer behaves as you expect. Plan for a period of extra background I/O, and treat any unexpected client slowdown as a reason to pause and adjust.

How long does balancing take as data volume grows?

Duration scales with the amount of data that must move, the number of OSDs involved and the movement limits you set, so no fixed timeline applies to every cluster. More stored data and a wider spread between the fullest and emptiest OSDs generally mean more movement and a longer run. Ask any partner to base the schedule on a measured baseline from your cluster, including current utilization spread, the target deviation and the allowed misplaced ratio. Treat a timeline that does not reference your data volume as a guess.

What should we ask a partner for in a fixed-scope quote?

Ask for a written scope that names the cluster, Ceph release, pools to be balanced and the balancer mode to be used. The quote should list the baseline measurements taken before work begins, the limits for misplaced ratio and deviation, the verification steps and the rollback approach. Confirm who runs the change, who monitors client performance, what evidence is delivered at the end and whether follow-up checks are included. Ask which items are one-time work and which would recur as a retainer, so you can compare the offer with your own team's time.

When should we bring in outside help for OSD balancing?

Consider outside help when the cluster carries production workloads that cannot tolerate slow requests, when recovery or rebalancing has stalled before, or when your team has no recent experience with the upmap or read balancers. Outside help also fits when a compliance review requires documented change records for storage operations. DeployOpen plans, implements, hardens and documents self-hosted platforms in infrastructure you control, so a fixed-scope engagement can cover the balancing plan, the change and the evidence. Your own team stays responsible for approvals and for day-to-day operation.

How do we check that balancing worked?

Compare the OSD utilization spread before and after the change, and confirm the result is within the deviation target you set. Watch cluster health, misplaced and degraded object counts, and client latency across the whole run, not only at the end. Record the balancer settings so the result can be reproduced or reversed. Ceph's cluster status commands and the Ceph documentation for your release are the starting point for these checks, and you should confirm the exact commands for your version before the window opens.

How DeployOpen scopes and delivers OSD balancing work

DeployOpen approaches OSD balancing as a scoped change to a live cluster: first we confirm the current distribution, the client and release constraints and the operating window, then we agree the balancer settings in writing. Engagements are fixed-scope, so the deliverables, the checks and the handover are defined before any change is made to your cluster.

How we scope

Written scope lists the clusters, pools, client release floor and balancer mode covered by the work.
We record current OSD utilization spread, PG counts per pool and any active recovery or backfill before proposing changes.
The scope states the misplaced ratio limit, deviation target and change window that you approve for the rebalance.

How we build

We confirm minimum client compatibility, then enable upmap mode and check that the balancer runs against your target settings.
We test read balancing for primary PGs only where your clients meet the Reef or later requirement you confirm.
We run the balancer in controlled steps and watch recovery load, client latency and OSD fullness throughout each step.

How we hand over

You receive a runbook with the commands used, the settings applied, the before and after utilization figures and the rollback steps.
Documentation explains how to read balancer status and when to re-run the balancer after drive or host changes.
We walk your operations team through the checks they need to repeat after expansions, failures or upgrades.

How we support

Ongoing support, upgrades and periodic balance reviews are available as a retainer after the fixed-scope work closes.
Retainer scope, response expectations and review frequency are agreed in writing, and quoted separately from the project.
Ask us to confirm which balancer behaviors apply to your Ceph release before you schedule a rebalance in production.

Balancing data across OSDs with the upmap and read balancers