Why the CRUSH tunables warning appears after an upgrade and when to act on it
After a cephadm upgrade, a cluster can report HEALTH_WARN crush map has non-optimal tunables. This is a planned decision, not a fault to clear quickly: moving to the optimal tunables profile shifts placement groups and moves data, and it changes which older clients the cluster will accept. Treat the warning as the start of a change window, not as an alert to silence. CRUSH tunables are settings in the CRUSH map that control how the algorithm chooses OSDs for each placement group and how it retries when a choice fails. The warning appears when the tunables in use are not at the optimal profile. Newer profiles can make OSDs and monitors reject clients that predate them, and the CRUSH documentation advises that old ceph-osd versions should not run once non-legacy tunables are in use.
The supported path is to inventory clients first, then change tunables inside a planned window. Before touching the map, list every consumer that talks to RADOS directly: kernel RBD and CephFS mounts, librados-based applications, hypervisors, and any gateway or tool with its own Ceph library. Record their kernel and library versions and compare them against the minimum requirements for the profile you intend to set, which the CRUSH map documentation describes. Confirm the requirements with Ceph's current documentation rather than relying on memory. The documentation also notes that moving to optimal may move up to about 10% of data, so schedule the change when that rebalancing can run without pressure on client I/O, and agree the rollback and client-upgrade sequence with the owners of each consumer before the window opens. Ceph CRUSH map documentation is the natural next read for profile names and their client requirements.
Which tunables changes affect clients and OSDs
This table compares each tunables-related change with its effect on OSDs, monitors and older clients, and the action a team should plan for. Read it before changing tunables, because the rows show where data moves and where clients can be cut off. The main takeaway is that the tunables change and the client upgrade belong in one plan, not two separate tasks.
| Change | Impact | Action needed |
|---|---|---|
| Newer tunables | OSDs and monitors reject older clients that do not support them, so those clients lose access to the cluster. | Inventory every kernel and librados client before switching, and upgrade or retire the ones that would be rejected. |
| Non-optimal tunables | The cluster reports HEALTH_WARN with the message crush map has non-optimal tunables, which signals that the placement settings are behind the current defaults. | Treat the warning as a planned change to schedule, not as noise to silence, and confirm the cause with the health detail output. |
| Non-legacy tunables in use | Old ceph-osd versions should not keep running once non-legacy tunables are active, so mixed-version OSD hosts become a risk. | Finish upgrading every OSD host before setting non-legacy tunables, and check the OSD versions across the cluster first. |
| Adjusting tunables to optimal | Placement groups shift to new OSDs, and moving to optimal may move up to about 10% of data (Ceph CRUSH map docs). | Schedule the change in a low-traffic window, watch recovery and client latency, and ask for the data movement estimate from a test cluster. |
Which older kernel and librados clients get cut off
Kernel clients map RBD images and mount CephFS inside the host kernel, so their supported feature set is fixed by the kernel build rather than by the Ceph packages installed on the host. When a tunables profile moves past what an older kernel client understands, the monitors and OSDs refuse the connection instead of degrading it, which shows up as hosts that can no longer map an image or mount a share after the change. To find a client's feature set, check the kernel release on each host and the feature information Ceph reports for connected sessions, then compare it with the requirements in the CRUSH map documentation for your release.
- Inventory every kernel mapper: list each host with
rbd mapor a CephFS kernel mount, not only the hypervisors you think of as Ceph clients. - Read the connected feature set: use the cluster's client feature reporting to see which feature bits each live session advertises.
- Watch for vendor kernels: distribution kernels often backport features, so the release number alone can mislead.
- Test a remap: map a test image on a representative host after the change in a staging cluster before production.
- Ask the storage owner: which hosts use kernel mappings, and whether any appliance or backup agent maps images on their behalf.
Confirm the exact minimum kernel versions for your release against the Ceph CRUSH map documentation before you schedule any change.
Applications that link librados or librbd use the client library version bundled with their runtime, container image or operating system package. These clients negotiate features with the cluster at connection time, so an application built against an older library can fail to connect or behave differently once non-legacy tunables are active. Failures often appear only in one service, batch job or container image, which makes them easy to miss in a cluster-level health check.
- List library consumers: search application images, language bindings and build manifests for librados and librbd dependencies.
- Record library versions: capture the library version inside each running container or host package, since the image tag may not match the bundled library.
- Reproduce the connection: run a small read and write test from each application image against a staging cluster with the target tunables.
- Plan rebuilds together: applications that cannot be upgraded may need a rebuilt image before the tunables change.
- Ask application owners: which services use direct librados or librbd access, who builds their images, and how long a rebuild takes in their release process.
Ask each owner for written confirmation that their service connected and completed I/O in staging with the planned profile.
The Ceph Object Gateway, RBD mirroring, CephFS MDS, NFS and other daemons run as Ceph components and speak the same protocol as the rest of the cluster. Because they are upgraded with the cluster, they are usually compatible when the cephadm upgrade completes, but their container images, sidecar tools and any external scripts that talk to the cluster directly may not be. Confirm that every daemon in the orchestrator list runs the target release before changing tunables, and that no external tool still uses an old library build.
- Check daemon versions: confirm each daemon type, including RGW, MDS and rbd-mirror, reports the target release after the upgrade.
- Review sidecar tools: backup agents, monitoring exporters and custom scripts often bundle their own librados build.
- Verify RGW clients separately: S3 and Swift clients talk HTTP to the gateway, so their compatibility depends on the gateway and their own SDK rather than on the RADOS tunables.
- Confirm daemon restarts: a daemon that was not restarted after an image change may still run the previous code.
- Ask the platform team: which external tools connect to RADOS directly, and who owns their packaging.
Record the daemon version list and the sidecar inventory in the change record before the tunables step starts.
Kubernetes clusters that consume Ceph through Rook use CSI drivers whose images and kernel mounts are managed by the operator, while hypervisors such as Proxmox VE or OpenStack Nova use librbd or kernel modules from the host or hypervisor image. These consumers sit in a different ownership model from the storage team, so a tunables change can break them even when the Ceph cluster looks healthy. Coordinate with the Kubernetes and virtualization owners early, because node pools and hypervisor hosts often have their own patch cycles.
- Map the CSI stack: note the Rook version, CSI driver image and whether volumes use kernel or user-space mounts.
- Check hypervisor library builds: each host's Ceph client packages and the hypervisor's own RBD driver can have different feature support.
- Drain and test by pool: move a test workload to a node or host running the target client build before changing tunables for the whole cluster.
- Plan node upgrades first: where a node pool or hypervisor is behind, schedule its upgrade ahead of the tunables change.
- Ask the virtualization and platform owners: which hosts run which client package versions, and what their maintenance window and rollback process look like.
Get written sign-off from each platform owner that their consumers were tested with the planned profile.
How to plan and apply the tunables change safely
This sequence moves a cluster from non-optimal CRUSH tunables to the chosen profile only after the cluster is healthy, every client can handle the change, and a tested rollback point exists. The output is a cluster on the new profile with placement groups back to a clean state and clients still connecting.
Confirm cluster health and daemon versions
The storage or platform lead checks that the cluster reports HEALTH_OK or documents any warning that remains, and confirms that every ceph-osd daemon runs the upgraded release. Use the version summary in the cluster tooling and check each OSD host, since a mixed set of OSD versions is the condition the tunables guidance warns against. Record the output so the team can show the starting state. The step is done when every OSD reports the target release and no unrelated health issue is open.
Inventory every client and its feature support
The platform team lists each consumer of the cluster: kernel RBD mappings, CephFS kernel mounts, librados applications, virtualization hosts, gateways and any custom code linked against older librados. For each one, record the kernel or library version and whether it supports the feature set the new profile requires. Ask each application owner to confirm their client version in writing, because a client that is not on the list is the one most likely to fail. The step is done when every client has a named owner and a supported or unsupported status.
Upgrade clients before changing tunables
Each client owner upgrades the clients flagged as too old before the cluster profile changes, starting with the clients that cannot be moved without a reboot or a maintenance window. Where a client cannot be upgraded, the team decides whether to keep the current profile or retire that workload first. Test one upgraded client per type to confirm it still maps volumes or mounts and reads and writes data. The step is done when every remaining client is on a version that supports the target profile, or the team has accepted the affected workloads in writing.
Export the current CRUSH map as a backup
Before any change, the storage administrator exports the compiled CRUSH map from the running cluster and stores a copy outside the cluster, with the date and the cluster identifier in the file name. Keep a second copy in the change record so the team can restore it quickly if needed. Confirm the file is readable by decompiling it on a separate host. The step is done when the saved map decompiles cleanly and is stored in at least two locations. Note that restoring this map later only reverts placement rules, not data already moved.
Test the change on a copy and estimate remapped data
The engineer takes the exported map, applies the target tunables offline, and compares the new placement against the current one to see which placement groups would move. Use this comparison to estimate how much data would be remapped, keeping in mind that the CRUSH map documentation says moving to optimal may move up to about 10% of data. Review the result with the capacity owner, because remapping consumes network and disk I/O while it runs. The step is done when the team has a written estimate of remapped data and has confirmed the cluster has enough free capacity to absorb the movement.
Set recovery and backfill throttles
Before applying the profile, the storage administrator lowers the recovery and backfill limits for OSDs so that data movement does not starve client I/O. Confirm the current values first, then set the new ones and record both. Check the limits on a sample OSD to confirm the change took effect. The exact values depend on the hardware and workload, so the team should choose them after reviewing the benchmark results they already have. The step is done when the throttle settings are recorded, applied and visible on the OSDs.
Apply the profile in a planned window
The storage administrator applies the chosen tunables profile during an agreed change window with the application owners present. Set the profile, then confirm the cluster reports the new profile and begins to remap placement groups. Watch for the health warning about non-optimal tunables clearing and for any client errors in the first minutes. The step is done when the cluster accepts the change and the change log records the time, profile name and operator.
Watch placement groups to completion
The operator monitors placement group states until all are active and clean, tracking recovery progress and client latency throughout. If client latency rises beyond the team's agreed threshold, the operator lowers throttles further rather than stopping the change mid-flight. Keep the progress view open and record checkpoints so the change owner can report status. The step is done when no placement groups remain in a degraded, undersized or remapped state and the cluster returns to its normal health report.
Verify client connectivity and record a rollback point
After the data movement finishes, each client owner runs a read and write test against their workload and confirms the mapping, mount or connection is working. Restore the throttle settings to their normal values and record the final CRUSH map and profile as the new baseline. Document the rollback point: restoring the saved map reverts placement rules only, so it does not undo data that has already moved, and it does not bring back clients that were cut off during the change. The step is done when every client owner signs off and the baseline is stored with the saved map.
What to prepare before changing tunables
The tunables change moves placement groups and cuts off older clients, so the work is mostly preparation. These six items connect the client inventory, the safety net, the rehearsal, the communication and the rollback decisions that should be settled before anyone runs the command.
Inventory every client
List each kernel RBD mapping, librados application, hypervisor and gateway that talks to the cluster, with its exact version. Older kernel clients are the usual reason the change is refused, so confirm which ones predate the tunables you plan to enable. Record the owner of each one so you know who must act.
Save and test a CRUSH map backup
Export the current CRUSH map before any change and store it outside the cluster, then confirm the file can be decompiled and read back. Keep the backup tied to the cluster state it describes so a restore is not guesswork. Treat this file as the reference for every later comparison.
Rehearse on a mirrored lab
Build a test cluster whose CRUSH hierarchy, device classes and pool rules match production, then apply the same tunables change there. Watch how many placement groups remap and how long recovery takes. Use the result to judge the production window, not to predict it exactly.
Plan the change window with client owners
Agree a window with the teams that own each client, and tell them which versions will stop connecting. Confirm that every client they run has an upgrade path before the cutoff date. Put the agreed contact list and rollback trigger in writing so both sides act on the same information.
Throttle recovery and watch placement groups
Set recovery and backfill limits before starting so client I/O keeps priority, and decide the thresholds that pause the change. Monitor placement group states, degraded and misplaced object counts, and slow requests during the move. Keep the health warning visible on a dashboard the operators watch throughout.
Write a rollback plan for moved data
Reverting the tunables does not undo data that has already moved, so the plan must cover the remapping that has happened. Define which state you return to, how long reverting would take, and who approves it. Test the revert path in the lab so the team knows its limits before production needs it.
What makes tunables planning take longer
This table lists the factors that stretch tunables planning and shows how each one changes the work. Read each row as a question to answer about your own cluster before estimating the window. The main takeaway is that client inventory and change control usually take longer than the CRUSH change itself.
| Driver | Why it adds effort | What to check |
|---|---|---|
| Number and diversity of clients | Each client type (kernel RBD, kernel CephFS, librados applications, ceph-fuse) has its own minimum version and upgrade path, so every type needs separate verification. | List every client type, its version and how it connects to the cluster. |
| Old kernels still in service | Kernel clients are updated through the operating system, often on a slower schedule than Ceph, so old kernels may be the hardest to move. | Identify hosts running kernels older than the tunables profile needs and confirm who owns each one. |
| Cluster fullness | Moving to optimal tunables shifts placement groups, so a cluster with little free space has less room to absorb the movement. | Check current utilization and headroom before scheduling the change, and confirm recovery settings with your operations team. |
| CRUSH hierarchy complexity | Deep or uneven hierarchies, custom rules and device classes make the expected data movement harder to predict. | Export the CRUSH map and review rules, buckets and device classes against the planned profile. |
| Number of failure domains and pools | Each pool may depend on different rules and replica or erasure coding settings, so the impact must be reviewed per pool and per failure domain. | Count pools and map each one to its CRUSH rule and failure domain. |
| Third-party managed clients | Hypervisors, appliances or vendor-managed applications may need a vendor to approve or deliver a client upgrade, which adds lead time outside your control. | Ask each vendor for a supported client version and its upgrade schedule in writing. |
| Change-control requirements | Formal approval, maintenance windows, rollback plans and evidence of testing add review cycles and stakeholder time. | Confirm which approvals apply, the required change record content and how a client outage would be handled. |
Questions teams ask before changing CRUSH tunables
How much effort and cost does a CRUSH tunables change add to an upgrade?
The direct cost is the engineering time to assess the cluster, plan the client cutover, run the change and verify the result. The main drivers are how many clients fall below the new tunables level, how many teams must upgrade those clients, and how much change control your organization requires. Cost is mostly one-time, because the tunables change itself is a single operation on the CRUSH map. Recurring cost appears only if you later need to keep older clients running or repeat the work for new hardware. Ask any partner to price the client inventory, the coordination with client owners, the maintenance windows and the post-change checks as separate line items in the written scope. Confirm the exact tunables profile and its client requirements in the Ceph CRUSH map documentation for your release before you size the work.
How long does changing CRUSH tunables take, and what sets the duration?
The duration depends mainly on how much data the new tunables move and how fast your cluster can rebalance without hurting client I/O. The Ceph CRUSH map documentation notes that adjusting tunables shifts placement groups, and that moving to optimal may move up to about 10% of data. Other drivers are the number of OSDs and failure domains, the recovery throttling you set, the size of the cluster's stored data, and the time needed to upgrade clients first. Plan for the data movement to run as a background process that you monitor until the cluster returns to a healthy state. Ask for a timeline that separates client upgrade dates from the rebalance window, and confirm the throttling settings with your storage team before the change window opens.
What does the change need from our storage and client teams?
The storage team owns the cluster side. It must check cluster health, confirm that all ceph-osd daemons are at a version that supports the target tunables, review the placement impact, set recovery limits and watch the rebalance. The client teams own the older kernel and librados consumers. Newer tunables make OSDs and monitors reject older clients, so every hypervisor, kernel mount, and application using librados must be inventoried and upgraded or scheduled before the change. Agree on a go or no-go checkpoint in which both teams confirm the client list is clear. Document which hosts and services are affected so that a rejected client is found quickly rather than discovered by users.
Can our own team change CRUSH tunables without a partner?
Yes, if your team already operates the cluster, understands CRUSH rules and has a tested client inventory. The change is a single cluster-level operation, but the risk sits in the surrounding work: finding every older client, sequencing upgrades, controlling data movement and proving the cluster is healthy afterward. Teams that lack recent Ceph operations experience, or that run many hypervisor and kernel client types, often benefit from a fixed-scope review of the plan. DeployOpen can plan, document and support this kind of change in infrastructure you control, with the scope and deliverables agreed in writing before work starts. Whichever route you choose, rehearse the sequence in a non-production cluster first.
Can we keep non-optimal tunables instead of changing them?
Usually yes, and this is a valid choice when older clients must keep working. A non-optimal tunables setting raises the HEALTH_WARN status crush map has non-optimal tunables, so the cluster will report the warning until you change the profile or accept it. The trade-off is that you keep compatibility with older clients but leave the cluster on a less current placement behavior and keep the warning in your monitoring. Record the decision, the affected clients and the date you will revisit it. Confirm with the Ceph documentation for your release that the warning can be safely suppressed or tracked in your monitoring tool before you rely on that approach.
How do we roll back a CRUSH tunables change if clients fail?
A tunables change is not simply reversed by a single command in the middle of a rebalance, so the rollback plan must be written before the change. The safe approach is to hold the change until every client is confirmed compatible, to keep a saved copy of the current CRUSH map, and to define the conditions under which you would stop data movement and restore the previous profile. Test the restore path in a non-production cluster. Also plan for the case where the rollback itself moves data again. Confirm the exact steps for your release in the Ceph CRUSH map documentation and agree the rollback criteria with your change board.
How DeployOpen scopes and delivers the tunables change
DeployOpen plans and implements the CRUSH tunables change inside your own infrastructure, whether that is your cloud accounts, data centers or an air-gapped network. The work starts with a written, fixed scope agreed before any cluster is touched. That scope normally covers four parts: a client inventory that lists every consumer of the cluster with its kernel or librados version, a CRUSH test that estimates data movement against a copy of your current map, a change window agreed with the teams that own the workloads, and verification after the change. Most engagements are fixed-scope, so the deliverables and timelines are written down before work begins.
The client inventory is the input that most often decides the plan. Ask your team to confirm which kernel RBD clients, CephFS mounts, librados applications and older ceph-osd daemons are still in use, and who owns each one. DeployOpen uses that list to decide whether the client upgrades must finish before the tunables change, and to write the rollback and verification steps into the scope. Confirm the exact client versions your environment needs with the Ceph CRUSH map documentation at Ceph CRUSH maps and your own testing, rather than relying on a general rule.
Ongoing support after the change is available as a retainer, which can cover monitoring of the health warning, follow-up client upgrades and later cluster upgrades. DeployOpen does not publish a price list. A quote follows from the written scope, so the inventory, the number of clusters and the number of client teams involved are the details to prepare before requesting one. Ask for the quote to show one-time work and any recurring retainer separately.
