Ceph RBD QoS limits for noisy neighbors

How Ceph RBD QoS limits stop one busy disk from starving the rest

Per-image QoS limits in librbd cap IOPS and throughput for a single RBD image, but only for clients that go through librbd, such as QEMU disks opened with librbd. In a shared cluster, the symptom is usually latency that climbs for every neighbor when one virtual machine runs a bulk copy, a database rebuild or a scan that keeps the OSDs saturated. Limits work at the client, so they cap the demand one workload places on shared OSDs. They do not add capacity, and they do not change how the cluster handles the requests it receives.

A limit is a ceiling only. It stops one image from exceeding its allowance, but it gives the other images no minimum IOPS and no latency target. If the cluster is undersized, every image can still be slow while each one stays within its limit. Recovery, scrubbing and rebalancing run inside the cluster, so image limits do not throttle them. Confirm the behavior of your release in the Ceph documentation before you rely on limits to protect latency during maintenance windows.

Set limits before buying hardware when one workload is unpredictable and your monitoring shows which images cause the spikes. Use your own benchmarks to decide which images need caps and what latency your other workloads tolerate. Confirm which clients use librbd, because a kernel-mapped image does not pick up these limits. Assign one team to own the settings, and review them when workloads change, since a limit only helps if it applies to every image that can cause the problem. Treat limits as a stopgap while you measure demand.

Which limit types fit which workload

This table compares combined IOPS and throughput caps, per-direction read and write caps, and burst allowances for Ceph RBD images. Read each row for what the limit caps, where it fits, and the failure mode to test for before you rely on it. Confirm exact option names and units against the documentation for your Ceph release before writing any configuration.

Limit type	What it caps and where it fits	Main risk
Combined IOPS and throughput	Caps total reads and writes for an image together. Fits a general-purpose VM disk where the goal is a fair share of the cluster rather than a fixed write profile.	A cap on combined I/O can hide a write-heavy neighbor, because a noisy image that mostly writes can consume the budget while reads look normal.
Per-direction read and write limits	Caps reads and writes separately. Fits workloads with a known shape, such as a log or database volume that must not flood the cluster with writes during backups or compaction.	Setting only one direction leaves the other unbounded, and a limit set too low on writes can slow commit paths enough to cause application timeouts.
Burst allowance	Lets an image exceed its steady limit for a short window before the base limit applies. Fits boot storms, package installs and other short spikes that are normal for the workload.	A burst that hides a sustained overrun. If the burst window refills too quickly, a neighbor that runs continuously can look compliant in short tests.
Steady limit without burst	Holds the image to a fixed ceiling at all times. Fits shared or multi-tenant pools where predictable behavior matters more than peak speed.	Users may see slow periods during legitimate peaks, so the ceiling needs measurement under real load before it is set.
Limits set on the client side	Applied by librbd in the client process, so the limit follows the image on that client. Fits QEMU or librbd-based attachments where the client path is controlled.	Clients that do not use librbd, such as the kernel RBD driver, do not enforce these limits, so a mixed client fleet can leave gaps.
Limits with no written evidence	Settings that exist on paper but were never tested under a neighbor workload. Fits nothing until a benchmark confirms the effect.	Teams assume protection that does not exist. Confirm the behavior with a controlled test that runs a noisy writer beside a latency-sensitive reader.

Choosing limits for multi-tenant, internal and database volumes

Tenants share one pool and cannot see each other's load, so limits protect the platform and also define the service levels you sell. Start from the plan you sell: map each tier to a fixed QoS profile and apply that profile to every image in the tier at provisioning, so limits are set by the catalog and not edited by hand per ticket.

Starting approach: define a small set of tiers, each with one combined and per-direction profile, and treat any change to a tier as a versioned platform change.
What to measure first: per-image IOPS, throughput and read/write mix for each tenant during busy hours, so the caps reflect real demand before they are published.
Signs the cap is too tight: a tenant's application latency rises while cluster capacity is idle, and that tenant's usage sits flat against its ceiling for long stretches.
Signs the cap is too loose: one tenant's backup or batch job slows neighbors on the same hosts, and cluster-wide latency tracks that tenant's activity.
Contract alignment: confirm that each tier's promised behavior matches the profile configured in Ceph, and retest after every tier change.

Before publishing tiers, check that every profile maps to a documented promise and that support staff can see which profile each image carries.

How to set RBD QoS limits on an image step by step

This sequence moves from confirming which clients honor librbd limits to capping one test image, verifying the cap from inside a guest, and rolling the settings out to the rest of the image set. Each step ends with a check so the team can show that the noisy neighbor is contained without affecting the other images.

Confirm the client path and release support
Start by listing how each image is consumed. Disks opened through librbd, such as QEMU with the librbd driver, can use Ceph's client-side QoS settings. Images mapped through the kernel RBD client cannot, so flag them separately. Then check the Ceph release documentation for your running version to confirm the exact configuration keys, their units and their burst options. The platform engineer owns this step. It is done when the inventory records the client path for every image and the keys are confirmed against your release.
Baseline image and pool performance
Measure each image that may be noisy under its normal production load before setting any cap. Record IOPS, throughput and latency for reads and writes, and note peaks as well as typical values. Also check pool-level utilization and recovery activity, because a cap that looks correct on a quiet cluster can starve clients during rebuilds. The storage engineer runs the measurements in a window that reflects real workload patterns. The step is complete when a baseline table exists for every candidate image and the pool headroom is written down.
Choose combined or per-direction limits
Use the baseline to decide whether one combined limit fits or whether reads and writes need separate caps. A workload that is heavy on writes during batch jobs may need a write-specific limit while reads stay free. Decide whether burst allowances are needed for short spikes, such as log rotation or backup starts, and set them only where the baseline shows them. Record the chosen IOPS and byte-rate values with their units. The step is done when the settings are written into a change record that a second engineer has reviewed.
Apply the limit to one test image
Choose a non-production test image and set the chosen keys with rbd config image set against that image only. Read the values back with rbd config image get or list them with rbd config image list to confirm the stored values match the change record. Check in the release documentation whether the running client must reopen the image before the new limit takes effect, and plan a guest or VM restart if it does. The step is done when the stored configuration reads back correctly and the client behavior is understood.
Verify the cap from inside the guest
Run the same workload profile from inside the guest that uses the test image, with a load generator such as fio, and set the tolerance you will accept before the test starts. Compare the achieved IOPS and throughput against the configured cap. Confirm that the sustained rate stays at or under the cap and that burst behavior matches the settings. Also run a quiet neighbor image on the same pool to confirm it is not affected. The step is done when the results are recorded side by side with the cap and the tolerance.
Confirm kernel-mapped images are not limited
For any image mapped with the kernel RBD client, map a test copy and run the same load to confirm whether the librbd limit is absent. If the throughput exceeds the intended cap, the kernel path needs another control. Options include Linux cgroup I/O limits applied to the block device on the host. Test that control with the same workload before relying on it, and confirm its behavior with your platform vendor or the Linux documentation for your kernel. The step is done when each kernel client has a documented, tested control.
Roll out and monitor the full image set
Apply the approved settings to the remaining images in batches, recording each change with the image name, the keys and the rollback command that removes them. After each batch, watch client latency, Ceph health output and the noisy image's achieved rate against its cap. Stop the rollout if a neighboring image shows regression. The operations lead signs off when every image in the inventory has a recorded setting, the monitoring shows no unexpected client regression, and the kernel-mapped exceptions are listed with their controls.

Measuring noisy neighbors before you cap anything

A QoS cap set from a guess fails in one of two ways. Set too low, it throttles a database or build job during its normal peak and shows up as application latency that nobody can trace back to storage. Set too high, it protects nothing, and the neighbor keeps degrading shared OSDs. Before you choose any limit, measure each image's demand and the latency it currently receives at both ends: the client (librbd, the kernel RBD driver or the hypervisor's view of the disk) and the cluster (per-pool and per-OSD statistics from the Ceph Dashboard or the ceph CLI). Capture IOPS, throughput and latency separately for reads and writes, over at least one full business cycle, including the nightly batch window. Averages hide the bursts a cap would clip, so keep the percentiles and the peak values as well.

Next, separate client I/O from background work. Recovery after a drive failure, backfill after adding capacity and scrubbing all consume the same OSD queues as your workloads, and they often start at the same hour as a backup job. If a latency spike lines up with recovery or scrub activity rather than with one image's throughput, a per-image cap will not fix it, and the cap may hide the real cause. Check the recovery and scrub settings for your release and confirm with your operations team how they are scheduled, since these controls differ between releases and should be verified against the documentation for the version you run. The Ceph hardware recommendations also advise sizing memory for peak use rather than calm periods, which matters here because a cluster that is already memory-starved during recovery will show latency that no QoS setting can repair.

Agree on the target latency for each workload class before you pick a cap. A payment database, a development VM and a log archive do not need the same numbers, and the owner of each workload should sign off on an acceptable latency band. Then benchmark the cluster under representative load, ideally before purchase or before a hardware expansion, because the hardware guidance notes that no two clusters are alike. The result is a documented baseline: demand per image, latency under load, and the headroom left once recovery runs. Only then does it make sense to set limits, and the baseline tells you whether a cap is the right tool or whether the cluster simply needs more capacity.

Trade-offs and failure modes of per-image caps

Per-image limits protect neighbors only when they match real demand, the cluster's current capacity and the client type in use. The cards below describe the failure modes that appear most often, the symptom each one produces and the check that exposes it before a tenant does.

Caps set below real peaks

A limit sized from average load will hold back legitimate bursts from databases, log writers and batch jobs. The symptom is higher client latency during known busy periods while the cluster itself looks healthy. Compare each image's observed peak demand over a representative period with its configured IOPS and throughput ceilings, and confirm the latency curve against the limit before tightening further.

Bursts that hide sustained overload

Burst allowances let an image run above its base rate for a short window, so a short benchmark can pass while a steady workload still starves its neighbors. Watch for images that repeatedly drain their burst budget and then sit at the base rate. Check how long each burst lasts under sustained load, and test with a run longer than the burst window.

Many images, one tenant

Limits apply per image, so a single tenant or application that creates dozens of images can still saturate the pool with the sum of their allowances. The symptom is a pool-wide latency rise that no single image's metrics explain. Aggregate usage by owner or project, not only by image, and compare that total with what the pool can deliver.

Recovery and rebalancing traffic

Client limits do not stop OSD recovery, backfill or scrubbing from competing for the same drives and network links. During a drive replacement or rebalance, a capped image can still slow down while the cluster repairs itself. Check recovery and client I/O together during a planned rebuild in a test window, and confirm the cluster's recovery throttling settings with your Ceph administrators.

Kernel clients ignore librbd caps

Limits enforced in librbd apply only to clients that use librbd, such as QEMU disks. Images mapped through the kernel RBD driver do not pass through those controls, so a limit that looks correct on paper may do nothing for that workload. Inventory how each client attaches to its images and verify the effective limit from the client side, not only from the configuration.

Limits left after changes

A limit chosen for last year's drives, network or workload can become wrong after an expansion, a media change or a new application moves onto the pool. Stale limits either throttle capacity that now exists or allow overload the cluster can no longer absorb. Schedule a review after each hardware change, after major workload onboarding and after each Ceph upgrade, and record the reason for each value.

Operating limits without creating a shared-storage blind spot

A QoS limit only holds if the people and systems that can change it are fewer than the people who can change the image or the hypervisor. Because the cap is applied by the client that opens the image, a workload owner who can edit the libvirt domain XML, the Proxmox VM configuration or the image's rbd settings can usually lift the limit without anyone else noticing. Decide in advance who may set image QoS keys (typically the storage team), who may set the matching client-side parameters (typically the platform team), and who may only read them. Restrict cephx capabilities so that application and tenant keys cannot write image metadata, and confirm with your hypervisor vendor and your own configuration management which settings a VM owner can edit without a review step.

Record every QoS change as a change request with the image name, the pool, the previous and new values, the reason and the rollback value. Store the desired limits in version-controlled configuration so that drift between the cluster and the hypervisor can be detected by comparing the two. Ask your team to confirm how the rbd QoS keys are read back from the cluster and whether your hypervisor exposes the effective values, because a limit that is set on the image but not applied on the client gives a false sense of control. Treat each change as a planned event: a limit that is tightened on a database volume during business hours can cause the same latency symptoms that the limit was meant to prevent.

Dashboards should show per-image throughput, IOPS and latency side by side with the configured cap, so that a flat line at the cap is visible as a state rather than a mystery. Alert on sustained saturation, meaning an image that stays at its limit for a period you choose, and on latency rising while the image is throttled, since that combination points to a limit set too low for the workload. Keep in mind that kernel RBD clients are not governed by the same image settings in every configuration, so the alert should also check the client type in your inventory. Confirm the metric names and collection path for your Ceph release in the Ceph documentation and your monitoring stack before building the rules.

What to confirm for your Ceph release and client stack

RBD QoS limits only work where the client enforces them, and that depends on the Ceph release, the client library and the hypervisor. Each check below should be run with your vendor or in your own lab before you rely on a limit in production.

Confirm the Ceph release is maintained

Start by identifying the exact Ceph release running in your cluster and checking its status on the active releases page. Maintained releases receive periodic backports and security fixes, so per-image limit behavior and option names can change between releases. Record the release, the latest point version you run, and the estimated end-of-life date. If your release is out of maintenance, plan an upgrade before you build QoS policy on it, because the settings you tune today may not exist or behave the same on the next version.

Map which clients use librbd

Limits set on an image are enforced by librbd inside the client process, so every consumer must be listed. Typical librbd consumers include QEMU disks attached through libvirt or Proxmox VE, and applications linked against the library. Build an inventory of each hypervisor, the client version it ships, and whether it opens images through librbd. Confirm this with your hypervisor vendor, because the answer depends on how each disk is attached.

Find kernel RBD clients

Kernel RBD clients, such as images mapped with the kernel driver on a Linux host, do not run librbd, so image-level limits set in the cluster do not govern their I/O in the same way. Identify every host that maps images through the kernel and decide which host-level controls apply, such as block-layer throttling or cgroup I/O limits. Test these controls in your own lab, because their behavior differs by kernel version and device type.

Verify exact setting names and units

Option names for RBD QoS, including combined and per-direction limits and burst settings, must be copied from the documentation for your release, not from older blog posts. Check each key, its unit (IOPS, bytes per second or a burst window), and its default value. A limit written in the wrong unit can cap a volume far below what you intended or leave it uncapped. Test each setting on a non-production image and measure the result with a benchmark before applying it broadly.

Check hypervisor interface support

Some hypervisors expose RBD limit settings in their management interface, while others require editing the disk definition directly. Confirm which fields your platform presents, whether it passes the limits to QEMU and librbd unchanged, and whether changes apply to running virtual machines or only after a restart. Ask the vendor how limits are stored and whether they survive live migration between hosts.

Confirm limits in your own lab

Every check in this set should be repeated in a lab that matches your production release, client versions and hosts before any limit is applied. Use a noisy-neighbor test with two images on the same pool and measure latency and IOPS for each. Keep the test results with the release record so that an upgrade triggers a retest rather than an assumption.

Questions buyers ask about RBD QoS before hiring help

Can our own team set RBD QoS limits without a partner?

Yes, most teams with Ceph operators can set per-image limits themselves, because the controls are image-level settings applied with the standard Ceph tooling. The effort lies in deciding the numbers. Your team needs to know which images belong to which workloads, what each workload actually drives in IOPS and throughput at peak, and which clients share a pool. Setting a cap is quick; choosing a cap that protects neighbors without starving the busiest VM is the part that takes measurement and review. Confirm the exact settings and their behavior against the Ceph release you run before you rely on them.

How does the number of images and clients change the effort?

Effort scales with how many distinct workload profiles you have, more than with the raw image count. Fifty images cloned from one golden template with identical behavior can be grouped into a few classes, while ten images belonging to ten different tenants each need their own review. Client count matters because kernel and librbd clients may sit on different hosts, versions and caching setups, which changes how you verify that a limit is enforced. Ask any partner to price the work by workload classes, client types and hypervisor or host groups, and to state which assumptions would move the quote.

What data do we need to provide before a QoS review starts?

You need three things: workload profiles, baseline benchmarks and monitoring history. Workload profiles describe read and write mix, block sizes, peak periods and which applications own each image. Benchmarks should be taken on the actual cluster, since the Ceph hardware recommendations say to benchmark before buying because no two clusters are alike. Monitoring history, covering OSD latency, client IOPS and throughput, and slow requests over several business cycles, shows where contention really happens. Without these, any limit is a guess, so ask for a discovery phase that produces them as a deliverable.

Does QoS replace adding capacity to the cluster?

No. QoS limits control how one client can consume shared capacity, but they do not create capacity. If the cluster is saturated, limits only decide which workloads slow down, and a cluster running near its limits will still show latency. Capacity planning, media choice and network design determine the ceiling, while QoS shapes access to that ceiling. Ask a partner to review both together, because a limit set to fit a cluster that is too small usually moves the problem rather than solving it.

Do kernel RBD clients get the same limits as librbd clients?

Not necessarily, and this is a common source of surprise. Image-level limits are applied by the librbd client path, so workloads that attach through the kernel RBD driver need other controls to be capped reliably. Those controls usually sit at the host, hypervisor or cgroup layer, and they must be designed and tested for each host type. Ask your team or a partner to confirm which client paths your VMs and hosts use and to test enforcement on each path before you assume the limit holds.

What would a fixed-scope QoS engagement cover?

A fixed-scope engagement covers a defined set of clusters, pools and image groups, with the scope, deliverables and timeline agreed in writing before work starts. Typical deliverables include a discovery review of profiles and monitoring, a proposed limit model, configuration changes applied in your environment, enforcement tests on each client path, and an operations note explaining how to adjust limits later. Items usually outside the scope are hardware purchases, application tuning and unrelated cluster repairs, unless you add them. DeployOpen does not publish prices; a quote follows from the written scope.

What recurring work follows after the limits go live?

Limits need periodic review because workloads change, new VMs arrive and Ceph releases alter behavior. Recurring work includes watching latency and slow request trends, revisiting limits after large onboarding events, and rechecking enforcement after cluster upgrades. This work can stay in-house with a runbook, or it can sit under a support retainer. Ask for the review cadence and the change process in writing, so you can compare the ongoing cost of your own team against a partner's retainer.

How DeployOpen scopes and delivers RBD QoS work

DeployOpen plans, implements, tests and documents Ceph RBD QoS limits inside your own infrastructure, starting from the workloads that share each pool. Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts. Ongoing support, upgrades and continuous testing are available as retainers.

How we scope

Inventory which VM disks, images and pools share OSDs and where contention has been observed.
Agree in writing which limit types, per-image targets and burst settings are in scope.
Confirm the Ceph release, client types and hypervisor versions that will enforce the limits.

How we build

Set baseline benchmarks on representative images before any limit is applied.
Apply combined and per-direction limits through the agreed configuration method, one image group at a time.
Test that limits hold under concurrent load and that kernel RBD clients get the separate controls agreed.

How we hand over

Document each limit, its owner, the clients it applies to and the rollback steps.
Deliver test results comparing baseline and post-change behavior for the agreed workloads.
Walk your operations team through monitoring signals that show a limit is throttling legitimate work.

How we support

Review limit settings when images are added, moved or resized, as part of an agreed change process.
Revisit limits during Ceph upgrades, since client behavior can change between releases you confirm.
Offer ongoing support, upgrades and continuous testing through a retainer if your environment keeps changing.