Which librbd cache setting fits your VM workloads
Virtual machine disks on RBD usually need two things at once: low latency for small writes and a guarantee that acknowledged data survives a failure. The librbd client cache sits between the guest disk and the cluster, inside the QEMU process on the hypervisor host, where it merges and batches I/O before it reaches the OSDs. Three policies control how that cache treats writes. Writeback acknowledges a write once it is held in the cache, *writethrough* acknowledges a write only after the cluster has stored it, and writearound sends writes straight to the cluster while the cache serves reads. Any team running VM disks on RBD has to pick one, because the choice decides what a guest can lose and how much latency it sees.
The cache belongs to the client, so its memory and crash behavior belong to the hypervisor host, not to the storage cluster. Dirty data held under writeback lives in host RAM until it is flushed, so a host crash, a kernel panic, an out-of-memory kill of the QEMU process or a power loss can drop writes the guest believed were durable. Live migration adds another question: confirm how your virtualization platform hands over an image that still holds cached data. Options such as rbd_cache_size and rbd_cache_max_dirty set how much data can be at risk at any moment, so host memory sizing has to account for them alongside the guest workloads.
Before you choose, check two things. First, confirm whether the guest operating system and applications issue flush requests, because writeback is only as safe as the guest's flush behavior. The rbd_cache_writethrough_until_flush option keeps the cache in writethrough mode until the first flush arrives, which protects guests that never flush, but confirm its default and its behavior in the Ceph release you run. Second, test a host crash on a non-production image and check what the guest filesystem reports afterward. Record the chosen policy for each pool or image in your runbook, and verify the exact option names and defaults in the Ceph documentation for your release before you change anything in production.
How each librbd cache mode handles reads and writes
This table compares writeback, writearound, writethrough and cache-off on read caching, write acknowledgement, data at risk during a host crash, and the VM workloads each suits. Read each row as a trade-off between speed and how much unflushed data a crashed client could lose. Confirm option names and defaults for your Ceph release before you set them.
| Behavior | Writeback | Writearound | Writethrough |
|---|---|---|---|
| Write acknowledgement | Writes can be acknowledged to the guest once stored in the client cache, before they reach the OSDs | Writes go straight to the cluster and are acknowledged there, bypassing the read cache for writes | Writes are acknowledged only after they reach the cluster, but reads can still be served from cache |
| Read caching | Reads are cached on the client and can be served locally after being fetched | Reads are cached on the client, but writes do not populate the cache | Reads are cached on the client; the cache is used for reads and written data is kept consistent with the cluster |
| Dirty data at risk on host crash | Acknowledged writes still held in the client cache can be lost if the client host fails before a flush | Little or no acknowledged-but-unflushed data, because writes are committed to the cluster on each request | Little or no acknowledged-but-unflushed data, because writes are committed before acknowledgement |
| Flush behavior | Dirty data is flushed in the background and on guest flush requests, subject to dirty limits | No dirty data is held for writes, so there is no write-back flush queue to manage | Cache is treated as read-only-safe until the guest issues a flush, after which write caching can apply |
| Suits workloads | Bulk or latency-sensitive writes where the guest issues flushes and the host is reliable | Write-heavy guests that need committed durability and still benefit from read caching | Guests that do not send flushes reliably, or where safety matters more than write latency |
| Cache-off comparison | Compare against cache-off to measure the gain before choosing writeback | Often close to cache-off for writes, while read behavior differs | Choose cache-off only if you need no client-side caching at all; confirm the exact behavior for your release |
Choosing a cache policy by workload
Application servers, build agents and utility VMs usually gain the most from client-side caching, because repeated small writes and metadata updates can be absorbed before they reach the cluster. Writeback is a common starting point for these guests, but only after the team has confirmed that the guest filesystem and the virtual disk driver pass flush requests down to librbd. The cache stays in hypervisor memory until a flush or eviction, so the trade-off is accepted only where a short window of lost writes after a host failure is tolerable.
- Starting point: writeback, with flush pass-through verified on every guest image in the pilot group.
- Safety cost: writes acknowledged to the guest may exist only in host memory until flushed, so a hypervisor crash can lose them.
- Verify: that Linux and Windows guest images issue flushes, and that the disk controller setting in each template does not hide them.
- Verify: memory headroom on each hypervisor for the combined cache across all VMs placed there, including during recovery.
- Watch: Windows guests with write caching enabled on the virtual disk, which already buffer data inside the guest.
Before changing the default for a fleet, run a power-loss test on a pilot VM and confirm the filesystem recovers cleanly.
Database engines commit a transaction only after the storage confirms that the log reached stable media, so the guarantee depends on every layer between the engine and the cluster honoring that confirmation. For transaction log volumes and any database where lost commits are unacceptable, writethrough is the conservative starting point: each write reaches the cluster before the guest is told it is complete. The price is latency on small synchronous writes, and that latency is often the main performance question for the team.
- Starting point: writethrough for log and data volumes, with writeback reserved for scratch or rebuildable data.
- Safety cost: no acknowledged write exists only in client memory, but each commit waits for a full round trip to the cluster.
- Verify: commit latency and fsync timing under production-like concurrency, measured inside the guest.
- Verify: that the database fsync and write-barrier settings are the values the vendor documents for your version.
- Watch: benchmarks that show high throughput from cache hits, which can hide the latency your transactions will see in production.
Confirm the recovery point you need from the database's own replication and backups, because the storage cache setting does not replace either.
Live migration moves a running guest between hypervisors while its writes continue, so the cache state has to be handled at cutover. With writeback, dirty data held in the source host's cache must be drained or transferred correctly before the destination takes over, and that path is where a configuration mismatch or a failed migration causes the most damage. Choose the cache mode that your hypervisor and libvirt versions have been tested with for migration, and treat the migration path as part of the cache decision rather than a separate concern.
- Starting point: the cache mode your hypervisor vendor or tooling documents for migration, often writethrough when the migration path is not proven.
- Safety cost: a writeback guest depends on a clean drain at cutover, so a failed migration carries more risk of lost or inconsistent writes.
- Verify: that the cache setting is identical on source and destination hosts for each guest.
- Verify: migrations under sustained write load, followed by checksum comparison of test files on the guest.
- Verify: behavior when a migration is interrupted or aborted, including whether the guest can resume on the source host.
Record the tested versions of hypervisor, libvirt and Ceph client libraries, because a change to any one of them can alter the result.
Writeback assumes that the guest sends flush requests when it needs data persisted. Some guests never send them, some send them only in certain code paths, and some are images built by teams outside your control. For these guests the safe starting point is writethrough, which keeps every acknowledged write on the cluster at the cost of cache benefit. The setting rbd_cache_writethrough_until_flush can also help, because it keeps the cache in writethrough mode until the first flush arrives, but it does not protect a guest that flushes once and then stops.
- Starting point: writethrough for any guest whose flush behavior has not been tested.
- Safety cost: the cache gives little write acceleration, so latency and throughput for these guests are closer to the uncached path.
- Verify: whether the guest kernel or driver sends flushes under normal load, and whether that behavior persists over a long run.
- Verify: a guest test image that ignores flush requests, to confirm which cache mode is in effect after the first flush.
- Watch: images that use a flush-disabling option in the disk configuration, which removes the protection entirely.
Require evidence of flush behavior from any image owner before a guest is moved off writethrough, and recheck it after each kernel or driver update.
What writeback costs when a host fails
Writeback mode acknowledges a guest write once librbd holds it in client memory, before the write reaches the OSDs. That is the source of its speed and its risk. If the hypervisor host loses power, the kernel panics, or the QEMU process is killed, any dirty data still in the cache never reaches the cluster. The guest may have received a successful completion for that data, so the application above it can believe a transaction was committed when it was not. The loss is bounded by what the cache held at the moment of the failure, but the size of that bound depends on your cache size, dirty limits and how often the guest flushes, so confirm the exact flush and dirty-data behavior for the Ceph release you run against the librbd documentation for that release before you write it into a recovery runbook.
The guest's flush requests decide how much of that risk is actually exposed. A filesystem or database inside the VM issues flush (cache flush) commands at points where it needs data on stable storage, such as after a journal commit. librbd honors these requests by writing out dirty data before it completes the flush, so in writeback mode the durability guarantee holds only at those flush boundaries. Writes between flushes are the exposed window. Guests that never flush, or that run with an option that disables write barriers, lose that protection even when the cache mode is correct. Writethrough mode sends every write to the cluster before acknowledging it, so a host failure cannot lose an acknowledged write, at the cost of a full round trip to the OSDs on every write. Writethrough also keeps the cache read-only in effect for writes, which is why it is the safer choice for a database volume that has no application-level recovery of its own.
Application sync settings are the third control. A database configured to fsync on every commit already forces flushes through the stack, so writeback loses less than the raw cache size suggests, but only if the flush reaches librbd and is not dropped by a guest driver setting. Ask your VM and application owners which volumes run with synchronous commit and which tolerate losing the last few seconds of acknowledged writes, then assign cache modes per volume rather than per host. Test the decision with a controlled QEMU kill during a write load on a non-production image, and record what was present on disk afterward. That test tells you more about your exposure than any general description of the cache.
Setting and verifying the cache mode on a VM disk
This sequence confirms the Ceph release, sets the librbd cache policy on a QEMU RBD disk, and proves with the running configuration and a controlled write test that the guest is using the intended cache behavior.
Confirm the Ceph release in use
Record the release of every client library and cluster daemon before changing anything. Run
ceph versionson a cluster node andrbd --versionplusqemu-img --versionon each hypervisor. Check the versions against the Ceph active releases page, which lists Squid (19.2.x) and Tentacle (20.2.x) as the actively maintained lines. Confirm that the client librbd build matches a supported line and that the option names in the next steps exist in that release by runningceph config help rbd_cache. Done when you have a written version record for the cluster and each hypervisor.Review the current cache settings
Before changing anything, capture the baseline. Run
ceph config get clientfor the rbd_cache family of options andceph config dumpto see which values are set centrally and which are inherited. Then read the disk definition withvirsh dumpxml <domain>and note thecacheattribute on thedriverelement of each RBD disk. Confirm which setting actually governs the disk, because a value in the cluster configuration does not override an explicit disk attribute. Done when you have a saved baseline file for the VM, which you will compare against at verification.Set the cache mode on the QEMU RBD disk
In libvirt, set the
cacheattribute on the RBD disk'sdriverelement, for examplecache='writeback'for write caching orcache='writethrough'for a policy that acknowledges writes only after they reach the cluster. Edit the definition withvirsh edit <domain>, keeping the cephx secret and the source pool and image unchanged. Explain the trade-off to the change owner before you save: writeback can acknowledge writes that are still held in client memory, so a host failure can lose recent writes. Done whenvirsh dumpxmlshows the new attribute andvirsh definereports success without warnings.Set client-side librbd cache options
Set the librbd options in the Ceph configuration for the
clientsection, usingceph config set client <option> <value>for each option you have decided to change. Typical options are the enable switch for the RBD cache, the cache size, the dirty data limits, and the writethrough-until-flush behavior. Use the values documented for your release fromceph config help, and do not carry values over from older clusters without checking them. Record each change with the reason, the old value and the new value. Done whenceph config dumplists exactly the intended client options and nothing else under the client section.Apply the change to the running guest
Most cache changes take effect only when QEMU opens the disk again. Plan a guest reboot in a maintenance window, or run a live migration to a host that already has the new definition, which causes QEMU to start on the target with the new settings. Stop the guest cleanly with
virsh shutdown <domain>and start it withvirsh start <domain>, or usevirsh migrate --livewith the destination configured. Confirm the guest runs and that the disk is attached before you continue. Done whenvirsh domstatereports running and the guest's filesystem mounts without errors.Verify the effective settings on the client
Confirm what librbd is actually using, not only what you configured. If the client exposes an admin socket for the QEMU process, run
ceph --admin-daemon <socket> config showand filter for the rbd_cache options. Compare the output with the values you set. Where the admin socket is unavailable in your build, confirm the settings through the client configuration and the disk definition, and record that the runtime check was not possible. Done when every option you changed matches the running value, or you have a documented exception with an owner.Run a controlled write test
Test write behavior with a known workload inside a non-production guest or a scratch volume. Use
fiowith a fixed block size, a fixed size, andfsyncorfdatasyncafter writes, then record throughput and latency for the new mode against the baseline run. For the data safety check, write a known pattern, callsync, and confirm the checksum on a read-back from the same disk after the guest restarts. Note that a clean read-back after a controlled restart does not prove behavior during a host crash, so treat that failure mode as a separate test that your team plans. Done when the test output, checksums and the approved cache mode are stored with the change record.
Sizing the cache and dirty limits per VM
Three librbd options decide how much write and read data a client holds in memory and for how long: rbd_cache_size sets the total cache for each open image, rbd_cache_max_dirty caps how many bytes of unflushed writes may accumulate, and rbd_cache_max_dirty_age sets how many seconds a dirty block may wait before it is written to the cluster. The dirty limit must stay below the cache size, and the dirty age sets the longest window in which acknowledged writes exist only on the client host. Before you set any of them, confirm the defaults and exact semantics for your Ceph release in the documentation for your version, because these options have changed behavior across releases.
Cache memory belongs to the hypervisor process, not to the cluster. Each QEMU process that opens an RBD disk through librbd allocates its own cache, so the host cost is the per-image size multiplied by the number of attached disks on that host, plus overhead. A host running 40 VMs with two disks each carries that multiplication in every value you choose. Size the host for peak use, not the quiet hours, because a backup window, a rebuild or a batch job can raise both the number of dirty bytes and the number of active images at the same time. The Ceph hardware recommendations make the same point about memory in general, and cache memory should be counted in that budget alongside guest memory and OSD memory on storage nodes.
The benefit of a larger cache depends on locality. A VM that rereads a hot working set, such as a database with frequently accessed indexes, may gain from a bigger read cache, while a log writer or bulk loader that streams sequentially may gain little beyond a modest buffer. Illustrative example: doubling rbd_cache_size on a streaming ingest VM may add memory use without moving latency, while the same change on a database VM may cut read latency for its hot set. Benchmark with your own profile before changing values. Run the same guest-level workload, such as fio with your block size, read/write mix and queue depth, and record latency percentiles, throughput and the host's resident memory for each QEMU process. Change one value at a time, and keep the dirty age and dirty limit values you tested with the VM's configuration.
Security and operational risks of the client cache
Where does the librbd cache live and who can reach it on the host?
The cache lives in the memory of the client process, usually the QEMU process that runs the VM, so anyone who can read that memory on the hypervisor can read cached guest data. Treat the hypervisor as part of the data path. Confirm which process holds the cache in your stack, restrict root and debugger (ptrace) access on hypervisors, and disable core dumps for VM processes or send them to encrypted storage. Check where cephx keyrings and libvirt secrets are stored, because a host compromise exposes them too. Under writeback, dirty data exists only in that memory until it is flushed, so a host crash can lose acknowledged writes that were never flushed. Ask your platform team to confirm the dump and debugging policy in writing.
What happens to cached data during a live migration?
Dirty data must be flushed from the source client before the destination takes over, or writes the guest believes are complete may not reach the cluster. Confirm with your hypervisor and QEMU versions that migration flushes the cache for the policy you run. The cases to test are a migration that fails midway and a source host that crashes during migration. Where guests may not send flush requests, review rbd_cache_writethrough_until_flush, which keeps writes going through to the cluster until the first flush arrives. Run migration tests with a realistic write load under the chosen policy, and keep the test steps and results as change evidence.
How do cephx client keys limit what a VM can access?
A cephx key limits which pools, namespaces and images a client can read and write, but the cache sits inside whatever the client is already authorized to touch. Cache settings do not change authorization. Create one key per hypervisor group or tenant, scope its caps to the pool or namespace it needs, and avoid broad capabilities such as allow *. For example (illustrative only): ceph auth get-or-create client.vm-tenant-a mon 'profile rbd' osd 'profile rbd pool=vms-tenant-a'. Confirm the exact cap syntax for your release, the key rotation process, and who approves new keys. A VM that shares a key with other VMs can reach their images, so key design matters more than the cache policy.
How do we monitor cache-related latency and errors for RBD clients?
Watch four layers and compare them over time:
- Client side: performance counters exposed through the client admin socket, if your Ceph version enables them for librbd in your deployment.
- Guest side: write latency and flush latency reported inside the VM, which show what the application actually experiences.
- Cluster side: OSD commit and apply latency, slow request warnings and health messages, which show whether writes are reaching the cluster slowly.
- Logs: client-side errors on flush, timeouts and reconnects.
Confirm the counter and log names in your release before you write alerts, and set baselines before any cache change so you can tell a regression from normal load.
How do we plan a change window for cache settings across a fleet?
Client cache settings are read when the client opens the image, so running VMs keep their old settings until they restart or migrate. Plan the change in waves. Record a performance and data-safety baseline, pilot on lower-risk hosts, and agree with application owners how much data loss each workload can tolerate. Apply the setting through your configuration management, verify each wave against the baseline, and schedule waves outside peak periods. Write the rollback path before the first wave, including how to return to the previous setting and which VMs must restart again. Ask your vendor and platform team to confirm which changes need a restart in your versions.
Can our own team tune the librbd cache, or do we need a partner?
Your team can change the settings, but the work that carries risk is testing: profiling real workloads, running crash and migration tests, and documenting the policy for each workload class. The main cost components are workload discovery, test environment time, the configuration change itself, and a runbook for operators. Costs rise with the number of hypervisors, the number of distinct workload types, and the depth of crash and migration testing. DeployOpen plans and tests cache policy inside your infrastructure and documents the result. Scope, deliverables and timelines are agreed in writing before work starts, and a quote follows from that scope. Ask any partner to show how they test flush behavior and migration under load.
What to confirm in your environment before choosing a mode
A cache mode is only as safe as the path that carries flush requests from the guest to the cluster. The checks below cover the Ceph release, the hypervisor stack, the guest workload, memory, migration and backups, and each one names where to confirm it.
Cache option names and defaults
The librbd cache keys, their defaults and how writeback and writethrough behave can change between Ceph releases, so the values in an older guide may not match your cluster. Confirm the exact option names and defaults for the Ceph release you run, including the cache size and dirty limit settings, before you write any configuration. Record the values you rely on so a later upgrade does not silently change write behavior. Check the release notes for your exact version and the client configuration reference in the Ceph documentation.
Flush pass-through in the hypervisor stack
A writeback cache only protects data if guest flush requests reach librbd. Confirm that your hypervisor, QEMU and libvirt versions pass flush commands through to the RBD disk and that the disk is not configured to ignore them. Test this on a non-production image by issuing a flush from the guest and checking the cache behavior. Ask the vendor of your virtualization platform to confirm this for the versions you run.
Guest filesystem and database sync behavior
Filesystems and databases issue sync and fsync calls at different points, and their durability depends on those calls completing. Confirm which guests run databases or journaling filesystems that depend on flush ordering, and what each one expects from the disk. Review the sync settings of each database engine and filesystem mount options in the guest. Guests that rely on barriers or write-through semantics need a mode that matches those expectations, not one chosen only for speed.
Host memory budget
The client cache consumes host memory for every RBD disk that uses it, and the total grows with the number of disks and the cache size you set. Confirm how much memory each hypervisor can spare after guests, the kernel and other services are counted, and size the cache against that budget. Plan for peak guest activity rather than the average, since the cache fills during bursts. Check the host memory figures on a loaded node, not on an idle one.
Effect of live migration
Moving a running VM to another host changes the client that holds the cache, and dirty data must reach the cluster before the move is safe. Confirm how your hypervisor flushes or drains the cache during live migration, and whether a writeback cache with unflushed data blocks or complicates the move. Test a migration under write load and verify the disk state on the destination. Ask your platform team to document the behavior for each cache mode you plan to use.
Backup tooling and on-disk consistency
Backups that read the image directly, or snapshots taken while the client holds dirty data, can capture a state the guest never wrote to disk. Confirm whether your backup tool flushes the guest or the cache before it reads, and whether it uses snapshots that quiesce the guest. Test a restore of a database volume taken during write activity and check that it starts cleanly. Document the sequence so the backup and cache mode are reviewed together when either changes.
Questions teams ask before changing librbd cache settings
How much effort does a librbd cache policy review take, and what drives it?
The effort depends mostly on how many distinct workloads share the hypervisors and how much of the current behavior is documented. A review covering one cluster, one hypervisor type and a handful of VM classes is a small exercise. Effort rises when hosts run mixed QEMU and kernel clients, when databases and general VMs share pools, or when nobody can say which cache mode each disk uses today. Ask any provider to scope the inventory of disks and cache settings first, because that step sets the rest of the work.
Is writeback caching safe for databases?
Writeback is safe only when the guest flushes data correctly and the cache is flushed before the guest reports a write as durable. With librbd, writeback holds dirty data in client memory until a flush, so a client crash can lose acknowledged writes that the application believed were stored. For databases, most teams start with writethrough or a conservative writeback setup and confirm that the guest issues flushes on commit. Verify the guest filesystem barrier settings and the database fsync behavior with your own tests before choosing a mode for production data.
How long does validation usually take, and what does it depend on?
Validation length depends on the number of workload classes, the test matrix, and how long you need to observe behavior under failure. A plan with a few workloads and a simple crash test can finish in a short cycle, while a plan that covers host crashes, live migration, and recovery under load needs more runs and more time for analysis. Ask providers how they will reproduce your production I/O patterns, which failure cases they will test, and what acceptance criteria will close the review. Confirm the schedule against your change calendar.
What must our team provide before a cache policy review starts?
A review goes faster when your team supplies accurate inputs. Prepare the following:
- A list of VM classes and the applications they run, with their write patterns and durability needs
- Hypervisor hosts, their QEMU and libvirt versions, and how they connect to the cluster
- The Ceph release in use, with the cluster's client configuration and any per-image cache settings
- Current benchmark results or a test window where benchmarks can run without touching production
- Failure and recovery procedures, including how guests are shut down and restarted after a host crash
Can our internal staff change the cache settings without outside help?
Yes, many teams change the cache settings themselves, as the settings are set in client configuration and per image. The harder part is proving the change is safe. Internal staff need time to test crash and flush behavior, to measure the performance effect, and to document the decision for auditors and application owners. Outside help is most useful when the environment is large, when databases carry critical data, or when the team has no test window. Whichever path you choose, record the chosen mode per workload class and keep it in version control.
Which Ceph release and client versions change the answer?
The Ceph release and the client library version determine which cache options exist and how they behave, so the review must name both. Check the Ceph releases page for the currently maintained releases, then confirm the librbd behavior for the client version your hypervisors actually run. Do not assume that a setting documented for one release applies to another. Plan a retest after any Ceph or hypervisor upgrade.
What does a cache policy change cost over time?
The one-time cost covers the review, testing, and documentation. The recurring cost comes from keeping settings consistent across new hosts, retesting after upgrades, and checking performance when workloads change. Ask any quote to separate the one-time review from any ongoing support, and to state what is included in retesting after each Ceph or hypervisor upgrade.
How DeployOpen scopes and delivers cache policy work
DeployOpen treats the librbd cache policy as a decision made with your workload, your hypervisor and your data safety requirements in view, not as a tuning default. We scope the cache mode, sizes and dirty limits in writing, test them against your VM workloads, and hand over settings and the reasoning behind them.
How we scope
- Inventory which VM workloads, hypervisors and client versions will use librbd and their write sensitivity
- Record the required crash consistency and flush behavior agreed with your application owners
- Define the cache modes to be tested, cache size and dirty limit ranges, and acceptance criteria in the written scope
- Confirm which client versions and cache settings your environment supports before the build starts
How we build
- Configure librbd cache options in the client configuration you control, per host or per image as agreed
- Test writeback, writearound and writethrough behavior against your guest filesystems and database workloads
- Verify that flush requests from guests reach the cluster under each cache mode and document the results
- Check the effect of cache sizes on host memory use alongside other VMs on the same hypervisor
How we hand over
- Deliver the final cache settings, the reasons for each choice and the workloads each setting covers
- Document the data safety trade-offs of each mode, including what can be lost after a host crash
- Provide a runbook for changing cache policy, validating a change and rolling it back on affected hosts
- Walk your operations and application teams through the test evidence and the open decisions
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer, with scope agreed in writing
- Re-test cache behavior after Ceph or client library upgrades, since defaults and options can change between releases
- Review cache settings when new VM workloads, hypervisors or pool layouts are added to the cluster
- Ask any provider to confirm the cache options and defaults for the exact Ceph release you run
