Ceph RBD for KVM and libvirt VMs

How libvirt and QEMU attach Ceph RBD disks to KVM guests

libvirt attaches a Ceph RBD image to a KVM guest through QEMU's librbd driver, which runs in the QEMU process on the hypervisor and talks to the cluster directly, without a kernel block device on the host. The cephx user that QEMU authenticates with is stored as a libvirt secret, and the disk definition in the domain XML refers to that secret by UUID, so the key itself stays out of the guest definition. libvirt decides what the guest sees, QEMU performs the I/O, and the Ceph cluster enforces authentication, pool permissions and placement. Each of the three has its own controls, and a plan that only configures one of them leaves the others open.

In practice the work splits into three tasks. On the Ceph side, you create a cephx user with capabilities limited to the pool or namespace that holds the VM images, and you confirm the monitor addresses the hypervisors will use. On each host, you define the libvirt secret of type ceph with the matching usage name and set the key from the cephx keyring. In the domain XML, the <disk type='network'> element names the pool and image, lists the monitor hosts, and references the secret. Confirm the exact element names and attribute values against the libvirt release your hosts run, since the schema has changed across versions. The secret and the cephx user are what you rotate, audit and revoke, so record which user backs which hypervisor cluster from the start.

Kernel RBD, mapped with rbd map, is a different client. It exposes the image as a host block device, so cache behavior, I/O limits and authentication are handled by the kernel client and the host, not by QEMU or libvirt. Settings described for librbd, such as QEMU-level throttles and cache modes, do not apply to a kernel-mapped disk, and a guest that uses a mapped device needs its own controls. Decide which client each workload uses before writing automation, because mixing both on the same image set makes troubleshooting and limit enforcement harder. Ask your implementation partner or your own team to confirm the libvirt, QEMU and Ceph release combinations you intend to run, as support for specific RBD options varies between them.

What a Ceph RBD and libvirt build needs before work starts

A Ceph RBD disk for a KVM guest depends on five things lining up before the first image is created: a supported Ceph release, a healthy cephadm-managed cluster, hypervisors that can reach the cluster, cephx identities with the right scope, and consistent clocks. Work through these cards in order, because a gap in one usually shows up later as a failed live migration or a stalled disk.

Choose a supported Ceph release

Two releases are actively maintained: Tentacle (20.2.x) and Squid (19.2.x). Tentacle is the newer line, with its first release on November 18, 2025 and an estimated end of life of June 1, 2027. Squid's estimated end of life is October 31, 2026, so a build that will run for several years should start on Tentacle unless a supplied constraint says otherwise. Confirm the current point release on the releases page before ordering hardware.

Run a cephadm-managed cluster

Plan the cluster around cephadm, which runs Ceph daemons as containers and manages their placement. It was introduced in Octopus (v15.2.0) and does not support older releases. Confirm that the MONs, managers and OSDs are placed by the orchestrator and that the cluster reports HEALTH_OK before any RBD pool is created. Hypervisors do not need Ceph daemons, but they need the client keyring and configuration.

Confirm libvirt and QEMU RBD support

The hypervisor's libvirt and QEMU builds must include RBD support, meaning QEMU links against librbd and libvirt can define network type disks that point at a Ceph pool. Distribution packages differ in whether this is enabled, so confirm the exact package versions with your distribution's documentation and check that qemu-img can open an image in the pool from each hypervisor. Also confirm that the Ceph client packages match the cluster release closely enough for your vendor's support statement.

Open network paths to MON and OSD addresses

Each hypervisor must reach every MON address and every OSD address on the cluster network, because RBD clients talk directly to OSDs. Confirm routing, firewall rules and MTU across the path, and bond links across two switches as the hardware guidance recommends. Clients connect to MONs on port 3300 when msgr2 is in use, and the legacy port 6789 may appear in older configuration. Test each path from every hypervisor before live migration is attempted.

Plan cephx keys for each client

Create one cephx user per consumer or hypervisor group, with capabilities limited to the pools it needs, and avoid sharing the admin key with guests or automation. Libvirt stores the Ceph key as a libvirt secret, so decide who holds the authority to define and rotate those secrets on each host. Record the key owner, pool scope and rotation procedure in the runbook before the first disk is attached. Confirm capability syntax against the current Ceph authentication documentation.

Set host access and time sync

The person or automation that defines libvirt secrets needs root or equivalent access on each hypervisor, and that access should be documented and limited to the hosts that run VMs. Time skew between MONs, OSDs and hypervisors causes authentication and cluster health problems, so run an NTP or chrony source on every node and confirm that all clocks agree before cephx tickets are relied on. Check the clock status as part of the acceptance test, not only after an incident.

Configuring cephx users, libvirt secrets and RBD disks in order

This sequence gives each KVM host a least-privilege Ceph identity, a libvirt secret that holds its key, and a domain definition that reads VM disks from RBD. Each step ends with a check so a fault shows up before any production guest depends on the disk.

Create a dedicated cephx client
On a Ceph admin node, create a client identity scoped to the pool that holds the VM images, for example with ceph auth get-or-create client.libvirt and caps limited to mon 'profile rbd' and osd 'profile rbd pool=vms'. Use a separate client for each hypervisor cluster so that revoking one does not affect the others. Confirm the result with ceph auth get client.libvirt, checking that the caps name only the intended pool and that no admin capabilities appear. Record the key in your secrets store, not in shell history or configuration management plaintext.
Define the libvirt secret with a fixed UUID
Write a secret XML file, for example ceph-secret.xml, with ephemeral='no', a private='yes' flag, a usage type of ceph, and a name that identifies the Ceph cluster and client. Choose the UUID once and reuse it on every host, because the domain XML refers to it. Run virsh secret-define ceph-secret.xml, then virsh secret-set-value with the base64 key from the previous step. Check the result with virsh secret-list, which should show the UUID and usage, and virsh secret-get-value should return the key only when run by a privileged operator.
Verify monitor and network reachability
Confirm that each hypervisor can reach every Ceph monitor on the v2 port 3300 and the v1 port 6789 if your cluster still uses it, and that the public network path is bonded and routed as your design intends. Use ceph mon dump to list monitor addresses and test from the host with a simple TCP connection check. Also run ceph -s from the hypervisor using a copied ceph.conf and the client keyring to prove the cephx login works before libvirt is involved. A failed check here usually points to firewall rules or a wrong public network, not to libvirt.
Define the RBD disk in domain XML
Add a <disk type='network' device='disk'> block to the guest definition with <source protocol='rbd' name='vms/guest01-disk0'>, one <host> element per monitor, and an <auth username='libvirt'> element that references the secret UUID with <secret type='ceph' uuid='...'/>. Set the <driver name='qemu' type='raw' cache='none'/> or the cache mode you chose from the cache policy guidance. Apply the change with virsh define guest01.xml, then run virsh dumpxml guest01 and confirm that the disk source, hosts and secret UUID match what you intended.
Start a test guest and confirm the disk
Start a disposable test guest with virsh start guest01 and open its console. Inside the guest, check that the virtual disk appears and that no I/O errors appear in dmesg. On the hypervisor, virsh domblklist guest01 should list the RBD source, and ceph osd pool stats vms or the client I/O view should show traffic from the libvirt client while the guest is busy. This confirms the data path works end to end, not just that the definition parses.
Run fio inside the guest
Inside the test guest, run a bounded fio job against a scratch file or a spare virtual disk, using a mix of random reads and writes at a modest queue depth and a fixed runtime. Record the IOPS, latency and any error counts, then compare them with a baseline taken from the same host class. Check ceph -s during the run for slow request warnings and confirm that the job completes without I/O errors in the guest or on the hypervisor. Keep the results with the configuration so that later changes can be measured against them.
Record the configuration for repeatable builds
Store the client name, pool, caps, secret UUID, monitor list and domain XML template in version control, with the cephx key kept in your secrets system. Document which hosts received the secret and confirm each one with virsh secret-list. Repeat the fio check on a second hypervisor before moving production guests so that live migration between hosts is tested with the same identity and secret layout. Confirm the first migration with virsh migrate --live only after the previous checks pass on both hosts.

Choosing cache modes and pool layout for VM disks

The librbd cache sits in the QEMU process on the hypervisor and decides when a guest write is acknowledged and when it reaches the cluster. Writeback caching acknowledges writes early and holds dirty data in memory until a flush, which improves throughput but puts unflushed data at risk if the host fails. Writethrough caching sends writes to the cluster before acknowledging them and only uses the cache for reads, which is safer for guests that do not send flushes. The effective behavior depends on guest flush semantics: a guest that issues flush requests at the right points (most modern Linux and Windows guests do) can use writeback safely, while a guest or image that never flushes should not be trusted with it. The following points summarize the trade-offs.

Flush-aware guests: Confirm that the guest OS and filesystem send flushes, and that the disk bus advertises a volatile write cache to the guest.
Writeback risk: Dirty data can be lost on a host crash before a flush, so databases and journaling filesystems need explicit flush paths tested.
Writethrough cost: Every write waits for the cluster, so latency-sensitive workloads see the full replication round trip.
Host memory: Cache size and dirty limits consume hypervisor RAM and should be sized per VM density.
Live migration: Confirm the cache mode and the migration settings your hypervisor uses before changing either on running guests.

Before choosing a mode, test a crash and power-loss scenario with the actual guest workload. For the full policy details, see the librbd cache settings page in this group.

Mapping libvirt disk fields to Ceph identities and pools

This table maps each libvirt disk element to the Ceph object, client identity or host setting it points to. Read it when building the domain XML, and check that the values listed as shared stay identical on every hypervisor that may run the guest.

libvirt field	What it points to in Ceph	Must match across hosts for live migration?
disk source name	Pool and RBD image name (for example, the pool name followed by the image name), plus optional namespace	Yes, the same pool, namespace and image name must resolve on every host
source namespace attribute	RADOS namespace inside the pool, used to separate tenants or VM groups within one pool	Yes, the namespace must be identical wherever the guest may run
auth username	The cephx client name without the client. prefix, for example a dedicated client for VM disks with access limited to the pool	Yes, the same client and capabilities must exist on every host
secret usage or UUID	A libvirt secret that holds the cephx key for the client, referenced by UUID from the disk definition	Secret UUID should match on every host so the same domain XML works; key material is set locally on each host
host name and port entries	Monitor addresses that librbd uses to reach the cluster, usually listed in the same order on every host	Addresses must be reachable from every hypervisor, with the msgr2 port (3300) open where the messenger v2 protocol is used
cache and driver attributes	Client-side QEMU and librbd behavior, not a cluster object	No, but cache policy should be consistent on hosts that may receive the same guest
Ceph configuration file path	The ceph.conf and keyring that librbd reads for global settings when no inline host list is used	Content should match on each host, though it may differ in local paths

Live migration between hosts when disks stay on RBD

With disks on RBD, the destination hypervisor opens the same image that the source is using, so disk contents do not need to be copied between hosts. Live migration then depends on every host presenting the same identity, reaching the same Ceph daemons, and running compatible software with a cache policy that permits it.

Identical cephx secrets on every host

Each host needs a libvirt secret that holds the same cephx key for the client user that owns the VM disks, registered under the same UUID. If the UUID or key differs between hosts, the destination cannot open the disk and the migration fails before the guest moves. Check that the secret definitions match on every hypervisor and that rotating the key is done on all hosts in one change window. Confirm the exact secret and auth syntax against the libvirt RBD documentation for your version.

Network reachability to MONs and OSDs

Both the source and destination hosts must reach every Ceph monitor and every OSD that holds the image's objects, on the addresses the cluster advertises. A destination host on a separate VLAN or behind a firewall rule that allows only the monitors will connect but stall on data I/O. Test from each hypervisor with a read and write against a scratch image before scheduling the first move. Ceph's messenger v2 uses port 3300 for monitors, so confirm that port is open alongside the OSD port ranges.

Matching QEMU and libvirt versions

Live migration assumes the destination can run the guest with the same QEMU machine type and a compatible libvirt version. Mixed-version hypervisor pools are common during upgrades, so record the versions on each host and confirm which combinations your distribution and the QEMU and libvirt release notes support for live migration with librbd disks. Treat version pairing as a gate in your upgrade plan, not as an assumption.

Cache settings that allow migration

Migration requires that the guest's disk cache mode be one the hypervisor will allow to move across hosts, and that no host-local cache state holds data the destination cannot see. Writeback caching on the client changes what is safe to hand off, so review the cache mode on each disk definition before the move. Confirm the cache rules for librbd in the QEMU and libvirt documentation for your versions and test a migration of a non-critical guest first.

Pool layout and image placement

Migration does not move data, but the pool layout decides how much I/O the destination takes on once the guest resumes. Keep VM images in a pool whose failure domains and media suit the workload, so both hosts see the same latency profile. Avoid placing the only copy of critical images on a single rack if hosts are spread across racks. Confirm your pool replication settings with your storage design before relying on cross-host failover.

Verify the move before you rely on it

Treat a successful migration as a test you run on a schedule, not a one-time check. Confirm the guest is running on the destination, that the disk path still points at the same RBD image, and that I/O resumes without errors in the guest and in Ceph health output. Keep a rollback plan for failed moves and record the libvirt and QEMU versions used for each test so later upgrades can be compared against it.

Common failures when attaching RBD disks to KVM guests

This table lists the failures teams most often hit when libvirt guests use RBD disks, with the cause, the check and the fix for each. Read it when a guest will not start, a host cannot reach the cluster, I/O is slow, or a live migration fails after a secret or network change.

Failure	Likely cause	Check	Fix
Authentication fails on guest start	The libvirt secret UUID in the domain XML does not match the secret defined on the host, or the secret holds an old cephx key	Compare the secret UUID in the domain XML with virsh secret-list, then confirm the stored value matches the key from ceph auth get for the client	Redefine the secret with the same UUID, set the current key with virsh secret-set-value, and restart the guest
Guest fails to start after key rotation	The cephx key was rotated on the cluster but the secret on one or more hypervisors was not updated	Run ceph auth get on the client and compare the key with each hypervisor's libvirt secret, host by host	Update the secret on every hypervisor before restarting guests, and roll rotations through a maintenance window
MON unreachable from a new host	The host lacks the monitor addresses in ceph.conf, or the public network route or firewall blocks the monitor ports	Test TCP reachability to each monitor address on the messenger ports, and confirm the host's ceph.conf lists current monitors	Add the monitor addresses to the client configuration, open the required ports or routes, then retest before defining the guest
Slow I/O inside the guest	Client traffic shares an undersized or single-path network with recovery, or the cache mode does not suit the workload	Check network throughput between hypervisor and OSD hosts during load, and compare the disk cache setting in the domain XML with the workload	Use bonded links across switches as the hardware guidance describes, separate recovery traffic where possible, and benchmark the chosen cache mode
Live migration fails between hosts	Secrets, keys or monitor lists differ between source and destination hosts, so the destination cannot open the RBD image	Run the same authentication and reachability checks on both hosts and compare secret UUIDs and key values	Make the secret UUID and key identical on every host in the migration set, then repeat a test migration of a non-production guest
Guest disks missing after host reboot	The secret or client configuration was defined at runtime only and was not persisted on the host	List secrets with virsh secret-list after a reboot and confirm the libvirt service starts before guests	Define secrets as persistent objects and keep the configuration in the host build or configuration management

Questions teams ask before attaching VM disks to Ceph

How much effort does a Ceph RBD build for libvirt take?

Effort depends mostly on whether the Ceph cluster already exists and meets the hardware and network guidance, then on how many hypervisor hosts attach to it and what cache and live migration behavior you need. A new cluster adds work for hardware sizing, a dedicated drive per OSD, WAL and DB placement on shared SSDs for HDD-based OSDs, and bonded network links across two switches, as the Ceph hardware recommendations describe. A cluster that already runs needs a smaller effort: cephx users, libvirt secrets, pool layout, and a tested migration path. Ask a partner to size the build from these inputs in writing before any date is agreed.

What does the client team need to provide before work starts?

The client team usually provides the following:

Access to the target hypervisor hosts, the Ceph cluster or the hardware plan for it, and the network design.
A list of VM workloads with their I/O profile, growth expectations and any latency requirements.
The cache policy and live migration requirements, along with the operators who will approve changes.
Named owners for Ceph, libvirt and the hypervisor hosts, plus change windows for test and production.

Ask the partner to list each dependency in the written scope so that missing inputs are visible before implementation.

Can a competent in-house team build this without a partner?

Yes, a team with Linux, storage and virtualization experience can build RBD for libvirt on its own, and the Ceph documentation covers the core steps. The areas that most often need extra attention are cephx capability design, keeping libvirt secrets consistent across hosts, choosing a librbd cache mode that matches your data safety needs, and proving live migration works under load. A partner is most useful when the team lacks recent Ceph operating experience or needs an independent validation before go-live. Compare the internal time cost against a fixed-scope quote.

How long does validation take under a fixed scope?

Validation length follows the scope you agree in writing, not a general rule. A fixed scope should name the hosts, pools, cache modes, migration tests and failure cases to be checked, and it should state the acceptance criteria for each. The duration then depends on how many of those tests need a full cluster, how much benchmarking is required, and how many change windows are available. No duration is set here; ask the partner to tie each test to a stated step and a stated owner.

Do Proxmox VE or OpenStack change the RBD method?

They change the surrounding tooling more than the storage principles. Proxmox VE and OpenStack each manage their own way of defining RBD storage, credentials and VM lifecycle, so the cephx users, pool layout and cache choices should be checked against that platform's configuration rather than copied from a plain libvirt setup. Live migration, snapshots and backups also depend on how the platform drives librbd. Confirm the exact integration version and its documented behavior with the platform vendor and your Ceph release before you decide the method.

How is the cost of this work quoted?

DeployOpen does not publish a price list. A quote follows from a written scope that lists the cluster readiness, number of hosts, cache and migration requirements, test cases and deliverables. Most engagements are fixed-scope, so the quote covers the agreed deliverables, while ongoing support or upgrades are priced separately as retainers. Ask for the quote to separate one-time build work, validation, documentation and handover from any recurring support. Ceph itself is LGPL-licensed with no paid edition from the project, so software licensing is not a cost line here.

Which Ceph release should a new RBD deployment target?

Target an actively maintained release. According to the Ceph releases page, Tentacle (20.2.x) and Squid (19.2.x) are the maintained series, and the page lists an estimated end of life for each. Tentacle's estimated end of life is June 1, 2027 and Squid's is October 31, 2026, so a deployment that runs past late 2026 should plan its release track. Confirm the current point release and upgrade path before the build starts.

How DeployOpen scopes and delivers a Ceph RBD for libvirt build

DeployOpen plans, builds, documents and supports Ceph RBD storage for KVM and libvirt virtual machines inside infrastructure you control. Each engagement starts from a written scope that covers cephx users, libvirt secrets, cache policy, pool layout and the live migration path your hosts need.

How we scope

Inventory your hypervisor hosts, libvirt versions, network paths and the VM workloads that will move onto RBD.
Agree which cephx users, pools and namespaces each host group needs, and document the capability limits in the written scope.
Record the cache mode, live migration requirements and QoS expectations per workload before any configuration starts.

How we build

Create the RBD pools, cephx keys and libvirt secret definitions on a lab or staging cluster before production hosts are touched.
Configure QEMU librbd disks and test live migration between hosts, including failure of a single host during a move.
Verify disk I/O, cache behavior and recovery under load against the benchmarks agreed in scope, and record the results.

How we hand over

Deliver the written configuration: key definitions, secret UUIDs, pool layout, libvirt domain templates and the rollback steps.
Walk your operations team through a live migration, a key rotation and a restore test, with runbooks for each step.
Document the open items and the settings that depend on your Ceph release, so your team knows what to recheck on upgrades.

How we support

Ongoing support, upgrades and testing are available as a retainer, and scope changes are agreed in writing before work starts.
Review cache settings, pool capacity and client versions when you add hosts, change Ceph releases or move to a new hardware generation.
Confirm with your team which support hours, escalation contacts and change windows apply before the retainer begins.

Ceph RBD for KVM and libvirt virtual machines