Ceph RBD primary storage for CloudStack: setup guide

How CloudStack consumes Ceph RBD as primary storage

If you run CloudStack KVM hosts and want Ceph block storage for VM root and data disks, the supported method is libvirt with librbd, with no separate CloudStack plugin to install. This page covers the dedicated pool and cephx user, adding RBD primary storage, disk offering tags and the monitor binding limits that differ between CloudStack versions. The split of responsibilities matters before you touch any of those settings. CloudStack decides which storage pool a volume belongs to and tells the hypervisor layer what to attach. libvirt holds the domain definition and the secret that unlocks the Ceph key, QEMU opens the disk through its RBD interface, and librbd on the host talks to the monitors and OSDs directly. The Ceph cluster itself knows nothing about CloudStack; it only sees authenticated client connections from the KVM hosts.

That layout has practical consequences. Every KVM host in the cluster needs network reach to the Ceph monitors and OSDs, and a working cephx key for the CloudStack user available to libvirt, so a host that cannot reach the cluster can still be scheduled a VM it cannot start. Failures can also originate on either side of the boundary. A CloudStack pool definition can be correct while a host's libvirt secret is stale, and a healthy host can still fail to place volumes if the disk offering does not match the pool tag. Confirm with your own environment which CloudStack release and host distribution you are targeting, and check the Ceph documentation for block devices and CloudStack for the exact configuration sequence for that release.

What you need before adding RBD primary storage

Adding Ceph RBD as CloudStack primary storage depends on four things lining up: the CloudStack release, the Ceph release, the hypervisor hosts and the network between them. Check each one before you create the pool or the storage in CloudStack, because a mismatch often shows up only when the first volume is created.

CloudStack version floor

The Ceph integration needs CloudStack 4.0 or later as a minimum. Your monitor configuration then depends on the exact release. Confirm the version you plan to run against the CloudStack downloads page, and note whether it is a regular release or an LTS line, because LTS releases receive bug and security fixes for a longer period.

Monitor list support by version

Through v4.17.1.0, CloudStack binds to one monitor at a time, so the documented workaround is round-robin DNS. Comma-separated monitor lists require CloudStack 4.18.0.0 or later. If you run an older release, plan a DNS name that resolves to your monitors and test a monitor failover before go-live.

Ceph release you run

Record the Ceph release your cluster runs and confirm it against the Ceph releases index before you choose client packages. Active releases receive backports and security fixes, and the client libraries on the hypervisors should match a release your cluster supports. Ask your Ceph administrator for the exact version rather than assuming it from the cluster name.

KVM hosts with libvirt and librbd

CloudStack reaches RBD through libvirt, which configures QEMU's interface to librbd, so each KVM host needs libvirt and a librbd build compatible with your Ceph release. Check that the host distribution is supported by your CloudStack version and that the Ceph client packages are installed on every host, including hosts added later.

Hypervisor-to-Ceph network paths

Every KVM host must reach the Ceph monitors on their listed ports and reach every OSD that serves the pool it will use. Verify routing, MTU and firewall rules from each hypervisor, not only from one test host. Confirm with your network team whether the storage traffic uses a dedicated VLAN and whether it crosses any firewall that inspects or limits traffic.

MySQL-compatible database

CloudStack stores its management state in a MySQL-compatible database, either MariaDB or MySQL. Confirm the version your CloudStack release documents, plan backups of that database, and check that the management server can reach it with the required privileges. Loss of this database is separate from Ceph data loss, so back it up on its own schedule.

Configure the Ceph pool, cephx user and CloudStack storage in order

This sequence takes a Ceph cluster from an empty state to a CloudStack zone where a dedicated RBD pool, a restricted cephx user and a tagged disk offering place VM disks on the intended pool.

Create a dedicated RBD pool
A Ceph administrator creates a pool used only by CloudStack, sets its replication or erasure coding to match the cluster's failure domains, and initializes it for RBD with the rbd application tag. Keeping the pool separate lets you apply capacity limits, quotas and placement rules without affecting other workloads. The step is complete when the pool appears in the cluster status and an rbd ls against it returns an empty list without errors.
Create a restricted cephx client
The administrator creates a cephx user whose capabilities grant access to the CloudStack pool and to nothing else, then exports its keyring. Ask your Ceph team to confirm the exact capability strings for their release, because the monitor and OSD caps differ between versions and between RBD and CephFS use. The step is complete when the user's capabilities listing shows only the intended pool, and a test authentication from a KVM host with that keyring succeeds.
Distribute keyring and ceph.conf to KVM hosts
Copy the client keyring and a minimal ceph.conf listing the monitor addresses to every KVM host that will run CloudStack VMs, with file permissions limited to the service account that libvirt and the CloudStack agent use. Confirm that each host can reach the monitor ports on the public network, since the Ceph messenger uses port 3300 for msgr2 and the legacy port 6789 for older clients. The step is complete when ceph status run with that keyring succeeds on every host.
Define the libvirt secret on each host
On each KVM host, the operator defines a libvirt secret for the cephx user, sets its value from the keyring, and records the secret UUID that CloudStack will use when it attaches disks. Use the same UUID on every host where the pool is available, because CloudStack refers to the secret through the pool configuration. The step is complete when virsh secret-list shows the secret as a Ceph usage entry and virsh secret-get-value returns the stored key.
Add RBD primary storage in CloudStack
A CloudStack administrator adds a primary storage pool with the RBD protocol, entering the pool name, the cephx user, the libvirt secret UUID and the monitor settings. Check the CloudStack version before you enter monitors: until v4.17.1.0 CloudStack binds to one monitor at a time, with round-robin DNS as the workaround, and comma-separated monitor lists need v4.18.0.0 or later. The step is complete when the pool shows as Up in the zone and the host connection test passes on every KVM host in the cluster.
Create a disk offering with a matching rbd tag
The administrator creates a disk offering whose storage tag matches the rbd tag on the new primary storage pool. Without the matching tag, the storage pool allocator can place volumes on another pool, so this check matters most in zones with several primary stores. The step is complete when the offering lists the tag and a volume created from it reports the CloudStack pool as its location.
Deploy a test VM onto the intended pool
Deploy a small test instance from a template using the new disk offering, then confirm on the Ceph side that its root disk exists as an image in the CloudStack pool, for example with rbd ls on the pool. Start, stop and delete the instance once to confirm the full volume lifecycle works. The step is complete when the image appears in the expected pool and is removed after the instance is destroyed, with no leftover images or errors in the CloudStack management server log.

How monitor binding limits change the design by CloudStack version

Until CloudStack v4.17.1.0, CloudStack binds to one Ceph monitor at a time when it uses an RBD primary storage pool. If that monitor becomes unavailable, the hypervisor side keeps working only as long as the address CloudStack holds still resolves to a healthy monitor. The documented workaround before that release is round-robin DNS: publish one name that resolves to several monitor addresses so a reconnect can land on a different monitor. Before you commit to that design, confirm the exact monitor address and DNS settings your CloudStack zone will store, test name resolution from every KVM host, and rehearse a monitor outage in a lab so you know how long your VMs and management plane tolerate a stalled connection.

CloudStack 4.18.0.0 and later accept a comma-separated list of monitors, which removes the dependence on DNS rotation for failover. For a new deployment on a current release, list every monitor the cluster exposes and treat DNS as an optional convenience rather than the failover mechanism. For an existing deployment on 4.17.1.0 or earlier, compare two options: keep the round-robin DNS workaround and document its limits, or plan a CloudStack upgrade that also moves the monitor list into the pool definition. An upgrade is usually the simpler fix when the DNS workaround needs extra records, monitor addresses change often, or the team cannot test failover convincingly. Confirm the upgrade path, required database and agent steps, and any Ceph client compatibility with the CloudStack release notes before scheduling it.

The version you run also decides how long you get fixes. The CloudStack downloads page lists 4.22.1.1 as the current LTS release and 4.20.3.1 as the latest maintenance release of the 4.20 LTS line, while 4.23.0.0 is a regular release. LTS releases receive bug and security fixes for 18 months after the main release, whereas regular releases get only critical bug fixes for a short period. Choose the line first, then pick the monitor configuration that line supports. For production storage, an LTS line is usually the safer base because it keeps the monitor binding behavior you tested under fix coverage. Check the support end date for your chosen line on the CloudStack downloads page before you design around it.

Why disk offerings must carry the rbd tag

When CloudStack has only one RBD primary storage pool, the tag requirement is simple but still easy to miss. A disk offering that does not carry the rbd tag gives the storage pool allocator no reason to prefer the Ceph pool, so it may place a volume on another primary storage pool that happens to have room. In a single-pool design, one tagged offering per service tier is usually enough, and the tag name only needs to be consistent across the offerings and the storage pool.

One tag, one pool: Use a single tag value, such as rbd, on every offering that should land on Ceph.
Offering count: Expect one offering per performance or size class you sell, not one per pool, because there is only one pool to match.
Untagged offerings: Identify any existing offering without the tag and decide whether it should be retired or retagged before users select it.
Testing: Create a test instance from each tagged offering and confirm in the CloudStack management interface that its root and data volumes sit on the RBD pool.
Ongoing checks: Recheck placement after adding local or NFS primary storage, since new pools can change allocator behavior.

Confirm the exact tag handling for your CloudStack version in the Ceph and CloudStack documentation before you publish offerings to users.

CloudStack settings mapped to Ceph objects

This table maps each CloudStack primary storage field, disk offering tag and libvirt secret attribute to the Ceph pool, CephX user, keyring and monitor addresses it refers to. Read the Match column first: exact matches cause most failed pool attachments, while free labels only need to be consistent within your CloudStack configuration.

CloudStack or libvirt setting	Ceph object it references	Match requirement	Notes for the operator
Primary storage pool name (RBD)	Ceph pool name (for example, the dedicated RBD pool)	Must match the pool created in Ceph	The pool should be used by CloudStack only so capacity reporting stays accurate.
Storage URL monitor address	Ceph monitor address or addresses	Must resolve to reachable monitors on hypervisor hosts	Before CloudStack 4.18.0.0, use one monitor or round-robin DNS; comma-separated lists need 4.18.0.0 or later.
Storage user name	CephX user name without the client. prefix	Must match the CephX user granted access to the pool	Grant that user only the capabilities the pool requires; confirm the exact capability syntax in the Ceph documentation.
libvirt secret UUID	CephX key stored in the libvirt secret	Must match the UUID used when defining the libvirt secret on each hypervisor	The same UUID should be used on every KVM host in the cluster so the attach process finds the secret.
Keyring file on hypervisor hosts	CephX keyring for the storage user	Keyring contents must match the user and key in Ceph	Confirm the keyring path and file permissions on each host with your Ceph and KVM administrators.
Disk offering storage tag	Primary storage pool tag (rbd)	Must match exactly	If the tag and pool tag differ, the allocator may place volumes on another pool.
Primary storage tag label	Free label chosen by the operator	Free label, must be consistent across the pool and offerings	Choose a descriptive name so administrators can tell Ceph pools apart in CloudStack.
Storage pool display name	Free label chosen by the operator	Free label	Used only in the CloudStack interface and API; it does not need to match any Ceph name.

Securing the cephx user, keyrings and storage network

CloudStack hosts hold the credentials that let them read and write every VM disk in the pool, so access and network design matter as much as pool layout. The cards below cover the cephx identity, the files that store its key, the network paths Ceph traffic uses, and the encryption option for data in transit. Confirm each setting against the Ceph release you run before you apply it.

A cephx user scoped to one pool

Create a dedicated cephx client for CloudStack and grant it access only to the RBD pool it uses for primary storage. A client limited to one pool cannot read or modify images belonging to other applications on the same cluster. Confirm the exact capabilities syntax for your Ceph release in the Ceph documentation before you write the caps, and test that the client can still create, map and delete images in its pool.

Keyring file permissions on KVM hosts

Libvirt needs the cephx secret on every KVM host that runs CloudStack VMs, so the keyring is copied to each hypervisor. Restrict the keyring file to root or the service account that reads it, and avoid world-readable modes. Record which hosts hold the key so that a rotation or host decommissioning removes the secret everywhere it was placed.

Avoiding shared admin keys

Do not give CloudStack the cluster admin keyring, even when it is the fastest way to get a working pool. An admin key lets any holder create users, change pools and alter cluster settings, which turns a compromised hypervisor into a cluster-wide incident. Keep one client identity per integration so that rotating or revoking the CloudStack key does not affect monitoring, backups or other platforms.

Separating public and management traffic

Ceph clients and OSDs exchange data on the public network, which should be a dedicated path from the KVM hosts to the monitors and OSDs. Keep CloudStack management, API and console traffic on separate interfaces so that heavy VM I/O does not contend with control-plane calls. Ask your network team to confirm bonding across two switches for the storage links, as the Ceph hardware guidance recommends.

Enabling msgr2 secure mode

The messenger v2 protocol listens on port 3300, and its secure mode encrypts all traffic after authentication with AES-GCM and adds integrity checks. Enabling it protects VM disk I/O crossing the storage network, though it adds CPU cost on hosts and must be tested with every client version, including libvirt on the hypervisors. Confirm the secure mode settings for your release and check client compatibility before enabling it cluster-wide.

Common CloudStack and Ceph RBD failures and fixes

This table lists the failures teams most often hit when wiring CloudStack to Ceph RBD primary storage, with the likely cause and the first check for each. Read it when a pool shows as unreachable, volumes land on the wrong pool, or hosts cannot attach disks, and start with the first check before changing configuration. Most of these problems trace back to monitor configuration, tag matching, or secrets and keyrings that differ between the management server and the KVM hosts.

Symptom	Likely cause	First check
Primary storage pool not reachable after adding RBD	CloudStack on a version before 4.18.0.0 was given a single monitor, and that monitor is down or unreachable from the host	Confirm the CloudStack version, then test TCP reachability from each KVM host to the monitor on port 6789 or 3300
Pool becomes unavailable when one monitor fails	Before v4.17.1.0 CloudStack binds to one monitor at a time, so there is no failover within the pool definition	Check whether round-robin DNS resolves the monitor name to several addresses, and plan an upgrade to 4.18.0.0 or later for comma-separated monitor lists
Volumes created on an unexpected pool	The disk offering has no rbd tag, or its tag does not match the tag on the intended RBD pool, so the storage pool allocator chose another pool	Compare the tag on the disk offering with the tag on each primary storage pool in the CloudStack UI or API
Volumes fail to allocate with no suitable pool	The disk offering tag matches a pool that is in maintenance, disabled, or has insufficient capacity	Check the state and free capacity of the tagged pool, then retry allocation for a new volume
libvirt cannot open the RBD disk, authentication error	The libvirt secret UUID or cephx key on the KVM host does not match the cephx user CloudStack was configured with	List the libvirt secrets on the host, compare the UUID with the one CloudStack stores, and confirm the key with the cephx user on the cluster
Keyring permission denied on the hypervisor	The keyring file for the CloudStack cephx user is readable only by root or has the wrong owner for the process that opens it	Check file ownership and mode on the keyring and the Ceph configuration file on each KVM host, then test access with the same cephx user
Host can reach monitors but I/O stalls or times out	MTU mismatch on the storage network, so large RADOS packets are dropped while small control packets pass	Ping between hosts and monitors with the largest expected payload and the do-not-fragment flag set
Connections to OSDs refused or intermittent	A host firewall or switch ACL blocks the OSD port range on the storage network, while monitor ports are open	Confirm that the OSD port range and monitor ports are open between every KVM host and every OSD host on the storage network

Running the integration after the first VM is live

The first VM booting from RBD proves very little about the next year. Day-two work for this integration centers on three things: watching pool capacity and cluster health in Ceph, proving that CloudStack survives a monitor outage, and sequencing upgrades so the two systems never drift outside a version pair both vendors support. Set alerts on the RBD data pool's used capacity and on Ceph health warnings, and confirm with your Ceph administrator which thresholds trigger expansion before the pool approaches full. Because the storage pool allocator places disks by disk offering tags, also check periodically that each rbd tag still maps to the pool you intended, especially after a pool is added or renamed.

Monitor failover needs its own test. Before CloudStack 4.18.0.0, the integration binds to one monitor at a time, and the documented workaround is round-robin DNS, so the behavior depends on how your DNS entry resolves and what happens when a monitor host drops. After CloudStack 4.18.0.0, a comma-separated monitor list is supported. Schedule a controlled monitor restart in a maintenance window, then confirm that running VMs keep I/O, that new volume creation succeeds, and that the CloudStack management server still reports the primary storage as up. Record which CloudStack version you run, since that determines which of these two behaviors you are testing.

Upgrades and key rotation are where this integration most often breaks. Plan CloudStack and Ceph upgrades as separate, ordered changes with a rollback point for each, and confirm the supported combinations with both the Ceph and Apache CloudStack documentation before you move either side. Rotating the CephX key used by CloudStack means updating the secret on every KVM host and in the CloudStack primary storage configuration, so schedule it as one change with a verification step that creates and deletes a test volume. Ask your team to confirm the exact rotation procedure with both vendors before the first rotation, not during an incident.

Questions buyers ask before connecting CloudStack to Ceph

What drives the cost of connecting CloudStack to an existing Ceph cluster?

The cost follows the number of hypervisor hosts, the number of CloudStack zones and clusters that will use RBD primary storage, and how much of the Ceph side already exists and is documented. Each additional KVM host needs libvirt and librbd configured against the same cephx user and pool, so the host count drives configuration and testing time. Cost is mostly one-time: design, configuration, verification and handover. Recurring costs appear only if you buy ongoing support or a retainer. Ask any provider for a quote that lists hosts, pools, CloudStack versions and the test plan separately, so you can see which items scale with your environment.

How many monitors and network paths does the CloudStack connection need?

Monitor count is set by your Ceph cluster design, not by CloudStack, but the CloudStack version decides how you can reference it. Up to and including v4.17.1.0, CloudStack binds to one monitor at a time, and round-robin DNS is the documented workaround. Comma-separated monitor lists need CloudStack 4.18.0.0 or later. Hypervisors must reach every monitor and the OSD network that serves their I/O, so confirm routing, firewall rules and MTU on the storage network before you add the pool. Check the Ceph documentation for the monitor and network requirements of your Ceph release.

Can our team set up CloudStack with Ceph using the official documentation alone?

A team with Ceph and KVM experience can usually follow the official Ceph guide for CloudStack, which covers creating a dedicated pool, a cephx user, libvirt secret configuration and adding RBD primary storage. The steps are documented, but the decisions behind them are not: pool sizing, placement group planning, how disk offerings map to pools, and how monitor binding behaves in your CloudStack version all need judgment. Teams without recent Ceph operations experience often benefit from an external review of the design before go-live. Your own engineers remain the best judges of whether they have the time and skills.

What does our team need to provide before the work starts?

The work needs administrative access to the CloudStack management server, the KVM hosts and the Ceph admin interface, or a documented way for the team to make changes under supervision. You should also provide the CloudStack and Ceph versions in use, the network layout for monitors and hypervisors, and an agreed change window for each host. A test pool, separate from production volumes, is essential so that disk offering tags, live migration and VM creation can be verified before any production workload moves. Confirm who approves changes and who holds the rollback decision.

How long does a CloudStack and Ceph RBD integration take?

Duration depends on scope: the number of hosts and clusters, whether the Ceph cluster already exists, how many CloudStack zones are affected, and how much testing your change process requires. No fixed duration applies to every environment, so DeployOpen does not quote one here. Work is typically fixed-scope and agreed in writing before it starts, with deliverables and timelines stated in that document. Ask for the written scope and timeline, and check that it names the CloudStack version, the test plan and the acceptance criteria.

Why does a disk offering need to match the rbd tag?

A disk offering must carry the same tag as the RBD storage pool, or CloudStack's storage pool allocator may place volumes on another pool. This is a configuration rule, not a performance tuning choice, and it is easy to miss when several pools exist. Plan your offering and tag names before you add primary storage, record them, and test that a new volume lands on the intended pool. Changing tags later means reviewing existing volumes and offerings, so decide the naming scheme during design.

What is not included when DeployOpen connects CloudStack to Ceph?

The integration covers configuring CloudStack primary storage on top of a Ceph cluster, including the dedicated pool and cephx user, libvirt and librbd configuration on the hypervisors, disk offering tags and verification. It does not include designing or building the Ceph cluster, selecting hardware, or installing CloudStack itself, and it does not include a CloudStack management server build or the migration of existing guest VMs unless the written scope says so. DeployOpen does not issue audit reports or certifications. Those items need separate scope, or they should be handled by your existing teams or another provider.

How do we keep the cephx credentials for CloudStack secure?

Create a dedicated cephx user for CloudStack with access limited to the single pool it uses, rather than reusing the admin key. Store the key in libvirt secrets on each hypervisor, restrict who can read the management server database, and record the key rotation process before go-live. Review the Ceph security guidance for your release, including client authentication and encryption in transit, and confirm that your hypervisor network segment is protected from untrusted hosts.

How DeployOpen scopes and delivers CloudStack and Ceph storage work

DeployOpen approaches Ceph RBD primary storage for Apache CloudStack as a written, fixed-scope engagement inside your own infrastructure. We plan the Ceph pool, CephX user, CloudStack configuration and monitor layout together, then test and document each step so your team can run the result.

How we scope

We confirm your CloudStack and Ceph versions, hypervisor hosts and whether monitor binding limits affect your release.
We define the RBD pool, CephX user, disk offering tags and primary storage cluster scope in writing.
We agree deliverables, exclusions and timelines before any work starts, since engagements are fixed-scope.

How we build

We configure libvirt and librbd access on KVM hosts and verify the Ceph client path before adding primary storage.
We map disk offerings to the rbd tag so the storage pool allocator places volumes in the intended pool.
We test VM creation, migration, snapshot and deletion against the RBD pool, and record the results.

How we hand over

We document the pool layout, CephX capabilities, monitor list, offering tags and the reasons behind each choice.
We deliver runbooks for adding hosts, rotating keys and checking Ceph health from the CloudStack side.
We walk your administrators through the configuration and the acceptance tests so they can repeat them.

How we support

Ongoing support, CloudStack upgrades and Ceph upgrades are available as a retainer after handover.
We can plan CloudStack LTS and Ceph release upgrades as separate written scopes, with testing before each change.
Continuous testing and security checks of the storage path are available under the same retainer model.

Ceph RBD primary storage for Apache CloudStack