How CephFS subvolumes back Kubernetes and Manila shares
The ceph-mgr volumes module provisions the volumes, subvolume groups and subvolumes that both Ceph-CSI and OpenStack Manila use to hand out CephFS shares. Teams need this layer whenever Kubernetes workloads or OpenStack tenants request shared file storage through an API and expect each claim or share to be created, sized, snapshotted and deleted without an administrator touching the file system by hand. The volumes module is the source of truth for those objects, so the CSI driver and Manila both read and write through it rather than managing raw directories on their own.
Raw CephFS directories still make sense for a fixed set of shared paths managed by one team, such as a lab scratch area or a small set of application mounts that never change. Once several tenants, namespaces or clusters need isolated paths, per-share size limits and independent snapshot lifecycles, the subvolume model keeps that bookkeeping in one place. Before committing, confirm which Ceph release your cluster runs and which Ceph-CSI and Manila versions you plan to deploy, because the supported combinations and the subvolume features they depend on are documented per release in the Ceph FS volumes documentation.
Plan the layer in the design phase, not after the first tenant arrives. Decide how subvolume groups map to Kubernetes clusters or OpenStack projects, who holds the credentials that Ceph-CSI and Manila use to authorize against the volumes module, and which team owns quota changes. Changing the grouping later means migrating shares that workloads already mount, which is far more disruptive than setting the layout correctly at the start.
What the volumes module manages: volumes, groups and subvolumes
The ceph-mgr volumes module is the single place where CephFS exports are defined, and both OpenStack Manila and Ceph-CSI ask it to create the shares your workloads mount. Understanding its three levels is the starting point for any design: an FS volume is the file system itself, a subvolume group is a named container inside a volume, and a subvolume is the directory that a client actually mounts. The module keeps one record of each of these objects, so a share created through Manila or through a Kubernetes StorageClass appears in the same model and can be listed, resized, snapshotted and removed through the same interface. Confirm your Ceph release's volume commands and output with the FS volumes documentation before you script against them, since command names and options are the parts most likely to change between releases.
Subvolume groups matter because they are where you apply settings that should be shared by many shares. A common pattern is one group per tenant, per Kubernetes cluster or per Manila back end, so that ownership, permissions and snapshot policy can be reasoned about at the group level rather than subvolume by subvolume. Group boundaries also make cleanup and reporting simpler: a team that retires a cluster can see which subvolumes belong to it. Decide the group layout before the first share is created, because moving existing subvolumes between groups is not a step to plan casually. Ask your storage team to write down the naming scheme and who may create groups, since Manila and the CSI driver each create subvolumes under the group you name in their configuration.
The size of a subvolume is set by a quota. A subvolume created without a quota has no size limit, so it can grow until the cluster runs out of space, and that affects every other tenant on the same pool. Treat the quota as a required part of every provisioning request rather than an optional tuning value, and confirm that your Manila share types or CSI StorageClass parameters pass a size to the module in every case. Quotas limit what the share can consume, but they do not reserve capacity on the cluster, so check the pool's total space and your monitoring thresholds as well. Test the behavior in a non-production file system by creating a subvolume with and without a quota and checking what a client sees, because the reported size and the enforced limit are what your applications and capacity planning will rely on.
Subvolume settings and when each fits
This table compares the CephFS volumes settings that shape how shares are provisioned for Kubernetes CSI and OpenStack Manila. Read each row as a decision: the option, what it isolates or limits, and which consumer usually suits it. The main takeaway is that a quota is the only thing that gives a subvolume a size limit, so size controls must be designed in from the start.
| Setting | Kubernetes CSI fit | Manila fit | Trade-off to confirm |
|---|---|---|---|
| Subvolume group per cluster | Shared group for many PVCs from one Ceph-CSI storage class | Common group for shares from a single Manila backend | Simple to run, but weaker separation between tenants sharing the group |
| Subvolume group per tenant | Separate group per namespace or team when storage classes differ | Separate group per OpenStack project or tenant | Clearer isolation and reporting, more objects for administrators to manage |
| Quota set at creation | Size requested in the PVC maps to a quota on the subvolume | Share size in gigabytes maps to a quota on the share | Without a quota the subvolume has no size limit, so confirm the CSI and Manila paths both set one |
| Subvolume mode and permissions | Mode and ownership must match the pod security context of the workload | Mode fits the guest or client access model for the share | Confirm exact mode options for your release before relying on them |
| Snapshots | Used for VolumeSnapshot and point-in-time copies of a PVC | Used for share snapshots before changes or restores | Snapshots are tied to the source subvolume, so plan retention and quota growth together |
| Clones from snapshots | Create new PVCs from a snapshot, such as for test copies | Create new shares from a snapshot, such as for a migrated tenant | Clones are asynchronous and usable only after completion, so workloads must wait for the clone to finish |
| Namespace isolation through the volumes module | Keeps CSI-provisioned shares in a defined area of the file system | Keeps Manila shares managed through one source of truth | Isolation depends on group layout and client capabilities, which you should verify in your environment |
How Ceph-CSI and Manila each use subvolumes
Ceph-CSI's CephFS driver asks the volumes module to create a subvolume for each PersistentVolumeClaim, so Kubernetes sees a dynamically provisioned volume while Ceph holds the directory, quota and snapshot state. The driver authenticates with a Ceph user whose capabilities should be scoped to the file system and the subvolume group it writes into. The cluster administrator must define the cluster connection that the StorageClass refers to, and the platform team must keep those two sides consistent.
- StorageClass parameters: Each class identifies the Ceph cluster, the file system and, where used, the subvolume group and data pool. Confirm the exact parameter names in the Ceph-CSI release you install, because they change between releases.
- Access modes: Shared access across pods is the main reason teams choose CephFS over block storage. Confirm which access modes your driver version offers for each class before workloads depend on them.
- Expansion through quota changes: Resizing a claim changes the subvolume quota, which is the size limit. Growing is a metadata change, while shrinking requires the data to fit under the new limit, so plan it as a deliberate operation.
- Credentials as Secrets: The Ceph user key lives in a Kubernetes Secret. Secrets are stored unencrypted in etcd by default, so enable encryption at rest and restrict list and watch access with least-privilege RBAC, as described in the Kubernetes Secrets good practices guide.
- Snapshots and clones: Volume snapshots map to subvolume snapshots. A restored clone is not usable until the asynchronous clone completes, so readiness checks in your restore runbook should wait for it.
Before production, confirm the driver version against your Kubernetes version, then test StorageClass behavior, Secret scope and clone timing in a non-production namespace.
Manila's CephFS driver maps each share to a subvolume, so a share's size, snapshots and clones are controlled by the same volumes module that serves Kubernetes. Manila runs inside OpenStack, where Keystone handles identity, so tenant isolation depends on how share types, share networks and access rules are assigned to projects. The cloud team owns the policy and the storage team owns the Ceph side, and both must agree on who can request what.
- Share types: Define a share type per backend and capability, such as the file system, the subvolume group and whether snapshots are allowed. Tenants choose from types, and operators decide what each type exposes.
- Share networks: A share network connects a share to a tenant network segment so clients can reach the Ceph metadata and data services. Confirm routing, firewall rules and MTU with the network team before the first share is created.
- Access rules: Access rules grant clients access through a Ceph identity or path, and each rule creates or updates a Ceph client with limited capabilities. Review who can add rules, because that decides who can mount a share.
- Tenant isolation: Scope each tenant to its own subvolume group or path, and never rely on share names for separation. Keystone federation maps external users to projects through mapping rules, so review those rules as part of isolation.
- Size and snapshots: Share size maps to a quota, so a share without a size limit is not a valid end state for a multi-tenant cloud. Confirm snapshot and share-from-snapshot behavior for your Manila release, since clones complete asynchronously.
Verify with a test tenant that one project cannot list, mount or read another project's share, and that revoking an access rule removes the client's access.
Kubernetes and Manila both create subvolumes in one volumes module, so a decision made for one consumer affects the other. The Ceph FS volumes documentation describes the module as the single source of truth for CephFS exports, which is why shared rules matter. These checks belong in the platform runbook, owned jointly by the platform team and the cloud team, and they apply before the first claim or share exists.
- Subvolume group layout: Separate groups per consumer or per tenant make quotas, snapshots and client key scope easier to reason about. Decide the layout before provisioning starts, because changing it later means migrating data.
- Quota discipline: Size comes from quotas, so a subvolume without one has no size limit. Make a quota a required part of provisioning for both consumers and check for missing quotas during reviews.
- Manual edits: Avoid changing subvolumes by hand when Kubernetes or Manila manages them, because the module is the shared record that both consumers rely on.
- Version alignment: Ceph, Ceph-CSI, Manila and Kubernetes each move on their own schedule. Check the compatibility guidance for each pair before an upgrade, and confirm which Kubernetes minor versions are in maintenance according to the project's release page.
- Monitoring: Alert on quotas approaching their limits and on clones that stay incomplete, since both problems reach users late if nobody watches them.
Write these rules into the runbook and confirm that both teams apply them the same way during a test restore.
Snapshots and asynchronous clones: what is usable and when
Clones in CephFS are created asynchronously from snapshots, and a clone is usable only after it completes. That single rule shapes every restore workflow built on top of it. A Kubernetes VolumeSnapshot restore or a Manila share-from-snapshot request can return a new object before its data is ready to mount, so the restore is not finished when the API call succeeds. Your runbooks, health checks and application cutover steps should treat clone completion as a separate gate. Confirm with your Ceph-CSI and Manila driver versions how each surfaces clone state, and whether the consuming pod or VM will start against a clone that is still in progress.
Before cutover, test completion directly rather than trusting the orchestration layer's status. On the Ceph side, check the clone state of the subvolume with the volumes manager commands described in the FS volumes documentation, then mount the clone read-only and verify file counts, sizes and a sample of application data against the source snapshot. Run this test on a clone created from a representative snapshot in a non-production subvolume group. Record how long the clone took so you can plan maintenance windows, and set alerts for clones that stay in progress past that baseline.
Large clones need extra attention. Clone work consumes cluster I/O while it runs, so schedule large restores away from peak client load and watch OSD latency and client request times during the copy. Confirm the quota on the target subvolume is set to the size you expect the restored data to need, because a subvolume's size comes from its quota and a clone that does not fit can fail or leave the workload short of space. If an application must not start until the clone is complete, block its startup in your deployment automation instead of relying on a timer. Keep the snapshot until the restored workload has run cleanly for a period you agree with the data owner.
Setup sequence for CSI and Manila on CephFS
This sequence takes a Ceph cluster from a confirmed release and deployment path to a tested CephFS backend that Kubernetes CSI and OpenStack Manila can provision, snapshot, clone and delete shares from. Each step ends with a check you can run before moving on.
Confirm the Ceph release and deployment path
Confirm which Ceph release the cluster runs and whether it is managed by cephadm or by the Rook operator. Check the Ceph releases page for the current maintained releases and their end-of-life dates, and confirm that the Ceph-CSI or Manila driver versions you plan to use support that release. Record the cluster's CephFS status with the ceph fs status command on a cluster with a running MDS. The step is done when the release, orchestrator and driver versions are written down together and the cluster reports HEALTH_OK or a warning you have explained.
Create the FS volume and subvolume group
Using the ceph-mgr volumes module, create the file system volume that both consumers will share, then create one subvolume group per consumer or tenant boundary, such as one for Kubernetes and one for Manila. Groups are the level where you set shared policy, so decide the names and layout before the first share exists, because renaming them later means migrating data. The ceph fs volume ls and ceph fs subvolumegroup ls commands show what was created. The step is done when both groups are listed under the volume and the MDS daemons are active.
Set a quota on each subvolume
Plan a size limit for each subvolume and set it as a quota when the subvolume is created or resized. A subvolume without a quota has no size limit, so this step matters for every provisioning path, including PVCs created on demand. Ask your storage owner which default sizes apply to each consumer, and confirm how the CSI storage class or Manila share type passes size through to the subvolume. The check is a test subvolume reporting the expected size with ceph fs subvolume info, and a write past the limit failing as intended.
Create a scoped CephX client
Create a dedicated CephX client for each consumer with capabilities limited to the file system, the subvolume group and the data pool it needs, rather than reusing the admin key. The Ceph-CSI and Manila drivers each need their own identity, and the MDS, OSD and monitor capabilities should be written to the narrowest set the driver documents for your release. Store the key in your secrets system, not in a plain manifest. The check is that the new client authenticates from a test host and that it can list the intended group but not the others.
Configure the Ceph-CSI or Manila driver
On the Kubernetes side, install the Ceph-CSI driver and create a CephFS storage class that points to the cluster identifier, the file system name, the subvolume group and the scoped CephX secret. On the OpenStack side, configure the Manila CephFS driver with the same file system, group and client, then create a share type that matches the backend. Confirm the Ceph monitor addresses and network path from the worker nodes and the Manila share servers first. The check is that the CSI driver pods and the Manila share service report ready, with no authentication errors in their logs.
Test provisioning from both consumers
Create a test PersistentVolumeClaim with the new storage class and a test share with the new share type. Mount the Kubernetes volume in a pod with the ReadWriteMany access mode you intend to use, and mount the Manila share on a test instance. Write a file from each side and confirm it appears on the other, which proves the subvolume is shared correctly. Then confirm that the subvolume appears under the expected subvolume group and that its quota matches the request. The step is done when both tests succeed and the subvolumes are visible in ceph fs subvolume ls.
Test snapshot, clone and deletion
Take a snapshot of the test subvolume from each consumer, then create a clone from that snapshot. Clones are created asynchronously and are usable only after the operation completes, so poll the clone status until it reports complete before mounting it. Test deletion of the clone, the snapshot and the original subvolume in that order, and confirm the space is released. Record the clone and deletion behavior in your runbook, since the same flow will run during restores and ticket cleanups. The step is done when the clone status shows complete, the cloned data matches the source, and the deleted objects no longer appear in the subvolume listing.
Document the baseline and hand over
Record the release, driver versions, CephX capabilities, subvolume group names, quota defaults and the test results in a configuration baseline that the platform and storage teams both keep. Ask your DeployOpen or internal team to confirm the upgrade order for Ceph, the CSI driver and the Manila service, since version compatibility between them drives the change plan. Set up monitoring for cluster health and MDS status before production consumers arrive. The step is done when the baseline is approved by both teams and the monitoring alerts have been tested once.
Security and operational considerations for shared subvolumes
A shared CephFS subvolume serves many consumers, so access, isolation, capacity and upkeep decisions made at provisioning time decide how safely the platform runs later. The cards below cover the controls to set on the Ceph side and the operational habits that keep Kubernetes and OpenStack consumers from affecting each other.
Scoped CephX users per consumer
Give each Kubernetes cluster, CSI driver deployment or Manila back end its own CephX user, limited to the volumes, subvolume groups and paths it needs. Avoid reusing the admin key for provisioning or for mounts, because a leaked shared key exposes every share on the cluster. Review the capabilities of each user when a consumer is added, changed or retired, and confirm the exact capability strings with the Ceph-CSI and Manila documentation for your release before you write them.
Network isolation between tenants
Subvolumes are reached over the public network, so tenants that share a Ceph cluster must not share an unrestricted path to the monitors and metadata servers. Place Kubernetes nodes and OpenStack compute or share-service hosts on networks that can reach only the Ceph addresses they need, and keep the back-end storage network, which carries OSD replication, off client segments. Confirm firewall rules and VLAN design with your network team, since the pattern depends on your environment.
Quota enforcement and noisy-neighbor control
A subvolume has no size limit unless it has a quota, so every provisioned share should carry an explicit quota set at creation time. Check that the CSI storage class or Manila share type passes a size value through to the quota, and test that a write beyond the limit fails as expected. Quotas limit capacity but not throughput, so decide how you will watch metadata load and client I/O when one tenant generates heavy traffic.
Snapshot retention and cleanup
Snapshots and clones consume space until someone removes them, and clones only become usable after their asynchronous copy completes. Set a retention rule for each share class, delete snapshots that no longer back a restore point, and check that a failed PersistentVolumeClaim or Manila share deletion does not leave orphaned subvolumes or snapshots behind. Ask your provider or internal team to document who owns cleanup and how it is audited.
Upgrade alignment across the stack
The Ceph release, the Ceph-CSI or Manila release and the Kubernetes or OpenStack release must be chosen together, since each pairing has its own compatibility matrix. Upgrade the Kubernetes control plane first and keep kubelets within the supported version skew, and confirm the matching Ceph-CSI or Manila version before moving the Ceph cluster. Record the pairings in a change plan so that a Ceph upgrade does not surprise a CSI driver or share service.
Logging and monitoring of provisioning failures
Provisioning failures often appear first as pending claims or stuck share creation rather than as Ceph health warnings. Collect logs from the CSI provisioner and node plugins, the Manila share service and the ceph-mgr volumes module, and alert on failed creates, failed deletes and clones that stay incomplete. Keep these logs in your central logging platform so that a failed request can be traced from the Kubernetes or OpenStack event to the Ceph operation.
Questions buyers ask before hiring help with CephFS subvolumes
How much effort does a CephFS subvolume setup take, and what drives it?
Effort depends less on creating a single subvolume than on how many tenants, consumers and release paths the design has to cover. Each tenant usually needs its own subvolume group or subvolume layout, quota, client capabilities and snapshot policy, so the count of tenants multiplies the testing and documentation work. Running Ceph-CSI for Kubernetes and Manila for OpenStack on the same cluster adds a second set of provisioning paths to verify. Upgrades add more work: the Ceph release in use, the Kubernetes release line and the OpenStack release each need a compatibility check before a change is scheduled. Ask for the tenant count, the consumer mix and the upgrade path in writing before you compare estimates.
Can our own team set up CephFS subvolumes without outside help?
Yes, if your team already runs Ceph and owns the consumer platform. The volumes manager module that provisions subvolumes is part of Ceph, so a team with cluster experience can create volumes, subvolume groups and subvolumes with the documented commands. The harder parts are the decisions around them: how quotas map to tenant sizing, how Ceph-CSI and Manila credentials are scoped, how snapshots and clones are retained, and how the setup is tested against each consumer. Teams without recent Ceph operations experience, or with no one who owns the Kubernetes or OpenStack side, often use an outside review to settle those decisions before building them.
What do we need to provide from our side for this work?
The work needs three groups of people and one set of data. Ceph administrators provide access to the cluster, the current release and the monitoring setup. Kubernetes or OpenStack owners provide the cluster configuration, storage classes or Manila share types, and the identity model used for tenants. Security or platform leads confirm credential handling and who approves changes. For testing, you need representative data sets or synthetic data of realistic size, plus a non-production environment that matches the production Ceph release. Confirm access paths, change windows and named contacts before the scope is written, because gaps in these usually extend the schedule more than the technical work does.
How long does a fixed-scope CephFS subvolume engagement take?
Duration is set in the written scope, not by a general rule. DeployOpen works mostly on fixed-scope engagements, so the scope document states the deliverables, the environments covered and the timeline before work starts. The scope usually reflects the number of consumers (Ceph-CSI, Manila or both), the number of subvolume groups and tenants, whether snapshot and clone policies are included, and whether upgrade planning is part of the work. Ask for the timeline as a list of phases with the dependencies each phase needs from your team, such as test environment access or approval windows.
What does a quote for this work depend on?
A quote follows from the written scope, and DeployOpen does not publish a price list. The main inputs are the consumers in scope, the tenant and subvolume group design, the number of Ceph, Kubernetes and OpenStack environments touched, the release and upgrade path, and the testing and documentation you require. Ask the provider to separate one-time work (design, build, validation, handover) from any recurring support or retainer, and to state which tasks your team performs. Confirm what is excluded, such as hardware changes or third-party licensing, so the comparison with your own team's effort is like for like.
What happens to snapshots and clones during a migration?
Clones are created asynchronously from snapshots and can be used only after the clone completes, so a migration plan must wait for that state before cutting over a workload. A subvolume's size comes from its quota, so a subvolume without one has no size limit, and that must be settled before data moves. Plan an inventory of existing snapshots and clones, confirm which ones consumers still reference, and decide whether they are copied, recreated or retired. Test a restore from a snapshot in the target environment before the source is decommissioned. The exact behavior should be confirmed against the FS volumes documentation for your Ceph release, for example through the CephFS volumes documentation.
Why do multi-release upgrades change the CephFS subvolume plan?
Each platform moves on its own schedule, so an upgrade can change what the subvolume layer must support. Ceph maintains a small set of active releases with periodic backports, so the target Ceph release should be checked against the Kubernetes and OpenStack versions in use. Kubernetes allows kubelets to be up to three minor versions older than kube-apiserver, and the control plane must be upgraded one minor version at a time, which affects the CSI driver rollout order. OpenStack upgrades between adjacent releases or between SLURP releases, which determines how often the Manila side must be retested. Ask for a compatibility matrix covering all three platforms before committing to a schedule.
Is the CephFS subvolume design open source, and what licenses apply?
The Ceph source is licensed under LGPL-2.1 or LGPL-3, and there is no paid edition from the Ceph project itself, so the storage layer carries no license fee. Kubernetes is licensed under Apache 2.0, and official OpenStack projects must be Apache License 2.0, so the platforms in this design are open source. Your costs therefore come from infrastructure, staff time and any support contract you choose. Confirm the license of any third-party CSI driver or Manila packaging you add, because bundled components in Ceph are under other licenses and your compliance team may ask for an inventory.
How DeployOpen scopes and supports CephFS subvolume work
DeployOpen is a Chainscore Labs company that plans, implements, hardens, documents and supports self-hosted Ceph inside your own infrastructure, including the CephFS volumes, subvolume groups and subvolumes that Manila and Ceph-CSI provision on top of it. The work runs in your cloud accounts, data centers or air-gapped networks, and your team keeps ownership of the cluster, the keys and the data throughout. Most engagements are fixed-scope, so the agreed deliverables, the quota and clone decisions, the client configuration for each consumer and the handover documentation are written down before work starts.
The written scope usually makes the open questions explicit: which Kubernetes and OpenStack releases are in play, which Ceph release line the cluster will run, how many subvolume groups map to tenants or storage classes, what quota policy applies to each share, and who owns snapshot and clone retention. It also records what you will provide, such as network access, hardware, identity integration and named contacts for approvals. A quote follows from that scope. DeployOpen does not publish a price list, so the fastest route to a useful quote is to bring the list of consumers, expected share sizes and your change-control rules to the first conversation.
Ongoing support, upgrades and continuous testing are available as retainers. This matters for subvolumes because Kubernetes and OpenStack each release on their own cycle, and Ceph upgrades change how the volumes manager and CSI driver behave, so a share layout that works today still needs checks after each change. A retainer covers planned work against the agreed scope rather than open-ended changes. DeployOpen does not claim vendor partner status with Ceph, Kubernetes or OpenStack, and you should confirm any vendor support arrangement directly with the relevant project or distributor before relying on it.
