CephFS ReadWriteMany volumes on Kubernetes with Rook

When teams need ReadWriteMany volumes with CephFS on Rook

Teams need a ReadWriteMany volume when several pods, often scheduled on different nodes, must read and write the same files at the same time. Typical cases are shared upload directories for a web tier, build caches reused by parallel CI jobs, and content pipelines where one stage writes files that later stages process. In Kubernetes, CephFS is the Ceph file system that serves this access mode through the Rook operator. Block volumes (RBD) do not provide it, because an RBD image is attached to one node at a time for read-write use, so an RBD-backed claim cannot be shared across nodes in this way.

This page covers the decisions that come after that choice: the CephFS StorageClass and its pools, subvolume groups that separate tenants or applications, metadata server sizing for workloads with many small files, and quotas that cap how much each volume can consume. DeployOpen plans and builds these volumes inside your own Ceph and Kubernetes clusters, so the data, keys and operating procedures stay under your control. Before you start, confirm with your storage team whether a shared file system is needed at all, since an object bucket or an application-level sharing design may fit some workloads better.

Start by listing each workload that needs shared access, the number of pods that will mount the volume at once, and the size and count of files they create. Those answers drive the metadata server design and the quota model more than raw capacity does. Ask your Rook and Ceph administrators which Ceph release your clusters run, since the Ceph releases page lists the maintained releases and their end-of-life dates, and plan your upgrade path before the shared volumes hold production data.

Configuration options for CephFS StorageClasses and subvolume groups

This table compares the choices a team makes before creating a CephFS StorageClass for shared ReadWriteMany volumes. Read each row as a decision: what it controls, when the option fits, and what it costs in isolation, administration or performance. Confirm parameter names against the Rook CephFS documentation for the release you run.

Option	When it fits	Trade-off to weigh
File system and data pool	One CephFS file system with a shared data pool suits teams that accept common capacity and a shared failure domain.	Shared pools are simpler to run, but one noisy workload or a full pool affects every volume on it.
Additional data pool per class	A separate data pool per StorageClass suits tenants with different replication, media or compliance needs.	Each pool adds placement group planning, capacity management and monitoring work for your Ceph administrators.
Metadata pool and media	Enterprise-class SSDs for the metadata pool suit clusters serving many small files, where metadata operations dominate.	Dedicated metadata media is extra hardware to buy and size, and it has to be planned before the file system is created.
Subvolume group	Separate subvolume groups isolate volume sets by team or application and let you apply group-level settings.	More groups mean more objects to name, track and clean up, which increases administrative overhead.
Reclaim policy	Delete suits ephemeral scratch volumes; Retain suits data that must survive PersistentVolumeClaim deletion.	Retain leaves data and quota consumption behind until someone removes it, so define an owner and a cleanup process.
Mount options and kernel or FUSE client	The kernel client is generally preferred where the node kernel supports the needed features; FUSE suits constrained kernels.	Client choice changes behavior, performance and upgrade paths. Confirm supported client versions for your Kubernetes and Ceph releases.
Per-volume quota	Quotas on each subvolume stop one workload from filling the shared file system.	Quotas limit capacity, not file count. Many small files still load the metadata server, so size the metadata tier separately.

Sizing the metadata server for many small files

Many small files load the metadata servers more than large sequential files do. With CephFS, every open, create, rename, attribute lookup and directory listing is a metadata operation handled by an MDS daemon before any data moves to or from the OSDs. A workload that writes a few large files spends most of its time moving bytes and generates relatively few metadata requests. A shared volume holding millions of small files, such as build artifacts, session files or container layers, can produce a high rate of metadata requests while moving very little data. That is why the MDS, not raw throughput, often sets the ceiling for a ReadWriteMany volume. Ask your application owners for file counts, average file size, directory fan-out and the number of pods that will mount the same volume at peak, and treat those answers as the input to every sizing decision in this section. The Ceph hardware recommendations advise placing the metadata pool on enterprise-class SSDs, and that advice matters more here than it does for bulk data. Metadata operations are small and latency-sensitive, so a slow metadata pool slows every client that depends on the file system. Keep the metadata pool on its own SSD-backed devices and confirm that the drives are enterprise-class with power-loss protection before you purchase them. On the MDS side, the number of active metadata servers is a decision you make with the file system design, not a fixed default. Multiple active MDS daemons can share the namespace load, but they add coordination overhead and failover behavior to test. Confirm with the Rook and Ceph versions you plan to run how many active ranks and standby daemons are supported and how the file system is configured for them. Memory for each MDS should be sized for peak use rather than calm periods, because the cache holding recently used metadata grows during bursts of file activity and the daemon needs headroom when a client surge arrives. Plan the limit for the worst hour you expect, such as a nightly batch job that touches the whole tree, not for an idle Tuesday. Then benchmark a representative workload before buying hardware, since the Ceph hardware recommendations note that no two clusters are alike. Replay your real file counts and access patterns against a test file system, measure request latency and MDS memory, and repeat the test with the number of pods you expect in production. Your benchmark results, not a generic rule, should set the final MDS count and memory limit.

Choosing CephFS or RBD by access pattern

A single pod that owns its data, such as a database, queue or search index, usually fits RBD. A block device is mounted by one node at a time, so the usual access mode is ReadWriteOnce, and the filesystem sits on top of the volume inside the pod. This keeps locking and caching simple because only one client writes.

Access mode: ReadWriteOnce for a block volume mounted by one node.
Fit: RBD gives the workload a dedicated device with predictable latency for database-style I/O.
Main risk: a volume attached to one node cannot be mounted by a replacement pod on another node until the old attachment is released, so failover plans need testing.
Snapshots: confirm how crash-consistent snapshots are taken for the application before relying on them for recovery.
Sizing: expansion and reclaim behavior are set per StorageClass, so review them with the application owner.

Check whether the application needs only one writer for its whole life, and whether it does its own locking. If it does, a block volume is the simpler and usually the faster choice.

When several pods across nodes must read and write the same directory tree, such as a content store, shared build cache or media processing pipeline, CephFS is the storage type built for that pattern. It presents a POSIX-style file system, so pods can mount the same volume with the ReadWriteMany access mode and see one set of files.

Access mode: ReadWriteMany for a file system mounted by pods on several nodes.
Fit: CephFS keeps one namespace across all clients, and the metadata server coordinates file access.
Main risk: concurrent writers still need application-level coordination, since a shared file system does not make conflicting edits safe.
Metadata load: workloads with many small files put pressure on the metadata server, so its sizing needs to be planned with the expected file counts.
Quotas: plan per-volume quotas and subvolume groups before many teams share one file system.

Ask the application owner which files are written by which pods and how conflicts are prevented. The answer decides whether shared file access is a fit or whether the application should move to object storage.

Some workloads need a raw or formatted block device with consistent write latency, such as transactional databases that manage their own on-disk layout or virtual machine disks. RBD is the usual fit here. These workloads often expect exclusive access to the device and may rely on specific flush behavior, so the storage choice should be validated with the application's own tests.

Access mode: ReadWriteOnce for exclusive use, with Block volume mode when the application needs a raw device.
Fit: RBD gives each volume its own image in a pool, which isolates I/O from other tenants more directly than a shared file system.
Main risk: a shared block device mounted read-write by several pods can corrupt data, so multi-writer requests for block volumes need to be refused in design.
Performance: latency depends on pool replication, media and network, so benchmark with the application's I/O pattern.
Recovery: test restores from snapshots and backups, not only the snapshot creation.

Confirm the database's own replication and failover model before choosing the StorageClass. Many database operators handle replicas themselves, which changes whether a shared volume is needed at all.

Shared file systems often run into trouble with workloads that create, rename and delete very large numbers of small files, such as package caches, CI workspaces or document conversion queues. CephFS can serve these patterns, but the metadata server becomes the limiting component, so the cost of the workload shows up as metadata activity rather than raw throughput.

Access mode: ReadWriteMany with CephFS, sized for the file count rather than the capacity alone.
Metadata: ask for metadata server sizing based on the expected number of files and the rate of metadata operations.
Subvolumes: separate teams or applications into subvolumes within subvolume groups so quotas and cleanup stay manageable.
Cleanup: deleting large directory trees can create a long metadata backlog, so plan retention and cleanup jobs.
Alternative: if the workload is mostly object-style, a bucket with an S3 client may suit it better than a shared file system.

Measure file counts and metadata operations in a representative test before setting quotas or sizing the metadata servers. Vendor limits and tested ceilings should be confirmed with the Rook and Ceph documentation for your release.

How to set up a CephFS StorageClass with quotas for Kubernetes

This sequence takes a Rook-managed Ceph cluster from a version check to a shared ReadWriteMany StorageClass with per-volume quotas, verified by two pods on different nodes reading and writing the same path. Each step closes with a check so you know it is safe to move on.

Confirm Ceph and Rook versions are supported together
Before creating anything, check the Rook release notes and compatibility table for the Rook version you run against the Ceph version in the cluster. Confirm the Ceph release is one the project still maintains, using the Ceph active releases page, and note which Kubernetes version the Rook release supports. Owner: the platform engineer, with DeployOpen if the work is scoped as a fixed engagement. Done when the Rook, Ceph and Kubernetes versions are written down together and each pair is listed as supported in the Rook documentation for the chosen release.
Create the CephFS file system and metadata pool
Create the CephFS file system through the CephFilesystem resource or the equivalent Rook manifest, defining a replicated metadata pool and one or more data pools. Metadata is latency-sensitive and small in bytes but heavy in operations, so the hardware recommendations call for enterprise-class SSDs for metadata; confirm your metadata pool sits on that media. Check the MDS pods are scheduled and that the file system reports as active with its metadata servers in an active state. Done when the file system health is OK and at least one MDS is active.
Size metadata servers for many small files
Set the number of active MDS daemons and standby daemons, and set CPU and memory requests for the MDS pods. Workloads with many small files consume more metadata server memory and cache, and the Ceph hardware recommendations advise sizing memory for peak use rather than calm periods. Benchmark with a representative file count before you commit to a number. Done when the MDS pods hold their requests under a test load and the file system reports one active and at least one standby daemon.
Create or confirm the subvolume group
Create a CephFS subvolume group for the Kubernetes volumes so they are isolated from other file system content, or confirm the existing group that the CSI driver expects. Set the group's data pool layout if your design separates workloads onto different pools. Check that the group exists and is listed for the file system through the Ceph tooling referenced in the Rook CephFS documentation. Done when the group is visible and empty, ready for the CSI driver to provision subvolumes into it.
Deploy the StorageClass and CSI secrets with scoped CephX access
Create the CephFS CSI secrets with a dedicated CephX user for provisioning and a separate user for node mounts, granting each only the capabilities the Rook CephFS documentation lists for that role. Apply the StorageClass that references the file system, the subvolume group and the secrets, and set the reclaim policy you want. Review the generated capabilities before applying them, because broad caps are the most common reason shared file systems are over-permissioned. Done when the CSI provisioner and nodeplugin pods are running and a test PersistentVolumeClaim with ReadWriteMany access binds to a volume.
Set per-volume quotas through the StorageClass
Configure the quota behavior the Rook CephFS StorageClass supports for your release, so each provisioned subvolume receives a size limit when its claim is created. Ask the vendor documentation for the exact parameter names for your release, and confirm that the quota is enforced on the subvolume rather than only reported. Create a claim with a requested size and inspect the resulting subvolume to confirm the limit was applied. Done when a writer inside the volume hits the limit at the configured size and the claim status shows the expected capacity.
Test shared access from two pods on different nodes
Deploy two pods that mount the same ReadWriteMany claim, scheduled onto different nodes using anti-affinity or node selectors. Have the first pod write a file to a known path, then have the second pod read it and append its own file. Repeat the cross-pod read after restarting one pod to confirm data persists on the volume. Done when both pods read each other's writes at the same path, the MDS stays active throughout, and the file system health remains OK.
Record the checks you will repeat in operations
Save the health commands, the MDS state check, the quota test and the cross-pod read as a runbook entry tied to the cluster and Rook release. Add the same checks to monitoring so a degraded file system or a missing MDS standby raises an alert. Review the CephFS and Rook documentation for the release you run before each Rook or Ceph upgrade, because quota handling and CSI capabilities can change between releases. Done when the runbook is stored with your platform documentation and the alerts fire in a controlled test.

Security and operational risks of shared file systems

A shared ReadWriteMany volume gives many pods the same data, so one weak credential, permissive claim or unmonitored metadata server affects every consumer at once. The cards below cover the controls to set before the StorageClass goes into production and the checks to keep running afterward.

Scope CephX credentials for the CSI driver

The Rook CSI driver authenticates to Ceph with CephX users. Create separate users for provisioning and for node mounts, and grant each only the caps its role needs on the filesystem and data pools. Ask your Ceph administrator to confirm that no CSI user holds administrative caps, and record which user belongs to which StorageClass so a credential rotation or revocation can be scoped to one consumer.

Isolate tenants with subvolume groups

Subvolume groups and separate data pools are how CephFS keeps one team's shared volume out of another team's namespace. Decide whether tenants share a pool with separate groups or get their own pools, and document the trade-off. Confirm with your storage team which option matches your failure domain and capacity planning, because pool layout is difficult to change once volumes exist.

Restrict who can request shared volumes

Any user who can create a PersistentVolumeClaim against the CephFS StorageClass can consume shared capacity. Use RBAC to limit create and update rights on PersistentVolumeClaims in each namespace to the teams that own those workloads. Review bindings with an audit of which service accounts can reference the StorageClass, and reject claims from namespaces that have no approved use.

Protect CSI secrets in etcd

Secrets are stored unencrypted in etcd by default, and the CephX key for the CSI driver lives there. Configure encryption at rest for the API server, restrict list and watch access on Secrets in the Rook namespace to the operator and CSI service accounts, and confirm that backups of etcd are encrypted. Where your policy requires it, consider an external secret store through the Secrets Store CSI Driver.

Apply Pod Security Standards to mounting pods

Pods that mount a shared volume can read and write every file the volume exposes to their group, so a compromised container becomes a path into other workloads' data. Enforce the baseline or restricted Pod Security Standard on namespaces that mount CephFS, set fsGroup and runAsNonRoot where the application allows it, and check that no pod runs privileged or mounts host paths that expose Ceph credentials.

Monitor MDS health and capacity

Metadata servers handle every namespace operation on a CephFS volume, so workloads with many small files stress them before they stress data capacity. Alert on MDS state changes, cache pressure and slow requests, and track quota use per subvolume. Ask your operations team which dashboards and alert thresholds they will own, and rehearse what happens when an MDS fails over during a busy write window.

Where per-volume quotas and subvolume groups stop protecting you

Per-volume quotas cap how much data a CephFS subvolume can hold, and subvolume groups let you apply that cap and a separate data pool placement per tenant. Neither control limits how hard a tenant works the shared metadata server (MDS), how many metadata operations it issues, or how much client throughput it pulls from the data pool. A workload that creates millions of small files can stay far under its byte quota and still saturate the MDS, because capacity and metadata load are different resources. Before promising a tenant that quotas isolate them, confirm with your Rook and Ceph release notes how quota enforcement works for that release, including whether quota changes are enforced by the client, by the MDS, or both, and what happens to writes already in flight when a limit is reached.

A noisy neighbor can still slow every other volume that shares an MDS rank and the same data pool. Metadata-heavy jobs such as builds, unpacking archives or scanning directory trees compete for the same MDS cache and request queue, so latency rises for volumes that never exceeded their quota. Recovery, scrubbing or rebalancing on the underlying OSDs can add further contention to the data pool. To detect this, watch MDS request latency and queue depth, client session counts, and cache pressure on each active MDS, and compare per-subvolume operation rates against the baseline you recorded before the tenant arrived. Also track slow requests and OSD latency for the data pool. Agree on these thresholds with the owning teams before go-live, so the first alert is not a dispute about whose workload caused it.

Limits to confirm in your environment before committing to shared volumes

Shared ReadWriteMany volumes depend on a chain of versions, network design, metadata capacity and application behavior. Each card below names one limit to confirm with Rook, Ceph and Kubernetes documentation or in a lab before you commit a workload to CephFS.

Rook and Ceph version pairing

Confirm that the Rook release you plan to run supports the Ceph release you intend to deploy. Ceph's actively maintained lines are Tentacle (20.2.x, latest 20.2.4, estimated end of life June 1, 2027) and Squid (19.2.x, latest 19.2.7, estimated end of life October 31, 2026). Ask the vendor or your Rook support contact which Rook minor releases support each line, and plan the upgrade path before you pick a line. A Squid end of life that falls before your planned retention period is a reason to choose Tentacle first.

Kubernetes version and support window

Check that your Kubernetes version is inside the window the CSI driver and Rook release support. Kubernetes maintains its three most recent minor releases, and the version skew policy limits how far kubelets may lag behind the API server. Confirm the CSI sidecar and CephFS driver versions that Rook deploys against your cluster version, and schedule Kubernetes upgrades so they do not overlap with a Ceph major upgrade. Test the CephFS mount on each node image before you roll the cluster forward.

Access modes and snapshot behavior

Shared ReadWriteMany volumes are only useful if the application and the storage class agree on access modes. Confirm that the CephFS StorageClass exposes ReadWriteMany for your workload, and that the application does not expect ReadWriteOnce semantics such as a single writer or block-level locking. Test volume snapshots and clones in a lab as well, because CephFS snapshot behavior and the snapshot class you use can differ from RBD. Ask the vendor which snapshot operations are supported for your version.

Network bonding and failure domains

A CephFS mount depends on the public network and on the metadata servers staying reachable. Ask whether your switches and hosts support bonded links split across two switches, so that one switch failure does not isolate a metadata server or a set of OSDs. Map your CRUSH failure domains to racks or switches, then test a switch failure in the lab and record how long client I/O stalls. Confirm that your monitors and MDS daemons are spread across failure domains as well.

Expected file count and MDS capacity

Workloads with many small files stress the metadata server more than large sequential writes. Estimate the file count per volume and across the filesystem, then ask the vendor or Ceph documentation how many active MDS daemons and how much memory your target count requires. Run a metadata benchmark with a realistic file tree before you buy hardware, because Ceph's hardware guidance says no two clusters are alike and memory should be sized for peak use. Confirm the MDS cache limit you plan to set.

Shared POSIX locking behavior

Many pods writing to one filesystem means your application must handle concurrent access, file locks and cache coherence the way its design expects. Confirm with the application vendor whether it relies on advisory or mandatory POSIX locks, whether it uses fsync for durability, and whether it tolerates stale reads across nodes. Run a concurrent-writer test in a lab that simulates pod restarts and node loss, and record any data loss or corruption before the workload goes to production.

What shared CephFS volumes cost and take to run

What drives the cost of setting up shared CephFS volumes with Rook?

The cost depends mostly on how much of the Ceph and Kubernetes stack is new to your organization and how many decisions still need to be made. The main components are design work (pool layout, replication, subvolume groups, quota model), building the StorageClasses and Rook manifests, metadata server sizing and testing against your real workload, and handover to your operators. Costs rise when the cluster is still being specified, when several Kubernetes clusters or an existing external Ceph cluster must be connected, or when many applications need migrating from another shared file system. Costs fall when hardware and the Ceph version are already settled and the target workloads are well described. Ask for the quote to list each component separately, state which items are one-time and which recur, and name who pays for hardware, licenses and support contracts, since Ceph itself is LGPL-licensed software with no paid edition from the project.

How long does it take to set up a CephFS StorageClass?

Duration varies with the state of your environment, and no fixed figure applies to every project. A single StorageClass on an existing, healthy CephFS file system is a small change. A new file system, metadata server tuning for many small files, quota testing and a migration plan take considerably longer. Timing is driven by hardware lead times, change windows, how quickly your team can review designs, the number of applications to onboard, and whether the Ceph and Rook versions must be upgraded first. Ask for a timeline tied to written milestones in the scope, with the client dependencies listed beside each one, so delays can be traced to a specific cause.

What does our team need to provide for a CephFS engagement?

The work needs named people who can make decisions and grant access. Typically this means a platform owner for the Kubernetes clusters, a storage owner who can approve pool and replication choices, a security contact for CephX client caps and network policy, and application owners who can describe access patterns, file counts and growth. You will also need working access to the Kubernetes API, the Ceph cluster or an agreed administrative path, the network segments between nodes and storage, and a test environment where workloads can run before production. Confirm these items during scoping, because gaps in access or decision makers are one of the most common reasons project timelines move.

Can our own team build and run shared CephFS volumes without outside help?

Yes, a capable platform team can build and run shared CephFS volumes with Rook, provided it has time to learn the Ceph storage model, CRUSH failure domains, metadata server behavior and the Rook operator lifecycle. The steepest parts are sizing the metadata servers for many small files, choosing pool and replication settings before data is written, and planning upgrades so that the operator and the Ceph release move in the right order. Outside help is most useful for a design review before purchase, for a one-time build with knowledge transfer, or for a second pair of eyes on quotas and access control. Ask any partner to show which parts of the work they would leave with your team and which they would keep.

What recurs after go-live, such as upgrades and capacity planning?

Shared file systems need continuing attention after the first deployment. Recurring work includes watching cluster health and metadata server load, reviewing subvolume quotas as teams grow, adding capacity, and planning upgrades. Kubernetes releases come about three times per year on a cycle of roughly 14 weeks, and the project maintains the three most recent minor releases, so the Kubernetes side needs a regular upgrade rhythm. On the Ceph side, Tentacle (20.2.x) and Squid (19.2.x) are the actively maintained releases, with estimated end of life dates of June 1, 2027 and October 31, 2026 respectively, so check which release your cluster runs before planning. Ongoing support, upgrades and continuous testing are available as retainers, and the scope should state what each retainer covers and how changes are priced.

Is the CephFS volume work quoted as a fixed scope?

Most DeployOpen engagements are fixed-scope, meaning the scope, deliverables and timelines are agreed in writing before work starts. A written scope for this topic should name the Ceph and Rook versions in use, the clusters and namespaces involved, the number of StorageClasses and subvolume groups to create, the metadata server sizing approach, the quota and access model, the verification tests, and the handover material. It should also state what is excluded, such as hardware procurement, network changes outside the cluster, or application migrations. Changes that appear during delivery are handled through a documented change request, so you can see the cost and schedule effect before accepting it. DeployOpen does not publish a price list, and a quote follows from the written scope.

Which CephFS settings should we confirm before asking for a quote?

Before requesting a quote, gather the facts that most affect the work. Confirm the Ceph release and the Rook operator version, whether the Ceph cluster is managed by cephadm or is external to Kubernetes, the hardware for metadata and data, the expected number of pods mounting the same file system, the average file count per volume, and the required quota and retention rules. Also confirm your Kubernetes version, since the version skew policy requires the kube-apiserver to be upgraded one minor version at a time before other control plane components and kubelets. A clear inventory lets the proposal reflect your environment instead of generic assumptions, and it reduces the number of scope changes after kickoff.

How DeployOpen scopes, builds and supports shared CephFS volumes

DeployOpen approaches shared ReadWriteMany volumes as a planned storage service, not a single StorageClass. We define the CephFS file systems, subvolume groups, metadata server capacity and quotas against your workloads, then build, test, document and hand over the result in infrastructure you control. Engagements are fixed-scope, with ongoing support available as a retainer.

How we scope

The written scope lists which CephFS file systems, StorageClasses and subvolume groups are included.
We record the expected number of pods, small-file volume and access patterns that the metadata server must serve.
The scope states which quota, snapshot and reclaim behavior is in and out, and which client workloads are covered.

How we build

We configure the CephFS StorageClass, secrets and per-volume quotas in a staging cluster before production changes.
We run load tests with many small files and concurrent writers, then tune metadata server settings from the results.
We verify mounts, permissions and quota enforcement from representative pods before sign-off.

How we hand over

Deliverables include the StorageClass manifests, subvolume group layout and quota conventions for your teams.
We document the sizing assumptions, the test results and the health checks to run before and after changes.
Your operators receive a runbook covering metadata server scaling, quota changes and recovery from common failures.

How we support

Ongoing support, upgrades and periodic re-testing of shared volumes are available as a retainer.
We recommend confirming Rook and Ceph version compatibility with your Kubernetes version before each upgrade.
Retainer reviews can revisit metadata server sizing as file counts and pod counts grow in your environment.

Shared ReadWriteMany volumes with Rook and CephFS