Rook external mode for Ceph on Kubernetes

How Rook external mode connects Kubernetes to your Ceph cluster

If you already run a cephadm-managed Ceph cluster and need it to serve block, file and object storage to Kubernetes workloads, Rook external mode is the supported method for this. In this model, the Rook operator runs inside Kubernetes and connects to a Ceph cluster that it does not deploy or manage. The Ceph cluster stays where it is, with its own daemons, data and upgrade lifecycle, and Kubernetes consumes it through the Rook operator and the Ceph CSI drivers.

The two sides have distinct roles. The Ceph cluster is the *provider*: your storage team prepares it with cephadm, creates the pools and CephX users that Kubernetes needs, and exports the connection details. The Kubernetes cluster is the *consumer*: your platform team imports those details into the Rook operator, which then provisions storage classes and volumes for applications. Each side keeps its own failure domains, capacity planning, monitoring and patching. Rook does not add or remove Ceph daemons in external mode, so a Kubernetes upgrade or a Rook redeploy does not change the storage cluster itself.

Ownership must be written down before the first export. The storage team owns the Ceph hardware, the cephadm host and daemon lifecycle, pool design and the CephX keys handed to Kubernetes. The platform team owns the Rook operator, the CSI configuration, the StorageClasses and the workloads that consume them. Confirm the supported Rook and Ceph combination in the Rook compatibility matrix before choosing versions, and use the Rook external cluster documentation for the exact export and import steps, since those commands and flags change between Rook releases. Agree which team rotates CephX keys when the Ceph release changes, because that step is easy to miss.

Versions and access Rook external mode needs

External mode connects a Kubernetes cluster to Ceph that already runs somewhere else. The cards below cover the versions, network paths and accounts that have to line up on both sides before you export or import any connection details.

A cephadm-managed Ceph cluster

The external cluster must be managed by cephadm, which was introduced in v15.2.0 (Octopus) and does not support older releases. If your cluster was deployed by hand or with an older tool, plan the migration to cephadm before connecting Kubernetes. Confirm with your Ceph administrator that the cluster's orchestrator reports healthy services, because the export steps depend on it.

A maintained Ceph release

Use a release that is still receiving backports and security fixes. Squid (19.2.x) has an estimated end of life of October 31, 2026, and Tentacle (20.2.x) has an estimated end of life of June 1, 2027. Pick the release the Rook version supports, and plan an upgrade of the Ceph side before the end of life date rather than after it.

A supported Kubernetes minor version

The Kubernetes project maintains the three most recent minor releases. As of September 2026 these are 1.35, 1.36 and 1.37, and 1.35 reaches end of life on February 28, 2027. Confirm the Kubernetes version of every consumer cluster against the Rook compatibility matrix, and remember that each upgrade of the consumer cluster can change the CSI driver images you need.

Network paths to monitors and OSDs

Every Kubernetes node that runs a Ceph CSI plugin must reach the Ceph monitors, and the CSI components must reach the OSD ports for data traffic. Ceph's messenger v2 listens on port 3300, so confirm firewall rules for that port and the OSD port range before anything else. Test reachability from a real node in the consumer cluster, not from a workstation, because pod networks and node firewalls often differ.

Rook operator pinned to the Ceph version

Rook and Ceph are released on separate schedules, so the operator version determines which Ceph releases it can manage. Pin the Rook version in your Helm values or manifests and record it next to the Ceph release. Confirm the pairing in the Rook documentation before any work starts, and treat that pairing as the source of truth for upgrade order later.

An admin account to create CephX users

The Ceph administrator needs an account with rights to create CephX users and read the cluster's fsid, monitor addresses and pool names. The export script creates restricted users for Kubernetes, so the admin key itself should not be copied into the consumer cluster. Ask who will hold this account, how the key is rotated, and where the exported output is stored.

Provider and consumer layouts teams choose between

In the simplest layout, a cephadm-managed cluster serves one Kubernetes cluster that does not run Ceph itself. (Illustrative) A storage team runs the provider cluster, and a platform team runs the Kubernetes cluster as the only consumer. The duties are clearly divided, so most of the planning concerns access, upgrade order and recovery.

Provider keeps: the monitors, OSDs, pools, CRUSH rules and the CephX keys that govern access to them.
Consumer receives: the monitor endpoints and a scoped CephX identity, which the Rook operator uses to create block and file storage classes.
Credentials: one identity for this Kubernetes cluster, limited to the pools and filesystems it needs and no others.
Blast radius: a fault in the Kubernetes cluster mostly affects its own volumes, while a provider outage affects every mounted volume.
Upgrades: the provider team sequences Ceph upgrades, and the platform team checks client compatibility before it upgrades Rook.

Before you adopt this layout, confirm which team can rotate the consumer credentials and where the exported connection details are stored.

Rook external mode setup sequence, step by step

This sequence connects a Kubernetes cluster to a cephadm-managed Ceph cluster so that Rook can provision block, file and object storage from pools you prepared in advance. Each step ends with a check you can run on both sides before moving on.

Confirm cluster health on the provider
The Ceph administrator starts on the cephadm-managed provider cluster and confirms it is healthy before anything is exported. Run ceph -s from an admin shell and check that the overall status is HEALTH_OK, or that any warnings are understood and accepted, and that monitors, managers and OSDs are all up and in. Record the cluster FSID and the Ceph release, because the Rook and Ceph versions on the consumer side must be compatible with them. Confirm the exact release compatibility matrix and the supported Rook versions in the Rook documentation for your release. The step is done when the health output is saved with the rest of your change record.
Create the pools consumers will use
Before the first import, the provider administrator creates every pool the Kubernetes clusters will consume, because the external connection details reference existing pools and file systems. Plan an RBD pool for block volumes, a CephFS file system if shared ReadWriteMany volumes are needed, and an RGW realm, zone and pools if S3-compatible object storage is in scope. Set replication or erasure coding and placement settings per pool according to your failure domain design, and enable the application tag each service expects. Verify with ceph osd pool ls detail and, for CephFS, ceph fs ls, and confirm each pool reports the expected size and application. The step is done when every planned pool is listed and its settings match the design.
Create dedicated CephX users with scoped caps
Create separate CephX users for the Rook external connection rather than reusing the admin key. Limit each user's capabilities to the monitors, the specific RBD pool, the CephFS file system and the RGW resources that were created in the previous step, and avoid granting admin caps to Rook consumers. Keep one user per consumer cluster when several Kubernetes clusters share a provider, so access can be revoked independently. Check the caps with ceph auth get for each user and confirm that no pool outside the planned list appears. Record the caps in your configuration repository, since upgrades later require reviewing them. The step is done when each user shows only the intended pools and services.
Export connection details from the provider
Run the export script that the Rook external-cluster documentation describes for your Rook version, passing the pool names, file system name and RGW settings you intend to use, and the CephX user you created. The script prints or writes the monitor endpoints, FSID and keys that the consumer needs. Confirm the exact script name, its flags and whether it supports the object storage options in your Rook release before running it, because these change between versions. Store the output in a restricted location, since it contains credentials. The step is done when the export file lists the expected monitor endpoints and every planned pool, and the file is handled as a secret.
Deploy the operator and external CephCluster
On the Kubernetes consumer cluster, install the Rook operator using the chart or manifests for the release you chose, then create the CephCluster resource configured for external mode. The operator will not connect to the provider until the connection secret exists, so this step deploys the control plane and prepares the resource for the import that follows. Confirm the operator pod is running in its namespace and check the operator logs for errors. The CephCluster status will remain pending until the next step, which is expected. The step is done when the operator is running and the CephCluster resource is created with external mode enabled.
Import connection details as a Secret
Run the import script from the Rook documentation against the export file, which creates the Secret and ConfigMap that the external CephCluster reads. Apply the generated resources to the namespace where the operator runs, and confirm the script flags for your version before running it. Once applied, the operator connects to the provider and the CephCluster status moves to a connected state. Verify with kubectl -n rook-ceph get cephcluster, check that the phase and health fields reflect the provider's status, and compare the reported health with the output from the first step. The step is done when both sides report the same health and the cluster shows the expected FSID.
Define StorageClasses for imported pools
Create StorageClasses that reference the imported RBD pool, the CephFS file system and, if in scope, the object store, using the parameters shown in the Rook documentation for your release. Each class should point to exactly one pool or file system so that workloads get predictable placement. Set the reclaim policy and expansion options deliberately, since changing them later affects existing volumes. Confirm the StorageClass names and parameter keys against the CSI driver version installed in your cluster. The step is done when kubectl get storageclass lists each class with the correct provisioner and no class points at an unplanned pool.
Test with a PVC and an object bucket
Create a test PersistentVolumeClaim against each StorageClass, and if object storage is in scope, create a test ObjectBucketClaim. Run a short pod that writes and reads data on the test volume, then delete the test resources. Verify with kubectl get pvc that the claim reaches Bound, and with kubectl get obc that the bucket and its generated Secret and ConfigMap exist. On the provider, confirm the new image or subvolume and bucket appear in the expected pools with rbd ls, ceph fs subvolume ls or the RGW admin tools, and that the Ceph health status is unchanged. The step is done when every test claim binds, the data round-trips, and the provider remains healthy.

How to test the consumer before any workload moves

Test the consumer side before any production workload points at the cluster. Once the Rook operator on the consumer cluster has imported the connection details, run a short sequence of block, file and object checks from inside Kubernetes, because a wrong monitor address or a missing CephX cap usually fails here first. Start with a block test: create a test PersistentVolumeClaim against an RBD StorageClass in the consumer cluster, mount it in a pod, write a known file, read it back, take a snapshot if your design uses snapshots, then delete the claim. Check the result with ceph health detail on the provider before and after each step, and record the health state so that any warning you see later can be tied to a specific test.

If CephFS is part of the design, add a file test that uses two pods. Mount the same ReadWriteMany volume in both, write from the first pod and confirm the second pod can read the file and write its own. A permission error at this step often points to a CephX capability granted to the wrong client or to a subvolume group that the consumer identity cannot reach. For the object test, run a pod that holds the bucket credentials and sends a simple S3 request to the RGW endpoint address you exported. Confirm that the request authenticates and that the bucket listing matches what the provider expects. Exposing the RGW endpoint outside the cluster, load balancing and TLS are separate work covered on the Ceph object storage pages, so keep this test on internal addresses only.

Run any deletion or cleanup test only against pools created for testing. Deleting a volume, bucket or pool on a shared cluster can remove data that other consumers depend on, and that data is not recoverable from the consumer side. Ask the provider team to confirm which pools and users are test-only before you start, and write down the names of every object you create so cleanup is complete. When all three tests pass and the health record shows no new warnings, the consumer is ready for a controlled first workload.

Which Ceph identities and pools map to which Kubernetes objects

This table pairs each Ceph identity or pool with the Kubernetes object that consumes it and the scope that object can reach. Read each row as a boundary: the narrower the CephX user and pool, the less a compromised consumer can touch. Confirm the exact object names and capabilities against the Rook and Ceph versions you run before you build the mapping.

Consumer	Ceph object	Kubernetes object	Scope or cap the mapping implies
Block (RBD) for stateful workloads	CephX user with access to one RBD pool	Secret holding the CephX credentials, consumed by an RBD StorageClass and the CSI driver	Read and write on the named RBD pool only. Use a separate user per cluster or per trust zone so one compromised consumer cannot reach another pool.
Block (RBD) for a second cluster	Separate CephX user and separate RBD pool	Second Secret and StorageClass in the consuming cluster	Scoped to its own pool so storage classes in different clusters never share a credential. Confirm how the import script names these users in your version.
File (CephFS) for shared volumes	CephFS file system with a CephX user limited to the needed subvolume group	Secret for the CephFS CSI driver and a CephFS StorageClass	Access to the named file system and subvolume group. Metadata server access is broader than data pool access, so review it with the Ceph security team.
Object (RGW) for buckets	RGW user or service account with bucket-level permissions	ObjectBucketClaim with its generated Secret and ConfigMap, backed by an object StorageClass	Permissions limited to the buckets or prefixes the claim creates. Confirm whether the bucket provisioner creates a new RGW user per claim in your version.
Admin or provisioning identity	CephX user with broader capabilities used by the import process	Credentials used only during export and import, not mounted into application pods	Broad cluster access. Create it for setup and rotation, keep it out of workload namespaces, and remove or restrict it after the mapping is verified.
Monitor endpoints	Monitor addresses and the msgr2 port (3300) the consumer connects to	Endpoint data included in the imported connection details	Network reachability only. Authentication still depends on CephX, so reachability does not grant data access. Confirm the port and secure mode settings for your release.

Updating CephX caps when Ceph and Rook upgrade

An upgrade is the moment an external-mode connection is most likely to break without anyone changing the Kubernetes side. A newer Ceph release or a newer Rook release can require different CephX capabilities for the consumer user that Rook's CSI drivers and the operator authenticate with. If those caps are missing after the upgrade, provisioning of new volumes or mounts of existing ones can fail, even though the Ceph cluster itself reports healthy. Treat cap changes as part of the upgrade plan, not as a follow-up after something breaks.

Work in this order. First, read the release notes for the target Ceph version on the Ceph releases index and note any changes to required caps or to the RBD and CephFS features the consumer uses. Second, check the Rook upgrade documentation for the Rook version you plan to run, because it defines the caps that version expects. Third, update the caps on the provider cluster using the procedure in the Rook documentation, and record the before and after output so you can roll back. Fourth, confirm that the Secret imported into the consumer cluster still matches the provider's keys and endpoints. Only then upgrade the Rook operator and the Ceph version, in the sequence the Rook upgrade guide requires for your versions. Confirm your current versions and the exact sequence against the documentation before you start.

Plan the Kubernetes side of the same window. The Kubernetes version skew policy requires kube-apiserver to be upgraded one minor version at a time and before kube-controller-manager, kube-scheduler and kubelets, and it allows kubelets to be up to three minor versions older than kube-apiserver. Those rules let you stage node upgrades around the Rook change, but they do not remove the need for a health check before and after each step. Confirm that new PVCs can be provisioned and that a test pod can mount an existing volume before you close the window, and keep the previous Secret and caps available until both checks pass.

Security controls that keep the provider and consumers separate

External mode puts two trust domains next to each other: the Ceph cluster that owns the data and the Kubernetes clusters that consume it. The controls below decide how much one side can reach into the other if a consumer is compromised or a credential leaks.

One CephX user per consumer

Create a separate CephX user for each Kubernetes cluster or consumer, and limit its caps to the named pools and namespaces it actually needs. A shared admin key means a single compromised cluster can read, change or delete data in every pool. Review the caps with ceph auth get after each new consumer is added, and confirm that no consumer holds monitor or manager capabilities beyond what the import process requires.

Connection details as credentials

The exported connection details contain CephX keys, so treat them as secrets from the moment they leave the provider. Kubernetes stores Secrets unencrypted in etcd by default, so confirm that encryption at rest is enabled on the API server, restrict list and watch access with least-privilege RBAC, and check who can read the Rook namespace. If your policy requires a central store, consider delivering the values through the Secrets Store CSI Driver instead.

Encryption in transit with msgr2

Ceph traffic between clients and daemons can use the msgr2 protocol in secure mode, which encrypts all traffic after authentication and adds integrity checks. Confirm which mode your cluster and the Rook consumer support, and whether monitors listen on the v2 port. Enable secure mode only after testing, and record which connections still negotiate a weaker option.

Pod Security Standards for the Rook namespace

The namespace that runs the Rook operator and CSI components needs enough privilege to manage storage, but the workloads that consume volumes do not. Apply Pod Security Standards at the namespace level, and keep consumer namespaces at a stricter level. Use the CIS Kubernetes Benchmark as a checklist for reviewing these settings, and record any exceptions that the CSI driver requires.

Network reachability for monitors and OSDs

Limit access to Ceph monitors and OSDs to the consumer node addresses that need them, using network policy in the Kubernetes clusters and firewall rules on the cephadm hosts. Monitors and OSDs should not be reachable from general workload networks. Document the allowed source ranges, and test that a pod outside the approved namespaces cannot connect to the monitor ports.

Key rotation after staff changes

Rotate CephX keys and the connection details when people with access leave the team, when a consumer cluster is rebuilt, or when a key may have been exposed. Plan the rotation so that the consumer secrets are updated in the same change window, and confirm that volumes keep working afterward. Keep a dated record of who holds the provider side and who can export new details.

Common failures in external mode and how to fix them

This table pairs common external mode symptoms with their likely causes and the check to run on the Ceph provider or the Kubernetes consumer side. Read each row by starting with the symptom, confirming the cause in the listed system, and applying the fix only after the check passes. Most failures trace back to connection details, CephX caps, or pool names that differ between the two clusters.

Symptom	Likely cause	Fix and where to check
Operator reports the external cluster as unreachable	Monitor addresses in the imported details do not match the provider, or the consumer cannot reach msgr2 port 3300	On the provider, confirm monitor addresses with ceph mon dump. From a consumer node, test TCP reachability to those addresses on port 3300, then re-export and re-import the connection details if they changed.
Operator reports the external cluster as unreachable	DNS names in the imported details do not resolve from inside the Kubernetes cluster	Resolve each monitor hostname from a pod or node in the consumer cluster. If resolution fails, use IP addresses in the import or fix DNS for the operator namespace.
PVCs stay in Pending	The StorageClass names a pool that does not exist on the provider or has a different name	Compare the pool parameter in the StorageClass with the output of ceph osd pool ls on the provider. Correct the StorageClass or create the pool, then recreate the pending claims if needed.
PVCs stay in Pending	The CephX user used for provisioning lacks caps for the RBD or CephFS pool	On the provider, review the caps for the provisioning user with ceph auth get. Confirm it has the caps described in the Rook external cluster documentation for the pool and application type, then check the provisioner logs for permission errors.
Mounts fail after a Ceph upgrade	CephX caps were not updated for the new release, so the client user lost a permission the CSI driver now needs	Re-run the caps export on the provider after the upgrade, import the updated script output on the consumer, and confirm the CSI node and provisioner pods restart with the new Secret. Check mount errors in the node plugin logs.
Object access is denied	The RGW user lacks permission on the bucket, or the bucket policy is narrower than the application expects	On the provider, check the RGW user and any bucket policy with radosgw-admin user info and bucket policy commands. Confirm the access key in the consumer Secret belongs to that user and matches the endpoint in use.
Ceph health warnings appear after import	Pools referenced by the import were not created on the provider before the import ran	Run ceph health detail and ceph osd pool ls on the provider. Create the missing pools with the expected application tag, then wait for the warning to clear before importing again.
Imported Secrets are out of date after a key rotation	The consumer still holds the previous CephX key because the import was not repeated after rotation	Compare the key in the Kubernetes Secret with the current output of ceph auth get on the provider. Re-run the import with the rotated key, then confirm the CSI pods reload the Secret.

What teams ask before connecting Kubernetes to Ceph

What drives the cost and effort of connecting Kubernetes to an existing Ceph cluster?

The main drivers are how many Kubernetes clusters will consume storage, how many storage classes each one needs, and how healthy the Ceph cluster is before any work begins. Each consumer cluster adds its own import, its own CephX identities and its own validation, and each storage class adds pool, replication and reclaim decisions. Scope matters too: adding object storage or CephFS on top of RBD means more pools, more users and more testing. Finally, the number of Rook and Ceph upgrade cycles you expect to run shapes the work, because CephX caps may need updating at each step. Ask for a quote that lists each of these items separately, plus any one-time setup and any recurring support you want, so you can compare it with your internal estimate.

Can our own team connect Kubernetes to Ceph without a partner?

Yes, a capable team can do this, provided it has the right skills in place. The work needs Ceph operations knowledge to create pools, read cluster health and manage CephX users with the correct caps. It needs Kubernetes RBAC to limit which namespaces and service accounts can read the connection details. It also needs working knowledge of CSI basics, including how StorageClasses, secrets and provisioners fit together. Teams without one of these skills usually benefit from a short review of their design before go-live. Check your skills against the Rook external mode documentation for your versions before you decide.

How long does a Rook external mode integration usually take?

Duration depends on the written scope, not on a fixed figure. The timing is set by the number of consumer clusters, the storage classes and pools to be created, whether object or file storage is included, and whether a change window is needed to validate the first upgrade. Delays most often come from waiting for Ceph administrator access, agreeing on network paths between the Kubernetes nodes and the Ceph monitors, and scheduling a test pool that can be used without affecting production. A fixed-scope engagement sets out deliverables and timelines in writing before work starts, so ask for the schedule as part of the quote.

What do we need to provide before the work starts?

The work needs several things from your side before it can begin. Provide access to a Ceph administrator account, or a process where administrators run the export steps with you, so the connection details can be generated. Confirm network paths between the Kubernetes nodes and the Ceph monitor and OSD addresses, including any firewall or routing changes. Reserve a test pool that is isolated from production data so the first storage classes can be validated safely. For upgrades, agree a change window with both the storage and platform teams, and confirm who approves rollback. Ask your partner to list each of these inputs in the written scope so nothing is discovered late.

Does external mode work with our Ceph and Kubernetes versions?

Compatibility depends on the specific Rook release, the Ceph release running on your cluster and the Kubernetes version on your consumer clusters. Ceph's active releases are listed on the Ceph releases page, and Kubernetes supports the three most recent minor releases, so check both lists against your environment. Then confirm the exact combination in the Rook compatibility matrix before choosing a Rook version. Do not assume that a combination works because each component is individually supported. Your integration plan should record the tested combination and the version you plan to move to next.

Which parts of Ceph and Rook does the engagement cover?

The scope is agreed in writing and typically includes the provider and consumer role design, exporting and importing connection details, creating the pools needed by the first storage classes, setting up CephX users with appropriate caps, and validating each consumer cluster with test workloads. It can also cover the procedure for updating CephX caps during upgrades and documentation for your operations team. Object storage, CephFS, snapshots and cluster-level Ceph hardening are separate areas, and each is included only if it appears in the scope. Ask which items are in and which are out so the boundary is clear before you sign.

What does DeployOpen deliver at the end of the engagement?

DeployOpen delivers the configured external mode integration described in the written scope. That typically includes the connection setup for each agreed consumer cluster, the pools and storage classes you asked for, documented CephAuth users and caps, validation results from test workloads, and an operations runbook covering upgrades and checks. Deliverables are agreed up front, so you can confirm them against the scope before work starts. DeployOpen does not issue audit reports or certifications, but it can prepare documentation and evidence that your auditors may review. Ongoing support and upgrades are available as a retainer if you want them after handover.

How DeployOpen scopes and delivers Rook external mode connections

DeployOpen plans and builds the connection between your Kubernetes clusters and an existing cephadm-managed Ceph cluster, inside infrastructure you control. Each engagement starts with a written scope that names the provider and consumer clusters, the CephX identities, the pools and the upgrade path before any credentials are exchanged.

How we scope

Inventory the cephadm-managed Ceph release, the Kubernetes clusters that will consume it, and who owns each side.
Agree in writing which pools, StorageClasses and CephX users each consumer cluster may access.
Confirm the Rook release, Helm chart approach and upgrade ordering that the written scope will cover.

How we build

Run the provider export and consumer import steps in a staged order, with each credential recorded and access limited to the agreed scope.
Create the pools and StorageClasses that workloads need before the first consumer connects, and verify them with test volumes.
Test failure cases such as a rejected key, a missing pool and a Ceph health warning, and record the expected output for each.

How we hand over

Deliver a runbook that lists each exported secret, its owner, its rotation point and the caps it carries.
Document the CephX cap update procedure to run before and after each Ceph or Rook upgrade, with checks to confirm success.
Walk your platform and storage teams through the connection, the verification steps and the rollback path.

How we support

Ongoing support, upgrades and continuous testing are available as a retainer if your team wants DeployOpen to stay involved after handover.
Ask for a written quote that lists the scope items, the clusters in scope and the handover documents included.
Confirm with your Ceph and Rook vendors or documentation which versions and cap changes apply to your environment before each upgrade.

Connecting Kubernetes to an existing Ceph cluster with Rook external mode