Why cephx capabilities decide what a compromised client can reach
A cephx user's capabilities decide how much damage a stolen or misused key can do. Least privilege in cephx means each client gets its own user, granted only the monitor, OSD and metadata access its job requires, and nothing more. Teams need this when many applications share a handful of broad keys, typically client.admin or one wide user copied into several configuration files. In that setup, compromising one web server, backup job or virtual machine host gives an attacker the same reach across the cluster as the most privileged administrator, and rotating the key breaks every consumer at once.
Each cephx user carries capability strings for three services. The mon cap controls what the user can ask the monitors to do, such as reading cluster maps or issuing commands. The osd cap controls which pools and objects the user can read and write, and it is where most of the blast radius is set. The mds cap governs CephFS metadata access. Pool-scoped grants restrict osd access to named pools, and namespace-scoped grants narrow that further to a RADOS namespace inside a pool, which matters when several tenants share one pool. Capability profiles, such as the RBD and RGW profiles described in the Ceph documentation, bundle common permissions so you do not hand-write long grant lists, but a profile can still be wider than one application needs, so read what it expands to on your release. Confirm exact cap syntax, supported profile names and namespace behavior against the documentation for the release you run before writing any grant, because these details have changed between releases. Start with the Ceph cephx authentication documentation to map each application to the narrowest set of caps it can work under.
Which capability pattern fits each client type
This table compares the cephx capability patterns most clusters use, from broad admin keys to read-only monitoring users. Read each row for the client type it fits, what the key can reach, and what it costs you in operations and automation.
| Pattern | When it fits | What it allows | Main trade-off |
|---|---|---|---|
| Broad admin key (client.admin) | Break-glass administration and initial cluster bootstrap by a small set of named operators | Full read and write across monitors, OSDs, pools and the auth database | Any leaked copy grants full cluster control, so keep it off application and automation hosts |
| Per-application user with pool-scoped OSD caps | Application or service that writes to one or a few named pools, such as a backup job or an analytics loader | Read and write on the named pools only, with monitor access to fetch the cluster map | Each new pool or namespace needs a matching user or cap update, which adds upkeep as the estate grows |
| RBD profile users (profile rbd) | Hypervisors, Kubernetes CSI drivers or other clients that create and map block images in a pool | Image operations in the bound pool, using built-in profiles rather than hand-written caps | Profiles are broad within their pool, so a compromised client can still touch every image there; scope by pool or namespace |
| CephFS users with path or filesystem caps | Clients that mount one filesystem or a subtree, such as a team share or a build cache | Mount and access limited to the named filesystem and, where configured, a specific path | Path restrictions need careful testing across client versions, and mistakes can expose or block data unexpectedly |
| Read-only monitoring user | Dashboards, exporters and health checks that only need cluster status and metrics | Read access to cluster state and status commands, without write or admin rights | Some tools expect more rights than they need, so confirm the exact caps each exporter requires before granting them |
How to create and verify a scoped cephx user
This sequence replaces broad or shared Ceph keys with one cephx user per application, each limited to the pools and operations it needs. Every step ends with a check so you can confirm the new user works and that the old access is gone.
Inventory clients and the pools they use
Start by listing every client that talks to the cluster: hypervisors, backup agents, application servers, gateways and administrators' workstations. For each one, record the protocol it uses (RBD, CephFS, RGW or raw RADOS), the pools or filesystems it touches, and whether it reads, writes or both. Pull the current key inventory with
ceph auth lsso you can see which users already exist and which clients appear to share a key. The step is done when every client maps to a pool list and every existing key has an owner. Ask your team to confirm the list against the hosts that actually connect, because configuration files often hold keys that nothing uses anymore.Decide one user per application
Assign a dedicated cephx entity to each application or workload, such as
client.app-billing-rbdorclient.backup-vm, rather than sharingclient.adminor one broad key across several services. Record the pool, namespace and capability set for each user in your configuration management or change record. Naming should make the owner and purpose obvious when you readceph auth lslater. The step is complete when each user has a written purpose, a single owning team and a list of pools. Confirm with the application owners that no workload depends on a key outside this list.Create the user with minimal capabilities
Create each user with
ceph auth get-or-createand grant only the capabilities that match the protocol. For RBD clients, use therbdprofile scoped to the specific pool, and add a namespace restriction where your design uses namespaces. For RGW or CephFS clients, grant the narrowest mon, osd and mds caps the documentation for that service describes. Avoidallow *and avoid granting mon or mgr access to application users that only need data pools. Save the returned keyring to a file with restrictive permissions. The step is done when the command returns a keyring for each user. Check the caps withceph auth get client.<name>and confirm the output lists only the pools and profiles you recorded in the previous step.Distribute keyrings to client hosts
Copy each keyring only to the host or service that needs it, placing it under
/etc/ceph/with the ceph.conf file and setting ownership so only the service account can read it. Do not copy keyrings into shared configuration repositories or into images that many workloads can pull. Where a secret store is used, store the key there and inject it at runtime. The step is done when each keyring exists on exactly the intended hosts. Verify with a directory listing andstaton each host, and confirm no other service account can read the file.Test read and write access from the client host
From the client host, run a read and a write using only the new keyring. For RBD, pass
--idand--keyringtorbd lsand create, write to and remove a test image in the permitted pool. For RADOS, userados -p <pool> putandgetwith a test object. The step is done when the permitted read and write both succeed and return the expected data. If an operation fails, compare the caps shown byceph auth getagainst the operation, because a missingrorwon the pool is the most common cause. Keep the test objects out of production data and remove them afterward.Confirm denied operations fail
Using the same keyring, attempt operations outside the granted scope: a write to a pool the user should not touch, a read from another application's image, or a command that needs admin rights such as
ceph osd pool lsorceph auth ls. Each of these should return a permission error. The step is done when every out-of-scope attempt is refused and the refusal is logged. A command that succeeds means the caps are broader than the design, so return to the create step and tighten them. Record the test results with the change ticket so an auditor can see the boundary was tested, not assumed.Remove or rotate broad keys
Once the scoped user passes both tests in production, switch the application to the new keyring and remove the old broad key with
ceph auth del, or rotate it if other systems still depend on it. Schedule the change with the owning team, because removing a key that a hidden job still uses will stop that job. Rotate keys that were shared across hosts, and revoke any key that was stored in configuration files outside the secret store. The step is done whenceph auth lsno longer lists the retired entity and the application still works. Check the cluster audit log and the application logs for authentication failures over the following days.Document and review the user list
Record each user's purpose, owner, pools, caps and keyring locations in your inventory, and set a review date with the owning teams. Export
ceph auth lson a schedule and compare it with the inventory to catch new keys created outside the process. The step is done when the inventory matches the cluster and each user has a named owner. Ask your provider or internal team to confirm how often reviews happen and who approves new keys, since that process matters more over time than the initial setup.
How to audit keys that already exist in your cluster
Start by listing every cephx entity in the cluster with ceph auth ls, then compare that list against what is actually running. Each entity in the output shows its key and its capabilities, and the most useful first pass is to flag entities whose mon, osd or mgr caps grant broad access, such as allow *, or access to pools the named application never uses. Admin-level caps granted to an application key are the highest-priority finding, because a leaked application key with those caps exposes the whole cluster. Also look for entities that were created for a one-off task, a retired host or a test pool and never removed. Record each finding in a register with the entity name, its caps, and the date it was last used, if your monitoring can show that.
Shared keys are the next thing to find. Map each entity to the hosts or services that authenticate with it, using the client configuration, the keyring files and your deployment inventory. A single key used by several hosts cannot be revoked for one host without affecting the others, and it makes audit logs ambiguous about which workload made a request. Every key should have a named owner and a workload it serves. Entities with no owner or no identifiable workload are candidates for review, not immediate deletion.
Keyring files on clients must be checked too, because the monitor's list does not show where a key has been copied. Search the hosts that run your applications for ceph.client.*.keyring files and compare their contents with the entities you have recorded. Then confirm with each owner which clients still use a key before you remove or rotate it, since a key may still be in use by a backup job, a scheduled task or a host that is powered off. Plan rotation in stages: create the replacement entity with narrower caps, move one workload at a time, watch for authentication errors in the logs, and only then remove the old entity. Keep the old key working until the last client has moved.
Security and operational risks of over-broad cephx keys
Most cephx exposure comes from keys that were convenient to create and never revisited. The cards below cover the common ways a client key ends up with more reach than its application needs, and the control and check for each. Work through them against your current key inventory, not just the design you intended.
Admin keys on application hosts
A client.admin keyring on an application server gives that host full control of the cluster, including every pool, the monitor configuration and user management. Replace it with a per-application user that holds only the capabilities the workload needs. Check each application host's /etc/ceph directory for a ceph.client.admin.keyring file and confirm which client identity its workloads actually use.
Shared keys across tenants
When several teams or customers authenticate with the same cephx user, one compromised client exposes every pool that key can reach, and audit logs cannot separate their activity. Issue one user per application or tenant, scoped to that tenant's pools or RBD namespaces. Check the key list for users referenced by more than one unrelated workload.
Loose keyring file permissions
Keyring files are readable secrets, and a world-readable keyring lets any local user on a host act as that client. Keep keyrings owned by the service account that uses them, with restrictive mode bits, and avoid leaving copies in home directories or shared temporary paths. Check file modes and owners across hosts with a scripted audit rather than sampling a few machines.
Keys copied into config management and CI
Keys pasted into Ansible variables, Terraform state, or CI/CD variables spread to every system that can read those stores, and their history often outlives the key itself. Store the keyring in a dedicated secrets system with access logging, and let pipelines fetch it at deploy time. Check whether any repository, pipeline log, or variable group contains a base64 key string or a keyring block.
No owner for key rotation
A key that nobody owns is never rotated, and a departed engineer may still know where it is copied. Assign each cephx user a named owner, a consuming system, and a rotation process that includes adding the new key, updating clients, and removing the old one. Ask for the owner field in your key inventory and flag any user without one.
Capabilities reaching more pools than intended
A capability string that names a broad pool pattern, or omits a pool restriction, can let a client read or write data it was never meant to touch. Scope osd capabilities to named pools or RBD namespaces, and keep mon capabilities to the minimum the client needs. Check each user with ceph auth get and compare the pool list against the application's documented data flows.
How keyrings reach clients under cephadm and outside it
On a cephadm-managed cluster, the orchestrator holds the cluster's administrative credentials and distributes configuration and keyrings to hosts it manages. The admin keyring is the most sensitive file in the cluster because it grants full control, so the question is which hosts are allowed to hold it at all. Cephadm can place the admin keyring on hosts labeled for that purpose, and the rest of your fleet should receive only the keys its daemons and clients need. Treat every host that holds the admin keyring as part of your cluster's trust boundary.
- Where the key lives: The admin keyring sits on the bootstrap host and on any host given the
_adminlabel, typically under/etc/ceph. Confirm which hosts carry that label and which users can log in to them. - Who can read it: Root on those hosts can read the file, and so can anyone with sudo to root. Confirm your SSH and sudo policy for these hosts is tighter than for application servers.
- How it rotates: Rotating the admin key is a deliberate operation, not a routine. Plan it with your change process and confirm every
_adminhost receives the new keyring. - Where clients get keys: Application users should be created with
ceph authand their keyrings exported to the application host, not copied from an admin host. - What to confirm: Your own inventory of
_adminhosts, the users with root on them, and whether the admin keyring appears in backups or configuration management repositories.
Check the list of _admin hosts against your current host inventory before any audit.
Application hosts should receive only the keyring for the cephx user that the application needs, scoped to the pools and namespaces it uses. The keyring file contains a secret, so it should be readable by the service account that runs the application and by no one else. Keep one keyring per application or per host role so that a compromised host exposes one identity, not the whole cluster. Delivery is usually done by your configuration management or secrets tooling, and the file is written at deploy time.
- Where the key lives: Usually in a file such as
/etc/ceph/ceph.client.<name>.keyringor in the application's own configuration directory. Confirm the path your client library reads. - Who can read it: Set the file owner to the application's service account with mode
0600or stricter. Confirm that backup agents, log collectors and shared administrators do not get read access by default. - How it rotates: Create a new key with
ceph auth, deploy it, verify the application reconnects, then remove the old key. Confirm the deploy process can do this without downtime for your workload. - Shared hosts: If several applications share a host, avoid sharing one keyring between them. Confirm each application's keyring is separate.
- What to confirm: Which configuration management tool writes these files, whether old keyrings remain on disk after removal, and whether container images bake keyrings into layers.
Run a file permission check on every application host after each deploy, since permissions often drift when files are copied by hand.
With Rook, the operator manages the cluster and creates Kubernetes Secrets that hold the keys workloads use for block, file and object access. The key material therefore moves through the Kubernetes API and the etcd datastore, not through host files. Access control becomes a Kubernetes RBAC question. Anyone who can read a Secret in a namespace can usually read the Ceph key inside it, so the access model for these Secrets matters as much as the cephx capabilities behind them.
- Where the key lives: In Kubernetes Secrets in the namespace where the consuming workload runs, and in the namespace where the Rook operator runs. Confirm which namespaces hold copies.
- Who can read it: Any subject with
getorliston Secrets in that namespace. Confirm your RBAC roles do not grant broad Secret access to developers or CI service accounts. - How it rotates: Rotation involves updating the cephx user, then the Secret, then restarting or remounting the consuming pods. Confirm your Rook version's procedure for this, since it varies by release.
- Encryption at rest: Confirm whether the Kubernetes cluster encrypts Secrets in etcd and whether etcd backups are protected accordingly.
- External clusters: When Rook consumes an external Ceph cluster, confirm which keys are imported and whether they are limited to the pools the Kubernetes cluster needs.
Review the Rook documentation for your release before designing rotation, and confirm the Secret access model with your Kubernetes platform team.
In an air-gapped environment, keys cannot be fetched from a remote secrets service or a hosted vault during deployment, so the transfer path is manual or runs through an approved internal channel. The main risks are that keyrings get copied onto removable media, emailed, or left in staging directories, and that rotation becomes so painful that keys are never changed. Plan the transfer, storage and rotation steps as part of the deployment design rather than as an afterthought, and record each key's owner and purpose.
- Where the key lives: On the cluster hosts, in the approved internal secrets store if your site has one, and nowhere else. Confirm that transfer media is wiped or destroyed after use under your site procedure.
- Who can read it: Limit the people who can export keys from the cluster to a named group. Confirm that each export is logged in your change records.
- How it rotates: Rotation requires a physical or approved transfer and a change window. Confirm the frequency your policy requires and how long the procedure takes at your site.
- Offline backups: Confirm whether keyrings are included in offline backups, and who can restore them.
- What to confirm: Your site's approved transfer method, your key inventory format, and whether your internal tooling can record key creation and deletion events without an outside service.
Write the transfer procedure down and test it once before the first real rotation.
Limits to confirm in your environment
The cephx model is simple to describe but its exact behavior depends on your release, your clients and how you change keys. The cards below list the points to test on a non-production cluster before you rewrite any user's capabilities in production.
Cap syntax and profile availability
Capability strings and profiles such as profile rbd and profile rbd-read-only are documented per release, and the exact set you can use depends on whether you run Squid (19.2.x) or Tentacle (20.2.x). Check the capability reference for your release and run ceph auth get-or-create in a lab with the exact strings you plan to use. Do not copy examples from older blog posts or from a different major release.
Client and driver support for capabilities
A narrow capability only helps if the client library, kernel driver or hypervisor integration actually uses the operations you restricted. Some tools need broader access to create images, read configuration or list pools than you expect. Test each consumer (librbd, the kernel RBD client, CephFS mounts, RGW) against its restricted key and record every operation it fails on.
Behavior of existing sessions after a key change
Changing a key or its capabilities does not necessarily end sessions that already authenticated with the old values. Confirm, in your release, when a revoked or narrowed capability takes effect for an open connection, and whether a client must reconnect or remount. Plan the rotation window so that a capability reduction does not leave an older session with broader access longer than you intend.
Gaps for specific tools
Backup products, monitoring exporters, provisioning automation and storage orchestrators often assume a shared administrative key. Inventory every tool that touches the cluster and identify which ones need mon or mgr access beyond their own pools. Where a tool cannot run with a restricted user, decide whether to isolate it on a dedicated host with its own keyring or to accept the broader grant in writing.
Keyring distribution and auditing
Restricted users only reduce risk if keyrings stay on the hosts that need them. Confirm where each keyring is copied, who can read it, and how you would detect an unexpected client.* user. Ask your team to show how existing keys are listed and reviewed, and to confirm which of those listings your operators can run without the admin keyring.
Effect of your upgrade path
Your upgrade path can change how capabilities are interpreted or what profiles are available. Read the release notes for every major release between your current version and your target before you rewrite grants. Schedule a capability review as a step in the upgrade plan, and rerun the client tests after each upgrade stop.
Questions teams ask before scoping cephx access
How much effort does a per-application cephx key model take?
The effort depends mostly on how many clients exist, how many pools they touch, and how much of key creation and distribution is automated. A cluster with a handful of clients sharing one pool is a small inventory exercise. A cluster where every application, virtual machine host and backup job needs its own user with pool-scoped capabilities involves mapping each workload to its real access, creating users, testing each one, and removing old shared keys. Each additional client adds an inventory entry, a capability review and a test. Ask for the client count, pool layout and existing automation before estimating anything.
Can our own team scope cephx access without a partner?
Often yes, if the team already runs the cluster and has someone who understands Ceph capabilities, pools and namespaces. The work is well bounded: audit existing keys with the cluster's own tools, define one user per workload, grant the narrowest capability that works, and verify each client. The usual gap is time and review discipline rather than knowledge. Teams that lack hands for testing across every application, or that need an independent review of the resulting access model, often bring in a partner for a fixed scope while keeping day-to-day ownership in-house.
What do application owners need to provide during a cephx review?
Application owners need to confirm what their workload actually does with the cluster. That means which pools or RBD images it reads and writes, whether it needs to create or delete objects, whether it uses snapshots or namespaces, and which hosts or service accounts run it. Without this, teams tend to copy broad capabilities from old keys and never narrow them. Owners should also name who approves credential changes and who is contacted when a client breaks after a change.
How long does a key rotation window take, and what sets it?
The window depends on how many clients must be switched, how many keyrings each client reads, and how easily each application reloads credentials without a restart. Clients with a single keyring file and a documented reload step move quickly. Clients embedded in images, hypervisor configuration or third-party tools need coordinated changes and may need a maintenance window. Plan a parallel period where old and new keys both work, confirm each client authenticates with its new user, and only then remove the old key. Ask the team to inventory clients before setting any date.
Do narrower cephx capabilities affect performance?
Capability checks happen when a client authenticates and when it performs operations, and a correctly scoped capability normally does not change the I/O path for allowed operations. The practical performance risks come from the work around the change: restarting clients, reloading keyrings, or retrying failed operations while a client still holds a stale key. Overly restrictive profiles can also cause failures that look like slowdowns. Benchmark representative workloads before and after the change to confirm this in your environment, since results depend on your hardware and network.
What should we request in a fixed-scope quote for cephx work?
Ask for a written scope that names the cluster release and deployment method, the number of clients and pools in scope, the expected user and capability model, and the keyring distribution method. Request the audit method, the list of application owners who must take part, the testing approach for each client, and the rotation plan with rollback steps. Confirm what is one-time work, such as the audit and initial model, and what recurs, such as reviews after new applications are onboarded. Also ask which deliverables are documentation and which are changes to your cluster.
Which cephx capabilities should we avoid granting by default?
Broad administrative capabilities on the monitors and admin-level access should be treated as exceptions, not defaults. Requests for full access across all pools usually signal that the application has not been mapped to its real needs. Confirm each capability against the Ceph authentication and capability documentation for your release, and test that the client still works with the narrower grant before rolling it out. Treat any key that can change cluster configuration as a high-risk item to review separately.
How DeployOpen scopes and delivers cephx hardening
DeployOpen performs cephx hardening inside your own infrastructure, on your clusters, under a written scope agreed before work starts. The scope names the clients to review (applications, hypervisors, Kubernetes consumers, backup tools and administrators), the pools and namespaces they touch, and the existing keys in scope, including any client.admin copies held on workstations or in automation. Most engagements are fixed-scope, so the scope document also states what is excluded, such as network-level controls or the dashboard, which are covered in sibling sections. Ask for the scope to be written against your current ceph auth ls output so that the starting point is recorded, not assumed.
The delivered work follows a set sequence. First comes an inventory of every key with its current capabilities, last-used evidence where your logs allow it, and an owner for each client. Next comes a capability model that maps each application to a named user, a pool-scoped or namespace-scoped grant, and a profile where a built-in profile fits the workload. The model is then applied in a change window, with a rotation runbook covering keyring distribution to hosts, how keys are replaced without a service outage, and how revoked keys are removed. Verification evidence records the before and after capability listings, failed-access tests against pools a client should not reach, and the audit trail of changes. Confirm with DeployOpen which of these artifacts you receive in your own repository and format.
After the initial delivery, DeployOpen offers retainers for ongoing review. A retainer typically covers periodic key audits, new client onboarding against the capability model, rotation support and re-testing after Ceph upgrades, because capability syntax and profiles can change between releases. DeployOpen does not issue audit reports or attestations. If an assessor will test access control, the evidence package prepares your team for that review, and the independent assessor makes the determination. Ask for the retainer scope, response expectations and what happens to the capability model when your Ceph release changes, and confirm these in writing alongside the fixed-scope quote.
