Ceph encryption at rest with encrypted OSDs

Ceph encryption at rest with encrypted OSDs: when your team needs it

If your organization must show that data on disks, including drives taken out of service, cannot be read by someone who gets physical custody of them, plan encryption at rest before the first OSD is created. Encrypted OSDs protect Ceph data at rest through LUKS-based device encryption, applied to each OSD's underlying device. The choice is made per cluster during planning, because it is set when OSDs are deployed from the cephadm service specification and is not something you switch on later without rebuilding the affected OSDs.

Encryption at rest is a separate control from msgr2 secure mode. Msgr2 secure mode encrypts all post-authentication traffic between Ceph daemons and clients with AES-GCM, which protects data while it moves across the network. It does nothing for a drive that leaves your data center in a box. An auditor testing storage encryption will usually ask about both, so your evidence should cover the disk layer and the wire separately, with each setting recorded in your cluster configuration.

The control has running costs that belong in your plan. Encryption keys must be stored, backed up and recovered, and losing them makes the data on those OSDs unreadable. Repair and recovery tools may need to be run with the device key available, and your runbooks should say who holds that access. Confirm with your Ceph version's documentation how TPM2 enrollment is supported on your release, and test a drive replacement and a full host rebuild in a non-production cluster before you rely on the design in production.

Encryption options for Ceph OSDs and when each fits

This table compares the main choices for encrypting Ceph OSDs: how much of the cluster to encrypt, how the keys are held, and whether existing OSDs must be rebuilt. Read each row as a decision, then confirm the option's exact fields and defaults in the official cephadm documentation before writing a spec.

Decision	Option	Fit and condition to confirm
Scope	Encrypt all OSDs	Fits when the whole cluster holds data in scope for your policy. Confirm in the cephadm OSD service spec documentation how encryption is set per spec and whether mixed spec sets are supported.
Scope	Encrypt a subset of OSDs	Fits when only some hosts or device classes hold sensitive data. Expect separate pools or placement rules to be needed, and confirm how mixed encrypted and unencrypted OSDs are reported in cluster health.
Key handling	Passphrase-based keys	Fits when you can manage passphrase custody and rotation under your own key procedures. Confirm where the passphrase is stored, how it reaches hosts at boot, and how a lost passphrase is recovered.
Key handling	TPM2 enrollment (Squid onward)	Fits when hosts have a TPM2 device and you want keys bound to the host hardware. Confirm the Squid release level you run, the TPM2 settings your cephadm version accepts, and the procedure for hardware replacement.
Existing hosts	New hosts only	Fits when you are adding capacity and can drain or retire older hosts over time. Confirm how new encrypted OSDs coexist with unencrypted ones during the transition and how the health view reports them.
Existing hosts	Rebuild existing OSDs	Fits when older OSDs must be encrypted and you can afford the data movement. Plan one OSD or host at a time, confirm the replacement procedure in the cephadm documentation, and check that cluster health returns before the next step.
Repair and recovery	Plan for repair tooling limits	Encrypted OSDs need their keys available to offline repair tools. Confirm with the vendor documentation for your release which tools accept encrypted devices and what key material they need before an incident.

Where the OSD encryption key lives: passphrase or TPM2

With a passphrase, the LUKS key is protected by a secret that an administrator or automation supplies when the device is opened. The people who create the passphrase and the system that stores it decide who can bring an OSD online. After a reboot, encrypted OSDs stay locked until the passphrase is supplied, so the host's data is unavailable until unlock runs. Confirm how your Cephadm documentation expects unlock to be handled on your release and how your boot process retrieves the secret.

Who holds the secret: the team that creates the passphrase and the vault or secrets service that stores it. Access to that store is access to the data on the drives.
After a reboot: OSDs wait for unlock, so plan boot ordering and expect slower recovery after a host outage.
After a hardware swap: a drive moved to another host can be opened with the same passphrase material, but the new host needs a working unlock path.
Recovery: if the passphrase is lost, the encrypted data cannot be recovered. Keep an escrowed copy under the same access controls as the drives.
Before production: test a host reboot, a drive replacement and an unlock from the escrowed copy.

Check where the passphrase is stored at unlock time and which people and services can read it.

Setting up encrypted OSDs with cephadm

This sequence takes a cephadm-managed Ceph cluster from a confirmed release and prepared hosts to OSDs that encrypt their data at rest, with checks at each stage so you know the encryption is active before data lands on the devices.

Confirm the release and management model
Your platform or storage lead confirms that the cluster runs Squid or Tentacle, the two actively maintained releases on the Ceph releases page, and that cephadm manages the cluster. Squid or Tentacle is the minimum for the TPM2 key method, and cephadm does not support releases older than v15.2.0, so confirm both before writing any spec. Check the running version with ceph versions and the cephadm status with ceph orch status. The step is done when every daemon reports a supported release and the orchestrator responds without errors. Confirm TPM2 availability separately in the next step if you plan to use it.
Prepare hosts and devices to the hardware guidance
The infrastructure team prepares each OSD host against the Ceph hardware recommendations: one dedicated drive per OSD, WAL and DB on shared enterprise-class SSDs for HDD-based OSDs, and bonded network links across two switches. Encryption adds a layer of work on each device, so confirm that the hosts have enough memory for peak recovery load, not just idle use, and that the team has benchmarked the hardware. Confirm that target devices hold no data you need, because the encryption setup overwrites them. Run ceph orch device ls from the admin host. The step is done when every intended device appears as available and no device carries a filesystem or LVM label you did not expect.
Define the OSD service spec with encryption
The storage engineer writes an OSD service specification (a YAML file applied through cephadm) that selects devices and sets encryption on. Confirm the exact field names, including the encryption flag and device selectors, in the cephadm documentation for your release, since they change between versions. Keep the device filters narrow enough that only the intended drives match, and keep the spec in version control so changes are reviewed. The step is done when the spec file parses and a dry run shows the expected devices as matches. Record the spec alongside the cluster's change log so that later OSD replacements use the same settings.
Choose and record the key method
The security lead chooses how OSD encryption keys are stored and unlocked. The options differ by release, and TPM2 enrollment is available from Squid onward, so the choice depends on what the hardware provides and how the hosts boot. Decide who holds recovery material, where it is stored, and how it is rotated, and document the procedure before any device is encrypted. Lost keys mean lost data, so the team should test recovery on a non-production host first. The step is done when the key method, the recovery procedure and the custodians are written down and approved, and a test host has been unlocked from that procedure.
Apply the spec and watch the OSDs deploy
The platform engineer applies the spec with ceph orch apply -i on a maintenance window agreed with the application owners, since OSD creation generates heavy I/O. Watch progress with ceph orch ps and ceph orch ls osd, and check the cephadm log for errors on each host. Expect the cluster to show rebalancing as new OSDs join. The step is done when every targeted OSD reports up and in, and the cluster has no unexpected daemons left in an error state.
Verify encryption on every OSD
The storage engineer confirms that each new OSD is actually encrypted, not just deployed. Use ceph osd metadata and ceph device ls to match OSD IDs to physical devices, then inspect each host's block devices and confirm the expected encrypted (dm-crypt or LUKS) mapping is present. Check that the key method matches the approved record. Do not rely on the spec alone, because a device that was skipped or matched a broader filter will not show encryption. The step is done when a written table lists every OSD, device and encryption state, and any mismatch is resolved before data is stored.
Confirm cluster health before accepting data
The operations team confirms that the cluster reaches a steady health state after the OSDs join. Run ceph -s and ceph health detail until the cluster reports HEALTH_OK, or until every remaining warning is understood and accepted. Confirm that placement groups have finished peering and backfilling and that no OSD is flapping. Record the baseline output and the spec version so later audits can tie the encryption setup to a specific change. The step is done when the baseline is saved, and the team has agreed that the cluster is ready for workloads.

How at-rest encryption pairs with msgr2 in transit

OSD encryption and msgr2 secure mode protect different things, so a design that handles sensitive data usually turns on both. OSD encryption protects data sitting on drives, so a stolen or decommissioned disk yields ciphertext instead of object contents. msgr2 secure mode protects data moving between Ceph daemons and clients. Ceph's messenger v2 protocol, introduced in Nautilus on port 3300, can encrypt all post-authentication traffic with AES-GCM and add integrity checks. Neither control replaces the other: encrypted disks do nothing against a sniffer on the cluster network, and secure mode does nothing for a drive pulled from a rack.

Confirm how secure mode is enabled before you size the rollout. The msgr2 documentation describes the connection modes and the settings that select them, and your team should verify those keys against the Ceph release you run, since the defaults and accepted modes can differ between daemons, clients and gateways. Check every consumer on the path, including kernel and FUSE CephFS clients, RBD clients in hypervisors, and the Object Gateway, because a client that cannot negotiate secure mode will either fail to connect or force a fallback you did not intend. Ask each integration vendor to state the Ceph client version it embeds and whether it supports the secure connection mode.

Plan for the cost of both controls in measured terms rather than assumptions. Encryption adds CPU work on the OSD, client and gateway hosts, and the effect depends on hardware acceleration, I/O size and network speed, so benchmark a representative workload with both controls enabled before you commit to a hardware design. Record the results alongside your key management and backup decisions, since the encrypted-at-rest keys and the transport settings both need documented owners and tested recovery procedures.

Security and operational risks to plan for

Encrypting OSDs protects data on discarded or stolen drives, but it adds dependencies on key material and on the host boot path. The cards below cover the decisions and tests that connect key handling, recovery work and unlock monitoring so that encryption does not become a source of data loss.

Key loss means permanent data loss

Each encrypted OSD depends on a LUKS key held by the cluster configuration. If the key store is lost or corrupted, the data on those OSDs cannot be read, even when the drives are intact. Decide where keys are escrowed, who can restore them, and test a full key restore on a non-production cluster before go-live. Confirm how keys are backed up and whether your backup policy covers them separately from the data.

Repair tools need the decrypted device

Offline repair and recovery utilities, such as those used after a corrupted OSD or a failed filesystem check, expect a plain block device. An encrypted OSD must be opened first, which changes the runbook and the people allowed to run it. Document the unlock step for each repair tool, limit who holds that access, and rehearse one repair on a test OSD so the team knows the steps under pressure.

Key handling on replacement drives

When a failed drive is replaced, the new OSD needs a fresh key and a clean enrollment path. Mistakes here can leave a drive encrypted with a key nobody retained, or leave old key material on a drive sent for return or disposal. Define the replacement procedure, confirm that old keys are removed from the cluster when a device is retired, and include drive destruction or secure erase in the decommissioning checklist.

TPM2 binding ties unlock to hardware

From Squid onward, keys can be enrolled against a TPM2 module so that OSDs unlock automatically on the host they were built for. This protects against drives moved to another chassis, but it also means a failed motherboard or TPM can block unlock. Confirm your Squid release with the Ceph releases page, verify the TPM2 setup on every host model you run, and decide the fallback process before hardware is refreshed.

Monitor unlock failures after reboot

An OSD that fails to unlock after a host reboot stays down, and the cluster rebuilds data it can no longer reach from that device. Alert on OSDs that do not return after boot, not only on cluster health warnings, and track which hosts depend on TPM2 or key-file unlock. Test a controlled reboot of each host type during commissioning so that unlock failures appear in the lab rather than during a maintenance window.

Documenting controls for auditors

Auditors will ask how keys are generated, stored, rotated, restored and destroyed, and how encryption coverage is proven for every OSD. Write these controls as runbooks with owners, record evidence from restore and decommissioning tests, and keep inventory output that maps each OSD to its encryption state. DeployOpen prepares this evidence, but independent auditors decide whether the controls meet the framework you are assessed against.

Questions buyers ask before encrypting Ceph OSDs

What does it cost to encrypt Ceph OSDs?

Encryption adds effort more than license cost, because Ceph itself is open source under LGPL terms and the project sells no paid edition. The cost comes from planning the key process, writing and testing cephadm OSD specs, rebuilding or draining OSDs, and validating performance on your hardware. Part of the work is one-time (design, key workflow, runbooks, test evidence). Part recurs (key rotation handling, replacing drives, and checking encrypted OSDs during upgrades). Ask any quote to split these items, to name who holds the keys and who is paid for each step, and to state whether hardware benchmarking is included.

Can an existing Ceph cluster be encrypted in place?

Usually not in a single step. Encryption is applied when an OSD is created, so existing OSDs are typically drained, removed and recreated as encrypted, one host or failure domain at a time, while the cluster rebalances. Whether this is practical depends on spare capacity, the current health of the cluster, and how much data must move during each cycle. Confirm the exact procedure for your release with the Ceph cephadm documentation and your own test cluster before committing to a schedule.

How long does an encryption rollout take, and what drives that time?

The duration depends mostly on how much data must be rebalanced and how fast the network and drives can absorb recovery traffic. Other drivers include the number of hosts, the size of each OSD, the number of maintenance windows your change process allows, and how many rounds of testing you run first. A useful quote will show the schedule as a sequence of host or failure-domain batches with checkpoints, not a single end date. Ask for the recovery throughput assumptions and the rollback steps for each batch.

What must our team provide before the work starts?

Expect to provide access to the hosts and cephadm environment, the network and change windows, and a decision on where encryption keys are held. The key process matters most: whether keys are stored on the host, enrolled through TPM2 on a release that supports it, or held in an external key service that you operate. You also need a test window on a non-production cluster, spare capacity for data movement, and named owners who can approve each batch. Confirm which of these items your environment already has before scoping.

Can our own team encrypt the OSDs without a partner?

Yes, if your team already runs cephadm clusters, understands CRUSH failure domains and can handle recovery events during business hours. The steps are well documented, but the risk sits in sequencing: one wrong drain or a missed health check can extend recovery for hours or days. Many teams keep the work in-house and buy a fixed-scope review of the spec, the key process and the rollback plan. DeployOpen plans, implements and documents this kind of work in infrastructure you control, and the scope is agreed in writing before work begins.

How does encryption affect repair and drive replacement?

Replacing a failed drive now also requires handling its encryption key material, so the runbook must cover how a new OSD is enrolled and unlocked. Some offline repair and recovery tools cannot read encrypted OSDs without the keys, which must be available to the operator at the time of repair. Keep backups of key material under a controlled process, test a replacement on a spare drive, and confirm with the Ceph documentation which tools and procedures support encrypted OSDs in your release.

Does encryption at rest replace encryption in transit?

No. Encryption at rest protects data on drives that are stored, removed or disposed of, while in-transit protection covers traffic between Ceph daemons and clients. Ceph's messenger v2 protocol on port 3300 offers a secure mode that encrypts post-authentication traffic with AES-GCM and adds integrity checks. Auditors and security teams usually expect both controls, so scope them together and confirm the settings for your release with the Ceph msgr2 documentation.

Limits to confirm in your environment

Encrypted OSDs change your release plan, your hardware, your repair runbooks and your audit evidence at the same time. The cards below list the items to verify with your own versions, hardware and compliance scope before you commit to a design.

Release lifecycle and Tentacle timing

Squid is the release where TPM2 enrollment is available, and its estimated end of life is October 31, 2026. Tentacle is the newer actively maintained line, with an estimated end of life of June 1, 2027. Confirm which release you will run at go-live, whether your patch window falls before Squid ends support, and how you will move to Tentacle with encrypted OSDs already in place. Ask your team to test an OSD upgrade on a non-production cluster first.

TPM2 hardware support

TPM2 enrollment depends on the hardware in every OSD host. Confirm that each server has a TPM 2.0 module that is enabled in firmware, that your hardware vendor supports it for the operating system you run, and that the module can be reached from the container runtime cephadm uses. Mixed fleets need a documented fallback key method for hosts without a usable TPM.

Repair and offline tools on encrypted devices

Offline tools, such as manual OSD repair or disk inspection utilities, may not read LUKS-encrypted devices unless the key is available to them. Build a runbook that states which tools your team uses, how each one obtains the key, and whether the procedure works during an incident when the key service may be unavailable. Test these procedures before you need them, since a failed repair with a missing key can delay recovery.

Performance on your hardware

Encryption adds CPU work to every OSD read and write, and the cost depends on your processors, drive types and network speed. Benchmark an encrypted and an unencrypted OSD on the same hardware with your real workload mix, including recovery and scrub activity. The Ceph hardware recommendations advise benchmarking before purchase because no two clusters behave alike, so use the results to size CPU and drives.

Key method and compliance scope

Your auditor or assessor decides whether the key method you choose meets the encryption-at-rest requirements in your scope. Document where keys are stored, who can release them, how keys rotate, and what evidence proves that an OSD is encrypted. DeployOpen can prepare configuration and evidence for review, but independent auditors decide whether the design is acceptable. Ask early, before the design is fixed.

Gateways and CephFS clients

Encryption at rest protects OSD devices, not every path a client uses. Confirm how RGW gateways, CephFS mounts, RBD clients and any NFS or SMB gateways handle encryption in transit and client-side encryption for your data. Each access path can carry its own configuration and its own assessment, so list them in your design and confirm each one separately before you describe the cluster as encrypted to your stakeholders.

How DeployOpen scopes and delivers encrypted Ceph OSDs

DeployOpen plans and delivers encrypted OSDs inside your own data centers or cloud accounts, starting from a written scope that names the hosts, devices, key handling and recovery procedures involved. The work is built around cephadm specs, tested on a staging cluster before production, and handed over with documentation your operators can run.

How we scope

Inventory every host, drive and OSD that will be encrypted, and note which devices carry WAL and DB volumes.
Agree in writing how encryption keys are stored, escrowed and rotated, and who holds recovery access.
Confirm TPM2 enrollment requirements against your hardware and the Ceph release you plan to run.

How we build

Write cephadm OSD service specs with encryption enabled and test them on a staging cluster first.
Validate that msgr2 secure mode is enabled alongside encrypted OSDs, so data is protected in transit and at rest.
Test drive replacement and host reboot procedures on encrypted OSDs before any production change.

How we hand over

Deliver the written spec files, key handling procedure and a runbook for replacing an encrypted drive.
Document the repair and recovery caveats your operators must follow, including tooling that needs key material.
Walk your operations team through a verification checklist that confirms encryption on every OSD.

How we support

Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
Ongoing support, upgrades and continuous testing are available as a retainer after handover.
Ask for a quote based on your written scope, since DeployOpen does not publish a price list.

Ceph encryption at rest with encrypted OSDs