How cephadm service specifications turn drives into Ceph OSDs
An OSD service specification tells cephadm which drives on which hosts should become Ceph OSDs, and it is the point where a team stops running one-off commands on each host. You need one when a new cluster or an expansion must produce the same drive layout on every host, because the alternative is a set of manual steps that drift apart over time. Instead of naming individual devices, you declare device filters for data devices and, where HDD OSDs use SSDs for metadata, for DB devices. cephadm evaluates those filters against the inventory of each host and creates OSDs that match. When you later add a host or replace a failed drive, the same specification applies again, so the layout does not depend on whoever happened to be running the commands.
The filters are where most of the design work sits. Typical selectors include whether a device is rotational (HDD) or not (SSD or NVMe), its size range, and its model string, and these can be combined so that, for example, HDDs of one capacity become data devices while a pair of enterprise SSDs becomes the shared DB devices. Before applying anything, run the specification as a dry run and read the planned actions for every host. Check that boot drives, drives already in use and devices you intended to exclude are not selected, and that each host's DB device count and sizes match the plan. The all-available-devices option is convenient for lab clusters, but it claims every eligible device it finds, so production teams usually write explicit specifications. Confirm the exact filter fields and their behavior for your release in the cephadm documentation, since the available options change across versions.
Once the specification is applied, cephadm keeps reconciling it as hosts and drives change, which is useful and also a risk. A newly inserted disk that matches a filter can be turned into an OSD without a review step, so decide in advance whether automatic creation is acceptable in your environment or whether devices should be admitted only after a check. Unmanaged mode and encryption settings also belong in this decision, because they determine who is responsible for an OSD's lifecycle and whether its keys are handled correctly. Record the specification in version control alongside your host inventory, and have someone verify the dry-run output for each host class before every change window.
Which OSD spec approach fits which cluster
This table compares explicit host or label specs, all-available-devices, unmanaged mode and encrypted OSD options against the cluster situations each suits and the risk each carries. Read each row as a decision: match your host count, hardware uniformity and change rate to a row, then check the risk column before you apply it.
| Approach | Best fit | Main risk |
|---|---|---|
| Explicit spec by host | Small or mixed clusters where each host has a known drive layout and you want one spec per group of hosts | Each new host needs its own spec or label match, so a host added without one gets no OSDs until someone acts |
| Explicit spec by label | Clusters with several hardware generations where hosts share a role label such as osd-hdd or osd-nvme | A host carrying the wrong label can claim drives you did not intend, so label discipline must be enforced |
| All-available-devices | Lab or proof-of-concept clusters with identical hosts and no shared SSDs for DB or WAL | Any newly inserted or unused disk on any host can be turned into an OSD, including disks that hold data for other systems |
| Unmanaged mode | Sites with custom provisioning scripts, unusual device layouts or a requirement to create OSDs by hand | No automatic OSD creation on new drives, so capacity and drive replacement depend on runbooks your team keeps current |
| Encrypted OSDs in a spec | Data at rest requirements where drive-level encryption is part of the design, with key handling agreed up front | Key custody and escrow decisions are easy to defer, and a lost key can make OSD data unrecoverable |
| Dry run before apply | Any cluster where a spec change could claim drives, which is most production changes | A dry run shows the planned result for the current inventory only, so re-check it after hardware changes |
Common drive layouts: all-flash, HDD with shared SSD, and mixed hosts
When every data drive on a host is solid state, the OSD specification is usually the simplest of the three patterns. Data and metadata share one tier, so there is no separate DB device layer to plan. The trade-off is that each failed drive removes a faster, more heavily used OSD, and the recovery work lands on the remaining OSDs in the same host and class.
- One data spec: Filter on the non-rotational flag so the spec selects every SSD on the host and nothing else.
- Dedicated drive per OSD: The hardware recommendations call for one drive per OSD, so the spec should never place two OSDs on one device.
- Enterprise-class media: Metadata-heavy workloads depend on enterprise-class SSDs, so confirm drive models in the dry run before you apply.
- Failure impact: A single drive loss triggers backfill to the other OSDs in the host group, so check network headroom and recovery capacity.
- Labels: One host label or pattern usually covers the whole tier, which keeps the spec count low.
Before applying, confirm that the dry run lists every intended SSD and excludes the operating system and boot drives.
HDD OSDs usually place their WAL and DB on a shared SSD. The specification therefore describes two device sets: rotational drives for data and a smaller set of flash devices for metadata. The two filters must select disjoint devices on each host, or an OSD can be created without its DB device or a flash device can be claimed twice.
- Two device filters: The data filter selects rotational drives, and the DB filter selects non-rotational drives. Check that the two sets never overlap.
- Shared SSD blast radius: Losing one shared SSD affects every HDD OSD that uses it, so the impact is larger than losing one data drive. The hardware recommendations advise enterprise-class SSDs for this role.
- Sharing ratio is a design input: How many OSDs share one SSD is decided during sizing and should be fixed before you write the DB filter.
- Dry run before apply: Preview the mapping of HDDs to SSD partitions with a dry run so you can see which flash device serves which data drives.
- Labels for mixed racks: If HDD and flash hosts share a rack, separate them with labels so each spec targets only its own group.
After deployment, confirm that each HDD OSD reports its DB device on the SSD you intended.
When hosts differ in drive count, drive model or media type, a single generic specification either matches too much or too little. Separate specifications, each bound to a host label or host pattern, keep every host's layout explicit. Hardware generation changes are the most common reason a cluster ends up with more than one layout.
- One spec per layout: Give each specification its own service ID so that
ceph orch lsshows which layout owns which OSDs. - Placement by label: Bind each spec to a host label or pattern so that a new host joins its layout by label alone, without edits to unrelated specs.
- Overlap checks: Two specifications that match the same host can compete for the same devices. Run dry runs for both and review the combined output.
- Model filters: Filter by drive model only when models are consistent within a group. Review any group that mixes models before applying.
- Hardware drift: Each new hardware batch may need its own specification, so record which generation each host belongs to.
Review the dry run output host by host, not only as a total device count. The cephadm documentation describes how specifications and placement interact.
Writing, dry-running and applying an OSD spec on cephadm
This sequence turns a set of bare hosts into OSDs placed by a reviewed declarative spec, so every drive that joins the cluster was previewed, approved and checked before it stored data.
Confirm the release and cephadm baseline
The platform engineer confirms that the target release is one cephadm supports. cephadm was introduced in v15.2.0 (Octopus) and does not support older releases, and the Ceph releases page lists Squid and Tentacle as the actively maintained releases. Record the exact version running on the cluster's container images and confirm that every host runs a compatible container runtime and the cephadm orchestrator is active. Done when
ceph orch statusreports the orchestrator as available and the version matches the change record.Inventory devices on every host
Run
ceph orch device lson the admin host, then review the output for each OSD host. Note the device path, type, rotational flag, size, model and whether each device is marked as available. Confirm that boot drives, operating system mirrors and any drives holding data from earlier use are marked unavailable, and that the hosts being added match the expected drive count from the hardware plan. Done when every host's inventory has been reviewed and each candidate data drive and each candidate shared SSD is identified by its model and size.Write the OSD service specification
Write the spec as a YAML file that names the service type
osd, aservice_id, aplacementblock that selects hosts by label or host name, and aspecblock withdata_devicesand, for HDD-based OSDs,db_devices. Filter devices byrotational,sizeormodelrather than by device path, because paths can change after a reboot. Addencrypted: trueonly when the design calls for it. Keep the file in version control. Done when the file passes a YAML lint and the reviewer agrees each filter matches the inventory identified in the previous step.Run a dry run to preview matches
Apply the spec with the dry-run option, for example
ceph orch apply -i osd-spec.yaml --dry-run, so cephadm reports which devices it would use and which hosts would receive OSDs without creating any. Compare the preview against the inventory: the count of data devices per host, the DB devices assigned to each HDD, and any device that matched unexpectedly. Adjust the filters and repeat until the preview contains only intended devices. Done when the preview is saved with the change record and a second reviewer has signed off on it.Apply the spec and watch deployment
Apply the spec for real with
ceph orch apply -i osd-spec.yaml. cephadm then creates the OSDs on the matched hosts, which takes time that depends on device count, device size and recovery load. Monitor progress withceph orch psandceph -sand keep the change window open until the cluster stops reporting new OSDs as starting. Done when the orchestrator reports the OSD service as running and no host shows a failed OSD deployment.Verify OSD count and host placement
Run
ceph osd treeandceph orch ls osdto confirm that the number of OSDs equals the number of matched devices in the dry run, and that each OSD sits under the host you expected in the CRUSH hierarchy. Checkceph orch psto confirm that the daemons run on the placed hosts only. Done when the counts and host mapping match the reviewed preview, and any gap is traced to a specific device in the inventory.Verify device class and encryption
Run
ceph osd treeorceph osd crush class lsto confirm that each OSD carries the device class you planned, such as hdd or ssd, because pools and CRUSH rules depend on that class. For encrypted OSDs, confirm the encryption setting in the applied spec and check that the OSD metadata reports it. Done when every new OSD has the expected class and encryption state recorded, and the results are added to the handover notes.Record the spec and confirm cluster health
Store the applied spec, the dry-run output and the verification results together. Run
ceph health detailand confirm the cluster is healthy or that any warning is understood and tracked. Ask your Ceph administrator to confirm the retention location and the process for changing the spec later, since future drive additions should go through the same dry-run review. Done when the record exists and the cluster health state is documented.
Security and operational risks when cephadm applies drive specs
A service specification tells cephadm which drives to consume, and cephadm acts on that match without a human confirming each device. The cards below cover the places where a loose filter, a lost encryption key, broad SSH access or an unreviewed spec edit can turn an OSD deployment into data loss or an outage.
Wrong device matches wipe data
A filter that matches on rotational flag, size or model can select a disk that already holds data, such as a boot device, a former cluster member or a drive carrying another system's volumes. Because cephadm zaps devices it adopts for OSDs, a broad match is destructive. Run the spec in dry-run mode on every host, compare the proposed devices with the hardware inventory, and confirm that no OS or boot disk appears in the output before applying.
Encrypted OSD key handling
When a spec enables encryption, the dm-crypt keys become part of recovery. If keys are stored only on the host that failed, an OSD cannot be reopened after a drive is moved to another server. Confirm where cephadm keeps the keys for your release, decide how they are backed up, and rehearse opening a moved encrypted drive in a test cluster so the procedure is known before a real failure.
SSH access used by cephadm
cephadm manages hosts over SSH using a key held by the cluster, so whoever controls the orchestrator can change drives on every managed host. Restrict the account it uses, limit which administrators can apply specs, and review the hosts and labels the orchestrator can reach. Hardening of host access is covered on the Ceph security hardening pages, and the planning decision belongs here because it determines who can trigger a drive wipe.
Message traffic encryption
OSD deployment and client traffic cross the Ceph messenger. Messenger v2 listens on port 3300 and offers a secure mode that encrypts all post-authentication traffic with AES-GCM and adds integrity checks. Confirm whether secure mode is enabled for your release and whether your clients and daemons negotiate it, because a compatible-mode default may still allow unencrypted connections.
Change control for spec edits
A spec is applied to the whole cluster, so a one-line change to a filter can add or remove many OSDs at once. Keep specs in version control, require a reviewed diff for each edit, and record the dry-run output that went with it. Treat spec changes like infrastructure changes with a named approver and a rollback plan, not like a configuration tweak.
Confirm settings for your release
Defaults for OSD specs, encryption and messenger modes change between Ceph releases, and cephadm behavior depends on the version you install. Read the cephadm documentation for the exact release you deploy, check the active release list for end-of-life dates, and test each spec against that version in a lab before production. Record the version and the confirmed settings in your change record.
Questions buyers ask about OSD service specifications
What drives the cost of writing OSD service specifications for a Ceph cluster?
The main drivers are host count, drive variety, the number of distinct specifications, and how much testing the change needs before production. Each host type that carries a different drive mix adds a layer of filter design, dry-run review and verification. A cluster of identical hosts with one drive model is a small job, while a fleet with several HDD models, SSDs for DB devices and mixed host generations takes more design and more checks. Ask any partner to price these items separately in the written scope: inventory review, spec design, dry runs on each host type, a test window on non-production hardware if you have one, and handover. Costs are mostly one-time labor. The recurring part is the work you take on when you add hardware later.
How long does it take to get OSD specs right before go-live?
Duration depends on how many host types you have, how inconsistent your drive inventory is, and how many change windows you can use for testing. Specs are quick to write once the inventory is clear, but confirming how cephadm matches devices on each host, running the preview and reviewing the results across every host type takes time. Plan for a dry run on each distinct host type before any apply, and ask for a schedule in the written scope rather than relying on an estimate in conversation. No duration is guaranteed here. Your inventory and the number of change windows you can secure will set the real timeline.
Can our own team write and apply OSD service specifications without a partner?
Yes, many teams can. The work needs someone who knows the Ceph cluster, can read the output of cephadm's device inventory, and is comfortable running a dry run before any change. The harder part is consistency across hosts, handling drives that enumerate differently after a reboot or replacement, and keeping the spec file in version control so future changes are reviewed. Teams with one or two engineers familiar with Ceph often manage this internally. Teams without that experience, or those with mixed hardware generations, benefit from a review before the first apply.
What do we need to provide before the OSD specification work starts?
You need an accurate hardware inventory, access to the cluster with administrative rights through cephadm, and agreed change windows. The inventory should list each host, its drive models, whether each drive is rotational, its capacity, which drives hold DB or WAL devices, and which drives are reserved for the operating system. You also need a current view of the cluster's health, a backup or recovery plan for the monitors, and a named person who approves each apply. Access should be scoped to the cluster and to the hosts involved, with a record of who ran each command.
Why does an all-available-devices spec cause problems in production?
An all-available-devices configuration tells cephadm to consume every eligible unused device it finds, which can include drives you intended to keep for other purposes. That is convenient on a lab cluster but risky in production, where a newly attached disk, a spare or a drive that is only temporarily idle could be claimed for an OSD. Explicit specs with filters for rotational type, size and model make the intent visible in version control and easier to review. Test any change to the spec with a dry run before applying it, because the preview shows which devices would be claimed.
When does a partner make the OSD spec work faster or safer?
A partner helps most when the cluster has many host types, inconsistent inventory, encrypted OSDs with key management decisions still open, or a hard change window that leaves little room for mistakes. It also helps when your team has not yet run a dry run and apply cycle on production-like hosts. DeployOpen plans, implements and documents this work on infrastructure you control, under a written scope agreed before work starts. Ask for the deliverables in writing, including the spec files, the dry-run records, the verification steps and the runbook for future drive replacements.
What changes when we use encrypted OSDs or unmanaged mode?
Encrypted OSDs add key handling to the design, since encryption keys must be stored and recovered reliably, and a lost key can make data unreadable. Unmanaged mode means you create the OSDs yourself with commands instead of letting a spec manage them, which gives control but moves the consistency work to your process. Both choices should be decided before the first apply because switching later means reviewing existing OSDs. Confirm the exact encryption and unmanaged options supported by your Ceph release in the cephadm documentation, and record the decision in your runbook.
How DeployOpen scopes OSD deployment work
DeployOpen begins OSD work with a drive layout review before any specification is written. The review inventories every host that will carry OSDs, records drive type, size, model and whether each device is rotational, and identifies which devices hold the operating system, which are reserved for other uses, and which will become data devices or BlueStore DB devices. The output is a written inventory that the client signs off, because a filter that matches one unexpected disk can consume hardware nobody meant to give to Ceph. The review also confirms the host layout that the cephadm deployment already assumes, so the spec matches the cluster as it will be built rather than as it was first drawn.
Specifications are then written and tested in a lab or staging cluster that matches the production hardware as closely as the client can provide. The team applies each service_type: osd spec with a dry run first, reads the planned device list that cephadm reports, and compares it line by line with the inventory. Where the plan differs, the filter is adjusted and the dry run repeated. The tested specs, the dry-run output, the host inventory and the change record are documented together, so the client's operators can see why each filter was chosen and what they should expect when a drive is replaced. Confirm with your own hardware which cephadm version and filter fields your environment supports, because the exact options available depend on the release you deploy.
