Mixing HDD, SSD and NVMe in Ceph with device classes

Mixing HDD, SSD and NVMe in Ceph with device classes: when a single cluster needs more than one media tier

A team that wants bulk capacity on HDD and low latency on flash can often run both in one Ceph cluster, but only if each pool is placed on the right media. Device classes are the mechanism that makes this possible. Ceph tags each OSD with a class, such as hdd, ssd or nvme, and CRUSH rules can then select only OSDs of a given class. A pool bound to a class-specific rule stores its data only on that media, so a latency-sensitive block pool and a large archive pool can share hosts, monitors and networks without competing for the same drives.

Whether to split pools by media depends on three things: the workload's latency and throughput needs, the failure domains you can build with the hardware you have, and the cost per usable terabyte in each tier. A small cluster with a handful of hosts per class may not have enough failure domains to place replicas safely in each tier, so the split can reduce resilience rather than improve it. Confirm the number of hosts that carry each class before you write the rules, and check the device class assigned to each OSD on your release, since automatic detection depends on what the drive reports. The Ceph releases in scope for new deployments are Tentacle and Squid, and the rule syntax should be checked against the documentation for the release you intend to run (Ceph active releases).

Which media layout fits which workload

This table compares three ways to place HDD, SSD and NVMe media in a Ceph cluster: single-class pools, hybrid rules with an SSD primary and HDD replicas, and all-flash pools. Read each row against your workload's latency needs, capacity target and tolerance for a slower read path after an SSD failure. The main takeaway is that the layout you choose sets both the performance you can promise and the capacity you pay for.

Layout	Workload fit	Latency and capacity effect	Main risk
Single-class HDD pool	Bulk capacity, archives, backups and throughput-tolerant data where latency is not the goal	Lowest cost per usable terabyte, with higher and less predictable latency under random I/O	Slow recovery and client response when many OSDs rebuild at once
Single-class SSD pool	Databases, virtual machine disks and metadata-heavy services that need consistent response times	Lower latency than HDD and higher cost per usable terabyte; capacity is set by the SSD tier	Cost growth if data that does not need flash lands in this pool by default
All-flash NVMe pool	Latency-sensitive workloads with high IOPS needs and a network that can keep up	Lowest latency of the three options, with the highest cost per usable terabyte	CPU, memory and network can become the bottleneck before the drives do
Hybrid rule, SSD primary and HDD replicas	Read-heavy data that needs fast reads from a bulk pool, where some write latency is acceptable	Reads are served by the SSD primary, while writes still wait for HDD replicas; capacity mostly follows the HDD count	If an SSD primary fails, reads fall back to a slower HDD replica until recovery completes
Mixed pools across device classes	Organizations that need several service levels on one cluster	Each pool keeps its own rule and replication, so capacity and latency are planned per pool	More rules and placement groups to design, monitor and explain to application owners

Why a hybrid rule with SSD primaries slows reads after a failure

A hybrid CRUSH rule places one replica on SSD and the others on HDD, and Ceph serves client reads from the primary OSD of each placement group. When that primary is an SSD and it fails, the next acting OSD is usually an HDD, so the same data is read from slower media until the cluster settles.

Primary selection

Ceph serves reads and writes through the primary OSD of each placement group, and the primary is the first OSD in the acting set. A hybrid rule that lists an SSD first makes that SSD the usual primary, so clients get SSD latency only while it stays healthy. Confirm in your release's CRUSH and primary-affinity behavior which OSD is chosen as primary before you rely on it for latency targets.

Failure impact on reads

When an SSD primary fails, the placement group peers and an HDD replica becomes primary. Reads for those objects then run at HDD seek and queue latency, which can be far higher than the SSD baseline your application was tested against. Measure read latency with the primary removed, not only during normal operation, and record the result as the failure-case baseline.

Recovery load

Rebuilding the failed SSD moves data onto the replacement while the HDD primary still serves client reads. Backfill and recovery compete with that client I/O on the same HDD OSDs, so latency can stay elevated for the full recovery window. Ask how recovery priority and throttling settings apply in your release, and test them under your expected read load.

Monitoring signals

Track per-OSD latency for the SSD and HDD members of each hybrid rule, the count of placement groups in a degraded or peering state, and which OSD is reported as primary for hot pools. Alert when a primary moves from an SSD to an HDD, because that change is the start of the slower read period. Confirm which counters and health states your release exposes before you build alerts on them.

Hybrid rule design

Decide whether the SSD copy should be primary at all, and whether reads must stay on SSD during failures. Some teams keep a separate all-SSD pool for latency-sensitive data and use hybrid rules only where a slower failover read is acceptable. Write the decision and its expected failure-case latency into the design record before the pool is created.

When to avoid hybrid rules

Avoid hybrid rules when workloads have strict read latency targets, when SSD failures are likely to be frequent in your hardware pool, or when the HDD failover read cost is not acceptable to the application owner. In those cases, use device classes to isolate each media type into its own pool. Confirm the behavior for the release in use with the Ceph documentation and a lab test before committing.

Setting up device classes and class-specific pools

This sequence tags each OSD with the right device class, builds CRUSH rules that select only that class, binds pools to those rules, and confirms with test data that each pool lands on the intended media. The result is separate HDD, SSD and NVMe pools that can be managed and sized on their own.

Inventory drives and detected device classes
The cluster administrator lists every OSD with its host, size, media type and the device class Ceph detected, using ceph osd tree and ceph osd crush class ls. Compare each OSD against the hardware inventory, paying attention to drives behind RAID controllers or virtual disks, which can be misclassified. The step is complete when every OSD has a class that matches its physical media and the class list shows only the classes you intend to use. Confirm the exact output format with the release you are running, since command output changes between releases.
Set or correct device classes on OSDs
Where Ceph detected the wrong class, the administrator removes the existing class from the affected OSDs with ceph osd crush rm-device-class and assigns the correct one with ceph osd crush set-device-class. Make changes one host at a time and let placement groups settle between hosts, because reclassifying OSDs moves data. The step is complete when ceph osd tree shows the intended class on every OSD and no OSD is left without a class. Check the device class procedure in the documentation for your release before running it on production hardware.
Create class-specific CRUSH rules
Create one replicated rule per class with ceph osd crush rule create-replicated, naming the root, the failure domain (for example host or rack) and the device class, such as hdd, ssd or nvme. Choose the failure domain with the same reasoning used for the whole cluster, because a rule that spreads replicas across too few hosts cannot survive the loss of a host. The step is complete when ceph osd crush rule ls lists each rule and ceph osd crush rule dump shows the expected class and failure domain in its steps. Confirm the rule syntax against the target release.
Create pools bound to each rule
Create each pool with ceph osd pool create and assign its CRUSH rule with ceph osd pool set <pool> crush_rule <rule>, or set the rule at creation time. Set the replica count and minimum replicas to match the failure domains the rule can satisfy, and leave placement group settings to the autoscaler or to the sizing decisions made earlier in the design. The step is complete when ceph osd pool get <pool> crush_rule returns the intended rule for every pool and the pools report a healthy state with no undersized placement groups. Note that changing a pool's rule later triggers data movement.
Write test data to each pool
Write a known volume of test objects to each pool using rados bench or a test RBD image, then record the object count and the bytes stored. Keep the test small enough to finish during a maintenance window, and delete it after verification. The step is complete when each pool holds the test data and the write finishes without errors, which confirms the pool can accept I/O before placement is checked.
Verify placement per device class
Use ceph osd map <pool> <object> on test objects and ceph pg ls-by-pool <pool> to confirm that the acting OSDs for each pool belong only to the intended device class and that replicas sit in different failure domains. Check per-class usage with ceph df and the per-class view in the dashboard or ceph osd df. The step is complete when every sampled placement group maps to OSDs of the expected class, and no HDD OSD appears in an SSD or NVMe pool or the reverse.
Confirm health and hand over the configuration
Run ceph health detail and ceph -s after placement is verified, watch for backfill or misplacement warnings, and wait until all placement groups are active and clean. Record the device class list, CRUSH rules, pool-to-rule mapping and the commands used in the cluster runbook so later hardware additions follow the same pattern. The step is complete when the cluster reports HEALTH_OK, the runbook is filed, and the administrator has confirmed that new OSDs will receive a class during deployment.

Security and operational considerations for mixed-media clusters

A mixed-media cluster splits one physical pool of failure risk across several device classes, and each class fills, recovers and gets changed in its own way. The cards below cover the controls that keep a fast tier from quietly running out of room or being edited by accident.

Headroom per device class

Each device class has its own usable capacity, so a full HDD class can block writes to HDD pools even when SSD and NVMe classes have space. Plan the full ratio and the headroom for a failed host separately for every class, not as one cluster total. Ask your team to confirm, for each class, how many hosts or OSDs can fail before the remaining capacity in that class crosses the nearfull threshold. Keep this worksheet with the capacity plan so it is revised whenever a class grows.

Per-class usage monitoring

Cluster-wide usage graphs hide the class that is filling first. Build alerts on utilization per device class and per pool, covering nearfull, backfillfull and full states, and route them to the team that owns the pools on each class. Confirm that your monitoring stack reads the per-class figures from Ceph and that alerts fire before a class reaches the point where writes stop.

Recovery traffic on the fast tier

When an NVMe or SSD OSD fails, its data rebuilds onto the remaining devices of the same class, and that rebuild competes with client I/O on the fast pools. Decide in advance how much recovery bandwidth each class may use and who can change those limits during an incident. Test a single-OSD failure in a non-production copy of the cluster and record how long client latency stays elevated, then use that record in the change plan.

Change control for CRUSH map edits

A mistaken device class assignment or rule edit can move data across the whole cluster, so treat CRUSH map changes as production changes. Export and store the current map before each edit, test the new map against the expected placement, and require a second reviewer for any change that affects which devices a rule selects. Ask your team to confirm the rollback steps and who can apply them outside business hours.

Access controls on pools and rules

Creating a pool or a CRUSH rule can route sensitive data onto a tier that has different hardware or failure behavior. Restrict administrative rights for pool creation, rule edits and device class changes to a small named group, and log each action. Review who holds these rights on a fixed schedule and when staff change roles, using the access model your organization already applies to other storage platforms.

Hybrid rule failure review

Hybrid rules that place primaries on SSD and replicas on HDD depend on the fast copy being available for reads. When an SSD primary fails, reads of those objects fall back to slower devices until recovery completes, so the latency change is a planned risk, not a surprise. Document which applications accept that penalty, test the behavior with your own workload, and confirm the expected read path with your vendor or the Ceph documentation before you approve the rule for production use.

Questions buyers ask about mixing HDD, SSD and NVMe in Ceph

What drives the cost of a device-class design in Ceph?

The main cost driver is the scope of the design, not the number of drives. A class-based layout needs a plan for which pools land on which media, the CRUSH rules that route them, and the failure domain each rule uses. Cost rises with the number of distinct media types, the number of pools that need their own rule, whether hybrid rules with SSD primaries are used, and how much of the hardware must be benchmarked before purchase. Ongoing cost comes from monitoring capacity per class, since a full HDD class can block writes even when the SSD class has room. In a quote, ask for the pool-to-class mapping, the rules to be written, the validation tests, and whether hardware benchmarking is included or priced separately.

How long does a class-based Ceph design take to set up and validate?

Duration depends on how many pools, media types and failure domains the design covers, and on how much of the hardware is already in place. The work usually runs in order: confirm the media inventory and device class assignments, write and test the CRUSH rules, create pools bound to those rules, then validate placement, recovery behavior and capacity per class. Validation takes longer when hybrid rules are tested under a simulated primary SSD failure, because the read behavior after failure needs to be measured, not assumed. Ask the partner for a written schedule that lists each stage and the acceptance tests, and confirm timing against your own change windows and hardware delivery dates.

What does the work need from our team?

Your team needs to provide accurate hardware inventory, including drive models, sizes and rotational type per host, and the network layout that connects the hosts. You also need to name the workloads that will use each pool, their latency and throughput expectations, and the data protection target, such as replica count or erasure coding profile. Someone must approve changes to production CRUSH maps and own the decision about which pools may use SSD primaries. Access to a lab or staging cluster for testing rules before production is strongly recommended. Confirm these responsibilities in the scope so that gaps appear before work starts, not during validation.

Can an internal team design and run mixed-media Ceph alone?

An internal team with working Ceph experience can build this design, provided it understands CRUSH, device classes, placement groups and how recovery traffic affects client I/O. The harder part is validation: testing rules under failure, reading the effect of a failed SSD primary on reads, and sizing each class for headroom. Teams with limited Ceph operations experience often benefit from an external review of the rules and test plan before go-live, then run the cluster themselves. Whether your team can do it alone depends on the skills available, the staffing for on-call work and how much time you can spend on testing. Ceph's hardware recommendations also advise benchmarking candidate hardware, which many teams underestimate.

When should we benchmark hardware before buying mixed media?

Benchmark before buying when the workload depends on latency or throughput that differs between media, when the drive or network mix is new to your team, or when the budget for SSD primaries depends on measured gains. Ceph's hardware recommendations say to benchmark before buying because no two clusters are alike. A benchmark should use drives, hosts and networks close to the production candidate, run the workloads that matter, and record results per device class. Without that data, a hybrid rule may look good on paper and disappoint under real read patterns or during recovery. Ask for the benchmark method and the test report as part of any quote.

When is it better to build separate Ceph clusters instead of device classes?

Separate clusters make sense when workloads need different failure domains, different upgrade schedules, different security boundaries or different operators. A single cluster with device classes shares its monitors, network and recovery load across all pools, so a heavy rebuild on the HDD class can affect the SSD class. If one tenant or application cannot tolerate that shared risk, or if compliance scope requires isolation, a second cluster is often the cleaner choice. Device classes remain the better fit when the workloads share an operations team, the network is shared, and recovery impact can be managed through rules and scheduling. Decide this before purchasing hardware, because changing the boundary later means migrating data.

How DeployOpen scopes and delivers a mixed-media Ceph design

DeployOpen starts a mixed-media Ceph engagement with a written scope agreed before any work begins. That scope names the pools that will split by media, the client protocols they serve (RBD, CephFS or RGW), the failure domain each pool needs, and the hardware that already exists or is being purchased. Most engagements are fixed-scope, so deliverables and timelines are settled in writing first, and a quote follows directly from that document. DeployOpen does not publish list prices, so the scope is what the buyer compares across proposals.

The design phase reviews the workload and the hardware together. The team checks which OSDs are HDD, SATA or SAS SSD, and NVMe, and whether HDD OSDs use shared SSDs for WAL and DB as the hardware recommendations advise. It then designs the device classes, which Ceph assigns automatically but which can be overridden with ceph osd crush set-device-class, and writes class-specific CRUSH rules with ceph osd crush rule create-replicated. Hybrid rules that place SSD primaries are the step most likely to need a deliberate decision about read behavior. Next, a test cluster or lab run validates the rules, placement and recovery behavior before production hardware is loaded. Handover includes the rule map, the pool-to-rule assignments, a runbook for adding a new media class, and the checks the client team should repeat after each upgrade.

Ongoing support, upgrade planning and continuous testing are available as retainers, so the operating team does not have to learn every rule change alone. Confirm with DeployOpen during scoping which parts of the handover the buyer's own staff will run, which will stay under retainer, and how the test cluster will be kept or retired. Ask every proposal to list the deliverables, the environments used for validation, the acceptance checks and the assumptions about hardware, so that differences between quotes come from the work described rather than from assumptions left unstated.

Mixing HDD, SSD and NVMe in Ceph with device classes