When an all-flash NVMe Ceph cluster is worth designing
An all-flash NVMe Ceph cluster is worth designing when your block or database workloads need low, predictable latency and a mixed HDD or SSD pool cannot hold that latency under load. If your workloads are capacity-driven, such as archives or large backup targets, and tolerate variable response times, HDD or hybrid pools usually stay cheaper per terabyte. The decision turns on the latency profile your applications require at peak, not on the headline IOPS a vendor quotes. Start by writing down the p99 latency targets, the queue depths and the recovery window you can accept, then test whether a candidate design meets them. What changes with all-flash is mostly the layout of each host. The Ceph hardware recommendations advise a dedicated drive per OSD, and they advise enterprise-class SSDs for metadata. In an all-flash design, the separate DB device question largely disappears, because there is no slower HDD data device that needs a faster DB or WAL partner. You still need to decide whether to colocate DB and WAL on the NVMe data drive or split them onto a second NVMe device, and that choice should come from benchmarks on your hardware. Each OSD also becomes a heavier consumer of CPU and memory, so the number of drives per host directly drives the compute you must buy. The network and the labor follow from those choices. Flash moves data faster than a typical 10 GbE link can carry, so recovery and backfill traffic can saturate the network and hurt client I/O, which is why the hardware guidance recommends bonding links across two switches. Ceph is licensed under LGPL-2.1 or LGPL-3.0, and the project offers no paid edition, so your cost sits in hardware, the engineering time to design and benchmark the cluster, and any support contract you buy from a vendor or partner. Ask any partner to price the benchmarking phase, the drive and network bill of materials, and the ongoing support separately so you can see which costs recur.
Media and layout options for flash OSDs
This table compares the main media and layout choices for an all-flash Ceph cluster, including OSD density per drive, NVMe versus SATA SSD, dedicated metadata drives and separate DB devices. Read each row for when the option fits and what it costs you, then confirm endurance and power-loss protection with the drive vendor before you buy.
| Option | When it fits | Main trade-off | What to confirm |
|---|---|---|---|
| One dedicated drive per OSD | The default layout for flash OSDs, where each drive holds exactly one OSD and failure impact stays contained to that device. | Each OSD needs its own CPU and memory budget, so larger drives mean fewer, heavier daemons per host. | Drive count per host and memory per OSD at peak load, using the vendor's benchmark data. |
| Several OSDs per NVMe drive | Very fast drives where a single OSD may not saturate the device, and the team wants more parallelism per drive. | More OSD processes multiply CPU and memory demand and make per-host sizing tighter. Treat this as a tuning decision to benchmark, not a default. | Benchmark results on your own hardware before choosing this layout. |
| Enterprise NVMe SSDs | Low-latency block and file workloads where the cluster network and CPU can keep pace with fast drives. | Higher cost per terabyte than SATA SSDs, and the cluster needs enough network bandwidth and CPU to use the speed. | Endurance rating, sustained write performance and power-loss protection from the drive vendor. |
| Enterprise SATA SSDs | Capacity-focused flash tiers or budgets where lower latency is not the main requirement. | Lower throughput and higher latency per OSD than NVMe, so the same host may be limited by the drive before the CPU. | Endurance rating, power-loss protection and whether the drive model is listed for enterprise write patterns. |
| Dedicated metadata drives | Where the hardware recommendations call for enterprise-class SSDs for metadata, typically for RGW index or metadata pools placed on faster media. | Extra devices add failure domains and procurement items, and mixed pools need careful CRUSH rules. | Which pools need metadata media, and the device class rules that keep them there. |
| Separate DB device on flash OSDs | Mostly useful on HDD-based OSDs, where the hardware recommendations suggest shared SSDs for WAL and DB. On all-flash OSDs the benefit is smaller, so test before adding one. | A shared DB device becomes a shared failure point for every OSD it serves, and it adds sizing and monitoring work. | Whether your drive mix gains anything in benchmarks, and how many OSDs would depend on each DB device. |
How CPU and memory per OSD set latency
On flash, the bottleneck moves from the drive to the host. An NVMe device can complete I/O fast enough that the work around each request becomes the limit: messenger processing, BlueStore metadata and checksums, and the fan-out of every write to its replica peers. Each OSD is a process that consumes CPU for all of this, so the question for an all-flash design is whether the cores available to each NVMe OSD can keep the drive busy without queuing. Check this in the lab by watching CPU saturation on the OSD process while you raise queue depth. If the process saturates before the drive does, adding drives to the same host will not lower latency, and the fix is more cores per OSD or fewer OSDs per host.
The Ceph hardware recommendations advise sizing memory for peak use rather than calm periods. For OSDs, peak means the working set during recovery, backfill and deep scrub, not steady client traffic. A host that is sized for normal load can start swapping, shrink its caches or lose OSDs to memory pressure at the moment a neighbor fails, which is when latency is already highest. Confirm how the osd_memory_target setting interacts with the memory you reserve for the operating system, monitors and any co-located services, and make sure the sum still fits during a failure. Recovery and backfill add their own load on top of client I/O. Decide in advance how much recovery concurrency each OSD may take, using settings such as osd_max_backfills and osd_recovery_max_active, and test that the cluster still meets its latency target while rebuilding.
No two clusters are alike, so the final CPU and memory figures should come from benchmarks on the hardware you intend to buy, not from a general rule. Build a representative test with small random reads and writes at the queue depths your applications produce, then repeat it with a simulated host failure so recovery traffic runs alongside client I/O. Record latency percentiles, not only averages, and note which resource saturates first. Ask your server and drive vendors to support that testing with the exact CPU model, memory population and drive model you will order, because a faster drive in the same chassis can move the bottleneck. The Ceph hardware recommendations are the baseline for these checks, and the results should decide the bill of materials.
Network choices that keep up with NVMe
NVMe drives can saturate a network long before the CPUs run out of headroom, so the network decisions for an all-flash cluster matter as much as the drives. The cards below cover the recommended link layout, whether to split traffic, the ports and MTU to confirm, and how to test throughput before purchase.
Bond links across two switches
The Ceph hardware recommendations advise bonding network links across two switches, so a single switch failure does not take a host out of the cluster. For NVMe hosts, the bond has to carry the full aggregate throughput of every drive in the node during recovery, not just the client load. Confirm that each host has two physical paths to separate switches and that the bond mode matches what your switches support. Ask your network team to document both paths before you order hardware.
Decide whether to split networks
A public network carries client traffic and a cluster network carries replication, recovery and backfill between OSDs. Splitting them keeps rebuild traffic off the path that serves your applications, which matters when a failed NVMe drive triggers heavy recovery. On a fast fabric, a single bonded network with enough bandwidth may be simpler to run. Write down the recovery traffic you expect during a host failure, then decide based on whether client latency stays acceptable during that event.
Open msgr2 ports 3300 and 6789
Monitors listen for the msgr2 protocol on port 3300, and the legacy v1 protocol on port 6789 remains in use for older clients. OSD, manager and metadata traffic use a port range that your firewall rules must also allow between cluster members. Confirm which ports each host and client subnet needs, and whether your policy permits the secure mode of msgr2 that encrypts post-authentication traffic. Test the rules from every failure domain, not only from the bootstrap host.
Confirm MTU and switch settings
Jumbo frames can reduce per-packet overhead on high-throughput links, but only when every NIC, bond member, switch port and VLAN in the path uses the same MTU. A single mismatch causes intermittent large-packet loss that looks like slow storage. Confirm flow control, LACP hashing, and any storm control or QoS policies on the switches. Ask your network team to run a path MTU test between every pair of hosts and record the result as part of the handover.
Benchmark the network before buying
The Ceph hardware recommendations say to benchmark before buying because no two clusters are alike. Measure raw throughput and latency between candidate hosts on the exact NICs, switches and cables you plan to use, then compare the result with the aggregate write and read bandwidth of the drives in each node. A network that is slower than the drives will cap the cluster regardless of drive quality. Ask the supplier or your integrator for test results on your hardware rather than vendor peak figures.
Do separate DB devices still help on all-flash?
A separate DB device helps less on all-flash than on HDD, and it can hurt if the device is slower than the data drive. The Ceph hardware recommendations advise moving WAL and DB to shared SSDs for HDD-based OSDs. That advice rests on a speed gap: the metadata and journal writes move to media far faster than the spinning disk underneath. When every data drive is already NVMe or enterprise SSD, that gap is gone or reversed, so the default carries over only if a benchmark shows a gain. Treat the split as a hypothesis to test, not a rule to copy from HDD designs.
The question to test is whether the DB and WAL device is faster than the data drive for your workload, and whether it adds a shared failure point. Build two candidate OSD layouts on the same hardware, one with co-located DB and WAL and one with a separate NVMe device, then compare small random write latency, client IOPS at your queue depth, and behavior during recovery. Ask your hardware vendor and Ceph engineers to confirm which benchmark tool and profile match your block, file or object workload. Record the results before you buy drives, because adding a dedicated device later changes the OSD layout.
Spillover is the risk to verify. When the DB fills, BlueStore places metadata on the slow device, which on flash can erase any gain and make latency less predictable. Check spillover warnings on the candidate layout under sustained load, and confirm how your release reports them. For how large the DB device should be and how many OSDs may share one, see the sibling topic on sizing BlueStore DB and WAL devices, which covers the sizing rule for HDD OSDs and why it differs here.
How to validate an all-flash design before purchase
This sequence turns a general all-flash idea into a tested design, so the bill of materials reflects measured latency, CPU and network behavior rather than assumptions about what NVMe hardware will deliver.
Define workload and latency targets
Start by writing down the workloads the cluster must serve: block volumes for virtual machines, databases, object or file access, and the mix of reads and writes, block sizes and queue depths each one produces. Record the latency percentiles the application owners need and the throughput they expect at peak, not at average load. Owners of the applications, with the storage team, sign off the document. The step is done when every workload has a written I/O profile and a latency target that someone outside the storage team has agreed to.
Shortlist drives and servers
Select candidate enterprise NVMe drives and server platforms that can meet the profile, checking that each drive has power-loss protection and the endurance rating your write volume requires. Ask the hardware vendor for drive and server models that have been tested with Ceph or with comparable software-defined storage, and confirm the drive firmware and controller revisions in writing. Keep at least two candidates per role so the benchmark can compare them. The output is a shortlist with drive models, endurance ratings, slot counts per server and the controller and PCIe layout of each platform.
Check CPU and memory per OSD at peak
Work out how many OSDs each host will run and compare the CPU cores and memory available per OSD against the Ceph hardware recommendations, which advise sizing memory for peak use rather than calm periods. Include recovery and backfill load, since a failed host pushes its data onto the remaining hosts at the same time as client traffic continues. Add CPU for any erasure coding or RGW gateway work planned on the same hosts. The check is complete when the team can state the memory and cores each OSD gets during a failure event, and that figure is written next to the server candidates.
Design the network for flash throughput
Size the network links for the aggregate throughput the drives can produce, not for the throughput of a single drive, and confirm that the switches and NICs on the candidate platforms support the speed chosen. Follow the hardware recommendation to bond links across two switches, and decide whether client and replication traffic need separate networks. Record the expected bandwidth during recovery alongside steady-state bandwidth. The step is done when a network diagram shows each host's bonded links, switch pairs and public and cluster networks, and the team has checked those figures against the drive throughput in the shortlist.
Benchmark a test cluster with representative I/O
Build a small test cluster from the shortlisted hardware, using the same drive models, firmware, network and Ceph release planned for production. Run benchmarks that reproduce the I/O profile from the first step, including mixed read and write workloads and a recovery scenario where one host is taken out during load. Record average and tail latency, throughput, CPU utilization per OSD, memory use, and recovery time, for each candidate. The step is done when the results are stored alongside the test configuration, so they can be repeated before the purchase and compared with the production cluster later.
Decide DB device placement from measured results
Compare the benchmark results with and without separate DB or WAL devices on the candidate configuration. On all-flash OSDs, the decision depends on whether the measured metadata latency and CPU cost justify extra devices, and whether those devices would become a shared failure point. Note where the test shows a benefit and where it does not, and do not rely on the HDD guidance for this decision. The check is that the team can explain the chosen layout using its own benchmark records rather than general recommendations.
Size the bill of materials
Convert the validated design into a bill of materials that lists servers, drives per server, memory, CPU, network cards, switches, optics and cables, spares for drives and power supplies, and any support contracts the vendor requires. Include headroom for a failed host and for the full ratio planned in capacity planning. Ask each supplier for a written quote that references the tested drive models and firmware. The step is complete when every line of the bill of materials traces back to a measured result or a stated requirement, and the list can be reviewed by the team that will operate the cluster.
Security and operations to plan into the hardware design
Several security and operations decisions for an all-flash cluster are fixed by hardware choices made early. Drive encryption, network isolation, firmware handling and wear monitoring each depend on what you buy and how you place it, so they belong in the design review rather than after go-live.
Encryption at rest for NVMe drives
Decide whether NVMe OSDs are encrypted at rest, and where the keys live, before you order drives. Ceph's OSD encryption is set per deployment, so changing it later usually means rebuilding OSDs. Confirm with your drive vendor whether self-encrypting drives are in scope and whether they interact with the OSD encryption you choose. Ask how key material is escrowed and rotated, and who can recover an OSD after a host failure.
Msgr2 secure mode on the wire
Ceph's messenger v2 protocol listens on port 3300 and, in secure mode, encrypts all traffic after authentication with AES-GCM and adds integrity checks. Flash clusters push high volumes between clients, OSDs and monitors, so confirm that your CPUs and network path can carry the encryption overhead at your target IOPS. Benchmark with secure mode enabled, not only with the default settings.
Management network isolation for cephadm hosts
cephadm manages hosts over SSH, so the management path is a privileged route into every node. Plan a separate management network or VLAN, limit which administrator hosts can reach it, and keep it apart from the public and cluster networks that carry client and replication traffic. Check which ports the Dashboard and Management Gateway expose and restrict them to that network.
Firmware and drive replacement procedures
NVMe firmware updates and drive swaps touch live OSDs, so write the procedure before the first failure. Agree with your vendor on firmware baselines, how updates are staged across hosts, and how long a replacement drive may stay unused. Plan for the recovery traffic a replacement triggers and confirm that spare drives of the same class and capacity are on hand.
Drive wear and health monitoring
Flash drives wear with write volume, and enterprise NVMe models report wear and media errors through SMART data. Decide which health attributes you collect, how often, and at what thresholds you schedule replacement. Confirm with your drive vendor which endurance metrics their NVMe models expose and how they map to the warranty. Feed these signals into the same alerting you use for Ceph health warnings.
Questions buyers ask about all-flash Ceph design
What drives the cost of an all-flash Ceph cluster?
The largest drivers are the number of NVMe drives, the CPU and memory each OSD needs, and the network fabric that carries replication and recovery traffic. Drive endurance class, host count, failure-domain layout and the replica or erasure-code scheme change how much raw capacity you buy, so they change the bill of materials. One-time costs include hardware, the design and benchmark phase, deployment and initial testing. Recurring costs include support contracts for hardware and software, spare drives, power and cooling, and any retainer for upgrades and monitoring. Ask any partner to split the quote into one-time and recurring lines and to show which hardware assumptions each line depends on.
How long does an all-flash Ceph design and benchmark phase take?
The duration depends on how many workload profiles need testing, how much of your hardware is already on hand, and how quickly benchmark results can be reviewed with your team. Phases stretch when workload data is incomplete, when candidate hardware must be borrowed or shipped, or when the network needs changes before tests are meaningful. Ask for a written schedule that lists each milestone, the client inputs each one needs, and the decision points where you approve hardware or move on. No fixed duration applies to every environment, so treat any single number as a claim to verify against your scope.
What does the work need from our team?
Expect to supply a clear description of the workloads, including block, file or object use, expected I/O sizes, read and write mix, latency targets and growth over the planned life of the cluster. You will also need access to candidate hardware or a test environment, network switch details, existing authentication and monitoring tools, and a named person who can approve decisions. Security and change-control reviewers should join early if the cluster will hold regulated data. Missing workload data is the most common reason a benchmark produces results that do not guide purchasing.
Can our internal team design and build an all-flash Ceph cluster alone?
Yes, a capable team can do it, provided it has people who understand CRUSH, placement groups, BlueStore, networking and benchmarking, and it can test hardware before buying. The harder part is sustaining the cluster afterward, including upgrades, recovery planning and capacity growth. Assess whether you have engineers available for the design phase and for ongoing operations, and whether you can get a second person trained on the cluster. A partner can shorten the design work or provide a second review, but the operating model should stay with the team that will run the cluster.
Is a separate DB device worth buying for NVMe data drives?
For NVMe OSDs, a separate DB device usually adds little benefit because the data drive is already flash and the DB can share the same fast media. The Ceph hardware recommendations focus separate DB and WAL devices on HDD-based OSDs, and they recommend enterprise-class SSDs for metadata. Whether a separate device helps depends on your drive count per host, how much spare NVMe capacity you have, and whether benchmarks show a gain on your workload. Confirm the current guidance for your release and test with and without a separate device before you commit hardware.
What should we ask for in a fixed-scope quote?
Ask for a written scope that names the workloads in scope, the hardware assumptions, the Ceph release, the number of hosts and drives, and the networks the design covers. The quote should list each deliverable, such as a design document, benchmark report, deployment runbook and handover session, along with the client inputs each deliverable needs. Look for acceptance criteria, a change process for scope shifts, and what happens if benchmark results contradict the hardware assumptions. Ask which costs are one-time, which recur, and which items are hardware you buy directly.
What happens if benchmark results do not meet our latency targets?
Benchmark results can show that a candidate design needs more CPU per OSD, more memory, a faster network, different drive models or a changed failure-domain layout. A sound design phase records those findings, models the cost of each change and lets you decide whether to adjust targets or hardware. Ask how the partner will report results, which tests will be repeated after changes, and how a shortfall affects the schedule and the quote. Your own latency requirements should be written down before testing, so results are judged against agreed criteria.
How DeployOpen scopes and delivers this design
We start by agreeing the scope of the hardware design in writing before any sizing work begins. The scope names the workloads the cluster must serve (block volumes for virtual machines, file, or S3-compatible object access through the Ceph Object Gateway), the latency targets that matter to your applications, the failure domains you need to survive, and the networks and sites in scope. Most engagements are fixed-scope, so the deliverables and timelines are settled up front, and a quote follows from that written scope. Ask any partner, including us, to state in writing which hardware decisions are included and which are assumed from your existing standards.
Our review runs in your environment, not on a generic reference build. We work with your team to collect the real I/O profile, then test candidate drives, CPU, memory per OSD and network links against it, because the Ceph hardware recommendations say to benchmark before buying since no two clusters are alike. We check that each enterprise NVMe drive sits in its own OSD, that the networks keep up with flash throughput, and that the decision on separate DB devices is justified by measured results rather than habit. The output is a bill of materials and a decision record that explains each choice, the alternatives considered, the evidence behind it, and the conditions that would trigger a revisit. Your auditors and future operators can then trace why the cluster looks the way it does.
Hardware decisions change as workloads grow, so we also offer retainers for ongoing support after handover, covering health reviews, capacity forecasts and re-benchmarking when you add new media or hosts. DeployOpen prepares designs, records and evidence for your compliance assessments, but we do not issue audit reports or certifications. Independent auditors and certification bodies make those determinations, and we will help you give them the material they need.
