Ceph hardware sizing: why the bill of materials starts with workload numbers
If you have a capacity target and a performance target, you need hardware, and the path from one to the other is sizing. Sizing converts workload numbers (usable capacity, growth, client IOPS and throughput, latency expectations, and how many hosts can fail at once) into hosts, drives per host, memory, CPU and network. Start with those numbers in writing. A bill of materials built from a vendor's standard server SKU and then checked against the workload usually misses the constraints that matter most, such as recovery time after a host failure or the cost of a slow network under load.
Ceph's hardware recommendations give several rules that shape the bill of materials. Use a dedicated drive per OSD, so no drive hosts an OSD alongside the operating system or another OSD. Put metadata on enterprise-class SSDs. Bond network links across two switches so a single switch failure does not isolate a host. Size memory for peak use rather than calm periods, because recovery and backfill raise demand well above normal operation. The same guidance says to benchmark candidate hardware before buying, because no two clusters behave alike. Each of these rules changes the count of hosts, drives and ports, so they belong in the first draft of the bill of materials, not in a later review.
Confirm with your hardware vendor the drive, controller, memory and NIC models you intend to buy, and ask for their support and warranty terms in writing. Then plan a benchmark on representative hardware before the purchase order is final. DeployOpen does not publish hardware prices. Hardware cost, licensing for supporting software, and the implementation effort are quoted separately, and the quote follows the written scope that defines the workload, the hardware list and the acceptance tests. Ask any partner to show how each cost line traces back to a workload number.
Which hardware choices fit which workloads
This table compares common Ceph hardware options by workload fit, cost, recovery time and latency. Read each row as a starting point: the right choice depends on your capacity targets, latency needs and failure tolerance, and the final decision should follow benchmarking of candidate hardware.
| Option | When it fits | Trade-offs in cost, recovery and latency |
|---|---|---|
| HDD OSDs with SSD DB/WAL | Large capacity pools such as archives, backups and bulk object data where throughput needs are moderate | Lowest cost per terabyte, but slow recovery and higher latency under load; the shared SSD becomes a failure point to plan for and sizing of DB devices affects spillover |
| All-SSD pools | Block workloads such as virtual machine disks that need consistent latency at moderate capacity | Higher cost per terabyte than HDD; faster recovery and lower latency; drive count per host and network bandwidth must keep pace with flash |
| NVMe OSDs | Latency-sensitive databases, analytics and high-IOPS block workloads with spare CPU and network capacity | Highest drive cost and the most demand on CPU, memory and networking per OSD; fastest recovery when hardware is balanced; confirm drive endurance with the vendor |
| Replicated pools | Workloads where write latency and fast recovery matter most and where capacity overhead is acceptable | Higher raw capacity overhead; simpler recovery and lower CPU use per write; the replica count sets how many hosts must be available |
| Erasure-coded pools | Capacity-driven data such as backups and object archives that can tolerate slower writes | Lower raw overhead per usable terabyte; more CPU for encoding and more network traffic during recovery; needs enough failure domains for the chosen profile |
| RGW-heavy hosts | Object gateways serving many concurrent S3 requests, with TLS, authentication and compression enabled | Adds CPU and memory per gateway and network throughput at the edge; scale gateways separately from OSD hosts so client load does not slow recovery |
| Dedicated monitor hosts | Large or multi-site clusters where monitor and manager load must stay isolated from OSD and gateway activity | Extra hosts and their operating cost; fewer shared resources to contend with; monitors still need odd counts and separate failure domains |
| Shared monitor and OSD hosts | Small clusters where hardware budget is tight and host count is limited | Lower hardware count and cost; monitor disk and memory compete with OSD workloads during recovery, so reserve headroom and monitor disk space |
From workload numbers to a benchmarked bill of materials
This sequence turns capacity, performance and resilience targets into a Ceph hardware specification that has been checked against failure scenarios and benchmarked on representative equipment before purchase.
Gather capacity, performance and latency targets
The workload owner and the platform team record the usable capacity needed in 12 to 36 months of growth, the client IOPS and throughput for each pool or service, the read and write mix, object sizes, and the latency the applications can tolerate. Block, file and RGW workloads each carry different profiles, so record them separately. The step is complete when every target is written down with an owner and a source, such as an existing system's metrics or a vendor-supplied estimate, and each figure is marked as measured or assumed. Ask the workload owners to confirm peak periods, because sizing for an average hour leaves the cluster short during the busiest one.
Choose the failure domain and data protection profile
Decide whether the failure domain is host, rack, row or room, and whether data is protected by replication or erasure coding. The choice sets the minimum number of hosts or racks you need before the first pool can be created, and it determines how much raw capacity turns into usable capacity. Record the replica count or erasure code profile for each pool, and note which pools need to survive a full room loss. The step is done when a written profile exists for every pool and the team agrees on the failure domain. Your Ceph design notes should match the profile before you order any hardware.
Derive OSD count and drive size
Divide the usable capacity target by the usable fraction the protection profile gives, then add headroom for the full ratios and for one failed host's data to rebuild elsewhere. Divide the resulting raw capacity by the drive size available from your supplier to get the drive count, then divide drives across hosts so that no host holds a disproportionate share. Use dedicated drives for each OSD, and note where BlueStore DB and WAL devices will sit if the OSDs are HDD-based. Check the result by confirming that usable capacity still meets the target after a single host is removed from the calculation. If it does not, add drives or hosts before continuing.
Size memory per OSD at peak
Multiply the number of OSDs per host by the memory target you configure for each OSD, then add memory for the operating system, monitors and managers if they share the host, and for any RGW or other gateway services. Size against peak use, such as recovery or backfill after a failure, rather than the memory used during quiet periods, because the Ceph hardware recommendations advise sizing for peak use. The output is a per-host memory figure with a stated margin. Confirm it in a test by simulating recovery on a representative host and recording the memory used at the busiest point. Do not add memory to the host until this check passes.
Add CPU for erasure coding and RGW
Estimate the CPU cores each OSD needs for its share of client and recovery work, then add cores for erasure coding if your profile uses it, and for each RGW instance that will serve S3 or Swift requests. Erasure coding and recovery are CPU-intensive, and gateway traffic adds request processing on top of data movement, so the two workloads should be sized separately and then combined per host. Compare the result with the processor options your supplier offers and note the clock speed, core count and socket count for each. The check is a benchmark run that includes erasure coding and an RGW load test on the candidate CPU, with CPU utilization recorded under the peak load you set in step one.
Plan bonded networks for client and replication traffic
Calculate the aggregate throughput needed for client traffic and for replication and recovery traffic at the peak targets, then decide whether to split public and cluster networks. Specify bonded links spread across two switches, which the hardware recommendations advise, and confirm that the switch fabric, cabling and host network cards support the chosen speed. Record the bond mode, the VLAN or subnet layout and the msgr2 ports that must be open between hosts and clients. The check is a link test on the bonded interface that confirms both links carry traffic and that the bond survives the loss of one switch. Ask the network team to confirm the switch model and uplink capacity in writing.
Benchmark candidate hardware before buying
Build a small test cluster or a representative subset of hosts using the candidate drives, memory, CPU and network cards, and run a benchmark that reflects your workload profile, including recovery and a host failure. The hardware recommendations say to benchmark before buying because no two clusters are alike. Record IOPS, throughput and latency at the targets from step one, along with CPU and memory use. The step is complete when the candidate configuration meets each target in the test, and when the team has noted any gap that needs a different drive class, network speed or CPU model. Confirm with the supplier whether a loan or evaluation unit is available before committing to a full order.
Finalize the bill of materials
Convert the validated configuration into a bill of materials that lists each host by role, with CPU model, memory modules, drive models and count, boot devices, DB or WAL devices, network cards, optics or cables, switches and spares. Include spare drives for the expected failure rate your supplier can confirm, and add a margin for growth that the capacity plan already accounts for. Ask the supplier to confirm lead times and warranty terms in writing, and ask whether they will sign off the bill of materials against the benchmark results. The sequence is complete when the bill of materials references the benchmark run and the failure-domain check, so that any later substitution must pass the same tests.
Security and operations considerations for the hardware layer
The hardware you buy for Ceph sets its security posture and its operating load for years. The cards below cover the drive, management, encryption, network and spares decisions that connect to CPU, memory and network sizing, so they can be settled in the same bill of materials rather than after installation.
Enterprise drives and power-loss behavior
Ceph's hardware recommendations call for enterprise-class SSDs for metadata, and the same reasoning applies to the drives that hold WAL and DB devices. Consumer drives may acknowledge writes before they are safely on flash, so a power event can leave OSD metadata inconsistent across hosts. Ask each vendor for power-loss protection specifications, and confirm the drive model against your Ceph release's supported hardware list before ordering.
Out-of-band management and firmware patching
Each Ceph host needs a baseboard management controller that you can reach when the operating system is down, and that network must sit apart from the Ceph public and cluster networks. Firmware for the HBA, NICs, drives and BMC is part of the same patch cycle as the OS. Plan the firmware baseline per hardware model, and test it on one host before rolling it across a cluster that is already holding data.
Encrypted OSDs and key management
Encrypted OSDs protect data at rest on drives that leave the building for repair or disposal, but the keys become a dependency. Ceph stores OSD encryption keys in the monitors' config-key store when you use dm-crypt with cephadm, so the monitors' disks and access controls matter as much as the drives. Confirm the key escrow approach with your security team, and test a node reboot and drive replacement on an encrypted OSD before you rely on it. Encryption also consumes CPU, so size the CPU headroom with it enabled.
Public and cluster network segmentation
Client traffic and replication traffic have different risks and different bandwidth profiles. During recovery, replication can saturate links that clients depend on, which is one reason to separate the public and cluster networks and to bond links across two switches. Segmenting these networks at the switch and VLAN level limits what a compromised client host can reach. Confirm which ports each network must allow before you write firewall rules for the cluster.
msgr2 secure mode and its CPU cost
The messenger v2 protocol listens on port 3300, and its secure mode encrypts all traffic after authentication with AES-GCM and adds integrity checks. That encryption runs on the hosts' CPUs, so its overhead lands on the same cores that serve OSDs, RGW and erasure coding. Benchmark a representative workload with secure mode on and off on candidate hardware, and confirm the throughput and latency figures with your Ceph release and your CPU vendor before you set the CPU budget.
Spares and drive replacement planning
Drive failures are routine in a large Ceph cluster, and a missing spare turns a routine repair into a delayed one while the cluster runs degraded. Decide how many spare drives each site holds, which models they match, and who can swap a drive at each location. Record the replacement procedure in your runbook, and confirm that replacement drives match the device classes and sizes your OSD specifications expect. Ask your vendor for lead times, because they set how many spares you should keep on hand.
Benchmarking candidate hardware before the purchase order
The Ceph hardware recommendations tell buyers to benchmark candidate hardware before purchasing because no two clusters are alike. Drive firmware, HBA behavior, CPU generation, NIC driver and switch configuration all change how Ceph performs under your load, so a vendor's published figures or a generic reference design cannot stand in for a test on the hardware you intend to buy. The test is also the cheapest point at which to discover that a chosen drive or network card does not suit the workload.
A useful pre-purchase test uses a representative drive, not a single sample from a catalog. Run it on a test host with the intended network: the same NIC model, bonding mode and switch pair you plan to order, with the public and cluster networks configured as they will be in production. Use a workload mix close to your production profile, including the read and write ratio, block size and client count you expect. Most importantly, record recovery behavior: time to recover after a failed OSD or host, the client latency during backfill, and how far recovery throttles under load. Peak throughput alone hides the failure case that drives your capacity headroom.
Ask the vendor or partner to deliver the test results in writing, with the exact hardware, firmware, kernel, Ceph release and configuration used, so the numbers can be reproduced. Ask for latency percentiles at the workload you care about, the recovery timeline for a drive and a host loss, and any tuning changes made to get the result. Confirm who pays for the test hardware and whether the same configuration is what will be quoted. Where the test shows a gap, the results should feed back into the bill of materials and the recovery planning for the cluster.
Questions teams ask before buying Ceph hardware
What does Ceph hardware sizing cost, and what drives the price up?
Sizing cost follows the written scope, not a fixed rate. The main drivers are the number of workloads to model, how many media classes and pools are involved, whether erasure coding or RGW gateways are in the design, how many candidate server configurations need benchmarking, and whether the work covers only a bill of materials or also procurement support. Hardware purchase, rack space, power, and support contracts are separate costs paid to vendors. Ask any partner for a quote that lists each deliverable, the hardware assumptions, and which items are one-time versus recurring, such as a later re-sizing when data growth changes. DeployOpen does not publish prices; a quote follows from the agreed written scope.
How long does a Ceph sizing exercise take, and what does the team need to provide?
Duration depends on how well the workload is already understood. A team with capacity history, client access patterns, and a clear growth forecast can move quickly, while a team starting from estimates will spend more time on data collection and benchmarking. The exercise needs a named technical owner, access to current storage and network metrics where they exist, input from application owners on growth and latency needs, and time with procurement to confirm lead times and approved vendors. Ask for the timeline in the written scope, with the points where the client must review or decide.
Can our own team size a Ceph cluster without a partner?
Yes, if the team has someone who can translate workload numbers into drive counts, memory, CPU, and network capacity, and who is willing to benchmark before buying. The Ceph hardware recommendations describe the core rules: a dedicated drive per OSD, shared SSDs for WAL and DB on HDD-based OSDs, enterprise-class SSDs for metadata, memory sized for peak use, bonded network links across two switches, and benchmarking because no two clusters are alike. The gap usually appears in peak-load modeling, failure-recovery headroom, and independent review of the bill of materials. A partner review is most useful when the purchase is large or the cluster must meet a regulated service level.
What workload data should we collect before sizing hardware?
Collect data that describes peak demand, not averages. Gather the expected usable capacity by pool, growth rate over the planned hardware life, read and write mix, object size distribution for S3 workloads, block device counts and I/O profile for RBD, and any erasure coding or replication intent. Add peak client throughput and IOPS, latency targets, and the expected recovery window after a host or drive failure. For RGW, record request rates and concurrency. Record which workloads share the cluster, since a noisy neighbor can change the required memory and network. Missing peak figures are the most common reason a bill of materials underperforms.
How do we read the hardware recommendations for our Ceph release?
Start from the release you plan to deploy. The Ceph releases page lists the actively maintained releases, and Tentacle and Squid are the two current entries at the time of writing, so confirm the status of each before you commit. Then read the hardware recommendations for the matching documentation version, because guidance on DB and WAL sizing and memory per OSD has changed between releases. Check each recommendation against your own media, failure domain, and network design. Ask your hardware vendor to confirm that the chosen drives, controllers, and NICs are supported with that release.
What does a fixed-scope Ceph hardware sizing engagement include?
A fixed-scope engagement usually covers the agreed workload inputs, a sizing model for usable capacity and performance, a bill of materials per host class, memory and CPU reasoning for each OSD and gateway role, network design with bonding, a benchmark plan for candidate hardware, and a written list of assumptions and open risks. DeployOpen plans and documents this work inside the client's infrastructure context. The exact deliverables, the number of candidate configurations, and any benchmarking performed on client hardware are fixed in the written scope before work starts. Anything outside that scope, such as procurement negotiation or later re-sizing, is quoted separately.
Limits to confirm in your environment before ordering
Hardware sizing numbers go stale quickly, so these checks should happen before a bill of materials is signed. Each one changes what you buy: the release sets memory and DB device rules, while the network, power and drive firmware decide whether the design you sized can actually be installed and supported.
Current Ceph release and end of life
Confirm whether you will deploy Tentacle (20.2.x) or Squid (19.2.x) and check the end-of-life date for that line against your planned service life. The Tentacle line was first released November 18, 2025, with an estimated end of life of June 1, 2027, and Squid's estimated end of life is October 31, 2026. A cluster bought for a five-year life on Squid will need an upgrade plan before it is retired, and that upgrade should be costed into the project.
Memory targets for the release in use
Ask the vendor and your Ceph reference to confirm the memory-per-OSD target for the release you chose, since the default OSD memory target and the peak-use guidance both affect the DIMM count per host. Size for recovery and backfill peaks, not idle nodes. Verify that the chosen memory configuration supports the target for every OSD on the host, including OSDs added later when you expand.
The 2.5 percent DB rule
Confirm whether the DB sizing rule for HDD OSDs applies to your chosen release before you buy SSDs for BlueStore DB and WAL. Older guidance sized DB devices differently, so a rule copied from an earlier design can leave you with too little or too much SSD. Ask for the DB size per OSD, the number of OSDs sharing one SSD, and the spillover behavior if the DB fills. Those answers set how many SSD slots each host needs.
Switch and NIC bonding support
Check that your NICs, switches and host OS support the bonding mode you plan to use across two switches, and that the switch pair supports the LACP or equivalent configuration. Confirm the link speed you need for recovery traffic, since a failed host returns its data over the cluster network. Ask the network team to verify port counts per host and the number of ports reserved for the public and cluster networks.
Rack power and cooling capacity
Confirm the power draw per node at full load, including drives, SSD and NIC cards, and check the circuit and cooling capacity of each rack against the planned node count. Add headroom for a node added later and for the drive count you may need for capacity growth. Power and thermal limits often decide how many drives fit in a rack, so ask the data center team for a written power budget before the bill of materials is final.
Drive firmware and controller support
Ask the drive vendor which firmware versions it certifies for your Ceph release and whether it supports the controller mode Ceph expects. Use enterprise-class drives for metadata and DB devices, and confirm that the controller passes drives through rather than presenting them as RAID volumes that hide SMART data. Request a written statement of firmware support and the process for updating firmware across a fleet.
What the sizing work changes for your purchase and rollout
A written sizing output changes the purchase in three ways: it reduces surprise upgrades after go-live, it gives a documented basis for drive and host counts, and it leaves a benchmark record that explains why the bill of materials looks the way it does. Without that record, procurement tends to buy against a vendor quote or a hardware list rather than against the workload, and the first capacity or memory shortfall then arrives as an unplanned order. The sizing plan should state which assumptions drive each number, so finance and operations can see what changes if the workload grows faster than expected.
The reader has to supply the inputs that the sizing depends on. That means workload data (client I/O mix, read and write ratios, object sizes, expected growth, and recovery expectations), rack constraints (power, cooling, rack units, and floor space per host), network limits (switch ports, optics and cabling already in place), and a budget ceiling. Ask for the Ceph recommendations and the vendor's drive, memory and NIC specifications to be cross-checked against these inputs, and confirm which Ceph release the cluster will run, because supported configurations and guidance change across releases. Missing inputs should be recorded as open assumptions, not filled with guesses.
DeployOpen prepares the sizing and benchmark plan under a written scope, so the deliverables, the candidate hardware to test and the acceptance criteria are agreed before work starts. The plan defines the test runs to execute on candidate hardware, the metrics to capture, and how results map back to drive counts, memory per OSD at peak and network capacity. Benchmarking remains the final check before buying, because the Ceph hardware recommendations note that no two clusters are alike. Ask for a quote that lists the sizing review, the test environment, the benchmark runs and the written report as separate line items, with any hardware loan or lab access called out.
How DeployOpen scopes Ceph sizing and delivers the bill of materials
Hardware sizing for Ceph is only as good as the workload numbers behind it. DeployOpen fixes the scope in writing, collects the inputs, tests candidate hardware where it is practical, and hands over a bill of materials with its assumptions written down so your team can challenge them.
Written sizing scope
The scope names the workloads (RBD, CephFS, RGW), the protection scheme, the target usable capacity, the failure tolerance you need, and the sites in play. It also states what is excluded, such as network switch selection or facility power. Because most engagements are fixed-scope, the scope, deliverables and timeline are agreed before any sizing work begins, which makes it clear what your own team must provide.
Workload data collection
Sizing starts from measured demand: current capacity growth, peak client IOPS and throughput, object counts, average object size, read and write ratios, and recovery windows you can tolerate. Where no history exists, DeployOpen works with your application owners to define those numbers and records each one as an assumption. Missing workload data is the most common reason a bill of materials has to be reworked later.
Memory sized for peak
Ceph's guidance is to size memory for peak use, not for calm periods, because recovery, backfill and scrubbing can raise per-OSD memory well above steady state. The plan sets memory per OSD using your chosen OSD memory target and adds headroom for monitors, managers, and gateway hosts. Confirm the memory target and the recovery behavior you expect with your own team before purchase.
Drives, DB devices and CPU
The bill of materials lists one dedicated drive per OSD, enterprise-class SSDs for metadata and BlueStore DB or WAL on HDD-based OSDs, and the number of OSDs each host can carry without starving CPU. CPU needs are set by erasure coding work, RGW request handling and recovery load, so the plan states the core count per OSD it assumes and why.
Bonded networks and failure headroom
The plan sizes the network from the traffic you expect during recovery, not only during normal operation. It documents bonded links spread across two switches, the public and cluster network split, and the spare capacity needed when one host or rack is down. Each assumption is written so that a network team can confirm switch ports and uplinks independently.
Benchmarks before purchase
Ceph's guidance is to benchmark before buying because no two clusters are alike. Where your budget and timeline allow, DeployOpen plans benchmark runs on candidate drives, hosts and networks, measuring the workload patterns from the data collection step. The results adjust the bill of materials, and the test method is recorded so the numbers can be repeated after delivery.
Bill of materials and assumptions
The deliverable is a bill of materials with an assumptions register: each row lists the component, the quantity, the sizing rule that produced it, and the input it depends on. A short list of risks, such as spillover on DB devices or a single switch failure, accompanies the table. Ask for this register in any quote so you can see which numbers would change if a workload assumption changes.
Support and retainers after deployment
Hardware choices affect operations for years, so the engagement can continue after the cluster is built. Ongoing support, upgrades and reviews of capacity against the original sizing are available as retainers. Ask whether a quote covers a re-sizing review at a set interval and what triggers one, such as a capacity threshold or a new workload.
