Why benchmarking a Ceph cluster before production matters
Benchmark the hardware and the pools before a new or expanded Ceph cluster takes production traffic, and record the results so every later change can be compared against them. A cluster that must meet workload targets cannot be sized on datasheet numbers alone. The Ceph hardware recommendations advise benchmarking before buying because no two clusters are alike: drive firmware, HBA and controller behavior, network bonding across two switches, CPU and memory per OSD, and the workload itself all change what the cluster delivers. Benchmarking is a measurement practice, not a single setting, so it has to cover the drives, the network, each OSD and the pools that will serve real clients.
Teams need this work at four points. The first is a new build, before the hardware is ordered or accepted, when drive and network baselines decide whether the design meets the target. The second is new media or a new drive model, because a different SSD or HDD changes write latency, WAL and DB placement results and recovery speed. The third is a new workload, such as moving virtual machine disks, S3 object traffic or small-file metadata onto the same pools, since block, object and file I/O stress different parts of the stack. The fourth is before and after each upgrade, so a regression in throughput or latency can be traced to the release rather than to guesswork. Ask your team or partner which of these points apply and who owns the baseline record.
DeployOpen does not publish benchmark figures. Any throughput, IOPS or latency numbers for your cluster come from tests run on your hardware, with your network and your workload profile, so ask for the test plan, the tools, the test duration and the recorded baseline as part of the written scope before work starts. Confirm with your hardware vendor which drive and controller settings they validated, and with your own operations team which service levels the results must meet. Keep the raw results, the cluster version and the configuration together, because those records are what let you judge a later change.
Which test fits which question
This table pairs each benchmark type with the question it answers and the point in the project where it fits. Read it as a sequence: establish drive and network baselines first, then test OSDs and pools, and finish with client-side tests that match how applications will connect. Each test answers a narrower question than the one before it, so a strong result on one layer does not prove the layers above it perform.
| Test | Question it answers | When it fits |
|---|---|---|
| Drive baseline with fio | Can each drive sustain the write and read patterns its role requires, with sync writes for journal and metadata devices? | Before cluster build, on every drive model and firmware you plan to buy or have on hand |
| Network throughput and latency | Do the host links, bonds and switch paths deliver the bandwidth and round-trip time the design assumes under parallel streams? | After cabling and bonding, before any Ceph daemon is deployed |
| OSD bench | What can a single OSD sustain on its own device, which is the input for OSD capacity settings used by the mClock scheduler? | After OSDs are created, with the cluster idle and no client load on the same drives |
| rados bench on replicated pools | What throughput and latency does the cluster deliver at the RADOS layer for a replicated pool with the chosen replica count? | After pools are created, to compare replica counts and failure domain choices before workloads arrive |
| rados bench on erasure-coded pools | What does the erasure-coded profile cost in CPU and latency for writes and degraded reads compared with replication? | Before choosing an erasure-coded profile for capacity-focused pools, using the same object sizes planned for production |
| RBD client-side test | What block performance do virtual machines or hosts see through the kernel or librbd client, including client cache and network effects? | Once an RBD pool and a representative client host exist, before onboarding hypervisors |
| RGW client-side test | How do S3 requests perform through the load balancer, gateway daemons and bucket layout that applications will use? | After gateways and load balancing are in place, using object sizes and concurrency from the application profile |
Running the baseline sequence on a new cluster
This sequence measures what the drives, network and OSDs can deliver before any client data lands on the cluster, so later tuning and capacity decisions have a recorded starting point. Each step builds on the one before it, and the final record is what you compare against after every change.
Freeze the test scope and workload profile
Before any test runs, the architect and the workload owners agree in writing on which hosts, pools and drive classes are in scope, and on the workload profile each pool must serve: block I/O for virtual disks, file metadata and throughput for shared file systems, or object sizes and request mix for S3 traffic. Record the read and write ratio, block sizes, concurrency and the latency target that matters to the application. The step is done when the owners sign off on the document and no one changes the scope mid-run. Confirm with your application owners which numbers they actually need, since a generic benchmark does not answer that question.
Confirm the cluster is healthy and isolated
Check that the cluster reports a healthy state, with all monitors in quorum, all expected OSDs up and in, and no outstanding scrub errors or recovery activity, before starting any measurement. Confirm that the test pools hold no production data and that the test clients are not shared with live workloads, because background load would invalidate the results. Use the cluster health output and the OSD tree as the evidence, and save both to the record. The step is complete when the health state and the isolation of the test environment are written down. Do not run benchmarks on a cluster that is still rebalancing or recovering from a failed drive.
Baseline each drive outside Ceph
Test every drive model on its own, before it joins Ceph, using a raw device benchmark that runs sequential and random reads and writes at the block sizes the workload uses. Keep the test read only where the device allows it, or use a disposable device, so no data is at risk. Compare results across drives of the same model, because a single slow unit often shows up here before it causes trouble in the cluster. The step produces a per-device table with throughput, IOPS and latency percentiles. Treat any device that falls well outside its peers as a hardware question to resolve before you continue. Check that write caching and power-loss protection settings match the vendor guidance for your model.
Test network paths and bond behavior
Measure throughput and latency host to host across every network path the cluster uses, including the public and cluster networks if you run them separately. Then pull one link of each bonded pair, and then one switch, and repeat the test to confirm that traffic fails over and that the throughput loss matches what the bond should deliver. Check MTU end to end, since a mismatch often appears only under load. The step is done when you have a path matrix and a failover result for each bond. A path that does not fail over cleanly should be fixed before OSD benchmarks begin, because it will skew every later number.
Run OSD capacity benchmarks
Run the OSD bench test on each OSD individually, without client load, to measure what each daemon can write at its configured block size. Compare OSDs on the same host and across hosts, and flag any that fall outside the group. Use the results to set realistic expectations for the cluster's aggregate throughput and to feed the mClock OSD capacity values later, when you tune recovery and client traffic. The step produces a per-OSD list with the measured values and the host each one sits on. It is complete when every OSD has a result and any outliers have been investigated or explained. Confirm the exact command and the output fields with the documentation for your Ceph release.
Run pool tests that match each workload
Create a test pool with the same replication or erasure coding profile, failure domain and placement group count as the production pool will use, then run a load generator that matches the profile recorded in step one. For block workloads, drive RBD images with a block benchmark tool. For file workloads, use a client that exercises metadata and data paths. For object workloads, send S3 requests with the object sizes and concurrency you expect. Run each test long enough for the numbers to settle, and repeat each one at least once to check consistency. The output is throughput, IOPS and latency at the percentiles the application cares about. Delete the test pool once results are recorded, so that no test data remains.
Record results with versions and configuration
Write each result into a single baseline record that includes the Ceph release and exact version, the cephadm-managed container image, host and drive models, memory and CPU per host, network configuration and bond modes, pool settings, placement group counts, and any non-default configuration values in effect during the tests. Attach the raw output of each benchmark, not only summary charts, so the comparison can be repeated later. Store the record where the operations team can find it and version it with the cluster's configuration. The step is complete when another engineer can read the record and reproduce each test without asking you a question.
Rerun the baseline after every change
Whenever the cluster changes in a way that affects performance, rerun the relevant tests from the same scope and profile, and compare the results against the recorded baseline. Changes that should trigger a rerun include a Ceph version upgrade, a drive or memory replacement, a network change, a new pool or a change to osd_memory_target, scheduler profiles or compression settings. Record the new results alongside the old ones and state which difference is due to the change. The step is done when each comparison is written up and any regression has a named owner. Reruns are how you learn whether a tuning change helped, so do not skip them to save time.
Matching pool tests to the workload you will run
A benchmark only predicts production if it resembles production. Each workload shape below changes the block size, queue depth, concurrency and redundancy scheme you should test, so the results can be compared with what applications will actually do. Confirm each profile with the application owner before running anything.
Small random writes for virtual machine disks
Virtual machine disks generate many small, random writes at moderate queue depth, often from several guests at once. Test 4 KiB and 8 KiB block sizes with varied queue depth and numjobs, on a replicated pool that matches the planned layout. Ask the application owner how many VMs share a pool, the read to write ratio and whether bursts happen at backup or patch windows.
Large sequential writes for backup targets
Backup jobs stream large sequential writes and care most about sustained throughput over a long run, not peak IOPS. Vary the block size toward 1 MiB or larger and run long enough to pass any write cache effects. Ask the backup team for the nightly window, the number of concurrent streams and the retention pattern, since deletions and expiry create their own load.
Metadata-heavy file access
CephFS workloads that create, stat and list many small files stress the metadata servers and the metadata pool more than the data pool. Test file creation and directory listing at realistic file counts, and measure metadata pool latency alongside client results. Confirm with the file owners the typical directory depth, file counts per directory and whether they run parallel build or scan jobs.
Many small RGW objects
Object workloads with millions of small objects load the bucket index and the RGW gateways, not only the OSDs. Run S3 tests with object sizes and concurrency that match the application, and include list and delete operations. Ask the application team the object size distribution, expected bucket counts and whether clients use multipart upload for larger files.
Mixed read and write
Most production pools serve reads and writes at the same time, and the ratio changes by hour and by application. Run a mixed profile with a fixed read to write split, then repeat it at more than one ratio to find where latency starts to climb. Confirm the expected ratio with the application owner and whether any reads are cache-friendly, since that changes what the test should count as a hit.
Recovery under load
A failed drive turns a quiet cluster into one that rebuilds while serving clients. Run the client workload while you deliberately mark an OSD out or drain a host in a test window, and record how client latency and recovery rate move together. Confirm with the operations team how much client slowdown the business accepts and which recovery settings they are prepared to change.
Recording results so changes can be compared
A benchmark only helps later if the record around it is complete enough to reproduce. Each run should capture the Ceph release and build, the kernel version, drive models and firmware revisions, NIC driver and firmware, the bonded network configuration and MTU, and the switch settings that affect it. Record the pool settings (replicated or erasure coded, size and min_size, the CRUSH rule and failure domain), the placement group count per pool, and the osd_memory_target value on every OSD host. Include the exact test commands with every parameter, such as rados bench or fio arguments, block size, queue depth, runtime, and the object or image names used. Store the output with the cluster state at the time of the run: ceph -s, ceph health detail, any recovery or backfill activity, scrub state, and the mClock profile in use. Without that state, a slow result cannot be told apart from a cluster that was rebalancing during the test.
Repeat runs under identical conditions are what make a comparison meaningful. Drives and networks vary between runs, caches warm up, and background work such as scrubbing or balancing can add noise. Run each test several times on a quiescent cluster, keep the number of runs and the spread of results in the record, and confirm the cluster returns to HEALTH_OK between runs. Changing one variable at a time, then re-running the same baseline, shows whether a tuning change helped. Ask your own team or partner to confirm how many repetitions and what spread they consider acceptable for your hardware, because that depends on the environment.
The changes themselves belong to the tuning pages in this group, including OSD memory, the mClock scheduler, compression and data balance. This section only covers how to record the before and after state so those changes can be judged. Keep the baseline record unchanged after tuning, and create a new record for each change, stored alongside the configuration that produced it. Confirm with the vendor or your own cephadm configuration history which settings are stored in the cluster configuration database, so the record can be rebuilt if the environment is lost.
Security and operational considerations during testing
Benchmarks write real data, open real connections and consume real capacity, so the test environment needs the same controls as production. The four cards below cover data isolation, client credentials, network placement and cleanup, which are the points most often skipped when a team is in a hurry to get numbers.
Isolate test pools from production data
Create benchmark pools that are separate from any pool holding business data, and give them their own names so that no script or dashboard confuses the two. A write benchmark can fill a pool, trigger recovery and change placement, so a shared pool can damage live data or skew production metrics. Before each run, confirm the target pool name and its CRUSH rule, and record both in the results so that a later reader knows exactly what was measured.
Limit test client credentials
Create a dedicated Ceph client identity for benchmarking with capabilities restricted to the test pools, instead of reusing the admin keyring or an application key. Ceph's authentication model grants access through per-client capabilities, so a narrow key limits what a mistyped command or leaked file can reach. Confirm which keyring the benchmark tools read, remove the key when testing ends, and keep it out of shared scripts and shell history.
Keep test traffic off production networks
Run load generators on the cluster or storage network that production will use, but keep the heavy test traffic from crossing links that carry client or replication traffic for live workloads. Benchmarks can saturate bonded links and starve monitors or heartbeats, which can cause false failure events. Agree a test window with application owners, and check that monitoring alerts are either paused or labeled so the test does not page the on-call team.
Clean up pools and images afterward
Delete benchmark pools, RBD images and test objects once the run is complete, and confirm the cluster returns to a healthy state before anyone reuses the hardware. Leftover data consumes capacity, keeps placement groups active and can confuse later capacity planning. Keep the raw output and the configuration used for each run, but remove the test data itself, and write the cleanup step into the test plan so it is not optional.
Questions teams ask before benchmarking Ceph
How much does a Ceph benchmarking engagement cost?
Cost follows from the written scope, and DeployOpen does not publish a price list. The main drivers are the number of nodes and drives to test, the number of pools and workload profiles, how many distinct hardware configurations need comparing, and how many repeat runs the results require. Part of the cost is one-time: the baseline and the first results report. Part may recur if you rerun tests after hardware or configuration changes. When you request a quote, ask for the list of test phases, the hardware and pools covered, the deliverables (raw results, a summary and a repeatable test procedure), and whether reruns are included or priced separately.
How long does benchmarking a Ceph cluster take?
Duration depends on how much hardware is in scope, how many pools and workload patterns must be tested, and how long each test must run to reach steady state. Drive and network baselines are quick compared with OSD capacity runs and pool-level tests that repeat across block sizes and read and write mixes. Scheduling also matters: you need a quiet test window with no production load, and the schedule depends on when that window is available. Ask for a phased timeline in the written scope that shows the baseline, the OSD and pool runs, analysis and handover, with the client's dependencies listed for each phase.
What does the client need to provide for a benchmarking engagement?
The work needs access to the hardware and the network that the cluster will use, a test window that stops production I/O on the nodes under test, and written workload profiles that describe the real access patterns: block sizes, read and write ratios, concurrency and expected growth. You also need named owners who can approve the test plan, accept the results and decide on configuration changes. Expect to provide change approval for the test window and a rollback plan if a test disturbs shared infrastructure. Confirm these items in the scope before work starts.
Can our own team run Ceph benchmarks without a partner?
Yes, if your team has the skills and time to do it. The work requires knowing how to isolate drives, OSDs and network links, how to run repeatable tests, and how to read results in the context of your workload. Many teams underestimate the discipline needed to record results consistently so that later comparisons mean something. A partner typically adds a documented method, a second set of eyes on the workload model and a clear report. Compare the effort of your team against the scope you would need to cover, and be explicit about who will own the results after handover.
How do benchmarks differ from production monitoring in Ceph?
Benchmarks are controlled, time-bounded tests that push known workloads through the cluster to measure what the hardware and configuration can deliver. Production monitoring observes live workloads continuously, without controlling them, and tells you what is happening now and when something drifts. You need both. Benchmarks give you a baseline and capacity figures to compare against, while monitoring shows whether production still behaves like that baseline. Ceph's own health and performance counters help with the second, but they do not replace a documented test that you can repeat after a change.
When should we rerun Ceph benchmarks?
Rerun benchmarks whenever something changes that would alter the results: a Ceph release upgrade, new drive or network hardware, a change in OSD memory or BlueStore cache settings, a change in pool layout or replication or erasure coding profile, a significant change in the workload, or an expansion that adds nodes. Also rerun after a performance complaint, to confirm whether the cause is the platform or the application. Keep the same test procedure and parameters so that the new results compare directly with the stored baseline. Ask in the written scope whether reruns are part of the agreed work or a separate item.
How DeployOpen scopes and delivers Ceph benchmarking
Benchmarking work for a Ceph cluster starts with a written, fixed scope agreed before any test runs. That scope names the hardware under test, the Ceph release, the networks, the pools to be measured and the results the client expects to receive. Because most engagements are fixed-scope, the deliverables and the acceptance conditions are settled up front, so the client knows what is measured, what is excluded and when the work is complete. Ask for the scope to list which drives, hosts and switch ports are covered, and whether the Ceph version is confirmed against the active releases before testing begins.
Before any load is generated, DeployOpen and the client agree on workload profiles. A profile describes the block sizes, read and write mix, queue depth, client count and whether the traffic is sequential or random, drawn from the applications that will actually run on the cluster. A database volume, a virtual machine disk pool and an S3 ingest pipeline stress the cluster differently, so each profile gets its own pool-level test. The client should confirm these profiles with the teams that own the workloads, since a benchmark that does not match production answers a question nobody asked.
All tests run inside the client's own infrastructure, using the client's hosts, networks and change windows. Drive and network baselines, OSD capacity measurements and pool tests are executed in that environment, and no client data leaves it. The output is a recorded baseline package: raw results, the exact commands and configuration used, hardware and firmware identifiers, and notes on anomalies. That package is handed over so the client's own team can repeat any test later. Optional retainer support covers reruns after changes such as a Ceph upgrade, a new drive model or a tuning change, so each comparison is made against the same recorded method.
