Why adding Ceph OSDs without hurting clients starts before the drives arrive
The safest way to add OSDs to a Ceph cluster without hurting clients is to decide the pace of data movement before any drive is racked. When a cluster nears its capacity limits, the fear is that expansion itself will slow writes and reads. That fear is justified, because adding OSDs changes where data belongs. CRUSH recalculates placement for placement groups, so some PGs must move to the new drives, and that movement is backfill. The work is not about adding capacity quickly. It is about pacing the backfill so client I/O keeps priority. Squid (19.2.x) and Tentacle (20.2.x) are the actively maintained release lines, and cephadm handles adding hosts and OSDs to a running cluster, so confirm which of these your cluster runs before you plan the change.
Before the hardware arrives, measure the current state. Record raw used capacity, the fullness of the most utilized OSDs, and the recovery and client load at normal peak hours. Check the nearfull, backfillfull and full ratios so you know how much headroom exists for data to move onto new drives while the cluster is still accepting writes. Confirm the failure domain layout that the new hosts will join, because a host added to the wrong CRUSH bucket can move far more data than intended. Also confirm the network path, since backfill traffic shares links with client traffic unless you have separated them.
Plan the maintenance window and the rollback as deliberately as the install. Decide which recovery settings you will adjust and who has authority to change them during the move, and write down the thresholds at which you pause backfill. Use a non-production cluster or a lab copy to rehearse the sequence if one exists. Ask your DeployOpen or internal team to confirm, for your release and hardware, which cephadm steps and recovery options apply, since behavior differs between releases and this page does not set defaults for your environment. Expansion planned this way, well before nearfull, is far less stressful than expansion forced by a full OSD.
Scale up with existing hosts or scale out with new hosts
Ceph places data across failure domains defined in the CRUSH map, usually at the host level. Adding OSDs to an existing host keeps the host count the same, so the failure domain layout does not change, but every new drive on that host adds weight to the same machine. If that host fails, more data depends on it. Adding a whole host adds a new failure domain, which spreads data more evenly and gives CRUSH more room to satisfy the replication or erasure coding rule without placing two copies on one machine.
- Host count sets the ceiling: With replicated pools at a size of three and a host failure domain, you need at least three failure domains with enough capacity to hold the copies.
- Weight imbalance: A host with many new OSDs receives proportionally more placement groups, which can make that host the most loaded in the cluster.
- Recovery scope: Losing one host with heavy expansion can trigger a large recovery event, so confirm recovery time and I/O impact in a test before you expand a single host.
- Erasure coded pools: Check the number of hosts against the chunk count of each profile, because an uneven host count can limit placement.
- Illustrative example: A cluster of five hosts that adds four drives to one host may place a disproportionate share of data on that host.
Before you choose, ask your team to map each pool's failure domain requirement against the planned host count.
Both expansion paths add load, but they load different resources. Adding drives to existing hosts raises the I/O, memory and network demand on those machines. Adding hosts brings new CPU, memory and network ports into the cluster, but it also adds backfill traffic between old and new hosts during rebalancing. The Ceph hardware recommendations advise sizing memory for peak use rather than calm periods, so plan for recovery, not only steady-state operation.
- OSD memory: Each OSD needs memory for its cache and metadata. Confirm the per-OSD memory target on your hosts before adding drives, since a host that was sized for its current drive count may swap or throttle after expansion.
- Network bonding: The hardware recommendations advise bonding network links across two switches. Check that new hosts join the same bonded design and that the switch ports have capacity for backfill.
- Backfill traffic: Data moves across the cluster network during rebalancing, so a new host can concentrate traffic on its uplinks for a long period.
- Monitor and manager placement: Adding hosts gives cephadm more places to schedule services, but monitor quorum depends on the count you configure, not on the host count alone.
- Verification: Measure CPU, memory and network utilization on existing hosts during a controlled backfill before you add the next batch.
Ask your hardware vendor and your Ceph engineers to confirm peak memory and uplink capacity for the planned end state, not only the day-one state.
The hardware recommendations advise a dedicated drive per OSD, so each new OSD should map to its own physical device. On HDD-based OSDs, the recommendations suggest moving the WAL and DB onto shared SSDs. This matters most when you add drives to existing hosts, because a new HDD may not have an SSD slot with free capacity or a matching DB layout. Hosts that were built with one DB device per group of HDDs need careful planning, since a new drive can push the DB device past its intended sizing.
- One OSD per drive: Avoid partitioning one drive into several OSDs unless your design explicitly calls for it and you have tested it.
- Shared DB capacity: Check how much free space the shared SSD has before you add HDD OSDs, and confirm the sizing with your Ceph team.
- Media mix: Mixing drive types in one host can create performance differences between OSDs, so keep device classes consistent where your pools depend on them.
- Metadata pools: The hardware recommendations advise enterprise-class SSDs for metadata, so do not place metadata on capacity drives in a new host to save slots.
- Illustrative example: A host with one shared SSD serving six HDD OSDs may run short of DB space after three more HDDs are added, which can lead to spillover later.
Before you add drives, confirm the device layout, DB sizing and device class for every new OSD in writing.
Neither path is correct in every case. The better choice depends on how many hosts you already run, how much headroom each host has, and whether your next growth step is measured in drives or in machines. Small clusters with few hosts often have the most to gain from adding a new host, since each host is a large share of total capacity. Larger clusters may prefer adding drives in batches to existing hosts that have spare slots and memory, provided failure domains stay balanced.
- Growth pattern: Decide whether you expect steady drive replacements, periodic batch expansions or a planned fleet refresh.
- Spare slots: Count free drive bays and check whether the chassis supports the planned drive model.
- Balance target: Set a rule for acceptable utilization differences between hosts, then check it against the balancer output after each step.
- Lead time: Ask your procurement team how long hardware takes to arrive, and plan the expansion before the cluster reaches nearfull.
- Benchmark first: The hardware recommendations advise benchmarking before buying, because no two clusters are alike.
Check your decision with a written expansion plan that names each host, drive, network port and memory change, and get sign-off from your storage and network owners.
Expansion settings and when each fits
This table compares four ways to bring new hosts and OSDs into a Ceph cluster and control how data moves afterward. Read each row for the situation it suits and the cost it carries, such as client latency during backfill or drive placement you must review by hand. The main takeaway is that the fastest option is rarely the safest, so pick the method that matches your drive inventory and your tolerance for recovery load.
| Setting or method | When it fits | Trade-off to plan for |
|---|---|---|
| cephadm OSD service spec with a drive selector | Hosts with uniform drives where you want new devices picked up automatically as they are added | Matches more devices than you intended if the selector is broad, so preview the spec before applying it and confirm the behavior in your release documentation |
| Manual ceph orch daemon add | One-off additions, or a specific drive you want to place by hand on a host with mixed media | Each OSD is a separate operator action, which is slower at scale and easy to skip on the next host |
| Balancer in upmap mode | Clusters where PG distribution is uneven after new OSDs join and you want to move data in controlled phases | Balancing moves data that costs I/O, so run it in batches during a window that suits your clients and confirm the mode and tuning in your release documentation |
| Recovery throttling for backfill | Production clusters where client latency must stay within agreed limits while data moves to new OSDs | Lower limits lengthen the time to reach full redundancy, so check the current recovery settings and their defaults for your release before changing them |
| Adding capacity before nearfull | Clusters with growth trends that will reach nearfull thresholds before the next procurement cycle | Early expansion means more idle capacity for a period, but it avoids emergency backfill under write pressure |
| Adding hosts in stages | Larger expansions where each host or group of OSDs can be added, balanced and verified before the next | Staging takes longer end to end, and each stage needs its own health check and client impact review |
How to add a host and its OSDs with cephadm
This sequence brings a new host into a cephadm-managed Ceph cluster in controlled stages, so its OSDs join without starving client I/O. Each step ends with a check that tells you whether to continue.
Confirm the release is cephadm-managed and supported
Before touching the new host, confirm that the cluster was deployed or adopted with cephadm, since cephadm does not support releases older than v15.2.0 (Octopus). Check that your release is one the project still maintains by reviewing the active releases page, and note any planned upgrade so the expansion does not overlap it. Your team or the partner runs
ceph orch psandceph versionsfrom an admin node. Done when every daemon reports the same supported release and the cluster shows HEALTH_OK or only warnings you have already explained.Prepare the new host's SSH, runtime, time and network
Install a supported container runtime, set up time synchronization with chrony or NTP, and configure network bonding across two switches so one switch failure does not isolate the host. Copy the cluster's public SSH key into the root account (or the configured cephadm user) on the new host, and confirm that the cluster can reach it over SSH. Ceph monitors depend on clocks that stay closely aligned, so verify time sync before anything joins. Done when the host passes a manual SSH login from the admin node, reports synchronized time, and both bond members show as active on each switch.
Add the host to the orchestrator inventory
Run
ceph orch host addwith the new host's name and address, and include a label such as the role or rack if your inventory uses labels for placement. Cephadm connects over SSH, installs the cephadm agent components it needs, and records the host in the cluster map. Keep the host in a state where no OSD service spec matches it yet if you want to finish checks first. Done whenceph orch host lslists the host as online with the expected labels and no pending errors.Verify device inventory before any OSD is created
Run
ceph orch device lsand confirm that each data drive shows as available, and that drives you plan to use for WAL and DB, if any, appear with the intended class. Check that system disks and drives belonging to other hosts are not listed as candidates. Compare the drive count and sizes against your design, because a wrong match here is the most common reason an expansion adds the wrong capacity. Done when every intended drive is listed as available and every excluded drive shows as unavailable for a stated reason.Apply the OSD service spec with throttled recovery
Before applying the spec, set recovery and backfill limits so client I/O keeps priority, using the recovery profile your team has agreed. Apply the OSD service spec with
ceph orch apply -iusing a spec that targets the new host by label or name, and add OSDs in batches if your spec allows it. Cephadm creates one OSD per drive, and data starts moving onto them as placement groups map to the new capacity. Done when the OSD count rises by the expected number, all new OSDs report up and in, andceph orch psshows their daemons running.Check health and PG states after each batch
After each batch of OSDs, run
ceph -sandceph pg statto confirm that placement groups are active and clean or backfilling at the expected rate, and that no new slow ops or nearfull warnings appear. Watch client latency on your dashboards, not only cluster counters, because the cluster can look healthy while users see delays. If latency or recovery stalls, pause the next batch and lower the backfill limits before continuing. Done when placement groups return to active+clean for the completed batch and client latency stays within the range agreed before the change.Let the balancer redistribute data in phases
Once the new OSDs are stable, enable or review the balancer and let it move data in phases rather than in one large pass, so recovery traffic stays within the profile you set. Confirm the balancer mode and the target distribution with
ceph balancer status, and check the per-OSD utilization spread withceph osd df. Schedule further moves for periods of low client activity, and confirm with your team how many placement groups are expected to move per phase. Done when utilization across the old and new OSDs converges, the balancer reports no pending changes, and the cluster returns to HEALTH_OK.
Pacing backfill with the balancer and recovery limits
The main lever for protecting clients during an expansion is how fast data moves, not whether it moves. When new OSDs join, CRUSH assigns them placement groups, and every reassigned PG has to backfill its objects across the network and onto the new drives while client I/O keeps arriving. Before adding anything, decide how much movement the cluster can absorb. Add the new OSDs with a reduced CRUSH weight, or set the norebalance and nobackfill flags so the OSDs join without data flowing yet. Then raise weights in small steps, one host or one failure domain at a time, and wait for the cluster to return to HEALTH_OK between steps. Confirm the exact flags and weight commands for your release in the cephadm and operations documentation before you run them.
The balancer is the second control. Its upmap mode moves individual PGs to even out utilization, and it can be configured to make small, bounded changes instead of large corrections. Run it in phases: check ceph balancer status and the planned changes with ceph balancer eval, let it work during low-traffic windows, and stop it if recovery begins to crowd out client work. Confirm with your own team which balancer mode and window settings your release supports, since these details change between versions.
Recovery and backfill concurrency protect client latency. The relevant settings include osd_max_backfills and the recovery and backfill sleep and priority options, which control how many operations each OSD runs at once and how hard background work competes with client requests. Start conservatively, change one value at a time, and record the baseline. Watch slow ops during the work with ceph -s, ceph health detail and the OSD performance counters. Pause the expansion, meaning clear the weight increases and set nobackfill or norecover, when slow request warnings persist, client latency rises beyond your agreed threshold, or any OSD approaches backfillfull. Resume only after the warnings clear. Ask your partner or vendor to show the exact thresholds and pause procedure in the runbook before the change window.
Timing expansion before nearfull
Adding hosts and OSDs is safest when it starts before the cluster is under pressure. The cards below cover the signals that should start a capacity decision, the cost of waiting until writes stop, and the checks that keep a new host from becoming the next incident.
Nearfull and backfillfull headroom
Treat the nearfull ratio as the start of the planning window, not the emergency threshold. Backfillfull matters just as much during expansion, because new OSDs receive data through backfill and an OSD that crosses that ratio can stall the movement you started. Review per-OSD utilization, not only cluster-wide usage, since one unbalanced OSD can reach these ratios long before the average looks high. Confirm the thresholds your release uses with the Ceph documentation before you set alerts.
Cost of expanding after writes stop
Expanding during a full-cluster event is harder than expanding early. Recovery competes with client writes, operators work under pressure, and each step is more likely to be done wrong. Ask your partner or internal team to estimate how long a planned expansion takes at your data volume, then compare that with the time available before the next ratio is crossed. The gap between those two numbers is your real margin.
Seasonal and batch load
Capacity planning should use peak growth, not average growth. Month-end batches, backup windows, log retention jobs and seasonal sales can add data and I/O at predictable times. Map those periods against your recovery schedule so that large data movement does not start just before a known peak. Write down which workloads write the most and when they run, and check that the expansion window avoids them.
Drive burn-in before joining
A new drive that fails shortly after joining the cluster costs a second round of data movement. Run vendor diagnostics and a sustained write test on every new drive before it becomes an OSD, and record SMART values as a baseline. Check that the host's drive layout matches the hardware recommendations, including separate devices for WAL and DB where your design calls for them. Hold back spare drives from the same batch so a failed replacement does not delay recovery.
Change windows and rollback
Schedule host and OSD additions inside a change window that your operations team can staff, with a named owner for stopping movement if client latency rises. Agree on the thresholds that pause the work before the window opens, such as client latency or recovery queue growth. Plan rollback, which for an added host usually means setting OSDs out and letting data drain. Confirm that your change process and your backup state are current before the first host joins.
Emergencies go to the recovery runbook
If a cluster is already nearfull, blocked or full, do not start an expansion from this page. Stabilize writes first, then follow the dedicated recovery guidance for nearfull and full OSDs before adding capacity. Keep the emergency steps separate from planned expansion so that a rushed change does not start a second incident. Use the planning checks here to avoid reaching that state again.
Questions buyers ask before expanding a Ceph cluster
What drives the cost of adding hosts and OSDs to a Ceph cluster?
The cost depends mostly on how much hardware you add, how much of the work your team already does, and how much data movement the expansion triggers. Hardware and licensing are separate lines: Ceph's source is licensed under LGPL-2.1 or LGPL-3, and the project does not sell a paid edition, so software cost is the engineering time to plan, deploy and verify. The main recurring costs are staff time for monitoring and for future expansions. Ask any partner for a quote that lists planning, cephadm deployment, the balancer and recovery plan, validation, and handover as separate items, with what is one-time and what needs ongoing support. The Ceph hardware recommendations advise benchmarking before buying because no two clusters are alike, so budget that step too.
Can our own team add hosts and OSDs without an outside partner?
Yes, if your team has operated Ceph before and can work through the cephadm workflow carefully. cephadm manages a cluster as containers, and it adds hosts and services to a cluster that started from a bootstrapped Monitor and Manager. Your team needs experience with CRUSH failure domains, placement group behavior, network bonding across two switches and recovery tuning, because a mistake during expansion is what usually causes client impact. A partner is most useful when the team is small, when the expansion is large relative to the existing cluster, or when a deadline makes a second pair of experienced hands valuable. Ask whether the work would be fixed-scope and what your team would be expected to review.
How long does backfill take after adding OSDs, and what controls its speed?
Backfill duration depends on how much data must move, the drive and network throughput of the cluster, and how much capacity the existing hosts can spare for recovery while still serving clients. There is no reliable general figure, so measure it on your own cluster during a planned step. The main controls are the number of OSDs you add at once, the balancer's pace in moving data in phases, and recovery profiles that limit how aggressively backfill competes with client I/O. Also plan for the window before the cluster reaches nearfull, because expanding late forces faster, riskier movement. Ask your partner to show the expected movement per phase and the thresholds at which the team pauses.
What does the work require from our infrastructure and operations teams?
The expansion needs access and decisions from several groups before any hardware is racked:
Which OSD and host choices change how a Ceph expansion behaves?
The choices that matter most are whether each OSD has its own dedicated drive, whether WAL and DB sit on shared SSDs for HDD-based OSDs, and whether the new hosts match the existing ones in memory and network speed. The Ceph hardware recommendations advise a dedicated drive per OSD, moving WAL and DB to shared SSDs on HDD-based OSDs, enterprise-class SSDs for metadata, and bonded network links across two switches. They also say to size memory for peak use rather than calm periods. Mismatched hosts can create uneven load after the balancer runs, so confirm the CRUSH placement and device classes before adding capacity. Check your current DB device sizing too, because new OSDs with undersized DB devices can cause spillover later.
How do we get a fixed-scope quote for expanding our Ceph cluster?
Most DeployOpen engagements are fixed-scope, and scope, deliverables and timelines are agreed in writing before work starts. To get a quote, prepare a written description of the current cluster: Ceph release, number of hosts and OSDs, drive types, network layout, failure domains, current utilization and the capacity you plan to add. State the target date and any change windows, and say whether you need the work done only, or also validation, documentation and handover. Ask for the quote to name the deliverables, the assumptions about your environment, what you must provide, and what is excluded. DeployOpen does not publish a price list, so the quote follows from that written scope. If you want continued help after the expansion, ask about a retainer separately.
Which Ceph release should we target before expanding the cluster?
Check your current release first, then choose a supported target before adding capacity, so you are not expanding a cluster that is about to need an upgrade. According to the Ceph releases page, the actively maintained releases are Tentacle (20.2.x) and Squid (19.2.x), and actively maintained releases receive periodic backports and security fixes. Squid's estimated end of life is October 31, 2026, and Tentacle's is June 1, 2027. Confirm the current end-of-life dates on the releases page before you plan, because the estimate can change. If your cluster runs an older release, treat the upgrade as a separate project and decide with your partner whether it comes before or after the expansion.
How DeployOpen scopes and delivers a Ceph expansion
DeployOpen starts an expansion with a written scope that names every host and OSD being added, the cephadm service specifications that will place them, the Ceph release the cluster runs, and the current raw capacity, used capacity and nearfull and full ratios that set the deadline. The scope also records what is out of bounds: whether the balancer may run during business hours, which recovery profile applies, what rollback looks like if a new host misbehaves, and who on the client side approves each phase. Ask for this document to state the success checks in plain terms, such as the cluster returning to HEALTH_OK and the client I/O latency that the operations team already tracks. A quote follows from this written scope, so changes to host count, media or network design after signing are handled as scope changes.
The change window is planned from the data movement the expansion will cause, not from the hour the hardware arrives. DeployOpen works with the client's operators to choose the recovery and backfill limits, to decide whether new OSDs join at a low CRUSH weight, and to schedule balancer phases so that each phase finishes inside an agreed window. Confirm with the client's own change board how long a phase may run, which monitoring alerts will be silenced and which will stay live, and who can stop the work. Work is performed in the client's cloud accounts, data centers or air-gapped networks, under the client's access controls, so the client's engineers keep the audit trail of every host join and every ceph orch command.
Support is offered as a separate retainer after the expansion is complete, covering monitoring review, Ceph point releases, and the next capacity review before the cluster reaches the nearfull threshold again. The retainer scope, response expectations and reporting cadence are agreed in writing, not assumed. Ask any partner, DeployOpen included, to name the people who will run each phase, the hours they will be on call and the handover documents that will remain with your team after the work ends.
