Why osd_memory_target sets how each Ceph host behaves under load
The `osd_memory_target` setting is the main control for how much memory each BlueStore OSD may use, and it must be sized for peak use, not average use. If your OSD hosts show latency spikes, swap activity or OSD processes killed during recovery or metadata-heavy traffic, the first question is whether the memory each daemon is allowed to use still fits what the host does at its busiest. Average memory use on a quiet afternoon tells you little, because recovery, backfill, scrubbing and bursts of small-object or metadata operations all push an OSD above its normal footprint. The Ceph hardware recommendations advise sizing memory for peak use and benchmarking before buying, since no two clusters behave alike. Treat the setting as one part of a host budget that also covers the operating system, the monitors and managers that may share the machine, and any co-located services.
Review osd_memory_target whenever the host or its workload changes. The usual triggers are new hardware with a different RAM-to-drive ratio, adding more OSDs to an existing host, moving to workloads with many small objects or heavy metadata, and any recovery or rebalancing event that changes how much memory OSDs use for a sustained period. Each OSD on a host draws from the same physical memory pool, so a host that was comfortable with a few OSDs can fall short after drives are added, even when the per-OSD value has not changed. Confirm the current value on each OSD with your Ceph tooling, compare total targets across the host against installed RAM, and check that the margin still holds during a simulated recovery on a test or maintenance window. Record the results so the next hardware change starts from measured figures rather than memory of past sizing.
Plan the review as part of capacity work, not as an emergency fix after an OSD is killed. Confirm with your hardware vendor and the Ceph documentation for your release how the setting interacts with the cache and autotuning behavior you run, because the exact defaults and accepted ranges can differ between releases. Ask your team to define what counts as acceptable headroom on each host, and make that figure part of the purchase specification for future nodes. Teams that want an independent check of their numbers can have DeployOpen review sizing and benchmark results against the written scope of their environment, but the figures must come from measurements in your own cluster.
How BlueStore cache and osd_memory_target relate
osd_memory_target is a target, not a hard cap. Each OSD tries to keep its total memory use near the value set in osd_memory_target, but the process can exceed it during peaks, recovery or compaction, and the setting does not stop allocation. Most of that memory is not spent on the OSD's code or buffers. It goes to BlueStore's caches, which hold object metadata (onodes), omap and RocksDB data, and object data. Autotuning, which is on by default in the releases we support, shifts the split between the metadata and data caches and adjusts their sizes so the OSD tracks the target. Treat the target as the budget that the caches fill, and the rest of the process as overhead you must leave room for. Confirm the autotuning and cache option names and defaults for your release in the Ceph documentation before you change them.
The cache matters most when the workload is metadata-heavy. Clusters with very many small objects, large omap usage, heavy RocksDB activity or many placement groups per OSD put pressure on the metadata side, so a cache that is too small forces more reads from the drive for every operation. Large sequential object workloads depend far less on cache size. This is why no general cache figure can be recommended. Measure before you change anything: record object count per OSD, PG count per OSD, omap and RocksDB size, and the cache hit behavior exposed through the OSD performance counters during a representative load. Then compare the results after each change, using the same benchmark method you used for the baseline.
Plan for memory pressure as well as cache size. Size host RAM for the peak demand of all OSDs on the node plus recovery, not for a quiet period, as the Ceph hardware recommendations advise. Check host swap and OOM killer logs after any change, because an OSD pushed past available memory can be killed or can start swapping, which slows the whole cluster. Ask your team or partner to confirm the minimum memory recommended for your Ceph release and drive type, and to record which OSDs the change applies to, so a rollback is simple if latency rises.
Which memory settings to change and when each fits
This table compares the memory settings a tuning engineer reviews for OSD hosts, from per-OSD memory targets to host headroom. Read each row as a decision: when the setting fits, what it trades away, and the first check to run before changing it. Confirm exact key names and defaults against the Ceph documentation for the release you run.
| Setting or decision | When it fits | Trade-off | First check |
|---|---|---|---|
| osd_memory_target (per OSD) | Most clusters that keep autotuning on; set it when a host has spare RAM and cache hit rates are low on metadata-heavy pools | A higher target raises cache hit rate but multiplies across every OSD on the host, shrinking headroom for the OS and other services | Compare the target times OSD count against usable host RAM, leaving room for peak recovery use |
| BlueStore cache autotuning | Default choice on most releases, so the cache follows the OSD memory target without manual sizing | Less direct control over how memory splits between metadata, data and key-value caches under unusual workloads | Confirm autotuning is enabled for the release in use and review the cache split in OSD performance counters |
| Manual BlueStore cache sizes (when autotuning is off) | Specialist cases where a fixed split between metadata and data caches is needed and measured | Fixed sizes do not adjust when memory pressure changes, so an oversized value can push the host into swap or OOM kills | Record the chosen cache sizes and check that the sum across all OSDs on the host stays within the memory budget |
| Host memory reserved for OS and services | Always; sized for the OS, monitors, managers, container runtime and any co-located workloads | Reserving more protects against swap and OOM kills but leaves less RAM for caching | List every process that shares the host and measure its resident memory during recovery, not only at idle |
| OSD count per host | Dense hosts need a lower per-OSD target; sparse hosts can afford a higher one | More OSDs spread I/O but each gets less memory, so cache hit rate per OSD drops | Divide usable host RAM by OSD count and confirm the per-OSD figure still meets the workload's metadata needs |
| Peak memory during recovery | Any cluster that rebuilds after drive failure or rebalancing, which is when memory use rises | Sizing for calm periods leaves no margin when recovery and client I/O overlap | Benchmark or observe memory during a controlled recovery and size against that peak, as the hardware guidance advises |
| Swap configuration | Use as a last-resort safety net only; OSD latency becomes unpredictable once paging starts | Swap can prevent an OOM kill but usually causes slow requests and timeouts first | Check whether any OSD host is swapping under load and treat any swap activity as a sizing signal |
How the right memory target differs for each workload
Object gateways and CephFS workloads with many small objects or files generate heavy metadata traffic. Each object or file touches onodes, extents and omap data that BlueStore tries to keep in its metadata cache, so hot working sets often exceed what the cache can hold. When that happens, reads fall back to the drive, latency climbs, and the cache churns without helping much.
- Cache pressure signs: rising BlueStore metadata cache misses, slow requests that cluster on specific OSDs, and latency that grows as the bucket index or directory tree gets larger.
- What to measure: OSD perf counters for the BlueStore cache sections over time, the trend of hit ratios during peak hours, and per-OSD op latency compared with the object count held on each OSD.
- Trade-off: more memory for cache can absorb the working set, but every gigabyte given to cache is one less gigabyte of headroom for spikes in client connections and for the RGW or MDS processes on the same host.
- Placement check: confirm whether RGW, MDS or monitors share the host, because their memory use adds to OSD demand.
- Ask your vendor or team: how the metadata share of cache is split for your Ceph release, and which counters confirm the working set fits.
Before raising the target, compare cache trends against the object count per OSD so you know whether the problem is cache size or access pattern.
Block workloads for virtual machine disks usually issue large, mostly sequential I/O. The metadata footprint per gigabyte is small compared with object or file storage, so a larger cache often gives little benefit. The bigger risk is that the host runs out of headroom when many virtual machines burst at once.
- Cache pressure signs: cache hit trends stay flat as cache grows, while latency is driven by drive throughput or network saturation rather than metadata lookups.
- What to measure: per-OSD commit and apply latency, the trend of OSD resident memory under peak virtual machine load, and client latency on the hypervisor side.
- Trade-off: keeping memory as host headroom protects against swap and out-of-memory kills during bursts, which hurt virtual machines far more than a modest drop in cache hit rate.
- Hypervisor interaction: if the hypervisor and OSDs share hosts, confirm that guest memory and OSD memory together leave room for the kernel and page cache.
- Ask your vendor or team: whether the hardware baseline shows that drives or the network, rather than metadata, set the latency ceiling.
If cache hit trends do not move when you change the target, favor headroom and review drive and network limits instead.
Clusters that serve clients and also rebuild data after a drive failure face a different profile. Recovery and backfill add reads, writes and memory use on the OSDs at the same time as client I/O, so the memory peak can be well above the calm-period figure. A target sized only for steady state can fail during the first hours of a rebuild.
- Cache pressure signs: memory use climbs during recovery, cache hit trends fall as recovery touches cold objects, and client latency rises even when drives are not saturated.
- What to measure: OSD memory during a controlled recovery test, client latency during that test, and recovery progress against the scheduler profile in use.
- Trade-off: a larger cache may help client reads during recovery, but recovery traffic can evict useful entries anyway, so the gain is often smaller than expected while the memory risk stays.
- Peak sizing: the Ceph hardware recommendations advise sizing memory for peak use rather than calm periods, so test the peak instead of assuming it.
- Ask your vendor or team: what recovery memory peak your hardware produces in a controlled drill, and how it compares with the host memory left after all services start.
Run a recovery drill on a non-production copy or a maintenance window before settling on a value, and record the peak per OSD.
Sizing osd_memory_target for your host, step by step
This sequence takes a host from raw hardware inventory to a tested osd_memory_target value that leaves room for the operating system, containers and peak recovery work. The output is a documented per-OSD memory setting that fits the host budget and has been checked for swap use under load.
Inventory host memory and OSD count
The cluster administrator records the physical RAM on every OSD host, the number of OSDs each host runs, and any other services scheduled on those hosts, such as monitors, managers, gateways or client workloads. Use the host's memory total and the list of OSD IDs per host from the orchestrator inventory, and note whether hosts differ in RAM or drive count. Done means you have one table with a RAM figure and an OSD count for every host, with hosts grouped by hardware type so that mismatched nodes stand out before any sizing begins.
Reserve memory for the operating system and containers
Set aside a fixed memory budget for the kernel, page cache needs, the container runtime, cephadm daemons and any co-located services on each host. Subtract this reserve from total RAM to get the memory available to OSDs. Ask your platform team to confirm the reserve for your OS image and to record any service that will be added later. Done means each host has a written budget figure, expressed as the RAM left for OSDs, that the team agrees to review whenever the host's role changes.
Measure peak OSD memory under real load
Run the cluster under representative client traffic, then trigger recovery and scrub activity, for example by taking one OSD out and bringing it back in a maintenance window, and sample each OSD process's resident memory during the peak. Record the highest value per OSD, not the average, because the Ceph hardware recommendations advise sizing memory for peak use rather than calm periods. Done means you have a peak figure per OSD and a note of which load pattern produced it, which you can compare against the reserved budget.
Set osd_memory_target and check the host sum
Choose a per-OSD value for osd_memory_target that keeps the sum of all OSD targets on the host below the memory budget from step two, with headroom for peaks above the target. Apply it through the Ceph configuration database, for example with the
ceph config setcommand for the osd scope, and confirm the exact syntax for your release in the Ceph documentation or with your DeployOpen engineer before running it. Done meansceph config getreturns the new value for the OSDs, and a written calculation shows that OSD count multiplied by the target fits inside the host budget on every host.Verify no swap use at peak
Repeat the peak test from step three with the new setting applied, and watch swap usage, page-out rates and OOM killer messages on each host, using the host's standard monitoring tools. OSDs that swap lose latency predictability and can be killed by the kernel, so any swap activity under peak is a failed result. Done means the peak run completes with no swap growth, no OOM events in the kernel log, and OSD resident memory stays near the chosen target, with the results saved next to the earlier baseline.
Document the setting and its assumptions
Write down the host budget, the OSD count, the chosen target, the peak figures and the test date for each host class, and store the record with your cluster configuration documentation. Include the assumptions that would invalidate the value, such as adding drives, moving monitors onto the host, or enabling new client workloads. Done means a colleague who was not part of the tests can read the record and reproduce the calculation from the inventory.
Re-check after adding OSDs or changing workloads
Repeat the budget calculation and a shorter peak test whenever you add OSDs to a host, move a service onto it, change the pool mix in a way that alters metadata or client load, or upgrade Ceph to a new release. Adding one OSD changes the host sum immediately, so the check should happen before the new drive joins the cluster. Done means the record from step six is updated with the new OSD count and sum, and the host still passes the swap and OOM check at peak.
Swap, OOM kills and host risks to plan for
Memory settings only hold if the host has room for them. The cards below cover the failure modes that show up most often when OSD memory is set too high for the host, or when something else on the host quietly takes memory the OSDs expected to have. Each card names the signal to watch and the control to apply.
Swap turning latency into timeouts
When an OSD host starts paging, OSD operations wait on disk reads that were never meant to happen, and client requests stall long enough to time out. This is often worse than a clean restart because the cluster sees slow OSDs rather than dead ones. Watch swap-in and swap-out activity during peak and recovery periods, not only average usage. Control it by keeping swap either disabled or small, and by sizing OSD memory so the host never needs to page under load.
OOM killer ending OSD processes
Recovery and backfill can push an OSD's resident memory well above its steady-state level, and the kernel's out-of-memory killer may then terminate the process. Each restart forces peering and more recovery work, which can feed the next memory spike. Check kernel logs for OOM events and correlate them with recovery windows. Control it by budgeting memory for peak recovery load, not calm periods, and by confirming the host has headroom before adding OSDs.
Co-located services competing for memory
Monitors, managers, RGW gateways, hypervisors or application containers sharing a host with OSDs compete with them for the same RAM. The OSD memory target only governs the OSD's own cache, so other processes can push the host into swap without any change to Ceph. Inventory every process on each host and its memory limit before setting targets. Control it by reserving memory for the operating system and co-located services, and by keeping dedicated OSD hosts where the workload justifies it.
Overcommitting OSDs per host
Adding more OSD daemons to a host multiplies the memory each one can use, even when each target looks reasonable on its own. Overcommitment usually appears after a capacity expansion when new drives are added to existing hosts without revisiting the memory budget. Calculate the total of all OSD targets plus system and service memory against physical RAM for each host. Control it by making the per-host memory check part of every drive addition or replacement plan.
Memory behavior changing across upgrades
A Ceph upgrade or a change in the operating system, container runtime or kernel can change how much memory OSDs use and how the cache is autotuned. Behavior that was stable on one release may shift on the next, so a host that ran safely before may start paging afterward. Record baseline memory use per OSD before the upgrade and compare it after each step. Control it by including memory checks in the upgrade test plan and by keeping a rollback point for host-level settings.
Monitoring gaps hiding memory growth
Cluster health views may stay green while one host's OSDs slowly grow in memory or a host begins swapping, because the usual alerts track OSD state rather than host memory. By the time an OOM kill or timeout appears, the trend has usually been building for days. Track per-OSD memory and host swap and memory pressure over time, and alert on sustained trends rather than single samples. Confirm with your monitoring team which exporters already collect these metrics before assuming coverage.
What memory tuning costs and what the work needs from your team
What does an osd_memory_target tuning engagement cover?
It covers reviewing the current memory settings on each OSD host, measuring how BlueStore cache is used under your real workload, setting a target that leaves room for peak use, rolling the change out in controlled steps and confirming the result. Scope is set in writing before work starts. The main inputs are the number of hosts and OSDs, whether a baseline benchmark exists, how many pools and workload types must be measured, and whether the team needs help with change control and rollback planning. A cluster with a recorded baseline and a small, uniform fleet usually has a narrower scope than a mixed fleet with no prior measurements.
What drives the cost of memory tuning work?
The cost is driven by effort, not by a fixed package. The largest drivers are:
- Cluster size: the number of hosts and OSDs to review and change.
- Benchmarking scope: how many workload types, pools and client patterns must be measured before and after the change.
- Baseline status: whether monitoring data and earlier benchmark results already exist, or must be collected first.
- Change control: how many maintenance windows, approvals and rollback steps your organization requires.
- Heterogeneity: whether hosts differ in RAM, drive count or OSD layout.
When requesting a quote, ask for each of these items to be priced separately so you can see which ones change if scope shrinks. DeployOpen provides a quote from the written scope.
Can our own team tune osd_memory_target without a partner?
Yes, if your team already runs Ceph day to day and has the skills to read OSD and BlueStore counters, interpret workload results and manage configuration changes through cephadm. The work is mostly careful measurement and staged change, so the team needs time for benchmarking and a safe way to revert. A partner adds value when the fleet is large, when no baseline exists, when workloads are metadata-heavy and hard to reproduce, or when the team cannot safely take OSD hosts out of service for testing. The decision usually comes down to available engineering time and how much risk your change process can absorb.
How long does memory tuning take on a production Ceph cluster?
There is no fixed duration, because the work depends on the benchmark window, the number of change windows your process allows and how many validation cycles the workload needs. Metadata-heavy workloads often need longer observation periods than steady block workloads, since cache effects appear over time. Ask for a timeline that separates baseline collection, change rollout and post-change verification, and confirm which of those steps depend on your maintenance calendar. Any schedule should be agreed in the written scope before work starts.
What must our team provide before memory tuning starts?
Your team should prepare the inputs that make the work measurable and safe:
- A host inventory with RAM, drive counts, OSD layout and any co-located services.
- Historical monitoring data for OSD memory use, swap activity, OOM events and client latency.
- Agreed maintenance windows and a rollback approach for each change.
- Access through cephadm to view and set configuration on the affected hosts and daemons.
Confirm with your Ceph administrators which dashboards and logs already exist, and whether you have a current benchmark record. Missing monitoring history usually extends the baseline phase.
Does changing osd_memory_target require an OSD restart?
Confirm this against the Ceph documentation for your release before planning, since behavior can differ between versions and configuration methods. In general, operators set the value through the cluster configuration database and then verify the effective value on each OSD. Where a change does not take effect on a running daemon, the affected OSDs must be restarted, which should be done one host or failure domain at a time so data stays available. Plan restarts inside maintenance windows and check cluster health between each step.
How do we confirm memory tuning results after a recovery event?
Compare post-recovery OSD memory use, swap activity, OOM events and client latency against the baseline you recorded before the change. Recovery and backfill temporarily change memory and I/O patterns, so measure after the cluster returns to a healthy state, not during recovery. Keep the same workload definitions used in the original benchmark so results are comparable. If results differ, check whether the recovery profile, placement group count or host load changed at the same time before attributing the difference to the memory setting.
What to confirm in your release and environment
Memory tuning depends on the exact Ceph release, the OSD media, and the container and kernel limits on each host. The cards below list what to check before changing any value, and where the official sources answer each question.
Option names and defaults
Confirm the exact memory option names and default values in the Ceph release you run, whether that is Squid or Tentacle. Defaults and option behavior can change between releases, so do not copy values from older blog posts or forum threads. Check the running value on each OSD with the configuration query commands and record it in your change log before you alter anything. Compare the result against the release documentation for your version.
Autotuning on your media
Verify that cache autotuning behaves as expected on your OSD media, because HDD, SATA SSD and NVMe OSDs can receive different cache budgets from the same host. Watch the cache distribution across the BlueStore metadata, key-value and data caches after a restart and under a metadata-heavy load. Confirm that the autotuner does not starve metadata caches on OSDs that serve small objects or many RBD snapshots. Document the result per media class.
Container and kernel memory accounting
Find out how your container runtime and kernel count OSD memory, including cgroup limits, page cache and any memory overhead outside the OSD process. A limit set below the memory target can trigger OOM kills even when the OSD appears healthy. Confirm which memory limits cephadm applies to each OSD container on your hosts. Test a node under recovery load before you rely on the numbers.
Chassis and swap limits
Check the physical memory your chassis supports, the number of OSDs each host carries, and how much memory the host keeps for the operating system, monitors, managers and other daemons. Multiply the per-OSD target by the OSD count and compare the total with installed RAM at peak use, not idle use. Confirm whether swap is enabled and how the host behaves when it is exhausted. The Ceph hardware recommendations advise sizing memory for peak use.
Release support status
Confirm that your Ceph release still receives backports and security fixes before you invest in tuning it. Check the estimated end-of-life date for your release and plan any upgrade so that tuning changes are not lost or repeated. Confirm the status with the releases page on the day you make the change, since maintenance windows move.
Vendor and hypervisor guidance
Check whether your hardware vendor, hypervisor or storage integrator publishes memory guidance for OSD hosts that overrides or narrows the upstream advice. Some platforms reserve memory for their own agents or set container limits by default. Ask your vendor to confirm supported configurations in writing before you change memory settings on production hosts. Keep a rollback plan for each change.
How DeployOpen scopes and delivers memory tuning for Ceph
DeployOpen plans, implements, documents and supports self-hosted open-source platforms inside your own infrastructure, including Ceph clusters. For OSD memory and BlueStore cache work, we start from a written scope that ties memory targets to the hosts, drive counts and workloads you actually run, then make changes in your environment with verification and handover built in.
How we scope
- Inventory of OSD hosts, RAM per host, OSD count per host and colocated services such as monitors, managers or gateways
- Workload profile covering block, file and object pools, with metadata-heavy and small-object patterns called out
- Written list of tuning changes, the test method, the rollback steps and the hosts in and out of scope
- Confirmation of which cephadm-managed release your cluster runs and which memory options it exposes
How we build
- Baseline capture of OSD memory use, swap activity, OOM events and BlueStore cache hit behavior before any change
- Staged changes to osd_memory_target and cache settings on one host first, then across the rest of the cluster
- Load tests that reproduce peak client I/O, recovery and scrub activity, not only idle or average conditions
- Configuration recorded in the cluster's own settings store so changes are traceable and reversible
How we hand over
- Written memory sizing record per host class, with the formula used and the headroom kept for peak use
- Runbook for checking memory use, swap and OOM signals, and for adjusting targets after adding drives or RAM
- Documented before-and-after measurements from your environment, with the test conditions stated
- Walkthrough with your operations team so they can repeat the checks without DeployOpen
How we support
- Fixed-scope engagements, with scope, deliverables and timelines agreed in writing before work starts
- Ongoing support, upgrades and periodic re-testing are available as a retainer after handover
- Review of memory settings when you change release, add OSDs, move WAL and DB devices or change colocation
- Quote follows from the written scope, so ask us to list each tuning task, test and document separately
