BlueStore compression for Ceph pools: when it saves capacity and when it only costs CPU
Compression is a per-pool choice that depends on the data, so treat it as a pool decision rather than a first step to take cluster-wide. Consider it when your cluster is running short on usable capacity or when a pool's OSD hosts are CPU-constrained, and then check which pools hold data that shrinks. BlueStore, the object store that OSDs use, compresses data as it is written to the drive, so the savings depend on how compressible each pool's data is. The same setting can save a large share of space on one workload and almost nothing on another. Compression is configured with pool options such as compression_mode, compression_algorithm and compression_required_ratio, and OSD-level defaults can also apply. Check which layer a value comes from before you assume a pool is compressing.
The cost lands on the OSD hosts. Compression and decompression consume CPU on the machines that already run OSD daemons, alongside recovery, scrubbing and client I/O. Data that is already compressed or encrypted, such as media files, archives or ciphertext, usually will not shrink enough to justify that work, so you pay the CPU cost for little or no capacity gain. Before you enable anything, measure CPU headroom on the hosts during peak client load and during recovery. Use the Ceph hardware recommendations to check host sizing, and confirm the supported algorithms and their behavior for your release in the Ceph documentation.
The sequence matters more than any single setting. Identify the pools with compressible data, confirm the release-specific compression behavior, and measure the host CPU cost. Then enable compression on one non-production pool and compare the capacity used and the OSD CPU before and after. Record the baseline first, because compression changes the figures you will later use for capacity planning. The sections that follow cover the algorithms and modes, the required ratio that decides whether a write is stored compressed, and how blob size affects the result.
Which pools suit BlueStore compression and which do not
Compression is a per-pool decision, and the right answer depends on what the data looks like and what the hosts can spare in CPU. The cards below cover the data types and pool roles where compression tends to help, where it tends to cost CPU for little return, and the check to run on a representative sample before enabling it in production.
Text-heavy object data
Logs, JSON exports, CSV extracts, source archives and plain-text documents usually have repeated patterns that compress well. These are the strongest candidates for an RGW data pool where most objects are large and written once. Before enabling compression, sample real objects from the bucket and compress a copy with the algorithm you plan to use. Measure the ratio on your own data, because a vendor's example figure says little about your files.
Backup and archive targets
Backup repositories are usually written in large sequential streams and read rarely. Many backup tools already compress or deduplicate their output, so the gain from BlueStore compression can be small. Check whether your backup software compresses before writing, and if it does, test whether a second pass saves enough capacity to justify the CPU use during backup windows.
Virtual machine disks with small writes
RBD pools backing virtual machine disks often see random small writes and mixed reads. Compression adds work to each write path, and small blobs may not reach the required ratio, so the capacity gain is often low while latency rises. Treat these pools as a default exclusion unless a benchmark on the actual workload shows a clear benefit.
Already compressed media
Video, JPEG, compressed archives and other media files have little redundancy left to remove. Compression attempts on these objects consume CPU and usually fail the required ratio, so BlueStore stores them uncompressed. Keep these buckets or pools out of compression unless you have measured a gain that justifies the overhead.
Encrypted or pre-compressed client data
Data encrypted by the client before it reaches Ceph looks random to the storage layer, so it will not compress. The same applies to data that the application has already compressed with a strong format. Ask application owners whether they encrypt or compress before upload. If they do, a compression policy on that pool adds CPU cost without a capacity return.
CPU-bound OSD hosts
OSD hosts that already run close to their CPU limit during recovery, scrubbing or peak client load should not take on extra compression work. Check CPU headroom under realistic load before enabling compression on a pool that shares those hosts. If the hosts have little spare CPU, the saved capacity may not be worth slower client I/O during peaks.
Compression options and when each one fits
This table compares the main BlueStore compression levers, from the pool or cluster mode down to blob sizes, and explains what each controls, when a team would change it and what it costs. Read it as a checklist: start with whether the data compresses at all, then tune the mode, ratio and blob sizes per pool rather than cluster-wide.
| Lever | What it controls and when to change it | Trade-off to weigh |
|---|---|---|
| Compression mode off | Stores data uncompressed. Choose it for pools holding encrypted, already compressed or random data such as media, archives and some backup payloads. | No capacity savings, and no CPU spent on compression or decompression. |
| Passive mode | Compresses only when the client hints that data is compressible. Suits mixed pools where some writers, such as database or log exporters, set hints and others do not. | Savings depend on client hints being set correctly, so results can be uneven. Verify hint behavior with your clients. |
| Aggressive mode | Attempts compression on writes unless the client says the data is incompressible. Suits pools dominated by text, logs, VM images with compressible guest data. | Spends CPU on every write, including data that will not shrink, which can slow OSD-bound workloads. |
| Force mode | Compresses everything written to the pool regardless of hints. Use only when the data is known to compress well. | Highest CPU cost for no benefit on incompressible blobs. Test on a pool copy before enabling. |
| Compression algorithm | Sets the codec used for compressing blobs. Change it to trade compression ratio against CPU time on your OSD hosts. | Faster codecs save less capacity; denser codecs cost more CPU. Confirm the supported list and defaults in the BlueStore documentation for your release. |
| Required ratio | Sets the minimum compressed-to-original size needed before BlueStore keeps the compressed copy. Raise it when savings are marginal and decompression on reads is not worth the effort. | A higher threshold stores fewer blobs compressed, so capacity savings shrink. A lower one keeps more small gains with more CPU work. |
| Minimum and maximum blob sizes | Bound the size of blobs eligible for compression. Small blobs compress poorly and add overhead; very large blobs make small reads decompress more data than needed. | Tuning these changes read amplification and fragmentation. Benchmark read latency for your workload before changing them. |
| Cluster defaults versus per-pool overrides | Cluster-wide settings apply to every pool unless overridden. Per-pool settings let you enable compression only where the data suits it. | Cluster-wide enablement is simpler to run but wastes CPU on unsuitable pools. Per-pool overrides need documenting so later changes do not surprise teams. |
How the required ratio and blob size decide what is stored compressed
BlueStore does not compress every blob it writes. It compresses a blob only when the compressed result meets a required ratio, and otherwise stores the blob uncompressed. That means the CPU cost of compression is paid on every write that is attempted, while the capacity saving appears only for data that clears the threshold. Before enabling compression on a pool, confirm how the required ratio is applied in your Ceph release, including whether the pool-level setting or the cluster-wide default governs it, and check the current values with ceph osd pool get and the BlueStore configuration on a test cluster. The ratio threshold is the first control to review, because a high threshold on data that compresses only modestly will produce CPU load with little capacity return.
Blob size shapes both sides of that trade. Larger blobs give the compressor more repeated patterns to work with, so they are more likely to reach the required ratio. The cost is on the read path: BlueStore must decompress a compressed blob to return any part of it, so a small read from a large compressed blob can touch far more data than the application requested. Small writes and small objects tend to compress poorly for the same reason, since there is little data for the algorithm to find patterns in, and many of them will be stored uncompressed after the attempt. Workloads dominated by small random reads, such as databases or virtual machine disks, deserve particular scrutiny. Confirm the blob size settings that apply to your release and test read latency alongside capacity savings.
Measure the achieved ratio on your own data before you rely on it. Write a representative sample of the pool's real object sizes and content types, then compare logical and stored bytes after compression is enabled. Reported savings from a generic dataset, or from a vendor example, do not predict how your files, database pages or already encrypted content will behave. Encrypted or previously compressed data usually gives little return, so identify those pools early. Record the results alongside the benchmark baseline so that a later change to the ratio or blob settings can be judged against measured numbers rather than expectations.
Enabling compression on a pool and checking the result
This sequence turns BlueStore compression on for one low-risk pool and measures whether it saves raw capacity without costing more in OSD CPU or client latency than the workload can accept. Each step ends with a check, so you can stop or roll back before the change reaches other pools.
Confirm the release and settings you plan to use
Check the Ceph release running in your cluster against the BlueStore compression documentation for your version, and confirm which algorithm, compression mode and related options it supports. Use the active releases page to confirm your release is still maintained, since Squid and Tentacle have different end-of-life dates. Record the release and the exact option names in your change ticket. Done means someone other than the author has reviewed the settings against the documentation for that version and signed off on them.
Record baseline capacity and OSD CPU use
Capture the current raw capacity, the used bytes per pool, and the stored data for the candidate pool, along with CPU utilization on the OSD hosts over a period that includes normal daily load. Use the cluster status and pool statistics output, and your host monitoring for CPU. Save the figures in a shared record. Done means you can state the pre-change stored and raw bytes for the candidate pool and the typical OSD CPU range under load.
Benchmark a representative workload before the change
Run a pool-level benchmark that matches the workload the candidate pool serves, such as small random writes for a database or large sequential writes for backups, and record throughput, latency and OSD CPU. Repeat the run so you can see variance. Done means you have a saved baseline you can compare against after compression is enabled. If your workload mix is unknown, ask the application owner to describe it before you choose the test.
Pick a non-production or low-risk pool
Select a pool that holds data that is compressible in principle, has a clear owner, and can tolerate a short period of observation. Avoid pools that already hold data encrypted or compressed by the application, and avoid pools that serve latency-sensitive production traffic on the first attempt. Done means the pool, its owner and the rollback owner are written down together.
Enable compression on the pool
Set the compression algorithm, mode and any ratio or blob size options on the chosen pool only, using the configuration keys confirmed in the first step. Verify the setting with the pool configuration query so the value on the pool matches what you intended. Done means the query returns the expected values for every compression option on that pool and no other pool has changed.
Let new writes arrive and check the result
Allow normal client writes to reach the pool. Existing data may not be rewritten automatically when you enable compression, so confirm for your release whether old objects stay uncompressed until they are rewritten by your application or a copy process. Compare stored bytes against raw bytes for the pool, along with OSD CPU and client latency against the baseline. Done means you have a comparison table showing savings only for data written after the change, plus any CPU or latency increase.
Decide to keep, tune or roll back
If savings are meaningful and CPU and latency stay within the limits the application owner accepted, keep the setting and document it for the pool. If CPU rises or latency breaks the limit, revert the pool's compression options to their previous values, confirm the pool configuration again, and monitor the OSD hosts until load returns to baseline. Rollback affects only new writes, so note that already compressed objects remain compressed until rewritten. Done means the pool's final configuration is recorded and the decision is linked to the measurements.
How compression changes behavior by workload
Object data written through the Ceph Object Gateway often arrives as large objects, sometimes uploaded in parts, and is frequently read back whole. Two separate layers can compress it: the gateway's own compression feature, configured on a placement target, and BlueStore compression on the RADOS pool that stores the data. Confirm which layer you are configuring before you change anything, because enabling both adds CPU work without adding savings on data that is already compressed. The Ceph Object Gateway documentation describes the gateway-level options.
- Capacity effect: Text, logs, JSON and uncompressed exports often shrink, while images, video and archives usually do not.
- CPU effect: Gateway compression uses gateway host CPU, and pool compression uses OSD host CPU, so the bottleneck moves depending on the layer you choose.
- Access pattern: Large sequential writes suit compression because the blobs are big enough to compress efficiently.
- What to measure: Stored bytes against raw bytes per pool, OSD CPU during peak PUT traffic, and GET latency for frequently read objects.
Record which layer compresses each bucket and placement target, so that later tuning does not compound two settings without anyone noticing.
Virtual machine disks are RBD images with mixed I/O: small random writes from databases and guest filesystems, plus periodic bursts from patching and logging. Small random writes produce small blobs, which can fail the required compression ratio, so the cluster pays the compression cost and stores the data uncompressed. Pool compression applies to every image in the pool, so mixing databases and general-purpose guests in one compressed pool makes the setting hard to tune for either group.
- Capacity effect: Guest operating system files and application data vary widely, and savings are often partial. Free space inside a guest only returns to the cluster if discard is passed through the whole stack, so confirm that setting.
- CPU effect: Compression competes with OSD work during the same hours as client I/O, which typically appears as higher write latency for guests.
- Access pattern: Small random writes create small blobs, so expect little saving for database-heavy images.
- What to measure: Write latency percentiles from a guest benchmark, OSD CPU during peak hours, and stored against allocated bytes after normal use.
Test the setting on a separate pool with copies of representative images before you change a production pool, and keep databases on an uncompressed pool unless testing shows a clear gain.
Backup and archive pools receive large, mostly sequential writes and are read rarely, which is the pattern where compression has the most room to help. Many backup products already compress or encrypt data before it reaches Ceph, though, and encrypted or pre-compressed streams gain little from a second layer. Check the backup tool's settings for compression, encryption and deduplication before you decide where compression belongs.
- Capacity effect: Database dumps and text-heavy file backups often compress well at the storage layer. Encrypted backups and compressed archive files usually do not.
- CPU effect: The cost falls mostly during ingest windows, which can be acceptable if backups run off-peak. Confirm that the backup window does not overlap with recovery or scrub traffic.
- Access pattern: Large sequential writes with infrequent reads suit compression. Restore reads are the path to test, because they run under pressure when someone needs the data back.
- What to measure: Stored against raw bytes after successive full and incremental runs, ingest throughput, and restore time for a sample data set.
Ask the backup vendor which layer already compresses and encrypts the data, then run a full restore test before you choose a mode for the pool.
CephFS keeps file contents in data pools and file system metadata in a separate metadata pool, which is usually not the target for compression. Data pools serve many clients with a mix of small files, log appends and large sequential files, so savings differ between home directories, build artifacts and scientific data sets. Because the application mix decides the outcome, measure each directory tree or application before applying a setting to a shared pool.
- Capacity effect: Source code, logs and documents often compress well. Media files and already compressed scientific formats rarely do.
- CPU effect: Workloads with many small files generate many small writes, which can add CPU cost with little saving.
- Access pattern: Large sequential reads and writes of big files suit compression. Small random updates inside a file do not.
- What to measure: Stored against raw bytes per data pool, OSD CPU during client peaks, and latency for small-file operations from a representative client.
Where possible, separate applications with different file patterns into their own data pools, so one workload does not decide the setting for the others.
Security and operational considerations when compressing data
Compression changes how data is laid out, checked and reported on disk, so it touches encryption, scrubbing, capacity planning and audit evidence. The cards below cover the decisions and checks that connect these areas for a BlueStore compression change.
Confirm order of encryption and compression
Compression only reduces size when data is compressible, and that depends on what sits beneath it. Confirm whether your data is encrypted by the client or application before it reaches RADOS, and whether any at-rest encryption is applied by the cluster. Ask your security team to review whether compressed sizes could reveal information about plaintext content for your data classes, and record the decision. Do not assume a side-channel risk is absent or present without that review.
Plan CPU headroom for recovery
Compression runs on the OSD host, so recovery and backfill compete with it for CPU during a drive failure. Measure CPU use on your OSD hosts under peak client load and during a test recovery before enabling compression on busy pools. Ask your hardware vendor and your Ceph engineers to confirm the headroom that remains with your chosen algorithm and blob sizes.
Track stored versus raw bytes
Capacity reports can hide whether compression is working. Track the logical bytes written by clients against the physical bytes allocated on OSDs for each compressed pool, over weeks rather than at a single point. Watch for pools where the ratio never reaches the configured threshold, since those pools pay CPU cost for no capacity gain. Set alerts on the trend so changes in workload are caught early.
Control changes to pool settings
Compression settings are pool properties, and a change affects how new writes are stored, so it belongs under change control. Record the current values, the intended values, the owner, the rollback step and the test result before any change. Apply the change to a non-production pool first where possible. Confirm with your Ceph version documentation how existing data is treated after a setting change.
Check integrity and scrub behavior
Scrubs verify that stored replicas or erasure-coded chunks match their checksums, and compression adds a further transformation that the cluster must reverse on read. After enabling compression on a pool, run a deep scrub on a sample of PGs and confirm that no new inconsistencies appear. Check scrub duration and OSD latency against your baseline. Ask your Ceph support contact to confirm how checksums and compressed blobs interact in your release.
Document the setting for auditors
Auditors ask how stored data is protected and whether the controls are the ones you say they are. Keep a written record of which pools compress, the algorithm, the mode, the threshold, the reason for each choice and the test evidence. Store scrub results and capacity trend reports alongside it. DeployOpen prepares this evidence for assessments, but independent auditors and certification bodies make the determination.
Questions teams ask before enabling BlueStore compression
What does BlueStore compression cost, and who pays for the CPU and testing?
The software itself is free: Ceph is open source under LGPL terms, so the cost is the CPU your OSD hosts spend compressing and decompressing data, the time to test, and the labor to run it. Compression uses host cores that would otherwise serve client I/O, so the CPU cost lands on your hardware budget. Testing is paid for either by your team's hours or by a fixed-scope engagement quoted from written scope. One-time costs include the benchmark and the pool changes; recurring costs include monitoring compression ratios and re-checking CPU headroom after hardware or workload changes. Ask any partner to itemize labor, test environment use and follow-up review in the quote. Any dollar figure should come from a vendor quote or your own benchmark results, not from general estimates.
How long does a compression trial take, and what drives that time?
The duration depends mostly on how long you need to observe representative data, not on the configuration change itself. A trial has to cover a full cycle of your real workload, including peak periods, scrubbing and recovery, because compression cost shows up under load. The main drivers are the number of pools under test, how many distinct data types you have (already compressed media behaves very differently from text logs or database files), the size of the test cluster compared with production, and how quickly you can collect and review results. Ask for a schedule that names the test window, the acceptance criteria and the rollback steps before work starts. No duration should be assumed without a plan built from your environment.
What must our team provide before a compression review starts?
A useful review needs three inputs. First, hardware details: drive types, OSDs per host, CPU model and core count, memory per OSD, and network layout, so the CPU cost can be judged against real headroom. Second, workload samples: which pools hold which data, typical object or block sizes, the share of data that is already compressed or encrypted, and the read and write mix. Third, baseline metrics collected before any change: OSD and host CPU utilization, client latency, recovery times and current raw versus stored usage per pool. Without a baseline, you cannot show whether compression helped. Your team should also name the owner who approves a pool change and the window in which changes are allowed.
Can our in-house team enable and manage BlueStore compression?
Yes, if the team already runs Ceph day to day and has access to a representative test environment. Enabling compression is a pool-level setting and a set of per-pool options, so the configuration work is small. The harder parts are the judgment calls: selecting pools with compressible data, choosing the algorithm and mode, setting the required ratio and checking blob size effects, then watching CPU and latency under real load. A team without benchmarking experience usually needs an outside review of those decisions. Whichever route you take, confirm the exact option names, supported modes and defaults for your Ceph release in the official documentation, because these settings change between releases.
How do we measure compression savings honestly?
Measure stored bytes after compression against the logical data written, per pool, over a full period that includes normal churn. Compare the same workload before and after the change, and separate compression from deletes, snapshots and growth so the numbers do not flatter the result. Track the compression ratio that Ceph reports for each pool, and check whether data below the required ratio is being stored uncompressed, since that data costs CPU for no capacity gain. Record CPU, client latency and recovery time alongside the savings, because a capacity gain that slows clients is a trade-off, not a win. Keep raw results so the comparison can be repeated after upgrades or hardware changes.
Does compression replace adding capacity to the cluster?
No. Compression only reduces the space used by data that compresses well, and the gain varies widely by pool. Encrypted, already compressed or random data often gains little or nothing, so it cannot stand in for planned capacity. Capacity planning also has to cover replication or erasure coding overhead, free space needed for recovery and rebalancing, and growth in data you have not measured yet. Treat compression as one lever that may defer a purchase for some pools, and confirm the expected gain with a benchmark on your own data before you change any capacity plan. Keep the cluster below the full and nearfull thresholds regardless of compression results.
Which pools should we avoid compressing, and why?
Avoid compressing pools whose data is already compressed or encrypted, such as media archives, backup files that use their own compression, and many database or virtual machine images with high entropy. These pools spend CPU and rarely meet the required ratio, so the data is stored uncompressed and the CPU cost buys nothing. Also review pools that serve latency-sensitive clients on hosts with little CPU headroom. Good candidates are pools with text-heavy or repetitive data, such as logs, documents and structured records, where compression ratios are usually easier to verify. Test each candidate pool separately rather than enabling compression cluster-wide.
Does enabling compression change security or compliance posture?
Compression does not provide encryption, and it does not change who can read data. Ceph's messenger v2 secure mode encrypts traffic between daemons and clients, but that is a separate setting from at-rest compression, so review both if you handle regulated data. Confirm that your data retention, deletion and backup procedures still work with compressed storage, and that any evidence you present to an assessor shows the configuration in force at each point in time. DeployOpen prepares systems, controls and evidence for compliance assessments, but independent auditors issue any audit report or attestation.
Limits to confirm in your environment before you enable compression
Compression changes what the cluster stores, how much CPU it spends and how pools behave under load. Each card below is a check to run against your own Ceph release, pool layout and hardware before you set a compression mode on production data.
Algorithms and modes in your release
Confirm which compression algorithms and modes your Ceph release supports for BlueStore, and which ones your cluster configuration exposes at the pool and OSD level. Do not assume a feature from a newer release exists on the version you run. Check the release notes for your exact version and record the supported options in your change plan. Note which mode you intend to use, since the choice determines when data is compressed at all.
Behavior on erasure-coded pools
Confirm how compression interacts with erasure-coded pools in your release before you apply it there. Erasure-coded data is split into chunks, so the compressible unit and the capacity saving may differ from a replicated pool. Ask your storage team to test a representative erasure-coded pool and measure the actual stored size, not the expected size.
Whether existing data is rewritten
Confirm whether enabling compression on a pool affects data already written, or only data written afterward. If existing objects are not rewritten, capacity savings will appear slowly as data churns. If your process requires a rewrite, plan the I/O and the time it adds to a production cluster before you start, and verify the result on a sample.
OSD media and host CPU headroom
Confirm that the OSD media and host CPU can absorb compression work at peak load, not just at idle. Compression runs in the write path, so CPU saturation shows up as higher client latency. Check whether your OSDs are on HDD with SSD-backed WAL and DB, or on flash, and measure CPU use per host during recovery and scrubbing, which also consume CPU.
Support status of your release
Confirm that the Ceph release you plan to run is still actively maintained before you build compression into long-term operations. Maintained releases receive periodic backports and security fixes, and a release reaching end of life changes your upgrade timing. Check the active releases list for the current status and plan any compression tuning around your next upgrade window.
Re-benchmarking after changes
Confirm that you will re-run your benchmarks after changes to OSD versions, kernel versions, drive firmware, network settings or the compression configuration itself. A baseline recorded before compression does not show how the cluster behaves after a kernel update or a drive replacement. Store results from each run so you can compare latency and throughput against the same workload.
How DeployOpen scopes and delivers compression changes for Ceph
DeployOpen treats BlueStore compression as a change to a running storage platform, not a single setting. We start from the pools, workloads and data types you hold, agree what changes in writing, and deliver the work inside your own infrastructure with measurements you can repeat after handover.
How we scope
- The written scope lists which pools are candidates, which are excluded, and why each decision was made.
- Scope records the current compression mode, algorithm, required ratio and blob settings on every pool in question.
- Engagements are fixed-scope, with deliverables and timelines agreed in writing before any pool setting changes.
- Ongoing support, retesting and tuning after the change are available as a retainer if your team wants them.
How we build
- We baseline capacity, CPU load and client latency on the affected OSD hosts before enabling compression on any pool.
- We test the chosen algorithm and mode on a non-production pool that holds representative data, then compare results.
- Changes are rolled out one pool at a time, with rollback steps written and rehearsed before production work starts.
- We check that compressible and incompressible data are handled as expected, using measurements from your own cluster.
How we hand over
- You receive a record of every pool setting changed, with the values before and after each change.
- Documentation explains which data types suit compression in your environment and which were deliberately excluded.
- Benchmark results are delivered in a form your team can rerun after future hardware or release changes.
- Your team gets a walkthrough of monitoring signals to watch for CPU pressure and falling compression ratios.
How we support
- Retainers can cover periodic review of compression ratios, CPU headroom and pool usage as your data grows.
- Changes needed after a Ceph upgrade are scoped and tested before they reach production pools.
- Support includes reviewing new workloads before they are placed on compressed pools.
- Retesting after hardware changes is handled as a defined piece of work, priced from a written scope.
