Ceph erasure coding design

Why Ceph erasure coding design decides capacity and failure tolerance

If you are planning raw capacity for Ceph, the first decision is whether a replicated pool or an erasure-coded pool fits the data you will store. Replication keeps complete copies of each object. Erasure coding splits each object into k data chunks and computes m coding chunks, then spreads all k+m chunks across different failure domains. Any k of those chunks are enough to rebuild the object, so the pool keeps serving data through up to m simultaneous chunk losses within the same failure domain rules you configure. The ratio of k to m sets both how much usable capacity you keep and how many losses the pool survives, so these two numbers belong in the design review before any hardware is ordered.

Teams usually reach for erasure coding when they hold large volumes of data that can tolerate somewhat slower writes and recovery, such as backup targets, archives and object stores, and when raw capacity cost is the main constraint. Replicated pools remain the common choice for latency-sensitive block workloads, such as virtual machine disks, because writes are simpler and recovery is typically less expensive per object. The right answer depends on your workload mix, so ask for a written comparison of usable capacity, tolerated losses and expected write behavior for each pool you plan to create.

Before you commit to a profile, confirm with your team and against the Ceph documentation for your release how the chosen k and m interact with your host count, the failure domain you intend to use, and the device classes you have available. A profile that tolerates the loss of one host is not the same as one that tolerates the loss of a rack or a site, and a profile that needs more failure domains than you can supply will not place data as planned. Treat this as a checkpoint: write down the failure scenarios you must survive, then test that the layout meets them before data arrives.

Choosing k, m, failure domain and overwrites for each pool

This table compares the layout choices each Ceph pool needs, from replicated versus erasure-coded layout to the k and m profile, failure domain, device class, overwrite support and compression. Read each row as a question to settle before the pool is created, because most of these settings are hard to change later without migrating data.

Decision	When it fits	Cost or trade-off to weigh	Confirm before you build
Replicated pool	Latency-sensitive block volumes, small metadata pools, and workloads where write speed matters more than raw capacity.	Raw capacity overhead equals the replica count, so usable space is lower than with erasure coding for the same data.	Replica count and minimum size for your release and failure domain.
Erasure-coded pool	Large, mostly cold or sequential data such as backups, archives and object storage where capacity efficiency is the priority.	Lower raw overhead than replication, but reads and writes touch more OSDs and recovery reads from k chunks, which lengthens rebuild time.	The k and m values, the overhead they imply, and whether your workload tolerates the extra I/O.
Profile k and m	Higher k lowers overhead but spreads each object across more OSDs. Higher m tolerates more simultaneous chunk losses.	A wide profile needs at least k plus m separate failure domains, so small clusters may not be able to place it.	The exact profile syntax, default plugin and minimum host count in your Ceph release.
CRUSH failure domain	Host is the usual minimum. Rack or room domains fit clusters that must survive losing a whole rack or site component.	A broader domain protects more but needs more hosts or racks with free capacity, and it can block placement entirely.	How the CRUSH rule is written for the domain you choose and whether your topology has enough buckets.
Device class	Separate HDD, SSD and NVMe classes keep erasure-coded data on the media you intend and stop it landing on the wrong tier.	Mixing classes in one pool can make placement uneven and complicate capacity forecasting.	Device class names on your OSDs and the rule that targets them.
Overwrite support for RBD	Block images on erasure-coded data pools, where in-place writes are needed for virtual disks and databases.	Overwrites add write cost and recovery work compared with append-only use, and they must be enabled per pool.	That overwrites are enabled on the pool, the data pool split from RBD metadata, and the release notes for your version.
Overwrite support for CephFS	File data on erasure-coded pools for CephFS, usually for capacity-heavy shares rather than hot metadata.	Overwrite-heavy access patterns cost more than append-style writes, so measure client behavior before committing.	How CephFS data pools are attached and the overwrite setting your release requires.
Compression	Compressible data such as logs, text or some backup formats, where capacity gain is worth the CPU cost.	Compression uses CPU on the OSD hosts and gains nothing on already compressed or encrypted data.	Compression mode, algorithm options and minimum ratio settings in your release, plus a benchmark on sample data.
FastEC changes in Tentacle	Teams considering Tentacle for erasure-coded performance improvements on their workload.	Benefits and behavior differences must be verified against your workload, and not every pool may use the new path.	Release notes for Tentacle and the FastEC settings that apply to your pool type before you plan a design around them.

How overwrites for RBD and CephFS change the pool choice

Erasure-coded pools were historically a good fit for workloads that write whole objects or mostly append data, such as backup targets and archives. Block images and file data are different. RBD and CephFS issue partial overwrites to existing objects as guests and clients update blocks and file ranges, and an erasure-coded pool can only accept those overwrites when the feature is explicitly enabled for that pool. Before you commit a capacity plan to erasure coding, decide which of your workloads write in place, because that decision determines whether an erasure-coded data pool is usable for them at all.

Confirm the details in the release notes for the exact Ceph version you will run. Check which overwrite modes are supported for RBD images and for CephFS data pools on that release, whether the setting must be applied at pool creation or can be changed later, and whether any client or kernel requirements apply. Also confirm which pools must remain replicated. CephFS metadata and the pools RBD uses for image metadata are typically kept on replicated storage, and the RBD and CephFS documentation for your release should list the exact pool layout they expect. Write that list into your design, then test a representative VM disk or home directory workload on a lab pool before you rely on it.

Treat overwrite support as a gate in the design review rather than a tuning detail. Record, for each pool, the client type, the expected write pattern and the replication or erasure-coding profile chosen, so the pool choice can be audited later. If a workload cannot use erasure coding for its write pattern, a replicated pool for that tier is the simpler answer, and the capacity model should reflect that split rather than assume one profile across the cluster.

Four factors that decide whether erasure coding fits your cluster

Erasure coding trades capacity for recovery behavior, so the choice of k and m only holds up when four conditions are checked together. Each factor constrains the others, and a layout that looks efficient on paper can fail the first time a host or rack is lost.

Failure domains your hardware can separate

A pool's k and m only protect data if CRUSH can place each chunk in a different failure domain, such as a host, rack or site. Count the hosts, racks and power or network feeds you can truly separate, then check that the chunk count fits within them. If you have fewer independent domains than chunks, a single failure can take out more than one chunk of the same object. Confirm your CRUSH map and the failure domain you intend to use before choosing any profile.

Write pattern of the workload

Erasure-coded pools handle small random overwrites with more work than replicated pools, because a partial write touches several chunks and parity must be recalculated. Identify whether the workload is mostly large sequential writes, such as backups or archives, or frequent small updates, such as virtual machine disks. Overwrite support for RBD and CephFS on erasure-coded pools depends on the release, so confirm the behavior for your client and version before you commit.

Recovery cost after an OSD or host fails

Recovery reads surviving chunks from many OSDs and rewrites the missing ones, which consumes network, CPU and disk capacity while clients keep working. Wider profiles spread data more thinly and can increase the number of reads needed to rebuild a single chunk. Model how long a rebuild would take on your network and drive types, and decide how much client performance you will accept during it. Measure this in a test cluster rather than relying on assumptions.

Release line in use

Erasure coding behavior changes between Ceph releases, and the release you run decides which features and fixes you can rely on. Squid (19.2.x) first shipped September 26, 2024, and Tentacle (20.2.x) first shipped November 18, 2025, with the FastEC changes arriving in Tentacle. Check the maintenance window for your line on the releases page, and plan your profile around the release you will run for the life of the cluster.

Erasure coding fit by workload: backup, VM disks, file data and object storage

Backup and archive pools are usually written once, read rarely and can tolerate slower restores, which makes them the most common candidate for erasure coding. The planning question is less about peak throughput and more about how much data sits behind each restore point and how long a full recovery may take.

Pool layout: Ask whether the backup software writes large objects in sequential chunks. Full-object writes suit erasure-coded pools better than many small updates.
Overwrites: Confirm whether the backup product ever rewrites existing objects in place. If it only creates new objects, overwrite support matters less.
Restore time: Ask how long a full restore of the largest data set must take, and whether degraded reads during a drive failure still fit that window.
Immutability: If the targets use object lock, confirm the retention workflow works on the erasure-coded pool.
Metadata: Catalog and index data that the backup tool queries often should stay on replicated pools.

Check the backup vendor's documented storage requirements, then test a restore from a degraded pool before committing to a layout.

How to set up an erasure-coded pool with checks at each stage

This sequence takes a Ceph erasure-coded pool from a checked failure-domain plan to a verified pool attached to a client service. Each stage ends with a check, so a problem surfaces before data is written.

Confirm host count and failure domains
Before any profile is created, count the hosts, racks or other failure domains that will hold chunks and compare that count with the k plus m chunk total you intend to use. Your team or the partner confirms that enough independent domains exist to place every chunk on a different one, and that the cluster has enough raw capacity for the overhead. The step is done when the planned profile fits the topology on paper and the CRUSH map shows the expected buckets with ceph osd tree.
Create the erasure-code profile
Define the profile with the chosen k, m and failure domain, for example with ceph osd erasure-code-profile set, and confirm the exact syntax for your running release in the Ceph documentation. Profile values such as the plugin and technique are fixed for the pool once it is created, so review them before committing. The check is ceph osd erasure-code-profile get on the profile name, confirming that k, m, crush-failure-domain and the plugin match the design.
Create the pool and its CRUSH rule
Create the erasure-coded pool using the profile, which generates a CRUSH rule that places each chunk according to the failure domain. Set placement group counts according to your sizing plan, and confirm the commands for your release. Your team or the partner then inspects the rule to confirm it selects distinct domains for every chunk. The step closes when ceph osd pool ls detail shows the erasure profile and rule, and the rule dump lists the expected number of chunks per domain.
Enable overwrites where the workload needs them
Erasure-coded pools do not accept in-place overwrites by default, so RBD and CephFS data pools need overwrites enabled explicitly, typically with ceph osd pool set and the allow_ec_overwrites flag. Confirm the flag and its effect on your release before turning it on, since it changes how writes are handled. The check is ceph osd pool get on the flag returning the expected value. Pools that only serve whole-object writes, such as many RGW workloads, may not need this step.
Attach the pool to the client service
Attach the pool to RBD, CephFS or RGW according to the service design. For RBD, the image metadata usually lives in a replicated pool while data sits in the erasure-coded pool, so create the image with the data pool specified. For CephFS, add the erasure-coded pool as an additional data pool and assign it to directories through layouts. For RGW, map the placement target to the pool in the zone configuration. Confirm each command against the release documentation, then check that the client can write and read a test object before real data arrives.
Verify placement and cluster health
Write a test dataset, then confirm placement with ceph pg ls-by-pool and by checking the acting OSD sets, which should contain chunks on distinct failure domains. Run ceph health detail and confirm that the cluster reports no unexpected warnings, such as undersized placement groups, pool application warnings or PG autoscaler messages. Your team records the output as the baseline for later comparison. Note that cephadm manages the daemons throughout, so service changes should go through its orchestrator and the commands should be confirmed for the running release.

Security and operational risks that erasure coding does not remove

Erasure coding changes how data survives drive loss, but it does not secure traffic, place chunks correctly or prove that data can be restored. The cards below cover the risks that remain after the pool profile is chosen, and each one connects to the hardware and network choices in Ceph's hardware recommendations.

Encryption in transit must be set

Erasure-coded pools still send client and OSD traffic over the network, and msgr2 secure mode is not enabled by default in every deployment. Confirm that secure mode is configured for the cluster and that clients support it before go-live. Check that the encryption in transit setting matches on every host, because a mismatch leaves some paths unprotected. Review this alongside the bonded links recommended for the cluster network, since a degraded link can push operators toward insecure fallback paths.

Chunk placement in one failure domain

An erasure-coded profile only protects data if its chunks land in separate failure domains. A CRUSH rule that spreads chunks across OSDs on the same host, rack or switch can lose more chunks than the profile tolerates in a single event. Review the CRUSH map and the rule used by each pool before loading data. Compare the planned host, rack and power layout with the failure domain named in the profile, and confirm it again after any hardware addition.

Shared SSDs for WAL and DB

The hardware recommendations advise a dedicated drive per OSD and moving WAL and DB onto shared SSDs for HDD-based OSDs. Placing many OSDs on one SSD means losing that SSD can take several OSDs down at once, which can exceed the chunk loss the profile tolerates. Plan how many OSDs share each SSD and what a single SSD failure would do to the pool. Ask the hardware vendor for the OSD-to-SSD ratio they support and the replacement lead time.

Disk and network sizing under recovery

Erasure-coded recovery reads surviving chunks from many OSDs and writes the rebuilt chunks, which adds heavy load to disks and network links. A cluster sized only for normal client traffic can see slow requests and long rebuild windows after a single drive failure. Size the cluster network and disks for recovery traffic, not only steady state. The hardware guidance also recommends bonding network links across two switches, so one switch failure does not remove the recovery path.

Tested restores and degraded PG monitoring

An erasure-coded pool that reports healthy is not the same as a pool that can be restored. Backups must be restored into a test environment on a schedule, and the restore time measured against the recovery objectives the business has agreed. Monitoring should alert on degraded and undersized placement groups and on scrub errors, not only on full disks. Confirm that alerting reaches the on-call team and that a runbook exists for each state before production use.

What recovery costs and what the FastEC changes in Tentacle affect

Recovery is where erasure coding shows its cost. When an OSD or host fails, Ceph cannot copy the lost data from a single surviving replica. It reads enough surviving chunks from the other OSDs in the placement group to reconstruct each missing chunk, then writes the rebuilt chunks to new locations. That read fan-out and rebuild write load crosses the network and hits the disks of every OSD involved, and it competes with client I/O while it runs. A wider profile such as a higher k spreads the work across more OSDs, and a degraded pool keeps serving reads only while enough chunks remain. Plan for this load in the design, not after the first failure.

The Ceph hardware recommendations advise sizing memory for peak use rather than calm periods, and they tell you to benchmark before buying because no two clusters are alike. Apply that to recovery. Ask your hardware and integration partner to model recovery time and client latency for your chosen k and m, your failure domain and your network, and to measure it on hardware that matches your production nodes. Recovery throttling settings and OSD memory limits decide how hard recovery pushes against client traffic, so record the values you will start from and who may change them.

Tentacle, the current release line, includes work on erasure coding under the name FastEC. Do not assume what that work changes for your pools. Read the Tentacle release notes on the Ceph active releases page and confirm which erasure coding behavior changed, whether it affects the write path, reads, recovery or overwrites, and whether it applies to the pool type, release and profile you intend to run. If your cluster runs Squid or an earlier release, confirm the same points for that release before you rely on any behavior in this section.

Limits to confirm before you commit to a k and m

A k and m choice is only as good as the environment it runs in. The checks below cover release support, overwrite modes, failure domains, stretch mode and device classes, and client compatibility. Each one has an owner who can confirm it, so the decision is verified before data lands on the pool.

Release line and support window

Confirm that your release line will still receive fixes for the life of the hardware. Per the Ceph releases page, Squid has an estimated end of life of October 31, 2026, and Tentacle has an estimated end of life of June 1, 2027. A pool designed around a release that ages out during your planned life needs an upgrade plan before go-live. Ask your Ceph support provider or Ceph maintainers to confirm the current estimates and the upgrade path to the next release line.

Overwrite mode for your clients

Partial overwrites on erasure-coded pools are what let RBD and CephFS use them for live data. Confirm that your release supports the overwrite behavior your clients need, and whether the Tentacle FastEC changes apply to your workload. Check this with your storage architect and the vendor or project documentation for each client, because a pool that accepts writes for one client may not suit another.

Failure domains your CRUSH map can supply

A k+m profile needs at least k+m distinct failure domains, usually hosts, to place each chunk safely. Count the hosts or racks in your CRUSH map that are available on day one, not only the target count. Confirm the number with the team that owns the cluster map, and test the rule with a crushtool simulation before you create the pool. Too few domains either blocks placement or forces chunks onto the same failure domain.

Stretch mode and device class limits

Stretch mode and device classes both constrain which erasure-coded profiles you can run. Confirm with your Ceph architect whether your planned two-site design supports the erasure-coded layout you want, and whether a device class rule can place chunks on the media tier you intend. Verify both in a lab cluster, since a rule that looks valid on paper can still leave a pool unable to reach full placement after a site loss.

Client and driver compatibility

Each client path has its own requirements. Confirm that the kernel, librbd, Ceph-CSI, hypervisor or CephFS client in your environment supports writing to an erasure-coded data pool. Ask the platform owner to test a pool at your planned k and m with the exact client versions you run, including any virtualization or Kubernetes layer. A client that falls back to a replicated metadata pool changes the capacity math.

Recovery behavior under your hardware

A wider profile increases the number of shards read during rebuild, so recovery load depends on your drives, network and recovery throttles. Ask your performance engineer to confirm recovery throughput on your hardware and the settings that protect client I/O during a rebuild. Treat the result as a measured value from your own cluster, not a figure from a vendor sheet.

Questions buyers ask about erasure coding design before hiring help

What drives the cost of designing an erasure-coded layout for Ceph?

The cost follows the amount of analysis the layout needs, not the length of the Ceph documentation. The main drivers are how many pool types you need (for example, RBD, CephFS data pools and RGW buckets), how many failure domains you must model (hosts, racks, rooms or sites), how many media classes and OSD counts are in play, and whether the design must hold up under a site or rack loss with a stated recovery target. Each added workload profile multiplies the testing. One-time work covers the design document, the pool and CRUSH rule definitions, and the pilot test plan. Recurring work appears later, when you add nodes, change k and m, or revisit the layout after a Ceph release changes erasure coding behavior. Ask any provider to price these as separate line items with their assumptions written down.

How long does an erasure coding design and pilot usually take?

Duration depends on how much of the inputs already exist and how many pilots you need. A design for a single cluster with a known hardware list and one workload is shorter than a design that must cover several pools, a second site, or a recovery target you have not measured. Pilot time is driven by hardware availability, the time needed to fill a test pool to a realistic size, and the time needed to simulate a failed host or OSD and measure recovery. The Ceph hardware recommendations advise benchmarking before buying because no two clusters behave alike, so a pilot is worth scheduling early. Ask a provider for the milestones, the inputs they need before day one, and which items depend on your staff or your procurement cycle.

What does the design work need from our team?

The design needs accurate records of what you already own and how it is connected. Expect to provide a hardware inventory with drive models, drive sizes, controller types, memory per node and the number of OSDs planned per host. Network diagrams should show switch layout, link bonding across switches, and any separation between client and cluster traffic. Workload profiles matter as much as hardware: block, file and object pools have different write patterns, and erasure coding reacts differently to small random writes. You also need to name the failure domains that matter to your business, such as separate racks, power feeds or buildings, and state your tolerance for reduced redundancy during recovery. Missing or outdated inputs are the most common reason design schedules slip.

Can our internal team design an erasure-coded layout without outside help?

Yes, a team with storage and Ceph operations experience can design a layout internally, and the Ceph documentation covers the core concepts. The questions to answer honestly are whether someone has run a recovery under load, whether that person can test failure scenarios without risking production data, and whether the design will be reviewed by someone who did not write it. Erasure coding decisions interact with pool type, CRUSH rules, placement group counts, device classes and the workloads that will use each pool, so a single mistake can be expensive to correct after data is written. Many teams keep the design in-house and use an outside review or a time-boxed pilot as a second check. Decide which parts you own before you start.

How do we compare vendor quotes for an erasure coding engagement?

Compare the written scope first, because two quotes for the same title can cover very different work. Check whether each quote names the Ceph release it targets, the number of pools and failure domains it covers, whether a pilot with failure testing is included, and what the deliverables are: a design document, configuration files, a test report and a runbook for recovery. Ask who performs the work, how changes to scope are handled and what the client must provide before the work starts. Note whether the price is fixed for the agreed scope or estimated, and what happens if the hardware inventory is different from what was described. DeployOpen works from a written scope and provides a quote from that scope, so you can use the same checklist when reviewing any proposal.

Does the erasure coding design change when we upgrade Ceph?

It can. Erasure coding behavior, including overwrite support and recovery handling, has changed across Ceph releases, and the active releases are Tentacle (20.2.x) and Squid (19.2.x). Before you finalize k and m, confirm the release you will run and read its release notes for erasure coded pool features relevant to your workloads, including RBD and CephFS overwrites. A design written for one release should be reviewed before a major upgrade, and a pilot on the target release is the most reliable check. Also confirm the end-of-life dates for your chosen release, since they affect how long a layout can stay in service without a planned move.

What is the recovery performance cost of a wide erasure coded profile?

Wider profiles store data more efficiently but recover differently. When an OSD or host fails, Ceph must read surviving chunks from several places and rebuild the missing ones, which consumes network, CPU and disk capacity that client I/O also needs. Larger k values spread each object across more OSDs, so recovery involves more peers and more reads per rebuilt chunk. The cost depends on your network speed, drive type, memory, and how many concurrent recoveries your settings allow. Ask for a recovery test on your hardware that measures client latency and recovery time together, rather than relying on a general statement about erasure coding overhead.

How DeployOpen scopes and delivers Ceph erasure coding design

DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside the client's own infrastructure. For Ceph erasure coding design, we start from your failure domains, capacity targets and workload mix, then turn those into a written, fixed-scope plan that your team can review and run.

How we scope

Inventory host, rack and site counts so k and m are tested against real failure domains before any pool is proposed.
Document which workloads need RBD overwrites or CephFS data pools, and which can stay on replicated pools.
Agree in writing on the recovery impact to accept, including how client I/O is protected during rebuilds.

How we build

Create test pools with the agreed profile and confirm CRUSH rules place chunks in separate failure domains as designed.
Benchmark degraded reads, writes and recovery time on your hardware, then adjust pool settings before production use.
Verify overwrite and metadata settings for each RBD and CephFS consumer, and record the results in the build log.

How we hand over

Deliver a design record covering profile choice, CRUSH rules, failure domain assumptions and the capacity model.
Provide runbooks for drive failure, host loss and rebalancing, with the commands your operators will use.
Walk your storage and operations staff through the design and confirm they can explain each trade-off back to us.

How we support

Review pool utilization and recovery behavior after each Ceph upgrade or hardware change to confirm the design still holds.
Plan capacity additions against the existing profile so new hosts keep failure domain counts intact.
Ongoing support, upgrades and continuous testing are available as a retainer, scoped in writing alongside the original engagement.