Ceph CRUSH rules and pool customization services

What Ceph customization covers and when teams need it

Ceph customization is the layer that sits between a cluster that installs and passes health checks and a cluster shaped to your hardware and workloads. It covers the CRUSH map hierarchy that describes hosts, racks and rooms, the CRUSH rules that decide where each copy or erasure-coded chunk lands, device classes that separate HDD, SSD and NVMe OSDs, the choice between replicated and erasure-coded pools for each use, placement group sizing per pool, and RGW storage classes that map object tiers to media and cost.

Most teams start this work when something in the defaults stops matching reality. Typical triggers are mixed HDD and SSD nodes sharing one default rule, a new workload such as backups or analytics that needs different durability or throughput, a failure domain that does not reflect how racks or power rooms are actually laid out, a pool that performs poorly under load, or object data that needs lifecycle tiers to control cost. Each trigger has a specific fix, and each fix changes where data lands, so it should be decided before production data is written.

Many teams with strong Linux and storage skills do this in-house, and that is a reasonable choice. A partner usually earns its place in two cases: when the CRUSH design must be validated before data lands, and when the team has not changed CRUSH on a production cluster before. Ask for a written review of the CRUSH hierarchy, rule set, pool parameters and a test plan that includes a simulated host or rack failure with recovery timing measured on your hardware.

What the work includes and what you receive

Each work package ends in a named artifact that your team can review, apply and operate. The scope, including the Ceph release your cluster runs or plans to run, is agreed in writing before work starts.

Current-state review of CRUSH and pools

We export and read your existing CRUSH map, pool definitions, device classes and rule assignments to find what has drifted from intent. The output is a findings report listing each rule, pool and placement issue, ranked by risk, with the evidence for each finding. Ask for the review to cover the exact release named in the scope document.

Failure domain and device class design

We define the CRUSH hierarchy (racks, rows, hosts, OSDs) that matches your power, network and switch boundaries, and assign device classes such as hdd, ssd or nvme so that workloads land on the right media. The deliverable is a reviewed CRUSH map file with a short rationale for each bucket and class. Confirm how your hardware groups map to physical failure domains before approval.

Rule and pool catalogue per workload

For each workload, we choose replicated or erasure-coded protection, set the CRUSH rule, failure domain and minimum size, and document the trade-off between usable capacity and fault tolerance. The deliverable is a rule and pool catalogue that lists every pool, its purpose, owner and settings. Ask who approves changes to the catalogue after handover.

Placement group planning

We size placement group counts per pool from the expected OSD count, replication or erasure-coding width and growth plan, and set the autoscaler policy where it fits your change process. The output is a placement group plan with the target count per pool and the conditions that would trigger a review. Check how the plan handles expansion, since adding OSDs changes the balance.

RGW storage class and placement target design

For Ceph Object Gateway, we map storage classes and placement targets to the pools and device classes they use, so that hot and cold object data land on the intended media. The deliverable is a storage class and placement target specification that the RGW configuration can be checked against. Confirm which RGW features your applications depend on, such as lifecycle transitions, before the design is fixed.

Change plan for controlled data movement

Applying a new CRUSH map or pool setting can trigger large backfill, so we sequence changes with throttles, maintenance windows and rollback points to limit uncontrolled data movement. The deliverable is a written change plan with pre-checks, step order, expected movement, abort criteria and a rollback procedure. Ask for the plan to be rehearsed on a non-production cluster where one exists.

Test and validation evidence

We test the new rules and pools on a lab or staging cluster by simulating device and host failures, checking that data placement and recovery follow the intended failure domains. The validation report records each test, its result and the cluster state captured before and after. Ask whether your environment can host a test cluster and what hardware it would use.

Runbooks and handover

We deliver operator runbooks for common tasks such as adding OSDs, changing device classes, creating pools and responding to health warnings, with a handover session for the engineers who will run the cluster. The deliverables are the runbook set and a training record of who attended. Confirm which runbooks your team needs before the scope is signed.

What drives cost and duration

This table lists the factors that change how much design, testing and change control a Ceph customization needs, with how to keep each one contained.

Driver	Why it matters	How to keep it down
Device types and generations	Each distinct device class (HDD, SATA SSD, NVMe SSD, mixed generations) needs its own device class, rule and pool design, and mixed hardware makes OSD performance uneven.	Inventory every drive model and generation before design starts, and group them into as few device classes as the hardware allows.
Hosts and failure domains	Every added rack, row, room or site adds levels to the CRUSH hierarchy and more ways a rule can fail to place data, so each layout must be tested against real topology.	Fix the failure domains the business needs (host, rack, room or site) first, and avoid hierarchy levels that exist only for labeling.
Distinct workloads needing separate rules or pools	Block, file, object, backup and archive data often need different replication or erasure coding, failure domains and placement groups, and each separate intent adds design and verification work.	Consolidate workloads that share durability and performance needs into one pool, and document the intent behind each pool before creating it.
Production status	Changing CRUSH rules or pool settings on a live cluster moves data, so changes need a migration window, throttled recovery and a tested rollback plan.	Make all rule and pool changes in a test cluster first, and ask for a written change plan with rollback steps before any production change.
RGW storage classes and placement targets	Each storage class or placement target maps to a pool and possibly a device class, and lifecycle transitions between them require configuration and validation.	Limit storage classes to the tiers that applications actually request, and record which bucket policies depend on each one.
Compliance, retention and location constraints	Regulatory or contractual rules can dictate durability level, copy location, encryption and deletion behavior, which constrains rule and pool choices and adds evidence work.	Collect the retention and location requirements in writing before design, and confirm with your assessor which records they expect to see.
Test environment availability	Without a test cluster that resembles production hardware and topology, CRUSH and pool changes can only be checked on paper, which raises the risk of rework.	Confirm whether a test cluster exists or can be provisioned, and agree on which topology and device classes it must reproduce.
Release and deployment variants	Running different Ceph releases, or a mix of cephadm and Rook environments, means rule syntax, tooling and upgrade behavior must be checked for each variant.	Standardize on one maintained release and one deployment method across clusters, and record any exceptions with a planned remediation.

Project phases and what each phase produces

The sequence moves a Ceph customization from a documented baseline through lab proof and staged change to a production layout your team can operate.

Inventory hardware, workloads and current map
We collect the release version, host and device inventory, network layout, existing CRUSH hierarchy, device classes, pool settings and placement group counts, and the workloads that will use each pool. You receive a written baseline showing what the cluster looks like today and which settings differ from your targets. The step closes when you have reviewed and signed off that baseline.
Design failure domains, device classes and rules
We map failure domains such as host, rack or room to your physical layout, assign device classes to drives, and decide per workload whether replicated or erasure-coded profiles fit. Placement group targets are set from pool size and OSD count. The output is a placement design document with one rule per workload, the reasoning for each, and open questions for your team to confirm.
Prove the design in a lab cluster
We test the proposed CRUSH map offline with crushtool-style simulation to check that each rule maps to the expected OSDs and failure domains, then run benchmark workloads on a lab or staging cluster built to match production hardware where possible. The output is a test report with simulation results, benchmark numbers from your environment, and any rule changes made as a result.
Plan the change order and rollback
We write the order in which map changes, pool changes and device class assignments will be applied, estimate expected data movement from the simulation, and set recovery and backfill throttling. The plan also covers the maintenance window, health checks, and a saved copy of the current CRUSH map and pool settings for rollback. Your team approves the written plan before any production change.
Apply changes to production in stages
We apply each change set separately, checking cluster health, data movement progress and client I/O after every step before moving on. If a check fails, the step is reverted using the saved map. The output is a change log recording each command, the health state before and after, and the final state of every pool and rule.
Configure RGW storage classes and placement targets
Where the workloads use the Object Gateway, we define placement targets and storage classes that point to the pools chosen in the design, set default placement for users and buckets, and test writes and reads with S3 clients from the applications that will use them. The step closes when client-side tests show objects landing in the intended pools with the expected storage class.
Hand over runbooks and monitoring checks
We deliver runbooks for adding a device class, creating a pool, changing a rule, and rolling back a map change, along with monitoring checks for placement group states, pool usage per class, and data movement. Your operators walk through the runbooks with us. Handover is complete when your team has run one supervised change using the runbook and the monitoring checks are live in your alerting system.

What teams keep in-house and what they outsource

This table shows who usually owns each part of Ceph CRUSH, pool and storage class customization and which skills each part needs, so you can split decisions and work between your team and a partner.

Work part	Usual owner	Skills needed
Hardware inventory and procurement input	In-house storage and infrastructure team	Drive, HBA, network and server inventory, plus knowledge of supplier lead times and device classes
CRUSH hierarchy and failure domain design	Shared: partner drafts, in-house team approves	Deep Ceph CRUSH map knowledge, rack and power layout, understanding of failure domains such as host, rack and row
Device class assignment and verification	In-house storage engineers, partner reviews	Ceph CLI for device classes, drive type identification, checks that each class maps to the intended OSDs
Pool and erasure-code profile selection against workload needs	Business owner sets targets, partner recommends profiles	Replicated versus erasure-coded tradeoffs, workload profiling for block, file and object use cases
PG sizing and autoscaler policy decisions	Partner designs, in-house team operates	PG math, autoscaler modes, understanding of per-OSD PG limits and rebalance impact
RGW storage class and placement target design	Partner designs with application owners, in-house team runs	RGW zonegroup and placement target configuration, lifecycle rules, data tiering and application access patterns
Simulation and benchmark testing	Partner leads, in-house team witnesses	Test lab setup, benchmarking tools, interpretation of latency and recovery results against hardware recommendations
Change execution in production	Partner runs staged changes with in-house change control	Staged rule rollout, data movement monitoring, rollback planning and maintenance window coordination
Ongoing rule governance and review after new hardware arrives	In-house team owns, partner reviews on request	Change records for CRUSH and pool settings, periodic map review, capacity forecasting after hardware additions

What usually goes wrong and how to prevent it

Most CRUSH and pool problems appear after a change, not during the first build. The patterns below are the ones teams most often meet, with the check that prevents each one.

Map changes trigger unplanned data movement

Editing the CRUSH map, moving hosts between buckets or changing a rule can remap large numbers of placement groups at once, which drives backfill traffic and client latency. Simulate the change against a copy of the current map before applying it, review the expected mapping delta, and stage it so backfill runs under throttled recovery settings.

Failure domain does not match physical risk

If the failure domain is set to host when racks share power or top-of-rack switches, one rack loss can take out more replicas than the pool was designed to survive. Compare each rule's failure domain with the actual power, network and physical layout, and confirm the hierarchy in the CRUSH tree matches the data center before pools are created.

Mixed device classes place data on wrong media

When HDD, SSD and NVMe OSDs share a host or root without device class constraints, rules can place data on media that does not match the workload's latency or cost target. Assign device classes explicitly, confirm each OSD reports the class you expect, and bind every rule to a class so placement is predictable.

Placement group counts misfit the OSD count

Pools with too few placement groups concentrate data and load on a small set of OSDs, while too many increase peering and memory overhead across the cluster. Size PG counts from the OSD count and the expected pool share, verify the autoscaler behavior you have chosen, and benchmark before the pool takes production data.

Erasure-coded profiles exceed host count

An erasure-coded profile needs enough independent failure domains to place every data and coding chunk. If the host count is too low for the chosen k and m with a host failure domain, placement stays degraded or cannot complete. Check the minimum host or rack count for the intended failure domain before choosing a profile, and confirm it in a test pool.

Device class changes missed by existing rules

Re-classifying OSDs or adding new drives with a different class does not update rules that were written against the old class. Data can stop matching the intended media, or new OSDs can receive no data. After any device class change, list every rule that references the class and confirm its placement matches the design.

RGW storage classes point at mismatched pools

A Ceph Object Gateway storage class that references a pool which does not exist, or a pool with weaker replication or erasure coding than the class promises, gives users a durability level the cluster does not provide. Verify each storage class maps to an existing pool, and document the durability of that pool beside the class name.

Performance expectations set without benchmarking

Teams that size pools and rules from vendor figures or spec sheets often find latency and throughput differ under their own workload and network. Run benchmarks on each pool type and device class with a representative block, file or object profile, record the results, and use them as the baseline for later changes.

No documented rollback for map changes

Without a saved copy of the previous CRUSH map and pool settings, reverting a bad change means reconstructing it under pressure. Export and store the current map and pool configuration before every change, record the exact rollback commands, and rehearse the reversal in a non-production cluster.

How the work differs by starting point

This starting point gives the most room to design. The CRUSH hierarchy, device classes, rules and pool layout can be settled before any data is written, so changes later are rare.

Failure domains first: Agree on host, rack and room boundaries and confirm which of them your hardware and power layout actually support.
Pools before data: Define replicated and erasure-coded pools, their rules and placement group counts before the first pool receives data.
Benchmark before buying: Ask whether the hardware plan has been tested against the expected workload, as the Ceph hardware recommendations advise.
Confirm the release: Check the maintained release line and its end-of-life date at the time of build.
Document the map: Keep the CRUSH map and rule definitions in version control so later reviews start from a known state.

What you need to provide before work starts

Start by collecting the inputs that decide the CRUSH and pool design. Before any CRUSH rule, pool or device class is drafted, DeployOpen needs cluster access with a named administrator and, if available, a test or staging cluster where rule changes can be trial-run. Provide the output of cluster status, the current CRUSH map (decompiled text form), and the settings of every existing pool, including replica counts, erasure-code profiles, pg_num and autoscaler mode, and the device class each OSD reports. Without these, the team cannot see what is already placed where.

The hardware inventory should list device types, sizes, drive classes (HDD, SATA or NVMe SSD), where WAL and DB devices sit, the network layout with bonded links and the switches they terminate on, and the rack or room placement of each host. Rack and room placement is what turns a host list into failure domains. Also state the Ceph release in use and the planned upgrade path, because CRUSH and device class behavior should be confirmed against that release.

For each workload, provide the performance goal, the capacity estimate, and the durability requirement, along with whether the data is block, file or object. If object storage is in scope, add the RGW usage pattern, the client list, and the storage classes clients expect. Name the change windows, the approvers for CRUSH map and pool changes, the monitoring access and alerting contacts, and one owner who can accept durability and cost trade-offs. Items with gaps in hardware or workload data usually extend the design phase, so DeployOpen confirms each item during discovery before the design is drafted.

Questions buyers ask before hiring outside help

What does a Ceph customization engagement cost, and what drives the quote?

The quote follows from a written scope, and DeployOpen does not publish a price list. The main cost components are design work on the CRUSH hierarchy, rules, pools and storage classes, the number of placement intents you need, the change plan and staged rollout, validation against your workloads, and the handover documentation. Ask the quote to separate one-time design and implementation from any recurring support, and to state which items you own, such as hardware, network changes and client-side testing. The number of storage classes and workload types is usually the largest variable.

How long does a Ceph CRUSH and pool customization project take?

The timeline is set in the written scope, and the main drivers are the number of placement intents, whether the cluster is already in production, and the change windows available to you. A lab or pre-production cluster moves faster than one serving live block, file or object workloads, because each rule change triggers data movement that must be staged. Clusters with many device classes or pools that need placement group resizing take longer to validate. Ask for milestones tied to test results and change windows rather than a single end date.

Can our own storage team customize CRUSH rules and pools without outside help?

Yes, if the team has real Ceph operations skill, a test environment that mirrors production, and a tested rollback plan. The skills that matter most are CRUSH map concepts such as buckets, failure domains and device classes, pool design for replicated versus erasure-coded data, placement group sizing and autoscaler behavior, cephadm operations, and reading cluster health and recovery metrics. Teams without a staging cluster or without someone who can explain the expected data movement before each change usually benefit from a partner review.

What does the work need from our team?

Expect to provide administrative access to the cluster through the agreed channels, a current inventory of hosts, OSDs, device classes and network layout, and workload data such as client types, capacity growth, performance targets and latency sensitivity. Your team must make decisions on failure domain tiers, which workloads get which storage class, and acceptable recovery impact. You also need to book change windows, approve the rollout plan and confirm backups before any change. Missing inventory or unclear workload requirements are the most common causes of rework.

Will changing the CRUSH map cause an outage?

Changing the CRUSH map does not cause an outage by itself, but it can move large amounts of data, which adds load to disks and network links and can slow client I/O. The risk is reduced by staging changes, throttling recovery and backfill, checking cluster health between steps, and changing one placement intent at a time. Plan for the data movement by estimating it in a test cluster first, and set clear stop conditions for health warnings and latency.

Do we need to upgrade Ceph before changing our CRUSH rules and pools?

Confirm the running release before design starts, because rule and pool options depend on features available in that release. The Ceph releases page lists the actively maintained releases, including Squid with an estimated end of life of October 31, 2026 and Tentacle with an estimated end of life of June 1, 2027, and those dates should be checked against the current page. If your cluster is on an older release, the design may need an upgrade first, which is a separate piece of work with its own change plan.

Does DeployOpen certify or endorse Ceph?

No. DeployOpen is a Chainscore Labs company that implements, customizes, hardens and supports self-hosted open-source platforms, and it is not an official partner, reseller or certified provider of Ceph or its vendors. Ceph is an LGPL-licensed open-source project. Any vendor support, certification or distribution terms for your Ceph version should be confirmed directly with the vendor that provides them.

How DeployOpen scopes, delivers and hands over Ceph customization

DeployOpen approaches Ceph customization as a planned change to a cluster you control, starting from your hardware, failure domains and workloads before any CRUSH rule, pool or storage class is written. Each step produces written, reviewable artifacts that your team can own after handover.

How we scope

Engagements are fixed-scope, with the CRUSH hierarchy, rule set, pool list and storage class map agreed in writing before work starts.
The written scope names each device class, failure domain and replicated or erasure-coded pool, along with the workloads each one serves.
We confirm placement group targets, RGW storage class needs and the Ceph release in use so the design matches your environment.

How we build

Changes are staged in a lab or non-production cluster first, so CRUSH edits and pool changes can be checked before they touch production data.
Device classes and CRUSH rules are applied and verified against the hierarchy, and data placement is checked against each failure domain.
Pool settings, placement group counts and RGW storage classes are tested with representative load and recovery scenarios.

How we hand over

You receive the CRUSH map, rule definitions, pool and storage class documentation, and the reasoning behind each choice.
Runbooks cover adding OSDs, changing device classes, rebalancing and recovery steps, written for the engineers who will run the cluster.
Knowledge transfer sessions walk your team through verification commands and the health checks to run after any placement change.

How we support

Ongoing support, upgrades and periodic review of placement design are available as a retainer, scoped in writing.
Reviews after hardware additions or workload changes confirm that rules, pools and storage classes still match the cluster's layout.
Ask any provider to confirm which Ceph releases they cover, how upgrades are planned, and who is responsible for each change.