Why FileStore OSDs must migrate to BlueStore before the Reef upgrade
If your cluster still runs FileStore OSDs, you cannot take it to Reef or any later release until those OSDs are converted to BlueStore. FileStore was deprecated in Reef, and FileStore OSDs cannot run on Reef or later releases, so the conversion is a prerequisite for the upgrade rather than a follow-up task. The practical consequence is that the upgrade plan has two parts that must be sequenced together: each FileStore OSD must be destroyed and rebuilt as a BlueStore OSD, and the cluster must then move through the release upgrade. Teams that treat these as separate projects often discover late that a release they planned to reach is blocked by hardware they had not scheduled for replacement.
Three migration methods are available. Mark-out replacement needs no spare hardware, but the data crosses the network twice because it is moved off each OSD and then back onto its replacement. Whole-host replacement needs a spare host, which keeps the rebuild contained to new machines. Per-OSD copy uses ceph-objectstore-tool, which the migration documentation describes as unsupported tooling, so it carries more risk and needs a clear support position before anyone uses it. Each option changes how much spare capacity, network headroom and maintenance time you need, and the choice also determines how many OSDs can be out at once. Confirm current behavior for your exact release with the Ceph BlueStore migration documentation before you commit.
The rest of this page explains how to choose among these methods, how to keep OSD destruction within a single CRUSH failure domain so that parallel reprovisioning cannot cause data loss, and how to sequence conversion with the release upgrade. Before reading further, inventory every FileStore OSD by host and failure domain, record current cluster health, and confirm that you have a tested recovery path, since each method leaves data in a temporarily reduced state while it is rebuilt.
What changes for FileStore OSDs and the upgrade path
This table sets out what changes for FileStore OSDs as Ceph moves from older releases to Reef and later, and what each change asks you to do before upgrading. Read it top to bottom as a sequence: confirm the release you are on, convert the FileStore OSDs, then move to a maintained release. The main takeaway is that conversion to BlueStore is a precondition for the Reef upgrade, not a step you can defer until afterward.
| Change | Impact | Action needed |
|---|---|---|
| FileStore deprecated in Reef | New FileStore deployments are no longer the recommended path, and existing FileStore OSDs are on a deprecated storage backend. | Inventory every OSD and record which ones use FileStore before planning any upgrade. |
| FileStore OSDs cannot run on Reef or later | A cluster with FileStore OSDs cannot complete the upgrade to Reef or any later release as-is. | Convert each FileStore OSD to BlueStore before the upgrade to Reef begins. |
| BlueStore required before upgrading to Reef | Conversion is a prerequisite that sits in front of the release upgrade, so the two cannot be treated as independent projects. | Sequence conversion ahead of the upgrade and confirm the cluster is healthy before each step. |
| Mark-out replacement | Data for each OSD is rebuilt elsewhere and then crosses the network twice, but no spare hardware is required. | Choose this when you have no spare host and can tolerate the extra data movement and recovery time. |
| Whole-host replacement | A full host is reprovisioned with BlueStore OSDs, which needs a spare host to take the load during the swap. | Confirm you have a spare host and a plan to return the original host to the cluster. |
| Per-OSD copy with ceph-objectstore-tool | Data is copied OSD by OSD with tooling the Ceph documentation describes as unsupported. | Treat this as a last resort and agree the risk, rollback and support position in writing before use. |
| Parallel reprovisioning | Destroying more than one OSD at a time risks data loss if the OSDs share a failure domain. | Destroy OSDs only within a single CRUSH failure domain at a time and check PG health between batches. |
| Squid and Tentacle are the maintained releases | Reef is no longer among the actively maintained releases, so the target of the path should be a maintained release such as Squid or Tentacle. | Confirm the current target on the Ceph active releases page and plan the path to a release that is still supported. |
Choosing mark-out replacement, whole-host replacement or per-OSD copy
Mark-out replacement removes a FileStore OSD from the cluster, lets Ceph recover its data onto the remaining OSDs, then rebuilds the drive as a BlueStore OSD and lets data flow back. It is the method that asks the least of your hardware, because you do not need spare servers or spare drives beyond what the cluster already has. The cost is network and time: the same data crosses the network twice, once out during recovery and once back when the new OSD fills.
- No spare hardware: the conversion runs on the existing hosts and drives, so there is no procurement step before work starts.
- Double data movement: each converted OSD's data is rebalanced away and then back, which adds recovery load on the cluster network.
- Reduced redundancy during recovery: while an OSD is out, its placement groups run with fewer copies until recovery completes.
- Pacing is in your hands: you can convert one OSD or a small batch at a time and wait for the cluster to return to a healthy state between steps.
- Standard tooling: the steps use the normal OSD removal and creation workflow rather than unsupported utilities.
Before you choose this method, confirm with your team how much recovery traffic your network and client workloads can absorb, and check that the failure domain layout can tolerate the OSDs that are out at any moment.
Whole-host replacement brings a new host, already configured with BlueStore OSDs, into the cluster and drains an old FileStore host so its data moves to the new one. You need a spare host or enough temporary capacity to hold a full host's worth of data while the old host is retired. This method keeps the conversion on a per-host basis, which makes the change easy to explain and easy to roll back at the host level, as long as the old host is not decommissioned until the new one is verified.
- Spare host required: you need a server with capacity and network connectivity that matches the cluster's existing host standard.
- Clear unit of change: each host is either fully old or fully new, which simplifies inventory and change records.
- Data movement is one pass: data moves from the old host to the new one rather than out to the cluster and back.
- Hardware standard matters: the spare host should follow current guidance on drives, WAL and DB devices and bonded network links.
- Retirement is a separate step: keep the old host available until health checks pass, then decommission it.
Confirm with your hardware vendor and your Ceph design whether the spare host can be added to the same failure domain as the host it replaces, because that determines how the cluster absorbs the transition.
Per-OSD copy uses ceph-objectstore-tool to move the contents of an individual FileStore OSD into a BlueStore OSD, without removing the OSD from the cluster's data placement first. It avoids the full data reshuffle of mark-out replacement and needs no spare host. However, the Ceph documentation describes this tooling as unsupported, so the risk sits with your team and any support contract you hold for the cluster.
- Unsupported tooling: the procedure is not a supported upgrade path, so confirm your support arrangements and change approval rules before using it.
- Careful stop and start control: each OSD must be stopped while its data is copied, which requires precise sequencing across daemons.
- Lower data movement: data is not rebalanced across the cluster in the same way, which can reduce recovery traffic.
- Higher operator skill requirement: errors in the copy step are harder to detect and recover from than in a standard OSD replacement.
- Limited scale: the method is practical for a small number of OSDs, not for a whole cluster converted in one program.
If you choose this method, test it on a non-production cluster that mirrors your OSD layout, and agree in writing who approves each step before it runs on production data.
Keeping OSD destruction inside a single CRUSH failure domain
When you reprovision FileStore OSDs in parallel, the rule that protects your data is simple: destroy OSDs only within a single CRUSH failure domain at a time. Each OSD you destroy and recreate as a BlueStore OSD temporarily removes a copy or shard of data. If the copies you are removing overlap with the copies you still need, the cluster can lose data that exists nowhere else. The failure domain you pick, usually host or rack, is the boundary that keeps those losses contained.
Start by mapping your physical layout to CRUSH buckets. Run ceph osd tree and confirm that every host, and every rack or chassis you intend to use as a boundary, appears as its own bucket under the expected root. If a rack or chassis is not modeled, add it before the migration begins, and move hosts under it with the appropriate CRUSH commands. Then check your CRUSH rules with ceph osd crush rule dump to confirm which bucket type each pool spreads its copies or shards across. The bucket type in the rule, not the one you intended, determines what the cluster actually protects against.
Before you destroy anything in a domain, confirm that its data is replicated outside it. For replicated pools, check that the pool size and min_size leave at least one copy in another domain once the domain under conversion is empty. For erasure-coded pools, compare the profile's k and m values with the number of failure domains in the rule, because the number of shards you can lose in one domain is limited by how shards are placed. Ask your team to confirm the batch size in writing against those values, and to verify that the cluster returns to HEALTH_OK after each batch before the next one starts. Replication factor and erasure-coding profile both change the safe batch size, so recalculate it whenever either changes.
What to verify before converting the first OSD
Conversion destroys and rebuilds OSDs, so every card below is a gate the cluster must pass before the first FileStore OSD is touched. Work through them in order: recovery and health come first, because a failed conversion on an unhealthy cluster is the hardest to undo.
Verified recent backup or recovery plan
Confirm that the data you cannot regenerate is backed up, and that someone has restored a sample from it recently. Conversion rebuilds data in place, so a backup is your only route back if the cluster ends up in a state you did not expect. Record where the backup lives, who can restore it, and how long a restore takes in your environment. Do not treat an unrestored backup as verified.
Cluster health at HEALTH_OK
Run ceph health detail and confirm the cluster reports HEALTH_OK before any OSD is removed. Clear warnings about scrub errors, degraded or undersized placement groups, and near-full OSDs first, because each one reduces the redundancy you rely on while data moves. A cluster already recovering from a failure is not a safe starting point for conversion.
Inventory of OSDs and backing devices
List every FileStore OSD with its host, CRUSH location, backing data device and journal device. Note which OSDs share a journal SSD, because losing that SSD affects every OSD on it. This inventory tells you which OSDs can be converted together, and it shows whether a journal device is also a candidate for reuse as a shared WAL and DB device.
Dedicated drives and shared WAL and DB SSDs
Check the target layout against the Ceph hardware recommendations: one dedicated drive per OSD, and WAL and DB on shared SSDs for HDD-based OSDs. Confirm the SSDs are enterprise-class, have enough capacity for the planned DB size, and do not already carry another OSD's journal you have not accounted for. Hardware that does not match this layout will limit the converted cluster.
Test cluster or lab run
Run the full conversion sequence on a lab cluster or a non-production pool that matches your OSD types, host layout and release path. Time each step, record which commands failed, and confirm that data returns to clean placement groups between OSDs. The lab run is where you find problems with device naming, container images and client compatibility before they can affect production data.
Written rollback decisions
Decide in writing what you will do if an OSD fails to start, if recovery stalls, or if a host must be taken out of the rotation mid-conversion. Name the person who makes that call, the point at which you stop converting, and the conditions for resuming. A written rollback plan turns an unexpected failure into a planned pause instead of an open-ended incident.
Sequencing FileStore conversion with the Reef upgrade
This sequence converts every FileStore OSD to BlueStore while the cluster is still on a release that supports FileStore, so the Reef upgrade starts from a cluster with no FileStore OSDs left. Each batch is limited to one CRUSH failure domain and closes only after the cluster returns to health.
Confirm current release and cluster health
Record the Ceph release each daemon is running, the OSD store type for every OSD, and the output of the cluster health check before any data moves. Your team or your implementation partner does this from a host with admin credentials, and the output goes into the change record. Confirm the health state is clean and that no OSD is already down or out, because a conversion started on a degraded cluster compounds the recovery load. Rollback point: nothing has changed yet, so the sequence can stop here at no cost to data.
Choose the conversion method for each host
Select mark-out replacement, whole-host replacement or per-OSD copy with ceph-objectstore-tool for each host, based on whether you have spare hardware and how much network traffic you can absorb. Mark-out needs no spare hardware, but data crosses the network twice. Whole-host replacement needs a spare host. Per-OSD copy uses tooling the Ceph documentation describes as unsupported, so treat it as a last option that needs a written sign-off. Record the decision per host. Rollback point: the method choice is a plan document, so it can be revised before any OSD is touched.
Define batches inside single failure domains
Group the FileStore OSDs into batches so that every OSD destroyed in one batch sits within a single CRUSH failure domain, such as one host or one rack, depending on your CRUSH map. Check the CRUSH hierarchy with the cluster tooling and write down each batch's members. This keeps the data for any placement group from losing more than one copy at once. Your team confirms each batch list in writing before work starts. Rollback point: batch lists are documents only, so they can be reshuffled at no cost.
Convert one batch and wait for recovery
Mark out or replace the OSDs in the first batch, recreate them as BlueStore, and let the cluster backfill and recover. Watch placement group states and recovery progress until the cluster reports health again with no degraded or misplaced objects. Do not start the next batch until this condition holds. Rollback point: if recovery stalls or the health state worsens, stop, keep the remaining FileStore OSDs in place, and investigate before you continue. Earlier batches stay converted and do not need to be reversed.
Verify each converted OSD
After each batch, confirm that every new OSD reports BlueStore as its store type, is up and in, and carries the expected weight and host placement in the CRUSH map. Check that no OSD from the batch is flapping or logging errors. Record the verification output against the batch name. Rollback point: an OSD that fails verification is taken out and replaced before the next batch begins, so a bad device never joins the cluster silently.
Repeat batches until no FileStore OSDs remain
Run the batch, recovery, and verification cycle for each remaining failure domain in order until an inventory of OSD store types shows no FileStore entries. Keep the change record updated with the host, batch, and verification result for each pass. Confirm the final count of FileStore OSDs is zero before moving on. Rollback point: while any FileStore OSD remains, the cluster is still on a release that supports it, so the upgrade has not started and you can pause the program at a batch boundary.
Start the Reef upgrade on a clean cluster
Only after the store inventory confirms zero FileStore OSDs, start the upgrade to Reef through the cephadm upgrade process, following the release notes for the exact source and target versions. Confirm the target release and version with the official releases documentation before you begin. Your team monitors daemon upgrades and health throughout. Rollback point: the cluster still has the pre-upgrade state recorded in step one, and you can halt the upgrade between daemon groups if health degrades.
Run post-conversion and post-upgrade checks
After the upgrade completes, confirm the cluster health state, the running release on every daemon, and the store type of every OSD. Run a scrub or review the scrub schedule, and confirm client I/O and monitoring dashboards behave as they did before the program started. Archive the change record with the health outputs, batch verifications, and version evidence. The sequence is complete when the record shows a clean cluster on Reef with no FileStore OSDs. Rollback point: any issue found here is handled as a new change request, not by reverting converted OSDs.
Where FileStore migrations stall or lose data
The most damaging FileStore migration mistake is destroying OSDs in more than one failure domain at once. The BlueStore migration documentation says that when you reprovision in parallel, OSDs should be destroyed only within a single CRUSH failure domain, because removing copies from several hosts, racks or rooms at the same time can leave placement groups with too few surviving replicas. The same concern applies to a team that runs a host replacement while another team converts OSDs on a second host in the same rack. Before each batch, check the CRUSH tree with ceph osd tree and confirm that every pending destruction sits under one failure domain, and that the cluster returns to HEALTH_OK before the next batch starts.
A second pattern is starting the Reef upgrade while FileStore OSDs are still present. FileStore cannot run on Reef or later, so an upgrade that proceeds with those OSDs in place leaves daemons that cannot come back up in the new release. Treat the inventory of FileStore OSDs as a gate: confirm the count is zero, or that the remaining ones are scheduled for conversion before the upgrade window opens, and record that check in the change ticket. Unsupported per-OSD copy with ceph-objectstore-tool is a third risk. Teams that choose it without a rehearsed restore path have no clean way back if a copy fails partway, so require a tested recovery procedure on a non-production OSD before using it on production data.
Finally, teams underestimate recovery traffic. Mark-out replacement moves data across the network twice, and whole-host replacement also rebuilds data onto new media while clients keep writing. On a shared front-side network this can slow client I/O for longer than planned. Ask the migration team to measure recovery throughput on a representative OSD, set recovery limits deliberately, and schedule conversion batches so that the cluster always has enough headroom to absorb one more failure while data is in motion.
What makes the FileStore migration longer
This table lists the factors that stretch a FileStore to BlueStore conversion and the questions to ask about each one. Read each row as a driver: the more of it your cluster has, the more time, network capacity and careful sequencing the conversion will need. Ask for these numbers from your own environment before comparing migration methods or quotes.
| Driver | Why it matters | What to ask |
|---|---|---|
| OSD count and capacity per OSD | Each FileStore OSD must be drained, destroyed and rebuilt as BlueStore, so more OSDs and larger drives mean more total data to move through the conversion. | How many FileStore OSDs remain, and how much raw and used capacity sits on each one? |
| Data volume to rebalance | Mark-out replacement moves data out and back in, so the amount of stored data sets how much traffic the conversion generates on the cluster. | What is the used data per host and in total, and how much of it is client-facing during business hours? |
| Network bandwidth and bonding | Recovery and backfill share links with client I/O, so slow or single-path links lengthen the window and raise contention. | Are cluster and public networks separated, is each link bonded across two switches, and what throughput is reserved for recovery? |
| Spare hardware availability | Whole-host replacement needs a spare host, and without spare capacity the cluster must absorb the data movement itself. | Is a spare host or enough free capacity available in the same site, and can it be racked and joined to cephadm before the conversion starts? |
| Failure domain layout | Destruction must stay within one CRUSH failure domain at a time, so the number of hosts, racks or zones determines how many OSDs can be converted in parallel. | How are hosts grouped into racks or zones, and how many failure domains can be converted at once without reducing redundancy? |
| Shared WAL and DB devices | A shared SSD holding DB and WAL for several HDD OSDs ties their conversion together, so one device failure or rebuild affects every OSD it serves. | Which OSDs share a WAL or DB device, and does conversion of one OSD require touching the shared device? |
| Custom CRUSH rules | Rules that pin pools to specific device classes or hosts limit where data can be placed, which constrains rebalance and slows recovery. | Which pools use custom rules or device classes, and how much free space does each target class have during conversion? |
| Size of the test environment | Without a representative test cluster, sequencing problems tend to surface on production data, where rollback is harder. | Can a test cluster with the same release, OSD layout and rules run the full conversion sequence first, and what does it leave out? |
Questions teams ask before converting FileStore OSDs
What drives the cost and effort of converting FileStore OSDs to BlueStore?
The main driver is how much data must move and how many hosts are involved. Each OSD must be drained or rebuilt, and the data it holds crosses the network again, so cluster capacity, fill level and network throughput set the workload. Other factors include whether you have spare hardware for whole-host replacement, how many OSDs each host carries, how much change control your organization requires, and how much pre-conversion health work is needed. Costs fall into one-time labor for planning, conversion and verification, plus any temporary hardware or network capacity. Ask for a quote that lists the migration method, the host and OSD count in scope, the assumed change windows and who performs each step.
Can our own team convert FileStore OSDs without outside help?
Yes, if your team already operates the cluster and understands CRUSH failure domains, cluster health states and cephadm. The Ceph migration documentation describes the supported methods, and a careful team can follow them. The questions to answer honestly are whether you have engineers who can sequence drains and rebuilds without pausing for long periods, whether someone can watch recovery and client latency overnight, and whether you can test a failed OSD path before it happens in production. A partner is often used for planning, a second pair of eyes on the sequence, or staffing peak hours. DeployOpen plans, implements and documents this work inside your infrastructure under a written scope, so you can compare that option with doing it internally.
How long does a FileStore to BlueStore migration take?
Duration depends on the amount of data to rebuild, the number of OSDs converted at once, and the recovery speed your network and drives can sustain while clients keep writing. Recovery is usually the longest step, so the schedule scales with stored data rather than with the number of hosts alone. Other drivers include how many batches you run, how long each batch must stay healthy before the next starts, the time needed for the release upgrade that follows, and any pauses for change approval. Benchmarking the cluster and measuring recovery behavior on a single host gives a more reliable estimate than any generic figure. Ask the partner to show the batch plan and the health checks that gate each batch.
What does the migration need from our staff and change windows?
You need named owners for the storage cluster, the network, and the change board, plus someone who can approve each batch based on health output. Staff must be available to watch recovery, respond to alerts, and stop the process if client latency rises. Change windows should cover the drain and rebuild periods, which can run for many hours or days depending on data volume, and they should leave room to pause between batches. Confirm whether your policy allows continuous monitoring during the window, how escalation works out of hours, and who can authorize a rollback to the previous state. Record these responsibilities in the scope so the quote reflects them.
Is the per-OSD copy method with ceph-objectstore-tool safe to use?
It can work, but it is the riskiest of the three methods and the Ceph documentation treats the tooling as unsupported. Per-OSD copy with ceph-objectstore-tool moves data for a single OSD without a full host rebuild, which can save spare hardware. The trade-off is that a mistake in the copy or in OSD identity can corrupt data that the cluster then cannot easily recover. If your team considers it, require a documented test on a non-production cluster, a verified backup or snapshot strategy for affected data, and a written decision from your storage owner. For most regulated environments, mark-out or whole-host replacement is easier to justify because both use supported workflows.
What happens if a conversion fails in the middle of a batch?
The cluster stops advancing the batch and you decide whether to pause, retry or roll back the affected OSDs. The first step is to check cluster health and confirm that no OSD in the batch was destroyed outside one CRUSH failure domain. Failures often come from slow recovery, a failing drive, a network fault, or a mismatch in OSD configuration. Your plan should define the stop conditions, the commands and checks used to confirm state, who decides on retry, and how long a partially converted host can run in a mixed state. Ask the partner to show how each batch is scoped so that a failure touches a limited set of OSDs, and to document the recovery steps before the first batch starts.
How do we request a quote for this migration?
Start with a written scope that names the cluster, the Ceph release you run now, the number of hosts and FileStore OSDs, the migration method you prefer, and the target release for the upgrade. Add the change windows you can offer, any spare hardware, and your monitoring and approval process. DeployOpen does not publish a price list. Most engagements are fixed-scope, so the quote follows from the written scope, with deliverables and timelines agreed before work starts. Ask the vendor to itemize planning, conversion, verification, documentation and any ongoing support, and to state which items are one-time and which recur.
What a written scope for the migration should include
A written scope for a FileStore to BlueStore migration should name each stage, what DeployOpen produces at the end of it, and who on your side signs off. The first stage is an inventory of every FileStore OSD, its host, its CRUSH location, its device class, and whether its journal sits on a separate SSD. The inventory also records the current release, the cephadm or non-cephadm management state of each daemon, and the free capacity in each failure domain. From that inventory the scope should state which migration method is recommended and why: mark-out replacement, whole-host replacement or per-OSD copy with ceph-objectstore-tool. Per-OSD copy uses tooling the Ceph documentation describes as unsupported, so the scope should say whether it is excluded or whether you accept that risk in writing.
The scope should then describe a test run on a single host or a small set of OSDs in a non-production or low-risk failure domain, with the checks that must pass before conversion continues. Next comes the conversion batch plan, which sets how many OSDs are destroyed and rebuilt at once and confirms that every batch stays within a single CRUSH failure domain, as the Ceph BlueStore migration documentation requires when reprovisioning in parallel. Verification criteria for each batch, such as cluster health returning to a clean state and data placement completing before the next batch starts, belong in the scope as acceptance tests rather than general intentions. The scope also covers Reef upgrade coordination, including the release gate that confirms no FileStore OSDs remain, and the handover pack of runbooks, batch records and rollback notes. Your team should confirm which supported Ceph release you are upgrading toward, because the scope has to match it.
Scope, deliverables and timelines are agreed in writing before work starts, and a quote follows from that written scope. Ask for the quote to show each stage separately, the assumptions about spare capacity and maintenance windows, and which items recur, such as post-upgrade support, rather than one-time migration work. Items that depend on your environment, including available spare hosts, network capacity during data movement, and change approval lead times, should be listed as client responsibilities so they can be priced and owned in advance.
How DeployOpen works on this
DeployOpen plans FileStore to BlueStore conversions as written, fixed-scope work inside your own infrastructure, starting from an inventory of every FileStore OSD and the release you need to reach. Each conversion is sequenced against your upgrade, with destruction limited to one failure domain at a time and verification at every step.
How we scope
- We inventory every FileStore OSD, host and CRUSH failure domain to confirm which OSDs must convert before Reef.
- We choose mark-out replacement, whole-host replacement or per-OSD copy with you, based on the spare hardware you have.
- The written scope states the failure domains, conversion order, rollback points and the release targets it covers.
How we build
- We run conversions in batches confined to a single CRUSH failure domain, so destroyed OSDs never hold the only copy of data.
- We wait for cluster health to return before each next batch, and we record recovery state after every step.
- We treat per-OSD copy with ceph-objectstore-tool as unsupported tooling and only use it after you accept that risk in writing.
How we hand over
- You receive a runbook with the conversion sequence, health checks, commands used and the batch log for each host.
- We document the upgrade sequencing against the Reef requirement and the checks that confirm each step completed.
- We walk your operations team through the post-conversion checks so they can repeat the procedure on future hardware.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer after the conversion is complete.
- A retainer can cover follow-on upgrades and point release rollouts on the release you settle on.
- Confirm with us what support window, response terms and escalation path apply before signing the written scope.
