Upgrading Rook and Ceph on Kubernetes

Upgrading Rook and Ceph on Kubernetes: when to move and which path is supported

A platform team running Rook-managed Ceph eventually has to move to a newer release, and the work is two separate steps: first the Rook operator, then the Ceph version it manages. Each step has its own health checks and rollback decision, so do not combine them into one change window. The reason to move at all is maintenance status. According to the Ceph releases page, actively maintained releases receive periodic backports and security fixes, and a release that has passed its end-of-life date stops receiving them. Before scheduling anything, confirm which Ceph release your cluster runs today and whether it is still on the maintained list.

The two lines maintained at the time of writing are Tentacle (20.2.x, first released November 18, 2025, latest 20.2.4, estimated end of life June 1, 2027) and Squid (19.2.x, first released September 26, 2024, latest 19.2.7, estimated end of life October 31, 2026). Squid's estimated end of life is the nearer deadline, so a cluster on Squid should plan its move to Tentacle well ahead of that date. These are estimates from the releases page, so check the current table before you commit a date. The Ceph version must match what the Rook release you choose supports, and that compatibility table lives in the Rook documentation, which you should read for each target version.

Rook minor releases must be stepped through in order rather than skipped. Move the operator from one minor release to the next, let the cluster settle, run the health checks, and only then take the next step. Skipping a minor release means the upgrade logic for the missed release never runs against your cluster, and the CRDs and CephCluster resource may be left in a state the next version does not expect. Plan for more than one operator change if your current Rook version is several minor releases behind, and ask your vendor or partner to confirm the exact intermediate versions for your path before you start.

What changes between Rook and Ceph releases, and what each change asks of you

This table compares the release facts that shape a Rook and Ceph upgrade, with the impact of each and the action to take before you move. Read each row as a planning checkpoint, and confirm the details against the target version's release notes and breaking-change list before you schedule the work.

Change	Impact	Action needed
Squid (19.2.x) approaching end of life	Squid's estimated end of life is October 31, 2026, per the Ceph releases page, so it receives fewer backports and security fixes over time	Confirm which Ceph release your cluster runs, and plan the move to Tentacle or another supported release before Squid's end of life
Tentacle (20.2.x) as the current line	Tentacle was first released November 18, 2025, with 20.2.4 as the latest version listed, and its estimated end of life is June 1, 2027	Check the latest Tentacle point release and its release notes before choosing a target version for the Ceph upgrade
Backports and security fixes	The releases page states that actively maintained releases receive periodic backports and security fixes, so point releases on your current line carry fixes you need	Track point releases on your current line and schedule regular minor updates within the supported release
Operator first, then Ceph	The Rook operator and its CRDs must support the target Ceph version before the cluster image changes, so the order of the two steps matters	Upgrade the operator and its chart or manifests first, confirm the operator is healthy, then change the Ceph image in the cluster resource
Rook minor releases	Stepping through Rook minor releases may be required, and the supported path can differ between operator versions	Read the Rook upgrade guide and breaking-change notes for each minor release between your current and target version
Kubernetes minor version skew	The kube-apiserver must be upgraded one minor version at a time, and kubelets may be up to three minor versions older than the apiserver, which limits how far the nodes under Rook can lag	Before upgrading Kubernetes, confirm that the Rook and Ceph target versions support the Kubernetes version you plan to run
Kubernetes support window	The project maintains the three most recent minor releases, so an older cluster can fall out of support during a Rook or Ceph project	Check the Kubernetes release page for end of life dates and include any required Kubernetes upgrade in the plan
Target version breaking changes	Release notes for a specific Rook or Ceph jump are not covered here, so changes to CRD fields, defaults or CephX capabilities can differ by version	Check the target version's release notes and breaking-change list before planning, and confirm each item against your own cluster configuration

Which upgrade path applies to your starting version

A cluster already on Squid (19.2.x) has the most direct route to Tentacle (20.2.x). Because Rook upgrades the operator and the Ceph version as two steps, the operator moves first and the Ceph image moves second, with a health check between them. The path stays short only if each gate passes before the next change starts.

Record the starting state: capture the Ceph version, the cluster health output, placement group states and the Rook operator version before any change, so you have a baseline to compare against.
Confirm Rook supports Tentacle: check the Rook compatibility documentation for your operator version. If the operator does not support Tentacle, stop here and upgrade the operator first.
Upgrade the operator: update the Helm release or operator manifests, let the operator reconcile, and confirm its pods are healthy before you change the Ceph image.
Change the Ceph image as a separate step: set the CephCluster image to a Tentacle 20.2.x release. The releases page listed 20.2.4 as the latest patch at the time of research, so confirm the current patch before you pin it.
Gate on health: do not start the next change until placement groups are active and clean and no new health warnings appear.

Before you close the window, compare the post-upgrade health output with the baseline and confirm that client I/O works on a test volume.

When the Kubernetes version changes in the same project, the Kubernetes steps follow the version skew policy, and the Ceph steps follow the Rook path for your starting version. Keeping the two sets of changes in separate windows means a failure has one likely cause. The Kubernetes version skew policy requires kube-apiserver to be upgraded one minor version at a time, before kube-controller-manager, kube-scheduler and the kubelets, and it allows kubelets to be up to three minor versions older than kube-apiserver.

Take an etcd snapshot first: back up etcd with etcdctl snapshot save before any control plane change, and confirm you can restore it with etcdutl in a test environment.
Upgrade the control plane in order: move kube-apiserver one minor version at a time, then kube-controller-manager and kube-scheduler, then the kubelets on each node.
Confirm Rook and CSI support for the target version: check the Rook compatibility documentation and the CSI driver versions that Rook deploys before you start the Kubernetes change.
Gate between steps: after each Kubernetes step, confirm that nodes are ready, Ceph CSI pods are running, volumes still attach and mount, and Ceph health has not changed.
Plan around the support window: Kubernetes 1.35 has an end of life of February 28, 2027 according to the Kubernetes releases page, so include the target version's support dates in your plan.

Do not start the Ceph upgrade until every node reports the target Kubernetes version and your storage workloads have passed a read and write test.

How to prepare before touching the operator or the Ceph version

Most failed upgrades trace back to work done, or skipped, before the first change. The cards below cover what to capture, test and assign so that a rolling restart of operator and daemons can be paused, checked and reversed. Each one feeds the health checks you run before and after every step.

Verified backups of data and custom resources

Confirm that you hold a restorable copy of Ceph data, not just a snapshot of the cluster state. Export the CephCluster, CephBlockPool, CephFilesystem and CephObjectStore manifests, plus the StorageClasses and any ObjectStoreUser and bucket definitions, and store them outside the cluster. Test a restore of at least one volume and one bucket before the change window opens, since a backup you have never restored is an assumption.

Inventory of customizations

List every deviation from the chart defaults: Helm values files, custom CRs, CSI and resource overrides, and CephX caps granted to external consumers. Each item needs an owner and a reason, because values that worked on the old operator can be renamed or dropped in a newer chart. Compare the rendered manifests against the new release before applying anything, and flag any cap that a consuming cluster depends on.

Test environment matching production hardware

Rehearse the full sequence on a cluster that uses the same drive types, OSD layout, network and failure domains as production. Ask the vendor or your own benchmarks how a restart behaves on that hardware class, since performance results from a small lab rarely carry over. Run the operator upgrade, then the Ceph version step, and record the time and any health warnings so the production window is sized from evidence.

Baseline health capture

Record the cluster state before the first change: Ceph health output, OSD up and in counts, monitor quorum, placement group states, scrub and recovery activity, and the Rook operator and CSI pod status. Keep the output with the timestamp and the version pair you started from. Without this baseline, you cannot tell whether a warning appeared during the upgrade or was already present.

Headroom for rolling restarts

Rolling restarts move load onto remaining daemons, so size the headroom before choosing a window. Ceph recommends a dedicated drive per OSD and bonded network links across two switches; a host with shared drives or a single link leaves less room for recovery traffic while OSDs restart. Confirm that the cluster can lose one failure domain's worth of OSDs without losing data, and avoid scheduling the change alongside a scrub or rebalance.

Change window and named rollback owner

Set a change window with a start, a stop point for each stage, and a named person who can decide to roll back. That owner should hold the backup location, the rollback procedure and the authority to halt the rollout between the operator and Ceph steps. Agree in advance which health signals trigger a pause, so the decision is not made under pressure during the restart.

The upgrade sequence, with a health check and rollback point at each step

This sequence moves the Rook operator and then the Ceph version in separate, verified stages, so each change has a known-good state to return to. Every step ends with a health check that decides whether the next step may start.

Freeze changes and capture baseline health
The platform owner pauses unrelated changes to the cluster, node pools and storage classes, and announces the window to teams that consume block, file and object storage. Run ceph status from the toolbox pod and record the health output, OSD and monitor quorum, placement group states and any existing warnings, then capture the CephCluster and CephObjectStore status from kubectl. This step is complete when the baseline is saved and every existing warning is understood. Rollback point: none is needed yet, but if the cluster is already in HEALTH_ERR, stop and resolve it first, because an upgrade will not fix it.
Back up data and custom resources
Confirm that application data has a current backup or snapshot that has been tested for restore, and export the Rook custom resources (CephCluster, CephBlockPool, CephFilesystem, CephObjectStore, StorageClass and user objects) along with the Helm values in use. Store the exports outside the cluster. The owner of each consuming workload signs off that its recovery path works. Rollback point: if a later step fails, the exported manifests and the Helm values let you reapply the previous state, while the data backup covers loss that a rollback cannot reverse.
Upgrade the Rook operator with the Helm chart
Upgrade the operator first through its Helm chart, changing only the operator version and keeping the values from the previous step. Before running the upgrade, read the Rook release notes for the target minor and check whether intermediate minor releases are required, because the upgrade path may need to pass through them. After the rollout, wait until the operator pod is ready and the operator logs show that it has reconciled the CephCluster without errors. Rollback point: if the operator fails to reconcile, roll the Helm release back to the previous revision, which leaves the running Ceph daemons untouched.
Apply manifest changes for the target Rook minor
Apply any CRD, RBAC or manifest changes the target Rook minor requires, in the order the release notes specify, and step through each intermediate minor where the documented path demands it. Re-run the cluster health check after each stepped minor, not only at the end. Confirm that the operator still reconciles and that the Ceph version has not changed in this step. Rollback point: a failed step is reversed by reapplying the saved manifests from the previous step, and you should not advance to the Ceph image change until the operator is stable on the target minor.
Update the Ceph image in the CephCluster
Change the Ceph image tag in the CephCluster resource to the target release and let Rook roll the components in its own order: monitors, managers, OSDs, then metadata servers and gateways. Watch the rollout with kubectl -n rook-ceph get pods and ceph versions until all daemons report the new version. Keep ceph status open throughout, because PGs may enter peering or recovery while OSDs restart, and wait for them to return to active and clean. Rollback point: if monitors or OSDs stall, or PGs do not return to clean within the window your runbook allows, revert the image tag and let Rook reconcile back, and do not change the Ceph version on any further component.
Run post-upgrade health checks and smoke tests
Compare the post-upgrade ceph status and CephCluster status against the baseline captured at the start, and confirm that the health output matches or improves on it. Run smoke tests against each consumer type: create, write, read and delete an RBD volume, mount a CephFS volume from a test pod, and put, get and list an object through the S3 endpoint with a test user. The step is complete when every test passes and the owners of the consuming workloads confirm their services are healthy. Rollback point: if a test fails or health regresses, keep the window open and investigate, because rolling back the Ceph image after clients have written data needs a separate decision.
Close the change window
Record the final Rook operator version, Ceph version, Helm revision and CRD state, and attach the baseline, post-upgrade health output and smoke test results to the change record. Lift the freeze on unrelated changes and notify the consuming teams that the window is closed. Schedule a follow-up check for the next business cycle to watch for slow requests, scrub errors or health warnings that appear only under normal load. The window is closed when the change record is complete and the platform owner has signed it off.

How to test the upgrade and roll it back if health does not return

Run the full upgrade in a staging cluster that mirrors production's Rook, Ceph and Kubernetes versions before touching production. The staging test should prove that the workloads your platform actually serves still work after each step. Create a test PersistentVolumeClaim on the RBD StorageClass, attach it to a pod, write data, take a snapshot through the snapshot class, and restore a clone from it. Mount a CephFS volume from a second pod to confirm shared access still works, and check that an ObjectBucketClaim issues credentials and that an S3 client can put and get an object. Finish with a controlled restart of one pod per workload type and one OSD or node drain, then confirm the cluster returns to HEALTH_OK. Record the results and the commands used, so the production runbook can repeat them exactly.

Rollback is narrower than most teams expect. You can usually pin the Rook operator chart back to the previous version, but only if the Rook release notes list that version as a supported downgrade path for your exact versions. The Ceph daemons are a separate decision: cephadm and Rook can only return a daemon to an earlier release when the target release allows it, and Ceph does not support arbitrary downgrades across major releases. Any data written by the new release, such as new on-disk formats or features enabled after the upgrade, may stop an older daemon from reading the cluster. Confirm the supported rollback path with the Ceph release notes and the Rook upgrade documentation for your versions, and write down the point of no return before you start.

Set the rollback decision criteria before the upgrade begins. For example, an illustrative rule could be that a failed PVC attach, a snapshot that never becomes ready, or a health warning that persists past the agreed recovery window triggers the rollback review. Keep the staging results and the production health output as evidence for that review. If health does not return, first isolate whether the problem is in the operator, the CSI driver, or a Ceph daemon, because each one has a different reversal path. Your team should confirm each of these answers with the vendor documentation for its exact versions before it is needed under pressure.

What makes a Rook and Ceph upgrade take longer

This table lists the factors that stretch an upgrade of the Rook operator and Ceph, and what each one changes in effort. Use it to size the inventory you need before agreeing a scope, since most of the schedule risk comes from items you can count in advance.

Driver	Effect on effort	What to inventory
OSD and host count	More OSDs and hosts means more rolling restarts and longer waits between steps while Ceph returns to health	Number of OSD pods, storage nodes, and the time each restart takes in a test cluster
Data volume and recovery	Large amounts of data slow recovery after each restart, so health checks may take longer to pass	Used capacity, degraded object counts during a test restart, and recovery throughput limits
CRDs and custom manifests	Chart and CRD changes must be applied in order, and local patches to manifests must be reapplied or retired	CephCluster, CephBlockPool, CephFilesystem and CephObjectStore resources, plus any Kustomize overlays or Helm values
External mode consumers	Each consuming cluster needs its CephX capabilities checked or updated before or alongside the provider upgrade	Every Kubernetes cluster importing the external connection, and the caps each one holds
Object gateway integrations	Multisite zones, Keystone or LDAP authentication and OPA settings each need their own compatibility check	RGW realms and zones, identity provider settings, and any clients that depend on specific S3 behavior
Kubernetes version skew	The Kubernetes version must fall within the Rook and Ceph support matrix, which can force a cluster upgrade first	Current control plane and node versions, planned Kubernetes target, and the Rook release notes for supported versions
Client workloads to validate	Each application class (block, shared file, object buckets, snapshots) needs a read and write test after the upgrade	Stateful workloads by StorageClass, bucket consumers, and snapshot or clone jobs that run during the window
Rook minor release stepping	Skipping minor releases is often unsupported, so each intermediate step adds a validation cycle	The current Rook version and the upgrade path listed in the Rook documentation for the target release

Questions teams ask before upgrading Rook and Ceph

How much does an upgrade of Rook and Ceph cost?

The cost depends on the written scope, which is agreed before work starts. The main drivers are the number of clusters and their environments, how many Rook minor releases sit between the current and target versions, the amount of customization in the Helm values and manifests, the size of the health baseline you need, and whether pre-upgrade testing runs in a separate environment. One-time work covers assessment, the upgrade runbook, the upgrade itself and post-upgrade verification. Recurring costs appear only if you add a retainer for ongoing support or upgrades. Ask for a quote that lists each cluster, each upgrade step and each verification check separately, so you can see what you are paying for.

How long does a Rook and Ceph upgrade take?

Duration depends on cluster size, the number of OSDs and nodes that must restart, how long Ceph takes to recover placement after each step, and how many customizations need checking. Recovery time is the variable most teams underestimate, because data rebalancing and scrubbing can continue long after the operator reports ready. Change windows also shape the calendar, since upgrades usually need approved maintenance periods. Ask for a timeline in the written scope that separates preparation, staged upgrades, waiting for health to return to normal and the final review, and confirm which parts depend on your change approvals.

What does the client team need to provide for an upgrade?

An upgrade needs more than a kubeconfig. Plan for the following:

Administrative access to the Kubernetes clusters, with the Rook namespace, Ceph cluster and storage nodes reachable from the workstation that runs the upgrade
Named owners for storage, Kubernetes platform and the applications that consume volumes or buckets
Approved change windows and a rollback decision owner for each cluster
A test environment that resembles production, or a documented way to validate upgrades before production
Current Helm values, manifests and a recent Ceph health report

Confirm these items with your team before the scope is signed, because missing access or approvals are a common cause of delay.

Can our internal team upgrade Rook and Ceph without a partner?

Yes, many teams do this in-house, provided they have people who understand Kubernetes operators, Helm, Ceph health output and CephX client access. The skills that matter most are reading ceph -s and health warnings, understanding how placement groups recover, checking Rook custom resources before and after each step, and knowing when to stop. Confirm with your own engineers whether they have hands-on experience with a Rook minor upgrade in a cluster of your size. A partner is useful when that experience is missing, when the schedule is tight, or when you want an independent runbook and verification record.

Can one step skip Rook minor releases?

Do not assume that a single step can skip Rook minor releases. Upgrade guidance for Rook is written per release, and each minor version can carry changes to CRDs, the operator, RBAC or Ceph version requirements. Plan the path release by release, read the upgrade notes for each step, and verify health between steps. Kubernetes also limits how far components can drift in version, so check the Kubernetes version skew policy alongside the Rook path. Confirm the exact supported sequence against the Rook documentation for your target version before scheduling any change.

What happens if Ceph health does not recover after an upgrade step?

Stop before the next step. The recovery path depends on what is failing. Some warnings clear as placement groups finish peering and recovery, which can take time on large clusters. Others need a fix, such as an operator configuration change, a CephX capability update or a rollback of the step. The runbook should define health thresholds, who decides to pause or roll back, and what data to collect, including ceph health detail, operator logs and recent events. Ask for these decision points in the written scope so that a stalled upgrade does not turn into an unplanned change.

Does the upgrade affect security settings or consumer access?

It can. Upgrades may change CephX capabilities that external consumers or Kubernetes clients rely on, and they may change how the dashboard, RGW or encrypted connections are configured. Review access before the upgrade, record current settings and test a consumer workload after each step. Confirm which consumers are external to the cluster and who owns their credentials, because those are the most likely to break silently.

How DeployOpen scopes, delivers and supports a Rook and Ceph upgrade

DeployOpen plans and delivers Rook and Ceph upgrades inside your own Kubernetes clusters and data centers. Each engagement treats the operator upgrade and the Ceph version upgrade as separate, checked steps, with health evidence captured before and after each one.

How we scope

Inventory current Rook, Ceph, Kubernetes and Helm versions and confirm the target release path with you
Agree in writing which clusters, CRDs, pools and consuming workloads are in the upgrade
Confirm change windows, rollback owners and who approves each step before work starts
Engagements are fixed-scope, so scope, deliverables and timelines are agreed in writing first

How we build

Prepare a runbook covering Helm chart order, manifest changes and the Rook minor release steps
Capture cluster health output before each stage and compare it with the results after
Test the sequence on a non-production cluster that matches your topology where you provide one
Record each change in version-controlled manifests so the upgrade path can be reviewed and repeated

How we hand over

Deliver the runbook, pre and post upgrade health reports, and rollback steps for each stage
Document known warnings, deferred items and the Kubernetes versions your cluster must stay within
Walk your platform and storage teams through the procedure and the checks they will repeat
Provide a list of open questions for your vendor or distribution to confirm before the next release

How we support

Ongoing support, upgrades and continuous testing are available as retainers after handover
Retainers can cover the next Rook minor release or Ceph release review on a schedule you approve
Quotes are based on the written scope, so ask for the components, environments and check counts included
Ask whether independent audit evidence is needed, since DeployOpen prepares evidence but does not issue audit reports

Upgrading Rook and Ceph on Kubernetes