Why an air-gapped Ceph upgrade starts with the image list
An air-gapped cephadm cluster that must move to a newer supported release follows the same orchestrated process as a connected upgrade. The difference is that every container image the upgrade needs must already sit in a local registry before anyone runs the command. cephadm runs each daemon as a container, so the Ceph image and the monitoring images must be reachable from every host. If one image is missing, the orchestrator stalls partway through, often after some daemons have already been restarted on the new version. The practical starting point is therefore a written list of images for the target release, built from the cluster's current configuration rather than from memory.
The supported destination is the newer maintained release. The Ceph releases page lists Tentacle (20.2.x) as the newer maintained line, with an estimated end of life of June 1, 2027, and Squid (19.2.x) with an estimated end of life of October 31, 2026. Before choosing a target, confirm with your own support plan which release line you will be on when Squid reaches end of life, and check the source release against the sibling upgrade pages, since the supported starting points differ by path. The Tentacle release notes describe upgrading with ceph orch upgrade start, and that is the command you will point at the local image once the registry is ready.
Confirm three things before the maintenance window opens. First, list the Ceph image and the Prometheus, Node Exporter, Grafana and Alertmanager images that your cluster deploys, using the tags your cephadm version reports rather than assumed defaults. Second, mirror each one into the local registry under the exact name and tag the cluster will request. Third, verify that every host can pull each image from that registry, not just the admin node. Ask your Ceph vendor or support contact to confirm the image set for the specific release you choose, because a mismatched tag is the most common reason an offline upgrade stops at its first step.
What changes when the upgrade runs without internet access
This table compares each change an isolated cephadm upgrade introduces against what it affects and what your team must do before the upgrade starts. Read it as a pre-flight list: every row names a dependency that a connected upgrade normally pulls in automatically. The main takeaway is that the target Ceph image and its monitoring images must already exist in your local registry, and every host must be able to pull them before the upgrade command runs.
| Change | Impact | Action needed |
|---|---|---|
| Custom container registry | cephadm pulls every daemon image from a registry you host, so no host can fall back to a public source during the upgrade. | Stand up or confirm a local registry reachable from all cephadm hosts, and confirm your install procedure against the cephadm install documentation. |
| Mirrored target Ceph image | The target release image must exist locally before the orchestrator starts, or daemons will fail to pull during the staggered restart. | Mirror the exact target Tentacle or Squid image tag into the local registry and confirm the tag matches the version you plan to upgrade to. |
| Prometheus image | cephadm redeploys the monitoring stack, so a missing or older Prometheus image can block the monitoring service during the upgrade. | Mirror the Prometheus image version that the target release expects and verify it can be pulled from each host that runs the service. |
| Node Exporter image | Node Exporter runs on every host, so one host that cannot pull it leaves that host unmonitored and can stall the upgrade check. | Mirror the image and test a pull on every cluster host, not only on the admin node. |
| Grafana image | Grafana is deployed as a container, so a pull failure affects dashboards and can surface as a health warning after the upgrade. | Mirror the Grafana image and confirm the dashboard service starts from the local registry after a test deployment. |
| Alertmanager image | Alertmanager handles alert routing, and a missing image leaves alerts silent during the window when operators most need them. | Mirror the image, confirm the alert routing configuration still points at your internal receivers, and test a pull on each host. |
| Upgrade command pointed at the local image | The orchestrator must be told to use the local image reference, or it will attempt to contact a public registry. | Run ceph orch upgrade start with the local image reference and record the exact command in your change record for the upgrade. |
| Release notes for the source and target | Version-specific behavior and breaking changes that apply to your source and target are not covered in this table and vary by release. | Read the Tentacle or Squid release notes for your source and target versions and confirm each breaking change and required step before scheduling. |
| Pull verification on every host | A successful orchestrator start does not prove that each host can reach the registry, so failures appear mid-upgrade. | Before starting the upgrade, pull each mirrored image from every host using the same registry name and credentials cephadm will use. |
What to prepare before the upgrade window opens
Most failed air-gapped upgrades fail before the first daemon restarts. The items below are the artifacts and checks your team should have in hand, so the upgrade itself is a controlled sequence rather than a search for missing images or unknown host states.
Image mirror inventory
Record the exact Ceph image reference and the Prometheus, Node Exporter, Grafana and Alertmanager image references that your cephadm deployment uses, with their tags and digests. The inventory is the list your registry must hold before the window opens. Compare it against the target release's requirements and confirm each tag exists in the local registry, not only in your staging notes. Any image missing from this list will block the upgrade on the first host that needs it.
Integrity checks on carried artifacts
Every package and container image crossing the boundary should carry a recorded checksum, and the transfer media should be logged from source to destination. Verify each item on the isolated side before it reaches the registry, and keep the verification output with the change record. This is the evidence regulated utilities need for NERC CIP-013-2 software integrity and authenticity expectations, and it also protects against a tampered image reaching production hosts.
Configuration and monitor data backup
Before changing anything, capture a cluster configuration export and a copy of monitor data, then test that you can restore from it in staging. Confirm that the backup is stored inside the isolated network and that someone other than the operator can read the restore procedure. Without a tested baseline, a failed Monitor upgrade leaves you reasoning from memory about the cluster's prior state.
Staging cluster with the production image set
Run the same upgrade path on a staging cluster or lab that uses the same local registry, image tags and host OS packages as production. Use it to confirm that the upgrade command resolves to the mirrored image and that the sequence of daemon restarts matches your plan. Record the differences you cannot reproduce, such as drive counts or network topology, because those are the risks the production window still carries.
Current host and runtime inventory
List every cephadm-managed host with its operating system, container runtime and version, and the daemons it runs. Check that each host can resolve and pull from the local registry using the same credentials cephadm will use, and record any host that differs from the rest. A host with an older runtime, a stale registry certificate or a missing pull secret is the most common reason a staged upgrade stops partway through.
Change window with named rollback owners
Set the change window with the operations, security and application teams, and name one owner for the upgrade, one for each rollback decision and one for communication to clients. Agree in advance which health states mean pause, which mean roll back, and who has authority to make each call. Write this into the change record so that a late-night decision follows a plan that already exists.
The upgrade sequence, step by step
The sequence moves a cephadm-managed cluster to a target Tentacle release using only images held in your local registry, with a rollback point before the upgrade starts and after each verified stage. Each step ends in a check you can record, so a stalled or partial upgrade is caught before it reaches the next stage.
Confirm cluster health and current versions
The cluster administrator records the output of ceph health detail, ceph versions and ceph orch ps before any change, and confirms the cluster is in HEALTH_OK or that every warning is understood and accepted. Save the output with the change record and take a verified backup of the cluster configuration and monitor data according to your own procedure. This is the first rollback point: if the cluster is already degraded, stop here and resolve the warnings, because an upgrade on top of unresolved faults makes the cause of any later failure hard to isolate. Done means the health state, daemon versions and backup location are written down and reviewed.
Mirror target images into the local registry
On a connected staging host, pull the target Ceph image and the Prometheus, Node Exporter, Grafana and Alertmanager images that cephadm uses, then push them to the registry inside the isolated network under the same tags your deployment will reference. Confirm the exact tags against the release notes for your target version, and record the image digests so the copies can be compared later. Transfer the images across your approved media process, scan or verify them as your security policy requires, and load them into the local registry. Done means every required image is present in the local registry with a matching digest and the list is stored with the change record.
Point cephadm at the local image location
Configure cephadm to use the target Ceph image from your local registry and set the monitoring images to the local registry paths, so that no daemon falls back to a public registry during the upgrade. Confirm the settings with the ceph config get mgr command for the container image and the monitoring image keys, and check that the values match the digests recorded in the previous step. Stop if any value still points to an external registry. Done means the configured image references resolve to the local registry and nothing in the configuration refers to an outside location.
Test image pulls on every host
Run a pull of each required image on every cluster host, not only on the admin node, using the container runtime that cephadm uses on that host. Include hosts that run only OSDs, monitors or gateways, because a host with a stale DNS entry, certificate trust gap or missing registry login can pass on the admin node and still fail during the upgrade. Record the result for each host and image. Stop and fix any failed pull before continuing, since cephadm will try to redeploy daemons on each host in turn. Done means every host returns a successful pull for every image.
Start the upgrade with the orchestrator
Immediately before starting, confirm the rollback point from the first step still applies and that no maintenance activity or hardware work is running. Start the upgrade with ceph orch upgrade start and the local image reference for the target version. Confirm the upgrade has been accepted by checking ceph orch upgrade status, which should show the target image and the first daemon being upgraded. Stop the upgrade immediately with ceph orch upgrade stop if it reports an image or pull error. Done means the orchestrator reports the upgrade in progress against the local image.
Watch progress and health until completion
Monitor ceph orch upgrade status, ceph health detail and ceph orch ps until the orchestrator reports that the upgrade is complete. Watch for daemons that stay in an error or unknown state, for health warnings that appear mid-upgrade and for OSDs that fail to restart. If a stage stalls or health degrades beyond what was expected in your plan, pause the upgrade rather than letting it continue, and investigate before resuming. Record a rollback point after each verified stage, meaning after the managers, monitors, metadata services, object gateways and OSDs have each reported the target version and the cluster is healthy again. Done means the orchestrator reports completion and the cluster returns to a healthy state.
Verify daemon versions and monitoring
Run ceph versions to confirm every daemon type reports the target release, and run ceph orch ps to confirm each daemon is running the expected image from the local registry. Check that Prometheus, Node Exporter, Grafana and Alertmanager are running from the local image references, that dashboards load and that alerts are delivered to your test receiver. Compare the results with the baseline from the first step and file the record with the change. Done means every daemon reports the target version, every monitoring component runs from the local registry, and the post-upgrade record is complete.
How to roll back depending on where the upgrade stopped
Before you run ceph orch upgrade start, no daemon has changed version, so the cluster is still in its original state. This is the cheapest point to stop. Your work here is verification, not reversal: confirm the mirrored images are present in the local registry, confirm each host can pull them, and confirm the cluster is healthy enough to upgrade at all. If a check fails, you stop and fix the input without touching daemons.
- Image presence: confirm the Ceph image and the Prometheus, Node Exporter, Grafana and Alertmanager images exist in the local registry under the exact tags you plan to use.
- Host pull test: pull each image from every cephadm-managed host, including hosts that run only monitoring services.
- Cluster health: run
ceph health detailand clear warnings that would block or confuse a rollback decision. - Backups: confirm a recent, restorable backup of the monitor store and of any configuration you would need to rebuild services.
- Orchestrator state: run
ceph orch upgrade checkagainst the target image and record the output.
Nothing has advanced yet, so reverting means discarding the plan rather than running a rollback. Confirm the supported rollback method for your exact versions in the release notes before you start, and do not assume one exists later in the process.
A partial upgrade means some daemons run the new release and others still run the old one. This is the stage where rollback is most often attempted and most often misjudged. Start with ceph orch upgrade status and ceph orch ps to see exactly which daemons have advanced and which are stuck, then decide whether the failure is in one daemon, one host or the image itself.
- Pause first: run
ceph orch upgrade pauseso the orchestrator stops moving to the next daemon while you investigate. - Failing daemon: check
ceph orch psand the daemon logs for pull failures, which often point to a missing image in the local registry rather than a Ceph defect. - Failing host: confirm the host can reach the local registry and resolve its name, since an air-gapped host that cannot pull will stall the upgrade.
- Reversible actions: stopping, pausing or resuming the upgrade, and redeploying a single daemon with its image corrected, are orchestrator operations you can run.
- Restore from backup: if monitors or OSDs have advanced in a way the orchestrator cannot reconcile, the recovery path is a restore, and the release notes define which versions allow that.
Do not assume a daemon can be moved back to the old release once a newer daemon has joined the cluster. Confirm the supported rollback method for your exact versions in the release notes before you act.
After the upgrade completes and the new release carries production traffic, rollback is no longer a simple orchestrator action. Daemons have advanced, data formats and cluster settings may have changed, and clients may depend on new behavior. Your job here is to confirm that the cluster is stable, document the state you now operate, and decide whether any change is still reversible. Most post-upgrade problems are handled by fixing forward, not by reverting.
- Stability checks: watch
ceph health detail, scrub status and client latency over a period your operations team agrees on before closing the change. - Client compatibility: confirm clients and kernel or librados versions still connect, since some post-upgrade settings can cut off older clients.
- Configuration changes: list settings that the upgrade or follow-on steps changed, such as minimum release requirements, and note which ones are one-way.
- Backup refresh: take a new backup of the upgraded cluster so that any future restore starts from the new state.
- Restore only: treat a return to the previous release as a full restore from backup, with the data written since the backup accepted as lost or replayed.
Confirm the supported rollback method for your exact versions in the release notes, and record that answer in your change record before you declare the upgrade complete.
What makes an air-gapped Ceph upgrade take longer
This table lists the factors that stretch an air-gapped cephadm upgrade and the effect each one has on effort. Use it to see which factors apply to your cluster before you ask for a quote, since most of them grow with the size and customization of your environment. The main takeaway is that the transfer and approval steps often set the timeline more than the upgrade command itself.
| Driver | Effect on effort |
|---|---|
| Number of hosts and daemons | Every host must pull the new images, and each daemon restarts in turn, so more hosts and daemons mean more staged checks. |
| Monitoring images that differ from defaults | Prometheus, Node Exporter, Grafana and Alertmanager images must all be mirrored, and any custom tags add mapping work and verification. |
| Custom container images | Each custom image must be rebuilt or retested against the new Ceph release and mirrored into the local registry before the upgrade starts. |
| Volume of images transferred and verified | Large image sets take longer to carry across the boundary and to check against published digests or checksums on each side. |
| Change-window limits | Narrow maintenance windows force a staggered upgrade across several sessions, which lengthens the total calendar time. |
| Customization in cephadm specifications | Heavily customized service specs, placement rules and custom configuration need review so they still apply after the upgrade. |
| Tested staging cluster | Without a staging cluster that matches production, rehearsal and rollback planning must happen on the live system, which adds risk and time. |
| Approval cycles in the isolated environment | Each approval gate for media, images or change requests adds waiting time, and these gates often run longer than the technical steps. |
What evidence an isolated network needs to keep
An assessor or internal control owner will ask for proof that the images running your cluster are the ones you intended to carry across the boundary, so the evidence starts at the transfer, not at the upgrade. Keep a transfer manifest for every batch: the Ceph image reference and digest, the Prometheus, Node Exporter, Grafana and Alertmanager image references and digests (the set the cephadm install guide lists for isolated deployments), the checksums of each exported archive, who approved the transfer, the date it crossed the boundary and the media it traveled on. Record the tags as well as the digests, because a tag alone can move while a digest cannot. Confirm with your own registry tooling which digest format it reports so the manifest matches what the orchestrator later resolves.
NERC CIP-013-2 asks regulated operators to have supply chain procurement processes that address verification of software integrity and authenticity for vendor software and patches. For an isolated Ceph upgrade, that expectation translates into three records you can show on request: the checksum and signature verification performed before import, the approval that allowed the batch in, and the scan or review result for the carried images. Ask your compliance lead which of these your internal procedure already defines, because the standard leaves the method to you. Store the manifest, approvals and verification output together under the change record so one person can trace a single image from source media to the running daemon.
Per-host pull test results show that every host can fetch the new images from the local registry before the orchestrator starts. Keep the output of each pull with the host name, the image reference attempted and the result, and the time of the run. After the upgrade, keep the ceph orch upgrade status output, the orchestrator log lines for each daemon restart, and ceph health detail captured before the first step and again after the last one. Those two health captures are the record that cluster state was known going in and verified coming out. DeployOpen prepares this evidence and structures it for your change process, but it does not issue audit reports, attestations or certifications. Independent auditors and certification bodies make those determinations, so agree with your auditor in advance which records they will accept and in what format.
Questions teams ask before an air-gapped Ceph upgrade
What drives the cost of an air-gapped Ceph upgrade?
The cost is driven by how many clusters and hosts are in scope, how much preparation the isolated network needs, and how many approval and transfer cycles the work requires. Most of the cost is labor, split between your team's time and any partner hours in the written scope. The one-time parts are the registry build, the first mirror of images, and the runbook and test plan. The recurring parts are mirroring each new release and monitoring image set, repeating the host pull checks, and running the upgrade itself. Ask any quote to itemize these separately, and to state which hosts, monitoring images and rollback checks are included.
How long does an air-gapped Ceph upgrade take?
The duration depends on the written scope, not on a fixed figure. The main effort drivers are the number of hosts and daemons, how quickly images can cross your transfer process, how many change approvals the isolated network requires, and whether a staging cluster exists for rehearsal. Health checks and staggered daemon upgrades add time on larger clusters, and any failed pull adds a fix-and-retry cycle. Ask for a schedule that lists each phase, its entry and exit checks, and the approval windows it depends on.
What does our team need to provide for the work?
You need to provide four things: access to the isolated network and the hosts that will run the upgrade, a defined staging transfer process for moving images and checksums across the boundary, the change approvals your policy requires for the maintenance window, and a named owner who can make decisions and sign off on each phase. Your team should also confirm the current Ceph release and the target release in the release notes before the work starts. Missing approvals and an unclear transfer path are the most common causes of schedule slips.
Can our in-house team do this upgrade without a partner?
Yes, if your team already runs cephadm clusters and can operate inside the isolated network. The upgrade itself uses ceph orch upgrade start as described in the Tentacle release notes, and the steps are documented. The harder parts are building a registry that every host can reach, mirroring images accurately, and proving that each host can pull before the upgrade begins. Teams that have not done a major upgrade in an isolated network often benefit from a rehearsal on a staging cluster first. Decide based on your staff's current workload and on whether a second person can review each step.
Why do the monitoring images need to be mirrored separately from the Ceph image?
Cephadm deploys monitoring services as their own containers, and those images come from different projects than the Ceph image. The cephadm installation guidance for isolated environments calls for a custom registry that holds the Ceph image along with the Prometheus, Node Exporter, Grafana and Alertmanager images. If any of those are missing, the Ceph daemons can upgrade while the monitoring stack fails to start or stays on old versions. Check the exact tags for each image against your cephadm configuration, and mirror them in the same staging batch as the Ceph image so the set stays consistent.
What happens if a host cannot pull an image during the upgrade?
Confirm the pull on every host before you start the upgrade, not after. A host that cannot reach the local registry, has a stale certificate, or lacks the right tag will fail to run its daemons on the new image. Check registry reachability, name resolution, TLS trust and the exact image tags from each host. If a pull fails mid-upgrade, pause, fix the registry path or the image set, and resume only after the failing host passes the same pull test as the rest. Your runbook should state who can stop the upgrade and what the health checks must show before it continues.
Does the upgrade need an internet connection at any point?
No. Once the Ceph and monitoring images are in your local registry and each host points there, the upgrade can run without internet access. The work moves the required images across the boundary in advance, through your approved transfer process. You still need to review the release notes and any point release fixes ahead of time, and record the checksums of the images you transferred. Keep a copy of the transfer record with the change ticket so auditors and internal reviewers can trace what entered the network.
How DeployOpen scopes and delivers an air-gapped Ceph upgrade
DeployOpen plans, builds and documents Ceph upgrades inside your own isolated network, so the upgrade runs from images and packages you have verified and mirrored locally. The work starts with a written scope that names the cluster, the target release and every image the cephadm upgrade will pull.
How we scope
- Inventory the current Ceph release, cephadm version, hosts and every daemon image in use.
- Confirm the target release path with the Ceph release notes before any image is mirrored.
- List the Ceph, Prometheus, Node Exporter, Grafana and Alertmanager images to transfer into the registry.
- Agree the written scope, deliverables and timelines in writing before work starts; engagements are fixed-scope.
How we build
- Stand up or validate the local container registry and confirm every cluster host can reach it.
- Mirror the images, record their digests and verify them against the source before they cross the boundary.
- Point the cephadm upgrade at the local image and run pre-upgrade health checks in a staging copy first.
- Test that each host pulls every required image before starting
ceph orch upgrade startin production.
How we hand over
- Deliver a runbook with the exact image list, digests, registry paths and the order of host checks.
- Document the rollback and pause steps, plus the checks to run between each daemon group.
- Record the evidence of image verification and pull tests for your change and compliance files.
- Walk your operations team through the procedure so they can repeat it for the next point release.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers after handover.
- Retainers can cover mirroring the next release and point-release image updates for the isolated network.
- Ask your DeployOpen contact to confirm response arrangements and what a retainer includes in writing.
- DeployOpen prepares systems and evidence for compliance assessments but does not issue audit reports or certifications.
