Why teams replace the Ceph iSCSI gateway and what the move involves
Teams planning this move run VMware or Linux hosts that reach Ceph block storage through the iSCSI gateway. The work is a planned migration of LUNs, initiators, multipath configuration and data onto a different access path, and the effort depends on the host mix, the volume of data to move and how much downtime each application can tolerate. The iSCSI gateway is in maintenance mode, which signals where the project's direction is heading, but confirm its current status in the official Ceph documentation and Ceph active releases before you write the plan. Support windows change between releases, so record the exact version your cluster runs.
The two destinations in this plan are different. VMware hosts are planned to move to the NVMe-oF gateway, so confirm its exact product name, the Ceph releases that ship it, and which features it supports for your hypervisor version, including multipath behavior, namespace limits and authentication. Linux hosts can move to native RBD, which uses the kernel client or librbd with cephx users and removes the gateway from the data path. Each destination changes what gets re-cabled, what gets re-authenticated and how multipath is tested. Ask for the supported host OS and hypervisor combinations in writing before inventory begins.
Decide early which LUNs can cut over in a maintenance window and which need parallel running with the old path still active. Inventory every initiator, its multipath settings and the LUN-to-host mapping, because a cutover that works on one host type can fail on another. Plan the data side as well: copying or re-attaching images, verifying application consistency, and keeping rollback available until the old path is retired. Most of the cost is engineer time for inventory, testing and cutover windows, so any quote should list each host group, the number of LUNs in it and the test plan for each group.
What maps cleanly from iSCSI LUNs to Ceph block storage
This table compares iSCSI gateway capabilities with their Ceph block equivalents and marks where the mapping is direct, where the client or protocol changes, and where you must confirm parity in the Ceph documentation. Read each row against your own host inventory, because the cutover path differs for VMware and Linux consumers.
| Capability | iSCSI gateway today | Ceph equivalent | Migration note |
|---|---|---|---|
| LUN export | Block LUN presented to initiators over iSCSI | RBD image in a pool | Direct conceptually. Each LUN becomes an image, but the transport changes, so every consumer needs a new attachment. |
| Initiator access control | Initiator IQN or address lists mapped to LUNs | cephx users with caps scoped to pool and image | Recreate access per consumer rather than copying the old lists. Confirm the cap syntax against the current Ceph client authentication documentation. |
| CHAP authentication | Per-target or per-initiator CHAP secrets | cephx keyring authentication | Not a one-to-one swap. Secrets move to keyrings, so plan rotation and distribution to each host. |
| Multipath | Multiple iSCSI paths via dm-multipath or ESXi path policies | Native RBD clients or NVMe-oF gateway paths, depending on the consumer | Path behavior changes with the client. Test failover on each host type and confirm path policy support with the NVMe-oF gateway documentation. |
| VMware datastore on the LUN | VMFS datastore presented over iSCSI | NVMe-oF gateway exposing RBD images to ESXi | The protocol and host configuration change. Confirm VMware version support and datastore migration steps for your release before scheduling cutover. |
| Linux filesystem on the LUN | Filesystem mounted from an iSCSI block device | Kernel RBD or librbd-attached image with the same filesystem | Data usually stays in place logically, but the device path and udev rules change. Verify mount options and fstab entries after attachment. |
| Snapshots | Array or gateway snapshots of a LUN | RBD snapshots and clones | Snapshot semantics differ by client and filesystem. Confirm crash consistency and quiesce steps with the Ceph RBD snapshot documentation. |
| Host-side tooling | Vendor initiator tools and array management UI | rbd CLI, cephadm-managed services and the Ceph Dashboard | Operational tooling is replaced, not carried over. Confirm dashboard and single sign-on options against the Dashboard documentation before standardizing on them. |
How VMware hosts and Linux hosts take different paths off iSCSI
VMware hosts consume the gateway as iSCSI datastores, so the move centers on how storage is presented to the hypervisor. The hypervisor team, the storage team and the Ceph operators need to work through each datastore and path together, because the change affects every virtual machine that sits on those datastores.
- Host connectivity: Confirm that every ESXi host in a cluster can reach the NVMe-oF gateway endpoints on the networks you intend to use before you present any storage.
- Datastore re-presentation: Plan how each existing datastore is re-presented. Moving VMFS datastores usually means copying virtual machines to new datastores, so confirm the capacity and time that storage migration will need.
- Multipath changes: Review the path policy and the number of paths per device after the change. Remove iSCSI paths only after the new paths have passed failover tests.
- Vendor and documentation confirmation: Ask VMware and your gateway vendor which ESXi versions, transports and multipath plugins they support together. Check the Ceph documentation for the gateway release you run.
Test a single path failure, a host reboot and a storage migration on a non-production datastore before any production datastore moves.
Linux hosts that mount images through the gateway can usually talk to the cluster directly with native RBD, which removes the gateway from the data path. The change happens in identity, client configuration and filesystem handling on each host, so it is best scoped host group by host group.
- cephx user creation: Create a dedicated cephx user for each client group, with capabilities limited to the pools and namespaces that group needs. Avoid sharing one key across unrelated hosts.
- Keyring distribution: Place the keyring and
ceph.confon each client with restrictive file permissions, and distribute them through your configuration management so that key rotation is repeatable. - Image mapping or librbd access: Map images with the kernel RBD client for block devices, or access them through librbd from the application or hypervisor that uses them. Test each method against the kernel and library versions you run.
- Filesystem and LVM consequences: Device names can change after a reboot, so mount by UUID or another stable identifier. Check LVM device filters on the client so that it does not scan RBD devices unexpectedly, and plan data moves from iSCSI LUNs with the filesystem unmounted or with a copy method you have tested.
- Client-side configuration: Set monitor addresses, authentication settings and timeouts in
ceph.conf, and confirm that each client supports the image features you enable.
Confirm each client's kernel and library support against the Ceph release you run before you map any production image.
Some tasks apply to both host groups. Network paths, sequencing, data consistency and rollback need one plan, so that a VMware wave and a Linux wave do not conflict on the same cluster or storage network.
- Network and messenger v2: Open the monitor and OSD paths that the new clients need. Messenger v2 uses port 3300, and its secure mode encrypts all traffic after authentication, so decide whether secure mode is required and confirm that every client supports it. See the Ceph messenger v2 documentation for the settings.
- Sequencing: Move a pilot group first, then move the remaining hosts in waves ordered by application tier, with a written rollback for each wave.
- Data consistency: Quiesce writes or take a snapshot before copying data between iSCSI and RBD, and verify the copied data after each wave.
- Inventory: Record every host, LUN, datastore, path and mount point before the first change, so that rollback and audit questions have a clear baseline.
- Monitoring: Watch cluster health and client latency during each wave, and hold the next wave if either degrades.
Ask your Ceph operators and the hypervisor vendor to sign off the rollback procedure before the first wave starts.
Documented tools and paths for the target side of the migration
The target cluster, its management plane and its sizing all rest on documented Ceph tooling. The four cards below cover the components you will touch during the cutover, and where the documentation stops and your own verification has to start.
cephadm for the target cluster
cephadm deploys and manages the target cluster as containers, starting from a single-node bootstrap and then adding hosts and services. It was introduced in v15.2.0 (Octopus) and does not support older releases, so confirm that your target release and your host operating systems fit this model before planning the build. Use it to deploy the NVMe-oF gateway service alongside the cluster, and check that the gateway version you choose is supported by the same release. Record the cephadm host access model, since it determines who can change the target during the migration window.
Ceph Dashboard and single sign-on
The Ceph Dashboard gives operators a view of cluster health, pools, images and services during the cutover. It supports single sign-on through SAML 2.0, and through OAuth2 when cephadm runs the Management Gateway and OAuth2 Proxy services. Decide which option matches your identity provider before the migration starts, because changing it mid-cutover adds access risk. Confirm the exact configuration steps in the dashboard documentation for your release rather than relying on older blog posts.
Hardware recommendations for sizing
The hardware recommendations are the input for sizing the target, not a substitute for it. They advise a dedicated drive per OSD, moving WAL and DB to shared SSDs for HDD-based OSDs, enterprise-class SSDs for metadata, and bonded network links across two switches. They also say to size memory for peak use and to benchmark before buying, because no two clusters are alike. Use your measured iSCSI gateway workload, including peak IOPS and throughput per LUN, as the baseline, and ask your hardware supplier to show benchmark results for that profile.
Confirm the copy, NVMe-oF and tooling docs
The documentation you must check before the plan is final covers three areas: the method used to copy data from iSCSI-backed images to the target, the NVMe-oF gateway configuration, and any import or export tooling. This page does not name a specific data copy tool, because the official documentation for your release must confirm one first. Read the release notes and gateway pages for your target version, record the exact commands and supported client versions, and test the chosen method in a lab that matches your production images.
How the native RBD data path changes security and network planning
When Linux hosts stop going through a gateway and talk to the cluster directly, the security boundary moves from the gateway to every client. Each host needs a cephx identity with capabilities scoped to the pools and images it uses, and that key becomes the thing you protect, rotate and revoke. Today the gateway holds the credentials and the initiators only see LUNs, so the gateway is a single place to manage access. With native RBD, each Linux host runs rbd map or librbd under its own cephx user, so you need a key per host or per workload, a documented rotation process, and a check that no client can reach pools it does not own. Confirm which cephx capabilities your current gateway configuration grants, because those are the starting point for the new users.
Network reachability changes too. Clients must reach the monitors and every OSD that holds data for the images they use, which is a much wider set of endpoints than the gateway addresses you may have exposed before. Ceph's messenger v2 protocol listens on port 3300 for monitors. Its secure mode encrypts all traffic after authentication with AES-GCM and adds integrity checks, which matters when client and cluster traffic crosses a segment you do not fully control. Whether secure mode is enabled in your cluster, and whether your kernel and library versions on each host support it, is something to confirm before cutover. The CPU cost and the configuration steps depend on your hardware and client versions, so benchmark a representative host before deciding on a rollout order.
The hardware recommendations advise bonding network links across two switches, and that advice becomes more important once every client depends on direct reachability to the OSDs. A single switch failure on the client side would cut off every VM disk on that host rather than a shared gateway pair that you could fail over. Plan the bond mode, the switch redundancy and the client VLAN layout together with the cephx scoping, and test a switch failure during a pilot cutover. Document which networks carry public and cluster traffic, which hosts can reach them, and how firewall rules are maintained as hosts are added.
The sequence from inventory to decommission with a parallel run
This sequence moves VMware and Linux hosts off the iSCSI gateway in controlled stages, so each host is verified on Ceph before its traffic is cut over. The result is a documented, reversible migration with a clear point at which the gateway is retired.
Inventory LUNs, initiators and multipath settings
The team lists every LUN exported through the iSCSI gateway, the initiators that map to each one, the multipath configuration on each host, and the capacity in use alongside the rate of change for each volume. Your storage and virtualization administrators provide access to the gateway configuration, host configurations and existing monitoring data, and DeployOpen returns a written inventory that flags unsupported or unusual configurations. The step is done when you have signed off on the inventory and every LUN has an owner, a host list and a target migration method.
Build the target cluster with cephadm
The team deploys the target Ceph cluster with cephadm on a release that is still in support. Based on the Ceph releases page, Tentacle (first released November 18, 2025) or Squid (estimated end of life October 31, 2026) are the options to evaluate, and you should confirm the current support status before choosing. The client provides hardware that meets the Ceph hardware recommendations, including network links bonded across two switches and separate drives for each OSD. The step is done when the cluster passes health checks, RBD pools and cephx users are created, and the team has run a benchmark that matches the sizing decisions.
Pilot with non-critical hosts
The team moves a small set of non-critical hosts to native RBD access, using the VMware NVMe-oF path or the Linux RBD client as appropriate for each host. Your operations team selects the pilot hosts and agrees on the test plan, including application checks, failover tests and rollback steps. The pilot is complete when the test plan has been run end to end, any issues are recorded with fixes, and your team has signed off that the pilot hosts behave as expected under load and during path failure.
Run both paths in parallel and verify writes
Both the iSCSI gateway and the Ceph target serve data during this stage. Each migrated volume is kept in sync with its source, and the team verifies that writes land on the target with matching checksums or application-level checks. Your team confirms which applications can tolerate this period and which need a stricter plan. The stage closes when every volume in scope has passed verification for the agreed period, and you have a record of the sync status and any discrepancies found.
Cut over in agreed windows with a rollback plan
Each host or application group is switched to the Ceph target during a maintenance window that you approve in advance. The rollback plan for each window names the steps to return the host to the iSCSI path, the data state at which rollback is still safe, and the people who make that decision. DeployOpen runs the cutover with your staff present, and the change record captures the start and end times, the verification results and any rollback events. The step is done when every host in the window passes its post-cutover checks and no host is still reading from the gateway.
Decommission the gateway after an observation period
After all hosts have been cut over, the iSCSI gateway stays in place, unused but available, for an observation period that you define in the plan. During this period, monitoring confirms that no initiators are connected and that the Ceph cluster performs as expected. The gateway is then removed, its configuration and final state are archived, and DNS, monitoring and backup jobs are updated. The step is done when your team confirms the decommission in writing and the archived configuration is stored where your audit process can reach it.
What drives the cost and duration of the migration
This table lists the factors that change how much work the iSCSI gateway replacement takes, how each one moves effort, and what to ask for in a written quote. Use it to compare your environment against each driver before requesting a scope, since most cost sits in the parallel run and cutover window rather than in the gateway configuration itself.
| Driver | Effect on effort | What to confirm in a quote |
|---|---|---|
| Number and type of hosts | Each VMware host and each Linux initiator needs its own path change, validation and rollback step. Mixed fleets need two procedures. | Host count by platform, ESXi and Linux versions, and whether hosts are in clusters with shared datastores. |
| Number and size of LUNs | More LUNs mean more mappings to recreate on RBD or NVMe-oF, more data copies and longer verification. Large LUNs extend copy time per volume. | LUN inventory with size, backing image layout, and whether thin provisioning is used. |
| Change rate during the parallel run | Writes that continue during copying force repeated sync passes or a longer final freeze. High change rates raise the risk of a missed cutover window. | Measured write rate per workload over a representative period and the proposed sync method. |
| Required downtime per workload | Workloads that accept a short outage are cheaper to cut over than those needing near-zero interruption, which need more staging and rehearsal. | Downtime tolerance per application, agreed maintenance windows, and who approves each window. |
| Multipath and datastore complexity | Multipath policies, datastore extents and path ordering must be mapped one by one. Shared datastores across many hosts add coordination work. | Current multipath configuration, datastore-to-LUN mapping and any unusual path policies. |
| Team skills on each host platform | Gaps in VMware or Linux storage administration lengthen design and testing, and increase the number of review cycles. | Named skills on the client side, who performs host changes, and whether knowledge transfer is included. |
| Target release and its support window | Choosing a release near end of life may force an earlier upgrade later. Confirming the current active release list before design avoids rework. | Target Ceph release from the Ceph releases page, its estimated end-of-life date, and the planned upgrade path. |
| Network changes, bonding and switch design | Moving NVMe-oF or RBD traffic may need new VLANs, bonded links across two switches, and MTU checks. Each change adds validation and change approvals. | Current network topology, bonding mode, switch redundancy, MTU and any new addressing requirements. |
| Testing scope | Functional tests, failover tests, performance comparison and rollback drills each add effort. A broader test matrix raises both one-time and cutover work. | Test cases per platform, acceptance criteria, performance baseline method and rollback rehearsal count. |
What usually goes wrong during an iSCSI to RBD cutover and how to prevent it
Most failed cutovers trace back to a few planning gaps: the target cluster was sized on paper, a single path carried the traffic, client credentials were improvised, or the rollback was never exercised. Each card below names one of these failures and a check the team can complete before any production host moves.
Unbenchmarked target cluster
Teams often size the RBD cluster from the iSCSI gateway's old capacity and skip measurement. The Ceph hardware recommendations advise benchmarking before buying, because no two clusters are alike, and sizing memory for peak use rather than calm periods. During a cutover, recovery, scrubbing and migrated VM load all land at once. Before cutover, run a benchmark that reproduces the peak I/O profile of the busiest iSCSI LUNs and confirm OSD memory headroom under recovery load.
Single network path during cutover
During cutover, hosts move traffic between the old gateway and the new cluster while both paths stay active. If the storage network depends on one switch or one uplink, a single failure stalls every migrated disk at once. The hardware recommendations call for bonding network links across two switches. Before the first host moves, pull one switch port or uplink in a test window and confirm that I/O continues on the surviving path without client timeouts.
Cephx credential misconfiguration
Hosts that connect to RBD need a cephx user with the right capabilities on the right pool, and a keyring that is present, readable and not stale. Copying a keyring from a test cluster or granting broad capabilities to save time often works until a second pool or namespace is added. Before cutover, confirm each host authenticates with its own cephx user and that the capabilities are limited to the pools that host uses, then test a read and a write on a non-production image.
Untested rollback path
A cutover plan is only as good as its way back. Teams often document the forward steps and assume the reverse will work, but a rollback to the iSCSI gateway depends on LUN mappings, initiator sessions and data written during the window. If writes land only on RBD, returning to the gateway can lose or split data. Before cutover, rehearse a full rollback on one host with a written trigger, a named decision owner and a timed reversal of the mapping.
Out-of-support release chosen for target
Choosing the target release on familiarity or on the version already in a lab can place a production cluster on a release that is no longer maintained. Ceph's releases page lists which releases receive periodic backports and security fixes, and each release has an estimated end of life. Before cutover, check the target release against the active releases list and confirm the planned operating window covers the cluster's expected life, including the next upgrade.
Questions buyers ask before replacing the iSCSI gateway
How much does it cost to replace a Ceph iSCSI gateway?
There is no fixed price. The cost follows from a written scope, and the main components are discovery and inventory, design of RBD images and pool layout, cephx user creation, multipath and initiator configuration on each host, test cutovers, the production cutover window, documentation and handover. Most of this is one-time work. Recurring costs appear only if you buy ongoing support or retainers after the migration. Cost rises with the number of hosts, the hypervisor mix, the number of LUNs and the volume of data to move, and falls when hosts are standard and the change calendar is flexible. When you request a quote, ask for an itemized scope, stated assumptions, explicit exclusions, who performs the host-side work, the rollback approach and the milestones that trigger payment.
How long does an iSCSI to RBD migration take?
The duration depends mainly on the number of hosts and how quickly the data changes while it is being copied. More hosts mean more path changes, test cycles and change windows. A high write rate on a LUN means more delta synchronization before a clean switch, which extends the schedule. Application dependencies, the number of virtual machines per host and the business calendar for change freezes also shape the plan. No general duration is published here, so ask the vendor documentation for your Ceph release and your chosen copy method to confirm the mechanics, then ask for a milestone-based schedule that shows host groups, pilot results and the dates on which each cutover window opens.
What does our internal team need to provide for the migration?
Your team supplies the facts and access that no outside party can reconstruct. Provide an inventory of LUNs, the hosts that map them, initiator identifiers, current multipath settings and the virtual machines on each datastore. Name an owner for every application and test owners who can confirm that a workload works after the switch. Agree change windows and an approval chain for each window. Grant hypervisor administration access, network access to the Ceph public network and the gateway, and the ability to change switch and VLAN settings if needed. Finally, confirm backup status and restore tests for every LUN before any data moves.
Can our own team replace the iSCSI gateway without outside help?
A capable team can do it, provided it has the skills and time to cover the full path. The work requires cephx capability design for RBD clients, RBD image creation and sizing, multipath configuration on Linux and on the hypervisors, NVMe-oF or native RBD initiator setup, sequencing of host cutovers and tested rollback. The highest risk sits in the cutover, where a missed path or an unexpected dependency can stall a workload. Many teams split the work: they keep planning, pilot hosts and test ownership in-house and bring in an implementation partner for the cutover sequence and the runbooks. Either way, run a pilot on a non-critical host before touching production, and make sure at least two engineers can execute each step.
Can we avoid downtime completely during the cutover?
Downtime cannot be guaranteed in advance, but a planned parallel run reduces its scope. You can build the new RBD targets next to the existing gateway, copy data, test each host against the new path and move hosts one group at a time, so the old path remains available for rollback. This limits each interruption to one host or one application. A parallel run does not remove every risk. Applications with in-memory state may need a brief quiesce, path failover behavior varies between initiators and hypervisors, and undocumented dependencies can surface only at the switch. Ask for a per-host cutover plan that states the expected interruption, the rollback trigger and the test that confirms success.
Is the NVMe-oF gateway the right target for every VMware host?
Not necessarily. Confirm the choice against the Ceph documentation for your release before you commit, starting with the active releases page to confirm which versions are maintained. Check whether the NVMe-oF gateway supports your cluster version, and whether the ESXi versions in your estate include an initiator that works with it, along with the multipath behavior you need. Hosts that cannot run a supported initiator may need another path, such as a different hypervisor build or a virtualization layer that talks to RBD directly. Build a compatibility matrix per host type, record the versions tested in your own lab, and treat any host without a confirmed combination as an open item rather than a finished migration.
How DeployOpen scopes, delivers and supports the migration
DeployOpen plans the move off the Ceph iSCSI gateway as a written, fixed-scope engagement inside your own infrastructure. The work starts with an inventory of your VMware and Linux initiators, then moves each host group to its target path in a sequence you can roll back at every step.
How we scope
- Inventory every iSCSI initiator, LUN, host group and application that depends on the gateway today.
- Record which hosts move to the NVMe-oF gateway, which move to native RBD, and which need a bridge period.
- Agree the cutover windows, rollback points and acceptance tests in writing before any host is changed.
How we build
- Stand up the target NVMe-oF gateway or RBD client configuration in a lab that mirrors your production network.
- Test multipath behavior, failover and performance on representative LUNs before any production data moves.
- Run each cutover in stages, keeping the iSCSI path available until the new path passes validation.
How we hand over
- Document the final host-to-target mapping, client configuration, credentials handling and cutover runbook.
- Walk your operations team through failover drills, health checks and how to retire the old gateway safely.
- Deliver the written record of what changed, what was tested and what remains open for your own sign-off.
How we support
- Ongoing support, upgrades and monitoring of the new paths are available as a retainer after handover.
- Confirm with your storage and hypervisor vendors which support statements apply to each target path.
- Ask any quote to name the owner of host-side changes, change approvals and post-cutover troubleshooting.
