Ceph NVMe over TCP gateway for VMware and Linux: when teams need SAN-style block storage from Ceph
The Ceph NVMe/TCP gateway presents RBD images to hosts as NVMe namespaces over standard TCP networking, so VMware and Linux initiators can use Ceph-backed block devices without installing a Ceph client on every host. It is the right path when an estate already runs iSCSI or Fibre Channel SAN LUNs, the hypervisors and database servers need block devices, and the storage team wants Ceph to hold the data. If your hosts can run the native RBD client, or your workloads only need files or objects, a gateway adds moving parts you may not need. Treat the gateway as a way to reach Ceph, not as a separate storage product with its own data protection.
The feature exports RBD images through NVMe subsystems that run on gateway nodes, grouped so that one node can take over for another. Each design decision follows from that. You need to decide how many gateway nodes to run, which ones serve active paths and which stand by, how initiators on each VMware cluster and Linux host discover and multipath to those paths, and how many subsystems and namespaces each gateway group will carry. Confirm every one of these limits, the supported initiator versions and the failover behavior against the gateway documentation for the Ceph release you plan to run, because these capabilities change between releases and the figures in older guides may no longer apply.
The gateway does not replace the cluster's own design work. Replication, failure domains, network sizing and recovery behavior still belong to the Ceph cluster, and RBD snapshots and mirroring still provide the copy and recovery features your backup plan depends on. Gateway services are deployed and placed through cephadm in the normal way, so the cephadm documentation is the next read for service placement. Before committing, list each SAN feature your estate uses today, such as multipath policy, snapshots, replication, quality of service and host access control, and check each one against the gateway's documented support. Any gap becomes a design decision or a cost item in the scope.
Which decisions shape the gateway design
Six decisions determine how an NVMe over TCP gateway behaves under load and failure. They interact: the number of gateways sets how much failover capacity you have, and the subsystem layout sets what each VMware or Linux host can see and how multipath behaves.
Gateway group size
Deciding how many gateway nodes serve each subsystem sets your failure tolerance and your per-node load. Two nodes give a simple standby pair, while more nodes spread I/O but add configuration and testing work. Check how many gateway nodes you can lose before a datastore goes read-only or stops responding, and confirm that every node can reach every OSD host on the storage network.
Active/standby or multi-gateway access
An active/standby pair sends each subsystem's traffic through one gateway at a time, which keeps path behavior simple but concentrates throughput on that node. Multi-gateway access lets hosts use several paths at once, which raises throughput and makes failover behavior more complex to test. Confirm with your gateway version which access modes are supported before choosing.
Subsystem and namespace layout
Subsystems group namespaces that a set of hosts can reach, and each namespace maps to one RBD image. Grouping by datastore or by tenant limits blast radius and makes access control clear, while one large subsystem is easier to manage but exposes more images to every connected host. Plan the subsystem count and the namespace count per subsystem against the gateway limits your version documents.
Storage network separation
NVMe/TCP traffic from hosts to gateways should run on its own VLAN or subnet, separate from management traffic and from Ceph's internal cluster network. Mixing these lets a backup job or a Ceph recovery burst starve datastore I/O. Check that MTU settings match on every switch port in the path and that the storage network is bonded across two switches, as the Ceph hardware recommendations advise.
Host multipath configuration
Each ESXi or Linux host needs multipath settings that match the gateway's failover model, or path failures will show up as long I/O stalls instead of quick switchover. Decide the path selection policy, the number of paths per namespace, and the timeouts before the first datastore goes live. Test by pulling a gateway node during a write workload and measuring what the guest sees.
Whether VMware or Linux drives the design
VMware datastores and Linux hosts impose different constraints, including how each platform discovers NVMe controllers, handles path state and reports errors to guests. If VMware drives the design, validate the exact ESXi build and its NVMe/TCP adapter support first. If Linux hosts dominate, the nvme-cli and kernel versions on each host become the controlling inputs. Confirm supported combinations with the vendors before committing.
Configuration options and when each fits
This table compares the main design choices for exporting RBD images over NVMe/TCP to VMware and Linux hosts, with the situations where each fits and the operational cost it adds. Read each row as a decision: pick the option that matches your host count, failure tolerance and change control, then confirm the specifics with your team before building.
| Option | When it fits | Operational cost |
|---|---|---|
| Gateway group sizing | Small estates with a few hosts and one storage team can start with a minimal gateway group. Larger VMware clusters or many Linux initiators call for more gateways so that a single gateway failure does not stall many datastores. | Each added gateway adds configuration to keep consistent, monitoring to watch and capacity to test under failover. Ask the vendor or your partner to confirm the supported gateway count for your release. |
| Failover model (active/standby) | Suits teams that want a predictable, documented takeover path and can accept a short I/O pause during path changes. Good fit when VMware and Linux multipath behavior is already tested in your environment. | Standby capacity sits idle, and failover must be rehearsed regularly. Confirm how initiators reconnect and how long I/O pauses are expected to last in your own test runs. |
| Subsystem grouping per cluster | Fits when one VMware cluster or one Linux application group should share the same access boundary, so host NQNs and images map cleanly to a single owner. | Many subsystems multiply names, access lists and documentation. Changes to shared images need coordination across each group. |
| Subsystem grouping per workload | Fits when different workloads need separate change windows, separate host access lists or separate image lifecycles, such as databases beside general virtual machines. | More subsystems mean more objects to audit, back up and reconcile. Automation for creation and removal becomes more valuable as the count grows. |
| Host access control by host NQN | Fits environments where each host must be named explicitly in an access list, which supports least-privilege mapping and clear audit evidence. Useful when hosts are added and retired through a controlled process. | Every host added, replaced or re-imaged needs an access list update. Missed updates cause connection failures, and stale entries can leave retired hosts with access if they are not removed. |
| Dedicated storage network path | Fits production datastores where latency and throughput must not compete with other traffic, and where you can provide separate switches or VLANs for NVMe/TCP. Aligns with the bonded, two-switch network approach in the Ceph hardware recommendations. | Separate switching, cabling, addressing and firewall rules add hardware and change work. Teams must maintain a second network design through every expansion. |
| Shared network path | Fits labs, pilots and smaller estates where the existing fabric has capacity and the team wants fewer moving parts. Can be acceptable early, provided traffic is measured under load. | Storage and virtualization traffic compete on the same links, so congestion troubleshooting becomes harder. Plan a benchmark before committing, since no two clusters behave alike. |
| Memory and sizing for gateway hosts | Sized for peak use rather than average use, so gateways keep serving during recovery, rebalancing and failover events. | Oversized gateways cost hardware and licensing you may not use. Undersized gateways fail under exactly the conditions when you need them most. Ask for a load test plan as part of the quote. |
How the NVMe/TCP gateway is set up, step by step
The sequence builds the gateway in dependency order, from a confirmed Ceph release to RBD images that VMware and Linux hosts can reach over NVMe/TCP, with a check at each stage so faults surface before production datastores depend on them.
Confirm the Ceph release and gateway support
The storage team confirms which Ceph release the cluster runs and whether the NVMe/TCP gateway is supported on that release and on the cephadm version in use. Tentacle (20.2.x) and Squid (19.2.x) are the actively maintained releases on the Ceph releases page, so check the gateway documentation for those branches. Record the supported VMware ESXi and Linux kernel or nvme-cli versions alongside it. Ask the vendor to confirm the multipath and failover behavior for your exact release. Done when the compatibility record is signed off by the storage and virtualization leads.
Deploy gateway services through cephadm
The Ceph administrator places gateway daemons on dedicated hosts using a cephadm service specification, so cephadm manages them as containers alongside the monitors and managers. Keep at least two gateway hosts on separate failure domains for active/standby or multipath paths. Apply the spec, then confirm the daemons start on each host. Check the cephadm documentation for the exact spec fields for your release. Done when orchestrator output shows every gateway daemon running and the health status is clean.
Create the RBD pool and images
The storage administrator creates a replicated RBD pool sized for the datastores and initializes it for the RBD application so the gateway can use its images. Create one image per datastore or LUN, with sizes agreed with the VMware team. Use the replication level and failure domains from your cluster design, since the gateway inherits that protection. Verify each image with the RBD listing and info commands, checking pool, size and features. Done when every planned image appears with the expected size.
Define subsystems and namespaces
Create one NVMe subsystem per host group or datastore set, then attach each RBD image as a namespace inside the right subsystem. Record each subsystem's NVMe Qualified Name (NQN) and namespace ID in the configuration record, because initiators and access rules depend on them. Keep subsystems small enough that a fault or access change affects a bounded set of datastores. Confirm any subsystem and namespace limits for your release before you finalize the layout. Done when the subsystem listing matches the design document one to one.
Set host access controls
The administrator restricts each subsystem to the host NQNs that should reach it, so unlisted initiators cannot connect. Collect the initiator NQNs from every Linux host and every ESXi host before applying the allow list, and document the owner of each entry. Review the list with the security team, since it is the main access boundary between hosts on the same network. Test with one unlisted initiator. Done when that initiator is refused and every listed host connects.
Configure NVMe/TCP initiators on Linux and VMware
On Linux hosts, the administrator installs nvme-cli, runs discovery against the gateway addresses, and connects each subsystem, then makes the connections persistent with the distribution's supported method. On VMware, the virtualization team enables the NVMe over TCP software adapter on each ESXi host, adds the gateway targets, and creates datastores from the discovered namespaces. Verify that every namespace appears on each host with the expected capacity. Done when the namespaces are visible on all hosts and a test datastore mounts cleanly.
Verify multipath and failover behavior
The team checks that each namespace shows one path per gateway through the initiator's subsystem and multipath tools, then runs a controlled test. With a test workload running, it stops one gateway daemon and confirms that I/O continues on the surviving path and that paths recover when the daemon returns. Repeat the test for each gateway host and record the observed failover time. Confirm the expected path states with the vendor documentation for your release. Done when the test results are recorded and accepted by the storage and virtualization leads.
VMware and Linux consumers need different checks
VMware administrators judge the gateway by what ESXi hosts and vCenter see, and by how the datastore behaves under load and during failover. Support for NVMe over TCP depends on the ESXi release, the software adapter and the driver build, so confirm each combination in VMware's current compatibility guidance before you commit a datastore to the design.
- Adapter and driver support: Confirm that your ESXi release supports NVMe over TCP, and record the adapter and driver versions your hosts need.
- Multipath behavior: Test how the host's multipathing handles path loss to each gateway, and record how long I/O pauses during a failover in your own test.
- vCenter integration: Verify what vCenter reports for the namespace device, datastore creation, capacity and alarms, and confirm which offload features apply to this path in VMware's documentation for your version.
- Guest behavior during failure: Check whether virtual machines stay responsive during a gateway restart, how the datastore marks paths as dead, and how HA events behave while I/O is paused.
Ask the VMware and gateway vendors to confirm these points in writing for your exact ESXi build before go-live.
Linux administrators own the initiator side: kernel modules, the nvme-cli toolchain, multipath configuration and whether connections survive reboots and network interruptions. Support varies by distribution and kernel release, so confirm the NVMe/TCP initiator support for your distribution and the gateway vendor's published initiator matrix before you standardize the build.
- Initiator modules: Confirm that the nvme_tcp kernel module is available for your kernel and that it is included in the image or initramfs your hosts boot from.
- Discovery and connect: Verify discovery against each gateway address, then connect with nvme-cli using the subsystem NQN the gateway publishes, once from each path interface.
- Multipath tooling: Choose native NVMe multipath or device-mapper multipath according to your distribution's default and the vendor's guidance, then confirm path states during a simulated gateway failure.
- Persistent connections: Configure connections to reconnect after a reboot or network interruption, and test a reboot with the gateway down and then up.
Record the kernel version, nvme-cli version and multipath settings for each host in your build documentation, so that a later kernel update can be checked against the same test.
Both teams depend on the same gateway group and network, so some checks belong to neither team alone. These checks show where failover happens, whether storage traffic has its own predictable path, and whether the gateway can carry the subsystems you plan to map. Use this list as the common acceptance list for the whole design.
- Path independence: Confirm that each host has paths through separate network interfaces and separate switches, following the bonded two-switch pattern in Ceph's hardware recommendations.
- Failover timing: Measure the time from gateway failure to resumed I/O for each workload, and compare the result with what your virtual machines and applications tolerate.
- MTU and congestion: If you use jumbo frames, verify the MTU end to end, then test under storage load alongside other traffic and watch for retransmissions.
- Subsystem and host limits: Confirm the maximum number of subsystems, namespaces and connected hosts for your gateway version before you map datastores and volumes to it.
Assign each failed check to one named owner before go-live, so that the VMware and Linux teams do not each assume the other is tracking it.
Security and operational considerations for exported block storage
An NVMe over TCP gateway turns RBD images into block devices that every connected host can reach, so the controls that matter sit around the export: who may connect, which network carries the traffic, how credentials are held and how changes are made. The cards below cover the decisions to settle before the first datastore goes live.
Host access control per initiator
Each subsystem should admit only the specific host NQNs that need a given image, not a wildcard. Map every ESXi host and Linux initiator to its own NQN and to the RBD images it may mount, and confirm how your gateway version enforces that list. Review the allowed-host list for each subsystem after every host is added or retired, because a stale entry is an open path to the volume.
Network isolation of NVMe/TCP traffic
NVMe/TCP carries unencrypted block I/O across the network, so place it on a dedicated storage VLAN or subnet that only the hosts, gateways and Ceph OSD networks can reach. Keep the front-facing gateway addresses off general user networks and firewall the NVMe/TCP listener port to known initiators. Confirm the port and the path redundancy with your gateway documentation before you write firewall rules.
Encryption in transit trade-offs
Ceph's messenger v2 secure mode encrypts traffic between Ceph daemons and clients with AES-GCM and adds integrity checks, but it covers the Ceph side only. The NVMe/TCP hop between initiators and gateways is a separate decision, and enabling msgr2 secure mode adds CPU cost on the gateway and OSD hosts. Measure throughput with both settings on your hardware before choosing, and document the hop you left unencrypted.
Credential handling for gateway services
Gateways authenticate to the Ceph cluster with cephx keys, and the dashboard or orchestration layer may hold further credentials. Create a dedicated Ceph user per gateway with only the capabilities its images require, and rotate those keys on a schedule you can execute. Store keyrings through the cephadm secret handling your team already uses, and avoid copying them into configuration management repositories.
Monitoring gateway health and failover events
Active/standby failover is only useful if someone sees it happen. Alert on gateway daemon state, path changes seen by initiators, and Ceph health warnings that affect the backing images, and log each failover with its timestamp. Confirm which metrics your gateway version exposes and how the dashboard surfaces them, then test an alert by stopping a gateway in a lab before production use.
Change control for subsystem edits
Subsystem, namespace and host mapping changes can detach a live datastore in seconds, so treat them as production changes. Keep the gateway configuration in version control, require a second reviewer, record the rollback command beside each change, and schedule edits outside backup windows and recovery work. Ask your implementation partner to include a change log and a tested rollback procedure in the handover.
Questions buyers ask before building an NVMe/TCP gateway
Does NVMe/TCP replace iSCSI for Ceph block storage?
Not automatically. NVMe/TCP is a separate transport that presents RBD images to hosts over the network, so it is a choice you make per workload alongside any existing iSCSI path. The decision depends on which hypervisors and Linux hosts you must serve and which initiator versions, multipath behavior and gateway releases each of them supports. Before committing, list every consumer, confirm its support in the current vendor compatibility documentation, and plan a phased cutover in which both paths remain available until testing is complete. Your Ceph cluster itself stays the same, because the gateway sits in front of the RBD images it exports.
What drives the cost and effort of an NVMe/TCP gateway?
The main one-time costs are the engagement itself, the gateway hosts, the network changes and the test hardware. Most engagements are fixed-scope, so the written scope sets the price, and DeployOpen does not publish a price list. Effort rises with the number of gateway nodes and gateway groups, the number of subsystems and namespaces, the mix of VMware and Linux initiators, any network redesign, and how deep the failover testing goes. Recurring costs include hardware refresh, patching and any support retainer. Ask for an itemized quote that separates design, build, testing, documentation and handover, and ask which items depend on your own hardware or network work.
How long does an NVMe/TCP gateway build take?
The duration depends mostly on factors outside the gateway software: the health of the existing cluster, how quickly network changes can be made, the number of consumers to validate, and how many change windows you can use. Lead times for switch ports, optics and new gateway hardware often matter more than configuration work. Testing cycles add time because each failover scenario needs to be run, observed and repeated after fixes. Ask for a milestone schedule in the written scope that names the dependencies you must provide, such as racks, VLANs, approvals and maintenance windows, so the timeline reflects your environment rather than a generic estimate.
What does the build need from our own teams?
Three groups usually carry work that a partner cannot do for you. The network team confirms VLANs, MTU settings and bonded links spread across two switches, which the Ceph hardware recommendations advise for cluster networks. The storage team provides access to the Ceph cluster, pool and RBD image decisions, and cephadm operations. The virtualization team owns VMware datastore design, multipath settings and host maintenance windows. Security and change management also need to approve access, credentials and production changes. Name a decision owner for each group at kickoff, and confirm who will hold the test hardware and lab access before the schedule is fixed.
Can our team build an NVMe/TCP gateway without a partner?
Yes, if your team already runs Ceph and has the time, lab hardware and skills to test failure cases carefully. Your team should be comfortable with cephadm, which deploys Ceph as containers and has been available since v15.2.0, and with RBD, multipath and the NVMe/TCP initiator on each host type. The harder part is usually not the first working configuration but the repeatable failover tests, runbooks and documentation that operations staff can follow later. A partner can supply that structure and an independent review, while your team keeps ownership. Decide which parts you will build, which you will review, and which you will hand over, and write that split into the scope.
How is failover tested before the gateway goes live?
Failover testing should be written as numbered cases with expected results, run in a non-production environment first and then repeated after any configuration change. Typical cases include stopping a gateway node, dropping one network path, losing one switch in a bonded pair, restarting a gateway service during active I/O, and bringing a failed node back into service. For each case, record what the VMware datastore and Linux hosts see, whether I/O resumes, how long recovery takes in your environment, and whether the Ceph cluster stays healthy throughout. Confirm the exact failover behavior with the current gateway documentation for your release, and keep the results as evidence for change approval.
What does ongoing support cover after the gateway is live?
Ongoing support is available as a retainer and typically covers monitoring review, patching, upgrades, updated documentation and periodic failover retesting. Upgrades matter because Ceph release lifecycles are finite: the Squid release has an estimated end of life of October 31, 2026, and Tentacle has an estimated end of life of June 1, 2027, according to the Ceph releases page. Ask the support provider which releases it covers, how it handles gateway and Ceph upgrades in change windows, who responds to incidents and within what hours, and whether retesting after each upgrade is included. Confirm the terms in writing, because support scope is set by the agreement and not by this page.
How DeployOpen scopes and delivers an NVMe/TCP gateway build
Every NVMe/TCP gateway build starts with a written scope, and DeployOpen agrees that scope, the deliverables and the timeline with you before any work begins. For this topic the scope should name the RBD images to be exported, the gateway hosts and their count, the subsystems and namespaces each VMware cluster or Linux host group will see, the failover behavior you expect, and who on your side owns the switches, host NICs and VMware configuration. Ask for the scope to list which pieces are excluded, such as storage-array migration, multipath tuning on every host or application testing. Most engagements are fixed-scope, so a change to any of these items should be recorded as a written change before it is built, and the quote follows directly from that document.
Before any gateway is deployed, DeployOpen runs a design review of the network and failure domains. The review checks that the gateway and host traffic runs on a dedicated, bonded network across two switches, as the Ceph hardware recommendations advise, and that gateway hosts do not share a single point of failure with the Ceph monitors or OSD hosts they serve. It also decides how gateway groups are placed and how active/standby ownership is expected to move during a host loss. Confirm with your network team which MTU, VLANs and routing your hosts will use, and confirm with your VMware and Linux owners which multipath settings and timeouts their hosts already run. The review output is a design document you keep, not a sales artifact.
Implementation happens inside your own infrastructure, using your accounts, data centers or air-gapped network, and your change windows. Verification is a required deliverable: failover of an active gateway to its standby is tested while I/O is running, host connectivity is checked from every initiator type in scope, and the results are recorded against the scope's acceptance criteria. Documentation and handover cover the deployed configuration, the runbook for adding namespaces, and the steps for failover and failback. Optional retainers can cover ongoing support, upgrades and periodic failover testing after handover. Ask any partner to state the retainer scope, response expectations and what happens to the design document when a Ceph release changes gateway behavior, and to confirm how the gateway release you run is supported by your Ceph version.
