Should you run separate public and cluster networks for Ceph?
A separate cluster network carries OSD replication, heartbeat and recovery traffic, while the public network carries client traffic and monitor traffic. The decision is whether those two flows should share one set of links or run on different ones. Ceph lets you set both networks at bootstrap through cephadm, using the public_network and cluster_network settings, so the choice should be made before hosts are added. Changing it later means reviewing the configuration on every node and confirming that OSDs reach their peers on the intended network. Before you order switches, confirm how your Ceph release documents these two roles, because the guidance on what runs where is the basis for the rest of the network design.
Teams usually consider the split when the cluster has enough write volume or enough drives per host that recovery could compete with client I/O. After a host or drive failure, OSDs move large amounts of data between themselves, and on a shared link that traffic can slow the reads and writes that applications depend on. A separate cluster network keeps that replication traffic off the links that clients use. The cost is more cabling, more switch ports, more addresses to plan and more configuration to verify. A small or lightly loaded cluster can often run well on one bonded network, and the trade-off should be tested with the hardware you plan to buy rather than assumed from the diagram.
If you split the networks, bond the links on each host and spread the bond members across two switches, as the Ceph hardware recommendations advise, so that one switch failure does not isolate a host. Monitors must be reachable on the public network, and clients connect to monitors on the msgr2 port 3300 unless your design changes it, so confirm that firewall rules allow that path. Confirm the port ranges that OSDs and managers use for your release before writing firewall rules, and test a host failure on the staging cluster to see which traffic moves where.
Network layouts and when each fits
This table compares three ways to lay out Ceph networking: one shared network, separate public and cluster networks, and bonded links across two switches. Read each row against your replication load, recovery windows and switch design. The main takeaway is that splitting traffic isolates replication from client I/O, while bonding across two switches protects both networks from a single switch failure.
| Layout | What it isolates | When it fits | Main trade-off |
|---|---|---|---|
| Single shared network | Nothing: client, monitor and OSD replication traffic share one path | Small or lab clusters with light writes and modest recovery needs, where the team accepts contention | Recovery and rebalancing can slow client I/O, and one congested link affects every traffic type |
| Separate public and cluster networks | Client and monitor traffic on the public network, OSD replication and heartbeats on the cluster network | Clusters where replication or recovery traffic is heavy enough that it should not compete with clients | Two address ranges, two sets of ports and switches to plan, and more cabling and configuration to verify |
| Dedicated cluster network with bonded links | Replication traffic on its own bonded pair, with link redundancy on each network | Production clusters where the hardware recommendations' advice on dedicated replication networks and bonding across two switches applies | More NICs, switch ports and bonding configuration to build, test and monitor |
| Bonded links across two switches on a shared network | Link and switch failure for the single network, but not traffic type | Clusters that need switch redundancy but cannot yet justify a second network | Redundancy improves, yet recovery traffic still competes with client I/O |
| Separate networks without bonding | Traffic type only, with no link or switch redundancy | Rarely a production target; some labs or staged builds | A single switch or NIC failure can take down one network, which the hardware guidance advises against |
Setting the cluster network at bootstrap or later
When you bootstrap with cephadm, the public and cluster networks can be set before any OSD exists, which is the simplest point to get them right. The bootstrap command takes the first monitor address on the client-facing network, and it can also take the replication subnet as a CIDR. Decide both subnets before the first host joins, because changing them later means editing configuration on a live cluster.
- Client network first: the monitor address sits on the public network that clients, the dashboard and the gateways use.
- Cluster network CIDR: pass the replication subnet to bootstrap on the dedicated interface, for example
10.20.0.0/24(illustrative). - Hosts follow the subnets: every host added later over SSH needs an address in both subnets, so confirm that before adding it.
- Bonded links: if the replication interface is a bond across two switches, configure the bond before bootstrap so the subnet sits on the bond device.
- Confirm flags: option names and syntax change between releases, so check the exact bootstrap flags in the cephadm documentation for your version.
Check that every host has interfaces configured in both subnets before you run bootstrap.
On an existing cluster, the networks are stored in the monitor configuration database and changed with the configuration commands. The two keys are public_network and cluster_network. Changing the cluster network affects how OSDs talk to each other, so the change must be rolled out one daemon at a time. Changing the public network affects monitors and every client, so it needs a planned window with the client owners informed.
- Central config keys: set
public_networkandcluster_networkwithceph config set, and read them back before you touch any daemon. - Record the current state: save the output of
ceph config dumpso you can restore the previous values if the change fails. - Rolling OSD restarts: OSDs pick up the cluster network when they start, so restart them host by host and wait for health to recover between hosts.
- Set flags for maintenance: setting the
nooutflag during restarts stops unnecessary rebalancing, and it must be cleared when the work finishes. - Monitor address changes: a change to the public network can also require changing monitor addresses, which is a larger task than the cluster network change.
Confirm the exact sequence and commands for your release in the cephadm documentation, and rehearse the change on a non-production cluster first.
After either path, check that the networks are what you intended, both in the configuration and on the wire. The configuration can show the right values while a host still advertises an address on the wrong subnet, or a firewall still blocks a port. Verification should happen before the cluster takes production load and again after any later change.
- Configuration readback: run
ceph config getfor both keys andceph config dumpto find any overrides on individual daemons. - OSD addresses: confirm that each OSD reports its client-facing address on the public network and its replication address on the cluster network.
- Monitor ports: monitors use the msgr2 port 3300, so allow it between monitors and clients, and confirm any other ports your release needs.
- Reachability and MTU: test traffic on each subnet from every host, and confirm that MTU matches across the path if jumbo frames are used.
- Recovery behavior: run a controlled recovery test and watch that replication traffic stays on the cluster network.
Record the results in the change ticket so auditors and the next operator can see the intended network design.
How recovery traffic changes the network decision
The case for a separate cluster network is strongest when you model what happens after a failure. When a drive or host fails, Ceph marks its placement groups degraded and starts backfill and recovery, copying data from surviving replicas or erasure-coded shards to other OSDs. That copying is replication traffic between OSDs, and on a single shared link it competes directly with the reads and writes your applications send. With a cluster_network defined at bootstrap, OSD-to-OSD replication and recovery use the cluster subnet, while clients keep talking to OSDs and monitors over the public network. Client I/O then stays on its own path while the cluster rebuilds.
The trade-off is headroom. A dedicated cluster network is only useful if it has enough capacity to absorb recovery at the speed you want, and it is idle most of the time, which is a cost you pay in extra ports, cabling and switch capacity. Running recovery on shared links is cheaper to build, but then your recovery settings become a live trade-off against client latency during every rebuild. Neither choice removes the need to decide how fast recovery should run. Ask your team to confirm the expected drive and host failure scenarios for your cluster, the aggregate throughput your recovery would need, and whether the network can sustain it alongside peak client traffic.
Do not size the network from a rule of thumb. Benchmark candidate hardware and network paths before purchase, and rehearse a host-down event on a staging cluster so you see the actual recovery rate and its effect on client latency. Tuning how recovery and client I/O share resources is covered on the Ceph performance tuning page, so this section treats the network as a fixed constraint to measure against. Plan the bonded links across two switches, as the Ceph hardware recommendations advise, so a single switch failure during recovery does not remove a network path.
How to plan and configure split networks step by step
This sequence takes a cluster from an addressing plan to a verified layout where client traffic and replication traffic run on separate networks, with each path checked before the cluster takes production load.
Confirm subnets and VLANs for each role
The network engineer and the Ceph lead agree on one public subnet for client, monitor and gateway traffic, and one cluster subnet for OSD replication, heartbeat and recovery traffic. Record the VLAN IDs, gateways, MTU and the address each host will use on each subnet in a single addressing sheet. Confirm that both subnets are routed where they need to be and that the cluster subnet is not reachable from client segments. Done when the sheet matches the switch configuration and every host has a reserved address on each subnet. Confirm your MTU approach with your switch vendor and your Ceph version's documentation before you set it on hosts.
Build bonds across two switches
The network team creates a bond on each host for each role, with one member link to each of two switches, so that a single switch failure does not remove a whole network. Configure the bond mode and LACP settings your switch platform supports, and confirm the switches are configured to match. The Ceph hardware recommendations advise bonding network links across two switches. Done when each host shows both member links up in the bond status and failing one link over a test window leaves both subnets reachable.
Bootstrap with the cluster network
On the first host, the Ceph administrator runs the cephadm bootstrap with the public network set to the client subnet and the cluster network set to the replication subnet, so the first monitor and manager are placed with both settings in place. Keep the bootstrap command and the resulting configuration in your change record. Changing these networks later means reconfiguring a running cluster, so treat this step as the point where the design is fixed. Done when the cluster configuration shows the public and cluster network values you planned and the first monitor binds on the public address.
Add hosts on both networks
The administrator adds each further host over SSH with an address on the public subnet, and confirms that each host also has an address on the cluster subnet and can reach the other hosts there. Check that the cephadm orchestrator sees every host and that monitors and managers are placed where your design calls for them. Done when every host is listed as online, and a reachability test from each host to every other host succeeds on both the public and the cluster subnet.
Verify OSD heartbeat and replication paths
After OSDs are deployed, the storage engineer checks that OSD heartbeat and replication traffic flows over the cluster subnet rather than the public one. Review the OSD network addresses in the cluster map, then watch interface counters on the cluster bond during a controlled write test to confirm traffic appears there. Confirm that the cluster reports no OSDs marked down because heartbeats are blocked. Done when the OSD addresses in use sit on the cluster subnet and the bond counters rise during replication while the public bond carries client traffic.
Confirm msgr2 ports are open on both networks
The network and security teams check that firewall rules allow the messenger v2 port, 3300, between monitors and clients on the public subnet, and that OSD and manager traffic on the cluster subnet is allowed between hosts. Use the Ceph messenger v2 documentation to confirm which port ranges each daemon type uses in your release, then test each path from a host on the relevant subnet. Keep the cluster subnet closed to any client segment. Done when every required connection succeeds from its intended subnet and a connection attempt from a client segment to the cluster subnet is refused.
Check health after a test node restart
The operations lead restarts one host during a planned window and observes how the cluster responds. Watch recovery traffic on the cluster bond, the client latency on the public bond, and the health output until placement groups return to active and clean. Record the time to healthy and any warnings. Done when the cluster returns to HEALTH_OK, client I/O stays within the levels your team agreed on before the test, and the findings are written into the runbook for future recovery planning.
Securing and segmenting Ceph network traffic
A split network only improves security when each network is restricted to the traffic it carries. The cards below cover the controls that keep replication traffic off client and management paths, and the operational habits that keep those controls intact after the cluster is handed over.
Restrict the cluster network to OSD hosts
The cluster network carries OSD-to-OSD replication, recovery and heartbeat traffic. Only OSD hosts should have an address on it, and the monitors, managers and gateways should not be routed onto it. Confirm which hosts hold a cluster network address and that no client subnet can route to it. Set the cluster network in your bootstrap configuration so the boundary is defined before the first OSD joins.
Firewall the msgr2 port 3300
Monitors listen for msgr2 connections on port 3300, and clients and OSDs reach them over the public network. Allow that port only from the subnets that need it, and deny it from everywhere else. OSD daemons also bind to a range of ports on each network, so your firewall rules must cover that range on both networks. Confirm the exact range in the Ceph configuration for your release before writing rules.
Enable msgr2 secure mode
Messenger v2 secure mode encrypts all traffic after authentication with AES-GCM and adds integrity checks, so a tap on either network cannot read or alter replicated data in transit. Test the setting on a non-production cluster first, because it adds CPU cost on busy OSD hosts. Ask the team to measure CPU use during recovery with secure mode enabled, since recovery is the heaviest traffic on the cluster network.
Keep management access off the data networks
SSH to cephadm hosts, the dashboard, and administrative API access should run on a management network that neither the public nor the cluster network can reach. Otherwise an attacker on a client subnet can reach the control plane. Confirm that the Dashboard and any single sign-on endpoints are bound to the management side, and that jump hosts are the only path into the storage hosts.
Change control for network edits
Changing a subnet, a VLAN, a bond or a firewall rule can split monitors from OSDs or silently block recovery traffic. Treat network edits as controlled changes with a written rollback, a pre-change health check, and a post-change check of cluster health and OSD heartbeats. Record which cluster network and public network values are in effect, so audits and future upgrades start from a known state.
Monitor link errors on both networks
Errors on one switch port often show up first as slow requests or flapping OSDs rather than as a network alert. Collect interface error counters, bond state and switch logs for both networks, and alert on sustained growth. Check that a failed link in a bond is reported, because a degraded bond keeps the cluster running and hides the fault until the second link fails.
Questions buyers ask about Ceph network separation
Is a separate cluster network required for Ceph?
No. A Ceph cluster can run with one network carrying both client and replication traffic, and that is a valid design for small or lightly loaded clusters. A separate cluster network is worth considering when replication, backfill and recovery traffic between OSDs could compete with client I/O on the same links. The Ceph hardware recommendations advise bonding network links across two switches in either case, so the redundancy question matters even when you skip the split. Decide with your own measurements: the expected client throughput, the number of OSDs per host, and how long you can tolerate a slow recovery after a host failure.
What does a separate cluster network add in NICs and switch ports?
The main cost is hardware duplicated per host: a second set of network interfaces, optics or cables, and switch ports on each switch that carries the cluster network. Costs rise with the number of hosts, the choice of link speed, and whether each network is bonded across two switches, which doubles the ports again. Cabling, rack space and power for the extra switches are often overlooked. Ask the quote to list each NIC, transceiver, cable and switch port separately, state whether the switches are new or existing, and show which hardware is one-time purchase and which is support or warranty that recurs.
How long does a Ceph network design and bootstrap take?
The time depends on the number of hosts, whether the switch configuration already exists, how many VLANs and subnets must be approved, and whether the client network team has to change firewall or routing rules. Bootstrap with cephadm is usually the shorter part once addressing and switching are ready. DeployOpen does not publish durations; the timeline comes from the written scope after the network inventory is reviewed. Ask for a schedule that separates design sign-off, switch changes, host configuration, bootstrap and the verification tests, and note which steps depend on change windows you control.
What do we need from our network team?
Plan for the network team to own the switch side and the approvals. They typically supply subnet and VLAN allocations for the public and cluster networks, switch configuration for bonded ports across two switches, MTU settings (jumbo frames must match end to end), routing rules between the networks, and firewall rules for the Ceph messenger ports. Ceph uses port 3300 for msgr2 and a range of ports for daemons, so confirm with the vendor documentation which ports your release needs. Agree who signs off each change and how rollback works before the work starts.
Can our own team set up separate networks without a partner?
Yes, if your team already runs Linux bonding, VLAN tagging and Ceph, and has a lab or staging cluster to test in. The cephadm bootstrap accepts a cluster network setting, and the Ceph documentation describes the configuration options. The risk is in the details: a single MTU mismatch, an unbonded link or a missing firewall rule can cause intermittent slow requests that are hard to trace. A partner is most useful when the team lacks that experience or the work must be completed inside a fixed window. Compare the internal effort against a fixed-scope quote that lists the same checks.
How do we ask for a fixed-scope quote for network separation?
Send a written description of the hosts, OSD counts per host, NIC models and speeds, switch models and free ports, the subnets and VLANs already allocated, and whether bonds must span two switches. Ask whether the cluster network is in scope or only a recommendation. DeployOpen works from written scope, deliverables and timelines agreed before work starts, so ask the quote to list the design review, switch configuration support, the cephadm bootstrap with network settings, connectivity and bandwidth tests, and handover documentation. Also ask what is excluded, such as hardware purchase or switch replacement.
How DeployOpen plans Ceph network separation
DeployOpen plans and implements self-hosted Ceph inside your own data centers, and the network layout is settled before any hosts are touched. We start from your traffic model: how many client-facing services will use the cluster, how many OSD hosts and switches exist, whether each host has enough ports for bonded links across two switches, and whether your change process allows a separate cluster network at all. The output is a written design that names the public network, the optional cluster network, the bond mode, VLAN and subnet assignments, and the firewall rules for the msgr2 ports, so your network team can review it against their own standards.
The network design is scoped in a written, fixed-scope agreement before work starts. That agreement lists the deliverables, such as the subnet plan, the cluster_network value passed to the cephadm bootstrap, host network configuration, validation runs and the handover notes, along with what you must provide: switch access, IP allocation, change windows and a test environment where recovery behavior can be measured. Ask for the scope in writing and confirm which switch vendor configuration, routing or monitoring work is included, because those items change the effort more than the Ceph settings do. Pricing follows from that written scope, and DeployOpen does not publish a price list.
Ongoing support is available as a retainer for after the cluster is in production. Network questions tend to return later, when a new rack is added, a switch is replaced or recovery traffic starts slowing client I/O. A retainer covers reviewing those changes against the agreed design, checking network health during upgrades and documenting decisions as the environment changes. DeployOpen does not hold certifications or partner status that would change this work, so confirm the engagement terms, team roles and the support hours you need directly in the proposal.
