Load balancing Ceph RGW with cephadm ingress

Why teams front Ceph RGW with the cephadm ingress service

Teams front several Ceph Object Gateway (RGW) daemons with the cephadm ingress service when S3 clients need one stable endpoint that survives a gateway host or daemon failure. The ingress service deploys haproxy to spread client connections across the RGW daemons and keepalived to hold one or more virtual IP addresses that move between hosts when a node fails. cephadm deploys, configures and restarts both as containers, so the balancer is managed alongside the gateways rather than maintained as a separate hand-built tier. Without it, every application would need to know individual gateway addresses, and a failed host would break clients until someone changed their configuration.

The decision to use it usually follows three questions. First, do clients need a single address or DNS name that stays the same while daemons are added, moved or replaced? Second, do you have, or are you willing to dedicate, hosts that can carry the virtual IP and the haproxy process without conflicting with other services on the same ports? Third, does your network allow keepalived to advertise a floating address on the subnet, which some cloud networks and tightly controlled data center segments do not permit. Confirm how your cephadm release handles the ingress specification, including the keys for virtual IP, monitor port and daemon placement, against the version-specific cephadm documentation before you draft a design, because these fields have changed across releases.

Read the Ceph Object Gateway documentation first to confirm which RGW features your clients depend on, since some behavior such as TLS handling and virtual-host bucket addressing affects where the balancer should terminate connections. Then check the cephadm documentation for the ingress service requirements in your release. Treat the ingress as one part of your availability plan: it does not replace multiple RGW daemons, and it does not by itself protect data or define a multisite failover path.

Ingress configuration options and when each fits

This table compares the main ways to front RGW daemons with the cephadm ingress service, from a single host pair to a dedicated external load balancer. Read each row as a decision: the first column names the choice, and the other columns show when it fits, what it costs you, and what to confirm in the cephadm documentation for your release.

Option	When it fits	Trade-off	What to confirm
Ingress on two dedicated hosts	Small or mid-sized clusters where RGW runs on separate hosts from OSDs and the team wants a single virtual IP for clients	Gateway hosts are extra machines to patch and monitor, but they keep haproxy load away from storage nodes	Host count and placement rules for the ingress service in the cephadm documentation for your release
Ingress co-located with RGW daemons	Lab or small deployments where spare hosts are not available and RGW already runs on enough hosts for failover	A host failure can remove both an RGW daemon and a load balancer instance at once, so failover capacity is reduced	Whether the placement spec allows haproxy and keepalived on RGW hosts in your version
One virtual IP for all RGW endpoints	Most S3 clients that use one endpoint name and do not need traffic separated by function	Simple to publish in DNS, but every client shares one failover point and one set of connection limits	How keepalived assigns the virtual IP and which interface and VRRP settings the spec exposes
Multiple virtual IPs across hosts	Larger deployments that want traffic spread across ingress hosts or separate names for separate tenants or regions	Better distribution and failure isolation, but DNS records, firewall rules and client configuration grow	Whether your release supports multiple virtual IPs in one ingress spec and how they are ordered in DNS
TLS terminated at haproxy	Sites that manage one certificate chain at the edge and want RGW behind plain HTTP on a private network	Certificates are handled in one place, but traffic between haproxy and RGW must be protected by network controls or a second TLS hop	Certificate fields accepted by the ingress spec and the RGW certificate options in the cephadm documentation
TLS terminated at RGW	Environments that require encryption end to end or that must present the same certificate to every backend	Certificates must be deployed to each RGW daemon, and renewal touches more components	Whether haproxy passes TLS through to RGW in your release and how health checks handle HTTPS backends
External load balancer instead	Organizations with an existing enterprise load balancer, strict change control for edge services, or hardware-based TLS requirements	Fewer components inside the Ceph cluster, but health checks, session behavior and failover must be configured in a system the storage team may not own	Health check path, timeouts and TLS settings your load balancer supports, and whether the cephadm ingress service is still needed for the cluster

Where to terminate TLS for RGW traffic

With this option, the ingress service accepts HTTPS on the virtual IP, decrypts it in haproxy, and forwards plain HTTP to the RGW daemons on the private network. The certificate and private key are supplied through the ingress specification, so cephadm places them where haproxy reads them and redeploys when they change. This keeps certificate handling in one place, which is why platform teams often choose it when the number of RGW daemons is large or daemons move between hosts.

Single certificate location: haproxy holds the only copy of the certificate that clients see, so a rotation touches one specification instead of every RGW daemon.
Plain traffic inside the cluster: requests reach RGW unencrypted, so the back-end network must be isolated and restricted to the ingress hosts.
Client address visibility: RGW sees haproxy as the peer, so confirm that forwarded client address headers are set and that access logs and any IP-based policy read them.
Wildcard coverage: the certificate presented at the virtual IP must include the wildcard name your virtual-host bucket style requires, or S3 clients will fail the TLS check.
Renewal testing: validate the chain and key format in the ingress specification before a rollover, because a bad certificate leaves every client without access at once.

Before you commit to this path, confirm with your certificate team how certificates will be issued and renewed, and check the field names against the cephadm documentation for your release.

When an external load balancer is the better choice

The cephadm ingress service, which runs haproxy and keepalived on your Ceph hosts, is often enough for RGW. The conditions below point toward a dedicated external load balancer instead. In each case, the ingress service still provides RGW daemon placement and health tracking if you keep it in place.

Existing enterprise load balancers

If your organization already runs a standard load balancer tier with its own certificate management, monitoring and change control, adding a second haproxy layer on the Ceph hosts creates a parallel system to operate. Route RGW through the existing tier when its operators can own health checks and backend pools. Confirm that the external balancer can reach the RGW daemons directly and that it can keep long-running S3 uploads open. The cephadm ingress can remain as a fallback or be removed.

Strict network segmentation rules

When policy places storage hosts in a separate zone and only a DMZ or front-end zone may accept client traffic, the keepalived virtual IP must live on a network the clients can reach. If that network cannot be shared with the Ceph public network, an external balancer in the permitted zone is usually the cleaner design. Check which firewall rules must open for the balancer to reach the RGW ports and for health probes to succeed.

Multi-site or geographic routing

A single virtual IP per cluster does not steer clients to the nearest site or fail over between data centers. When you run RGW multisite or need DNS-based or anycast routing across regions, a global traffic manager or external balancer usually owns that decision. Define which zone serves which clients and how failover interacts with the multisite runbook before you choose.

Required health check or WAF features

Ingress health checks cover whether an RGW daemon answers, but they may not cover the checks your security team requires, such as authenticated probes, custom response matching or web application firewall inspection of S3 requests. If you must inspect requests at layer 7 or run bot and rate controls at the edge, place those controls on an external product. Confirm the exact HTTP behavior your product needs before choosing.

Team ownership of networking

The cephadm ingress is configured through the orchestrator, so the storage team owns its virtual IPs, haproxy settings and certificates. If a network team owns load balancing, VIP allocation and TLS policy, splitting ownership across two teams causes change conflicts. Agree on who approves virtual IP ranges, certificate renewals and timeout changes, and write those owners into the runbook.

Cluster scale and throughput

Large S3 workloads with many concurrent clients may need more gateway hosts, dedicated balancer capacity and separate scaling from the storage nodes. Running balancer processes on hosts that also run OSDs or monitors ties their resource use together. Benchmark the planned client load against both designs, and ask your vendor or the Ceph documentation to confirm the limits of the design you choose.

Deploying the ingress service with cephadm

This sequence takes a set of running Ceph RGW daemons and places them behind a virtual IP address that clients can use as a single S3 endpoint, with haproxy distributing requests and keepalived moving the address between hosts when one fails. Each step ends with a check so you can confirm the layer works before a client depends on it.

Confirm RGW daemons are healthy
Before adding anything in front of RGW, check that the cluster is in a healthy state and that every RGW daemon you plan to load balance is running on its host. Run ceph -s to confirm the cluster reports HEALTH_OK or understand any warning, and run ceph orch ps --daemon-type rgw to list the daemons and their status. Send a direct request to each daemon's own port with an S3 client and confirm it returns a valid response. The step is done when every daemon answers directly, because a failing backend will look like a load balancer fault later.
Choose hosts and virtual IP addresses
Select the hosts that will run the haproxy and keepalived daemons, and reserve one or more virtual IP addresses on a routable subnet that those hosts share. Confirm with your network team that the addresses are not in use, that the subnet is reachable from the clients that will call the endpoint, and that the hosts can exchange VRRP traffic on that segment. Ask the team to confirm whether their switches or firewalls filter the protocol. The step is done when each candidate address is documented, unassigned, and reachable from a test host on the client network.
Write the ingress service specification
Create a service specification file that names the RGW service, the virtual IP address and prefix, the frontend port, and the number of haproxy and keepalived instances. Keep the file under version control so later changes are reviewed. Check the field names and accepted values against the cephadm documentation for your release, since they can change between versions. The step is done when the file parses and lists the hosts and addresses you chose in the previous step, with no placeholder values left in it.
Apply the ingress specification
Apply the file with the orchestrator, then watch the service come up. Run ceph orch apply -i with the specification file, and then ceph orch ls ingress and ceph orch ps to see the haproxy and keepalived containers being deployed. The orchestrator will report the status of each daemon. If a daemon stays in an error state, read its logs with cephadm before changing the specification. The step is done when all ingress daemons show as running and the orchestrator no longer reports pending placement for them.
Verify the virtual IP and haproxy listeners
Confirm that the virtual IP is bound to exactly one host at a time and that haproxy listens on the frontend port on the address. On the active host, check the address with ip addr show and confirm haproxy is listening with ss -tlnp. From a client on the same network, open a TCP connection to the virtual IP and frontend port. Then send an S3 request through the endpoint and confirm the response comes from an RGW backend, for example by checking the backend log entries on the RGW hosts. The step is done when the endpoint answers and requests are visible on more than one backend.
Test keepalived failover
Simulate a failure of the host that holds the virtual IP by stopping its ingress daemon or taking it out of the network in a maintenance window. Confirm that keepalived moves the address to another ingress host and that the S3 client test continues to succeed, possibly after a brief interruption for in-flight connections. Restore the original host and confirm the address behaves as your configured priorities expect. Record how long clients saw errors, and ask the team whether that behavior meets the availability needs of your applications. The step is done when failover and failback both complete and the result is written down.
Run an S3 client test through the endpoint
Run the S3 clients your applications use against the virtual IP endpoint, including uploads, downloads, multipart uploads, listing and deletion. Include the TLS configuration your clients expect if you terminate TLS at the ingress layer. Compare results with the same operations sent directly to an RGW daemon to confirm the load balancer changes nothing. If you rely on virtual-host style bucket addressing, test it, because DNS and certificate names must match the endpoint. The step is done when the test suite passes through the virtual IP with the same results as direct access.

Security and operational considerations for the ingress layer

The ingress layer sits in front of every S3 request, so it carries credentials, network exposure and audit evidence for the whole gateway. The considerations below cover what to lock down, what to protect, where to place traffic and how to keep the layer patched and observable alongside the RGW daemons.

Restrict the stats and admin interfaces

The haproxy stats page exposes backend health, server names and traffic counters, and it should not be reachable from client networks. Bind it to a management address or allow only named operator subnets, and require authentication on it. Check the bind address and access rules in the ingress specification before the first deployment, then confirm from an external host that the page does not answer.

Protect the haproxy private keys

When TLS terminates at haproxy, the private key and certificate chain sit in the ingress specification and on every ingress host. Decide who can read the service specification, limit access to the cephadm admin account, and plan rotation before expiry. Ask the team that owns certificates whether keys must come from an internal CA or a hardware-backed store, since that changes where the key material lives.

Place virtual IPs on a controlled network

Keepalived announces the virtual IP with VRRP, which is link-local multicast traffic that should stay on the ingress subnet. Keep VRRP off routed and client-facing segments, confirm that switches on the path allow the protocol, and use a unique virtual router ID per cluster so unrelated keepalived instances do not contend for the address. Ask your network team to document which hosts may send advertisements.

Monitor haproxy and keepalived

Ingress failures often look like RGW failures to clients, so monitor each layer separately. Track haproxy backend health and error rates, confirm which host holds the virtual IP, and alert on failover events. Feed haproxy logs and keepalived state changes into your central monitoring so an unexpected move of the address is visible within your normal alerting window.

Patch ingress images with the cluster

The haproxy and keepalived images are versioned with the cluster and must be updated on the same change calendar as RGW. Include the ingress daemons in your upgrade plan and test them in a staging cluster first. Ask your vendor or DeployOpen to confirm which image versions match your Ceph release, because a mismatch between the ingress layer and the gateway can cause hard-to-trace request failures.

Log client source addresses for audit

Behind a load balancer, RGW may see only the ingress address, so audit records lose the true client origin unless forwarding is configured. Decide whether haproxy will pass the client address in forwarded headers, confirm that RGW logs and trusts them only from the ingress hosts, and keep haproxy access logs for the same retention period as gateway logs. Test the chain with a known source before relying on it for investigations.

Questions teams ask before fronting RGW

What drives the cost of fronting Ceph RGW with cephadm ingress?

The main cost driver is the number of hosts and the network design around them, because the ingress service runs haproxy and keepalived on hosts you choose and needs a virtual IP that clients can reach. Costs fall into three groups: the hardware or virtual machines that carry the ingress layer, the engineering time to design the network path, TLS and failover behavior, and the testing that proves clients keep working when a host or daemon fails. Recurring costs are patching, certificate renewal and periodic failover tests. Ask any provider to price these separately, state which hosts are included, and show who maintains the haproxy and keepalived configuration after handover.

How long does it take to set up RGW load balancing?

Duration depends on how many decisions are already made. A team with a working RGW service, a free virtual IP range, DNS control and an agreed certificate method can often move quickly, while a team that must first settle network segments, switch configuration and certificate authority will spend most of the time waiting on other departments. Testing with real S3 clients, including large uploads and failover during writes, usually takes longer than the configuration itself. Ask for a timeline that lists each dependency and its owner, and confirm it against your own change calendar.

What does the client team need to provide for the ingress setup?

The work needs decisions and access from your network, DNS and security teams before any configuration starts. Typically you provide:

A virtual IP on each ingress subnet, reserved so no other host uses it
DNS records for the public RGW endpoint and, if you use virtual-host buckets, a wildcard record that resolves to the virtual IP
Certificates for those names, either issued by your internal authority or signed through cephadm, with a renewal owner
Firewall rules allowing clients to reach the virtual IP and the ingress hosts to reach the RGW daemons

Confirm these items with your network and security teams early, since they often set the critical path.

Can our own team set up RGW ingress using only the documentation?

Yes, a capable team can follow the cephadm documentation to deploy the ingress service, but the difficult part is not the initial command. The harder work is choosing host counts, keepalived priorities, health checks, TLS termination and how the configuration is versioned and upgraded alongside Ceph. Teams without recent haproxy or keepalived experience usually find that failover testing and troubleshooting of client timeouts take the most effort. Before deciding, check whether your engineers can run failover drills on a staging cluster and own the result. You can also engage a partner for a fixed-scope design and test, then operate the setup yourselves.

When should we use an external load balancer instead of cephadm ingress?

Use an external load balancer when your organization already runs a standard load balancing platform with its own change control, monitoring and high-availability design, or when several services share the same front door. The cephadm ingress service is simpler to keep close to the cluster, but it places haproxy and keepalived on Ceph-managed hosts, which may conflict with network or security policies. An external balancer also suits TLS offload at a central point or wide-area traffic management. Whichever you choose, confirm that health checks, connection limits and timeouts for large object transfers are set in that layer.

How does multisite change the RGW load balancing design?

In a multisite deployment each zone has its own endpoints, so you usually build one ingress layer per zone and decide how clients reach each one. Replication traffic between zones also crosses these endpoints, which means load balancing rules, timeouts and certificates affect both client access and synchronization. Failover is more involved, because a virtual IP that moves within one site does not switch clients to another zone by itself. Plan zone endpoints, DNS behavior and the failover runbook together, and test a zone outage before relying on the design.

Which tests prove the ingress layer handles failover correctly?

Tests should show that clients keep working when a host, keepalived instance or RGW daemon stops. Useful checks include stopping the active ingress host during a sustained upload, restarting an RGW daemon under read load, and confirming that virtual-host bucket requests succeed through the wildcard name. Record how long clients see errors and whether retries from your SDKs cover the gap. Ask your provider to deliver the test scripts and results as part of the scope, and repeat the tests after every Ceph upgrade.

How DeployOpen scopes and supports RGW ingress

DeployOpen plans and builds the RGW ingress layer as part of a self-hosted Ceph deployment inside your own infrastructure. We agree the scope in writing before work starts, then test, document and hand over the virtual IPs, haproxy and keepalived configuration and the TLS arrangement so your team can operate it.

How we scope

Confirm the RGW daemon count, host placement and the virtual IP addresses your network team will allocate.
Record the TLS termination choice, certificate sources and whether any external load balancer sits in front of the ingress service.
Agree in writing which failover tests, client SDK checks and documentation deliverables are included before the fixed-scope work starts.

How we build

Deploy the cephadm ingress service with haproxy and keepalived on the hosts you designate, using your network and bonding layout.
Configure health checks against each RGW daemon so failed backends are removed from rotation during the test window.
Validate S3 requests through the virtual IP, including virtual-host style bucket addressing, against the client endpoints you name.

How we hand over

Provide a written record of the deployed service specification, virtual IP assignments, certificate locations and host placement.
Document the failover tests performed, the observed results and the steps to repeat them after maintenance or host changes.
Walk your platform and network teams through the runbook for adding RGW daemons, rotating certificates and moving the virtual IP.

How we support

Ongoing support, upgrades and continuous testing for the ingress layer are available as a retainer, scoped separately from the build.
Ask any partner to confirm how cephadm ingress changes are handled across Ceph upgrades on the release you run.
Confirm with your network and security teams who owns certificate renewal and virtual IP changes after handover.

Load balancing Ceph RGW with the cephadm ingress service