When a Ceph RGW multisite failover is the right move
A Ceph RGW multisite failover is the right response when your master zone is unavailable for long enough that writes to S3 buckets can no longer wait for it to return, and you have decided to accept the consequences of moving the master role to a secondary zone. It changes which zone accepts metadata changes for the zone group, so it is a control-plane decision with effects on every client that depends on bucket and user metadata. It is not a switch that the gateway flips on its own when a site goes dark, and nothing in the multisite design promotes a zone automatically.
The Ceph Object Gateway documentation describes multisite as a set of *zones* grouped into a *zone group*, where one zone is the master and the others are secondaries that replicate data and metadata from it. The *period* is the versioned configuration record that every zone in the realm must agree on. Promoting a secondary means changing the master assignment and committing a new period, and that commit is the step that makes the change take effect across the realm. Exact commands, flags and the order of steps differ between releases, so confirm each one against the Ceph Object Gateway documentation for the release you run, whether that is the Squid (19.2.x) or Tentacle (20.2.x) line.
Before you start, decide who has authority to declare the failover and what evidence triggers it. Record the last replicated state you can confirm on the secondary, because objects written to the failed master after that point may not exist in the new master. Confirm that your clients can reach the secondary endpoint and that your team has tested the runbook steps in a lab with the same release. If the outage is expected to be short, waiting for the master is often the safer choice, and that decision should be written down before anyone runs a command.
What fails while the master zone is down
While the master zone is unreachable, anything that must be committed through the master cannot succeed. In a Ceph RGW multisite design, the metadata master is the zone that owns the period and accepts changes to users, buckets, quotas and other metadata. Until you promote a secondary zone, those administrative writes fail or stop at the site that is down. Client object writes to buckets whose data lives only in the failed zone also fail, and the secondary zone will not accept them unless you have deliberately changed the zone configuration. Applications that send S3 writes to the failed endpoint get connection errors, so the first thing to check is which endpoints, load balancers and DNS names your clients actually use, since many still point at the old master after the outage.
The most important distinction is between data that already replicated and changes that never left the failed site. Objects and metadata changes that completed sync to the secondary zone are available there, subject to the sync position at the moment of failure. Anything written to the master after its last successful sync, including object uploads, deletes, bucket creation and quota changes, exists only on the failed hardware. Replication lag therefore sets your effective recovery point. Your own cluster determines that lag, so measure it before you decide anything: run radosgw-admin sync status on the secondary zone and review the metadata and data sync lines, and compare the timestamps of recent writes on the master with what the secondary reports. Record the result in your runbook before you promote anything.
Plan for the gap explicitly. Decide whether your recovery point objective accepts the lag you measured, and whether any writes lost in the gap must be reconciled by hand from application logs, event streams or upstream sources once the master returns. Confirm with your application owners how they handle duplicate or missing objects, and check whether any bucket notification or lifecycle process ran on the master during the outage. Also confirm the behavior of your specific Ceph release with the official Ceph Object Gateway documentation before relying on it, because multisite handling of failed zones changes between versions and must be tested in your environment rather than assumed.
Failover decisions by outage situation
The master zone is healthy during planned work, so the default is to finish the work without promotion. Promotion changes the period and must be reversed later, which adds risk to a change that did not need it. Promote only when the planned outage is longer than the business can accept and the cutover has been rehearsed in a non-production environment.
- Wait by default: Schedule the work so that the master keeps serving clients and the secondary catches up afterward.
- Promote only with a written cutover plan: Record the steps, the named owner, the rollback owner and the point at which you stop and revert.
- Freeze writes before promotion: Stop client writes at the load balancer or endpoint layer so nothing lands on the zone that is about to lose master status.
- Confirm sync is caught up: Check metadata and data sync status on both zones before changing the period.
- Evidence for the incident lead: A sync status report from both zones, the list of client endpoints affected, and a confirmed rollback owner.
Before the change starts, check that the window, the sync status and the rollback owner are written down. Confirm the exact period and sync commands against the release documentation for your Ceph version in the Object Gateway documentation.
An unplanned outage where the old master is still partly reachable is the highest-risk case. Two zones may accept writes at the same time, which is the situation that produces divergent data. Waiting is safer while the outage is short and the old site is confirmed unable to serve clients. Promotion is justified only after the old master has been fenced and the business has accepted the data-loss window.
- Fence the old master first: Block client traffic at the load balancers, DNS or network paths for the old site before any promotion.
- Do not restart the old site unattended: A returning master must not resume serving clients until its role has been decided.
- Gather reachability from several vantage points: One network view cannot prove the old site is unreachable to all clients, so check from more than one location.
- Record the last sync point: Note the last confirmed sync time and the last known write, since these define the data-loss window.
- Evidence for the incident lead: Reachability results from multiple vantage points, the fencing record, the last sync timestamp, and the decision owner's sign-off on data loss.
Do not promote until the fencing record and sync point are on paper. Confirm how your load balancer and DNS controls can be changed during an outage before you need them.
In a partial failure, the master still accepts writes from some clients, so promotion is almost always the wrong first move. Promoting the secondary while the master keeps writing creates two active zones. The better approach is to repair the failing component, such as gateway daemons, the ingress layer or a subset of storage, while the master continues to serve healthy clients.
- Repair before promoting: Restart or replace the failing gateway daemons and check the ingress layer before considering a role change.
- Drain only the failing endpoints: Remove the broken endpoints from rotation and leave healthy ones in place.
- Check writes accepted during the fault: Objects written to the master during partial failure still need to replicate, so confirm sync for those buckets.
- Watch for errors on the secondary: Track error rates on both zones to see whether replication is falling behind.
- Evidence for the incident lead: Per-endpoint error rates, the list of affected clients, sync lag for the affected buckets, and a timeline of the repair.
Promote only if the master cannot be restored to serving clients within the outage limit that the decision owner has accepted. Confirm with your Ceph version's release notes which daemon and sync behaviors apply during degraded operation.
Failover options and when each fits
This table compares three ways to respond when a Ceph RGW master zone is lost: waiting for recovery, promoting the secondary zone, or serving a limited service from the secondary. Read each column against your own recovery point and recovery time objectives, which you should set before choosing a path. The main takeaway is that promotion trades data currency and rollback effort for faster client service, so confirm the replication lag and rollback steps with your team before an incident.
| Option | Data currency | Client impact | Operational effort | Rollback path |
|---|---|---|---|---|
| Wait for the master zone to recover | Full, once the master returns and replication catches up | Writes and metadata changes stop for the whole zone until recovery | Low during the outage, but the outage length is outside your control | None needed; the original master keeps its period and zone roles |
| Promote the secondary with the period committed | Limited to objects and metadata not yet replicated when the master failed | Reads and writes resume on the secondary after clients are repointed | High: period commit, endpoint changes, and monitoring of divergent writes | Requires a planned failback that reconciles writes made during the outage |
| Limited read-only service from the secondary | Reads reflect the last replicated state, so recent writes may be missing | Read access continues; writes are rejected or queued by the application | Medium: endpoint routing and application behavior for rejected writes | Simpler, since the master's data is not changed, but queued writes still need handling |
| Confirm with your RPO | Ask how much replication lag you accept at the moment of failure | Confirm which applications can tolerate read-only or delayed writes | Confirm who can approve promotion and how long that approval takes | Confirm that failback is rehearsed in a test environment before production use |
Promoting the secondary zone: sequence and checks
This sequence moves write authority for an RGW zone group from a failed master to a surviving secondary zone in a controlled order, so that each change is verified before the next one starts. The output is a new master that accepts writes, a recorded sync baseline, and clients pointed at the zone that is now authoritative.
Confirm the master zone is down
Before any change, the on-call engineer and the application owner confirm that the original master is unavailable and not just partitioned from the monitoring network. Check the RGW endpoints of the master zone from more than one network path, check the monitors and OSDs of that cluster from its own site if reachable, and record the time of the outage and the last known health state. Done when the team agrees in writing that the master will not return within the window they have set for this decision, and the decision is logged with the reason.
Record the multisite sync state
Capture the current multisite sync status from the secondary zone, including data and metadata sync markers, any shards behind, and the last period that zone applied. Save the output with a timestamp, because it is the only evidence of how far the secondary is behind the failed master and which objects may be missing after promotion. Done when the saved status is attached to the incident record and the team has a list of buckets known to be behind.
Isolate the original master
Stop the original master zone from accepting writes before the secondary becomes authoritative, so that a master that comes back online cannot accept writes that conflict with the new master. Typical isolation options include disabling the RGW endpoints at the load balancer, removing the zone's endpoints from DNS, or blocking access at the network layer. Confirm the exact isolation method for your release and topology in your own runbook. Done when a client test against the old endpoints fails and the isolation change is recorded.
Promote the secondary zone and update the period
Promote the secondary zone to master by following the documented period update and commit procedure for your Ceph release. The period update changes the zone group configuration that every zone uses to determine which zone is master, so this step must be run against the secondary zone and checked by reading the period back. Confirm the exact commands, flags and any release-specific prerequisites in the Ceph Object Gateway documentation for your version before running them. Done when the period reports the secondary zone as the master zone and the change is recorded with the new period ID.
Verify the new master accepts writes
Run a controlled write test against the new master using an S3 client and a test bucket with credentials that the team has prepared in advance. Confirm that PUT, GET, LIST and DELETE succeed, that the zone reports itself as master, and that the RGW daemons on the new master are healthy in the cluster status. Done when the test results are saved and the write path is confirmed on the new master before any production client is redirected.
Confirm replication from the new master
Check that the surviving secondary zones, if your topology has more than one, begin syncing from the new master, and that sync status shows progress rather than errors. Write a test object on the new master and confirm it appears in the other zones within the time the team considers acceptable for the workload. Note that this verification is about the new master's replication path, so any zone that was never in sync with the old master needs its own review. Done when sync status is healthy and a test object is confirmed on a remote zone.
Repoint clients, load balancers and DNS
Update the endpoints that applications use, including DNS records for the RGW service name, load balancer back-ends, and any client configuration that hard-codes the old master's address. Lower any DNS caching that would delay the change, and confirm the TLS certificates on the new endpoints cover the names clients use, including wildcard names for virtual-host style buckets. Done when a sample of application and SDK clients reach the new master, the old endpoints receive no traffic, and the change record lists every endpoint that was moved.
Risks that appear during and after promotion
Most multisite failover incidents are not caused by the promotion itself. They come from what happens around it: a returning master that still believes it is in charge, writes that never reached the secondary, and clients that keep calling the old site. Each card below names one risk, how to detect it, and how to contain it before it spreads.
Split brain after the old master returns
If the original master comes back online before it is reconfigured, it can accept writes that the promoted zone never sees, and the two zones diverge. Before the old master serves any traffic, confirm the realm period on each side and check which zone each one reports as master. Keep the old zone read-only or fenced at the load balancer until the failback plan says otherwise. Treat any write accepted by the returning zone as unreconciled until you have compared it.
Lost or conflicting writes from lag
Multisite replication is asynchronous, so objects written to the master shortly before the outage may not exist on the secondary. Compare the sync status on the secondary before promotion and record the gap in writing, including the last window you believe was replicated. Applications that retry writes after a timeout can also create two versions of the same key. Agree with application owners how to identify and resolve those objects, for example by comparing ETags or object versions.
Stale users, keys and policies
Metadata such as users, access keys, quotas and bucket policies replicates through the period, so a secondary that was lagging may lack a user created shortly before the outage or still hold a revoked key. Diff the user list and access keys between zones before you open traffic, and confirm that a credential revoked on the master is also rejected on the promoted zone. Review account and quota settings for any tenant that changed during the incident window.
Clients still pointing at the failed site
Applications often hold hard-coded endpoint names, DNS caches or SDK configuration that still resolves to the failed zone. Their requests then time out, retry, or fail with errors that look like application bugs rather than a site outage. Build a list of every client, its endpoint, and its owner before the drill, and test each one against the promoted zone. Check DNS TTLs in advance, since a long TTL extends the period in which clients keep failing.
Load balancer and certificate names mismatch
Virtual IPs, load balancer back ends and TLS certificates are often built for one zone's hostname. After promotion, the new active endpoint may present a certificate that does not list its name, or the ingress may still forward to daemons in the failed zone. Verify the subject alternative names for the endpoint and the virtual-host bucket wildcard on the promoted zone, and confirm that health checks only route to running gateways. Confirm the certificate chain with your security team before the cutover, not during it.
Lifecycle and event jobs run twice or not at all
Lifecycle expirations and bucket notifications are tied to the zone that processes them. During an outage, scheduled expirations may be missed, and when the failed zone returns, the same events may fire again on both sides. Downstream consumers such as Kafka topics or webhook receivers may then see duplicates or gaps. List every bucket with a lifecycle rule or notification, decide which zone owns each job after promotion, and make consumers idempotent before the failback window opens.
Restoring the original master without losing writes
This sequence brings the original master zone back into the zone group only after it has caught up with the promoted zone, so no writes made during the outage are lost or silently overwritten. It ends with a controlled cutover that the team has rehearsed beforehand.
Confirm the promoted zone is stable
Before touching the original master, the storage team confirms the promoted secondary is serving all client traffic, is the current master in the period, and has stopped accepting failover changes. Record the period ID, realm and zone group state as the baseline for rollback. Done when the team has a written snapshot of the current multisite topology that everyone agrees on.
Isolate the original master zone
When the original master's hardware or network returns, its RGW endpoints stay out of client DNS and load balancer pools. Operators hold the zone read-only or isolated so no client writes land on it while its metadata and data are behind. Done when a test write through the original endpoints is refused or never reaches a client, and the isolation is documented in the change ticket.
Resynchronize the original master from the promoted zone
The team lets the original zone pull metadata and data changes from the promoted zone, then watches multisite sync status until both zones report they are up to date. Any sync errors, such as objects that fail to replicate, are investigated and resolved before the next step. Done when sync status shows no pending metadata or data shards for the zone.
Verify metadata and object consistency
Before any write moves back, the team compares bucket lists, user and account metadata, bucket policies, and sample object counts and checksums between the two zones for the buckets that changed during the outage. Discrepancies are reconciled from the promoted zone, which holds the writes made during the incident. Done when the comparison report shows no unexplained differences and the team signs off in writing.
Run the controlled cutover window
In an agreed maintenance window, the team freezes writes on the promoted zone, waits for a final sync pass to drain, and confirms no pending changes remain. The original master is then recommitted as the master in the period, and client DNS or load balancer pools are switched back. Done when a final sync check shows zero backlog and writes succeed against the restored master.
Recommit the period and redirect clients
The team commits the period so the original master is restored in the zone group, then updates client endpoints, SDK configurations and monitoring to point at it. The promoted zone returns to secondary status and sync resumes in the original direction. Done when every zone reports the expected master in the period, sync is healthy in both directions as intended, and client smoke tests pass.
Keep a rollback plan and rehearse the failback
Failback is a planned change with its own rollback: if verification or cutover fails, the team reverts to the promoted zone as master using the baseline snapshot. Operators should rehearse the full sequence in a non-production realm or lab before an incident, and confirm the exact multisite commands with the Ceph Object Gateway documentation for their release. Done when the rehearsal produces a runbook with timings and rollback steps that the on-call team has walked through.
Security and operational controls for failover
A zone failover changes who can alter the realm, which credentials are valid where, and which endpoints are exposed during an incident. The controls below cover those points. Confirm each one against the Ceph release you run, because multisite behavior and dashboard options differ between releases.
Restrict promotion and period commits
Promoting a secondary zone and committing a period change the topology for every zone in the realm, so only a small named group should hold the rights to run them. Keep these actions behind a documented approval step, and record who holds the admin keys used for the radosgw-admin commands. Before an incident, confirm which accounts can run zone promotion and which ones can only read realm status.
Keep access keys in sync
Users, access keys and system users are replicated through the period, so a failover depends on the secondary zone already holding the credentials your applications use. Check that the system user used for zone sync exists and works in every zone, and that application keys match across zones before you need them. Test a key against the promoted zone as part of the drill, not during the incident.
Encrypt replication traffic
Zone sync moves object data and metadata between sites, often across networks you do not fully control. Ceph's messenger v2 protocol uses port 3300 and offers a secure mode that encrypts post-authentication traffic with AES-GCM and adds integrity checks. Confirm whether secure mode applies to your sync path and how it is configured for the inter-site links in your release.
Protect the dashboard and admin APIs
During an incident, operators often reach for the dashboard or the RGW admin API, which are exposed on management networks and can change zone state. Limit their listening addresses to an administrative network, require strong authentication, and confirm whether single sign-on through SAML 2.0 or OAuth2 is available for your cephadm deployment. Rotate any dashboard or admin credentials used during the drill.
Audit every failover action
A failover and failback sequence involves many commands across several sites, and the record of who ran what is needed for incident review and for compliance evidence. Capture the period and realm state before and after each step, along with the operator, change ticket and time. Store these records outside the zones that may be down, so the evidence survives the event it describes.
Keep SSE key access working
If objects use server-side encryption with an external key manager, the promoted zone must reach the same keys or reads will fail even though the data is present. Confirm that the key backend, its credentials and its network path are available from the secondary site, and that the key manager itself has a recovery plan. Test a read of an encrypted object from the promoted zone during the drill.
How much does a Ceph RGW multisite failover runbook cost and take
What drives the cost of designing and rehearsing a Ceph RGW failover runbook?
The main driver is how many zones, realms and workloads the runbook must cover, and how many failure scenarios the team wants rehearsed. Cost comes from the people doing the work: architecture review of the zone topology, writing step-by-step procedures for promotion and failback, building a test environment that mirrors production closely enough to be useful, and running the rehearsal with your operations staff present. Costs rise with more zones, more client applications that depend on the endpoints, stricter change control, and any need to validate application behavior during failover. Costs fall when the topology is simple, documentation of the existing setup is current, and the test environment already exists. Ask any quote to itemize design, test environment, rehearsal sessions, runbook documents and handover as separate lines, so you can see what is one-time and what is repeated later.
Can our own team write and rehearse a multisite failover runbook in-house?
Yes, many teams do, provided they have engineers who understand RGW realms, zonegroups, zones, periods and the replication state of their own clusters. The work is mostly careful procedure writing and disciplined testing, so the constraint is usually time and access to a non-production environment that resembles production. A partner adds value when the team lacks recent multisite experience, when there is no safe place to practice promotion and failback, or when an independent reviewer should challenge the procedure before it is trusted. Be honest about the skills on your side. Practice must include the failback path, because that is where many runbooks are weakest.
What does our team need to provide before a failover runbook engagement starts?
Plan to supply the zone topology, including realm, zonegroup and zone names, endpoints, and which zone is the current master. You will also need recent replication status from each zone, the Ceph release running in each site, network paths between sites, and the list of applications and clients that use the endpoints. Provide access to a non-production environment or a window to build one, change windows that allow rehearsals, and named contacts who can approve steps. Most importantly, agree on your recovery point objective (RPO) and recovery time objective (RTO) in writing before design starts, because the runbook's choices depend on them. Ask the provider to confirm exactly which of these inputs they need.
How long does a failover runbook and rehearsal usually take?
Duration depends on scope, not on a standard figure. The timeline is driven by how many zones are involved, whether a suitable test environment exists, how quickly your team can approve changes, and how many rehearsals are needed to prove both promotion and failback. A single zone pair with an existing lab usually moves faster than a multi-zone design that needs a new environment. No duration should be accepted without a written schedule tied to the agreed deliverables. Ask the provider to show the planned rehearsal count, the time reserved for fixes found during rehearsal, and the dependencies on your team's availability.
What changes between a planned failover and an unplanned failover?
A planned failover starts while the master zone is still reachable, so you can stop writes, confirm the secondary has caught up, and then promote it in a controlled order. An unplanned failover happens when the master is already down, so the secondary may be behind by an amount that depends on replication lag at the time of the outage. That gap is what your RPO has to cover, and the runbook must say who decides to promote, how data loss is recorded, and how the old master is handled when it returns. Runbooks should cover both cases with separate procedures and checks. Rehearsing the unplanned path is harder because it needs a simulated outage, not a clean handover.
How is a fixed-scope failover engagement quoted?
Most engagements are fixed-scope, which means scope, deliverables and timelines are agreed in writing before work starts. A quote should follow directly from that written scope, so ask that it lists the zones and platforms covered, the scenarios to be documented (planned, unplanned, failback), the test environment responsibilities, the number of rehearsal sessions, the documents delivered and the support period after handover. Ask what is excluded, such as application testing, changes to client code, or work on a zone that is not in scope. DeployOpen does not publish a price list, so compare quotes on the written deliverables rather than on headline totals. Ongoing rehearsals or upgrade support are typically handled as a separate retainer.
Limits to confirm in your environment
The failover steps in this runbook assume a specific Ceph release, a working multisite configuration and a recoverable master zone. Each card below names one condition to verify in your own cluster before a real failover, and explains how to confirm it.
Release line and support window
Confirm which Ceph release your zones run and whether that release is still in its maintained window. The Ceph releases page lists Tentacle (20.2.x) and Squid (19.2.x) as actively maintained, with estimated end-of-life dates for each. Run ceph versions on every zone and compare the output against the release table. If a zone runs a release outside the maintained window, plan an upgrade before you depend on failover, because backports and security fixes will not reach it.
Multisite behavior on your release
Multisite promotion and period handling are documented per release, and command behavior can differ between versions. Read the Object Gateway multisite section for the release you run, and confirm that the zonegroup, zone and period commands you plan to use exist with the options your runbook expects. Rehearse promotion and failback in a lab zone group built from the same release before you rely on it in production.
Sync lag and replication limits
Failover is only as current as the secondary zone. Measure the sync state between zones with radosgw-admin sync status and record the typical lag under your write load. Confirm the bucket and object counts the sync process can handle at your size, including how resharding interacts with multisite, and agree with application owners on the maximum data loss window you will accept. Objects written to the master but not yet synced are not present on the promoted zone.
S3 feature behavior during failover
Lifecycle rules, bucket notifications and Object Lock do not behave the same way on a promoted zone as they did on the master. Lifecycle processing and notification delivery depend on where the work runs and what state has synced, so list the features your applications use and test each one against a promoted zone. Confirm how retention and legal holds replicated before the outage are enforced, and document which expirations or event triggers may be delayed or repeated after failback.
Load balancing and certificates per zone
Each zone needs its own endpoint, virtual IP and TLS certificate that clients reach after the switch. Check that the ingress service or external load balancer for the secondary zone is configured, healthy and routing to RGW daemons, and that its certificate covers the names your clients use, including wildcard names for virtual-host buckets. Confirm DNS records and their TTLs so applications can move to the secondary endpoint without manual changes.
Recovery posture of the original master
When the original master returns, its data may be stale or diverged from writes accepted on the promoted zone. Confirm how the master's data and metadata will be reconciled before it rejoins, which backups or snapshots exist, and who approves the decision to resync or rebuild the zone. Run radosgw-admin period get on each zone to record the current period and realm state, and keep a copy of it before any change so failback can be verified.
How DeployOpen scopes and rehearses RGW failover
DeployOpen plans, implements, documents and supports self-hosted open-source platforms inside your own infrastructure, so the failover runbook, the rehearsal and the handover all run in your environment. Each engagement starts from a written scope that names the zones, the period and the failure cases you need covered.
How we scope
- Inventory both zones, the zone group, the current period and the realm each site belongs to
- Agree in writing which failure cases the runbook covers, such as a master zone outage or a partial site loss
- Confirm which client applications, replication lag tolerances and sync checks define a safe promotion point
- Engagements are fixed-scope, with deliverables and timelines agreed before any work starts
How we build
- Write the promotion procedure, including committing the period from the secondary zone and the checks before it
- Document what fails while the master zone is down, such as writes to the old master and metadata changes
- Rehearse promotion and failback in a non-production copy of your multisite setup before any production change
- Record the commands, config keys and expected outputs so each step can be repeated and checked by your team
How we hand over
- Deliver the runbook, rollback steps and a failback procedure for restoring the original master zone
- Walk your operations and storage teams through the rehearsal results and the open risks they must accept
- Provide a checklist for confirming replication health after promotion and again after failback
- Agree a list of items to confirm with your Ceph vendor or your own environment before go-live
How we support
- Ongoing support, upgrades and continuous testing are available as retainers, so the runbook stays current
- Review the runbook when the Ceph release in either zone changes, using the release status shown on the Ceph releases page
- Repeat rehearsals on a schedule you agree, and after changes to zone configuration, endpoints or certificates
- Keep evidence of each rehearsal to support your internal reviews and any assessment your auditors request
