When a GitLab Geo planned failover is the right move
A planned failover is a controlled switchover that promotes a Geo secondary site to become the primary while the current primary is still healthy enough to be stopped cleanly. You reach for it when the primary has to move for maintenance, when a region problem is forecast or underway, or when your team wants to rehearse the move before it is needed. Unlike an emergency response, the old primary is still available to drain writes, so the process can be sequenced and checked at each step. GitLab Geo provides a warm-standby secondary in another region that is continuously synchronized for disaster recovery, according to the GitLab Geo documentation. The secondary is kept current by replication, but it is not a substitute for a backup. Geo replicates the state of the instance, so an error, a bad migration or an accidental deletion can replicate to the secondary as well. That is why Geo still needs backups of the primary, and why your recovery plan should name both the Geo site and the backup archive it falls back to. Geo is available in the Premium and Ultimate tiers for self-managed GitLab, so confirm the tier of every instance involved before you plan around it.
Before you commit to a switchover, confirm three things with your own team: that the secondary is healthy and has caught up with the primary, that you have a current backup that could restore the instance if the switchover goes wrong, and that someone owns the decision to put the primary into maintenance mode. Check the replication status on the secondary, record the time you start, and agree who communicates with users. If any of these is unclear, a planned failover is probably not the right move yet, and a rehearsal in a non-production environment will show where the gaps are.
Planned switchover versus emergency failover
A Geo failover can start from a healthy primary or from a primary that has already failed, and the two paths carry very different data-loss exposure. The cards below set out what separates them and why the planned path is the one worth rehearsing before an incident forces the emergency path.
Planned switchover defined
A planned switchover runs while the primary is healthy. Writes are paused through maintenance mode, the secondary is allowed to catch up with replication, and only then is the secondary promoted. The old primary is demoted and becomes a secondary. Because the primary is still reachable, the team can confirm replication is drained before the cutover. Check that your change window covers the full catch-up time, which depends on your replication backlog at the moment you start.
Emergency promotion after primary loss
An unplanned promotion happens when the primary is down or unusable and the secondary must take over. Changes that the primary had not yet replicated may be lost, so the data-loss exposure is the replication lag at the moment of failure. Recovery is also slower because the team is diagnosing the outage while promoting under pressure. Expect operators to make decisions with incomplete information about what did and did not reach the secondary.
Data-loss exposure compared
In a planned switchover, the goal is to drain replication so that no committed change is left behind on the old primary. In an emergency promotion, unreplicated writes are the exposure, and their size is only known after the fact. Ask your vendor or your own tests to confirm how Geo reports replication position and what recovery point you should expect in each path, rather than assuming parity between the two.
Why rehearse the planned path
The planned path is the one your team can practice safely, with the primary healthy, a known maintenance window and reversible steps. Rehearsals expose gaps in DNS changes, CI runner endpoints, certificates and access controls that would otherwise surface during an outage. Run a planned switchover in a non-production Geo pair first, then record timings and the exact commands used.
Re-adding the old primary
After an emergency promotion, the former primary may hold data the new primary never received, so it cannot simply rejoin as a secondary. Plan how that divergent state is inspected, preserved and resolved before the old site is re-added. A planned switchover avoids this problem because the old primary is drained first, which makes re-adding it a routine step in the procedure.
Confirm recovery behavior first
Before you write a failover runbook, confirm in the Geo documentation which replication components must be healthy, how replication lag is reported and what happens to in-flight writes during promotion. Record the version you run, because Geo behavior and commands change between releases. Keep backups in place as well, since Geo replicates data but does not replace a restore plan.
Which failover path fits your site layout
This table compares how the promotion steps change across three Geo site layouts: a single-node secondary, a multi-node site built on the reference architecture pattern, and a site that relies on an external or managed database and object storage. Read each row against your own topology, then use the confirmation column to build the list of questions for your GitLab contact before the rehearsal.
| Layout | What changes in promotion | What to confirm before the rehearsal |
|---|---|---|
| Single-node Geo site | Promotion runs on one Linux package host per site, so the replica database, Gitaly and object data are handled by the same node. Fewer components means fewer steps, but the whole site is affected by each change. | Confirm the exact promotion command and its order for your installed version, and whether the secondary needs a restart after promotion. |
| Multi-node site (reference architecture pattern) | Promotion touches each role separately: PostgreSQL with Patroni, Gitaly Cluster with Praefect, Redis Sentinel and application nodes. Steps must be run in the order your version documents, and each role must reach a healthy state before the next starts. | Confirm which promotion steps run on which node types, the Patroni and Praefect checks that must pass first, and how Consul and Sentinel are expected to behave during the switch. |
| Sites with external or managed database | Replication and promotion of PostgreSQL happen in the managed service, not on GitLab nodes. The GitLab side changes DNS, Geo configuration and application role, so the database provider's failover steps must be sequenced with GitLab's. | Ask the database provider how promotion of a replica works, how long writes are blocked, and whether GitLab's Geo replication is supported on that service for your version. |
| Sites with external object storage | Object data such as uploads, LFS, artifacts and registry blobs is replicated by the storage provider or by Geo, depending on configuration. Confirm the replication lag on the target bucket before promotion, because missing objects are not visible in the database check. | Confirm which object types Geo replicates for your setup, how to check replication status per bucket, and what the provider's replication guarantees and lag are. |
| Sites using a mix of external services and GitLab-managed components | Each external service adds its own promotion sequence, and failure in one can leave the other in a split state. Plan a single runbook with named owners for each component and a stop point if any check fails. | Confirm the supported combinations of external and bundled services for your version and whether any component must stay on the old primary during the switch. |
| Re-adding the old primary | In every layout the old primary becomes a secondary and must resync. The resync path differs by layout, so the rebuild plan should match the same topology used for promotion. | Confirm the re-add procedure for your layout and version, the expected resync method, and whether any manual cleanup is needed on the old primary before it rejoins. |
How to run the switchover in order
The sequence moves write traffic from the current primary to a verified Geo secondary with no unreplicated changes left behind, then brings the old primary back as a secondary so the site pair is protected again.
Confirm replication health and lag on the secondary
Before anything changes, the operator checks the Geo sites dashboard on the primary and the replication status on the secondary. Every replicable data type should show as synced, and the database and repository lag should be small and falling. The Geo documentation lists the health check tasks and the commands for each topology. Do not continue while any data type shows failures, because those changes would be lost at promotion. Record the current state so you can compare it after the switch.
Enable maintenance mode on the primary
Turn on maintenance mode on the primary so users and API clients stop making writes. This is the step that stops new data from being created on the site you are leaving. Confirm that the instance reports maintenance mode as active and that a test write from a non-admin account is refused. Pipelines already running will continue, so decide in advance whether you wait for them or cancel them. Exact settings and the maintenance mode commands are in the Geo documentation.
Let replication catch up to zero backlog
With writes stopped, wait until the secondary has received every change the primary has recorded. Watch the replication queues and verification status on the secondary until no items remain pending for any data type. A backlog that does not shrink usually points to a failing component, which should be fixed before promotion. Only proceed when the queues are empty and the secondary reports the same counts as the primary, because anything still queued at promotion will not reach the new primary.
Promote the secondary using the topology command
Run the promotion command that matches your site layout. The Geo documentation gives one procedure for single-node sites and a separate one for multi-node sites, where the steps differ for database, Gitaly and application nodes. Run it only on the secondary and only after the previous step is confirmed. Expect the secondary to restart services during promotion. The step is complete when the promoted site reports itself as the primary and accepts writes after maintenance mode is lifted.
Update DNS or the access path to the new primary
Point users, runners and integrations at the promoted site. This may mean changing the DNS record for the GitLab hostname, updating a load balancer or unified URL, or both, depending on how your site is published. Lower any DNS caching time ahead of the change if your environment allows it, and confirm the TTL with your network team. Check that the new hostname resolves to the promoted site from each location that uses GitLab, and that the TLS certificate covers that name.
Verify the new primary before lifting maintenance mode
Test the promoted site while it is still in maintenance mode. Sign in, push and pull a test repository, open an issue, run a small pipeline on a runner, and confirm that container registry and artifact actions behave as expected for your configuration. Check the logs and the health checks in the Geo documentation. Once these pass, lift maintenance mode and watch for errors during the first hour. Keep the old primary in its current state until these checks succeed.
Re-add the old primary as a secondary
Bring the former primary back as a Geo secondary of the new primary so the site pair protects data again. The re-add procedure in the Geo documentation may require the old site to be reset or its data to be replaced, which removes writes it made after the switch, so confirm this with your team before running it. Replication then has to catch up from scratch or from the point of divergence. The step is complete when the old primary shows as a healthy secondary with replication lag returning to normal.
Re-adding the old primary as a secondary
The former primary cannot simply be restarted and put back into service, because after a planned switchover it holds data that the new primary may have moved past. Writes accepted before the switch may not have been replicated, and the old primary's database, Git repositories and object storage can diverge from the site that now takes changes. Treat it as a site to be resynchronized from the new primary, and plan that work before the failover window opens rather than during it. Confirm the exact re-add sequence, including any data-reset or replication-setup steps, in the Geo documentation for your installed version, since these steps differ across releases and between single-node and multi-node sites.
Before the old primary serves as a secondary again, check what state it may hold. Its PostgreSQL data, Gitaly repository storage, uploads and any object storage it used may contain records that were never replicated, and some of those records may conflict with work done on the new primary. Confirm with your own runbook which data the procedure discards or rebuilds, and whether anything must be exported first for review. Also confirm that the gitlab-secrets.json file on the old primary matches the one on the new primary, because a mismatch leaves encrypted values unreadable on the secondary.
Verify consistency through the Geo status views and replication checks described in the documentation, not by inspection alone. Confirm that the secondary reports replication and verification for repositories, uploads and other data types as healthy and that the backlog is draining. Run a test clone and a test push against the new primary, and compare a sample of recent merge requests, issues and pipeline records across both sites. Record the results, because the same checks are the evidence you will need before you plan the next switchover in the opposite direction.
What goes wrong during a switchover
A planned switchover fails in predictable places: the new primary is promoted before it has the data, traffic still reaches the old site, or the two sites no longer agree on version or secrets. Each card below names one of these failure modes and the check that catches it before cutover.
Replication lag at promotion
Promoting the secondary while it still trails the primary means recent writes exist only on the old site. Before entering maintenance mode, confirm the Geo sync and verification status on the secondary shows nothing outstanding, and repeat the check after the primary stops accepting writes. Treat any remaining gap as a data-loss decision to make in writing, not a detail to fix after promotion. The final sync must finish before promotion starts.
Writes bypassing maintenance mode
Maintenance mode blocks web writes, but Git over SSH, API clients, CI jobs and scheduled tasks can still change data if they are not stopped. Check that runners are paused, webhooks and integrations that push data are disabled, and that a test push over SSH and an API write both fail on the primary. Record which paths you tested so the cutover evidence shows the freeze held.
DNS and certificates still pointing at the old primary
Users, runners and integrations resolve the GitLab hostname through DNS, and a stale record or a certificate installed only on the old site will send traffic to the wrong place. Lower record TTLs before the window, confirm the new site serves a valid certificate for the external URL, and verify resolution from the networks where runners and offices sit. Ask your network team to confirm which records they must change.
Mismatched GitLab versions across sites
A secondary running a different GitLab version from the primary cannot be relied on for a clean promotion, and a version mismatch can also block the re-add of the old primary. Compare the installed version on every node at both sites before the window. If they differ, stop and align them using the documented upgrade path rather than promoting across the gap. Keep runner versions aligned as well.
Secrets files that differ between sites
The gitlab-secrets.json file holds the keys that decrypt database values such as CI variables and two-factor secrets. If the secondary was built with a different copy, promoted data may fail to decrypt. Before the window, compare a checksum of gitlab-secrets.json on each node with the primary and confirm the copies match. Keep the secrets file in your backup set as well, since a mismatch found during an incident is much harder to fix.
Security and operations during failover
During a planned switchover, the secondary changes from a read-only replica to the site that accepts writes. Access controls that protected the primary must be checked on the secondary before users and integrations arrive, because a site that was only a replica may have had looser network or administrator settings.
- Network path and allowlists: Confirm that firewall rules, load balancer entries and DNS records for the secondary match the policy your team applies to the primary. Confirm which replication ports stay open after the switchover and which close.
- Identity provider configuration: Verify that SAML, LDAP, generic OAuth 2.0 and other sign-in providers point to the new primary's external URL, and that group mappings and SCIM provisioning still apply.
- Administrator accounts: Limit who can run promotion commands and who holds administrator roles during the window. Confirm that privileged actions on both sites reach your central log store.
- Tokens and runner trust: Check which runners, deploy tokens and API tokens remain valid after promotion. Rotate any credential that may have been exposed during the event.
Before sign-off, ask for a test sign-in from each user group and proof that removed users are denied access on the new primary.
GitLab keeps encryption keys for database columns and other sensitive data in gitlab-secrets.json. If the secondary holds a different copy, data that the new primary must decrypt may fail to load after promotion. The GitLab backup and restore documentation explains why the secrets file must match the data it protects, and the same discipline applies across Geo sites.
- Single source of truth: Confirm that gitlab-secrets.json on the secondary matches the primary before the switchover. Compare checksums and do not assume the file synced.
- Change control: Treat any change to the secrets file or gitlab.rb on either site as a change request, with the same review you apply to other production configuration.
- Key storage: Keep copies of the secrets file in an access-controlled vault that is separate from the GitLab hosts, with restricted read access and logged retrieval.
- Certificates and SSH host keys: Confirm that the TLS certificate for the new primary's hostname is installed and that the SSH host keys are the ones your users' clients already trust, so the switchover does not trigger host key warnings.
Ask your GitLab administrator to show where the secrets file lives on each site, when it was last compared and who can read it.
Change management and incident records need proof of what happened, when and by whom. A planned switchover leaves evidence in GitLab logs, in host and infrastructure logs and in the change ticket. Collect that evidence while the work is underway, because logs rotate and terminal history is easy to lose. DeployOpen prepares systems, controls and evidence for compliance assessments, but independent auditors and certification bodies issue any opinion.
- Change record: Link the change ticket to the replication status checks, the maintenance notice, the promotion command output and the post-switchover verification results.
- Command transcripts: Record the terminal session or automation job used for each promotion step, with the operator identity and timestamps.
- Log retention: Confirm that GitLab logs and host audit logs from both sites are shipped to a store with retention set by your policy before the window opens.
- Approval trail: Keep written approval from the business owner and security sign-off, along with the rollback criteria agreed before the window.
Check that a reviewer who did not take part in the switchover can rebuild the timeline from the evidence alone.
Maintenance mode stops users from making changes, so operators must tell people and systems what to expect. The notice should give start and end times, what remains available and who to contact. Confirm the maintenance and switchover behavior for your GitLab version in the GitLab Geo documentation before writing the notice.
- Audience list: Name every group that needs notice, including developers, release managers, support staff and owners of pipelines or scripts that call the instance during the window.
- Pipeline and merge freeze: Ask teams to pause scheduled pipelines and merges that would fail during the window. List external systems that call the GitLab API so their owners can plan for failures.
- Status channel: Publish updates on a channel that still works if GitLab is unavailable, such as an email list or a status page hosted outside the GitLab sites.
- Next update time: Announce the time of the next update and the time of the go or no-go decision. Do not extend the window silently.
Ask the operator for the notice template, the distribution list and the name of the person who approves the final go or no-go.
Operators need to know whether the new primary is healthy before users return, and they need to decide in advance when to go back. Watch replication state and the health of both sites during the switchover and for a defined period afterward, using GitLab's health checks, logs and Prometheus metrics where they are enabled.
- Replication before cutover: Hold the switchover until replication has caught up to the point named in the change record. Write down the value you observed.
- Post-promotion checks: Test web access, Git operations, runner job pickup and sign-in on the new primary against a written checklist.
- Error watch: Monitor logs and metrics for failed jobs, authentication errors and growing background job backlogs for the agreed period after the window.
- Rollback decision points: Agree in advance which failures trigger a return to the original site, who makes that call and the latest time it can be made. After users write data to the new primary, a return needs its own plan.
Confirm that the rollback runbook has been rehearsed in a non-production environment and that the decision owner is reachable for the whole window.
Limits to confirm in your environment
A planned Geo switchover only works when the two sites match in ways the documentation cannot check for you. The items below are confirmations to make against your own environment and vendor documentation, not figures supplied here. Work through them before scheduling the rehearsal, because each one can turn a planned switchover into an unplanned outage.
Version and upgrade stop alignment
Confirm that the primary and every secondary run the same GitLab version, and that both sites sit on the same side of any required upgrade stop. GitLab requires upgrades to pass through stops such as x.2, x.5, x.8 and x.11 since 17.5, so a site that lags behind can block promotion or replication. Check the current version on each node and the planned path before the window opens. Use the same version on both sites as the gating check.
Geo tier entitlement
Confirm that your license covers Geo on every site involved. Geo is available in the Premium and Ultimate tiers for self-managed GitLab, so a Free instance cannot run a secondary at all. Ask your licensing contact whether the secondary site is counted under the same subscription and whether the license file is installed on both sites. A missing entitlement often surfaces only when the secondary is activated.
Network ports and bandwidth between sites
Confirm the ports and throughput that replication and the failover path need between the primary and secondary, including HTTPS for the Geo tracking traffic and database replication. Measure sustained bandwidth under your real change rate, not a single speed test, because replication lag grows when write volume exceeds link capacity. Ask your network team to confirm firewall rules in both directions before the rehearsal, not during the window.
Object storage replication
Confirm how artifacts, LFS objects, uploads, packages and registry blobs are stored and replicated. If they sit in object storage, Geo replication of those objects depends on your storage provider's replication or versioning configuration, which the GitLab backup command does not cover. Ask the storage team which buckets replicate, how long replication lag can be, and whether the secondary can read objects that were written just before the switch.
Runner and integration endpoints
List every runner registration, webhook, Kubernetes cluster, chat integration and external CI system that points at the primary hostname. Each one must repoint, be reachable from the new primary, or be paused during the switch. Runners that keep polling the old address can run jobs against a site you believe is read-only. Confirm which endpoints use the shared hostname and which use the site-specific one before the rehearsal.
Rehearsal coverage of real data volume
Confirm that the rehearsal runs against a copy with a database size, repository count and object volume close to production, because replication catch-up time scales with data. A rehearsal on a small test instance can show a clean switch that fails at production size. Record how long the primary took to quiesce and how long the secondary took to catch up, then compare those timings with your agreed maintenance window.
Questions teams ask before a Geo switchover
How much effort does a rehearsed Geo switchover take?
The effort depends on how many sites are involved, how much data sits in the primary, how many DNS records and integrations point at the instance, and how many approvals the change needs. A single primary with one secondary and a short list of integrations is a very different job from a multi-node site with several secondaries, custom load balancing and external systems that cache the hostname. Ask any provider to break the estimate into discovery, rehearsal in a non-production copy, the production switchover, and the re-add of the old primary, because the rehearsals usually take more time than the live event. Confirm the exact replication status checks and promotion commands against the version you run in the GitLab Geo documentation before you accept any figure.
What does our team need to provide for a switchover engagement?
The work needs named owners on your side, usually someone for the GitLab instance, one for DNS and networking, one for each connected integration such as runners, identity or chat, and one person with authority to approve the change and call a rollback. It also needs administrative access to the primary, every secondary, the load balancers and the DNS zone, plus a maintenance window agreed with the business. Expect to provide current configuration files, the list of runners and external systems that point at GitLab, and recent backup evidence. Your change approval process decides how many review cycles fit between rehearsal and production.
Can our in-house team run a Geo switchover without a partner?
Yes, if your team has operators who have run Geo promotions before and a rehearsed runbook that matches your topology. Most of the risk sits in steps that are easy to skip under pressure: pausing replication correctly, confirming the secondary has caught up, updating the external URL, and repairing the old primary so it can rejoin as a secondary. A partner adds value mainly as an independent rehearsal, a second operator during the live window and written evidence for your change record. Decide based on whether your staff have done this recently, not on whether they could read the documentation.
How long does a planned Geo switchover take?
The duration is set by how long replication takes to catch up, how long maintenance mode must stay on, and how long your DNS and integrations take to reflect the new site. Those values depend on your instance size, write volume at the time of the change and your DNS time-to-live settings, so no single number applies across environments. Ask for the duration of each phase separately and for the assumptions behind each one, and check replication lag during your rehearsal rather than trusting a planning figure.
How is the cost of a Geo switchover structured?
A fixed-scope engagement is priced from a written scope that lists sites, topology, integrations, rehearsal count, the production window and the re-add of the old primary. DeployOpen does not publish a price list, so the quote follows from that scope. Ask whether the quote is one-time or includes follow-up work, whether rehearsals are included or billed separately, and what happens if the rehearsal finds issues that need more work before the live window. Your internal cost also includes staff time for approvals, testing and the maintenance window, which a quote will not show.
Does Geo replace the need for backups?
No. A Geo secondary is a warm standby that is continuously synchronized, which helps with failover, but it does not replace point-in-time backups. Replication copies changes, including mistakes and deletions, so a bad change can reach every site. Keep regular backups, including the gitlab-secrets.json file and configuration, and test restores separately. The GitLab backup and restore documentation describes the backup side of the picture, and the Geo switchover plan should reference it.
What is the main risk during a switchover, and how is it controlled?
The main risk is data loss or split-brain, where two sites accept writes at once. It is controlled by stopping writes through maintenance mode, confirming the secondary has replicated everything before promotion, and keeping the old primary read-only or offline until it is rebuilt as a secondary. A tested rollback path also matters, because the point of no return should be written down before the window opens. Ask any provider how they will verify replication state at each gate and who has authority to stop the change.
Should we check the upgrade path before a switchover?
Yes. Sites in a Geo pair must run compatible versions, so an upgrade planned close to a switchover can change the sequence of steps. Check the required upgrade stops in the GitLab upgrade paths documentation and confirm that both sites are on a version that supports the promotion steps you plan to use. Schedule the switchover and any upgrade in separate windows where possible.
How DeployOpen works on this
DeployOpen plans and delivers GitLab Geo planned failover inside your own infrastructure, from the switchover runbook to the re-added former primary. Engagements are fixed-scope, with the steps, deliverables and timelines agreed in writing before work starts.
How we scope
- Inventory your Geo sites, GitLab versions, deployment layout and whether each site is single-node or multi-node
- Agree which components switch over, including the database, Gitaly, object storage and load balancers
- Confirm the maintenance window, the replication lag you will accept and who approves the cutover
How we build
- Write a step-by-step switchover runbook covering maintenance mode, replication catch-up checks and promotion commands
- Test the promotion sequence on a non-production Geo pair that matches your topology before go-live
- Document the rollback path and the steps for re-adding the old primary as a secondary
How we hand over
- Deliver the tested runbook, verification checklist and a record of the rehearsal results
- Walk your operators through a live rehearsal so the team runs the failover without us
- Provide a post-switchover checklist for confirming repositories, CI and user access work at the new primary
How we support
- Review the runbook after each GitLab upgrade, since version changes can alter promotion steps
- Available as a retainer for periodic failover rehearsals and updates to the procedure
- Scope any post-incident review or runbook change as a separate written piece of work
