Upgrading GitLab Geo primary and secondary sites

Upgrading GitLab Geo sites without losing your disaster recovery position

Upgrade the primary site first, then each secondary, and decide before you start how replication will be handled, because that decision sets the recovery checkpoint your disaster recovery plan depends on. Geo is available in the Premium and Ultimate tiers for self-managed GitLab and provides a warm-standby secondary site in another region that is continuously synchronized. Each site runs a full GitLab installation, so every site must move along the same supported version path. The supported path passes through the required upgrade stops at x.2, x.5, x.8 and x.11, so a Geo estate several versions behind works through those stops on the primary before any secondary follows.

The Geo procedure is layered on top of the normal multi-node upgrade rather than replacing it. Each site still needs its own load balancer handling, component order and post-deployment migrations, and the general steps for a multi-node setup require downtime. Pausing replication is optional. Pausing it on a secondary protects that secondary's disaster recovery checkpoint while the site is changing, so a failover during the window still lands on a known state. Confirm with your team how long the secondary can stay paused and what recovery point your business accepts if the primary fails mid-upgrade. Plan the sequence so that replication resumes and catches up before the next stop.

Two prerequisites are easy to miss. The /etc/gitlab/gitlab-secrets.json file must match between the primary and every secondary, so check it on each site before the first hop and record where the copy is kept. A PostgreSQL major version upgrade also requires re-initializing replication to the secondaries, which can extend downtime well beyond a normal minor step. Ask your GitLab contact or review the Geo upgrade documentation to confirm the exact steps for your version pair, and budget a separate window for any database major change.

What changes in a Geo upgrade and what to do about it

This table compares the changes that affect a Geo upgrade, what each one does to your sites and replication, and the action to take before or during the work. Read each row as a checkpoint for your change plan, and treat the primary-first order and the secrets match as non-negotiable.

Change	Impact	Action needed
Primary before secondary	GitLab upgrades the primary site first, then each secondary, so sites run different versions during the window.	Write the upgrade sequence with named sites, and confirm the primary is healthy before starting any secondary.
Optional replication pause	Pausing replication protects a secondary's disaster recovery checkpoint, but the secondary falls behind while paused.	Decide before the change whether to pause, record the checkpoint, and set a resume and catch-up check after the upgrade.
Matching gitlab-secrets.json	The file in /etc/gitlab/gitlab-secrets.json must match between primary and secondary sites or Geo components will fail to work together.	Compare the secrets file across sites before and after the upgrade, and store a secure copy under your backup controls.
PostgreSQL major version upgrade	A major PostgreSQL upgrade requires re-initializing replication to secondaries, which can extend downtime beyond a normal multi-node upgrade.	Plan a separate change window for the database, confirm with your DBA how long re-initialization will take in your environment, and do not bundle it with a routine patch.
Required upgrade stops	Upgrades must pass through the x.2, x.5, x.8 and x.11 stops, so a site behind several versions needs more than one hop.	Map the hop sequence for every site before scheduling, and confirm each stop is tested on the same version across primary and secondaries.
Monthly minor and biweekly patch cadence	Minor releases ship on the third Thursday of each month and patches twice a month, which affects which target version you choose.	Pick a target that is a supported stop and not a fresh patch, and confirm it against the maintenance policy and your change calendar.

What to back up and verify before touching either site

A Geo upgrade touches the primary and every secondary, so a mistake on one site can leave the others unable to replicate or recover. The checks below establish a known-good restore point, a documented configuration baseline and a rehearsed path before any node is changed.

Current full backup and restore test

Take a full backup of the primary and confirm it can be restored somewhere other than production. An untested backup gives no restore point if the upgrade fails midway. Record the backup time, its size and where it is stored, and confirm the restore completes and the restored instance starts. Include repositories, the database and object storage in the test, since they are backed up through different paths.

Inventory of gitlab.rb and secrets

Keep a written copy of /etc/gitlab/gitlab.rb and /etc/gitlab/gitlab-secrets.json from every site, and record the version each file was taken from. The secrets file must match between primary and secondary sites, so compare checksums rather than assuming the copies agree. Store the copies in a location that is itself backed up and access-controlled, because they contain encryption keys.

Current replication state and lag

Before changing anything, confirm that each secondary is fully synchronized and note the replication lag for repositories, uploads and the database. A secondary that is already behind will fall further behind during the upgrade. Record this baseline so you can tell later whether the upgrade or an existing problem caused any gap in the disaster recovery checkpoint.

Staging copy of both sites

Build a staging environment that mirrors the primary and at least one secondary, using a recent copy of production data where your policy allows it. Run the full upgrade sequence there first, including the secrets check and the replication restart. Time each step and record every deviation from the documented procedure, since those notes become your production runbook.

Load balancer and maintenance window

For a multi-node setup, decide how traffic is drained from each node, which component is upgraded first, and how users are told about the window. The upgrade steps for multi-node installations require downtime, so agree the window with the teams that depend on GitLab, including CI pipelines and any runner fleet that talks to the instance. Confirm the load balancer can route around nodes being restarted.

Target version against the upgrade path

Confirm the target release against the upgrade path and the maintenance policy before scheduling. Required upgrade stops, such as x.2, x.5, x.8 and x.11 since GitLab 17.5, must be passed in order, and the maintenance policy shows which releases currently receive fixes. Also confirm the PostgreSQL version the target release needs, because a major database upgrade changes the downtime plan.

When to pause replication to keep a clean checkpoint

Pausing replication to a secondary site is optional, and you should decide it per site before the primary is touched. Its purpose is to protect the secondary's disaster recovery checkpoint: while replication is paused, the secondary keeps a known-good state from before the upgrade started. If the primary upgrade goes wrong, that secondary is the copy you can still fail over to or restore from without having absorbed the broken changes. The trade-off is lag. While paused, the secondary falls further behind the primary, so every change made on the primary during the window is missing from that site until replication resumes and catches up.

If you keep replication running, the secondary stays current, which is what you want for a fast recovery point. The risk is that a failed upgrade on the primary, including a bad migration or corrupted data, can be replicated to the secondary before anyone notices. At that point the two sites share the same faulty state, and rolling back one does not give you a clean copy of the other. Choose between these based on how far behind you can tolerate the secondary being, how long the primary upgrade window is, and how well you can validate the primary before replication resumes. Write those criteria down in the change plan so the decision is not made under pressure during the maintenance window.

Confirm the exact pause and resume steps, and the state checks that show replication has caught up, in the Geo upgrade documentation for the version you are moving to, because these commands change between releases. Ask your vendor or internal Geo owner to show you the replication status output you will read before and after the pause, and rehearse the resume on a non-production pair if you can. Also confirm that your backup of the primary is taken and verified separately, since a paused secondary is not a substitute for a backup.

Upgrade sequence for the primary and each secondary

This sequence moves a Geo deployment from its current version to the target version one site at a time, starting with the primary. It keeps secrets matched, protects each secondary's disaster recovery checkpoint and leaves a verified rollback point after every site.

Confirm the upgrade path and target version
The platform team checks the GitLab upgrade paths documentation and lists every required upgrade stop between the current version and the target. Confirm that the target is a supported version, that each hop has been tested in a non-production Geo environment and that any background migrations from the previous stop have finished. Done means a written upgrade plan names each stop, the order of sites and the owner of each step. Rollback point: none yet, because no change has been made.
Back up the primary and record secrets
The operations lead runs a full backup on the primary site and confirms it completes without errors. Copy /etc/gitlab/gitlab-secrets.json to a restricted, offline location and record its checksum so the file can be compared later. Done means the backup is restorable in a test environment and the secrets file is stored under access control. Rollback point: the verified backup and the recorded secrets file together form the first restore point for the primary.
Verify secrets match on each secondary
Before any upgrade begins, compare gitlab-secrets.json on the primary with the copy on every secondary site. The files must match, and a mismatch must be resolved before work continues because the sites will not share encrypted data correctly. Confirm this with a checksum comparison and record the result for each secondary. Done means every secondary has a recorded match. Rollback point: a signed-off checksum record for each site.
Decide on pausing replication per secondary
For each secondary, the Geo owner decides whether to pause replication during the upgrade. Pausing is optional, but it protects that secondary's disaster recovery checkpoint, so the team should weigh how far behind that checkpoint could fall against the upgrade window. Record the decision per site with the reason and who approved it. Done means every secondary has a documented pause or no-pause decision. Rollback point: the recorded decision and the checkpoint state at the time of the decision.
Upgrade the primary and run health checks
Upgrade the primary site to the next stop in the plan, following the Geo upgrade documentation, which requires downtime for multi-node setups. Once the services are back, run GitLab's health checks and confirm the web application, Git operations, CI/CD and background jobs work as expected. Done means the primary reports healthy and the upgrade record lists the version reached. Rollback point: a fresh backup of the upgraded primary, taken before the secondaries are changed.
Upgrade each secondary and check replication
Upgrade one secondary at a time in the order set in the plan, keeping its secrets file unchanged. After each upgrade, check replication and verification status in the Geo admin area and confirm the secondary is fully synchronized with the primary. Done means the secondary reports healthy replication without errors before the next site starts. Rollback point: a backup or recorded state for that secondary before the next site is touched.
Resume replication and confirm sync
If replication was paused for any secondary, resume it and watch the backlog until it clears. Confirm that repositories, uploads and other replicated data are in sync on every site and that no replication errors remain. Done means every secondary reports synchronized status at the target version. Rollback point: a documented end state for each site, including the version, the secrets checksum and the time of sync, which becomes the baseline for the next upgrade.

How a PostgreSQL major upgrade changes the downtime plan

A PostgreSQL major version upgrade is the step in a Geo upgrade that most often breaks the assumption that the sites can be upgraded in one pass. The Geo upgrade documentation states that a PostgreSQL major version upgrade requires re-initializing replication to secondaries, and that this can mean longer downtime than the standard procedure. Plan it as its own window, with its own approval, rather than folding it into the GitLab version change on the same night.

A PostgreSQL minor version change is a binary-compatible update, so replication can usually continue across it under the normal Geo procedure. A major version change is different because the on-disk data format changes and the secondary's database must be rebuilt from the primary to match. Your team should confirm with the current Geo upgrade documentation exactly which PostgreSQL steps trigger re-initialization for your versions, and whether any step requires the secondary to be taken out of service while that happens.

Before you commit to a date, ask the team three questions. First, is the database bundled with the Linux package or an external PostgreSQL cluster, since that decides who runs the database steps and how the cluster is reached during the change. Second, what is the database topology on each site, including any standby nodes that also need rebuilding. Third, how long re-initialization takes for your data size. No duration is given here. Measure it by running the full procedure on a staging copy of the primary and a secondary built from production-sized data, and record the time for each step.

Keeping secrets and configuration aligned across sites

The file /etc/gitlab/gitlab-secrets.json holds the keys GitLab uses to encrypt data at rest and to sign internal tokens. Geo expects the primary and every secondary to carry matching copies, because a secondary that cannot decrypt replicated values or verify tokens issued by the primary will fail in ways that look like replication or application faults. A mismatch often surfaces only after a failover test or during a restore, which is the worst time to find it.

Compare checksums, not file names: Hash the file on each node with a command such as sha256sum and compare the results before and after each site upgrade.
Treat the file as a restore prerequisite: Confirm that the copy in your backup set matches the live file, since a backup without the matching secrets file may not restore cleanly.
Restrict copy paths: Move the file only over channels your security team approves, and record each transfer in the change ticket.
Check after reconfiguration: Re-verify the checksum after any gitlab-ctl reconfigure or package change that could regenerate local files.
Plan rotation as a Geo change: Any secret rotation must be applied to all sites in one controlled window, not site by site.

Sign-off on this tab belongs to the security owner of the secrets, with the platform lead confirming the comparison was run on every node.

Most of /etc/gitlab/gitlab.rb should be identical across sites, but some settings are intentionally different. The primary and each secondary have distinct roles, external URLs, and Geo node definitions, so a blind copy from primary to secondary is as risky as ignoring the file altogether. The goal is a reviewed list of intended differences, and anything outside that list should be treated as drift.

Record the intended differences: Keep a short register of each setting that differs by site, such as the external URL, Geo role and node name, and the reason for each.
Diff the rest: Compare the full file between sites after removing the register entries, and investigate every remaining line.
Watch component toggles: Confirm that services enabled on one site, such as a registry or monitoring component, are enabled deliberately on the other.
Version the file: Store gitlab.rb under configuration management so each reconfigure is traceable to a reviewed commit.
Reconfigure after changes: Apply the file with gitlab-ctl reconfigure on each node and keep the output with the change record.

The platform lead owns the register and the diff. The application owner for Geo or the infrastructure architect should approve any difference that is not already listed.

What makes a Geo upgrade take longer

This table lists the factors that stretch a Geo upgrade beyond the base procedure and what each one does to effort. Read each row as a question to answer about your own estimate before agreeing a schedule. The main takeaway is that a PostgreSQL major version change and extra secondaries add the most time, so confirm both early.

Driver	Effect on effort	How to measure it
Data volume and replication lag	Large repositories and a backlog of unsynced data lengthen the window before secondaries are current and make checkpoint decisions harder.	Check replication status on each secondary in the Geo admin view and record lag before the change window opens.
Number of secondaries	Each secondary is upgraded and verified in turn, so effort grows with site count and coordination across teams or regions.	Count secondaries and note which ones must stay within a disaster recovery checkpoint during the work.
Multi-node topology with load balancers	Separate application, Gitaly and database nodes need ordered upgrades, and the general Geo steps require downtime in multi-node setups.	Inventory every node per site and confirm with your team which nodes serve traffic and which hold state.
Bundled versus external PostgreSQL	Bundled databases follow the Linux package upgrade path, while an external cluster adds a separate change owned by your database team.	Confirm where the database runs and who controls its maintenance windows and backups.
Matching gitlab-secrets.json across sites	Mismatched secrets block secondaries from working correctly, so each change needs a verified copy and a check before restart.	Compare checksums of the secrets file on each site and record who holds the approved copy.
Custom integrations and runner fleets	Integrations, webhooks, registry consumers and runners that must match the GitLab version add test and sequencing work across other teams.	List integrations, runner versions and external consumers, then confirm which must be upgraded alongside the sites.
PostgreSQL major upgrade included	A major version change requires re-initializing replication to secondaries, which can mean longer downtime and a full resync.	Confirm with the GitLab upgrade documentation whether the target release changes the PostgreSQL major version, and plan the resync window.

Questions teams ask before upgrading Geo sites

How much does an outside partner cost for a Geo upgrade?

The cost is set by the written scope, not by a list price. DeployOpen does not publish prices, and a quote follows from the scope agreed in writing before work starts. The main drivers are the number of sites (one primary and each secondary), whether your deployment is single-node or multi-node, how many environments need rehearsal, whether a PostgreSQL major upgrade is included, and how much of the backup, secrets and monitoring work is already in place. Ask for the quote to show one-time work (planning, rehearsal, the upgrade itself, rollback testing) separately from any recurring support or retainer, and ask which items are excluded.

How long does a Geo upgrade take, and what does the window depend on?

The window depends on the number of sites, the topology, and the size of the data that must be re-checked or re-synchronized after each site is upgraded. GitLab upgrades the primary site first and then each secondary, and the general steps require downtime in a multi-node setup, so the outage plan has to cover every site in order. Other drivers include the upgrade path between your current version and the target (which may include required upgrade stops), any background migrations still running, and how long your last rehearsal took. Confirm the window against a staging copy that matches production, not against a generic estimate.

What must our team provide before the engagement starts?

Most of the input is access and evidence. You need administrative access to every site, the ability to take and verify a full backup of the primary, a staging environment that mirrors the Geo topology closely enough to rehearse each step, and a named owner for change approvals and the maintenance window. You also need current copies of /etc/gitlab/gitlab-secrets.json from each site, a list of external dependencies such as load balancers, DNS, object storage and runners, and the monitoring you will use to confirm replication health. Ask the partner to list these items in the written scope so gaps are visible before the change date.

Can an internal team upgrade Geo sites without outside help?

Yes, if the team has the skills and time to run the procedure carefully. The work needs people who understand GitLab's Linux package or chart deployment, PostgreSQL replication and failover, Redis, the Geo replication status views, secrets handling and load balancer changes. The harder part is usually rehearsal and rollback planning, because a mistake on a secondary can damage its disaster recovery checkpoint. Assess whether your team has done a multi-node GitLab upgrade before and whether the same people can also cover on-call during the window. A partner can be engaged for the whole job or only for planning, rehearsal or review.

Do we have to pause replication during the upgrade?

No, pausing replication is optional. According to the GitLab Geo upgrade documentation, pausing replication protects a secondary's disaster recovery checkpoint, which is the state you would fail over to if the primary were lost. Pausing keeps that checkpoint clean while the site is changing, but it also means the secondary falls further behind until replication resumes. Decide based on your recovery objectives: if losing recent changes is more costly than a stale checkpoint, plan the pause and the catch-up period together and confirm both in rehearsal.

What happens if a secondary site falls out of sync?

A secondary that falls behind keeps serving the data it has, and it catches up once replication resumes and the primary is reachable. The practical risks are a stale disaster recovery checkpoint, a failover that loses more recent changes than expected, and longer catch-up time after a long pause. Check replication status on every secondary before and after each step, and set a clear threshold for when the change is stopped. If a secondary cannot resynchronize, the recovery path depends on your topology and the GitLab version, so confirm the procedure with the vendor documentation and your own rehearsal results before the window.

Does a PostgreSQL major upgrade add downtime to the Geo upgrade?

Yes, it can. GitLab documentation states that a PostgreSQL major version upgrade requires re-initializing replication to secondaries, which can mean longer downtime than a standard upgrade. The extra time depends on database size, the number of secondaries that must be re-initialized, network throughput between sites, and whether the new major version is run on each site in the same window. Treat the database upgrade as a separate planning item with its own rehearsal, and confirm the expected downtime for each secondary in writing before the change is approved.

How do we roll back safely if the upgrade goes wrong?

Plan rollback before the upgrade starts, because a Geo rollback is different from a single-node rollback. Keep a verified full backup of the primary taken immediately before the change, a matching copy of /etc/gitlab/gitlab-secrets.json for every site, and a written decision point after which you will fix forward rather than revert. Rehearse the rollback on staging, including how secondaries rejoin replication. Also confirm which steps can be reversed and which, such as a PostgreSQL major upgrade, change data formats and may need a restore instead. Ask the partner to show the rollback steps in the scope document.

How DeployOpen scopes and delivers Geo upgrades

DeployOpen plans, implements, tests, documents and supports self-hosted open-source platforms inside your own infrastructure, including the upgrade of a Geo primary and its secondaries. Engagements are fixed-scope, with the scope, deliverables and timeline agreed in writing before work starts.

How we scope

Inventory every Geo site, its GitLab version, node layout, PostgreSQL version and replication state before quoting.
Confirm whether secondaries will be paused, and which disaster recovery checkpoint you need to keep during the window.
Agree in writing who verifies matching gitlab-secrets.json files and who approves the PostgreSQL major upgrade downtime.

How we build

Write a site-by-site runbook that upgrades the primary first, then each secondary, with the order and checks for every node.
Rehearse the procedure in a non-production Geo environment that mirrors your topology, and record the observed downtime.
Test failover and replication health after each stage, and keep rollback steps for each site ready before the change window.

How we hand over

Deliver the runbook, rehearsal results, verification checklist and a record of every configuration change made during the upgrade.
Walk your platform and operations team through replication status checks and the steps to resume paused secondaries.
Document any open items, such as background migrations or re-initialized replication, with owners and dates agreed with you.

How we support

Offer ongoing support, upgrades and continuous testing for your Geo deployment as a retainer, scoped separately from the upgrade.
Cover the next required upgrade stop in the retainer plan so each future Geo change follows the same tested procedure.
Confirm with your team the support hours and response expectations that apply to your Geo sites before signing.