GitLab Geo disaster recovery

Set the recovery point objective (RPO) and recovery time objective (RTO) for your Geo secondary as two measured numbers per data class, not as defaults copied from a template. The RPO is how much recent data your organization can afford to lose if the primary site fails. The RTO is how long users can be without GitLab before service is back on the secondary. Both numbers depend on your own write volume, network path and promotion procedure, so the reader has to derive them in their environment before they can be agreed or tested. Geo replicates asynchronously, so the secondary is always some distance behind the primary. The RPO you actually get is the replication lag at the moment of failure, which moves with commit and upload volume, peak hours and the health of the link between regions. Measure that lag under normal load and during your busiest period, then look at the worst values seen, not the average. Confirm the replication lag metric names and the commands that expose them in the Geo documentation for your exact GitLab version, and do not assume names carried over from older releases or from monitoring examples.

Data does not all carry the same business weight, so the RPO should not be one number for the whole instance. Ask the business owners to agree an acceptable data loss per data class, for example repository content, merge request and issue metadata, CI job logs and artifacts, and package or container registry content. Some of these may be acceptable to re-create from a developer's local clone or a pipeline rerun, while others may not. Record each agreed value next to the lag you measured for that class. If the measured lag exceeds the agreed value for any class, you have a decision to make: reduce lag through capacity or network changes, accept the gap in writing, or add a separate backup for that class.

The RTO comes from the promotion runbook, and it is only credible when each step has been timed. Break the runbook into stages: detecting the outage, deciding to fail over and who holds that authority, confirming the secondary is healthy, promoting it, switching traffic through DNS or your load balancer, running verification checks and reopening access. Several of these steps are human decisions, and their duration depends on your on-call process and escalation paths, so include them in the estimate. Name the person who can authorize promotion and confirm that they can be reached at any hour. A runbook that has never been executed will consistently understate the time required.

Run a rehearsal to replace estimates with observed values. Schedule a planned promotion drill in a window your business has approved, or use a non-production copy that mirrors your topology, and record the lag immediately before the drill, the elapsed time for each runbook step and the total time until users can complete a standard set of actions, such as cloning a repository, opening a merge request and running a pipeline. Compare the results with the RPO and RTO targets. If the drill misses a target, change the architecture, the runbook or the target itself, and rehearse again. Plan the drill so that it also checks the secondary's ability to serve traffic, not only that the promotion command completed.

Keep the targets under change control. Record each RPO and RTO value with the GitLab version, the date of measurement, the lag metric used and the drill results behind it. Re-measure after GitLab upgrades, significant growth in user or pipeline volume, and changes to network capacity between regions, because any of these can move lag and promotion time. Be clear with the business that Geo protects against site loss by keeping a current copy of data, but a replica also copies unwanted changes such as accidental deletions or corruption. Those cases need the separate backup and restore capability covered elsewhere in this page, and the RPO for them is governed by backup frequency, not by replication lag.

A Geo failover is only safe if you have tested it against your own data, users and network path before you need it. The verification plan starts with replication health on the primary and the secondary site, checked before any test begins and again after each step. Use the Geo sites status view and the replication and verification status for each data type in scope, such as Git repositories, wikis, uploads, LFS objects, CI artifacts and the container and package registries where they are replicated. Record which components show as healthy, which show as pending, and which show verification failures. A failover test that starts from an unhealthy baseline tells you nothing about the failover itself, so treat a red status as a stop condition for the test window. Run the planned failover as a rehearsal in a non-production window with a controlled write freeze on the primary. Announce the freeze, stop pushes, merges and pipeline creation, and wait until the secondary has caught up with the last replicated event before promotion. Then promote the secondary and point users, runners and integrations at it according to your DNS and load balancer runbook. Once the promoted site is serving traffic, validate the work that matters: clone and push to a sample of repositories, open and merge a merge request, run a pipeline that uses a runner registered to the new site, confirm that a representative set of users can sign in through your identity provider, and check that webhooks and API tokens behave as expected. Test with named accounts from each group that carries real work, because group membership and permissions are part of what must survive promotion. Confirm that no data was silently lost by comparing what you expected to exist with what the promoted site holds. Pick a set of marker events before the freeze, such as a known commit SHA, a merge request ID, a pipeline ID and a newly created user, and check that each is present after promotion. Compare repository counts and last-activity timestamps against the pre-freeze inventory, and review the verification checksums that Geo reports for each data type. Any gap should be recorded as a finding, not explained away. Record the actual replication lag at freeze time and the elapsed time for each step, from freeze to promotion to first successful pipeline, and compare those numbers with the recovery point and recovery time targets your business has agreed. If the targets are not written down yet, set them before the test so the result has something to be measured against. A failed promotion must leave the primary intact. Rehearse this deliberately: simulate a promotion that stops partway, such as an unreachable database or a misconfigured external URL, and confirm that the primary still accepts writes after you lift the freeze and that users can continue working there. Ask the implementation team to show exactly which steps are reversible and which are not before the rehearsal starts. Then test failback as its own exercise. Re-establish replication from the promoted site back to the original primary, wait for verification to complete, run the same validation checks in the reverse direction, and only then switch traffic back. Note any drift in configuration, secrets or runner registration that appeared during the time on the secondary. Geo is a replication and promotion mechanism, not a backup. Keep the gitlab-backup process and a tested restore procedure as a separate safety net, because a replicated deletion or a corrupted database is copied to the secondary just as faithfully as valid data. Save the secrets and configuration files that the backup documentation says must be stored separately, and confirm the restore steps for the exact GitLab version you run, since the procedure can change between releases. Schedule a restore test on a separate environment and record how long it takes. Your verification plan is complete only when both the Geo rehearsal and the restore test have passed and their results are written down with the owner of each follow-up.