Backup strategy for large GitLab instances starts with your recovery targets
A large self-managed GitLab instance needs a backup strategy chosen from three things: its size, the recovery time and data-loss tolerance the business accepts, and its architecture, whether that is a single node, a multi-node deployment with Gitaly, or a Geo setup with a secondary site. A single scheduled backup task that worked when the database and Git data were small stops being a sound plan once backup windows, restore times and storage growth start to matter. The GitLab backup and restore documentation is the starting point for every option discussed here, but it is not the whole answer. Before you pick a method, confirm the current size thresholds and the guidance for your exact version in that documentation, because recommendations change between releases and should not be taken from older articles or from this page.
Backup archives, disk snapshots and Geo secondaries solve different problems, and a mature plan usually combines them. A backup archive is a portable, logical copy that can restore to a different host and is the reference point for version and edition rules. Disk snapshots can be fast to take but depend on consistency: they need the instance quiesced, or a coordinated read-only window, so that the database, Git repositories and configuration are captured at the same point. A Geo secondary is a warm standby that replicates continuously, which shortens failover but does not protect against logical damage such as an accidental deletion, which replicates too. For that reason, Geo still needs backups. Ask your team which failure modes each control covers, how long a restore takes on your hardware, and who is authorized to run each step.
Which backup approach fits each large-instance situation
This table compares five backup approaches for large self-managed GitLab instances, from the backup task alone to Geo warm standby combined with backups. Read each row for what it protects, what it costs in downtime or operator effort, what it depends on, and when it fits. Geo does not replace backups, so most large instances need a backup approach in addition to any replication.
| Approach | What it protects and costs | Depends on and fits when |
|---|---|---|
| GitLab backup task alone | Produces application data archives from the database and Git repositories. Creating a backup can pause or limit writes depending on the snapshot method, so plan a maintenance window sized by your own timing tests. | Depends on matching GitLab versions and editions at restore time and on enough local or remote space for archives. Fits smaller instances or teams that can tolerate a scheduled window and restore time measured in their own tests. |
| Backup task with a read-only snapshot window | Combines a short read-only period with disk or database snapshots so the database and Git data are captured at one consistent point. Operator effort is higher because the snapshot steps must be scripted and verified. | Depends on storage that supports consistent snapshots across the database and Gitaly volumes and on the snapshot procedure in the backup documentation. Fits instances whose database or Git data has grown past the point where the plain backup task is comfortable, provided the snapshot tooling is tested. |
| Server-side or split repository backups | Lets Gitaly send repository backups directly to object storage and can split repository backups across multiple storages. Lowers network load on the application server, but object storage data such as artifacts and LFS still needs its own protection. | Depends on object storage access, Gitaly configuration and the GitLab version in use, so confirm feature availability and configuration keys for your release. Fits large Git data volumes where the archive host is the bottleneck. |
| Disk snapshots on infrastructure you control | Captures whole virtual machine or volume state, including secrets and configuration if they are on the same disks. Restore is faster to reason about, but snapshots taken without GitLab's consistency steps may not be application-consistent. | Depends on your hypervisor or storage platform, snapshot consistency features and retention policy. Fits environments already running reliable snapshots, with restore drills that prove GitLab starts and data is intact. |
| Geo warm standby combined with backups | A secondary site in another region is continuously synchronized for disaster recovery, reducing time to switch over. Logical deletions and corruption replicate, so backups remain necessary for point-in-time recovery. | Depends on a second site, Premium or Ultimate licensing, network capacity between sites and matching versions across primary and secondary. Fits organizations with strict recovery time goals that also keep independent backups. Confirm replication lag and failover steps for your release. |
How the strategy changes with instance layout
On a single Linux package node, PostgreSQL, Redis, Sidekiq, Gitaly and repository data share one host, so one backup run and one disk snapshot can capture most of the state together. The main gap is that the backup and restore archive does not hold everything: the secrets file and configuration files sit on the same disk and must be preserved alongside each archive.
- Database and repositories together: The backup task writes database content and Git repositories in one run, so the archive is only consistent when no writes happen during it.
- Snapshots with read-only mode: For large data sets, a disk snapshot taken while GitLab is in maintenance or read-only mode gives one point in time across the database and Git data.
- Secrets and configuration: Copy
/etc/gitlab/gitlab-secrets.jsonand/etc/gitlab/gitlab.rbwith every archive, because encrypted values stored in the database, such as two-factor secrets and CI variables, cannot be decrypted without them. - Object storage gaps: If artifacts, uploads or registry blobs live in object storage, the archive skips them, so protect them with provider versioning or replication.
- Headroom on the host: The backup competes with GitLab for CPU and disk I/O on the same machine, so schedule it for a window where that capacity exists.
Confirm that the archive and the snapshot were taken in the same maintenance window and that you can restore both to a spare host running the same GitLab version.
When Gitaly and PostgreSQL run on separate hosts, the database and repository storage are captured at different moments unless writes are paused. The reference architectures recommend a high availability strategy for 3,000 or more users, using Gitaly Cluster with Praefect and PostgreSQL with Patroni, so many teams at this layout will also be running failover components that affect where backups can safely read from.
- Coordinated stop point: Set a maintenance window or read-only mode before starting, so the database and Gitaly data describe the same moment.
- Where the work runs: Repository backups can run from an application node that calls the backup task, or Gitaly can stream them to object storage. Confirm the supported method for your Gitaly layout before choosing.
- Secrets on every application node:
gitlab.rbandgitlab-secrets.jsonmust match on the Rails and Sidekiq nodes, and one verified copy belongs in the backup set. - Failover during a run: A PostgreSQL failover mid-backup can produce an incomplete or inconsistent archive, so check the backup log for errors and rerun when the output is not clean.
- Restore order: Restore the secrets and configuration first, then the database and repositories, then run the reconfigure and verification steps before reopening access.
Check that your restore test covers every node type in the layout, not only the node that runs the backup task.
A Geo deployment gives a warm-standby secondary in another location that is continuously synchronized for disaster recovery. That synchronization changes the recovery question: the secondary protects against losing a site, but it does not protect against losing data that was replicated in error. Backups still run on their own schedule and must be kept independently of the replication path.
- Replication is not a backup: A bad deletion or a corrupted repository is copied to the secondary as well, so keep archives that are older than the replication window.
- Replication lag sets the exposure: Data that has not yet reached the secondary is what a failover could lose, so monitor sync status before and during backup windows.
- Where backups run: Take archives from the primary, which holds the authoritative data, unless your Geo documentation confirms a supported procedure on the secondary.
- Shared secrets across sites: The secondary needs the same
gitlab-secrets.jsonas the primary, so the secrets backup protects both sites and must be kept current. - Separate drills: Test restores from archives and test promotion of the secondary as different exercises, so a failover does not leave you without a verified backup.
Confirm your recovery time and data loss targets against measured replication lag and restore test results, not against the presence of a secondary alone.
Why snapshots need a read-only window to be consistent
A disk snapshot of a self-managed GitLab instance is only consistent if every component is frozen at the same moment. Writes that continue while the snapshot runs can leave the PostgreSQL database, the Git repositories on Gitaly and the uploaded files at different points in time. A restore from that set may show a merge request whose commits are missing, a repository that references a database record that no longer exists, or attachments that the database does not know about. None of these failures is obvious at restore time, which is why the window has to be planned rather than assumed. The usual approach is to place the instance into read-only or maintenance mode for the duration of the snapshot. Users can still browse projects, read issues and view pipelines, but pushes, comments, merges and CI job updates are refused or queued until the window closes. Developers should expect failed pushes and a message that the instance is in maintenance, and CI jobs that depend on the instance may stall. The length of the window depends on how quickly the storage layer can complete snapshots of the database, Gitaly and object or file storage, and on the size of the instance, since larger data sets take longer to snapshot and to bring to a consistent state. Storage speed and the number of volumes involved therefore matter more than the nominal data size. Before relying on this method, confirm the exact procedure, the storage layouts it supports and the order of operations in the current GitLab backup and restore documentation, since these details change between releases and depend on whether data lives on local disks or in object storage. Then run the full sequence on a copy of the instance, time the read-only window, and perform a restore from the snapshots, checking that repositories, merge requests and attachments all resolve correctly. Record the measured window and the restore steps in the runbook so the planned downtime is based on evidence.
How to build and test a large-instance backup plan
This sequence turns a backup habit into a tested recovery capability: every data location is accounted for, recovery targets are agreed with the business, and a full restore proves the copies work at the same GitLab version and edition.
Inventory every data location
Map where GitLab keeps state: the PostgreSQL database, Git repository storage (Gitaly), uploads, CI artifacts, LFS objects, the container and package registry, and any object storage buckets behind them. Note which items the GitLab backup task covers and which live in external object storage, where provider versioning or replication must cover them instead. Also record the configuration and secrets files on each node. The output is a single inventory signed off by the platform owner, with each location tagged with its backup mechanism owner. Confirm the list against your own deployment, because a Geo, Praefect or object storage setup changes what is local.
Agree recovery point and recovery time targets
Meet the business owners of the projects, the release and compliance teams, and the operations leads to agree how much data loss and how long an outage each service can tolerate. Translate those answers into a recovery point objective and a recovery time objective for the instance as a whole and for the most sensitive data types. Record the targets in writing with their owners. The check is that every target maps to a backup mechanism in the next step, and any target that no mechanism can meet is escalated before build starts.
Choose a backup mechanism per data type
For each inventory item, select the method that meets its targets: the GitLab backup task for the database and local repositories, disk snapshots taken with GitLab in read-only mode where the instance is large enough to need them, incremental repository backups, server-side repository backups to object storage, Geo secondaries for warm standby, and provider versioning or replication for object storage content. Document why each method was chosen and what it does not cover. Confirm with your storage and hypervisor vendors that snapshot consistency meets your requirements. The output is a mechanism matrix that a reviewer can challenge.
Separate secrets and configuration backup
Back up gitlab-secrets.json and gitlab.rb on every node into a store with its own access controls, separate from the database and repository archives. Include SSH host keys and TLS material needed for a full rebuild. Restores fail in confusing ways when the secrets file does not match the database, because encrypted fields such as CI variables and tokens become unreadable. The check is a documented copy of each file, stored separately, with a dated record of when it was last captured and who can read it.
Schedule and monitor backup jobs
Set the backup schedule for each mechanism so the windows match your recovery point objective, and stagger heavy jobs so they do not compete with peak CI or push traffic. Alert on failed jobs, missing archives, jobs that run longer than their previous successful run, and snapshots that were not confirmed. Make sure someone receives those alerts during working and out-of-hours periods. The check is a dashboard or report that shows the last successful run of every mechanism, reviewed on a set cadence.
Set retention for each backup type
Define how long each archive, snapshot, incremental and object storage version is kept, and align the retention with the recovery point objective and any legal or contractual hold requirements. Remember that incremental repository backups form a chain to a previous backup ID, so deleting an earlier link can make later increments unusable. Enforce retention with automated policies where possible. The output is a retention table, approved by the data owner, and a test that a restore point from the oldest retained date still exists.
Run a full restore in a non-production environment
Restore the secrets and configuration first, then the database and repository data, into a separate environment built from the same GitLab version and edition as the source. Follow the same-version, same-edition rule and keep matching packages available. Time the restore and record the actual duration against the recovery time objective. Verify the result: the restored instance signs in with a test account, a sample of repositories clones over HTTPS and SSH, a merge request shows its history, and a CI job runs and can read a project variable. Log every failure, because the restore is only proven when these checks pass.
Document the runbook and rehearse it
Write the recovery runbook from the restore run: the order of steps, the commands used, the credentials and access needed, the decision owner for failover, and the expected checks at the end. Include the steps for Geo promotion if a secondary site is in use. Schedule periodic rehearsals so the runbook keeps pace with GitLab upgrades and architecture changes. The output is a versioned runbook that a second engineer can follow without help, plus a log of the last rehearsal showing the measured restore time and any gaps found.
Protecting backups and the keys they depend on
A large GitLab backup is only useful if its encryption keys, configuration and object storage data restore alongside it. The cards below cover the items that most often go missing, and each one names what breaks on restore when it is skipped.
Back up the secrets file
The gitlab-secrets.json file holds the keys that decrypt database columns, CI variables and two-factor secrets. The backup archive does not contain it, so a restore onto a fresh install without this file leaves encrypted values unreadable. Store a copy with the archive under the same access controls, and confirm that the file is included in every snapshot plan. Test this by restoring to a scratch host and checking that CI variables and two-factor logins still work.
Keep configuration under version control
Files such as gitlab.rb and the SSH host keys define how the instance behaves and how clients recognize it. Without them, a rebuilt server may use different ports, URLs or host fingerprints, and developers see host key warnings. Record configuration changes in a Git repository with restricted write access, and copy the host keys with the other recovery material. Check that the restored node uses the same external URL before users reconnect.
Encrypt archives and snapshots at rest
Backup archives contain source code, issue text, and credentials that may be stored in project data. Encrypt them with keys held outside the GitLab host, and encrypt snapshot volumes at the storage layer. Confirm who can read the decryption keys and whether those keys are backed up. A restore fails if the key material is lost, so treat key escrow as part of the backup design.
Restrict access to backup storage
A backup location with broad read access exposes the whole instance to anyone with a copy of the archive. Limit write and read permissions to the backup service account and a small restore group, and separate them from general administrator access to the GitLab server. Log access to backup buckets or directories. Review these permissions when staff change roles, since backup access is often granted once and never revoked.
Cover object storage data separately
When artifacts, LFS objects, uploads or registry blobs live in object storage, the GitLab backup task may skip them. A restore then succeeds for Git data but leaves broken links to attachments and missing build artifacts. Use provider versioning or cross-region replication for these buckets, and test a restore that reads a sample of each object type. Document the bucket names and retention rules in the same runbook as the archive.
Alert on failed backup jobs
A backup job that fails silently can leave the team with weeks of gaps that are only discovered during an incident. Monitor the job exit status and the age of the newest archive, then alert the on-call channel when either is out of tolerance. Also track archive size: a sudden drop often means a partial backup. Pair these checks with a scheduled restore test so that a successful job is also a usable backup.
What large-instance backups cost and what they need from your team
What drives the cost and effort of a large GitLab backup design?
Scope and instance shape drive the cost most: database size, Git data volume, how many storage nodes hold repositories, whether artifacts and registry blobs sit in object storage, and whether the instance is single-node or multi-node. The work splits into four components: design (choosing snapshot, Geo, file system transfer or split repository backup approaches), implementation (scripts, schedules, storage and access changes), testing (restore rehearsals, which are the main proof the design works) and ongoing operation (monitoring, retention, version changes). Design and implementation are largely one-time. Testing and operation recur, and they grow with each GitLab upgrade and each new storage node. Ask any provider for a quote that lists each component separately, names who performs each one, and states which items assume your existing infrastructure.
Can our team run the GitLab backup task alone?
Many teams can run the basic GitLab backup task alone on a small instance, but large instances need more planning than the default task covers. The task itself is documented in the GitLab backup and restore documentation, and it skips data that lives in object storage, so that data needs its own provider-side versioning or replication. Whether your team can take on the rest depends on skills you already have: snapshot tooling for your hypervisor or cloud, PostgreSQL and Gitaly familiarity, scripting for scheduled jobs and experience with restore drills. Ask yourself who would run a restore at 2 a.m. without the original author, and whether that person has done it recently.
How long does a restore test take, and what makes it longer?
A restore test takes as long as it takes to move your data, rebuild the services and verify the result, and no fixed duration applies across instances. The main drivers are total data volume (database plus Git repositories), storage throughput between the backup location and the target, the number of Gitaly nodes to rebuild, whether object storage data must be re-attached, and how many people must check results. Restore tests also need a separate target environment, which adds infrastructure cost. Ask for a plan that states the environment used, the data set tested, the pass criteria and the steps you will check afterward. Your own measured run on a copy of production is the only reliable number for your instance.
What does Geo add to ongoing cost and operational load?
Geo adds a second site that stays synchronized with the primary, so it adds recurring infrastructure cost for compute, storage and network in a second region, plus the ongoing work of monitoring replication, testing failover and keeping both sites on the same GitLab version. The GitLab Geo documentation describes a warm-standby secondary for disaster recovery, and it is available in the Premium and Ultimate tiers for self-managed GitLab. Geo does not replace backups, so you still pay for backup storage and restore testing. Ask for the expected replication lag monitoring approach, the failover runbook owner and the cost of keeping a secondary through upgrades, since each upgrade must be applied to both sites.
What does the work need from our side?
The work needs named owners and access more than it needs a large budget. Plan for the following:
- Storage: a backup target with enough capacity for retained full and incremental backups, and the snapshot capability of your platform.
- Network: bandwidth and routing between the GitLab nodes, the backup target and any Geo site, and firewall rules for the ports those links use.
- Access: service accounts with the privileges needed for snapshots, database and Gitaly data, plus change approval for production changes.
- Maintenance windows: time when GitLab enters maintenance or read-only mode for consistent snapshots.
- Owners: one person for backups, one for restores, one for the secrets file and one for sign-off on test results.
Confirm each item against your own environment before the design starts.
Must a restore match the exact GitLab version?
Yes, a restore must use the same GitLab version and edition that created the backup. The GitLab backup and restore documentation requires matching versions, so an archive from one release cannot be restored onto a different release. This affects planning in two ways: keep the packages for the version that produced each backup, and treat every upgrade as a point where your backup and restore procedure must be retested. Upgrades also pass through required upgrade stops, documented in the GitLab upgrade paths documentation, so a restore plan must account for the stop versions you pass through. Confirm the rule and the exact package handling with GitLab documentation before you finalize the runbook.
Limits to confirm before you commit to a backup design
A backup design for a large GitLab instance depends on details that change between releases and deployments. The cards below list the points to confirm with GitLab documentation and your own restore tests before you choose snapshots, Geo or object storage handling as your recovery path.
Backup and restore rules for your version
Restore rules are tied to your exact GitLab version and edition. Confirm that the backup you take can be restored onto the same version and edition, and that you can keep matching packages for that version available for the life of the backup. Check the restore procedure in the backup and restore documentation for your release, then run a full restore into a separate environment at least once before you rely on the design. Record the version in every backup manifest so an incident team does not have to guess.
Snapshot support for your storage layout
Disk snapshots taken with read-only mode only give a consistent recovery point if every stateful component is covered by the same snapshot set. Confirm whether your storage layout, including Gitaly storage, PostgreSQL data, Redis and any separate volumes, can be snapshotted together, and how your hypervisor or cloud provider sequences those snapshots. Ask whether restoring a snapshot needs any GitLab-specific steps before services start. Test the restore on a copy, because a snapshot that looks complete can still be inconsistent.
Geo and your chosen backup mechanism
A Geo secondary is a warm standby that synchronizes continuously, so it protects against site loss but does not replace a backup. Confirm how replication behaves when you take a backup or snapshot on the primary, and whether your procedure needs the secondary to be paused or excluded. Ask how a bad change or deleted repository replicates, since a Geo secondary can copy the mistake. Keep backups off the Geo sites' shared failure domain so one incident cannot remove both copies.
Object storage data outside the archive
When artifacts, LFS objects, uploads or registry blobs live in S3, GCS or Azure, the GitLab backup command does not capture them. Confirm which data types in your instance use object storage, and which provider features cover them, such as versioning, replication or lifecycle rules. Check how those copies are restored and whether they match the database and Git data at the same point in time. Without that match, a restored instance can show links to files that no longer exist.
Upgrade stops affecting restore targets
GitLab requires upgrades to pass through required upgrade stops, so a backup taken on one version may need to be restored onto a version that sits on a specific stop. Confirm the path from your backup version to your intended restore target, and whether your target release is one that GitLab currently maintains. Plan the restore target as a stop along the upgrade path, not as the latest release. Keep the upgrade path checklist with the backup records so the recovery team can follow it under pressure.
Runner and CI recovery needs
Restoring the GitLab instance does not automatically rebuild the CI/CD execution layer. Confirm which Runners are self-hosted, how their registration tokens and configuration are stored, and whether their images, autoscaling settings and cloud resources can be recreated. Keep Runner major and minor versions aligned with GitLab during restore. Decide whether Runners need their own recovery runbook so pipelines can resume after the instance is back, and test that sequence.
How DeployOpen works on this
DeployOpen plans, implements, backs up, documents and supports self-hosted open-source platforms inside your own infrastructure, including the cloud accounts, data centers or air-gapped networks you run. For large GitLab instances, we begin with a written scope that fixes what the backup strategy covers and what the restore tests prove before any change is made.
How we scope
- Inventory database size, Git data volume, object storage use and backup windows before recommending a method
- Agree in writing which recovery approach applies: snapshots with read-only mode, Geo, or file system transfers
- Define the recovery point and recovery time targets your organization must confirm with its own owners
- Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts
How we build
- Configure scheduled backups, including secrets and configuration files, on the hosts you control
- Set up snapshot or Geo procedures against the storage platform you already operate
- Split repository backup jobs across storages and confirm each job completes within its window
- Run timed test restores on a separate environment and record the results for your review
How we hand over
- Document the backup schedule, retention rules, storage locations and the exact restore order
- Provide runbooks for read-only mode, failover and re-adding the old primary, with named owners
- Walk your operations team through a restore drill so they can run it without us
- Deliver the scope, test evidence and configuration in a form your auditors can review
How we support
- Offer ongoing support, upgrades and continuous testing as a retainer after handover
- Re-run restore tests after GitLab upgrades, storage changes or growth in database or repository size
- Review backup job health and capacity trends with your team on a schedule agreed in the retainer
- Confirm with your vendor and auditors which retention and evidence requirements apply to your environment
