Should your GitLab Gitaly tier use Gitaly Cluster or sharded Gitaly?
Gitaly Cluster with Praefect replicates each repository across several Gitaly nodes so that a node failure does not take a repository offline, while sharded Gitaly spreads repositories across storage nodes with no replication between them, so each repository lives on one node. If you run or plan self-managed GitLab and must decide how repositories are stored and kept available, that difference is the core choice. GitLab's high availability guidance generally recommends Gitaly Cluster for deployments of 3,000 users or more, but the number of users is only the starting point. The choice also shapes how you back up repositories, how you schedule upgrades, and how much operational work your team takes on.
Start by deciding what outage you can tolerate for a single storage node. If losing one node for a period means developers cannot push or clone the repositories it holds, the replication that Praefect coordinates is the feature you are paying for. Before committing, confirm with GitLab the current limitations of Gitaly Cluster, the supported replication and failover behavior, and how Praefect metadata is backed up and restored, since the GitLab backup and restore documentation treats repository data and cluster state as separate concerns. Also confirm the current status of running Gitaly in Kubernetes, because the Cloud Native Hybrid pattern keeps stateful services such as Gitaly on VMs and that placement affects which option is practical for your platform.
Sharded Gitaly is simpler to reason about, but it moves the availability risk onto each storage node. A single shard failure affects every repository on that shard until you restore it, so your restore procedure, storage capacity planning and monitoring must be tested rather than assumed. Ask your team or a partner to write a short decision record that names your user count, your largest repositories, your recovery time target and the staff who will operate the cluster. Then compare that record against the reference architecture your sizing points to. If you are unsure, validate the layout in a non-production environment before go-live.
Which Gitaly layout fits your environment and risk tolerance
Teams that need repository-level failover at 3,000 users or more usually fall into the high availability reference architecture, where GitLab documents Gitaly Cluster with Praefect as the recommended Gitaly strategy. Praefect keeps multiple copies of each repository on separate Gitaly nodes and can route requests to a healthy replica when one node fails. This gives you a failover mechanism that sharded Gitaly does not provide on its own.
- Replication: Writes are replicated across Gitaly nodes, so a single node loss does not have to mean a repository outage.
- Known limitations: Confirm the current limitations list in the GitLab documentation for your target version before you commit, since behavior and supported features change between releases.
- Backups: Plan backups around Gitaly Cluster explicitly, because replicas are not a substitute for a restorable backup.
- Operations cost: Expect additional nodes, Praefect and its database dependency to monitor, patch and test failover against.
- Upgrades: Confirm how zero-downtime upgrades treat the cluster before scheduling maintenance windows.
Ask your implementation partner to show the failover test plan and the restore procedure for a single repository, not only the replication diagram.
Smaller or single-site instances usually do not need Gitaly Cluster to meet their availability goals. A single Linux package server or a modest multi-node layout can run Gitaly locally, with recovery depending on backups and a tested restore procedure rather than live replication. The question is less whether HA is possible and more whether the failover machinery matches the risk your organization accepts.
- Simpler topology: A local Gitaly instance has fewer moving parts to patch, monitor and explain to auditors.
- Recovery model: Your recovery plan rests on backups, so restore time and backup frequency matter more than replica count.
- Growth path: Plan how you would move repositories to a sharded or clustered layout if user count or CI load grows.
- Operations cost: Lower day-to-day overhead, but the restore drill becomes the critical test.
- Verification: Ask for evidence of a restore into a clean environment, not just a successful backup job.
Confirm with the GitLab sizing documentation which reference architecture your user count and API load point to before you decide the layout is sufficient.
Kubernetes and regulated networks add constraints that change the Gitaly decision. GitLab's reference architecture guidance says stateful components such as PostgreSQL and Redis are not supported inside Kubernetes, and the Cloud Native Hybrid pattern keeps stateful services on VMs or managed services. Gitaly itself is a stateful service, so the layout must place Gitaly outside the cluster or confirm the current supported pattern for your version.
- Beta status: Confirm the current maturity of any Kubernetes-hosted Gitaly Cluster pattern in the GitLab documentation before you rely on it.
- Stateful placement: Keep Gitaly storage on VMs or dedicated hosts that your storage team can back up and restore.
- Air-gapped operation: Offline installs need the same Gitaly artifacts and backup tooling without external dependencies, so test them before the network is isolated.
- Evidence: Auditors will ask how repository replicas are protected and restored, so keep the design and test records together.
- Snapshot backups: Confirm with GitLab whether snapshot-based backups of Gitaly storage are supported for your layout, since GitLab's backup tooling is the documented path.
Ask for the supported-versus-unsupported matrix in writing before the design review.
Gitaly Cluster, sharded Gitaly and the questions that separate them
This table compares Gitaly Cluster, which uses Praefect to route requests and keep replicas of each repository, with sharded Gitaly, where each repository lives on one storage node. Read each row as a question to answer for your own environment. The main takeaway is that Gitaly Cluster adds repository-level redundancy at the cost of more components to run, while sharded Gitaly is simpler but leaves recovery to your backup and storage plan.
| Decision point | Gitaly Cluster (Praefect) | Sharded Gitaly |
|---|---|---|
| Replication model | Praefect routes Git traffic and coordinates replicas of each repository across Gitaly nodes. | Each repository is stored on one Gitaly shard. No built-in replica is kept for it. |
| Failover behavior | Designed to keep repositories available when a Gitaly node fails. Confirm current failover behavior and known limitations in the GitLab documentation. | A failed shard makes its repositories unavailable until the node is restored or recovered from backup. |
| Backup approach | Snapshot-based backups of the cluster are not a documented option. Plan with GitLab backup tooling and confirm current guidance. | Backup and restore run per shard. Confirm the procedure and restore timing against your own data volumes. |
| Node count and operational load | Adds Praefect nodes and a database for Praefect metadata, plus replicas to monitor and upgrade. | Fewer moving parts, but more shards to plan, balance and monitor as repositories grow. |
| Kubernetes support | Gitaly Cluster on Kubernetes has been labeled beta. Confirm current status before choosing it for a cluster. | Gitaly runs outside the cluster in the Cloud Native Hybrid pattern, on VMs or managed infrastructure. |
| Fit with HA stack | Reference architectures pair it with Patroni PostgreSQL, Redis Sentinel and Consul, typically from 3,000 users. | Works with the same stack, but HA for repositories depends on your storage and backup design. |
| When it fits | Choose it when repository-level redundancy and automatic failover matter and your team can run the extra components. | Choose it when simplicity matters most and recovery through backups and storage protection meets your objectives. |
| Questions to confirm | Current Kubernetes status, supported replica counts, and the known limitations listed for your GitLab version. | How repositories are placed across shards, how they rebalance, and how you restore a single failed shard. |
How Praefect routes and replicates repository writes
Praefect is the routing layer in front of a Gitaly Cluster. Clients send Git operations to Praefect rather than to a single Gitaly node, and Praefect decides which storage nodes hold each repository and which node serves each request. The important unit here is the repository, not the cluster or the individual server: each repository is assigned to a set of Gitaly nodes, and replication happens per repository. That means two repositories on the same cluster can have different primaries and different replica sets, and your capacity planning should account for that distribution rather than assuming every node holds a full copy of everything.
For writes, Praefect coordinates replication so that changes are applied to the replicas assigned to a repository, and it tracks which replicas are up to date. The reader should confirm in the current GitLab Gitaly Cluster documentation exactly how write acknowledgement works for their GitLab version, which replicas must confirm a write before the client sees success, and how the replication queue behaves when a replica is slow or unreachable. Do not assume that every replica is always consistent at every moment. Ask the vendor for the documented consistency guarantees, then test a write during a simulated node outage in a non-production environment and record what the clients see.
Failover is where in-flight operations need attention. When a primary becomes unavailable, Praefect promotes a healthy replica, but operations that were running at the moment of failure may fail, need to be retried, or may leave a repository in a state that requires reconciliation. Plan for client-side retries, check how long the failover takes in your test environment, and confirm how Praefect reports replicas that lag behind the primary. Because Gitaly Cluster does not provide snapshot backups, the backup plan for repository data must be designed separately, as covered in the backup section.
Deciding and standing up the Gitaly tier step by step
This sequence takes a team from sizing targets through a go or no-go decision on Gitaly Cluster to a tested Praefect deployment, so repository-level high availability is adopted on evidence rather than assumption. Each step ends with a check that tells you whether to continue.
Confirm user and API request targets
Start by matching your expected user count and peak API request rate to the published GitLab reference architectures, which range from 1,000 users at 20 requests per second to 50,000 users at 1,000 requests per second. Your platform or DevOps lead should measure current API traffic from existing load balancer or GitLab logs rather than estimate it from headcount alone. The step is complete when you have a documented user count, a peak request rate and the reference architecture row they map to. Confirm the figures with your GitLab account team or the reference architecture documentation before you commit to hardware.
Decide whether repository-level HA is required
The reference architecture documentation generally recommends a high availability strategy at 3,000 or more users, and that strategy uses Gitaly Cluster with Praefect. Below that scale, or where your tolerance for repository downtime is higher, sharded Gitaly may meet the need with fewer moving parts. Record the business case for HA, the recovery time your teams expect for Git operations, and the RTO and RPO your risk owner has accepted. The output is a signed decision that either requires Gitaly Cluster or selects sharded Gitaly, with the reasons written down.
Size Gitaly nodes and repository storage
Size the Gitaly tier from the chosen reference architecture and the repository footprint, including the largest monorepos and the expected CI clone load, since both drive disk I/O and memory. Confirm the exact node counts and specifications for your row in the reference architecture documentation rather than copying figures from another deployment. Check that each Gitaly node has storage with headroom for growth and that the network path between Gitaly, Rails and Praefect is low latency. The step closes when the sizing sheet is reviewed against the reference architecture and approved by the platform owner.
Deploy Praefect with its database
Deploy the Praefect service on its own nodes and provision the PostgreSQL database that Praefect uses to track replication state. Keep Praefect's database separate from the GitLab application database unless your design review explicitly accepts sharing it, and plan its availability as carefully as the Gitaly nodes it governs. Install the components with the method your team has standardized on, such as the Linux package, and confirm that Praefect can reach every Gitaly node. The check is a healthy Praefect status report that lists all configured Gitaly storages as reachable.
Configure replication factor and storage names
In gitlab.rb, define each Gitaly storage name, the Praefect virtual storage that Rails uses, and the replication factor for that virtual storage. Use storage names that are stable and documented, because changing them later complicates repository paths and backups. Apply the configuration with the standard reconfigure step on each affected node, and make sure the configuration is stored in version control so changes are reviewed. The check is that Rails reports the virtual storage as available and that a test repository created through it is written to the expected physical nodes.
Migrate repositories into the virtual storage
Move existing repositories from their current storage into the Praefect virtual storage in controlled batches, starting with a small set of non-critical projects. Schedule the batches around your maintenance window and confirm, before each batch, that a fresh backup exists and that the repository counts match the source. Track any repositories that fail to move, investigate them individually, and record the cause. The step is complete when repository counts, sizes and sampled clone and push results match the source environment.
Verify failover and backup restore
Test a Gitaly node failure on a non-production instance that mirrors your configuration. Stop one storage node, confirm that Git reads and writes continue through Praefect, then bring the node back and confirm it resynchronizes. Because Gitaly Cluster does not provide snapshot backups, also run a full backup and restore of the instance and verify the restored repositories match expected checksums or spot-checks. Document the observed behavior and any gaps against your recovery targets. The check is a written test report signed by the operations owner before production cutover.
Security and operations to plan for each Gitaly layout
Gitaly holds every repository, so whichever layout you choose, the network path, credentials, encryption and upgrade order around it carry real risk. The cards below cover what to plan before go-live and what to keep checking afterward, with Gitaly Cluster (Praefect) and sharded Gitaly noted where they differ.
Isolate Gitaly traffic from Rails
Gitaly nodes should accept connections only from the Rails, Sidekiq and Praefect hosts that need them, over a private network segment. Confirm which ports and addresses each Gitaly node listens on and that no Gitaly endpoint is reachable from user or CI networks. Gitaly Cluster adds Praefect as another listener, so its firewall rules need the same review.
Manage Praefect and Gitaly tokens
Gitaly authentication uses shared tokens that Rails, Praefect and each Gitaly node must agree on. Store them in your secrets manager or the Linux package secrets file with restricted file permissions, and record who can read them. Plan a rotation procedure before you need one, because a token change touches every node in the layout.
Encrypt repository storage at rest
Repository data on Gitaly disks is not encrypted by GitLab itself, so encryption depends on your storage layer. Ask your platform team whether volumes use full-disk or volume-level encryption and who holds the keys. Apply the same standard to replicas in Gitaly Cluster, since each node holds a full copy of every repository it serves.
Monitor replication health
In Gitaly Cluster, Praefect tracks which replicas are behind, so a failed or lagging node can leave repositories with unequal copies. Alert on unhealthy storage nodes and on replication queue backlog, and check that failover tests have been run. Sharded Gitaly has no replica to lag, so its monitoring focuses on per-node disk, latency and availability.
Order upgrades across Gitaly nodes
Upgrade Gitaly nodes in a sequence that keeps enough healthy storage to serve repositories, and keep the Gitaly version aligned with the GitLab version on the Rails side. Confirm the required upgrade stops for your target release before scheduling, since a Gitaly Cluster upgrade touches Praefect as well as each storage node. Rehearse the sequence on a non-production copy.
Apply hardening in stages
The GitLab hardening guidance is a work in progress and should be rolled out gradually, so pick a sequence that fits your change process. Start with application settings and operating system controls on Gitaly hosts, then review configuration files such as gitlab.rb. Treat Geo as a disaster recovery option to evaluate separately from the layout decision, not as a substitute for HA.
Why backups change when repositories are replicated
Replication is not a backup. Gitaly Cluster writes each repository change to multiple Gitaly nodes through Praefect, so a force push, an accidental deletion or a corrupted repository can be copied to every replica within moments. The replicas protect you from losing a node, but they do not give you a earlier version of the data to return to. Gitaly Cluster also does not support snapshot backups, so a disk-level snapshot of the Gitaly nodes cannot be your repository recovery path. Before you commit to Gitaly Cluster, confirm with the current backup documentation which repository data the GitLab backup task captures on your version and how it does so for Praefect-managed storage.
A recovery plan for this topology has three parts. First, the GitLab backup and restore task, run on a schedule, covers the database and the other application data it supports, and you should keep those archives off the host that produced them. Second, Gitaly data needs its own explicit check: confirm in the GitLab backup and restore documentation how repositories are included, how long a backup run takes on your repository volume, and what must be backed up separately, such as the secrets file and the gitlab.rb configuration. Third, schedule restore tests on a non-production instance, then clone sample repositories, confirm pipelines still run and record the elapsed time. A backup that has never been restored is an assumption, not a recovery plan.
Geo is a separate option for disaster recovery. According to the GitLab Geo documentation, it is available in the Premium and Ultimate tiers for self-managed GitLab and provides a warm-standby secondary site in another region that is continuously synchronized. Continuous synchronization means Geo protects against losing a site, but like replication, it will carry a bad change to the secondary as well. It also adds a second environment to run, network paths between sites and a tested failover procedure. Assess its fit as its own decision, using your recovery time and recovery point targets, your tier and your operating capacity, and keep the backup plan in place alongside it.
Questions buyers ask about Gitaly Cluster and sharded Gitaly
Is Gitaly Cluster required once a GitLab instance passes 3,000 users?
No. GitLab's reference architectures recommend a high availability strategy for 3,000 or more users, and the documentation generally pairs that strategy with Gitaly Cluster using Praefect. The reference architecture is a starting point, though, and your own availability target decides the layout. Repository-level replication keeps copies of each repository on several nodes, so a node failure does not take a repository offline. Sharded Gitaly spreads repositories across nodes without replicas, so a node failure affects the repositories stored on it. Compare both layouts against your recovery time and recovery point objectives, your repository growth, and your team's ability to operate the extra components. Confirm the current guidance on the reference architectures page before you commit.
What drives the cost and effort of each Gitaly layout?
The main cost is the number of nodes and the storage on each, and Gitaly Cluster needs more of both because every repository is stored on several replicas. Cost and effort rise with:
- Node count and size for the Gitaly tier, plus the Praefect layer and its coordination data
- Storage per replica and the growth rate of large repositories and monorepos
- Load testing and failover testing before go-live
- Upgrades, which must follow GitLab's required upgrade stops across every node
- Monitoring and on-call coverage for more moving parts
Sharded Gitaly usually needs fewer copies of data, but it still needs careful repository placement and capacity planning. One-time costs include design, migration and testing. Recurring costs include infrastructure, operations time and upgrade work. Ask any quote to show node counts, storage per replica, the test plan and who performs each upgrade.
How long do migration and testing take for our repository volume?
The duration depends mainly on total repository size, the number of repositories, the number of active users and how long you can pause writes during cutover. No general timeline applies to every instance, so ask for a scoped plan built from a repository inventory. That inventory should list each repository's size, commit history, branch count and recent write activity. A scoped plan should also cover integrity checks after the copy, load testing against the reference architecture targets, a rollback rehearsal and a restore test. The GitLab Performance Tool can validate the environment before go-live. Each of these steps adds time, and the largest repositories usually set the schedule. Treat any duration without that inventory as an unverified estimate.
Can our own team run Gitaly Cluster without an outside partner?
It can, if the team has clear ownership for every tier and tested runbooks for failure cases. Gitaly Cluster sits inside a larger high availability layout that also includes Rails and Sidekiq nodes, Patroni-managed PostgreSQL, Redis Sentinel and Consul. Each of those components needs an owner who understands its failover behavior, its upgrade order and its logs. Your team should be able to run the required upgrade stops, rehearse a node failure and restore from backup without guesswork. Before you decide, ask for named owners, on-call coverage, documented runbooks and a restore test you have run yourselves. DeployOpen can plan, implement, document and support self-managed GitLab in your infrastructure under a fixed scope, or provide ongoing support under a retainer.
Is Kubernetes a safe place to run Gitaly for our GitLab instance?
Treat Gitaly as a stateful service and check the current guidance before you place it in a cluster. GitLab's reference architecture documentation says that running stateful components such as PostgreSQL and Redis inside Kubernetes is not supported. Its Cloud Native Hybrid architecture runs stateless components, such as Webservice and Sidekiq, on Kubernetes and keeps stateful services on virtual machines or managed services. Gitaly stores repository data, so the same caution applies to it. GitLab's installation methods include a Helm chart and an Operator, but an install method does not decide whether a stateful component is supported in a cluster. Confirm the current status of Gitaly on Kubernetes in GitLab's documentation, and plan for a Cloud Native Hybrid or virtual machine layout unless that guidance changes.
How do backups differ between Gitaly Cluster and sharded Gitaly?
Replication protects against node loss, but it is not a backup, so both layouts need GitLab backup and restore procedures. Gitaly Cluster does not support snapshot backups, so you cannot rely on storage snapshots alone to recover it. Replicas also copy mistakes, such as an accidental deletion or a corrupted write, to every node. Plan for:
- A backup method that GitLab documents for your layout, tested by restoring to a separate environment
- Backups of the database, repository data, uploads and the secrets file, kept on separate storage
- Restore timing measured against your recovery objectives
- A check of the backup approach for sharded Gitaly before you choose that layout
For disaster recovery across regions, GitLab Geo provides a warm-standby secondary site that is continuously synchronized. Geo is available in the Premium tier and higher for self-managed GitLab, so confirm its fit and cost with your license terms.
Does Gitaly Cluster replace a disaster recovery site?
No. Gitaly Cluster keeps repository copies within your environment, so it helps with node failures in one location but does not protect against losing that location. A disaster recovery design needs a separate site with its own power, network and storage. GitLab Geo can maintain a warm-standby secondary site in another region, synchronized continuously, and it is documented for self-managed installations in the Premium tier and higher. Geo adds its own operational work, including failover testing, monitoring of replication lag and a clear decision on who promotes the secondary site. Ask your provider to show how a site failure would be detected, who approves failover and how the team would fail back. Confirm the details in the Geo documentation before you design around it.
What DeployOpen scopes and what your team keeps
A fixed-scope Gitaly engagement divides work between DeployOpen and your team, and the written scope fixes that split before any work starts. DeployOpen's part typically covers the Gitaly layout decision, the Praefect and storage node configuration, the migration of existing repositories onto the chosen layout, testing of failover and restore, and handover documentation that your operators can run without us. Each of those items should appear as a named deliverable in the statement of work, with acceptance criteria your team agrees to, such as which Praefect failover test cases were run and which restore procedure was rehearsed.
Your team keeps the parts that only your organization can own. That includes the servers, virtual machines or cloud accounts that host the Gitaly nodes, the network and storage that sit underneath them, and the approval of production changes under your change management process. Ongoing GitLab upgrades, including the required upgrade stops documented in the upgrade paths, remain with your team unless a separate support or upgrade retainer is agreed in writing. Confirm early who holds administrator credentials, who signs off on the maintenance window for the cutover, and who will receive the on-call alerts after handover.
Before you sign, ask the partner to put the boundary in writing, including what happens if the chosen layout changes during the build, who pays for extra storage or compute, and what is excluded from the timeline. Timelines and scope depend on your environment, repository volume and the availability of your staff, so they are set in the agreed statement of work rather than assumed. Ask for the handover package in the quote, and check that it includes the backup and restore runbook, the failover test results and the on-call escalation path.
How DeployOpen scopes and delivers Gitaly decisions
DeployOpen treats the Gitaly layout as a written design decision, not a default. We scope the repository availability requirement, the backup approach and the Kubernetes footprint first, then build and hand over only the topology your organization can run and recover.
How we scope
- Our written scope records your repository availability target, recovery expectations and the Gitaly layout you choose.
- We document which repository storage paths, Praefect nodes and Gitaly shards are in scope before any build starts.
- Engagements are fixed-scope, so deliverables and timelines are agreed in writing before work begins.
How we build
- We install the chosen Gitaly topology inside your own cloud accounts, data centers or air-gapped network.
- We configure replication, storage placement and failover behavior, then test them against the scope you approved.
- We align backup tooling with the layout, since Gitaly Cluster does not provide snapshot backups and needs a planned alternative.
How we hand over
- You receive runbooks for failover, repository recovery, node replacement and sharded storage moves.
- Architecture notes record why the chosen layout fits your requirements and which known limitations remain open.
- Your team gets documented verification steps to rerun after upgrades, runner changes or GitLab version stops.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer after handover.
- We can review Gitaly topology when user counts, CI load or monorepo size change, and when GitLab releases affect storage.
- Confirm with DeployOpen which support level covers Gitaly incidents, backups and upgrade stops before signing.
