When a 3,000-user GitLab needs a high availability layout
GitLab's documentation generally recommends a high availability strategy once an organization reaches 3,000 users or more, using Gitaly Cluster with Praefect for repository storage, PostgreSQL with Patroni for the database, Redis Sentinel for caching and queues, and Consul for service discovery and failover coordination. At this size, source control and CI stop being a convenience and become a dependency for releases, incident response and audit evidence, so a single server failure or a long maintenance window can block many teams at once. The high availability layout is designed to remove single points of failure across the Rails, Sidekiq, Gitaly, database and Redis tiers so that one node can fail or be taken out for patching without taking the whole instance down.
The single-node Linux package deployment and the smaller reference architectures rely on one machine or a few machines per role, and their recovery plan is mostly restoring from backup. That plan is usually acceptable for a smaller team, but it is measured in hours of downtime, and it does not give you a way to upgrade without a service interruption. Check your own numbers before deciding: confirm your user count, peak API requests per second and CI job concurrency against the reference architecture targets, and note how many engineers would stop work during a restore. When the answers point to many teams blocked by one outage, or when your change windows cannot absorb a restore, the high availability design is the next step to evaluate. Confirm with GitLab's reference architectures documentation which sizing applies to your load.
What each node role does in the 3,000-user layout
The 3,000-user reference architecture splits GitLab into separate roles so that each stateful service can fail, be patched and be scaled on its own. The cards below cover what each role does, what breaks when it is lost, and what must stay separate from it. Confirm exact node counts and sizes against the reference architecture page for your GitLab version before you buy hardware.
Rails and Webservice nodes
These nodes serve the web UI, the REST and GraphQL APIs, and Git over HTTPS. They are stateless with respect to repository data, so losing one reduces capacity and in-flight requests but does not lose code. Keep them off the database and Redis hosts so that application load does not compete with the stateful tiers. Check that the load balancer health checks match the endpoints GitLab exposes, or failed nodes will keep receiving traffic.
Sidekiq nodes
Sidekiq runs background jobs such as notifications, webhooks, repository housekeeping and CI-related processing. Losing a Sidekiq node delays jobs rather than corrupting data, but a backlog can grow quickly under heavy CI load. Run Sidekiq on dedicated nodes instead of co-locating it with Rails so that a memory-heavy job cannot starve web requests. Ask how queued jobs are monitored before go-live so that a stalled queue is noticed early.
Gitaly Cluster nodes with Praefect
Gitaly stores the Git repositories, and Praefect is the routing layer that sends each request to the correct Gitaly node and tracks replication. Repositories are replicated across Gitaly nodes, so a single node failure should not take repositories offline, but the replication and failover behavior has documented limits you must review. Keep these nodes dedicated to Gitaly and do not share their disks with the database or object storage. Plan separate backup procedures, because Gitaly Cluster does not support snapshot-based backups in the way many teams expect.
Patroni PostgreSQL nodes
Patroni manages a PostgreSQL primary and its replicas, promoting a replica when the primary fails. Losing the primary triggers a failover, but applications may see brief errors during the switch, so the connection path matters. Do not place these nodes on hosts that also run Redis, Consul or Gitaly, because disk and memory contention on the database is hard to diagnose. Confirm the failover timing you expect and test it before the instance serves real users.
Redis Sentinel nodes
Redis holds cache and session data and queues used by GitLab. Sentinel watches the Redis primary and promotes a replica when it becomes unavailable. Losing a Redis node can drop cached sessions and interrupt in-flight jobs, which users notice as forced sign-ins or retries. Keep Redis separate from the database and application hosts, and confirm the Redis version your GitLab release supports before you deploy.
Consul nodes
Consul provides service discovery and cluster membership so that GitLab components can find the current PostgreSQL primary and related services. Losing a single Consul member is tolerable, but losing quorum stops membership changes and can block failover. Run Consul with an odd number of members on separate hosts, and never co-locate all Consul members on one hypervisor. Include Consul health in your monitoring so that quorum loss is alerted before a database failover is needed.
How Gitaly Cluster and Patroni keep data available
Gitaly Cluster and Patroni each protect a different data tier, and the consistency promises you can make to your developers depend on how each one fails over. Praefect sits in front of the Gitaly storage nodes and routes repository access across the replicas in the cluster. When a Gitaly node is unavailable, Praefect directs requests to the healthy replicas, so repository reads and writes can continue while the failed node is repaired. Patroni manages PostgreSQL in a similar role: it watches the cluster members, uses a distributed configuration store (Consul in this layout) to hold leadership, and promotes a replica when the primary is lost. Before you go live, confirm with the current reference architecture documentation exactly which replication behavior, failover settings and recovery steps apply to your GitLab version, and do not assume a guarantee the docs do not state.
Consistency differs between the two tiers, so plan for each separately. Repository data is stored on several Gitaly nodes, and Praefect tracks which replica holds the current version of each repository. Ask your GitLab support or the reference architecture owner which replication mode your version uses, what happens to writes accepted during a node outage, and how a repaired node catches up. PostgreSQL holds the metadata that ties repositories, merge requests, pipelines and permissions together. A promoted replica can only be as current as the replication stream allowed at the moment of failure, so confirm the replication and synchronous commit settings your Patroni configuration uses and test a forced failover in a non-production environment before you accept the design.
The database tier does not work alone. Patroni and PostgreSQL depend on Redis Sentinel for cached and queued state, and the Rails and Sidekiq tiers depend on both. If PostgreSQL fails over while Redis is unhealthy, or Redis fails over while the database is still electing a leader, requests can fail in ways that look like Gitaly or application faults. Treat the three tiers as one failure domain for testing: document the order of recovery, the health checks each tier uses, and who on your team responds to each alert. Your runbook should state which failover events you have rehearsed and which you have only read about.
Full HA layout compared with the scaled-down variant
This table sets the standard 3,000-user HA layout against the scaled-down variant across node roles, what is combined, the operational effort each implies and the workloads each suits. Use it to decide how many separate hosts your team can operate, then confirm exact node counts and specs against the current reference architecture page before committing.
| Layout area | Standard 3,000-user HA layout | Scaled-down variant |
|---|---|---|
| Rails and Sidekiq | Separate node groups for Rails (web, API, Git over HTTP and SSH) and Sidekiq background jobs | Rails and Sidekiq may share hosts, so background job load competes with web and API traffic |
| Gitaly Cluster with Praefect | Dedicated Gitaly nodes behind Praefect for repository replication and failover | Confirm whether the variant keeps Gitaly Cluster or uses a different Gitaly layout, since this changes repository HA |
| PostgreSQL with Patroni | Dedicated PostgreSQL nodes managed by Patroni for leader election and failover | Database role is still HA-managed, but node placement and pooling choices should be confirmed against the reference architecture |
| Redis with Sentinel | Dedicated Redis nodes with Sentinel for failover | Confirm whether Redis roles are co-located with other services and how Sentinel quorum is kept |
| Consul | Dedicated service discovery nodes supporting the HA components | Confirm whether Consul runs on separate hosts or alongside another role |
| Operational effort | More hosts to patch, monitor, back up and upgrade in a fixed order | Fewer hosts to manage, but each shared host has more interacting services to test during upgrades |
| Upgrade behavior | Zero-downtime upgrades depend on each role being upgraded in sequence across its node group | Zero-downtime upgrades still need a tested rolling sequence, and co-located roles must be included in the plan |
| Workloads it fits | Large user bases, heavy API traffic and sustained CI load with separate scaling needs per role | Smaller user populations or lower API load where scaling roles separately is not yet needed |
Build sequence for the 3,000-user HA layout
This sequence takes a 3,000-user GitLab deployment from an agreed reference architecture to a load-tested high availability environment, building the stateful tiers first so the application nodes have something reliable to connect to.
Confirm the reference architecture and request target
Before any server is ordered, agree the reference architecture, the expected API request rate and the CI concurrency with the platform owner. The GitLab reference architectures start at 1,000 users (20 API requests per second) and reach 50,000 users (1,000 API requests per second), and the documentation generally recommends high availability from 3,000 users. Confirm whether your workload matches that profile or needs a larger one. The step is complete when the target and the node inventory are written down and signed off by the people who will run and pay for the platform.
Provision the PostgreSQL and Patroni tier
Build the database nodes first, because every other tier depends on them. Deploy PostgreSQL managed by Patroni across the nodes in the agreed layout, with a connection pooler in front if your design uses one, and confirm the database and extension versions against the reference architecture documentation for your GitLab release. The step is complete when the cluster elects a leader, a replica is streaming, and a forced leader switch completes with GitLab able to reconnect. Record the failover time you observe as your baseline.
Provision Redis Sentinel and Consul
Deploy the Redis nodes with Sentinel for failover, and the Consul servers that GitLab uses for service discovery in this layout. Keep the Redis eviction and persistence settings as the reference documentation describes for your version, and confirm that Redis Cluster and managed serverless offerings are not part of the design. The step is complete when Sentinel reports a primary and replicas, Consul reports a healthy quorum, and stopping the Redis primary triggers a failover that the application tier can follow.
Provision the Gitaly Cluster with Praefect
Build the Gitaly Cluster nodes and the Praefect routing layer, placing repository storage across the nodes so that a single node loss does not stop Git operations. Verify that every storage node is reachable from Praefect and that a test repository is replicated to the expected nodes. Also confirm the backup approach for repositories, because Gitaly Cluster does not support snapshot backups and needs its own backup procedure. The step is complete when a test push and clone succeed through Praefect and a storage node outage leaves repositories available.
Install and configure Rails and Sidekiq nodes
Install the application tier with the Linux package or your chosen install method, separating Rails web and API nodes from Sidekiq nodes as the reference layout requires. Point every node at the database, Redis, Consul and Gitaly Cluster endpoints, and use the same GitLab version and secrets on all of them. Configure external URL, load balancer health checks and Sidekiq queue settings in the configuration file. The step is complete when every node reports healthy, background jobs are processed, and a login and a repository operation succeed through the load balancer.
Connect object storage and backups
Connect object storage for artifacts, LFS, uploads and packages, and configure the backup job so it writes to storage outside the GitLab nodes. Back up the configuration and secrets files separately, since a database backup alone does not restore a working instance. The step is complete when a test backup is written to the target, a restore is performed on an isolated environment, and the restored instance can open a project, a pipeline log and an uploaded artifact. Keep the restore time you measure in the runbook.
Test failover and run a load test before go-live
Before production traffic arrives, test each failure you designed for: stop a database leader, a Redis primary, a Gitaly storage node and a Rails node in turn, and record whether users see errors and how long recovery takes. Then run a load test against the target request rate with the GitLab Performance Tool to validate the environment against the reference architecture numbers. The step is complete when each failover leaves the platform usable, the load results meet the agreed target, and the findings are recorded with any tuning changes before go-live.
What zero-downtime upgrades require of this layout
Zero-downtime upgrades in a multi-node GitLab layout depend on the node roles being separated and redundant. Rails and Sidekiq nodes can be taken out of service one at a time only if another node of the same role keeps serving requests, and Gitaly Cluster, Patroni PostgreSQL, Redis Sentinel and Consul each need enough members to keep quorum and leadership while one member restarts. A scaled-down variant that collapses roles onto fewer hosts removes that room to maneuver. Before committing to a design, ask the vendor documentation which node counts and rolling order its zero-downtime procedure assumes, and confirm that your environment can run it as written.
Required upgrade stops still apply. Since GitLab 17.5, upgrades must pass through the stops at x.2, x.5, x.8 and x.11, for example 18.2, 18.5, 18.8, 18.11, 19.2, 19.5, 19.8 and 19.11, so a jump from one stop to a later release must be planned as a sequence of supported hops. Each hop needs its own rehearsal in a staging copy of the HA environment, with a current backup taken first. The GitLab upgrade paths documentation is the authority for the exact sequence from your current version.
Plan upgrades on a regular cadence rather than waiting for a crisis. The GitLab maintenance policy backports bug fixes only to the current stable release, while security fixes also go to the previous two monthly releases. An instance that falls several monthly releases behind may therefore be left with a known fix it cannot receive without a larger jump. Keep GitLab Runner aligned with the GitLab version as well, because the documentation advises matching major and minor versions. Runner fleets are often upgraded separately, so give them their own step in each upgrade window and confirm the target Runner version before the GitLab hop begins.
Security and operational risks of the HA layout
A multi-node layout adds more hosts, more credentials and more network paths than a single server. The cards below cover where that extra surface sits and which controls, backups and checks keep the HA design from becoming a larger place to fail.
Larger attack surface and node secrets
Every Rails, Sidekiq, Gitaly, PostgreSQL, Redis and Consul node holds some part of the trust chain, and the Linux package shares the configuration file gitlab.rb and the secrets file across them. A copy of the secrets file that is out of date on one node causes authentication failures that look like network faults. Confirm how each node receives its secrets, who can read them, and how a rotation is applied to every node in the same change window.
TLS and network segmentation between tiers
Traffic moves between the load balancer, Rails, Gitaly, the database pooler, PostgreSQL, Redis and Consul, and each path needs its own rule. Open only the ports each role needs, restrict Gitaly and Redis to the application subnet, and decide where TLS terminates for each hop. Test each boundary with a failed connection, not only a successful one, before go-live.
Backups across the stateful tiers
The backup task covers repositories and database content, but a Gitaly Cluster does not produce snapshot backups in the same way, and secrets files sit outside the archive. Plan separate backups for the database, the repository data, object storage and the secrets, then run a restore into a test environment on a schedule. Ask your implementation partner which backup method they will use for each stateful node.
Geo for disaster recovery
A single HA cluster protects against a node failure, not against losing the data center. GitLab Geo provides a warm-standby secondary site in another region that is continuously synchronized, and it is available in the Premium and Ultimate tiers. Decide early whether a secondary site is in scope, because it changes the network, storage and failover runbook that the 3,000-user layout needs.
Gradual rollout of hardening
GitLab's hardening guidance covers application settings, CI/CD, configuration files, the operating system and NIST SP 800-53 alignment, and it describes itself as a work in progress to be rolled out gradually. Apply changes one tier at a time, record each setting in version control, and test the CI/CD effects in a non-production instance before the change reaches production. Ask for a written list of settings changed per release.
Monitoring each tier
A healthy Rails tier can hide a degraded Gitaly node, a lagging PostgreSQL replica or a Sentinel failover that has already happened. Watch each tier with its own signals, including Prometheus metrics, replication lag, Redis failover events, Consul membership and Sidekiq queue depth. Set alerts on trends, not only outages, and confirm that someone is paged for each tier before go-live.
Questions buyers ask about the 3,000-user HA build
What drives the effort and cost of a 3,000-user GitLab HA build?
The biggest drivers are the number of nodes you must provision and harden, the tier you buy, the skills your team already has, and the environment the build runs in. A layout with separate Rails, Sidekiq, Gitaly Cluster, Patroni PostgreSQL, Redis Sentinel and Consul nodes has many more moving parts than a single server, so each role adds build, test and runbook work. Tier matters because Premium adds features such as advanced CI/CD and priority support, while Ultimate adds security and compliance features, and each feature you enable needs configuration and validation. The environment matters too: cloud accounts, data center networks and air-gapped sites each change how images, secrets and updates reach the nodes. Ask any partner for a quote that lists each node role, each tier-specific feature, the environment assumptions and what is excluded. The GitLab pricing page shows the license tiers, but license cost is separate from implementation effort.
Can our internal team build a 3,000-user GitLab HA environment alone?
Often yes, if the team already runs Linux, PostgreSQL, Redis and Git infrastructure in production and has time to own the build as a project. The hard parts are rarely the first install. They are getting Patroni failover, Gitaly Cluster with Praefect, Redis Sentinel and Consul to behave correctly together, testing failure cases, writing backup and restore runbooks, and proving that upgrades can run without downtime. Check whether your team has done a failover drill for each stateful role and has someone who can read the GitLab reference architecture documentation closely. A useful self-assessment is to list who will patch each node, who answers a page at night, and who has restored a backup in the last year. Gaps in those answers are the reason many internal builds stall. A fixed-scope engagement can cover the build while your team takes over operations, and DeployOpen can outline that split in a written scope.
How long does a 3,000-user HA build take?
Duration depends on the scope you agree, not on a standard number, so DeployOpen does not quote one here. The main variables are how many environments you need (for example, staging and production), whether you use a cloud provider or your own data center, how much of the platform you configure beyond the base install, whether you integrate identity, runners and backups in the same phase, and how quickly your teams provide access and approvals. Ask for a fixed-scope plan with phases, a testing window and named client dependencies, since those show where delays usually happen. If your timeline is tied to an upgrade stop or an audit date, state that up front so the plan can be sequenced against it.
What do we need to provide for the build to succeed?
The build needs a clear set of inputs before work starts. You should be ready to provide:
- Cloud accounts, virtual machine or hypervisor capacity, and network ranges with agreed DNS names and load balancer placement
- Named owners for identity, storage, backups, secrets and change approval
- Access for the build team under your security policy, with a plan for hardening the operating systems and GitLab settings
- Decisions on object storage, certificate authority, outbound network rules and whether the site is offline
- A test plan that includes failover and restore drills, and a reviewer for the final acceptance checks
Missing any of these typically moves work into later phases. Confirm each item with your own infrastructure and security teams before the statement of work is signed.
Can a managed cloud database replace Patroni for GitLab HA?
It can be a valid choice, but it changes the design and must be checked against GitLab's documented requirements for your version. Patroni manages PostgreSQL failover on nodes you operate, so you control the replication topology, the failover behavior and the upgrade sequence. A managed database moves much of that work to the provider, but you must confirm that the service is supported for your GitLab version, that the required extensions and settings are available, how failover is tested, how connections are pooled, and how backups and point-in-time recovery work across regions or sites. Ask the provider and your GitLab reference to confirm these points in writing. Tell your partner early if you plan to use a managed service, since the rest of the HA layout, including Redis Sentinel and Consul, is still needed unless the documentation supports a different approach for your case.
How should GitLab upgrades be scheduled on a 3,000-user HA layout?
Plan upgrades around the required upgrade stops and a documented zero-downtime procedure, then test each stop in a non-production copy before production. GitLab ships a minor release monthly and patch releases twice a month, and it requires upgrades to pass through stops at versions x.2, x.5, x.8 and x.11 since GitLab 17.5. Examples from the upgrade paths documentation include 18.2, 18.5, 18.8, 18.11 and 19.2, 19.5, 19.8 and 19.11. Security fixes are backported to more recent releases than bug fixes, so a practical schedule keeps you on a supported line and avoids skipping stops. Each upgrade should include a fresh backup, a rollout order for Rails, Sidekiq, Gitaly Cluster and the database tier, and a rollback decision point. Check the current maintenance policy and upgrade paths before setting dates.
Does the 3,000-user HA design support air-gapped or regulated networks?
GitLab documents running self-managed instances in offline, air-gapped environments, so the HA layout can be built without outbound internet access if the install, updates and any scanners are staged in advance. The design work includes how packages, container images and advisories reach the network, how certificates and secrets are handled, and how backups leave the site. Scanner and AI features may need separate offline guidance, so confirm those features against your tier before committing to them. Your compliance team should decide which controls the evidence must support. DeployOpen can prepare systems and evidence for an assessment, but independent auditors and certification bodies issue the actual audit reports and certifications.
How DeployOpen scopes and delivers GitLab HA
DeployOpen plans, implements, hardens, documents and supports self-managed GitLab inside your own cloud accounts, data centers or air-gapped networks. For a 3,000-user high availability layout, the work starts with a written scope that fixes node roles, component versions and the upgrade path before any build begins.
How we scope
- Confirm user count, API request rate, monorepo size and CI concurrency before choosing the HA layout.
- Agree in writing which node roles are in scope: Rails, Sidekiq, Gitaly Cluster, Patroni, Redis Sentinel and Consul.
- Record the upgrade stops and maintenance windows the design must support, including zero-downtime requirements.
- Engagements are fixed-scope, so deliverables and timelines are agreed in writing before work starts.
How we build
- Provision the node topology in your environment, with load balancing, network rules and separate roles for each component.
- Configure Gitaly Cluster with Praefect, Patroni PostgreSQL, Redis Sentinel and Consul against the agreed design.
- Validate failover for each stateful component and record the results before the environment is accepted.
- Check runner placement, backup coverage and secrets handling as part of the build, not after go-live.
How we hand over
- Deliver an architecture document covering node roles, versions, dependencies, network paths and recovery order.
- Provide runbooks for routine upgrades, node replacement, failover testing and restoring from backup.
- Walk your platform and database teams through operations so they can run the stack without DeployOpen present.
- Confirm with your team which checks, metrics and alerts move into your existing monitoring stack.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers after handover.
- Retainers can cover required upgrade stops, patch releases and re-running failover tests after infrastructure changes.
- Ask any support provider to confirm response expectations and maintenance windows in writing before signing.
- Quotes follow from the written scope, so ask for each component, recurring service and one-time task to be priced separately.
