Why load test GitLab with the GitLab Performance Tool before go-live
The GitLab Performance Tool is a load testing toolkit built on k6 that drives API traffic against a self-managed GitLab instance and reports how it performs against the reference architecture it was sized for. Teams need it at two points: before users arrive, to confirm the environment meets the request rate its design assumes, and after changes that alter the load path, such as new node sizes, a database upgrade or a different Gitaly layout. Sizing on paper is a claim. A load run is the first evidence that the claim holds for your hardware, network and storage, and it is far cheaper to find a shortfall in a test window than in the first week of production.
The reference architectures set the targets. GitLab's documented range runs from 1,000 users at 20 API requests per second up to 50,000 users at 1,000 API requests per second, and each size has its own node layout. Before the run, confirm which row matches your user count and peak API load, then write down the target and the topology it applies to. The reference architectures documentation gives the exact figures for each size. Results only mean something when the target is fixed in advance, because a team that chooses the bar after seeing the numbers has tested nothing. Agree the pass criteria with the people who own the service levels, and record the tool and GitLab versions with them so a later run can be compared fairly.
Repeat the test after any change to the load path, including GitLab version upgrades, node resizing, database or Redis changes and Gitaly topology changes. Treat each run as a change record: keep the tool version, the GitLab version, the topology and the results together. That way a slowdown after an upgrade appears as a measurable difference between two runs and is caught before users report it. A load test does not replace production monitoring, but it gives the change board evidence that the environment still meets its target before the change is approved.
Choosing where to run the test: staging, production-like or production
A production-like staging copy is the usual choice when the test must not touch users. It should mirror the reference architecture you sized for, with the same node roles, Gitaly, PostgreSQL, Redis and Sidekiq layout, so that the requests per second you measure can be compared with the published targets for your user count. Its main weakness is data. A copy with empty or thin repositories will show latency that production does not have, especially for Gitaly and the database, so the copy needs representative data before the results mean anything.
- Data volume: Confirm how much repository, issue and pipeline data the copy holds and how it was seeded before trusting any latency figure.
- User impact: Production traffic is unaffected as long as the copy has its own network paths, credentials and runners that cannot reach live systems.
- Approval: The platform owner approves the environment, and security and the data owner approve any use of production data in the copy.
- Drift: Check that GitLab versions, runner settings and object storage configuration match production, because a mismatch changes the result.
- Cost: A standing staging tier recurs as infrastructure spend, while building and seeding the copy is usually one-time work.
Before the run, confirm that the topology and versions match production and record every known difference in the test report.
A pre-go-live environment is built for a launch that has not happened yet. No users depend on it, so you can run harder scenarios and rebuild components when results show a bottleneck. The trade-off is data. The instance may hold synthetic data or a partial migration, so the test only reflects load on the data you actually loaded. Settle the migration plan first, because repository and project counts drive Gitaly and database behavior more than user counts alone.
- Data volume: Record whether the data is synthetic, a trial migration or empty, and repeat the run after the final import.
- User impact: None on current users, but confirm that SSO, email, runners and webhooks are not wired to live systems during the test.
- Approval: The go-live sponsor, the operations lead and the security owner should agree on the scenarios, the stop conditions and the test window.
- Backups: Take and verify a backup before any large import or configuration change, following the backup and restore documentation.
- Rework: Fixes that change topology, such as adding a node or changing a Gitaly layout, require a fresh run, which is effort that recurs with each change.
Finish by confirming that the final configuration, data load and topology are the ones that go live, and rerun the test if any of them change afterward.
Testing a live instance after a change gives the most accurate data, because the repositories, users and pipelines are already present. It also carries the most risk. Synthetic load runs against people who are working, so the test is only justified when a change needs confirmation under real conditions and a staging copy cannot represent it. Limit the scope to the affected component where you can, and define the conditions for stopping before the run starts.
- Data volume: The data is already in place, so no seeding is needed, but confirm that load scripts create only test projects and users.
- User impact: Load can slow or fail live work, so set the request rate ceiling and the error and latency thresholds in advance.
- Approval: The service owner, the change advisory process and the on-call lead approve the window, and users are told about it.
- Restore point: Verify a recent backup before the test, since a change that goes wrong may need it.
- Scope: Start with a low request rate and raise it only while the thresholds hold, and keep a named person able to stop the run.
Before starting, confirm the rollback path, the abort thresholds and who can end the test, and write those into the change record.
Test options and when each fits
This table compares the main ways to scope a GitLab Performance Tool run, from choosing a reference architecture target to pairing baseline and post-change runs. Read each row for what the run proves and what it leaves open, then confirm the scenario list and run options in the tool's current documentation before you plan the test.
| Test option | When it fits | What it proves | What it cannot prove |
|---|---|---|---|
| Reference architecture target (users and RPS) | Before go-live, to test against the size closest to your user count and peak API load | Whether the environment meets the published RPS target for that size, such as 20 RPS at 1,000 users | That your real traffic matches the test mix. Confirm your peak request rate from logs first |
| Baseline run before a change | Before an upgrade, hardware change or topology change, while the current setup is still stable | Current performance under the same test profile, giving later runs a reference point | Anything about the changed setup. A post-change run is still required |
| Post-change regression run | After an upgrade, scaling step or configuration change, compared against the stored baseline | Whether the change moved response times or throughput for the scenarios that were run | Load patterns outside the chosen scenarios, or the root cause of any regression |
| API-only scope | When the goal is API throughput against an RPS target and a fast comparison is needed | API capacity under the selected API scenarios | Git over SSH and HTTP traffic, CI job load, web UI paths or background Sidekiq throughput |
| Broader scenario scope (web, Git and API) | When the change touches Gitaly, repository access or UI paths, or before a go-live sign-off | Behavior across several request types in the same run | Your exact workload mix. It also takes longer and needs more environment capacity. Confirm scenario options with the tool docs |
| Single run versus repeated runs | Repeated runs when results look close to a target or vary between attempts | Whether results are consistent across runs under the same settings | Stability over days of normal use, or behavior during failures and maintenance windows |
Setup sequence for a GitLab Performance Tool run
This sequence takes a self-managed GitLab environment from a documented target to a full load run with captured results. Each step closes with a check, so setup problems surface before the full run and the results can be trusted.
Confirm the reference architecture and target RPS
The platform lead and the performance owner match the test environment to the reference architecture it was built to, using the reference architectures documentation. They then write down the user count and the requests per second the run should reach, along with the GitLab version under test. A test plan that cannot name the architecture, the target and the version is not ready. Check: a one-page test plan signed off by both owners states the architecture size, the RPS target, the GitLab version and the node layout.
Prepare an isolated environment with agreed data
The platform team builds or restores a non-production instance whose topology mirrors the planned production layout. They seed the projects, groups, users and repositories that match the agreed data profile, and they confirm that no production integrations, webhooks or other traffic can reach it during the test. Check: the topology review matches the test plan, the seeded data matches the agreed profile, and health checks pass on every node before any load is applied.
Deploy the tool and k6 runner outside GitLab nodes
The team installs the GitLab Performance Tool and its k6-based runner on a separate host that has a network path to the GitLab endpoint over the same route users take. The host should not share nodes with Rails, Sidekiq, Gitaly, PostgreSQL or Redis, so the load generator does not consume the resources being measured. Check: the runner reaches the instance through the intended path, its clock is synchronized with the GitLab nodes, and it shows enough spare CPU and memory to generate the target load without saturating.
Configure credentials for a dedicated test account
The team creates a dedicated test user with only the permissions the scenarios need, and avoids administrator rights unless a scenario requires them. The access token is stored in your secrets manager and supplied to the runner at execution time, so it does not appear in shell history, job logs or the test repository. Confirm the exact configuration keys and token scopes in the tool's own documentation before you proceed. Check: the account signs in, reaches every test project, and a search of the test configuration and logs finds no plain-text token.
Run a short smoke test
The operator runs a short, low-intensity pass across the configured scenarios to confirm that requests succeed, responses are well formed and results are written to the output location. Any failure is fixed and the smoke test is repeated before the team moves on, because a broken scenario produces misleading full-run numbers. Check: the smoke test completes without the runner failing, the error output is reviewed and explained, and GitLab metrics and logs show data arriving for the smoke test window.
Confirm metric capture before the full run
The monitoring owner confirms that Prometheus scrapes and node logs cover every component in the test topology, including Rails, Sidekiq, Gitaly, PostgreSQL and Redis. Clocks on all hosts must agree so that the test window can be matched to the metrics window. Check: dashboards for each component show data for the smoke test window, and the team records the exact start and end times that the full run will use.
Execute the full run at target RPS
The operator starts the full scenario set at the target load and records the run window. During the run, the team watches the runner as well as the GitLab nodes, because a saturated load generator would make the target look unmet when the environment is not the cause. Check: the run completes, the runner shows no sign of being the bottleneck, and the recorded window matches the metric captures taken on each component.
Archive the run record for comparison
The team saves the test plan, the GitLab version, the environment and node layout, the tool configuration with secrets removed, the raw results and the exported metrics for the run window in one location. A complete record is what allows a later run after an upgrade or hardware change to be compared fairly. Check: someone outside the team can open the archive and reproduce the run conditions from the record alone.
Reading results against RPS targets without fooling yourself
Hitting the target request rate is not a pass if errors and latency climb with it. When you read a run from the GitLab Performance Tool, look at throughput, error rate and latency together for the same window. A run that reaches the target requests per second while error responses rise, or while response times drift upward through the run, has found a limit in the environment, not proven capacity. Record the ramp profile and the point where errors first appear, because that point is more useful to the sizing discussion than the peak number. The reference architecture targets are stated in API requests per second, so compare like with like: confirm which request mix the tool generated and whether the environment under test matches the architecture size you intended to validate.
The load generator can be the bottleneck. The tool runs on k6, and a generator that runs out of CPU, memory, network bandwidth or open connections will report lower throughput and higher latency even when GitLab is healthy. Before trusting a result, check the generator host's own utilization during the run, and confirm it has capacity to spare at the target rate. Run the generator from a network position close to the instance, as it would be used in practice, and note where it sits. If the generator saturates first, the result says nothing about GitLab, and a rerun on a larger or distributed generator is the honest next step.
Correlate every run with server-side metrics. A request rate result only explains itself when it is lined up against what each stateful and worker component was doing. Watch Gitaly for request latency and saturation on the repository nodes, PostgreSQL for connection use, lock waits and replication lag, Redis for memory and latency, and Sidekiq for queue growth and job durations, using the Prometheus metrics and dashboards from your monitoring stack. Keep the run timestamps and the metric windows aligned. Confirm with your team which metrics and alert thresholds they will treat as the stop signal, since this page does not supply pass thresholds. A result that cannot be traced to one saturated component should be rerun with narrower changes, not accepted.
Security and operational risks of running load tests
A load test sends real traffic at a real system, so the same environment that validates capacity can expose data, credentials and alerting to harm. The cards below cover the risks we see most often when teams run the GitLab Performance Tool, each with the control that keeps the test useful without creating a new incident.
Running against production data
Load tests create, clone and delete projects, issues and merge requests at volume. Pointed at a production instance or a restored copy with live customer repositories, the run can mutate records that the business relies on. Use a dedicated environment seeded with synthetic projects and users, and confirm with the data owner in writing which environment the test targets before the first run.
Exposing test credentials
The tool authenticates as test users and often needs personal access tokens or an OAuth client to drive API and Git traffic. Tokens left in shell history, CI variables visible to all projects or a shared load generator image can outlive the test. Store them in a secrets manager, scope them to the test group, and rotate or revoke them when the run ends.
Leaving test projects or users behind
Each run can leave behind groups, projects, runners, users and tokens that carry no owner and no review. These accumulate across repeated runs and can later be mistaken for real workloads or invite access. Name every test object with a run identifier, record what the run created, and include a teardown step that removes it and confirms the cleanup in the results.
Load that triggers rate limits or alerts
Sustained request rates can trip GitLab rate limits, WAF rules or cloud provider throttles, and they will page on-call engineers if the monitoring stack is live. A test that fails for these reasons tells you about your controls, not your capacity. Agree the target rate with the operations team, allowlist the load generator where policy allows, and run during a window when alert routing is paused or redirected.
Load generators reaching internet endpoints
In air-gapped or restricted networks, a load generator with default settings may try to pull images, packages or test assets from public endpoints. That breaks network policy and can leak test metadata outward. Build the generator image and its dependencies in advance, block egress except to the instance under test, and check GitLab's offline environment guidance before the run.
Retaining test logs with sensitive content
Load runs produce request logs, GitLab production logs, k6 output and sometimes captured responses that contain repository names, usernames, tokens or fragments of code. Keeping these in a shared bucket with default retention turns a test into a data store. Set a retention period for the run artifacts, restrict access to the performance team, and scrub or exclude response bodies before the results are archived.
Questions teams ask before running GitLab load tests
What drives the cost of a GitLab load test engagement?
Cost follows the amount of work in four areas: building the test environment, designing the test scenarios, analyzing results, and remediating what the tests find. Environment build is often the largest one-time component when no staging copy of the production layout exists, because the environment must match the reference architecture being validated. Test design rises with the number of user journeys, API endpoints and CI patterns you want exercised. Analysis scales with how many runs you need to compare and how deep the bottleneck investigation goes. Remediation is separate and optional: fixing Gitaly, PostgreSQL, Redis or Sidekiq findings is a different scope from measuring them. Ask any partner to price each area separately so you can see which parts recur when you retest after changes.
How long does one GitLab load test cycle take?
Duration depends on scope, not on a fixed template. A single run against a known reference architecture is shorter than a cycle that includes environment setup, several test profiles, a rerun after tuning, and a written findings report. The biggest variables are how much of the environment must be built or reconfigured, how many scenarios cover your real workflows, and how long each test window needs to run to produce stable numbers. Ask for a timeline that separates environment readiness, test execution, analysis and any retest, and confirm which of those dates depend on your team returning access or feedback.
What does our team need to provide for a load test?
A load test needs four things from your side. First, a staging environment that matches the target reference architecture closely enough for results to mean something, including the same node roles and sizing. Second, test accounts and seeded data such as projects, groups, repositories of realistic size and CI configuration. Third, monitoring access to the GitLab metrics endpoints, host metrics and database statistics, so bottlenecks can be seen while the test runs. Fourth, a named owner on your side who can approve changes to the environment, schedule test windows and make decisions about findings. Confirm the staging access model and data handling rules in writing before work starts.
Can an internal team run the GitLab Performance Tool alone?
Yes, many internal platform teams can run the tool without a partner, provided they have engineers who understand GitLab's architecture, Linux performance tuning and Kubernetes or VM operations, depending on the install method. The tool is open tooling built on k6, so the harder skill is not the test runner but the interpretation: knowing whether a result reflects a sizing problem, a misconfigured Gitaly, a saturated PostgreSQL connection pool or a test-harness limit. Teams that run tests only once a year often lose the context between runs. Consider whether you have someone who can own the test plan across upgrades, not just the first run.
What should we ask a partner for in a fixed-scope load test quote?
A fixed-scope quote should state the target reference architecture and user or request-per-second level being validated, the environment you will provide and what the partner will build, the list of test profiles and workflows, the number of test runs included, and the format of the findings report. It should also name what is out of scope, such as fixing findings, capacity changes to infrastructure, or retesting after a GitLab upgrade. Ask how the partner will handle data in the staging environment, who on each side approves changes, and which deliverables are final. DeployOpen agrees scope, deliverables and timelines in writing before work starts, so each of these points should appear in that document.
Does a load test prove our GitLab instance is secure or compliant?
No. A load test measures performance against reference architecture targets, such as requests per second at a given user load. It does not test security controls, validate configuration hardening or produce an audit result. Those are separate work streams with their own evidence requirements. Performance findings can still matter to compliance programs, since availability and change management records often reference capacity testing, so you may want to keep the test report alongside your change records. Independent auditors and certification bodies decide what counts as sufficient evidence for your program, not the testing team.
When should we repeat a GitLab load test?
Repeat a test before go-live and after changes that could move the performance profile. Common triggers include a change to the reference architecture or node sizes, a major or minor GitLab upgrade that moves through a required upgrade stop, changes to the database or Redis tier, a significant growth in users or CI concurrency, and changes to Gitaly storage or repository layout. Confirm with your platform team how each trigger maps to a retest, since a retest after an upgrade is often a smaller scope than a first baseline. Keep the previous report so the new run can be compared against the same profile.
How DeployOpen scopes and delivers GitLab performance testing
DeployOpen plans, runs and documents GitLab load tests inside the infrastructure you control, so the results reflect your own network, storage and identity setup. Each engagement starts from a written scope tied to the reference architecture you are validating against, and ends with a report your platform and operations teams can act on.
How we scope
- Confirm the reference architecture and user or request-rate target the environment is meant to meet before testing starts.
- Agree in writing which GitLab workloads, endpoints and test scenarios the GitLab Performance Tool will exercise.
- Define the pass criteria, the test window and which changes must be frozen during each run.
- Engagements are fixed-scope, so deliverables and timelines are agreed in writing before any work begins.
How we build
- Prepare the GitLab Performance Tool runner host and its k6-based configuration in your own network.
- Set up the test data, seed users and projects so load profiles match your expected usage patterns.
- Instrument the environment with the monitoring you already run so server-side metrics line up with test results.
- Execute baseline and repeat runs, then record each run's configuration and environment state for comparison.
How we hand over
- Deliver a written report comparing measured throughput and latency against the RPS targets in the agreed scope.
- Document the test configuration, scripts and data so your team can rerun the same profile after changes.
- Flag bottlenecks by component, such as application servers, Gitaly, PostgreSQL or Redis, with the evidence behind each finding.
- Walk your engineers through the results and list open questions for your architecture or capacity owners.
How we support
- Offer retesting before go-live and after upgrades, scaling changes or major configuration updates, scoped as new work.
- Provide ongoing support and continuous testing as a retainer if your platform changes often.
- Confirm with your team which environments, tools and approval steps must be available before each test window.
