How many GPUs does GitLab Duo Self-Hosted need?
GPU count follows from three inputs: the size of the model you host, the number of developers who use Duo at the same time, and the Duo features you turn on. GitLab publishes example sizing that maps these inputs to accelerator counts, and you should read the example that matches the GitLab version you deploy, because guidance changes between releases. Treat those examples as a starting baseline for budgeting. Before you request GPU servers, write down the model family and parameter size you intend to serve, the peak number of concurrent users per team and per site, and whether Code Suggestions, Chat and the Agent Platform will share the same hardware. Those decisions determine which row of the example you start from.
Confirm exact figures in GitLab's current documentation for your version, including the base system minimums for each component you run. Those minimums cover CPU, memory and storage for the GitLab instance and the self-hosted AI gateway, and they are listed separately from accelerator counts. Also confirm the accelerator model, memory per device and the inference configuration the documentation assumes. A different GPU generation or serving setup can change how many devices a given model needs. Ask your GitLab account team or implementation partner to confirm any figure that the public documentation does not state for your version, and get that answer in writing before you place a hardware order.
The final number comes from a load test in your own environment. Replay representative prompts at your planned peak concurrency, record response latency and queueing, and adjust the device count until the measured results meet the experience your developers need. Then add headroom for failover, maintenance windows and upgrades. If the sites will be disconnected, check GitLab's offline guidance for Duo Self-Hosted before you fix the hardware plan, because model artifacts, licensing and gateway updates must reach the environment by a route you control.
Which sizing choices change the GPU plan
This table compares the sizing choices that move the GPU plan for Duo Self-Hosted, with the fit and trade-off of each option. Read each row against your own feature list and concurrency target, then take the actual memory, accelerator counts and throughput figures from GitLab's published examples and vendor specifications. The main takeaway is that the model tier and endpoint layout usually matter more to hardware cost than the accelerator brand.
| Sizing choice | When it fits | Trade-off |
|---|---|---|
| Model size class | Smaller models suit lighter Code Suggestions use. Larger models suit Chat and agentic features that need more reasoning. | Larger models need more accelerator memory and more accelerators per replica, which raises hardware and power cost. Confirm feature support per model in GitLab's support matrix. |
| Single-node versus multi-node placement | Single-node fits a smaller developer group and a model that fits the memory of one server. Multi-node fits larger models or higher concurrency. | Multi-node adds network, orchestration and failure-domain work. Check how your serving engine splits a model across nodes before buying hardware. |
| A100 versus H200 accelerators | A100 can fit a planned budget and model set confirmed in GitLab's examples. H200 suits larger models or tighter consolidation, if its specifications meet your targets. | Accelerator choice changes per-unit memory, availability and lead time. Compare vendor specifications for memory and throughput rather than assuming one generation fits every workload. |
| One shared model endpoint versus separate endpoints per feature | A shared endpoint suits a single approved model used across features and keeps the footprint small. Separate endpoints suit features that need different models or isolation. | Separate endpoints duplicate memory use and add operations work. A shared endpoint concentrates load, so plan capacity for peak concurrent developers. |
| On-premises GPUs versus a private cloud model platform | On-premises GPUs suit strict data residency, offline operation or existing data center capacity. A private cloud platform suits teams that want managed capacity and accept that cloud region. | On-premises needs purchase, cooling, rack and staff time, mostly one-time with ongoing operation. Cloud shifts cost to usage and needs the residency terms reviewed with your compliance team. |
How to size the GPU plan from GitLab's examples
This sequence turns GitLab's published examples into a GPU plan you can defend: a confirmed version and tier, a model listed in the support matrix, a measured peak load, and tested headroom. Each step ends with a check that the platform and security teams can review before purchase.
Confirm the GitLab version and license tier
The platform owner records the GitLab version you will run, the tier you hold and any add-ons your Duo features require. Confirm the add-ons with GitLab's pricing and licensing material, and check the version against GitLab's maintenance policy so you target a release that still receives fixes. If an upgrade stop lies between your current version and the target, schedule it before ordering hardware. Done when the sizing record lists version, tier and add-ons, and the target version is one GitLab currently maintains.
Choose Duo features and the model family
The platform team selects which Duo features will run on self-hosted models, such as Code Suggestions, Chat or the Agent Platform, and proposes a model family for each. Check every candidate model against GitLab's support matrix for your exact GitLab version, and note the support rating listed for each feature. If a model is missing for your version, change the version or the model before continuing, since sizing depends on it. Done when each enabled feature maps to one model the matrix lists for that version.
Measure concurrent developers and peak hours
Engineering leads count the developers who will use each feature, then estimate concurrent use during peak working hours using your own CI, commit and working-hour data. Split the estimate by feature, because agentic sessions send far more requests per user than single completions. Record each site's time zone and any batch windows that overlap with peaks. Done when you have a table of peak concurrent users per feature per site, reviewed by a team lead who knows the actual working pattern.
Read GitLab's published hardware examples
Using GitLab's reference architecture documentation and its guidance for the same version, find the published example closest to your model size and concurrency. Record the GPU type, GPU count and base system minimums that the example lists. Confirm the example applies to your version and model, because guidance can change between releases. Done when you have a written note linking each planned model to one example, with the source version and page recorded for the review.
Derive the GPU count with peak and failover headroom
Starting from the matched example, calculate how many GPUs serve your peak concurrent load for each model deployment. Add capacity so the service still meets peak demand when one GPU or one host is unavailable, and add room for growth you have agreed with stakeholders. Write down the headroom rule and the reason for it. Done when a test shows that total capacity with one failed host still covers the peak figure from the previous step.
Size the base servers around the GPUs
The infrastructure team chooses host chassis, CPU, memory, local storage and networking so each server carries the planned GPU count and meets the base system minimums from the examples. Confirm power, cooling and rack space in each data center, and separate the GPU hosts from the GitLab instance, which needs its own sizing. Costs include one-time hardware purchase and recurring power, support and replacement parts. Done when a bill of materials is reviewed against the power and cooling survey and approved by the platform owner.
Run a load test before purchase
Build a test from realistic agentic sessions, mixed feature requests and the peak concurrency from the earlier step, then run it against a staging deployment or a loaner system of the same class. Measure response latency, queueing, errors and GPU memory at peak. If the environment is fully offline, run the test without outbound connectivity. Done when the test meets latency and error targets agreed with stakeholders at planned peak, and removing one host during the test still keeps the service within those targets.
Why base system minimums matter as much as the GPUs
A GPU server that meets the accelerator count for a model still fails in practice if the host around it is undersized. The base system includes CPU, memory, local storage, operating system and network requirements, and GitLab publishes these minimums alongside its GPU examples. Pull them from the published examples for the exact GitLab version you will run, not from a summary elsewhere, because the requirements change between releases. Ask your hardware vendor to confirm that the chosen server, drivers and operating system combination is supported for the accelerator and model format you plan to use before you order.
Sizing the GitLab instance is a separate exercise from sizing the model servers. Your GitLab users, API requests per second and CI load determine the reference architecture, which is documented in the reference architectures guidance and covers components such as Gitaly, PostgreSQL, Redis and Sidekiq. The AI gateway sits between GitLab and your models, so its placement, its network path to both sides and its version alignment with the GitLab instance add requirements of their own. Confirm the required version pairing in the GitLab documentation before you build, and plan for an upgrade of the gateway alongside each GitLab upgrade stop.
Treat the base system and the GitLab sizing as line items in the same plan as the GPUs. Record who owns each component, which network zones the gateway, model servers and GitLab instance need to reach, and how you will apply patches and version changes in an environment that may be offline. A version mismatch between GitLab and the gateway is one of the most avoidable failures in this stack, so write the pairing into your change calendar. Ask any implementation partner to quote the base system, GitLab sizing and gateway work as explicit scope items, so that the GPU budget does not hide the cost of the surrounding infrastructure.
Four inputs that move the GPU count up or down
GPU count for self-hosted Duo is not a single number taken from a table. It follows from four inputs that interact: the model you run, how many developers use it at the same time, which Duo features are switched on, and how much spare capacity you keep for failover and maintenance. Measure each one in your own environment before you ask for a quote.
Model size and precision
Larger parameter counts need more GPU memory to hold the weights, and the precision you deploy at (for example, 16-bit or 8-bit quantized weights) changes how much memory each parameter takes. Moving to a lower precision can reduce the number of GPUs needed, but only if GitLab's support matrix rates the feature as acceptable at that precision. Check the model size and precision GitLab lists for each Duo feature, then confirm the memory footprint with your serving engine before buying hardware.
Concurrent developers and peak hours
The number of developers who use Duo at the same moment drives throughput, not the total headcount with licenses. Pull login and request counts from your identity provider, IDE telemetry or GitLab logs for a typical working day, then identify the busiest hour. Size for that peak plus growth, since queued requests during a morning stand-up will be the first complaint you hear. Ask the vendor or your partner to state the assumed concurrency behind any quoted GPU count.
Enabled Duo features and request patterns
Code Suggestions sends many short requests as developers type, while Chat sends fewer, longer prompts that wait for a full answer. The Agent Platform adds multi-step loops in which one user action can trigger several model calls. Each feature you enable adds its own load profile on the same GPUs. Enable features in stages, measure request rate and token counts per feature, and size for the combined profile rather than for an average request.
Context length and failover headroom
Longer context windows reserve more memory per active request, so a deployment that allows large prompts holds fewer concurrent sessions per GPU. Headroom is the capacity you keep free so that a failed server, a driver upgrade or a GitLab minor release window does not stop Duo for everyone. Decide with your operations team how many GPU servers may be offline at once and what degraded service is acceptable, then add that spare capacity to the base count.
Planning with A100 or H200 accelerators
The A100 is a common choice for teams that already own GPU servers or can source them through existing data center channels, so it is often the first option to test against. Before sizing, confirm how GitLab's published examples name the accelerator, including the exact variant, because one family name can cover several memory configurations and form factors. Then read the per-device memory from the vendor's specification for that variant and work out how many devices your chosen model needs at your target concurrency.
- Exact variant: match the SKU or model string used in the examples to the line your hardware vendor actually sells.
- Memory per device: take the figure from NVIDIA's datasheet for that variant, not from secondhand summaries.
- Power and cooling per server: confirm the chassis power budget, PDU capacity and airflow for the planned number of cards per server.
- Lead time and support: ask for delivery windows and confirm the warranty and driver support cover the operating system and container runtime you run.
- Mixed fleets: confirm with GitLab's documentation and your model's support matrix whether one serving pool can span A100 devices with other accelerator types.
Ask your hardware vendor to put the variant, power draw and delivery window in writing before the sizing is final.
The H200 is typically considered when a model or a concurrency target pushes the per-device memory requirement or the number of servers beyond what an existing estate can host. Before sizing, confirm how GitLab's published examples name the accelerator, and check that the variant you buy matches the memory and form factor the examples assume. Verify the per-device memory from NVIDIA's specification for that variant, then calculate device and server counts from that figure rather than from memory of earlier generations.
- Exact variant: confirm the SKU and form factor, since server chassis support depends on it.
- Memory per device: use the datasheet figure for the specific variant you will order and record it in the sizing worksheet.
- Power and cooling per server: confirm that your facility can supply and cool the full chassis load at peak, including any planned growth.
- Lead time and allocation: ask the vendor whether supply is allocated, what the delivery window is and what happens to the order if the window moves.
- Mixed fleets: check whether the chosen model and serving engine accept H200 devices alongside other accelerator types in the same pool.
Do not treat a delivery estimate as a commitment until the vendor confirms it in the order terms.
Several questions apply to both families and often decide the project timeline more than the accelerator itself. Hardware cost is a one-time item, while power, cooling, support contracts, spares and staff time recur for the life of the system. Those recurring costs depend on the chosen family and server count, so the comparison should be made on the full ownership picture rather than the purchase line alone. Ask each vendor to quote the same configuration so the figures can be compared directly.
- Facility readiness: confirm rack space, power feeds, cooling capacity and network links between GPU servers and the GitLab instance.
- Sourcing and offline updates: if the deployment is air-gapped, plan how drivers, firmware and container images reach the network. GitLab documents offline guidance for Duo Self-Hosted in its offline environments documentation.
- Spares and replacement: agree on replacement times for failed devices and whether spare parts are held locally.
- Support boundaries: clarify whether the hardware vendor, the platform team or a partner handles driver, firmware and operating system issues.
- Quote contents: ask for the accelerator variant, memory per device, server count, power draw, delivery window, warranty term and support level in one written document.
Confirm all of these items with the hardware vendor and your facilities team before comparing totals.
The choice between the two families should follow from the sizing results, the facility and the sourcing position rather than from a general preference. Choose the A100 when existing inventory or lead times favor it and your chosen model fits within the planned device count at the target concurrency. Choose the H200 when the model or concurrency requires more memory per device or fewer servers, and the facility can supply the power and cooling at peak. Where neither clearly wins, the decision usually turns on delivery risk and on how many servers the facility can accommodate.
- Model requirements: start with the model you intend to serve, because its memory needs set the minimum device count for each family.
- Existing estate: favor hardware you already operate if it meets the sizing, since it avoids a new procurement cycle.
- Facility limits: choose the option that fits the power and cooling you can deliver without a facility upgrade.
- Concurrency growth: size for the developer count you expect in the next planning cycle, not only for launch day.
- Serving stack support: confirm that the serving engine and driver versions you plan to run support the family you select.
Re-run the sizing whenever the model, the concurrency target or the hardware order changes, and test the candidate hardware before committing the full order.
Security and operational risks of GPU servers for Duo
GPU servers for self-hosted Duo hold model weights, carry developer prompts and sit on the request path between GitLab and developers. The cards below cover the controls and checks that shape how many GPUs you size, where they run and how they are operated over time.
Keep weights and prompts inside the boundary
Model weights and prompt traffic should never leave the network segment you control, which is the main reason to choose on-premises GPUs over a hosted model platform. Confirm that the AI gateway, model servers and any logging sinks resolve only to internal addresses, and that no component falls back to an external endpoint. Ask GitLab which Duo features still need outbound connectivity in your chosen license mode, and review the offline guidance before finalizing the design.
Restrict access to the endpoint and gateway
The model endpoint and the self-hosted AI gateway are high-value targets because anyone who reaches them can run inference on your GPUs. Place them on a restricted subnet, allow only the GitLab instance and the gateway to call the model server, and authenticate every gateway request. Review who can administer the GPU hosts, since shell access there exposes weights and logs alike.
Patch drivers and serving software on schedule
GPU drivers, container runtimes and the model serving engine each need their own patch cadence, and each can break inference after an update. Align the serving stack with the GitLab version your instance runs, and test a driver or engine change in a staging GPU host before production. Record the approved combinations so that a security patch does not silently move the stack out of its tested state.
Decide what prompts and responses are logged
Prompt and response logs are useful for troubleshooting and audit, but they often contain source code, secrets pasted by mistake and personal data. Decide which fields the gateway and model servers record, who can read them, and how long they are kept, then align that with your data retention policy. Confirm the logging controls in GitLab's hardening guidance and in the serving software you choose.
Monitor utilization and queue depth
Sizing assumes a concurrency level, so you need evidence that the cluster stays near it. Track GPU memory use, GPU utilization, request latency and queue depth per model, and alert when queues grow for sustained periods. These signals show whether you need more capacity, a different model size or fewer concurrent users, and they are the first place to look when developers report slow responses.
Document hardware and model versions as evidence
A change record should show which GPU models, driver versions, serving engine release, model version and GitLab version were running at each point in time. This evidence helps during security reviews, incident analysis and compliance assessments, and it makes a later sizing review faster. Keep the inventory in the same change system you use for GitLab itself, and confirm supported version pairs with GitLab before each change.
How to test the GPU count before buying hardware
Confirm the GPU count with a load test before you place a hardware order. A sizing estimate built from published examples describes a workload profile, not your workload. Your developers may ask more Chat questions, run longer Code Suggestions sessions or trigger more agentic runs than the profile assumes. A test built from your own traffic shows whether the planned accelerators hold the concurrency you need, and it gives you numbers you can defend to finance and to the teams that depend on the service. Build the test around the model you intend to serve, the inference engine and the AI gateway version you plan to run, since those three together determine throughput. Confirm the exact versions and hardware configuration with GitLab's documentation and your serving vendor before you start, because a test on a different stack does not validate your purchase.
Replay representative traffic rather than synthetic bursts. Capture or script a mix of Chat questions and Code Suggestions requests in roughly the proportions you expect across your teams, with realistic prompt and response lengths. If you plan to enable the Duo Agent Platform, include multi-step agentic runs as a separate traffic class, because each run holds capacity for longer and fans out into several model calls. Ramp concurrency in steps toward your target, and measure time to first token, full response latency, request queueing at the gateway and GPU utilization at each step. Record where latency starts to rise sharply, since that knee, not average throughput, marks your usable capacity. Interpret the results against the latency your developers will accept for interactive Chat and for inline suggestions, which are usually held to different thresholds. Agree those thresholds with the engineering leaders who own the developer experience before the test, not after the numbers arrive.
Treat a failed test as a signal about the sizing inputs, not only about the hardware. If latency breaks early, check whether the concurrency assumption, the model size, the context length, the share of agentic traffic or the quantization choice was wrong before adding a GPU. Repeat the test after each GitLab version upgrade and after any change to the model or serving stack, because release changes can alter prompt handling and throughput. Keep the test scripts, datasets and results under version control so the next sizing exercise starts from evidence.
Limits to confirm before you commit to a GPU plan
GPU sizing examples are only useful when they match the GitLab version, feature set, license model and environment you actually deploy. The cards below list the points to confirm with GitLab, your vendor and your own infrastructure team before hardware is ordered.
Match the example to your version
GitLab's published sizing examples change between releases, so a plan written against one version can be wrong for the version you install. Before you size anything, record the GitLab version you will run, then ask GitLab for the Duo Self-Hosted model and hardware guidance for that exact release. Keep the reference with your change records so a later upgrade triggers a fresh check. Plan the upgrade stops in the same review, because the GPU plan may need to change with them.
Confirm Agent Platform support by version
The Duo Agent Platform is generally available on self-hosted deployments from GitLab 18.8. If you run an earlier version, agentic features are not covered by the same hardware reasoning and need a separate check. Ask GitLab which features, models and GPU profiles apply to your version and whether any features require a later release. Do not assume a sizing example for chat or code suggestions also covers agent workloads, which can make different demands on the model endpoint.
Keep licensing separate from hardware
Hardware sizing tells you what GPUs to buy, but it does not tell you which Duo add-on or license model you need. Seat-based Duo Enterprise, the Agent Platform Self-Hosted add-on for offline licenses and usage-based billing for online licenses each follow different commercial terms. Ask GitLab or your account contact to confirm which license applies to your deployment before the GPU order is placed, so that the capacity plan and the commercial plan are approved together.
Confirm offline component support
Sites with no internet connectivity need their own confirmation of which Duo components, models, licenses and scanners work without outside access. GitLab documents offline guidance for GitLab Duo Self-Hosted, but a sizing example does not prove that every feature runs fully offline. Ask GitLab to confirm in writing which components your version supports offline, how model weights and container images reach the site, and how licenses are activated without an online connection.
Check GPU supply and lead times
GitLab's documentation does not cover the availability, lead time or allocation of A100 or H200 accelerators, and these change with market conditions and your hardware vendor's allocation. Ask your server vendor for a written quote with the stated lead time, the exact accelerator model and the validated driver and CUDA stack. Confirm that the chassis, power and cooling in your facility can support the chosen configuration before the order is confirmed.
Treat examples as starting points
GitLab's published examples describe starting configurations, not guarantees for your workload. Concurrent developer counts, prompt length, repository context and agent activity all change how much GPU capacity you need. Before you commit, run a pilot with representative users and measure latency, queueing and error rates at peak load. Ask GitLab which measurements it expects from a pilot, and treat the result as the basis for the production order.
Questions buyers ask about GPU sizing for Duo
How much does a GPU plan for GitLab Duo Self-Hosted cost?
The cost depends on the number of GPU servers, the accelerator type and count, the hosting model (purchased hardware in your data center or rented capacity in your cloud account), and the engineering time needed to size, install and validate the setup. Hardware or cloud capacity is paid to your vendor or cloud provider, and sizing work is paid to whoever performs it. Hardware is largely one-time, while cloud capacity, power, support contracts and GitLab licensing recur. No figure is given here because none is supplied for this topic. Ask for a written scope that lists the models you plan to run, the concurrent developer count you expect, the accelerator options considered and the cost components each party pays for.
How long does GPU sizing and procurement take?
Sizing is usually shorter than procurement. The timeline is driven by how quickly you can confirm usage data, how many model candidates you need to compare, whether accelerators must be ordered with lead times from a vendor, and how long your change and security approvals take. Air-gapped or offline deployments add steps for moving software and model files into the environment. Confirm current accelerator lead times with your hardware supplier or cloud provider, and ask any partner for a fixed-scope timeline with named milestones before work starts.
What does our team need to provide for a sizing exercise?
A useful sizing exercise needs an honest picture of demand and a place to test. Typical inputs include the GitLab version you run or plan to run, the number of developers who will use Duo features and when they work, the features in scope (such as Code Suggestions or Chat), the models you are considering, and any requirements for latency, data residency or offline operation. You also need access to a test environment with GPU capacity or a way to provision it. Ask the partner to list these inputs in the written scope so gaps are visible early.
Can our internal team size GPUs for Duo without outside help?
Yes, if your team has people who understand GitLab's documented requirements, model serving, GPU memory and throughput, and network placement for the AI gateway. The work is mostly measurement and comparison: reading the published examples, testing candidate models on the hardware you intend to buy, and measuring behavior under concurrent load. Teams without recent GPU or inference experience often find the load testing and capacity arithmetic the slowest part. A partner can take on that work under a fixed scope, while your team keeps ownership of the decisions.
Does a private cloud model platform change the GPU requirement?
It can change where the GPU cost sits, and it may remove the need to buy accelerators, but it does not remove the need to size capacity. Managed model platforms bill for usage or reserved throughput instead of hardware, and their availability, quotas and data handling terms differ from on-premises GPUs. Choose the platform based on data residency, offline requirements and contractual terms, then size against that platform's limits. See the sibling topic on AWS Bedrock or Azure OpenAI for the trade-offs.
How do GitLab's published examples apply to our own workload?
GitLab's published examples are starting points, not a sizing guarantee for your organization. They show the kind of hardware and model combinations GitLab describes, but your real demand depends on how many developers use each feature at the same time, the prompt and response sizes your teams generate, the model you choose and your latency expectations. Treat the examples as the baseline for a load test, then measure throughput and response times with a representative workload. Confirm the exact example values against the current GitLab documentation for your version before you rely on them.
What does a fixed-scope GPU sizing engagement deliver?
A fixed-scope engagement should deliver a written sizing report that states the assumptions, the models and GitLab version assessed, the hardware options compared, the load test method and results, the recommended configuration and the risks that remain open. It should also list what the client must supply and what is out of scope, such as procurement, installation of the production cluster or model licensing. Deliverables and timelines are agreed in writing before work starts, and a quote follows from that written scope.
How DeployOpen scopes and delivers Duo GPU sizing
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure, including air-gapped networks. For GPU sizing for GitLab Duo Self-Hosted, we turn GitLab's published examples into a written sizing decision for your models, your concurrent developers and your environment.
How we scope
- We confirm which Duo features and self-hosted models your teams need before any hardware is named.
- We record your expected concurrent developers, peak usage windows and target model sizes in the written scope.
- We agree the GPU model, count, network and storage assumptions and list what the scope excludes, in writing before work starts.
How we build
- We check the chosen GPU servers against GitLab's documented base system minimums and the model support matrix for your GitLab version.
- We size the self-hosted AI gateway, model serving layer and GitLab instance together, so version alignment is planned rather than assumed.
- We test response times and throughput under a load profile that matches your agreed concurrent developer count before sign-off.
How we hand over
- You receive a sizing document that states each assumption, the GitLab version it targets and the conditions that would trigger a resize.
- You receive build and configuration records for the serving stack, including model placement, capacity settings and access controls.
- We walk your platform and security teams through monitoring signals, capacity thresholds and the steps to add GPUs later.
How we support
- Ongoing support, upgrades and capacity reviews are available as a retainer after handover, scoped in writing.
- We can re-run sizing when you move GitLab to a later version, change model families or grow your developer population.
- Any future change is quoted from a written scope, so the cost of each review follows from the work agreed in advance.
