GitLab Duo Self-Hosted: Setup and Planning Guide

When GitLab Duo Self-Hosted fits your organization

Choose GitLab Duo Self-Hosted when your team runs self-managed GitLab and wants AI assistance for code and workflow tasks without sending source code or prompts to a public model service. The feature routes Duo requests from your instance to language models you host or procure yourself, through a self-hosted AI gateway that you deploy inside your own infrastructure. Duo is sold as an add-on, so the decision has two parts: whether your organization wants the AI capability on its self-managed instance, and where the models and gateway will run. Confirm with GitLab which Duo add-on subscription applies to your self-managed tier, which Duo features are available with self-hosted models, and whether any feature depends on a specific GitLab version. Those answers determine whether the feature covers the workflows your developers actually need.

Offline operation is the requirement to test first. If your instances sit in networks with no route to the internet, such as air-gapped government or defense enclaves, or in regulated environments that prohibit external AI traffic, you need the full chain to work without outbound calls: the GitLab instance, the AI gateway, the model runtime and any scanners you plan to use. GitLab documents offline environments and separate guidance for running GitLab Duo Self-Hosted offline, so start there and ask GitLab to confirm the offline setup for your version and feature set. If your policy only prohibits sending data to third parties, a connected design may be acceptable, but that is a policy decision to settle with your security and legal teams before the architecture is chosen.

Before you commit, run a short fit check. List the Duo features your teams would use, confirm each one is supported with self-hosted models, identify the model hosting option you will use (for example, a self-managed inference server such as vLLM or a cloud model platform you control), and assign an owner for model capacity, gateway upgrades and access policy. Plan these alongside your GitLab upgrade stops, because the AI gateway and the instance must stay compatible across each upgrade. If the fit check shows gaps in features, hardware or staffing, the right answer may be to delay the rollout rather than to change the scope of the security review.

Model hosting options and when each fits

This table compares self-hosted vLLM, a cloud model platform in your own cloud account and fully offline operation across the factors that usually decide the choice: where data sits, how the network is isolated, who runs the service and where accelerator capacity comes from. Read each row as a question to answer for your organization before you pick a path, since the rows interact.

Decision factor	Self-hosted vLLM	Cloud model platform in your account	Fully offline operation
Data location	Prompts and code stay on hardware you control, in your data center or private cloud	Inference runs in your cloud account, but the model provider's service boundary and data terms apply	Prompts, code and model weights never leave the isolated network
Network isolation	Serving nodes can sit on a private network behind a reverse proxy that allowlists user-facing endpoints	Depends on private endpoints and network rules you configure in the cloud account; confirm them with the provider	Full air-gap, with model files and GitLab components transferred in by approved media
Operational burden	Your team or a partner patches the serving stack, monitors it and rotates keys and certificates	Provider manages the serving platform; your team manages identity, networking and access policy	Highest: every update, model artifact and security fix is carried across the air gap by hand
Accelerator capacity	You procure and size GPUs or other accelerators; vLLM supports NVIDIA CUDA, AMD ROCm and Intel XPU	Capacity is drawn from the platform's offerings, subject to regional availability and quota	You procure, install and maintain all accelerator capacity on site
Who runs the service	Your platform team, with DeployOpen implementation and support if contracted	Cloud provider and your cloud team, with the model platform's own operations	Your team alone, with no external connectivity for vendor support or updates
Licensing and cost drivers	vLLM is Apache-2.0 licensed with no paid edition; cost comes from accelerators, infrastructure, implementation and operations	Platform consumption charges and the cloud account; confirm the pricing model in writing before you commit	Same drivers as self-hosted vLLM, plus media handling, offline update procedures and longer change cycles
Multi-node scaling	Tensor parallelism within a node and pipeline parallelism across nodes; multi-node traffic must sit on an isolated network	Scaling is handled by the platform; confirm model sizes and limits it supports	Same constraints as self-hosted vLLM, with every node and dependency sourced offline
Best fit	Teams that need control over data location and serving stack and can staff accelerator operations	Teams that want managed serving inside their cloud boundary and accept the provider's service model	Environments where no outbound connection is permitted, such as some public sector and critical infrastructure networks

Setting up the self-hosted AI gateway and model connection

This sequence connects your self-managed GitLab instance to an AI gateway and a model server that you run, then enables Duo features in stages. Each step closes with a check, so a failed check stops the work before the next dependency is built. The result is a tested path from GitLab to models you control.

Confirm the Duo add-on license and tier
Ask your GitLab account contact to confirm in writing that your subscription includes the GitLab Duo add-on for self-managed use, and record the seat count and the features it covers. Confirm that your base tier supports the features you plan to enable, since feature sets differ by tier on the pricing page. Ask whether a license file must be applied to the instance before the AI features appear. Done when you hold the written entitlement and seat count, which you will need in step six.
Check the GitLab version and upgrade path
Record the GitLab version your instance runs and compare it with the minimum version your Duo release requires, confirming that requirement in the GitLab documentation for that release. If the instance is behind, plan the move through required upgrade stops, which since GitLab 17.5 fall at versions x.2, x.5, x.8 and x.11. Take and test a backup before each stop. Done when the target version and every required stop are written into the change record.
Deploy the self-hosted AI gateway
Install the AI gateway in your network using the method your GitLab release documents, and place it where both GitLab and the model server can reach it on an isolated segment. Decide how it authenticates to GitLab and whether it sits behind your existing TLS and access controls. For air-gapped sites, use the separate offline guidance for GitLab Duo Self-Hosted, which is not covered by the standard install steps. Done when the gateway starts, responds to a basic health check from an approved host, and its logs meet your retention rules.
Deploy and validate the model server
Deploy the model server, such as vLLM, on accelerators sized for the models and request volume you plan to serve, and confirm the chosen model loads on that hardware. Start the OpenAI-compatible server and send a chat completion and an embeddings request from the gateway network. Keep the server off general networks, because vLLM's multi-node communication is insecure by default. If you use an API key, remember it does not cover every endpoint, so place a reverse proxy in front that allows only the endpoints users need. Done when direct test requests return sensible output and the metrics endpoint reports request counts.
Connect the gateway to the model server
Point the gateway at the model server's internal address and set the model identifier exactly as the server reports it. Allow only the endpoints the gateway calls, and apply the reverse-proxy allowlist from the previous step. Map each Duo feature to the model and endpoint it will call, and confirm that mapping before enabling anything. Done when a request sent through the gateway returns output matching the direct test, and the gateway logs show the expected model name.
Enable Duo features for groups or the instance
Assign add-on seats to a small pilot group first. Then turn on the self-hosted model configuration at the instance level and enable individual features for groups as your rollout requires, following the GitLab documentation for your release. Feature availability varies by feature and release, so check each one before you widen access. Record who holds administrator rights to change these settings. Done when a pilot user sees the expected Duo features in a test project and the instance settings match the change record.
Verify with test requests and monitoring
Run a fixed set of test prompts for each enabled feature, covering normal use, an oversized input, and a request from a user without a seat, and record the results. Confirm that GitLab, the gateway and the model server each log the request and that model server metrics are being collected. Assign an owner to the alerts and set a review date for the test set. Done when the test set passes, the results are filed as evidence, and alerting has a named owner.

Choosing and sizing the vLLM serving layer

vLLM is the component that answers GitLab Duo's model requests, so its sizing decides how many developers you can serve. vLLM exposes OpenAI-compatible endpoints, including /v1/completions, /v1/chat/completions, /v1/responses and /v1/embeddings, started with vllm serve. Because the interface follows the OpenAI API, your self-hosted AI gateway talks to it the same way it would talk to a hosted model service. vLLM runs on NVIDIA CUDA, AMD ROCm and Intel XPU GPUs, and on several CPU architectures, with further accelerators available through third-party plugins. Before committing to hardware, confirm that your chosen accelerator, driver stack and operating system are supported by the release you intend to run, and that your candidate model loads and serves at the context lengths your developers will use.

Capacity planning depends on how the model is split across accelerators. vLLM uses tensor parallelism to spread a model across GPUs within one node and pipeline parallelism to span nodes, and it can run data parallel deployments for mixture-of-experts models. A model that fits on one node should usually stay on one node, because cross-node communication adds latency and complexity. If you need multiple nodes, the vLLM security documentation states that traffic between nodes is insecure by default, with no authorization or encryption, so those nodes must sit on an isolated network segment. Treat that network as part of the AI service's security boundary, and record it in your architecture review. Ask your team to measure throughput and KV cache use with the Prometheus metrics vLLM exposes at /metrics before deciding how many replicas you need.

Pin the vLLM version and plan a test cycle before you deploy. vLLM ships new minor releases about every two weeks, so an unpinned install can change behavior between one deployment and the next. Choose a version, test it against your model, your GitLab Duo feature set and your load profile, and promote upgrades through a staging environment on the same cadence you apply GitLab patches. Also protect the endpoints: the --api-key option does not cover /metrics, /health or /invocations, so place a reverse proxy in front of vLLM that allows only the paths GitLab needs. Confirm the exact endpoint list and the current API key behavior with the vLLM security documentation for your release, since these details change across versions.

Securing model endpoints, prompts and metrics

A self-hosted GitLab Duo deployment is only as contained as the model server behind it. The vLLM endpoints, the metrics port and the traffic between nodes each need a deliberate boundary, and the secrets that connect GitLab to the gateway need their own handling. Treat each card below as a check to complete before users reach the service.

API key covers only some routes

The vLLM --api-key option protects endpoints under /v1, /v2, /inference and /cohere, but not /invocations, /pause, /resume, /abort_requests, /metrics or /health. A key alone is not a perimeter. Place a reverse proxy in front of vLLM and allowlist only the routes users and the AI gateway need, then test every other path from outside the network to confirm it is refused.

Allowlist user-facing routes

Define the exact set of paths the self-hosted AI gateway calls, such as the chat and completion endpoints, and deny the rest at the proxy. Control and administrative paths such as pausing or aborting requests should be reachable only from the operations network. Record the allowlist as configuration in version control so reviews can compare it against the running proxy.

Metrics expose operational data

The Prometheus endpoint at /metrics reports request counts, token throughput, latency histograms and KV cache use. That data reveals load patterns and capacity limits to anyone who can read it. Bind it to a monitoring network or require authentication at the proxy, and confirm that your Prometheus scraper is the only client that reaches it.

Isolate multi-node traffic

The vLLM security docs state that communication between nodes in a multi-node deployment is insecure by default, with no authorization or encryption. Place all nodes on an isolated network segment with no route from user subnets, and confirm that Ray or the multiprocessing backend ports are not reachable from outside that segment.

Protect the GitLab to gateway secret

The credential that GitLab uses to call the self-hosted AI gateway, and the gateway's key to the model server, should live in a managed secrets store rather than in plain configuration files. Plan rotation and confirm with your GitLab administrators where the value is stored, who can read it and how a compromised value would be replaced without a full outage.

Log access to model endpoints

Record every request to the gateway and the model server with the time, calling identity, route and response status, while keeping prompt content handling a documented decision. Forward these logs to your central log platform so that unexpected routes or spikes in use can be investigated. Confirm retention with your compliance owner before enabling detailed logging.

What changes when the environment is fully offline

A connected deployment can pull model weights, container images and packages from public sources during installation and routine maintenance. The network path is the least constrained of the three models, so planning matters more than network design. Even so, pin every version and keep a private copy of what you deployed, so an upgrade or rollback does not depend on a remote source being available.

Model weights: Download weights into storage you control and record the exact model revision and checksum used in each environment.
Container images: Pull GitLab and vLLM images by version tag, then keep a copy in a private registry for rollbacks.
vLLM releases: vLLM ships minor releases about every two weeks, so set an internal rule for which version you test and when you move to it.
GitLab upgrades: Follow the GitLab maintenance policy and required upgrade stops, which apply the same way whether or not the network is connected.
Pinned configuration: Keep the Duo and AI gateway settings in version control so a rebuilt host matches the one you tested.

Confirm in writing with GitLab which Duo components, if any, still call external services under your configuration.

Restricted egress allows outbound traffic only to named destinations, usually through a proxy or allowlist. Duo Self-Hosted can still need access to package repositories, registries and update endpoints during install and upgrade. The work is to build a complete destination list, then decide which destinations are mirrored internally and which are opened through the proxy.

Destination inventory: List every host each component contacts during installation, first start and upgrade, and test those flows against the egress rules you plan to enforce.
Model transfer: Load weights through an approved transfer path so serving hosts do not need access to public model hubs.
Internal registries: Mirror container images and operating system packages into an internal registry that the GitLab and vLLM hosts can reach.
vLLM exposure: Keep the vLLM server on an isolated network and place a reverse proxy in front of it, because the --api-key option does not cover every route. See the vLLM security documentation.
Proxy rules: Record each permitted destination with its owner, purpose and review date so auditors and operators can trace it.

Ask GitLab in writing for the destinations each Duo component calls, and confirm which of them your proxy must allow.

Operating the stack after go-live

Once GitLab Duo Self-Hosted serves users, the GitLab instance, the self-hosted AI gateway and the model server each change on their own schedule. The cards below cover the recurring work that keeps those three components compatible, observable and recoverable, and the checks to agree on before go-live.

Upgrading through required stops

GitLab requires upgrades to pass through required upgrade stops, which since GitLab 17.5 fall at versions x.2, x.5, x.8 and x.11. Plan each GitLab upgrade as a path through those stops, not a jump to the latest patch. Confirm the gateway release you run matches the GitLab version in use before you schedule the change window, and rehearse the path on a copy of the instance first.

Matching gateway and vLLM versions

The self-hosted AI gateway, the model server and the GitLab release must stay compatible as each one moves. Record the exact gateway image, vLLM version and GitLab version together in one change record. Because vLLM ships minor releases about every two weeks, do not upgrade it on its own schedule. Validate each pairing in a staging environment against the model you serve before promoting it.

Monitoring vLLM through Prometheus

vLLM exposes Prometheus-format metrics at /metrics, covering request counts, token throughput, latency histograms and KV cache use. Scrape these into the same monitoring stack you use for GitLab, and alert on sustained KV cache pressure and rising latency before users notice queueing. Histogram buckets are configurable, so set them to match the latency targets your teams have agreed.

Capacity reviews as usage grows

Usage of AI features tends to grow after launch, and accelerator capacity is the largest recurring cost in this design. Hold a scheduled review that compares request volume, token throughput and latency against the sizing you approved. Ask your team or partner to state, in writing, the threshold at which you add GPUs, add a serving replica or change the model. Include the cost of that capacity in the review.

Backing up gateway configuration

Back up the gateway configuration, including the settings that point GitLab at the gateway, model endpoint definitions, and the secrets used to reach the model server. Store these in your secrets manager, not in plain files on the gateway host. Keep the backup separate from the GitLab backup task, and test a restore into a clean host so that you know the configuration can be rebuilt without a manual reconstruction.

Rolling back failed releases

Define a rollback plan before any change to a model, gateway or GitLab release. Keep the previous model artifacts, gateway image and configuration available, and write the trigger for rollback in advance, such as failed validation prompts, error rates or latency targets missed after cutover. Note that the GitLab backup is a restore point for GitLab data, not for the model server, so the rollback must cover each component separately.

Questions buyers ask before hiring help

What drives the cost of running GitLab Duo on self-hosted models?

The main drivers are accelerator capacity, the number of users and concurrent requests, whether the environment must run offline, the integration work around identity and networking, and the people who operate the stack. Each of these is paid for differently. GitLab charges for the add-on licence, and the vLLM serving engine has no paid edition, so the model-serving cost comes from GPU or other accelerator capacity, the surrounding infrastructure, implementation labour and ongoing operations. Ask any partner for a quote that lists each of these components separately, with the assumptions behind the user count and peak request load stated in writing.

Which costs are one-time and which recur?

One-time costs usually include scoping, the initial build of the AI gateway and model serving, network and identity integration, and acceptance testing. Recurring costs include GitLab subscriptions billed per user, accelerator capacity whether owned or rented, power and hosting, monitoring, patching, and the time spent updating models and GitLab versions. Model updates recur because newer model releases and serving engine releases arrive on their own schedules. vLLM ships new minor releases about every two weeks, so a quote should state who tracks those releases, who tests them, and how often a change window is planned.

How long does a self-hosted Duo implementation take?

Duration depends on scope more than on the GitLab installation itself. The work that usually sets the timeline is securing accelerator capacity, getting network and firewall approvals, connecting identity, and validating that the chosen model behaves acceptably for the team's tasks. An offline deployment adds packaging and transfer steps for software and models. Ask for a timeline built from a written scope with named milestones and dependencies, and ask which items the client must complete before the partner's work can start, since those often decide the schedule.

What must our team provide before implementation starts?

Expect to provide GPU or accelerator access, either hardware in your data center or cloud capacity in an account you control, along with network approvals for the gateway, model servers and any outbound paths. You will also need GitLab administrator access to the instance, and access to your identity provider so users can sign in and groups can map to roles. Confirm early who approves production changes and who holds the credentials for the model servers, because the vLLM security guidance says node-to-node traffic is insecure by default and must sit on an isolated network.

Can our internal team deliver this without a partner?

A capable platform team can build this, provided it has experience with GitLab administration, GPU-backed inference and network segmentation, and it can allocate time for testing and operations after go-live. The harder parts are usually the security boundary around model endpoints, capacity planning, and the ongoing upkeep of GitLab, GitLab Runner and the serving stack together. Compare your own estimate against a written partner scope, and be explicit about who will own upgrades, on-call response and model changes once the initial work is finished.

Does offline operation change the scope of the work?

Yes. GitLab documents running self-managed instances in offline, air-gapped environments and gives separate guidance for running GitLab Duo Self-Hosted offline. The offline path needs a planned way to transfer GitLab packages, model artifacts and container images into the isolated network, and a way to apply updates through the same route. It also limits the external services a design can depend on, so identity, logging and secrets must be hosted inside the boundary. Ask whether the quote covers the transfer process for each future update, not only the first install.

Which GitLab tier do we need for Duo self-hosted?

GitLab Duo is sold as an add-on, so confirm the licence tier and add-on terms with GitLab before scoping the work. GitLab Free is listed at 0 US dollars, Premium at 29 US dollars per user per month billed annually, and Ultimate is custom priced, and all three are available for self-managed installations. Pricing and add-on terms change, so check the current pricing page and get written confirmation from GitLab before budgeting. Ask your partner to confirm which features your chosen tier includes in the design.

What questions should we ask about the model and endpoint security?

Ask how the model server is exposed, which endpoints are reachable by users, and how access is limited. The vLLM security documentation notes that its API key option does not protect every endpoint, including metrics and health paths, and recommends a reverse proxy that allows only the endpoints meant for users. Ask for the proxy rules, the network placement of each node, how keys are rotated, and how metrics are scraped without exposing them. A good quote names these controls as deliverables with acceptance tests.

How DeployOpen scopes and delivers GitLab Duo Self-Hosted

DeployOpen plans, implements, integrates, hardens, documents and supports self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks. For GitLab Duo Self-Hosted, we start by confirming which GitLab tier, Duo add-on and model platform your organization can actually use, then build and test the stack in your environment.

How we scope

Written scope confirms your GitLab edition, Duo add-on licensing and the instance version it will run against
Scope names the model platform, whether vLLM or a cloud model service, and where it is hosted
Scope states whether the deployment is connected or fully offline, with the network rules that follow from that choice
Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts

How we build

We install the self-hosted AI gateway and connect it to your GitLab instance and chosen model endpoints
We place model-serving endpoints behind a reverse proxy that exposes only the paths users need
We test Duo features against your models, covering access control, latency and failure behavior before handover
For offline builds, we follow GitLab's offline guidance and verify every dependency is available internally

How we hand over

You receive architecture diagrams, configuration records and the access model for the AI gateway and model servers
Runbooks cover model changes, GitLab upgrade alignment, gateway restarts and rollback steps
Test results show which Duo features were verified, with known limits and items for your team to confirm
Knowledge transfer sessions walk your platform and security teams through daily operation

How we support

Ongoing support, upgrades and continuous testing are available as retainers after handover
Retainers can cover GitLab version upgrades, including required upgrade stops, and matching gateway and model server changes
Periodic reviews check endpoint exposure, API key handling and model server patching
Quotes follow from the written scope, so ask for the support tasks and response expectations to be listed explicitly

GitLab Duo Self-Hosted