How vLLM serves GitLab Duo models in your infrastructure
An organization that must keep Duo model inference on its own GPUs needs a serving layer that sits between the model weights and the self-hosted AI gateway. vLLM, an open-source inference engine licensed under Apache-2.0, fills that role. Started with vllm serve, it exposes an OpenAI-compatible server that the self-hosted AI gateway can call as if it were a hosted model API. The work divides into four parts: the GPU host and its drivers, the model endpoint itself, the connection from the AI gateway to that endpoint, and the day-two operation that keeps all three running alongside your GitLab instance. Each part has an owner, and the handoffs between them are where most unplanned work appears.
The first decisions sit with the platform team: which accelerator family you run (vLLM supports NVIDIA CUDA, AMD ROCm and Intel XPU GPUs among others), how many GPUs one model needs, and whether the endpoint runs on bare metal, VMs or Kubernetes. Which models and model versions Duo supports on a self-hosted endpoint, and which vLLM and AI gateway versions GitLab expects alongside them, must be confirmed in GitLab's own documentation before any hardware is ordered, because those requirements change between GitLab releases. Plan for a staging endpoint that the AI gateway can reach, so that model and version changes are tested before they reach developers.
Operating the endpoint is the part teams underestimate. vLLM's own security documentation states that node-to-node traffic in a multi-node deployment is unauthenticated and unencrypted by default, so multi-GPU hosts must sit on an isolated network. Its --api-key option does not cover every route, so place a reverse proxy in front that allowlists only the paths the gateway uses and exposes nothing else. Scrape the Prometheus metrics at /metrics for request load, latency and KV cache pressure, and set alerts before developers report slow completions. Ask your GitLab and vLLM contacts for a written compatibility matrix and an upgrade plan, since vLLM ships a new minor release about every two weeks.
What you need before vLLM can serve Duo
A vLLM endpoint only helps Duo once several components agree on versions, network paths and trust. The cards below cover the hardware, the GitLab and gateway versions, the model choice, the network boundaries, the certificates and the offline path to settle before installation starts.
GPU or accelerator capacity
Confirm the accelerator platform first: NVIDIA CUDA, AMD ROCm or Intel XPU are the GPU families vLLM supports for serving. Check driver, CUDA or ROCm versions against the vLLM installation guide for the release you plan to pin, since the serving engine ships new minor releases about every two weeks. Confirm that a single node holds the chosen model with the parallelism you need. Sizing the GPU count is covered in the GPU sizing topic.
Matching GitLab and AI gateway versions
The self-managed GitLab instance and the self-hosted AI gateway must run at matching versions, so plan both upgrades together rather than treating the gateway as a separate project. Check the GitLab upgrade path and maintenance policy before choosing a target release, because upgrades must stop at required versions. Record the version pair in your change record so a later GitLab upgrade does not break model routing.
Supported model family
Pick the model from GitLab's support matrix before you size hardware or write routing rules. The matrix rates each model per feature, so a model that serves Code Suggestions may not cover Chat or the Agent Platform equally. Confirm that the model weights and tokenizer you plan to download are the ones named for your GitLab version. Choosing the model is covered in the sibling model topic.
Network isolation between components
Place the vLLM nodes, the AI gateway and the GitLab instance on separate, deliberately routed network segments. The vLLM security guidance states that communication between nodes in a multi-node deployment is insecure by default, so multi-node traffic must sit on an isolated network. Allow only the AI gateway to reach the inference endpoint, and document each permitted flow for your firewall reviewers.
TLS certificates and internal DNS
Issue certificates from your internal certificate authority for the inference endpoint and the AI gateway, and give each a stable internal DNS name. Confirm that the gateway trusts your CA chain, because a name mismatch or an untrusted certificate often appears only after the rest of the stack is working. Plan renewal before go-live, and record who owns each certificate and where its private key is stored.
Offline package path for air-gapped sites
If the environment has no internet access, plan how model weights, container images, vLLM packages and GitLab offline artifacts reach the network before installation. GitLab documents offline environments and separate guidance for running Duo Self-Hosted offline, so read that guidance first. Confirm with GitLab which offline licensing and add-on terms apply, and build a transfer process that your security team can verify and log.
How to configure the vLLM endpoint and connect it to the gateway
This sequence takes a GPU host from bare metal to a Duo model served by vLLM, reachable only through a filtered endpoint and wired to the self-hosted AI gateway. Each step closes with a check you can repeat after driver, vLLM or gateway upgrades.
Prepare the GPU host and drivers
The platform or infrastructure team installs the operating system, the vendor GPU driver and the matching CUDA or ROCm runtime on the host, then confirms every accelerator is visible to the system. vLLM runs on NVIDIA CUDA, AMD ROCm and Intel XPU GPUs, so confirm your accelerator family appears in its installation documentation before you allocate hardware. Place the host on a network segment that only the gateway and the proxy can reach. Done when the driver's device query lists all GPUs with the expected memory and the kernel log shows no GPU errors.
Install vLLM and confirm the model loads
The engineer creates a dedicated Python environment or container image, installs vLLM at a pinned version and downloads the Duo model weights you selected from GitLab's support matrix. vLLM is licensed under Apache-2.0 and ships new minor releases about every two weeks, so record the pinned version and retest after each change. Run a short load first and watch memory allocation in the logs. Done when the model finishes loading and the logs report the server is ready on its configured port.
Start vllm serve with tensor parallelism
Launch the server with vllm serve and set tensor parallelism to the number of GPUs in one node, for example a tensor parallel size of 2 on a two-GPU host (illustrative). Bind the server to a private address and set --api-key as a baseline control, understanding that the vLLM security docs warn against relying on it alone. Multi-node pipeline parallelism needs an isolated network because inter-node traffic is unauthenticated and unencrypted by default. Done when the server log shows the model loaded and a request to /v1/chat/completions from the host returns a completion.
Place a reverse proxy with an endpoint allowlist
Deploy a reverse proxy in front of vLLM that forwards only the paths users and the gateway need, such as /v1/chat/completions and /v1/completions, and refuses everything else. The vLLM docs state that --api-key does not cover /metrics, /health, /pause, /resume, /abort_requests or /invocations, so block those paths at the proxy and allow metrics scraping only from the monitoring network. Terminate TLS at the proxy and restrict source addresses to the gateway. Done when a request to /metrics from the gateway network is refused and an allowed path still returns a valid response.
Point the AI gateway at the endpoint
The team configures the self-hosted AI gateway with the proxy's internal URL, the model name as vLLM reports it and the API key, then confirms the gateway and GitLab instance run compatible versions. Check the gateway's configuration reference for current setting names, since they can change between releases. Keep the key in your secrets manager rather than in configuration files stored in source control. Done when the gateway's health check or status view shows the model endpoint as reachable. Confirm the exact health check path in the gateway documentation.
Run a test request end to end
Send a short prompt from a GitLab test account with Duo enabled, using a Code Suggestions or Chat request on a sample project, then trace it through the proxy log, the gateway log and the vLLM log. Also send a direct chat completion request through the proxy with a test key to confirm the path works without GitLab in the loop. Capture request latency and token counts from the vLLM metrics endpoint as your baseline. Done when the test user receives a response and every hop logs the same request, which becomes the reference for later checks.
How the AI gateway calls vLLM and what GitLab sees
The request path runs from a developer's IDE or the GitLab instance to the self-hosted AI gateway, then to your vLLM server, and back again. The developer's editor or the GitLab instance sends a Duo request to the AI gateway, which holds the connection details for the model. The gateway forwards the request in the OpenAI-compatible format that vllm serve accepts on /v1/chat/completions or /v1/completions, and the response returns along the same path. GitLab sees the gateway as the only model endpoint, so the vLLM server stays behind the gateway and off the users' network path.
Three settings decide how this path behaves: authentication, model naming and timeouts. On the vLLM side, the --api-key option protects the /v1 routes, but the vLLM security documentation warns against relying on it alone and recommends a reverse proxy that allowlists only the endpoints users need. On the gateway side, the key that authorizes calls to vLLM and the model identifier the gateway sends must match what vLLM was started with. Timeouts matter because long generations on a busy GPU can outlast a default proxy or gateway limit, so set them at every hop and test them under load. Confirm the exact configuration keys, the supported model identifiers and the gateway's version alignment with GitLab in the GitLab Duo Self-Hosted documentation for the version you run, because these details change between GitLab releases.
Before go-live, ask your team to verify three things: which endpoints the reverse proxy exposes, which model name each developer-facing feature sends, and what the timeout chain looks like from the IDE to the GPU. Record the answers with the GitLab and vLLM versions you deployed, since both move on their own release schedules. Treat any mismatch as a blocking issue, because a wrong model identifier or an open /metrics or /health route will fail either in confusing ways or in ways that expose your model server.
Which settings map between vLLM, the gateway and GitLab
This table traces five values (endpoint URL, API key, model name, context length and timeout) through the vLLM server, the AI gateway and the GitLab instance. Read each row left to right to see where a value is set, who owns it and where you confirm it before go-live. Most mismatches come from one value being entered in two places with different spellings or limits.
| Setting | Where it is set | Owner and confirmation |
|---|---|---|
| Endpoint URL | vLLM serves OpenAI-compatible routes such as /v1/chat/completions; the gateway points at that base URL as its upstream | Platform team owns the internal address; confirm with a test request from the gateway host to the vLLM route |
| API key | vLLM --api-key option; the same secret is configured as the upstream credential in the gateway | Security team owns issuance and rotation; store in a secret manager and confirm rejected requests fail as expected |
| Model name | Name passed to vllm serve, mapped to a model identifier in the gateway routing configuration, and selected in GitLab Duo settings | ML or platform engineer owns the name; confirm it matches exactly across all three, including case and path separators |
| Context length | vLLM maximum model length at serve time; gateway request limits; GitLab feature expectations for prompt size | Model owner and platform team; confirm the vLLM limit against the model's documented support in GitLab's support matrix |
| Timeout values | Client timeout in GitLab, gateway upstream timeout, and vLLM request handling under concurrent load | Platform team owns the chain; set the outermost timeout longest and confirm with load tests on your own hardware |
| Reverse proxy allowlist | Proxy in front of vLLM permits only user-facing routes; /metrics and /health stay internal | Security team owns the allowlist; confirm that blocked paths return errors from outside the trusted network |
Securing the vLLM endpoint and its node network
A vLLM server that answers GitLab AI gateway requests is only as protected as the routes it exposes and the network between its nodes. The cards below cover the limits of the built-in API key, the reverse proxy allowlist, isolation of multi-node traffic, metrics access, key handling and transport encryption. Work through them together before the endpoint accepts traffic from the gateway.
Know the limits of --api-key
The --api-key option protects the endpoints under /v1, /v2, /inference and /cohere, but it does not cover /invocations, /pause, /resume, /abort_requests, /metrics or /health. The vLLM security documentation warns against relying on this key alone. Confirm which routes your vLLM version exposes before you decide what the key protects, and treat it as one layer rather than the whole access control.
Allowlist routes at a reverse proxy
Place a reverse proxy in front of vLLM and forward only the paths the AI gateway needs, such as the OpenAI-compatible chat and completions routes. Deny everything else by default, including the control routes that pause, resume or abort requests. Test the allowlist from outside the gateway network, because a route left open at the proxy is reachable regardless of the vLLM key.
Isolate inter-node traffic
The vLLM security documentation states that communication between nodes in a multi-node deployment is insecure by default, with no authorization or encryption. Place all nodes on an isolated network segment that only inference nodes can reach, and block that segment from user and gateway traffic. Review this design whenever you add a node or change the parallelism layout, since tensor and pipeline traffic crosses the same boundary.
Restrict access to metrics
vLLM serves Prometheus-format metrics at /metrics, including request counts, token throughput, latency histograms and KV cache use. These values reveal load, user patterns and capacity, and the route is not covered by --api-key. Allow only your Prometheus server to scrape it, either by network policy or by proxy rules, and confirm that no public route forwards to it.
Keep keys in a secret manager
Store the vLLM API key and any gateway credentials in a secret manager rather than in shell history, unit files or plain configuration in the repository. Rotate the key on a schedule you define and after any staff change, and update the gateway configuration in the same change window. Ask your platform team which secret store and rotation process they will operate, since the key is only useful if both sides receive the new value together.
Encrypt traffic from gateway to endpoint
Requests from the self-hosted AI gateway carry prompts and source code context, so the connection to the vLLM endpoint should use TLS, even on a private network. Terminate TLS at the reverse proxy or on the vLLM host, and confirm that the gateway trusts the internal certificate authority. Verify the chain with a test request before go-live and record the certificate renewal owner.
Operating vLLM alongside GitLab after go-live
The vLLM server exposes Prometheus-format metrics at /metrics on its API server. Scrape that endpoint from your existing Prometheus setup and build dashboards and alerts around the signals that show whether developers are getting timely completions. The metrics cover request counts, token throughput, latency histograms and KV cache use, and the histogram buckets can be configured, so confirm the bucket boundaries match the latency targets your team actually agreed on.
- Request counts and errors: Alert on sustained error rates and on a drop in successful requests, which often signals that the gateway or network path has broken before the model has.
- Token throughput: Track generated and processed tokens over time to see how much work the GPUs carry at peak hours.
- Latency histograms: Watch tail latency rather than averages, since slow completions are what developers notice first.
- KV cache use: Sustained high cache use means requests queue or get pushed out under load, which is a signal to plan capacity.
- Endpoint exposure:
/metricsis not protected by--api-key, so restrict it at the reverse proxy and to the monitoring network.
Before go-live, confirm that these alerts reach the team that owns the GPUs and that someone is named to act on them.
Two release streams move at different speeds. vLLM ships new minor releases about every two weeks, so waiting for a quarterly change window will leave the serving stack far behind. GitLab ships a minor release on the third Thursday of each month, and patch releases twice a month, and it requires upgrades to pass through required upgrade stops, which fall at versions x.2, x.5, x.8 and x.11 since GitLab 17.5. The GitLab maintenance policy also limits backports: bug fixes go only to the current stable release, while security fixes reach the previous two monthly releases as well. Plan your GitLab upgrade path around those stops and treat each vLLM upgrade as a change to the model service.
- Pin versions: Record the vLLM version, model revision and GitLab AI gateway version together so every change is traceable.
- Regression test model output: After each vLLM upgrade, run a fixed set of prompts and compare outputs against an accepted baseline, since a new release can change tokenization, sampling or defaults.
- Check gateway compatibility: Confirm the self-hosted AI gateway still reaches the OpenAI-compatible endpoints it uses after each change.
- Stage the rollout: Upgrade a non-production endpoint first and keep the previous release available for rollback.
- Align with GitLab stops: Schedule GitLab upgrades so the GitLab version and the gateway version stay supported together, and confirm the pairing in the GitLab documentation before each step.
Before each upgrade, confirm which GitLab and gateway versions are supported together and who signs off on the regression results.
Capacity planning does not end at go-live. As more developers adopt Duo features, concurrent requests rise, and the GPU sizing that worked for the pilot may no longer meet latency targets. vLLM scales a model across GPUs with tensor parallelism inside a node and pipeline parallelism across nodes, and multi-node runs need Ray or the vLLM multiprocessing backend. Adding nodes is therefore a networking and design change, not only a hardware purchase. Growth should be driven by measured load from the monitoring data, not by guesses.
- Measure peak concurrency: Use request and KV cache trends to find the busy hours before you add hardware.
- Scale within a node first: Adding GPUs to a node through tensor parallelism is usually simpler to operate than spreading the model across nodes.
- Isolate multi-node traffic: vLLM's security documentation states that node-to-node communication is insecure by default, so place those nodes on an isolated network before expanding.
- Re-test after resizing: Repeat the regression tests and latency checks after any change in parallelism or hardware.
- Plan for model changes: A larger or different model needs new sizing, so treat model swaps as capacity projects.
Before expanding, ask your vendor or partner for a written sizing estimate tied to your measured peak load, with the hardware, network and operating costs spelled out.
Common vLLM and gateway failures and how to fix them
This table lists the failures teams most often hit when a vLLM endpoint sits behind the self-hosted AI gateway, with the likely cause and fix for each. Read it as a first-response checklist: match the symptom you see, check the cause column, then apply the fix in your own environment before changing model or hardware settings.
| Failure | Likely cause | Fix |
|---|---|---|
| Model does not load or fails with out-of-memory errors | The model weights plus KV cache exceed available GPU memory, or tensor parallelism is set lower than the GPUs needed for the model. | Confirm the model and quantization against GitLab's support matrix, then raise tensor parallelism across the GPUs in one node or lower the maximum context length and memory utilization setting. |
| Gateway timeouts on long completions | The gateway or reverse proxy closes idle connections before a long generation finishes, or the default request timeout is shorter than typical Duo responses. | Raise read and idle timeouts on the gateway and proxy for the chat and completions routes, and confirm the timeout values with your GitLab version's documentation. |
| 401 Unauthorized from the key check | The API key configured in vLLM does not match the key the gateway sends, or the key was rotated on one side only. | Align the key in the vLLM startup configuration with the gateway's model configuration, store it in a secret manager, and restart both services after rotation. |
| Admin or control endpoints reachable by users | vLLM's API key protects only the /v1, /v2, /inference and /cohere paths, so /metrics, /health, /pause, /resume and /abort_requests remain open unless blocked. | Place a reverse proxy in front of vLLM that allowlists only the endpoints the gateway needs and keep the vLLM port off user-facing networks. |
| Version drift between GitLab, the AI gateway and vLLM | Components were upgraded independently, so request formats or supported endpoints no longer match what GitLab expects. | Pin and record versions for all three components, check GitLab's compatibility guidance before each upgrade, and roll out changes in a test environment first. |
| 404 on the chat or completions path | The gateway routes to a path that vLLM does not expose, the model name in the request does not match the served model name, or the base URL is missing its /v1 prefix. | Compare the path against the endpoints vLLM serves, set the served model name to match the gateway configuration, and confirm the base URL includes the correct prefix. |
Questions buyers ask about serving Duo with vLLM
How much does it cost to serve GitLab Duo models with vLLM?
vLLM itself has no license fee, since it is Apache-2.0 licensed with no paid edition. The cost comes from four places: GPU or other accelerator capacity, the surrounding infrastructure (servers, storage, networking, a reverse proxy and monitoring), the implementation work, and the people who operate the stack afterward. GPU capacity is usually the largest recurring cost, whether it is bought or rented. Ask any partner for a fixed-scope quote built from a written scope that names the models, the number of concurrent developers, the hosting location and the operating model. DeployOpen does not publish a price list, and a quote follows from that written scope.
How long does a vLLM deployment for Duo take?
The timeline depends on how much of the environment already exists. Teams that already run GPU servers, a GitLab instance and an AI gateway have less to build than teams starting from bare hardware. The main drivers are accelerator procurement and delivery, driver and CUDA or ROCm alignment, model selection against GitLab's support matrix, network isolation, and testing with the self-hosted AI gateway before developers use it. Ask for a timeline tied to named milestones in the written scope rather than a single number, and confirm which GitLab and gateway versions the plan targets.
What does our team need to provide for this work?
Your team needs to supply the environment and the decisions that only you can make. That includes access to the GPU hosts or the cloud accounts that will run them, the network design that places the model endpoint and the AI gateway, a named owner for the GitLab instance and its version alignment, identity and access rules for who can reach the endpoint, and the approval path for model choices. You will also need someone who can answer questions about your change windows, logging rules and data handling. A written scope should list each of these as a client responsibility so gaps show up before work starts.
Can our own team build and run this without a partner?
Yes, if your team has experience with GPU hosts, Linux administration, container or service management, reverse proxies and GitLab administration. The vLLM server is started with vllm serve and exposes OpenAI-compatible endpoints, so the serving layer itself is not the hard part. The harder work is securing the endpoint, since vLLM's --api-key option does not cover every route, placing the gateway and model in a controlled network, collecting metrics, and keeping model, vLLM and GitLab versions aligned over time. Most in-house teams underestimate that ongoing operation. A partner can take on the build, the hardening and the runbooks, or cover a smaller part of the work.
Can GitLab Duo run with vLLM in a fully offline deployment?
Offline operation is possible in principle, but it depends on the GitLab feature set and licensing you choose. GitLab documents offline and air-gapped environments and has separate guidance for running GitLab Duo Self-Hosted offline. vLLM can run without internet access once its model weights and container images are staged inside your network. Confirm with GitLab which Duo features and licensing path apply to a fully offline setup, and plan how models, packages and updates will be transferred across the boundary. Keep in mind that vLLM's multi-node communication is insecure by default and must sit on an isolated network.
Should we use AWS Bedrock or Azure OpenAI instead of our own GPUs?
Managed model platforms are a reasonable alternative when a private cloud arrangement meets your data residency, compliance and contractual needs. The trade-off is between paying a cloud provider per use or per capacity and carrying your own GPU capacity and operations. Self-hosting on GPUs keeps prompts and model traffic inside infrastructure you control, which matters for some regulated workloads. Managed platforms reduce hardware and operations work but add a provider dependency and their own data terms. Compare both options against the same written requirements for residency, offline needs, model choice, staffing and total recurring cost, and confirm the details with GitLab's current support material.
What ongoing work follows after the first deployment?
Plan for recurring work from the start. vLLM ships new minor releases about every two weeks, so you will need a cadence for testing and applying updates, along with checks that each new version still works with your chosen model and GPU drivers. GitLab's own releases, required upgrade stops and gateway version alignment add further work. You also need to watch Prometheus-format metrics at /metrics, rotate keys, review access, and keep backups of configuration. Ask for a written operations scope that states who performs each of these tasks and how often, and whether ongoing support is handled through a retainer.
How DeployOpen scopes and supports a vLLM serving deployment
DeployOpen scopes a vLLM serving deployment as a written, fixed-scope engagement agreed before any work starts. The scope normally covers four parts: the GPU host, the vLLM endpoint, the connection to the self-hosted AI gateway, and security hardening and testing. For the GPU host, the scope should name the accelerator vendor and driver stack, the operating system, the network segment and how the host is patched. For the vLLM endpoint, it should name the model, the serving flags, the parallelism layout and the reverse proxy that allowlists only the paths meant for users, since the vLLM security documentation warns against relying on --api-key alone. For the gateway connection, it should name the base URL, the model name exposed to GitLab and the version alignment with the GitLab instance. Each item should carry a deliverable and an acceptance check, so the client knows what done means.
Hardening and testing belong in the same written scope rather than being added at the end. Typical items include isolating multi-node traffic on a segregated network, restricting access to /metrics and /health, reviewing what the gateway logs, and testing the endpoint for unauthorized access. DeployOpen also documents the runbook for the deployment, including upgrade steps for vLLM, which ships new minor releases roughly every two weeks, so the client can see how the version will be tracked. Where the client needs evidence for an assessment, DeployOpen prepares the configuration and records, but independent auditors issue any attestation.
DeployOpen does not publish prices. A quote follows from the written scope, and the client should ask the quote to separate one-time build work from recurring items. Ongoing support, upgrade planning and continuous testing can be added as an optional retainer once the initial deployment is live. Ask for the scope to state which hardware is supplied by the client, which is provided through a cloud account, and what is excluded, such as GPU procurement or model licensing. Confirm the GPU capacity and model choice with the GitLab support matrix and your own load tests before the scope is signed, since those decisions set the cost more than any labor line.
