Deploying the GitLab AI gateway

What the GitLab AI gateway does and when you need it

The GitLab AI gateway is the self-hosted service that receives GitLab Duo requests from your GitLab instance and forwards them to the language models that answer them. It sits between GitLab and the model servers so that GitLab reaches your models through an endpoint you run, instead of calling an external provider. For a team running self-managed GitLab that wants Duo on its own models, the gateway is the component that turns a model server on your network into a service GitLab can use. Its placement, its network paths and the model servers behind it decide what data leaves your environment and which teams must approve each connection. Treat the gateway as part of the Duo design from the first planning session.

A self-hosted gateway stops being optional once your Duo features must run on self-hosted models, because GitLab then needs an endpoint inside your network to send Duo requests to. It is also central to fully offline deployments, where the hybrid option would otherwise send some features to services outside your environment. In those cases the gateway and the models must both sit inside the offline boundary, and every feature you enable needs a path that stays within it. Confirm with GitLab which Duo features depend on the gateway in your configuration, and check the GitLab offline environments guidance for components that need separate handling. Record the answer before you choose models, because it sets the hardware and network scope for the rest of the project.

Exact request paths, component names and version rules are tied to specific GitLab releases, so confirm them in the GitLab Duo Self-Hosted documentation for the release your instance runs before you draw network diagrams or write firewall rules. Version alignment is the rule to plan around: the gateway must match the GitLab instance it serves, and because GitLab requires upgrades to pass through required upgrade stops, the gateway version is scheduled alongside each GitLab upgrade step. Gather four inputs first: your current GitLab version, the planned upgrade path, the network segments that must reach the gateway, and your offline licensing position. Those answers determine whether the gateway can be placed where your security team expects it.

Placement options for the GitLab AI gateway

This table compares three places to run the self-hosted AI gateway: beside the GitLab instance, near the model servers, or on a Kubernetes cluster you already run. Read each row against your own network, team and separation requirements, then confirm the supported layout with GitLab and your model vendor before you choose.

Decision factor	Dedicated VM beside GitLab	Shared tier near model servers	Kubernetes-hosted deployment
Latency to GitLab	Lowest path to the GitLab instance when both sit in the same network segment	Depends on the network path back to GitLab, so measure it from the gateway host	Depends on cluster placement and ingress routing, so confirm the traffic path to GitLab
Latency to model servers	Depends on the network path to the model tier, which may cross a segment boundary	Lowest path to the models when both sit in the same zone or rack	Depends on whether the cluster and model servers share a network
Operating team	The team that already runs the GitLab host, usually platform or infrastructure	The team that runs the model platform, often the AI or infrastructure group	The platform team that already operates the cluster, with GitLab and model teams as consumers
Separation from the model tier	Separate host and firewall rules, but shares the trust zone with GitLab	Often shares a trust zone with the models, so access controls must be explicit	Separation depends on namespaces, network policies and node pools you configure
Install method to confirm	Confirm the supported package and version alignment for the gateway in the GitLab installation documentation	Confirm the supported deployment method and any model endpoint format the gateway expects	Confirm the Helm or container method and version pinning in the GitLab documentation before you build
Offline support	Confirm offline package delivery and image handling for air-gapped hosts	Confirm that the model tier and gateway can run without internet access	Confirm that required images and charts can be mirrored into your registry
Version alignment	Plan gateway upgrades alongside each GitLab upgrade stop	Plan gateway upgrades alongside GitLab and model server releases	Plan gateway upgrades alongside GitLab and cluster upgrade windows
What to ask GitLab and the model vendor	Is this layout supported, and which network paths must stay open?	Is a shared trust zone supported, and how are credentials scoped?	Is this cluster layout supported, and which components stay outside Kubernetes?

Which network paths the gateway needs before anything runs

Before installing the self-hosted AI gateway, write down every network path it needs and get each one approved. There are three groups: inbound traffic from your GitLab instance to the gateway, outbound traffic from the gateway to the model endpoint that serves your Duo models, and any connections to licensing or update services. Each path needs a named source, a named destination, a port and a protocol. Treat the list as a change record, because the security team will ask for it during review and it becomes the baseline for firewall rules later.

Allow each path explicitly in firewall and security group rules rather than relying on broad internal ranges, then test it with the hostnames the deployment actually uses, not IP addresses copied from a diagram. Confirm the exact ports, hostnames and certificate requirements in the GitLab Duo Self-Hosted documentation for the GitLab release you are targeting, since these details change between releases. Ask your GitLab team to confirm which licensing or telemetry endpoints apply to your license type, because an offline deployment has a different list from a connected one.

Expect the first connection attempt to fail for reasons unrelated to the gateway itself. DNS resolution that works from your workstation may not work from the gateway host, an internal certificate authority may not be trusted by the service making the call, and an HTTP proxy may intercept traffic or exclude the internal hostnames you need. Check each of these from the gateway host with the same tools and accounts the service will use, and record the results. Resolving these issues before installation saves a repeat of the whole change window.

Setup sequence for the GitLab AI gateway

This sequence takes a self-managed GitLab instance from a version check to a verified gateway that serves licensed users, with a check closing each step so problems surface before the next one begins. The order matters because the gateway must match the GitLab release and reach a model endpoint that supports it.

Confirm the GitLab version and upgrade stop position
The platform engineering lead checks the running GitLab version against the upgrade paths documentation and records the most recent required upgrade stop the instance has passed. If the instance sits between stops, plan the upgrade first, because the gateway release is aligned to the GitLab version and the upgrade sequence can change which gateway release is valid. Done means you have a written version record, the next required stop, and a confirmed target release for the gateway that matches the instance. Confirm the exact stop list for your target release in the official upgrade documentation before you commit to a date.
Confirm the model endpoint and compatibility
The AI or ML platform owner confirms which model endpoint the gateway will call and checks that the model family, serving software and version are listed as supported for the chosen GitLab release in GitLab's Duo support documentation. Record any rating limits for the features your teams intend to use, since some features may be rated as partial or limited for certain models. Done means you have a written compatibility note naming the model, the endpoint, and the GitLab release it was validated against. Ask GitLab or your account contact to confirm support for your exact combination if the matrix does not list it.
Provision hosts or cluster capacity
The infrastructure team allocates the hosts or Kubernetes capacity for the gateway in the same network zone as the GitLab instance and within reach of the model endpoint, with no route to the public internet if the deployment is offline. Confirm the operating system, container runtime or cluster version, DNS names, TLS certificates and firewall rules against the gateway's installation requirements for the release. Done means the target environment answers on the required ports, resolves the names you will use, and passes a basic outbound test to the model endpoint and an inbound test from the GitLab instance. Confirm sizing with the vendor documentation for your release, since capacity needs depend on expected concurrent users.
Install the gateway by a supported method
The platform engineer installs the gateway using one of the methods the official setup guide for that release documents, and pins the image or package version to the release that matches your GitLab instance. Avoid mixing a current gateway build with an older GitLab instance, and keep the install record so the same version can be rebuilt during recovery. Done means the service starts cleanly, the process or pods report ready, and the logs show no startup errors. The exact commands, image names and environment variables must come from the official setup guide for your release, not from this summary.
Configure connections to GitLab and the model endpoint
The platform engineer sets the gateway's connection values for the GitLab instance and the model endpoint, including the base URLs, authentication secrets and TLS trust settings, and stores secrets in your approved secrets manager rather than in plain configuration files. Configure the gateway to trust your internal certificate authority if the endpoints use one. Done means the gateway reports successful connections to both systems in its logs or status output, and the secrets are restricted to the service account that needs them. Use the configuration key names from the official guide for your release, and confirm each one against that version.
Verify health and a request from a licensed user
The operations team checks the gateway's health endpoint, reviews the logs for authentication or routing errors, and then has a user with a valid Duo seat or add-on send a test request through a supported feature in the GitLab interface. Confirm the request reaches your model endpoint, returns a response, and appears in the logs and metrics you have configured. Done means the health check passes, the test request completes end to end, and the test result, release versions and configuration owner are recorded in the change record. Repeat the check after each gateway or GitLab upgrade, and confirm the health endpoint path and expected responses in the setup guide for your release.

Keeping the gateway and GitLab on compatible versions

The AI gateway, the GitLab instance, the model server and the Runner all move on their own schedules, and a mismatch between them can break Duo features without an obvious error. The cards below cover the decisions that keep these components aligned through each upgrade, from required upgrade stops to the monthly release rhythm.

Crossing a required upgrade stop

GitLab requires upgrades to pass through required upgrade stops, which since GitLab 17.5 fall at versions x.2, x.5, x.8 and x.11, for example 18.2, 18.5, 18.8, 18.11, 19.2, 19.5, 19.8 and 19.11. Moving GitLab from one stop to a version beyond the next without stopping can leave the gateway calling API behavior that no longer exists. Before any GitLab upgrade, confirm the gateway release that GitLab's documentation pairs with your target version. Plan each stop as its own change with a rollback point.

Running a gateway that does not match

The gateway is a separate component that must be on a release that GitLab supports for your instance version. A gateway release ahead of or behind the GitLab instance can cause Duo requests to fail in ways that look like model problems. Record the gateway version next to the GitLab version in your change log, and confirm the compatibility matrix with GitLab before each change to either side.

Upgrading the model server separately

The model server, such as a vLLM deployment behind an OpenAI-compatible endpoint, can be updated on its own schedule, which is a common source of drift. A new model server build or model file can change response formats or supported models that the gateway expects. Treat the model server as part of the same change set, test a Duo request end to end after each update, and keep the previous model artifact available for rollback.

Matching the Runner and instance versions

GitLab Runner executes the jobs that GitLab schedules, and the Runner documentation advises keeping its major and minor versions aligned with GitLab. A Runner that lags behind a newly upgraded instance can behave differently from the rest of your pipeline fleet. Include Runner hosts in the upgrade plan so that CI and Duo changes are not tested against different versions of GitLab.

Timing windows around the monthly release

GitLab ships a minor release on the third Thursday of each month and patch releases twice a month. Security fixes are backported to the previous two monthly releases, while bug fixes go only to the current stable release, so a version that is behind can still receive security fixes but not functional ones. Schedule gateway upgrades in a fixed window that follows the monthly release, and avoid moving the gateway and GitLab in the same week.

Staging the change in an offline environment

In air-gapped deployments, you cannot pull a newer gateway or container image on demand, so each release must be mirrored into your registry before the window opens. Confirm which packages and images the offline guidance requires for your GitLab version, and verify the checksums of the artifacts you import. Rehearse the upgrade on a non-production copy of the instance, the gateway and the model server together.

Connected and offline gateway deployments compared

In a connected deployment, the AI gateway runs inside your infrastructure but may reach GitLab-managed services, depending on the configuration you choose. Your team can install from network sources, follow GitLab's release cadence directly and activate an online license. The main design variable is outbound access, so the architecture depends on which destinations your security team approves.

Installation sources: Packages and images can be pulled over the network, but you must list the exact registries and endpoints your proxy or firewall needs to allow.
Update flow: GitLab publishes patch releases twice a month and a minor release on the third Thursday of each month, so you can schedule pulls into a fixed change window.
Licensing: Online licenses are activated against GitLab. Ask GitLab whether usage-based billing applies to your Duo entitlement and how activation is renewed.
Outbound data: Confirm, feature by feature, which requests leave your network under your chosen configuration. Do not assume that a feature stays internal until GitLab confirms it in writing.

Before firewall rules are written, obtain GitLab's list of outbound destinations for your configuration and record the decision in your architecture file.

Security controls for the gateway boundary

The AI gateway sees developer prompts, code context and model responses, so it sits inside your security perimeter as a trust boundary. The controls below work together: they limit who can reach the gateway, protect the credentials it uses, and keep its logs and hosts under the same discipline as the GitLab instance it serves.

Restrict which hosts reach the gateway

Only the GitLab instance and the model-serving hosts should be able to open connections to the AI gateway. Define these as explicit source rules in your firewall or security groups, and block direct access from developer workstations and general user subnets. Confirm with your GitLab Duo Self-Hosted documentation which outbound and inbound paths the gateway needs, and test that every other path is refused. Review the rules whenever the GitLab or model hosts change addresses.

Manage service credentials and tokens

Requests between GitLab and the gateway are authenticated with service credentials. Store them in your approved secrets manager rather than in plain configuration files, restrict who can read them, and record a rotation procedure before go-live. Confirm with the GitLab documentation exactly which token types the self-hosted gateway uses, how they are issued, and how a rotation is applied without a full outage.

Protect logs that hold prompts and code

Gateway logs may capture prompts, code snippets and model output, which can expose proprietary source or personal data. Decide what is logged at each level, set retention that matches your data policy, and restrict log access to a small operations group. Confirm whether prompt content can be disabled in the logging configuration for your version, and test that secrets do not appear in log output before you enable verbose modes.

Separate administrative access from user traffic

Administrative interfaces, configuration changes and host shell access should run on a separate management path from the traffic developers generate. Use dedicated administrator accounts, require multi-factor authentication for them, and keep administrative sessions logged. Users who only consume Duo features should never have a route to change gateway configuration or read its host.

Keep hosts and images patched

The gateway host operating system and any container images need a defined patch cycle that runs alongside GitLab's own release schedule. Pin image versions, scan images before deployment and rebuild them when a fix is released. Plan a maintenance window for gateway updates and confirm that the gateway version still matches the GitLab instance after each change.

Review the GitLab hardening guidance

GitLab publishes hardening guidance covering application settings, CI/CD, configuration files such as gitlab.rb and the operating system. The guidance is described as a work in progress, so roll it out gradually and record which recommendations you adopted, deferred or rejected, with reasons. Compliance evidence for these controls should come from independent assessors, since DeployOpen prepares systems and records but does not issue audit reports or certifications.

Operating the gateway after go-live

The gateway needs its own operating runbook, and a named owner who understands both the GitLab instance and the model tier behind it. Most failures after go-live sit at the seam between the two: a GitLab upgrade that changes what the gateway expects, a model endpoint that slows down while GitLab reports nothing wrong, or a rotated credential that one side still holds. Assign one person accountable for the gateway's health, with a named backup, and make sure that person can read the GitLab administration logs and the model server's metrics alike. Confirm the exact gateway settings, health endpoints and log locations against the version you installed, since these change between releases.

Monitoring should run alongside your existing GitLab monitoring rather than in a separate tool that nobody watches. Track gateway availability, request latency and error rates, and alert on the gap between what GitLab sends and what the gateway returns. Credentials for the gateway and its model connections also need a rotation schedule with a tested change procedure, so a rotation does not become an outage. Schedule upgrades against the GitLab maintenance policy, which ships minor releases monthly and patch releases twice a month. Align gateway versions with the GitLab instance each time, and check the GitLab upgrade paths for required stops, because a jump across one can leave the gateway behind.

Before the first production upgrade, rehearse the full runbook on a non-production copy: the upgrade order, the rollback steps for both the gateway and its configuration, and the checks that confirm Duo features respond afterward. Include the gateway and model tier in backup and disaster recovery planning, covering its configuration, secrets and any model artifacts you store, and record how you would bring the gateway back in a second location. Ask your DeployOpen team or your own platform staff to confirm recovery times for your environment rather than assuming them. A rehearsed rollback that has never been run is a plan on paper, so schedule it as a recurring exercise.

Limits to confirm in your environment before you commit

The gateway sits between your GitLab instance and your models, so its install method, version, model support, network posture, load behavior and logging all depend on choices made outside this page. Treat each card below as a question to put to GitLab or your model vendor in writing before you fix a design.

Which installation methods are supported

The gateway may not be offered through every method GitLab supports for the core instance, which include the Linux package, the Helm chart, the GitLab Operator, Docker and source installs. Ask GitLab which methods are supported for the AI gateway at your target version, and whether your chosen method for the main instance can host it. If your platform team standardizes on Kubernetes, confirm that before you plan node pools and ingress.

Exact version pairing with GitLab

Confirm which gateway release pairs with your target GitLab release, including whether the pairing changes across patch releases. Also ask how the pairing interacts with required upgrade stops, since a GitLab upgrade may force a gateway upgrade in the same change window. Write the pairing into your change plan so that a GitLab upgrade is never approved without it.

Supported model servers and accelerators

Ask GitLab which model servers and serving endpoints the gateway accepts for self-hosted Duo, and which accelerator types and driver versions those models have been validated on. Ask your model vendor whether the serving layer you intend to run is on that list. Do not assume that an OpenAI-compatible endpoint is sufficient without confirmation from GitLab that it is supported for your features.

Operation in a fully offline network

Ask GitLab whether the gateway, its container images, its dependencies and its license checks run with no outbound internet access. GitLab documents offline guidance for Duo Self-Hosted, but confirm whether every component of your chosen feature set is covered. Any feature that calls an external service would need a documented alternative before your air-gapped design is approved.

Behavior at your concurrent developer load

Ask GitLab for the gateway capacity model, including how many concurrent developer requests one gateway instance handles, how it scales horizontally, and which upstream model capacity is needed to match it. Ask for the figures in writing for your expected concurrency. This page gives no sizing figures, so the answer must come from GitLab's published guidance for your version.

What the gateway logs and where it sends it

Ask GitLab what request, prompt and response data the gateway writes to its logs, how long those logs are retained, and whether prompt content is recorded by default. Confirm how logs leave the host, whether through a file path, syslog or your monitoring stack, and whether that destination is inside your approved boundary. Your data classification policy should decide the answer before the gateway is deployed.

What does deploying the GitLab AI gateway cost and take?

What drives the cost of deploying the GitLab AI gateway?

The biggest drivers are how much of your environment the gateway must reach, whether the network is offline, and how many GitLab and gateway versions must be kept in step. A gateway placed in a simple network next to a connected GitLab instance is a small piece of work. A deployment that needs to reach a self-hosted model endpoint, sit behind inspection proxies, or run with no internet access adds design, testing and evidence work. Ask the partner to price each of these separately so you can see which choices move the total. Cost also rises when the gateway must be supported across several GitLab upgrades, because each upgrade requires a version check and retest.

Can our own team deploy the AI gateway without a partner?

Yes, if you have people who already run GitLab and can operate a Linux service, a reverse proxy and a model endpoint. The work is mostly configuration, network rules and version alignment, so the main gap is usually experience with the GitLab release cadence and the offline procedures, not the install itself. Teams that lack this experience tend to lose time on version mismatches and on network paths that look correct but fail under load or behind a proxy. A partner is most useful when the deployment is fully offline, when several environments must match, or when your staff cannot take on the testing and documentation in the same period.

What do we need to provide before the gateway work starts?

Plan for four inputs. First, network access: the gateway must reach your GitLab instance, and the model endpoint it calls must be reachable from its subnet. Second, model endpoint readiness: a self-hosted model service or a supported cloud model platform must already be running and tested, because the gateway cannot be validated without it. Third, a GitLab administrator who can change instance settings and confirm the installed version. Fourth, a security reviewer who approves the network rules, certificates and logging. Confirm the exact requirements for your GitLab version with GitLab's own documentation and your environment before you request a quote.

How long does the AI gateway deployment take?

Duration depends on the network path, how many environments you need, whether the gateway is offline, and how quickly your reviewers and model endpoint are ready. A single connected environment with a working model endpoint is usually much shorter than a multi-environment offline rollout with formal change approval. No duration is given here because the input that sets it, your environment and approval cycle, is not known until the scope is written. Ask for the timeline as part of a fixed-scope proposal, with dependencies on your reviewers and model endpoint listed separately.

What is one-time work and what recurs after the gateway is live?

The initial build, network configuration, integration testing and documentation are one-time work. Recurring work comes from GitLab's release cycle. GitLab ships a minor release on the third Thursday of each month and patch releases twice a month, and the gateway must stay aligned with the GitLab version you run. Upgrades that cross a required upgrade stop need more planning than routine patches. Monitoring, log review and periodic testing of the model path also recur. Ask any partner whether upgrades, monitoring and testing are included in the fixed scope or offered as a separate retainer.

How do version changes and upgrade stops affect the cost?

Version changes are one of the most important cost drivers because the gateway must match the GitLab instance it serves. GitLab requires upgrades to pass through required upgrade stops, which fall at versions x.2, x.5, x.8 and x.11 since GitLab 17.5, according to GitLab's upgrade paths documentation. If your GitLab instance is several stops behind, the gateway work includes both the platform upgrade and the gateway change. Ask the partner to state which GitLab versions the quote assumes and what happens if an upgrade is needed during the engagement.

What should a fixed-scope written quote for this work include?

A useful quote states the environment, the GitLab version, the network zones, whether the deployment is connected, hybrid or fully offline, the model endpoint it connects to, and the number of environments. It should list deliverables such as the installed gateway, configuration records, test results, runbook and handover. It should separate one-time build work from any recurring upgrade or monitoring service, and name the client inputs that the timeline depends on. Ask for assumptions in writing, so any change in network design or version is handled through a documented change to scope rather than an unplanned increase in cost. DeployOpen does not publish a price list, so the quote follows from the written scope.

How DeployOpen works on this

DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure. For the GitLab AI gateway, we start by confirming which GitLab version, network zones and model endpoints the gateway must serve, then build and hand over a version-aligned deployment you operate.

How we scope

Written scope confirms the GitLab instance version and the gateway version it must match
Scope maps network zones, DNS names, TLS certificates and which GitLab hosts reach the gateway
Scope records the model endpoints the gateway routes to and whether any traffic may leave your network
Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts

How we build

We install the gateway in your cloud accounts, data centers or air-gapped network, not in ours
We configure routing, authentication between GitLab and the gateway, and logging to your existing tooling
We test each connection path, including offline paths, and record results for your change records
We run security testing against the gateway and its exposed interfaces before handover

How we hand over

You receive runbooks for installation, version pinning, upgrades and rollback of the gateway
Documentation covers the upgrade sequence to follow when GitLab moves across a required upgrade stop
Your team is walked through health checks, logs and the steps to verify a working Duo connection
Configuration and evidence are delivered in your repositories so your team owns them after handover

How we support

Ongoing support, upgrades and continuous testing are available as a retainer after handover
Retainers can cover gateway version alignment each time GitLab ships a planned minor or patch release
Support scope is agreed in writing, so response expectations are known before an incident occurs
Ask for a quote built from the written scope, including what recurs and what is one-time