When a GitLab Runner Kubernetes executor setup fits your CI estate
Your self-managed GitLab already carries a steady CI load, and your compute already runs on a Kubernetes cluster, or is moving to one. In that situation the Kubernetes executor lets GitLab Runner run each CI job as a pod in a cluster you control. A job pod is built from a build container that runs your scripts, a helper container that handles Git checkout, cache and artifacts, and any service containers such as databases the job needs. Because each job gets its own pod, the cluster's scheduler, quotas and node pools decide how much parallel CI you can run, rather than a fixed set of VMs you size by hand.
The executor fits best when the cluster is already your platform of record and your teams can work within its isolation model. It is a weaker choice when your jobs need the whole host, a Windows or macOS toolchain, or privileged Docker-in-Docker builds that your cluster policy would not allow. In those cases a VM-based approach such as the Docker Autoscaler may be the simpler path. Ask your platform team which node pools, namespaces and Pod Security levels already exist, and whether CI jobs can run under them. Those answers shape the design more than any executor setting.
Plan version alignment before you install anything. GitLab Runner is distributed under the MIT license, and the GitLab documentation advises keeping the Runner's major and minor versions aligned with your GitLab instance. Check the version of your GitLab instance against the GitLab Runner documentation and pin the Runner chart or image to a matching release. Also confirm which Kubernetes versions your cluster runs, since the Kubernetes support window is short. Upgrades on both sides should be scheduled together, not discovered during a job failure.
Which configuration choices fit each team
A shared namespace is the simplest layout: every team's job pods run in one namespace under one runner configuration and one service account. Per-team namespaces separate workloads, quotas and RBAC bindings, at the cost of more objects to create and keep consistent. The choice mostly decides how far a misconfigured job can see and how many places an operator must change.
- Isolation: A shared namespace lets any job pod reach Secrets, ConfigMaps and other pods that the runner's service account can read. Per-team namespaces with their own RBAC bindings limit that reach to one team's objects.
- Operations effort: One namespace means one set of quotas, limit ranges and policies. Per-team namespaces multiply those objects, so templating and automation become necessary.
- Blast radius: A runaway job in a shared namespace can exhaust quota for every team. Per-team quotas contain that failure to the team that caused it.
- Cost visibility: Per-team namespaces make resource consumption easier to attribute to a team in cluster reporting.
- Ask your team: Confirm how many teams share the cluster, whether they need separate secrets, and who owns quota decisions.
Confirm the namespace model against your Kubernetes distribution's RBAC and quota features before you commit to a layout, and check the Kubernetes authentication documentation for how identities map to those bindings.
Privileged job pods run with elevated container permissions, which some builds need, for example to run Docker-in-Docker. Unprivileged pods avoid that elevation and are the safer default for most workloads. The trade-off is compatibility: builds that depend on container-level privileges need a separate path, and that path must be isolated from ordinary jobs.
- Isolation: A privileged container weakens the boundary between the job and the node. Unprivileged pods keep that boundary in place for ordinary builds.
- Operations effort: Privileged jobs usually need their own runner, tags and namespace, which adds configuration and monitoring to maintain.
- Blast radius: A compromised privileged job can reach the node and, from there, other pods. Restricting privileged runners to trusted projects limits that exposure.
- Alternatives: Rootless or daemonless build tools can remove the need for privileged pods in some pipelines. Confirm whether your build tooling supports them.
- Ask your team: List which jobs truly need elevation, which projects submit them, and whether protected branches gate those jobs.
Review the pod security settings your cluster enforces, including the Pod Security Standards, before you allow any privileged job to run in a shared cluster.
Runner scope decides which projects can use a runner: the whole instance, a group, or a single project. Instance runners are the easiest to share and the easiest to over-trust. Group and project runners narrow the audience, which reduces exposure but multiplies the registrations and configurations you must manage.
- Isolation: Instance runners can execute jobs from any project on the instance. Project runners limit execution to one repository, which fits sensitive deployment work.
- Operations effort: Instance runners are configured once. Group and project runners need registration, tagging and configuration for each scope, so automation matters more as the count grows.
- Blast radius: A compromised instance runner can affect every project that uses it. A compromised project runner affects only that project.
- Cluster sharing: Several runner scopes can point at the same cluster, so the cluster's namespaces and RBAC still set the real limits.
- Ask your team: Confirm which projects are trusted, which deploy to production, and whether runner tags separate workloads cleanly.
Before you choose a scope, read the GitLab Runner documentation on registration and runner types, and check how your chosen scope interacts with the namespace and privilege decisions above.
Kubernetes executor settings and when each fits
This table lists the main [runners.kubernetes] settings, when each one fits a team's workload and what fails when it is set wrong. Confirm each key name against the GitLab Runner configuration reference for the Runner version you deploy before you write it into config.toml. Read the last column first, because most failures in this executor come from permissions, resource sizing and node placement rather than from the job script itself.
| Setting | config.toml key | When it fits | What breaks if it is wrong |
|---|---|---|---|
| Namespace | namespace | One namespace per team or trust level, so RBAC, quotas and cleanup scope cleanly. | Teams share a namespace, and one team's jobs can see or affect another team's pods and secrets. |
| Service account and RBAC | service_account | A dedicated account per runner, limited to pod, exec, attach and secret verbs in its own namespace. | The default account or broad cluster roles let jobs reach other workloads in the cluster. |
| CPU and memory requests | cpu_request, memory_request | Defaults sized for a typical job, so the scheduler can place pods on suitable nodes. | Missing or low requests lead to overcommitted nodes and slow or evicted jobs under load. |
| CPU and memory limits | cpu_limit, memory_limit | Limits that cover normal build peaks while protecting neighboring pods on the node. | Limits set too low cause out-of-memory kills in builds that otherwise pass locally. |
| Overwrite limits | cpu_request_overwrite_max_allowed, memory_limit_overwrite_max_allowed | Teams set per-job resources through variables, with a ceiling enforced by the runner. | No ceiling lets one job claim a whole node. Overwrites disabled makes heavy jobs fail or fall back to defaults. |
| Helper container resources | helper_cpu_request, helper_memory_limit | Sized for Git clones, cache restore and artifact upload, and raised for very large repositories. | An undersized helper fails on clone or upload while the job pod looks healthy. An oversized one wastes capacity across concurrent jobs. |
| Image pull policy | pull_policy | Always for mutable tags, if-not-present for pinned digests or versions in air-gapped clusters. | Always adds registry load and fails when the registry is unreachable. If-not-present can run stale images. |
| Node selectors and tolerations | node_selector, node_tolerations | Route jobs to a dedicated build pool or to nodes with specific hardware. | A wrong selector leaves jobs Pending. A missing toleration keeps pods off tainted build nodes. |
| Privileged containers | privileged | Only for workloads such as Docker-in-Docker, on a dedicated node pool with a separate namespace. | Privileged jobs can reach host resources, so they should never share a namespace with untrusted projects. |
How to set up the Kubernetes executor in order
This sequence takes a Kubernetes cluster from a version check to a registered GitLab Runner that runs CI jobs as pods in separate namespaces per team, with permissions, resource limits and helper sizing set on purpose rather than left at defaults.
Confirm cluster version and skew
Before anything is deployed, the platform team checks the cluster's control plane and node versions against the Kubernetes version skew policy. The kube-apiserver must be at or ahead of the other components, and kubelets may be up to three minor versions older than the API server, so confirm each node pool sits inside that window. Also confirm that the Kubernetes minor version is one the project still maintains. Done means you have a written record of the apiserver, controller manager, scheduler and kubelet versions, and each one falls within the policy.
Create the runner namespace and service account
Create a dedicated namespace for the runner manager, separate from application workloads, and a dedicated service account for it. Do not reuse the default service account or an administrator's token. The manager pod uses this identity to create job pods, so keeping it isolated makes its permissions easy to audit later. Done means the namespace and service account exist, and kubectl auth can-i run as that account returns the expected answers for a small set of test verbs.
Grant least-privilege RBAC for pods, exec and logs
Bind a Role, scoped to the job namespaces only, that allows the manager to create, get, list, watch and delete pods, to create pods/exec and pods/attach for running job scripts, and to read pods/log for job output. Avoid cluster-wide roles and wildcard verbs. Review the Kubernetes Secrets good practices guidance before granting any read access to Secrets, because the runner needs them only if your job configuration mounts them. Done means a dry run from the service account can perform the required actions in the job namespace and nothing outside it.
Register the runner with an authentication token
Create the runner in the GitLab UI or API, which issues a token that begins with glrt-, and store it in a Kubernetes Secret in the runner namespace. Use the authentication token workflow rather than the legacy registration token, since the Helm chart and runner configuration reference the secret. Confirm the Runner version matches your GitLab version, as the Runner documentation advises keeping major and minor versions aligned. Done means the runner appears as online in the GitLab project or group runner list.
Set requests, limits and overwrite rules
In the runner configuration, set default CPU and memory requests and limits for build containers, then decide which job variables may override them. Limit overrides to an allowlist so a job cannot request arbitrary capacity from the cluster. Set node selectors and tolerations if builds should land on a dedicated pool. Ask your cluster team to confirm that a ResourceQuota and LimitRange cover each job namespace. Done means a test job that tries to exceed the allowed override is rejected or capped, and a permitted override is applied to the pod spec.
Size the helper container
The helper container handles Git clone, cache and artifact transfer for each job, so it needs its own requests and limits in addition to the build container. Start from the defaults in the runner documentation, then adjust based on repository size and artifact volume observed in test runs. Confirm which helper image your registry mirror provides, since air-gapped or restricted networks often need an internal copy. Done means a job with a large clone and artifact upload completes without the helper being evicted or throttled.
Run a test job in each team namespace
Create one job namespace per team with its own quota and RBAC binding, then run a pipeline from a project in each team. Check that pods are created in the correct namespace, that logs stream back to the GitLab job page, that artifacts and caches are stored, and that pods are removed after completion. Also confirm that a job from one team cannot list pods in another team's namespace. Done means each team's test pipeline passes and the cross-namespace check fails as expected.
Security controls for runner pods on shared clusters
A Kubernetes executor runs untrusted CI code as pods next to the cluster's own workloads, so the runner's permissions, the job namespace's policy, and the network paths out of each pod decide how much damage a malicious or careless job can do. The six controls below are the ones to design and verify before the first production pipeline runs.
Least-privilege RBAC for the runner
The runner service account creates, watches and deletes job pods, and it needs only the verbs and namespaces that job creation requires. Grant a Role per job namespace rather than a ClusterRole, and avoid rights to read Secrets or exec into pods beyond what the executor documentation requires. Confirm with kubectl auth can-i --list run as that service account, because a broad binding here is inherited by every job.
Pod Security Standards on job namespaces
Apply a Pod Security Standards level to each namespace where jobs run, starting with the baseline profile and tightening where the job workloads allow it. Privileged containers, host namespaces and hostPath mounts are the settings that let a job reach the node. Test each namespace label against a real pipeline before enforcement, since some build tooling needs capabilities that the stricter profiles block.
Secrets unencrypted in etcd
Kubernetes stores Secrets unencrypted in etcd by default, and the runner's CI variables are passed to job pods through Secrets. Enable encryption at rest for etcd, restrict list and watch access to Secrets, and consider an external store through the Secrets Store CSI Driver if your policy requires keys outside the cluster. Ask the cluster team who can read etcd snapshots, because backups carry the same data.
Isolating untrusted projects from privileged jobs
A fork, an external contributor or a newly created project should never run on the runner that deploys to production or holds cluster-scoped access. Give untrusted projects their own runners with separate namespaces, service accounts and tags, and restrict privileged jobs to protected branches. Confirm in GitLab which projects can use each runner, because the placement of a runner decides which code can reach its credentials.
Runner and GitLab version alignment
GitLab documents that Runner major and minor versions should match the GitLab instance, so the Kubernetes executor and the Helm chart need to move with each GitLab release. Plan runner upgrades into the same change window as GitLab upgrades, and check the GitLab upgrade stops before choosing the target version. A runner that lags behind can fail in ways that look like pipeline bugs.
Egress from job pods
Job pods can usually reach the internet, the cluster API, and internal services unless network policies say otherwise. Define which destinations a build needs, such as the GitLab instance, package mirrors and the container registry, and deny the rest with NetworkPolicy objects. Block access to the cluster API and to cloud metadata endpoints from job namespaces, and test the policy with a job that tries each path.
Why resource requests and overwrite limits decide job stability
Resource requests decide where a CI job can run, and limits decide what it can take once it is there. With the Kubernetes executor, each job pod is scheduled by the cluster scheduler like any other workload, so the CPU and memory requests set for the build container and the helper container determine whether a node has room for the pod at all. Requests set too low let the scheduler pack many jobs onto a node that then struggles under load, while requests set too high leave jobs pending even when the cluster looks idle. Limits in the runner's config.toml are the ceiling enforced by the kubelet. A limit that is too tight will not slow a job down gracefully. It will usually end the job with an out-of-memory kill, which shows up in the pipeline as a generic failure and sends engineers chasing flaky tests that are really starved of memory.
The runner lets job authors adjust these values through overwrite variables, such as the KUBERNETES_CPU_LIMIT family of variables, but only within the bounds the platform team has allowed. Confirm in the runner documentation which overwrite keys your runner version accepts and how the overwrite_max_allowed settings constrain them, because a permissive overwrite policy turns the .gitlab-ci.yml file into a way to claim the whole node. Decide early whether job authors may raise limits at all, and for which projects, since that choice connects directly to namespace layout and RBAC. The helper container deserves the same attention. It runs the cache restore, artifact upload and Git fetch steps, so an undersized helper can fail a job whose build step passed comfortably, and large artifacts or caches are usually where that failure appears. Measure helper resource use on your heaviest pipelines before you settle on a single value for every team.
Tight limits and job stability trade against each other, and the right position depends on your workloads. Generous limits reduce failures but let one noisy job degrade neighbors on the same node, while strict limits protect the cluster but generate retries and support tickets. A practical approach is to set requests from observed usage, set memory limits with headroom above typical peaks, and track OOM-killed jobs as a metric so the numbers are revised from evidence. Ask the team operating the cluster to confirm node sizes, any cluster-level quota or LimitRange objects, and whether autoscaling will respond to pending job pods, since those settings change what a request actually costs.
Limits to confirm in your environment before you commit
A Kubernetes executor runs CI jobs on a cluster you already operate, so the cluster's own constraints set the limits of the runner. The cards below cover the items to verify with your platform team before you commit to a design, since each one changes what the runner can do.
Kubernetes version support window
Kubernetes maintains only the three most recent minor releases, so a cluster on an older minor may already be outside support when the runner goes live. Check the current support table on the releases page and confirm your patch cadence can keep the cluster inside its window. Ask your platform team which minor version they plan to run for the next twelve months, and whether the runner chart you pick supports it.
Control plane topology and etcd backups
Confirm whether your clusters use a stacked control plane or an external etcd cluster, and how many control plane nodes sit behind the load balancer. Runner pods depend on the API server, so an etcd failure stops every CI job, not only deployments. Ask who takes and tests etcd snapshots, where they are stored, and how a restore is rehearsed.
CNCF conformance of your distribution
Confirm that your Kubernetes distribution appears as CNCF Certified Kubernetes software conformant, or that your vendor can show the required APIs behave as upstream. The Kubernetes executor relies on standard pod, secret and service account APIs, so a non-conformant distribution can break job pods in ways that are hard to trace. Ask for the conformance listing and the version it covers.
Registry and image pulls in air-gapped networks
Every job pod pulls a build image and the Runner helper image, so an air-gapped cluster needs an internal registry mirror for both before the first pipeline runs. Confirm which registry hosts the cluster can reach, how image pull secrets are issued to namespaces, and who mirrors new image versions. Missing images show up as pods stuck in pull errors, which look like runner failures.
GitLab and Runner version alignment
GitLab documents keeping Runner major and minor versions aligned with the GitLab instance, so each GitLab upgrade also needs a Runner chart and image update. Check which GitLab version you will run, the upgrade stop you will pass through, and when the Runner Helm chart is released to match. Confirm this in the GitLab upgrade paths before you pin a Runner version.
Runner manager placement
Decide whether the runner manager pod runs on job nodes or on a separate node pool. If the manager shares nodes with job pods, a busy build can starve it of CPU and memory, and it may be evicted with the jobs it controls. Ask whether your cluster has a node pool that can be tainted for the manager, and whether that pool is included in your autoscaling and patch windows.
Questions buyers ask before hiring help with the Kubernetes executor
What drives the effort and cost of a Kubernetes executor setup?
The main drivers are how many teams share the cluster, how many distinct trust levels the jobs have, and how much of the cluster platform already exists. A cluster that is already hardened, with identity, storage and monitoring in place, needs far less work than a greenfield build. Cost has two parts: one-time work for RBAC design, namespace layout, resource limits, helper container sizing and verification, and recurring work for upgrades, runner version alignment and capacity changes. Ask any partner for a quote built from a written scope that lists the clusters, teams, runner types and the verification steps included.
How long does a Kubernetes executor setup take?
Duration depends on the number of namespaces, the number of teams and runner tags, whether the cluster is new, and how many approval steps your security and platform teams require. No supplied figure gives a fixed duration, so treat any timeline as provisional until the scope is written. Ask for the phases, the dependencies on your cluster team, and the acceptance tests that close each phase. Fixed-scope engagements agree deliverables and timelines in writing before work starts.
What does the client team need to provide for the work?
Plan to supply cluster access with the rights needed to create service accounts, roles and namespaces, a GitLab instance with a project or group that will own the runner configuration, and named owners for each team's namespace. Security reviewers should be available to approve the permission model and the image registry used for job and helper images. Confirm whether your organization requires change tickets for cluster changes, because that affects scheduling more than the technical work does.
Can an internal platform team set up the Kubernetes executor alone?
Yes, if the team already runs Kubernetes in production, understands GitLab Runner configuration and has time to test permission boundaries. The hardest parts are not the first install but the decisions that are expensive to reverse: how service accounts are scoped, how limits are enforced when jobs can override them, and how namespaces map to teams. Many internal teams use an outside review of the design or a second pair of eyes on the RBAC model before rollout. Confirm your team's Kubernetes and GitLab Runner skills honestly before deciding.
How does the Kubernetes executor differ from autoscaling on cloud VMs?
With the Kubernetes executor, each CI job runs as a pod scheduled by the cluster, so capacity, isolation and resource accounting follow Kubernetes rules. With autoscaling on cloud VMs, such as the Docker Autoscaler executor, the runner manager creates and removes virtual machines that run Docker containers for jobs. The Kubernetes approach suits teams that already run platform capacity in a cluster and want one scheduler. The VM approach suits jobs that need a full host or a separate machine boundary. Compare both on isolation requirements, operational skills and the cache design before you choose.
How are Kubernetes and GitLab Runner upgrades scheduled?
Upgrade order matters. Kubernetes requires the kube-apiserver to be upgraded one minor version at a time before the controller manager, scheduler and kubelets, and kubelets may be up to three minor versions older than the API server. The Kubernetes project maintains only its three most recent minor releases, so plan a cluster upgrade cadence. GitLab Runner should stay aligned with GitLab's major and minor versions, and GitLab itself follows required upgrade stops. Ask for a written upgrade calendar that covers both products, a test cluster step and a rollback plan for each change.
What security checks should a Kubernetes executor setup include?
Start with the service account permissions the runner needs, then confirm that job pods cannot read secrets or other namespaces they do not need. Kubernetes stores Secrets unencrypted in etcd by default, so confirm encryption at rest and restrict list and watch access with least-privilege RBAC, or use an external store through the Secrets Store CSI Driver. Apply Pod Security Standards to job namespaces and review whether any job can override security-relevant settings. Ask the partner to show how each control is tested, not only configured.
Keeping the runner working through Kubernetes and GitLab upgrades
Upgrade order decides whether the Kubernetes executor keeps running jobs. The Kubernetes version skew policy requires kube-apiserver to be upgraded one minor version at a time, before kube-controller-manager, kube-scheduler and kubelets, and kubelets may be up to three minor versions older than the API server. Plan the cluster upgrade as a sequence of single-minor steps, then confirm the node pool that hosts runner job pods is still inside the allowed skew before you move on. The Kubernetes project maintains the three most recent minor releases and gives each patch line about one year of support, so a cluster that falls behind will lose security fixes before it loses function. Check the current support dates on the Kubernetes releases page as part of every upgrade plan.
Keep the GitLab Runner version aligned with the GitLab minor version. The Runner documentation advises matching major and minor versions, and the Helm chart or Operator image tag that deploys the Runner manager should be changed in the same change window as the GitLab upgrade. Use the GitLab maintenance policy to confirm which GitLab releases are currently maintained before you pick the target. Remember that GitLab requires upgrades to pass through required upgrade stops, so the Runner should follow the stop you actually land on, not the latest release.
Test the Runner upgrade in one team namespace first. Point a single team's runner registration at a canary Runner deployment with the new version, using a dedicated tag or a separate runner, and run that team's representative pipelines, including jobs that create helper containers and use their service account. Confirm pod creation, resource overwrite limits, cache access and log output before moving other namespaces. Ask your platform owner to write down the rollback steps, since reverting a Runner manager is quicker when the previous image tag and the config.toml are retained. Confirm the exact rollout behavior of your chart version in your own environment before you rely on it.
How DeployOpen scopes, delivers and supports GitLab Runner on Kubernetes
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure, including GitLab Runner on Kubernetes. We start from a written scope that sets out cluster ownership, namespace layout, permissions and resource limits before any runner is deployed.
How we scope
- Inventory the target clusters, their Kubernetes versions, network paths to the GitLab instance and the teams that will share the runners.
- Agree in writing how service accounts, RBAC roles, namespaces and resource overwrite limits map to each team and trust level.
- Confirm which GitLab and Runner versions are in use, and agree the deliverables and timeline before work starts, as engagements are fixed-scope.
How we build
- Configure the Runner Helm chart or Operator values, including helper container sizing, pod security settings and build volume handling.
- Test pipelines that exercise services, caches, artifacts and deliberately exceeded limits to confirm jobs fail and clean up as designed.
- Review cluster-level secret storage, encryption at rest and least-privilege access to pod logs and Secrets used by CI jobs.
How we hand over
- Deliver a runbook covering namespace layout, RBAC bindings, resource limit policy, version alignment steps and how to add a new team.
- Provide configuration as code in your repositories, with the values files, manifests and change steps your engineers will maintain.
- Walk your platform and security engineers through verification results, known limits and the decisions recorded during scoping.
How we support
- Offer ongoing support, Runner and cluster upgrade planning and continuous testing as a retainer, separate from the initial fixed-scope build.
- Plan Runner upgrades alongside GitLab upgrades and Kubernetes version changes, so the versions in use stay aligned and supported.
- Confirm with your team which support hours, response paths and upgrade windows apply before a retainer is signed.
