Why GitLab runner isolation decides how far a compromised job can reach
Isolating GitLab runners is a runner-fleet design decision, not a setting you flip after the instance is running. On a self-managed GitLab instance where merge requests from many contributors trigger CI jobs, the question that matters is how far a single job can reach if its code is malicious or has been tampered with. Some of those jobs need privileged mode for Docker builds, and that is where the risk concentrates. The answer depends on which executor each runner uses, which host it runs on, which projects it serves and how long its environment survives between jobs.
The trust problem is simple to state. Any job that runs code can attack the host that runs it and any other job that shares that host. A merge request from an outside contributor can add a script that runs under the runner's account, reads files left by earlier jobs, or uses credentials the runner holds. The GitLab Runner documentation describes shell, Docker and autoscaling executors, and each one draws the boundary in a different place. A shell executor runs jobs directly on the host, so the host itself is the boundary. Docker and autoscaling executors add a layer, but privileged containers weaken that layer. GitLab Runner is distributed under the MIT license, so the executor choice is yours to make and test, not something the license decides for you.
Teams need this topic as soon as one runner serves more than one trust level, for example a shared runner that builds forks, a runner that handles privileged Docker builds, or a runner that deploys to production. Before you design the fleet, confirm which projects send jobs to which runners, whether privileged mode is enabled anywhere, and whether any runner keeps state between jobs. Those three answers set the blast radius you must plan for, and they decide whether ephemeral VMs, protected-branch-only runners or a separate shell runner for trusted builds are needed.
Which jobs need separate runners and why
Not every CI/CD job carries the same risk, and a single shared runner fleet forces every job to trust every other job. The categories below separate jobs by what they can reach, so each group can get its own executor, host and pull policy. Use them to decide which jobs need their own runners before you choose an executor.
Privileged Docker-in-Docker jobs
Jobs that run privileged containers can typically reach the host kernel and devices, which weakens the boundary between the job and the machine running it. A job that builds images with Docker-in-Docker may need that privilege, so it should not share a host with unrelated projects. Check whether each build truly needs privileged mode, and whether rootless builds or a dedicated ephemeral VM can replace it.
Merge requests from forks
Pipelines triggered by merge requests from forks or untrusted contributors run code that someone outside your organization wrote. Such a job can read any variable the runner can reach, and can use the runner's network position to probe internal services. Keep these pipelines on runners that hold no deployment credentials and no access to internal networks, and confirm how your project settings control who can trigger them.
Deployment jobs with production credentials
Deployment jobs hold the keys to production, cloud accounts or registries, so a compromise of any other job on the same runner becomes a compromise of production. Restrict these jobs to protected branches and protected environments, and run them only on runners that no untrusted pipeline can reach. Ask which credentials each deploy job needs and whether a short-lived token would work instead.
Shell executor jobs on shared hosts
The shell executor runs jobs directly as the runner's user on the host, so one job can change files, installed tools and caches that the next job uses. On a shared host, a job from one project can affect another project's builds. Keep the shell executor for trusted builds on hosts dedicated to them, and confirm which projects can target those runners.
Jobs that pull arbitrary images
A job that pulls any image from a public registry runs whatever that image contains, including tools and scripts you never reviewed. Pull policies control whether a runner reuses cached images, which affects both trust and reproducibility. Decide which registries are approved for each runner group, pin images by digest where possible, and confirm how your pull policy treats cached images on mixed-trust fleets.
Choosing an isolation pattern for each job type
The autoscaling executor provisions a virtual machine for each job or small set of jobs and removes it afterward, so anything a job writes to disk, memory or the Docker daemon disappears with the machine. This is the strongest containment available for privileged work such as Docker-in-Docker, because a compromised build can only damage a machine that is about to be destroyed. Blast radius is limited to the job and the credentials it was given.
- Per-job disposal: Each machine starts clean, so residue from earlier jobs cannot be read by later ones.
- Network placement: Put the autoscaled machines in a segment that can reach the package mirrors and registries they need but not internal databases or admin interfaces.
- Scoped credentials: Give each runner only the tokens and cloud roles its jobs require, and rotate them on a schedule your security team sets.
- Operational cost: Your team owns image builds, machine lifecycle, capacity limits and cloud spend, and cold starts add time to every job.
- Fit: Mixed-trust fleets, regulated builds that run container builds and any workload where a compromised job must not reach shared state.
Confirm with your platform team which cloud or hypervisor driver your Runner version supports and how it handles machine cleanup after failures.
Protected-branch-only runners accept jobs only from protected branches and protected tags, so code that has passed review and merge controls is the only code that can reach them. Contributors working on feature branches cannot run jobs on these runners, which removes the path where an unreviewed change exfiltrates a deployment secret. The control depends on branch protection being configured correctly, so it is a policy gate rather than an isolation boundary on its own.
- Control point: The runner setting and the project's protected branch rules must agree, or the gate is weaker than it looks.
- Separate fleet: Run these runners on hosts with access to production deployment targets, and keep them apart from runners that serve untrusted branches.
- Review dependency: Protected branches are only as trustworthy as the merge approval rules behind them, so pair this with approval requirements and protected environments.
- Operational cost: Low in machine terms, but you must keep branch protection, runner tags and job tags documented and reviewed as projects change.
- Fit: Release and deployment pipelines in regulated environments where production credentials must never run against unreviewed code.
Check that no project can disable the protected-branch restriction on its runners and that forks and merge request pipelines cannot pick up these runners.
The pull policy controls whether a runner fetches a container image on each job, only when it is missing, or never, and it matters most when one fleet serves trusted and untrusted work. A runner that reuses cached images can run a previously pulled image that a later job should not share, while an always-pull policy costs bandwidth and time but forces each job to start from the registry's current content. Pull policy alone does not isolate jobs, so it is a tuning decision inside a fleet design.
- Always pull: Fetches the image on each job, which makes the image source explicit at the cost of registry traffic and job start time.
- If not present: Reuses local images, which speeds up jobs but lets cached images persist between jobs on the same host.
- Never pull: Relies only on images already on the host, which suits air-gapped hosts that load images from a controlled store.
- Mixed-trust rule: Use stricter pulls on shared hosts and reserve cached policies for dedicated fleets with the same trust level.
- Fit: Shared fleets where cache reuse is a convenience, and offline environments that must control exactly which images run.
Confirm the pull policy values your Runner version supports and how they interact with the executor you choose.
The shell executor runs jobs directly as a user on the runner host, with no container or VM between the job and the machine. It is the simplest executor to operate and has the least overhead, but every job shares the host's file system, processes, installed tools and any credentials stored on it. For that reason it should be reserved for trusted builds, typically code from your own maintainers on repositories you control, and never for privileged jobs or untrusted contributions.
- No isolation: A job can read other jobs' workspaces and any secret on the host, so the host is effectively part of the trust boundary.
- Dedicated host: Run shell runners on a host used only for this purpose, with no production credentials stored locally.
- Restricted scope: Limit it with protected branches, runner tags and project access so only approved projects can use it.
- Low cost: Little infrastructure to build, but the host needs careful hardening, patching and access review.
- Fit: Internal builds of trusted code, such as documentation or tooling that never runs privileged containers.
Check which projects and groups can register or use these runners, and confirm that no job on this host requires Docker-in-Docker or other privileged modes.
Runner options and when each fits
This table compares the runner executor and configuration choices that set how far a CI/CD job can reach into the host and network, along with the operational work each one creates. Read the isolation column first, then check effort, because the strongest option for privileged work is rarely the cheapest to run. Confirm exact setting names against the current Runner documentation before you write configuration.
| Option | Isolation strength | Operational effort | Typical fit |
|---|---|---|---|
| Shell executor | Weak: jobs run as the runner's OS user on the host | Low: no container or VM layer to manage | Trusted builds from protected branches only, on hosts dedicated to that purpose |
| Docker executor (not privileged) | Moderate: jobs share the host kernel but run in containers | Low to moderate: image management and host hardening | Mixed workloads from trusted projects where Docker-in-Docker is not required |
| Docker executor with privileged flag | Weak: a privileged container can reach host devices and the host kernel | Moderate: needs compensating controls around the host | Image builds that need Docker-in-Docker, with the runner kept away from untrusted code |
| Autoscaling on cloud or hypervisor VMs | Strong: each job or group of jobs gets a fresh VM that is discarded afterward | Moderate to high: images, scaling limits and cloud accounts to maintain | Mixed-trust fleets and privileged jobs that need a clean host for every run |
| Kubernetes executor | Moderate: pods are isolated from each other, but the cluster is shared unless namespaces and policies are strict | High: cluster policy, node pools and privileged pod rules to design | Teams already operating a hardened Kubernetes platform with clear namespace boundaries |
| Runner tags and protected runners | Depends on what the tag routes to: tags direct jobs but do not isolate them | Low: configuration and naming discipline | Keeping release jobs on runners that only protected-branch pipelines can reach |
| Pull policy: always | Not an isolation control: forces a fresh image pull for each job | Low: slower jobs and more registry traffic | Images that change under the same tag and where freshness matters more than speed |
| Pull policy: if-not-present or never | Not an isolation control: reuses cached images, so a poisoned cached image persists until removed | Moderate: cache hygiene and image pinning by digest | Faster jobs on controlled images, with digests pinned and runners rebuilt on a schedule |
Setting up isolated runner pools in sequence
This sequence turns a single shared runner fleet into separate pools, one for trusted builds and one for untrusted or privileged work, so that a fork or feature branch cannot reach the infrastructure that deploys to production. Each step ends with a check you can run before moving on.
Classify every job by trust level
Before touching any runner, your platform team and security lead list the jobs your pipelines run and label each one. Typical trust tiers are untrusted (merge requests from forks and external contributors), project-trusted (internal branches that run tests and builds) and deployment (jobs that touch production credentials or environments). Record which jobs need Docker-in-Docker or other privileged containers, since those drive the isolation requirements. Done when every pipeline job has a trust label in a shared inventory and the security lead has signed off on it.
Create separate runner pools with tags
Register a distinct runner pool for each trust tier, using runner tags that match the classification from the previous step. Keep the untrusted pool and the deployment pool on separate hosts or VM groups, and do not register one runner to multiple tiers. Use the GitLab Runner documentation to confirm the registration options for your chosen executor. Done when each pool shows its own tags in the Admin area and a test job tagged for one pool is picked up only by that pool.
Restrict trusted runners to protected branches
Configure the deployment pool so it runs only for protected branches and protected environments, and enable the setting that stops it from executing jobs on unprotected refs. Pair this with protected environments so that only approved roles can trigger production deployments. Confirm that the protected branch list matches your release process. Done when a job from an unprotected branch that requests the deployment pool stays pending or is rejected, and a job from a protected branch runs.
Configure the autoscaler with ephemeral VMs
For the untrusted and privileged-job pools, configure the autoscaler so each job runs on a fresh virtual machine that is removed after the job ends. This prevents a compromised build from leaving files, tokens or Docker state behind for the next job. Confirm the autoscaling options your cloud or hypervisor supports with the GitLab Runner autoscaling documentation, and set machine limits your capacity plan allows. Done when two consecutive jobs on the pool report different machine identifiers and the old machine no longer exists after the job.
Set pull policy and privileged settings per pool
Set the image pull policy and privileged container flag separately for each pool. Enable privileged mode only on the pool that needs Docker-in-Docker, and keep it off on the deployment pool. For the untrusted pool, restrict pull policy so jobs use images from your approved registry, and confirm that mounted host paths and Docker socket access are not configured. Keep the shell executor only for trusted builds on the internal pool. Done when a job on the deployment pool that tries to run a privileged container fails, and the runner configuration files show the flags you intended for each pool.
Verify isolation with an untrusted fork pipeline
Run a test pipeline from a fork or external test account that uses the untrusted tags and attempts to reach the deployment pool, including by requesting its tags, pointing at its variables and running on a protected branch name. Your security team should record the result of each attempt. Expected outcome: the fork job runs only on the untrusted pool, cannot read deployment variables or environments, and cannot be scheduled on the deployment pool. Any success is a failed control and should block go-live until fixed. Done when the test results are written up and retained as evidence.
How ephemeral VMs and tags keep one job from reaching another
A job reaches a runner only after GitLab matches it. The job's tags must match the runner's tags, and a runner set to accept only protected jobs will not pick up a job unless that job runs on a protected branch or tag. Once a match is made, the runner manager asks the autoscaler to provision a fresh virtual machine or container for that job. The job runs, and the machine is then removed, so files, process state and Docker daemon contents from one job do not carry into the next. Confirm in the GitLab Runner documentation which autoscaling executors and cloud drivers you plan to use, and how removal timing and failed-cleanup handling work for each, because these details vary by executor.
Secrets reach a job only at the moment GitLab hands it variables. Protected variables are released only to jobs running on protected branches or tags, so a pipeline from a feature branch never receives them. Three trust zones are involved: GitLab holds project configuration and secrets, the runner manager holds the registration credentials and the cloud credentials it uses to create machines, and the job host executes code from the pipeline. Treat the job host as the least trusted zone. Assume that anything a job can read, including variables it was given and any instance metadata reachable from its VM, is visible to whoever can push code that runs there. Confirm which cloud credentials the manager needs and whether job hosts can reach the cloud metadata service.
Before relying on this model, test it in a non-production project. First, confirm that a job without the right tags cannot land on the privileged runner, and that untagged jobs are not picked up by it unless you intend that. Second, run a feature-branch pipeline that checks whether a protected variable is set, and confirm the answer is no. Third, watch the autoscaler's instance list during a run and confirm that each job's machine is removed before the next job starts. Ephemeral VMs only isolate jobs if the matching rules send each job to the intended pool, so the tag and protection settings deserve the same review as the VM template.
Security and operational risks to control
Runners execute code from every branch and merge request, so a single weak runner can expose the host, its credentials and other teams' jobs. The cards below pair each risk with the control that contains it and the recurring task that keeps that control in place.
Docker socket and privileged mode
Docker-in-Docker jobs usually need privileged = true in the runner configuration, and a privileged container can reach the host kernel. Mounting the host Docker socket is worse, because any job can start containers that run as root on the host. Keep privileged mode on runners that only serve protected branches, and review the [runners.docker] section of config.toml after every change. Ask the team to confirm which jobs truly need privilege before granting it to a shared runner.
Runner tokens and registration
A runner authentication token lets its holder pick up jobs for every project the runner serves, and registration tokens that anyone can copy create rogue runners. Restrict who can register runners, scope each runner to specific projects or groups, and rotate tokens when staff change roles. Run a periodic review of the runner list in the admin area to remove runners nobody recognizes or that no longer have an owner.
Shell executor host sharing
The shell executor runs jobs directly as the runner's user on the host, so one job can read files, SSH keys or caches left by another job. This suits trusted builds on dedicated machines but not mixed-trust fleets. Keep shell runners limited to projects your organization fully controls, and confirm the runner user has no access to production credentials. Review the host's user accounts and filesystem permissions when the runner is provisioned and after any change to the build toolchain.
Cache and artifact leakage
Caches and artifacts from one job can be restored into another job on the same runner, which lets a merge request from an untrusted branch read data written by a protected pipeline. Use cache keys that include the branch or protected status, avoid storing secrets in artifacts, and set short artifact expiry. Check the runner's cache configuration, including any shared cache server, whenever you add a new project class to the fleet.
Runner version drift
Runners that fall behind the GitLab version can break job handling and miss security fixes, and each host may run a different version. Keep Runner major and minor versions aligned with GitLab as the documentation advises, and track the release cadence so updates land within your patch window. Assign one owner to inventory every runner host, record its version and schedule upgrades together with GitLab upgrade stops.
Questions buyers ask before isolating GitLab runners
How much effort does splitting a GitLab runner fleet take?
The effort depends mostly on how many distinct trust levels your pipelines need and how much of the fleet is already automated. A split with one trusted shell tier and one ephemeral tier for everything else is a bounded piece of work. Effort rises when privileged jobs such as Docker-in-Docker are used across many projects, when builds need different images or network zones, or when the current runners were registered by hand on shared hosts. Each runner group needs its own tags, executor settings, registration token handling, and a test that proves jobs from one tier cannot reach the other. Ask any partner to break the estimate into inventory, design, build, cutover and verification, so you can see which parts your team already covers.
Can an internal platform team isolate runners without outside help?
Yes, if the team has hands-on experience with GitLab Runner, the executors it supports, and the virtualization or cloud platform that will host ephemeral machines. The GitLab Runner documentation covers shell, Docker and autoscaling executors, so the knowledge exists in public docs. The harder parts are usually organizational: agreeing which projects are trusted, deciding who can change runner tags, and writing pull policies and protected-branch rules that developers will accept. Many teams do the build internally and bring in a partner for a security review of the result, or the reverse. Decide early who owns the runner configuration after go-live, because that owner will need to keep up with GitLab Runner versions.
What access does a runner isolation project need from our team?
Expect to provide a short list of access and decisions before work starts:
- Cloud account or hypervisor access with permission to create images, instances, networks and autoscaling groups in a non-production area first
- GitLab administrator or group owner rights to register runners, set tags and configure protected branches and environments
- Read access to current
config.tomlfiles, job logs and the projects that use privileged services - A test plan agreed with the people who own release pipelines, including sample jobs for each trust tier
The test plan matters most. Without it, nobody can say when the split is finished. Your team should also confirm which network paths runners may use to reach registries, caches and deployment targets.
How long does a runner split take for different fleet sizes?
Duration follows fleet size, the number of trust tiers and how many pipelines must be re-tested. A small fleet with a handful of projects is mostly configuration and verification. A larger fleet with many projects, several executor types and cloud autoscaling takes longer because each group of projects must be mapped, moved and checked. Image build pipelines, deployment jobs and projects that depend on Docker-in-Docker often need the most time. No duration is given here because none has been supplied as a researched figure. Ask for a schedule in the written scope that ties milestones to the number of runner groups and the test cases agreed up front.
Can we avoid privileged Docker jobs altogether?
Often you can reduce them, but not always remove them. Several common needs for privileged mode can be replaced: building container images with rootless tools, using a shared cache service, or moving image builds to a dedicated, ephemeral builder. Some workflows still need a Docker daemon, such as integration tests that start service containers. For those, the safer pattern is to run them only on ephemeral virtual machines that are destroyed after each job, and only for protected branches or trusted projects. Inventory every job that sets privileged mode before deciding. Then classify each one as removable, movable to an isolated tier, or still needing privileged execution with compensating controls.
How do GitLab and runner upgrades affect isolated runners?
Upgrades touch both the GitLab instance and the runners, and the docs advise keeping Runner major and minor versions aligned with GitLab. Each tier is a separate runner deployment, so an upgrade plan needs to cover every tier, the images used by ephemeral machines and the autoscaling templates. Required upgrade stops on the GitLab side mean you may need to move through intermediate versions, and runner versions must follow. Ephemeral fleets help here, because a new image can be rolled out and tested on one tier before others. Confirm with your partner or internal team how image rebuilds, runner registration and rollback will be handled and tested.
What does a fixed-scope quote for runner isolation cover?
A fixed-scope quote should list the work in writing before it starts. Look for the number of GitLab instances and runner groups in scope, the trust tiers and executor types to be built, the inventory of privileged jobs, the cloud or hypervisor targets, the test cases and acceptance criteria, the handover documentation and any runbooks for upgrades and image rebuilds. It should also say what the client provides, such as access and test pipelines, and what falls outside scope, such as changes to application code. DeployOpen does not publish a price list, so the quote follows from that written scope. Ongoing support, upgrades and continuous testing are available as retainers and should be quoted separately from the one-time build.
How DeployOpen scopes and delivers runner isolation
DeployOpen plans and builds runner isolation for self-managed GitLab inside infrastructure you control, starting from the jobs that need privilege and the trust level of each branch and project. The work is delivered under a written, fixed scope with documented runner configuration, tested isolation boundaries and a handover your team can operate.
How we scope
- Inventory which projects run Docker-in-Docker, privileged containers or host-level jobs, and which branches can reach them.
- Map trust levels for each project group, protected branch and environment before choosing executors or tags.
- Agree in writing which runner pools, executor types, pull policies and tag rules are in and out of the engagement.
How we build
- Configure ephemeral runner VMs or autoscaled pools so privileged jobs never share a host with untrusted work.
- Restrict high-trust runners to protected branches and tags, and set pull policies that match each image's trust source.
- Keep the shell executor limited to trusted builds on hosts that hold no shared credentials or cached state from other projects.
How we hand over
- Document each runner pool with its tags, executor, scope, registration owner and the jobs it is permitted to run.
- Test isolation boundaries with deliberately unsafe test jobs and record which containment checks passed or failed.
- Provide runbooks for adding runners, rotating registration tokens and responding when a job leaves its expected boundary.
How we support
- Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
- Ongoing runner upgrades, version alignment with GitLab and periodic isolation retesting are available as a retainer.
- Confirm with DeployOpen during scoping which GitLab and Runner versions your environment must support.
