How a GitLab Runner distributed cache in object storage keeps fresh instances fast
A distributed cache lets a job on a brand-new autoscaled VM restore its dependencies from shared storage instead of starting with an empty disk. Autoscaled runners create and remove machines as demand changes, so a cache kept on one VM rarely helps the next job. GitLab Runner's distributed cache stores cache archives in an S3-compatible object storage bucket that every runner in the fleet can reach. When a job starts, the runner downloads the archive that matches the job's cache key if one exists. When the job finishes, it uploads an updated archive for later jobs. Fresh machines can then skip re-downloading dependencies whenever a matching archive is present. Confirm the exact behavior for your runner version in the GitLab Runner documentation.
The *local cache* lives on the runner's own disk, so it only helps when a later job lands on the same host, which autoscaling makes unreliable. Teams usually need the distributed cache when runners are autoscaled, run on ephemeral VMs, or should share dependency archives across projects. The Shared and Path settings decide how runners and projects divide the cache inside the bucket. Before relying on either, confirm with your runner version's documentation how those settings interact with cache keys. A poor key design either shares archives too broadly or stores many near-duplicate archives, and both problems are hard to see until the bucket grows.
Before the first production pipeline, make three checks. Confirm that the runner manager and the fleet run versions aligned with your GitLab instance, since the documentation advises keeping major and minor versions matched. Confirm that the bucket exists in the region your VMs use, and that the runner credentials can reach only that bucket. Agree on a retention policy for old archives and confirm that your object storage platform can enforce it. Cache archives can hold dependency directories from several projects, so treat the bucket as a shared trust boundary. Finally, run two consecutive pipelines on a fresh instance and check the job log for cache restore entries. That is the simplest evidence that the object storage path works end to end.
Cache options and when each fits
This table compares the three cache settings that decide where autoscaled jobs store and restore dependencies, and how far that cache is shared. Read each row as a choice between isolation and reuse, then match it to how many teams, projects and branches use the same runner fleet. Exact key names and defaults must be confirmed against the GitLab Runner documentation for your version before you write the configuration.
| Setting | What it controls | When to use it | Isolation versus reuse trade-off |
|---|---|---|---|
| Distributed cache backend (S3-compatible) | Moves the cache from the runner manager's local disk to a shared object store, so a job on a fresh autoscaled instance can still restore dependencies. | Any autoscaled fleet where instances are discarded after jobs. Local-only caching gives little benefit when instances are short-lived. | Reuse rises across instances. Access control depends on the bucket credentials and policy, so confirm who can read and write the cache bucket. |
| Bucket endpoint and credentials | Sets the S3-compatible endpoint, bucket and access keys the runners use. Compatibility with non-AWS object stores must be confirmed for your product. | When you run an on-premises object store such as Ceph RGW or another S3-compatible service inside your own network. | Scoping credentials to one bucket per trust zone limits blast radius. One shared bucket with broad keys makes every project's cache readable by every runner. |
| Shared setting | Controls whether cache entries are shared between runners with the same cache configuration, rather than kept per runner. | Use it when many runners serve the same projects and you want cache hits across the fleet. Leave it off where runners serve different trust levels. | Enabling it increases hit rates and speeds jobs. It also lets a job on one runner read cache written by another, so it widens the exposure of cached secrets or tampered entries. |
| Path setting | Sets the key prefix or path under which cache is stored, which determines how entries are grouped and named in the bucket. | Use distinct paths per project, group or runner pool to keep caches separate while still sharing a bucket. Keep paths stable so restores hit. | Narrow paths isolate projects and reduce cross-project pollution. Overly narrow paths or paths that change every pipeline cause misses and extra storage cost. |
| Cache key design in .gitlab-ci.yml | Determines which cache entry a job restores and writes, typically keyed on files such as lock files or the branch name. | Keep keys tied to dependency files for reuse across branches. Add branch-specific keys only where branches must not affect each other. | Branch-keyed caches isolate work in progress but duplicate storage. Shared keys reuse dependencies but can carry a broken state from one branch to another. |
| Cache expiry and cleanup | Governs how long entries stay in the bucket and how old objects are removed, usually through bucket lifecycle rules or runner settings to confirm. | Always on a fleet with a bucket that grows over time. Confirm the retention behavior for your runner version before relying on it. | Longer retention improves hit rates. It also increases storage use and the time sensitive data remains available to other jobs. |
| Who can write the cache | Any job that can run on a runner with access to the bucket can write cache entries under the paths it is allowed to use. | Review this for untrusted merge requests or forks. Pair cache isolation with runner isolation rules from your fleet design. | Strict isolation usually means separate runners, buckets or paths per trust level, which adds configuration and operating effort. |
How the runner manager, cache server and jobs exchange cache archives
The cache archive moves between the job and the bucket on every run. When a job starts, the runner helper looks up the cache key that .gitlab-ci.yml defines for the job, downloads the matching archive from the S3-compatible bucket and unpacks it into the paths named in cache:paths. When the job finishes, the helper packs those paths into a new archive and uploads it under the same key. The runner manager holds the bucket endpoint and credentials in the [runners.cache] section of config.toml, so the instance running the job needs network access to the bucket but no local state from earlier jobs. Before go-live, confirm with your runner version which settings the helper reads from the manager and which it reads from the job definition, because the two can disagree.
The reason the cache lives outside the instance is that autoscaled instances are short-lived. When the fleet scales in, any archive stored on a VM's disk is lost with it, so the next job on a fresh instance would start cold. Keeping the archive in object storage lets any instance restore it. In an illustrative pipeline, a dependency install that takes several minutes on a cold run might finish in seconds on a warm restore, but only if the key matches. A cache miss means no archive exists for the key. The job still runs and passes, it simply downloads and builds dependencies from scratch, then uploads a new archive that later jobs can reuse. Misses also follow from a changed key, expired or deleted objects, or a bucket that the runner cannot reach, so the job log should be checked for cache restore messages after rollout.
The design decision is how keys map to archives, because that decides how often jobs share a cache. A key built from a lockfile hash gives each dependency set its own archive, while a fixed key lets all branches and runners overwrite one archive. Test both on a representative pipeline and watch the hit rate in job logs before choosing. Confirm with your own environment how the bucket credentials are issued to the runner, who can read and write objects under the cache prefix, and how long archives are retained. Shared and Path settings then determine whether runners on different fleets see the same archives, which the GitLab Runner documentation describes in detail.
Setting up the shared cache step by step
This sequence takes a runner fleet from no shared cache to a verified setup in which a job on one autoscaled instance restores dependencies written by a job on another. Each step closes with a check you can run before moving on.
Confirm runner and GitLab versions match
Your platform or infrastructure engineer checks the GitLab version on the self-managed instance and the GitLab Runner version on every manager host. GitLab's Runner documentation advises keeping the major and minor versions aligned with GitLab, so a mismatch is fixed before any cache work begins. Done means the runner manager reports a version that matches the GitLab instance's major and minor release, and the team has recorded both values in its change record. If the versions differ, plan the runner upgrade first, because cache settings tested on a mismatched runner may not behave as the documentation describes.
Provision an S3-compatible bucket for the cache
Your cloud or storage team creates a dedicated bucket in the object storage service you run, whether that is a public cloud, an on-premises S3-compatible system or a platform such as Ceph. Use a bucket name that signals its purpose, keep it separate from GitLab's own backup and artifact buckets, and confirm the region or endpoint that the runner managers can reach. Done means the bucket exists, the team can list it from a runner manager host using the endpoint it will configure, and the bucket's lifecycle and encryption settings match your data handling policy. Ask your storage vendor which expiration and encryption features apply to your deployment.
Create a least-privilege credential for the cache
A security engineer creates an access key or equivalent credential that can read and write objects only in the cache bucket or prefix. It should not carry permissions for other buckets, instance administration or the runner's own provisioning. Store the secret in your secrets manager and deliver it to the runner manager through the mechanism you already use for runner configuration. Done means a test upload and download succeed with the new credential, and a request to a different bucket is denied. Record the policy so auditors can trace which identity reads and writes cache objects.
Add the cache block to the runner configuration
The runner administrator adds a
[runners.cache]section to theconfig.tomlfile on the runner manager, setting the cache type to S3, the bucket name, the endpoint and region, and the access key references, then restarts or reloads the runner service. For autoscaled fleets, confirm that the cache settings apply to the runner section that launches the instances. Done means the runner manager starts cleanly, the log shows no cache connection errors on startup, and a quick job that writes a test file to the cache completes successfully.Set Shared and Path for each runner
The administrator decides whether runners share one cache or split it, then sets
SharedandPathin the cache block accordingly. WithSharedenabled, runners with the same cache configuration can read each other's cache entries, which suits a common dependency set. ThePathvalue sets the prefix under which entries are stored, so separate runner groups can keep distinct caches in one bucket. Done means the team has written down which runners share a cache and which are separated by path, and the config reflects that decision. Confirm the exact behavior of these keys against the Runner documentation for your version before relying on them.Define the cache key in the pipeline
A developer or pipeline maintainer adds a
cache:block to a test project's.gitlab-ci.ymlfile, choosing a key such as the branch name or a lock file hash, and listing the directories to store, such as a package manager's download folder. The key determines when a cache is reused and when it is rebuilt, so a poorly chosen key either misses every time or serves stale dependencies. Done means the job log shows the cache key and the paths being archived, and the team has agreed which key scheme the project will use.Run two jobs on different instances to verify a hit
The team runs the test pipeline twice, forcing the second job onto a different autoscaled instance, which you can confirm by comparing the instance or machine identifiers in the job logs. The first job writes the cache and the second job should report that it restored the cache from the bucket. Done means the second job's log shows a cache restore that completes, and the object appears in the bucket under the expected prefix. If the second job reports a miss, check the key, the Shared and Path values, and the credential's permissions in that order.
Securing the cache bucket, credentials and cache contents
A shared cache sits between every job on the runner fleet and the bucket it writes to, so one weak setting can expose dependencies across projects. The cards below cover the bucket, the credentials, the cache contents and the audit trail, and tie each one to the runner isolation guidance in GitLab's hardening documentation. Roll these controls out gradually and test each one on a non-production runner first.
Bucket access policy and encryption
Give the cache bucket a policy that allows only the runner manager and the autoscaled instances that need it to read and write, and deny public access. Confirm that server-side encryption is enabled with keys your team controls, and check that the bucket is not reachable from networks outside the runner subnets. Ask your storage provider how the policy is evaluated, because a permissive default on a parent account can override a narrow bucket policy.
Credential scope and rotation
The access key the runners use for the cache should be a dedicated identity limited to the cache bucket and prefix, not a general administrator key. Define a rotation schedule and document who performs it and how runners pick up the new key without a fleet-wide outage. Review the key's permissions after each change to the autoscaling group, since new instance roles often inherit broader access than intended.
Cache poisoning between projects
When Shared is enabled, jobs from different projects on the same runner can read and overwrite the same cache keys, so a malicious merge request can plant a dependency that a protected branch later restores. Decide which projects may share a cache and which need separate Path values or dedicated runners. Treat any runner that serves untrusted projects as a boundary that the shared cache should not cross.
Untrusted jobs writing to shared paths
Jobs from forks or external contributors should not run on runners that write to the cache used by privileged pipelines. Check the runner tags, protected-branch restrictions and the cache:key values in each project's pipeline file. Pipelines that build a key from a branch name or a user-controlled variable let an untrusted job choose which cache entry it writes, so restrict those variables before enabling the cache for shared runners.
Retention and deletion of cache objects
Cache objects accumulate across branches and keys, and old entries can keep dependency versions that a team has since removed from its lock files. Set a lifecycle rule on the bucket that expires objects after a period your security team approves, and confirm the GitLab Runner cache settings do not depend on objects the lifecycle rule removes. Test a restore after deleting a sample of objects so you know a missing entry produces a clean cache miss rather than a failed job.
Logging of bucket access
Enable access logging or the equivalent audit trail on the cache bucket so you can see which runner identity read or wrote which prefix and when. Route those logs to the same system that holds your runner and GitLab logs, and set alerts for unusual volume, writes from unexpected networks or deletes outside the lifecycle rule. Confirm with your provider how long audit logs are retained, because the investigation window depends on it.
Questions teams ask before configuring a runner cache
Do autoscaled runners need a shared cache at all?
Not always, but without one, every job on a fresh instance starts with an empty cache and downloads its dependencies again. A shared cache in S3-compatible object storage lets a job on a newly created runner restore packages, build outputs or tool directories that an earlier job saved. The question to answer first is how much your pipelines spend on dependency downloads and how often runners are replaced. If jobs are short, dependencies are small, or runners are long-lived, a local cache may be enough. If instances are created and destroyed often and builds pull large dependency trees, the shared cache usually pays for its setup. Measure job durations before and after a pilot rather than assuming the gain.
What drives the effort of a runner cache setup?
Four factors shape the work most: the storage platform, how credentials are issued and stored, the number of teams sharing the fleet, and the executors in use. An S3-compatible service you already run, such as Ceph or a cloud provider's object store, needs different endpoint, TLS and bucket policy work than a new deployment. Credentials that must be scoped per team or rotated add design and testing. Shared caches across many projects need a clear key and path policy so one team cannot overwrite another's data. Docker Autoscaler, the Kubernetes executor and the Instance executor each pass cache settings differently, so the configuration and verification differ by executor.
Can our own team configure the runner cache without a partner?
Yes, a team with object storage experience, access to the runner host and someone who can edit the runner configuration can usually complete the setup. The steps are documented in the GitLab Runner documentation, and the configuration keys are few. The work that often takes longer than expected is not the configuration itself but the decisions around it: bucket layout, retention, which projects share a cache, how credentials are issued and who owns changes afterward. A partner is most useful when those decisions must be made across several teams, when the storage platform is unfamiliar, or when the cache must be proven to work in an air-gapped or regulated environment. Compare the internal estimate with a written scope before deciding.
How long does it take to verify a shared cache works?
Verification takes less time than design, but it should cover more than one successful job. A reasonable check runs the same pipeline on two different runner instances, confirms that the second job restores what the first saved, and confirms that a job in another project cannot read or overwrite the cache it should not reach. It also checks behavior when a runner is terminated mid-job and when credentials are invalid. The duration depends on how many executors, runner groups and teams you test, and on how quickly your pipelines run. Ask any partner to state the verification steps and the test environment in the written scope, and confirm the expected duration with them.
What do we need to provide for the cache work?
The work needs four things from your side: access to the bucket or create rights for it, network routes from runner hosts to the object storage endpoint, a set of credentials with the narrowest permissions the cache requires, and a named owner for the runner configuration. The owner approves changes to config.toml, reviews which projects use which cache settings and handles incidents. Network needs often surprise teams, especially where runners sit in separate VPCs or segments with restricted egress. Confirm these items early, because waiting on bucket access or firewall changes usually delays a pilot more than the configuration does.
What recurs after go-live for the runner cache?
Three recurring tasks matter most. Credentials for the object store need scheduled rotation, and the rotation process must update every runner configuration that uses them. Runner versions need to stay aligned with your GitLab version, which the GitLab Runner documentation advises for both major and minor releases, so the cache configuration must be retested after runner upgrades. Storage costs recur with the volume of cached data and any retention rules you apply. Plan who performs each task, how often they review cache size, and how a broken cache is detected, since a failing cache often slows jobs silently instead of failing them.
How is the cost of a runner cache set?
The cost has two parts: one-time work and recurring infrastructure. One-time work covers design, configuration, testing and documentation, and is priced by the scope agreed in writing. Recurring costs include storage and data transfer charged by your object storage provider, compute for the runners, and the internal time to rotate credentials and upgrade runners. DeployOpen does not publish a price list, so a quote follows from the written scope. When requesting one, ask for the storage platform assumptions, the number of executors and teams covered, the verification steps, the handover of documentation, and which items are excluded, such as storage capacity planning or runner fleet changes.
Should different teams share one cache or split them?
Share a cache when projects build the same dependencies and the trust level between them is similar. Split caches when teams have different trust levels, different dependency versions that could conflict, or compliance needs to separate their data. The shared and path settings in the runner cache configuration control this trade-off: a cache shared across projects reduces downloads, while separate caches limit what one project can read or overwrite. Decide this with the team that owns the runner fleet, document the policy, and test a cross-project read before you roll it out to every pipeline.
Operating the cache after go-live
Once the shared cache is live, the work shifts to keeping the bucket predictable and the Runner fleet in step with GitLab. Start by watching bucket growth over time, since every distinct cache key written by a job adds objects that nobody deletes automatically. Set object lifecycle rules on the bucket so stale cache objects expire on a schedule your team agrees, and confirm with your storage provider how lifecycle expiration interacts with versioning and with the cache paths your jobs write. Review the bucket size and object count at a regular interval, and record the baseline so a sudden jump is visible before it becomes a cost problem.
Storage credentials need a rotation procedure before the first rotation is due. Create the access key for the Runner manager with the narrowest bucket policy that still allows cache reads and writes, then plan a dual-key window so the manager can switch to a new key without failing jobs mid-run. Update the [[runners.cache.s3]] settings in config.toml on every manager, and confirm the change with a test job that writes and restores a cache. Keep Runner versions aligned with GitLab during each upgrade, following the GitLab maintenance policy for the release cadence and the GitLab upgrade stops, so cache behavior is tested against the Runner version you actually operate.
Cache misses are the quiet failure mode. After any change to cache:key, cache:paths or the Runner Shared and Path settings, run the affected pipeline on two fresh instances and check the job log for restore and miss messages, since a changed key silently makes every job start cold. Keep a short register that lists which projects, groups and runner tags share which caches, and update it whenever a team changes its key. That register is the document an auditor or new engineer will need to explain why two teams can or cannot read each other's dependencies, and it is far easier to keep current than to reconstruct later.
How DeployOpen scopes and delivers the runner cache
DeployOpen plans, builds, documents and supports self-hosted platforms inside the client's own cloud accounts, data centers or air-gapped networks. For the runner cache, that means agreeing the bucket layout, Shared and Path settings and access model in writing before any configuration changes.
How we scope
- Written scope lists the runner fleets, executors and cache buckets that the cache work will cover.
- Scope records which projects share caches and which must stay isolated, using Shared and Path settings.
- Scope names the S3-compatible storage service, its access policy and who owns credentials and retention.
How we build
- Build sequence configures the cache in a test runner fleet first, then moves the same settings to production.
- Tests confirm that a job on a fresh autoscaled instance restores dependencies from the shared cache.
- Access tests check that cache credentials cannot read buckets or paths outside the agreed scope.
How we hand over
- Handover documents the config.toml cache block, bucket policy, key naming and retention settings used.
- Runbook explains how to rotate cache credentials and how to clear or inspect cache objects safely.
- Operators receive a verification checklist to repeat after runner upgrades or changes to the storage service.
How we support
- Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
- Ongoing support, upgrades and continuous testing for the cache are available as a retainer.
- Runner and GitLab version alignment is checked when the cache configuration is revisited during upgrades.
