How Prometheus monitoring fits a self-managed GitLab instance
A platform team that runs GitLab in its own infrastructure needs to see saturation before developers do, because a slow Gitaly node or a backed-up Sidekiq queue shows up first as delayed pushes and pipelines that sit in pending. Prometheus is the metrics layer that makes those trends visible over time, and Grafana is the view your team reads during an incident or a capacity review. GitLab's bundled exporters expose metrics for its components, and Prometheus pulls them over HTTP from each target on a schedule, so the GitLab node, the Gitaly hosts and the database and Redis tiers each need a scrape job that your team defines and secures. Grafana then queries Prometheus with PromQL and draws the dashboards your operators, not only your administrators, will use.
Lighter health checks are often enough for a small instance with a single application node. The readiness and liveness endpoints tell a load balancer whether a node is up, but they do not tell you how close queues, storage latency or connection pools are to their limits. A team should move to a full Prometheus and Grafana setup when it runs multiple nodes, when CI load has grown enough that runner queues matter, or when it must show service levels to the business. Confirm your reference architecture size with the GitLab reference architectures documentation before you choose the scope, since the published architectures for larger user counts already split Gitaly, PostgreSQL and Redis onto separate hosts.
Prometheus is licensed under Apache-2.0 and has no paid enterprise edition, so the cost of this setup sits in three places: the infrastructure that runs the Prometheus and Grafana servers, the storage that holds metrics beyond the default 15-day retention, and the people who build, secure, upgrade and maintain the stack. Grafana is a different case because its licensing and enterprise features differ by edition, so confirm which Grafana features your team needs before you plan its authentication and high availability. A partner quote should separate the one-time build, such as scrape configuration, dashboards and alert rules, from the recurring work of upgrades, alert tuning and storage management.
Which monitoring layout fits a single node, a multi-node reference architecture or Kubernetes
A single-node GitLab installed with the Linux package bundles PostgreSQL, Redis, Sidekiq and Gitaly on one host, so one machine exposes most of the metrics endpoints you need. Prometheus can run on a separate monitoring host or on the same server, but a separate host is the safer choice because a GitLab outage should not also remove your visibility into it. The scrape configuration is a static file on the Prometheus host that lists the GitLab host and the exporter ports enabled in gitlab.rb. Retention is driven by disk: Prometheus stores data locally in two-hour blocks, and its default retention is 15 days, which you can change by time or size.
- Targets: the GitLab host's bundled exporter endpoints, enabled through the Linux package configuration.
- Scrape config: a static job on the Prometheus host, typically a single
prometheus.ymlfile. - Storage: one local volume sized for your retention window and series count.
- Alertmanager: a single instance is usually enough at this size, with Prometheus pointing at it directly.
- Failure domain: if the host or its disk fails, metric history is lost unless you ship data to external storage.
Before you rely on this layout, confirm the exact exporter endpoints and ports for your installation method in the GitLab documentation, and confirm how metrics endpoints are authenticated.
A reference architecture splits GitLab across several hosts, typically with Gitaly, PostgreSQL, Redis and application nodes separated. Prometheus must reach each host and each tier, and the tiers fail independently, so you need one scrape job per role rather than one job for the whole installation. Scrape configuration usually moves into a version-controlled file managed with the same process as the GitLab configuration. Storage and retention scale with the number of series and the number of hosts, so plan disk for the larger total before you raise retention.
- Targets: separate endpoints on the application, Gitaly, PostgreSQL and Redis hosts, with labels that identify each role.
- Scrape config: several jobs grouped by tier, kept in version control and reviewed like other infrastructure code.
- Storage: a larger local volume or a plan for remote write to external long-term storage, since local storage is not clustered or replicated.
- High availability: the reference architectures for larger user counts use Gitaly Cluster, Patroni and Redis Sentinel, so monitor the tier-level health signals of each component, not only the application.
- Alerting: Alertmanager can run as a cluster, and Prometheus should point at every instance rather than a load balancer.
Confirm with your architecture design which exporter endpoints exist on each host, because the layout depends on how the components are placed.
A Helm chart deployment runs GitLab's stateless components on Kubernetes, and the GitLab reference architectures state that stateful components such as PostgreSQL and Redis are not supported inside Kubernetes. Your Prometheus target discovery therefore covers two environments: the cluster for stateless services and the VMs or managed services that hold state. Kubernetes service discovery can find pods that expose metrics, while the stateful tiers are usually scraped as static targets. The scrape configuration is typically managed through the cluster's monitoring tooling, and storage is a persistent volume whose size depends on series count and retention.
- Targets: pods and services in the GitLab namespace found through Kubernetes service discovery, plus static targets for the stateful hosts.
- Scrape config: defined as Kubernetes resources or Prometheus configuration managed alongside the chart values.
- Storage: a persistent volume with a retention window set in Prometheus, plus a plan for backups of the volume.
- Network access: Prometheus needs a route to the pod network and to the stateful hosts, which often requires firewall and policy changes.
- Log handling: stdout logs from the chart are a separate concern from metrics, so check the log forwarding approach at the same time.
Confirm the exporter endpoints and service names for your chart version with the chart documentation before you write scrape rules.
The right layout follows from where GitLab runs, not from the monitoring tool. A single node needs one scrape job and one storage volume. A reference architecture needs role-based jobs, version-controlled configuration and a deliberate storage and remote write plan. A Helm deployment needs discovery across the cluster and static targets for the stateful services that stay outside Kubernetes. Teams often underestimate how much the storage plan changes between these layouts, because local Prometheus storage is single-node by design and must be managed like any other single-node database.
- Count the targets: list every host or pod that exposes metrics before choosing a job structure.
- Separate the stateful tiers: PostgreSQL and Redis need their own jobs and alerts, even when they share a host with other services.
- Size storage from series and retention: estimate from your installation's actual series count, not from a default.
- Plan for Prometheus failure: decide whether a second Prometheus or external storage is needed for your recovery targets.
- Document the configuration source: record where each scrape job is defined and who approves changes.
Before you commit to a layout, confirm the exporter endpoints, authentication requirements and network paths with your own GitLab installation method.
Prometheus, Alertmanager and Grafana options and when each fits
This table compares the main setup choices for monitoring self-managed GitLab with Prometheus, Alertmanager and Grafana, and the trade-off each one carries. Read each row against your team's retention needs, alert ownership and dashboard change process. Prometheus favors reliability over precision, so the choices below are about keeping alerts dependable rather than perfectly exact.
| Decision | Option | Trade-off | When it fits |
|---|---|---|---|
| Prometheus storage | Local storage with default 15-day retention | Simple to run, but local storage is not clustered or replicated and must be managed like a single-node database | Small teams with short trend needs and a tested disk backup or rebuild plan |
| Prometheus storage | Remote write to long-term storage | Longer history and separation from the scraping host, with an extra system to size, secure and operate | Capacity reviews, audits or incident analysis that need data beyond local retention |
| Alertmanager | Single instance | One component to run, but a failure of that instance silences all notifications | Single-site instances where a short notification gap is tolerable and someone watches the stack |
| Alertmanager | Clustered using its cluster flags | More moving parts, but Prometheus points at every Alertmanager instance rather than a load balancer | Production GitLab where paging must survive a single Alertmanager failure |
| Grafana configuration database | SQLite (default) | Zero setup, but the Grafana docs say it does not scale for production or high availability | Evaluation or lab environments only |
| Grafana configuration database | PostgreSQL or MySQL | Extra database to back up and patch, but required for production and high availability | Production dashboards shared by several teams or run as two or more Grafana servers |
| Dashboards | Manual edits in the Grafana UI | Fast to start, but changes are hard to review, and rebuilding a lost instance means recreating them by hand | Short-lived exploration or a single operator who keeps notes |
| Dashboards | Provisioned or Git-managed as code | Needs a review process and file layout, but dashboards and data sources can be rebuilt from version control | Teams that must show change history or redeploy monitoring across environments |
Setting up GitLab exporter scraping, alert rules and dashboards
This sequence turns GitLab's bundled metrics endpoints into a monitored system: scraped by Prometheus, alerted through Alertmanager and visualized in Grafana, with each tier confirmed before the next step begins. The output is a working pipeline that has been tested with a real alert.
Confirm the metrics endpoint authentication setting
Before any scrape job is written, the platform engineer confirms how the GitLab metrics endpoints are exposed on the chosen installation method, and whether the instance requires authentication to read them. Check the monitoring settings in the admin area and the relevant configuration file on your installation, such as gitlab.rb for Linux package installs, and record which exporters are enabled. The step is done when you have a written list of the endpoints you intend to scrape, their ports and whether each one requires a token. Also confirm the component layout on your installation method, because Gitaly, PostgreSQL, Redis and Sidekiq expose metrics on different hosts in a multi-node deployment.
Restrict monitoring endpoints to Prometheus hosts
The GitLab administrator allows access to the monitoring endpoints only from the IP addresses of the Prometheus servers, and blocks other sources at the host firewall or network policy. Where the endpoints support it, require an authentication token and store that token in your secrets manager rather than in the Prometheus configuration file in plain text. Test from a host outside the allowlist to confirm the request is refused, and from a Prometheus host to confirm the request succeeds. The step is complete when the allowed host returns metrics in the Prometheus text format and the disallowed host does not.
Define scrape jobs for each tier
The monitoring engineer writes separate scrape jobs for the application, Gitaly, PostgreSQL and Redis tiers, rather than one job for everything. Separate jobs keep labels readable, let each tier have its own scrape interval and authentication, and make it clear which exporter failed when a target drops out. Use static configuration or service discovery that matches how your hosts are named. Confirm that Prometheus is pulling from every expected target by reviewing the targets page, and that the number of targets per job matches the number of nodes in your reference architecture.
Load alert rules for each tier
The engineer loads alert rule files into Prometheus that cover availability of each scrape target, and the saturation and error signals each tier exposes on your installation. Keep rules grouped by tier so that a Gitaly alert does not get lost among PostgreSQL alerts. Validate the rule files with the Prometheus tooling before loading, then reload the configuration. The step is complete when the rules page shows every group as loaded without evaluation errors, and each rule has an owner and a documented response in your runbook.
Route alerts through Alertmanager
The engineer points Prometheus at every Alertmanager instance rather than at a load balancer in front of them, then configures grouping, inhibition and silences to suit your on-call rotation. If you run Alertmanager as a cluster for high availability, enable its cluster flags and confirm the peers can see each other. Set receivers for your paging and ticketing tools and keep notification details in secrets storage. The step is done when Alertmanager lists all configured Prometheus connections as active and the routing tree sends each alert to the intended receiver.
Provision Grafana data sources and dashboards
The platform engineer provisions Grafana with the Prometheus data source and the GitLab dashboards as files loaded at startup, instead of building them by hand in the user interface. Store these files in version control so changes are reviewed like other configuration. For production, confirm that Grafana uses a MySQL or PostgreSQL database rather than the default SQLite, and check the authentication options for your edition before deciding how operators sign in. The step is complete when the data source test succeeds and the dashboards load from provisioned files after a Grafana restart.
Verify targets are up and fire a test alert
The operator checks that every scrape target reports as up on the Prometheus targets page, then triggers a controlled test to prove the alert path works end to end. A safe method is to temporarily lower a threshold on a non-production instance, or to stop a non-critical exporter for a short period under a change record. Confirm the alert moves from pending to firing in Prometheus, appears in Alertmanager, reaches the test receiver and shows on the Grafana dashboard. Record the test in your change log, then restore the original thresholds and exporter state. The step is complete when the full path has been observed and documented.
Which GitLab metrics are worth alerting on
Alert on the signals that tell a user something is wrong now, and treat everything else as a dashboard panel. For self-managed GitLab, that means four groups: whether each scrape target is up, request latency and error rates from the web and API tiers, Sidekiq queue growth, and disk and resource saturation on Gitaly storage. A target that stops answering Prometheus is the first alert to build, because every other alert depends on Prometheus seeing the instance at all. Keep the initial rule set small so that each page leads to a clear action by the on-call engineer.
Latency and error rates should be alerted on as a sustained condition, not a single bad sample. Request duration and HTTP status metrics from the web and API endpoints show what developers feel when they push, review or run pipelines. Sidekiq queue growth matters because a backlog delays merge request notifications, pipeline status updates and mirroring without producing any visible error page. For Gitaly, watch free space and I/O saturation on the volumes that hold repositories, since a full repository volume fails writes in ways users report as broken pushes. Confirm the exact metric names and their meaning in GitLab's current monitoring documentation before writing rules, because metric names change across releases.
Thresholds must come from this instance's own baseline. Collect at least several weeks of normal data for request rates, queue depth and disk growth, then set warning and critical levels above normal peaks, including your heaviest CI hours. Prometheus values reliability over precision, so pick alert windows long enough to avoid flapping. Record each threshold with the reason for it, and review them after each GitLab upgrade, since new releases can change the metrics you depend on. This page does not supply numeric threshold values, so any figure you adopt should be justified from your own measurements and the vendor's current guidance.
How to monitor Gitaly, PostgreSQL and Redis as separate tiers
Each stateful tier behind self-managed GitLab fails in its own way, so each needs its own scrape job, dashboard and alert set. The cards below cover the tiers in a high availability layout, plus the monitoring stack itself, since a monitor that fails silently leaves the other tiers unwatched.
Gitaly node health and storage latency
Gitaly serves Git reads and writes, and in high availability layouts it runs as Gitaly Cluster behind Praefect. Monitor each Gitaly node as its own target so a single slow disk shows up as one node's latency rather than a vague slowdown across all repositories. Watch request latency and error rates per node, and disk latency on the volumes holding repository data. Confirm with your reference architecture which metrics your Gitaly version exposes and how Praefect health is reported.
PostgreSQL replication and connection pressure
The database tier carries replication and connection load that rarely appears in application dashboards. Where Patroni manages failover, track which node is leader, replication lag on each replica and whether failover events have occurred. Track active connections against the configured maximum, since pooled connections can hide saturation until requests start queuing. Confirm with your database team which exporter and Patroni endpoints your build exposes before writing alert rules against them.
Redis Sentinel state
Redis holds queues and cache data for the application, and in the high availability reference design it is managed by Sentinel. Monitor the Sentinel view of the primary, the number of Sentinels that agree on it, and any failover in progress. A Redis that is up but has lost its primary is a common partial failure, so alert on role changes and on quorum loss, not only on the process being down. Confirm the exporter and Sentinel metrics your Redis version provides.
Sidekiq and application tier
Sidekiq runs background jobs such as CI pipeline processing and notifications, and the Puma web workers serve requests. A growing Sidekiq queue often shows up as delayed pipelines long before any host looks unhealthy, so track queue size and job latency per queue. Pair these with request latency and error rates from the web tier. Confirm which application metrics your installed GitLab version exposes and whether they require enabling in gitlab.rb.
Prometheus self-monitoring
A metrics stack that stops collecting gives no warning by itself. Prometheus keeps a single-node local store that is not clustered or replicated, so monitor its own disk usage, scrape failures and rule evaluation errors. Set retention deliberately, since the default is 15 days, and decide whether long-term storage through remote write is needed. Alertmanager should receive alerts from every instance, not through a load balancer, and a dead-man alert confirms the path works end to end.
Grafana high availability
Grafana configuration lives in a database, so production and high availability setups need MySQL or PostgreSQL rather than SQLite. Two or more active-active servers behind a load balancer share that database, but alert notifications need extra configuration to avoid duplicates, and Grafana Live has limitations in this mode. Confirm how dashboards and alert rules are provisioned from code, so a rebuilt node matches the others. Check which authentication methods your edition supports before planning SSO.
What monitoring GitLab with Prometheus costs and requires
What does it cost to monitor self-managed GitLab with Prometheus?
Prometheus has no license fee, because it is licensed under Apache-2.0 and has no paid enterprise edition, so the cost comes from infrastructure, storage, implementation and the people who run it. The main recurring costs are the hosts for Prometheus, Alertmanager and Grafana, disk for metric history, and staff time for upgrades, alert tuning and on-call. The main one-time cost is building the scrape configuration, dashboards and alert rules. Ask a partner to quote these separately: the build, the hosts and storage you will provision, and the ongoing support, if you want it. Any GitLab license tier is a separate decision and is not required for the metrics themselves.
How long does a Prometheus monitoring setup for GitLab take?
Duration depends mostly on how many GitLab components you expose, whether you run a single Linux package instance or a multi-node reference architecture, and how much of the Gitaly, PostgreSQL and Redis tiers you want monitored separately. Other drivers are whether Prometheus and Grafana already run in your environment, how your network rules allow scraping, and how many alerts must be reviewed with the people who will receive them. Ask any partner for a written timeline that lists the scrape targets, the dashboards and the alert rules as separate deliverables, so you can see what each phase includes.
Can an internal team set up GitLab monitoring alone?
Often yes, if the team already runs Linux services, understands PromQL and has someone who can own alert rules after go-live. The work is not large in concept, but it needs skills across GitLab's exporters, Prometheus service discovery and storage, Alertmanager routing and Grafana administration. Gaps usually appear in alert tuning, in multi-node GitLab topology and in keeping dashboards aligned across GitLab upgrades. Assess your team against those skills before deciding, and ask a partner to take on whatever you cannot staff in time.
What does monitoring GitLab require from our infrastructure and teams?
You need hosts for Prometheus and Alertmanager, network rules that let Prometheus reach GitLab's metrics endpoints, and a clear owner for alerts that page someone. Grafana also needs a database for production use, because its docs say SQLite does not scale for production and that MySQL or PostgreSQL is required for production and high availability. Plan for:
- Hosts for Prometheus, Alertmanager and Grafana, separate from GitLab where possible
- Firewall rules allowing scrapes from the monitoring hosts to GitLab and its exporters
- An on-call rota that receives alerts and a runbook for each alert
- Access control on the metrics endpoints, since Grafana's hardening guidance recommends requiring authentication for its metrics endpoint
How should we size storage and retention for GitLab metrics?
Size storage from the number of series and the retention you choose, not from a fixed rule. Prometheus stores data locally in two-hour blocks with a write-ahead log, and its default retention is 15 days, configurable by time or size. Local storage is not clustered or replicated, so it must be backed up and managed like any other single-node database. For longer history, Prometheus supports remote write and remote read to external long-term storage. Ask a partner to measure series counts in your environment before quoting disk, and to state the retention period and backup method in writing.
Is Grafana Cloud an option for monitoring GitLab?
Yes, Grafana Cloud is an option if your policy allows metrics to leave your infrastructure, which many regulated organizations will not accept for GitLab data. Grafana's pricing page lists a free tier, a Pro tier with a 19 US dollar monthly platform fee plus usage, and an Enterprise tier with a minimum commitment of 25,000 US dollars per year. Self-hosting Grafana avoids that dependency, but then you pay for the hosts and database yourself. Confirm the data residency, retention and access terms with the vendor before choosing, and ask a partner to compare both models against your policy.
How does monitoring change across GitLab upgrades?
Monitoring must be checked at each GitLab upgrade, because metric names, exporter settings and dashboard panels can change between releases. GitLab ships a minor release on the third Thursday of each month, patch releases twice a month, and a major release yearly, and it requires upgrades to pass through required upgral stops such as x.2, x.5, x.8 and x.11 since GitLab 17.5. Each stop is a chance to re-run your dashboards and alert rules against the new version. Also keep Prometheus and Grafana on supported lines: Prometheus long-term support releases get one year of bug fixes, and Grafana minor releases are supported for 9 months. Ask a partner to include monitoring checks in the upgrade plan.
What should we ask a partner for in a monitoring quote?
Ask for a written scope that names each scrape target, including GitLab's bundled exporters and the Gitaly, PostgreSQL and Redis tiers if you run them separately. Ask for the list of dashboards and alert rules, the Alertmanager routing design, the storage and retention plan with backup method, and the hosting model for Prometheus and Grafana. Also ask who maintains alerts after handover, how upgrades are handled, and what the quote excludes. DeployOpen works on fixed scope agreed in writing before work starts, and ongoing support is available as a retainer, so the quote should separate the one-time build from any recurring support.
How DeployOpen scopes and delivers GitLab monitoring
DeployOpen scopes monitoring work in writing before any build starts. For a Prometheus and Grafana setup around self-managed GitLab, the written scope names which GitLab exporters are scraped, which Prometheus and Grafana instances are in place, where they run, which hosts are monitored (application nodes, Gitaly, PostgreSQL, Redis and Runner hosts), and which alert receivers are connected. The scope also lists the deliverables: the configured scrape jobs, the dashboards, the alert rules, and the runbook for each alert. Most engagements of this kind are fixed-scope, so the agreed deliverables and timelines are fixed in the same document, and changes to the scope are handled through a written change to that document.
The work happens inside your infrastructure. DeployOpen builds the monitoring stack in your cloud accounts, data centers or air-gapped network, so metrics, dashboards and alert history stay under your control. Prometheus storage is local and not clustered or replicated, so the scope should state how that single-node store is backed up or paired with long-term storage, and who owns Alertmanager high availability. Grafana's own configuration database, authentication settings and provisioning files are also in scope, so that dashboards can be rebuilt from code rather than recreated by hand. Each alert gets a runbook that tells the on-call engineer what the signal means and what to check first, and those runbooks are handed over with the dashboards.
Quotes follow from the written scope, and DeployOpen does not publish a price list. Ask each vendor for the fixed-scope deliverables, the list of assumptions about your environment, and the items that are excluded, such as work on the GitLab instance itself or Runner fleet changes. After handover, many teams want continuing help with Prometheus and Grafana version changes, new exporters after a GitLab upgrade, and periodic alert review. DeployOpen offers these as optional retainers covering ongoing upgrades and continuous testing, and these are priced separately from the initial build.
