Monitoring Graylog with Prometheus and Grafana: when external metrics are worth the effort
If your Graylog cluster stalls and the first signal is a user asking why a dashboard is empty or a security team noticing missing events, you are monitoring the wrong layer. Scraping Graylog's metrics with Prometheus and building dashboards and alerts in Grafana gives you visibility from outside the Graylog UI, so a growing journal or a slowing output can page someone before ingestion stops. Prometheus is an *open-source* monitoring and alerting toolkit licensed under Apache-2.0. It pulls metrics over HTTP from targets it discovers or is configured to scrape, and it is queried with PromQL. Grafana is a visualization and alerting front end licensed under AGPLv3 since Grafana Labs relicensed it in April 2021, and it reads from Prometheus and other data sources.
Neither tool carries a paid edition of its own, so the cost sits in infrastructure and people. You pay for the hosts that run Prometheus, its local storage, the Grafana server and its database, the time to write scrape configuration and alert rules, and the on-call effort that follows. Graylog Open is licensed under SSPL-1.0, which is not an OSI-approved license, and that does not block using external monitoring of it. Before you commit, confirm with your vendor or your own test cluster that the Prometheus exporter and the gl_ metric names you plan to alert on exist in the exact Graylog version you run, and whether any metric depends on an Enterprise or Security license. Ask for that answer in writing as part of the scope.
The decision is whether an external stack is worth running alongside the Graylog UI and its own system indicators. It usually is when several clusters need one view, when the team on call is not the team that operates Graylog, or when stalls have already caused lost messages. It is harder to justify for a single small node where journal and buffer checks in the UI are enough. Plan the monitoring stack as a service in its own right, with its own backups, upgrade path and access controls, because a monitoring system that fails silently during the incident it was built for is worse than none.
Configuration options and when each fits
This table compares the main choices for exposing Graylog metrics, scraping them, storing them and building dashboards. Read each row for the situation it suits and the running cost it adds, then pick the combination that matches your network rules and on-call model. Most teams start with a loopback or management-network bind, one Prometheus server and provisioned dashboards, and add redundancy only when a stalled scrape would be unacceptable.
| Choice | Option | When it fits | What it costs to run |
|---|---|---|---|
| Exporter bind address | Loopback (127.0.0.1) with a local agent or proxy | Single-host or Docker setups where the scraper runs on the same machine | Extra agent to maintain, but no new network exposure |
| Exporter bind address | Management network interface | Most production clusters, where Prometheus reaches each node over a separate monitoring or management VLAN | Firewall rules and routing between the monitoring subnet and every Graylog node |
| Exporter bind address | All interfaces (0.0.0.0) | Only when no restricted network exists and access control is enforced upstream | Metrics reachable from any network the host touches, so it needs authentication or strict filtering |
| Access control | Authenticated metrics endpoint | Shared or segmented networks, or where auditors expect access to telemetry to be controlled | Credential handling in the scrape config and rotation work |
| Access control | Network-restricted access only | Dedicated monitoring segments with tightly scoped firewall rules and a small number of scrapers | Lower setup effort, but every change to scraper IPs needs a firewall update |
| Prometheus topology | Single Prometheus server | Small or medium clusters where short gaps in data during a restart are acceptable | One host to patch and monitor, with a single point of failure for alerts |
| Prometheus topology | Redundant pair scraping the same targets | Teams that cannot miss an ingestion stall alert during maintenance or host failure | Two sets of storage and retention, plus deduplication handled in Alertmanager |
| Storage | Local time series storage | Dashboards and alerts that need only the default retention window or a configured local size limit | Disk sized for retention, with backups of the data directory if history matters |
| Storage | Remote write to long-term storage | Trend analysis, capacity planning or retention beyond what the local disk holds | Second storage system to operate, network bandwidth for samples, and licensing of that system |
| Dashboards | File-provisioned Grafana dashboards | Teams that want reviewed, versioned dashboards deployed the same way in every environment | Repository workflow and review effort for each dashboard change |
| Dashboards | UI-built Grafana dashboards | Early exploration or one-off investigations by individual operators | Changes can drift between environments and are harder to audit or restore |
Setting up the exporter, scrape job and first dashboard
This sequence takes a running Graylog cluster from an enabled metrics endpoint to a Grafana dashboard and alert rules that show journal and buffer health before ingestion stalls. Each step ends with a check so the team can confirm one link in the chain before moving to the next. The work spans Graylog, Prometheus, Grafana and Alertmanager, and each step names who owns it.
Confirm the Graylog version and exporter option
The Graylog administrator reads the documentation for the exact release running in production before changing anything. The current Graylog documentation covers version 7.2, so confirm that your running version matches the guide you follow. Write down the exporter setting name, its default port, and whether your version offers authentication on the endpoint. Check: the version, setting and port are recorded, and the Graylog server logs show the exporter starting after a restart.
Bind the metrics endpoint to a restricted interface
The Graylog administrator sets the exporter to listen on an internal management interface or one private address. The host firewall then allows the port only from the Prometheus server. If your version does not authenticate requests to the endpoint, the firewall rule carries that control, so record it as a dependency. Check: a request from the Prometheus host returns plain-text metrics, and the same request from a user workstation fails to connect.
Add a scrape job for each Graylog node
The Prometheus administrator adds a scrape job with one target for each Graylog server node, using the host name or private address and the exporter port from the first step. Give the job a clear name such as graylog so PromQL queries and dashboards can filter on it. Validate the configuration file before reloading Prometheus. Check: under the targets page in the Prometheus web interface, every Graylog node shows as UP, and a node you stop for a moment shows as DOWN.
Confirm gl_ series with PromQL
Run a query in the Prometheus expression browser that lists metric names beginning with gl_. Look for the families covering journal, buffers and throughput, and confirm that each node appears as its own instance label. Record the exact metric names your version exports, because dashboards and alert rules depend on them and names can change between releases. Check: each node returns series for the journal and buffer metrics, and the values change between two queries made while Graylog is ingesting.
Provision the Prometheus data source in Grafana
The Grafana administrator defines the Prometheus data source in a provisioning YAML file so it loads at startup and can be rebuilt without manual steps. The file sets the URL of the Prometheus server and the access mode your network allows. Keep the file in version control with the rest of the Grafana configuration. Check: the data source test in Grafana reports success, and the Explore view returns the gl_ metric names confirmed in the previous step.
Build the first dashboard from provisioned files
The dashboard owner builds panels for ingestion throughput, journal utilization, and process and output buffer levels, with a variable that filters by node. Export the dashboard JSON and place it in the dashboard provisioning path that your Grafana version expects, using the provisioning documentation for the file layout. Check: every panel shows data for every node after a refresh, and redeploying Grafana from its provisioning files recreates the dashboard with no manual steps.
Define alert rules on journal and buffer growth
The monitoring owner writes alert rules for journal utilization climbing toward its limit and for buffers that keep growing without draining. Set thresholds from your own ingest baseline and journal sizing, since the Graylog architecture guidance ties journal size to daily ingest. Point Prometheus at every Alertmanager instance and route the alerts to your receivers. Check: the rules page in Prometheus loads each rule without errors, and a temporarily lowered test threshold moves the alert to firing and delivers it to the receiver.
Which gl_ metrics matter for throughput, journal and buffers
Graylog exposes its internal state through the Prometheus exporter, and a handful of metric families explain nearly every ingestion stall. The cards below group them by the question each one answers, so your dashboards and alerts follow what operators need to decide. Confirm each exact name against your own /metrics output and the Graylog metrics reference for your version before writing alert rules.
Input and processing throughput
Start with how many messages enter Graylog and how many leave processing. A gap between inbound and processed rates means work is queuing somewhere, and it is the earliest sign of a stall. Chart the rates per node, since one overloaded node can hide behind a healthy cluster total. Use the metric families listed in the Graylog metrics reference for your version, and confirm the exact gl_ names in your /metrics output before building panels.
Journal utilization and uncommitted messages
The message journal is the disk buffer that holds messages before Graylog writes them to the search backend. Track journal utilization as a percentage of its configured size, and track uncommitted messages, which are messages not yet acknowledged. A journal that grows steadily points to a backend or processing problem, not a traffic spike. Alert well before the journal fills, because the journal size is a fixed limit, not an elastic one.
Process and output buffer fill
Process buffers hold messages waiting for pipeline rules, and output buffers hold messages waiting to be written. Sustained high fill in either buffer shows which stage is the bottleneck. A full process buffer with an idle output buffer points at expensive rules or regular expressions, while the reverse points at the backend. Compare fill across nodes and across time windows so you can separate a brief burst from a persistent constraint.
Failed or stalled outputs to the search backend
Count failed write attempts and watch for outputs that stop making progress while messages still arrive. These metrics separate a search backend that is slow or unreachable from a Graylog node that is simply busy. Pair output failures with the backend's own health, such as cluster status and disk pressure on Data Node or OpenSearch. Alert on a flat output rate combined with rising input, since that combination usually precedes journal growth.
Node availability and leadership
Confirm every Graylog node is up and reporting metrics, since a node that stops exporting looks like a quiet node unless you alert on its absence. In a multi-node cluster, also confirm that exactly one node is acting as leader and that follower nodes are configured with is_leader set to false. Alert when a scrape target disappears, and track the scrape status in Prometheus itself so a silent exporter is not mistaken for a healthy one.
Alerting before ingestion stalls
Alert on the direction of journal and buffer utilization, not only on the level. A Graylog journal that is filling steadily is the earliest sign that search backends, pipelines or disks are falling behind, and by the time utilization reaches its limit, messages may already be at risk. In Prometheus, write alert rules that compare the current value of the journal and process or output buffer gauges exposed under the gl_ prefix with their value some minutes earlier, and fire when the increase is sustained across several scrape intervals. Set the for duration deliberately so short bursts from a log source do not page anyone, and confirm the exact metric names against the metrics your Graylog version exposes before you write the rules, since names change between releases. Keep a separate, simpler rule for the absolute journal level so that a slow but constant climb still produces a warning.
Prometheus evaluates the rules and sends firing alerts to Alertmanager, which handles grouping, inhibition and silences. Grouping keeps a single failing Graylog node or a shared search backend outage from producing dozens of pages, and inhibition lets you suppress downstream alerts such as ingestion rate drops while a higher-level backend alert is already active. Run Alertmanager as a cluster using its --cluster flags if you need high availability, and configure Prometheus with every Alertmanager instance in its alerting section rather than placing a load balancer in front of them. Prometheus documents that local storage is not clustered, so decide early how alert history and silences are kept, and test that a silence created on one Alertmanager instance is honored after one instance is stopped.
If you also use Grafana alerting, plan for high availability separately. The Grafana high availability documentation notes that alerting needs additional configuration to avoid duplicate notifications when several Grafana servers share one database, so either keep alert evaluation on one path or confirm the HA settings in your Grafana version before relying on it for paging. Before go-live, trigger each rule deliberately on a test node, confirm the notification reaches the on-call channel, and document who owns each rule and its runbook.
Securing the metrics endpoint and the monitoring stack
A metrics endpoint, a Prometheus server and a Grafana instance each expose a different surface, and each can reveal how your Graylog cluster works. The cards below cover who can reach each component, how the endpoints authenticate, which Grafana and Prometheus settings to harden and what should never appear in a metric label.
Restrict who reaches the Graylog metrics endpoint
The Graylog Prometheus exporter listens on a bind address you choose, so it can be limited to a management interface or to the Prometheus server's address rather than every network that can reach the node. Decide which hosts scrape it and block the rest at the host firewall and network layer. Confirm in the Graylog configuration reference how the exporter's bind address and any authentication options are set for your Graylog version before you expose it beyond the local host.
Authenticate scrapes and Grafana access
Treat the metrics endpoint as an interface that can disclose cluster topology, index names and traffic volume. Where the Graylog version and your network design allow, require credentials or restrict scrapes to a dedicated network path, and put the scrape credential in a secrets store rather than in a plain scrape file. Grafana should then authenticate its own users through a method your identity team already runs, not through shared accounts.
Harden Grafana cookies and headers
Grafana's security hardening guidance recommends secure and SameSite cookies, a Content Security Policy and Trusted Types. Enable these settings in the Grafana configuration for any instance reached through a browser, and serve it only over TLS. Test dashboards and plugins after enabling a strict Content Security Policy, because some panels may need an exception.
Hide versions and require authentication for metrics
Grafana's hardening guidance also recommends hiding the version from unauthenticated users and requiring authentication for its own metrics endpoint. A visible version lets an attacker match the instance against published vulnerabilities, so make sure the setting is applied on every node in a high-availability pair. Verify from an unauthenticated client that the version and metrics routes return no detail.
Protect the Prometheus API and web UI
Prometheus does not ship with built-in user authentication for its API or web UI, so access control usually sits in front of it. Review Prometheus's security model to decide who may query, reload configuration or use the admin API, and place the server behind a reverse proxy or network boundary. Keep the administrative endpoints disabled unless a specific workflow needs them, and restrict the Alertmanager endpoint to the Prometheus server.
Keep sensitive data out of metric labels
Labels create separate time series, so a label carrying user names, client IP addresses, email addresses or message content exposes that data in every query, dashboard and retained sample. Limit labels to bounded values such as node name, input type or stream category, and check exporter and scrape configurations for relabeling rules that drop sensitive values. Review retention settings too, because stored samples keep whatever labels were recorded.
Questions teams ask before monitoring Graylog
What does it cost to monitor Graylog with Prometheus and Grafana?
Prometheus has no license fee, since it is licensed under Apache-2.0 and has no paid enterprise edition. The cost comes from four places: the servers or virtual machines that run Prometheus, Alertmanager and Grafana, the disk that holds metrics, the implementation work to configure exporters, scrape jobs, dashboards and alerts, and the people who keep it running. Setup is largely one-time, while storage, hosting, patching and alert tuning recur. Storage grows with the number of series and the retention you choose. Ask any quote to separate one-time build effort from recurring operations, and to state which hosts, retention period and alert channels are in scope.
How long does a Graylog monitoring setup take?
Duration depends on how many Graylog server nodes and backends need scraping, whether Prometheus and Grafana already exist in your environment, how many alerts and dashboards you need, and how your change process handles approvals in production. A single cluster with an existing monitoring stack is a much smaller job than a new stack across several sites. Your team should first confirm the Graylog nodes can reach the exporter port, that network rules and authentication for the metrics endpoint are agreed, and who receives alerts. DeployOpen states a timeline only in the written scope, after those facts are known.
Can our own team set this up without a partner?
Yes, a team with Linux administration, basic networking and some PromQL experience can enable the Graylog Prometheus exporter, write a scrape configuration and build a Grafana dashboard. The harder parts are choosing alert thresholds that catch a stalling journal before messages are at risk, securing the metrics endpoint, running Prometheus as a single node with its documented storage limits, and keeping pace with releases. Useful skills include Prometheus and Alertmanager operation, Grafana administration, MongoDB and OpenSearch or Data Node basics, and change management for production. A partner is most useful when the team lacks one of these skills or lacks time.
Do we need Grafana Enterprise or Grafana Cloud to monitor Graylog?
No. According to the Grafana authentication documentation, all editions support LDAP, generic OAuth, Entra ID, Okta, GitHub, GitLab, Google OAuth, auth proxy and JWT. The features reserved for Grafana Enterprise and Grafana Cloud in that documentation are SAML, team sync, active sync and SCIM provisioning. Dashboards, Prometheus data sources and alerting work in the open-source edition. Confirm the exact feature list against the current Grafana documentation before you commit, because it changes with releases. If your identity provider only supports SAML, you will need to account for that licensing cost or choose another login method.
How do we upgrade Prometheus and Grafana without breaking monitoring?
Plan upgrades around each project's release cadence. Prometheus minor releases arrive every 6 weeks, and after that window a minor release generally no longer receives bug fixes. Prometheus long-term support releases receive bug fixes for one year, and the current LTS line is 3.13, supported until July 31, 2027. Grafana publishes major releases once a year, minor releases about every other month and patch releases at least monthly, with each minor release supported for 9 months. A common approach is to run an LTS line for Prometheus, apply Grafana patches promptly, and test each upgrade in a non-production copy that uses the same provisioned dashboards and alert rules.
Should we keep monitoring data for the same period as our logs?
Usually not. Prometheus stores metrics in a local database with a default retention of 15 days, and the storage documentation says local storage is not clustered or replicated. Graylog's own log retention is set by index rotation and retention rules, which serve a different purpose. Metric retention should match how far back your team needs to compare throughput or journal trends, and longer periods can be sent through remote write to external long-term storage. Ask your provider to explain how retention, disk sizing and backups for Prometheus will be handled, and whether a remote storage system is in scope.
How does DeployOpen scope a Graylog monitoring engagement?
DeployOpen works under fixed-scope engagements, where scope, deliverables and timelines are agreed in writing before work starts. For monitoring, the written scope typically names the Graylog nodes and backends to monitor, the Prometheus and Grafana hosting model, the dashboards and alert rules to build, the notification channels and the documentation handover. Ongoing support, upgrades and testing can be added as a retainer. DeployOpen does not publish a price list, so a quote follows from the written scope. Ask for the scope document to list what is excluded, such as external long-term storage or identity integration, so you can compare it with your own estimate.
Official sources to read before you build
Read the Prometheus documentation before you write a single scrape job, because several of the choices that shape this build are stated there in plain terms. The Prometheus overview explains the pull model, service discovery and PromQL, and it says Prometheus favors reliability over precision. That trade-off matters for a monitoring stack that watches Graylog: a dropped sample is usually acceptable, but a silent gap in the journal metrics during an incident is not. The Prometheus storage page is equally important. It states that local storage is not clustered or replicated, so the team must decide how long the local retention window should be and whether to add remote write to long-term storage before the first dashboard goes live.
Next, read the Prometheus release cycle and long-term support page and the Alertmanager documentation. The release cycle page tells you that a minor release generally stops receiving bug fixes after six weeks, so the team should choose between tracking minor releases or pinning to the long-term support line and budget for that upgrade cadence. The Alertmanager page covers grouping, inhibition, silences and cluster flags, and it says Prometheus should point at every Alertmanager instance rather than a load balancer. Confirm that routing design with your on-call process before you define alert rules for ingestion stalls.
On the Grafana side, read the Grafana provisioning docs so that data sources and dashboards are stored as files in version control rather than clicked together by hand. Then review the Grafana high availability guide, which requires a shared MySQL or PostgreSQL database and extra alerting configuration to avoid duplicate notifications. Finally, work through the Grafana security hardening page, especially the advice to require authentication on the metrics endpoint. Record the decisions these pages force, and ask your security team to sign off on them before the build starts.
How DeployOpen scopes, delivers and supports Graylog monitoring
DeployOpen plans and builds Graylog monitoring as a written scope inside your own infrastructure. The work covers how Prometheus scrapes Graylog nodes, how Grafana presents the gl_ metrics, and how alerts reach your on-call team before ingestion stalls. Each step ends with documentation your operators can run without us.
How we scope
- We confirm your Graylog version, node count and whether the Prometheus exporter is already enabled on each node.
- We agree which gl_ metrics, Grafana dashboards and alert thresholds are in the written scope before work starts.
- We record network paths for scraping, the bind address choice and who owns the Prometheus and Grafana hosts.
How we build
- We enable and bind the exporter on each Graylog node, then write the scrape configuration for your Prometheus servers.
- We build Grafana dashboards for throughput, journal utilization and buffer usage, and test them against live traffic.
- We configure alert rules and Alertmanager routing, then test that notifications reach the right people on each alert.
How we hand over
- We deliver configuration files, dashboard definitions and alert rules in a form your team can version and review.
- We document each alert, its threshold, the likely cause and the first checks an operator should run.
- We walk your operators through the setup and confirm they can change a scrape target or threshold on their own.
How we support
- Engagements are fixed-scope, and ongoing support, upgrades and tuning are available as a retainer.
- Retainers can cover Graylog, Prometheus and Grafana version changes so dashboards and alerts keep working after upgrades.
- Confirm with us during scoping which parts of the stack, and which of your teams, the support agreement covers.
