Why monitor Grafana itself and how to monitor Grafana with Prometheus
Monitor Grafana itself so you hear about slow dashboards and failing data sources before the incident channel does. A shared Grafana becomes part of the incident response toolkit, so when it is slow or a data source stops answering, responders lose the view they need at the worst moment. The practical approach is to have Prometheus scrape the Grafana server's /metrics endpoint on a fixed interval, then build dashboards and alert rules on the signals that endpoint exposes. Confirm the exact metric names against your installed version before writing alert rules. Add OpenTelemetry tracing only when metrics show a problem but not its cause, because traces record individual HTTP API calls and their timing. This is a day-two capability layered on a working deployment, so it needs its own owner, scrape configuration and review after the instance reaches production.
Decide who can read the endpoint before you open it to Prometheus. The Grafana security hardening guidance recommends requiring authentication for the metrics endpoint and hiding the version from unauthenticated users, so the scrape job needs a credential and a network path your security team has approved. Confirm which metrics your installed plugins expose, since plugin metrics vary by plugin and should not be assumed to appear on the core endpoint. For tracing, check the OpenTelemetry settings in your Grafana version for the OTLP exporter endpoint, protocol, sampling rate and whether credentials are sent. Grafana sends spans but does not store them, so you also need a trace backend. The OpenTelemetry Collector can receive the spans, apply processing and export them to your chosen store, with authentication configured on the receiver and the exporter.
Plan this work for multi-user production, high availability and any environment where alerts drive on-call work. Grafana high availability uses two or more active-active servers behind a load balancer sharing one MySQL or PostgreSQL database, so scrape every node individually and keep the instance label on each series. Otherwise a failing node can hide behind a healthy load balancer. Watch alert delivery too: the high availability documentation says alerting needs additional configuration to avoid duplicate notifications, and that check belongs on your dashboard. Prometheus stores its own data on a single node without clustering or replication, and its default retention is 15 days, so decide how long you need Grafana's health history and whether to send samples to external long-term storage through remote write. Finally, watch the watcher: test the Prometheus and Alertmanager routes that report on Grafana.
Which signals to collect: metrics, traces and logs
Metrics from the Grafana /metrics endpoint show whether the server is healthy and how busy it is. They cover request volume and latency at the HTTP layer, process state and internal counters that reflect how Grafana works with its database and caches. A Prometheus server scrapes the endpoint on a schedule, so the data becomes a time series you can graph, alert on and keep for trend analysis. The platform team owns this signal, and it is the first place to look when dashboards slow down for everyone at once.
- What it answers: Is the server up, how many requests is it handling and how long do they take?
- Who uses it: Platform and SRE teams for dashboards and alert rules, and capacity planners for trend data.
- What it does not cover: Why one specific query is slow, which user caused a spike or what a dashboard displays.
- Protection: The Grafana security hardening guidance recommends requiring authentication for the metrics endpoint, so the scrape job needs a credential or an approved network path.
- Retention: Prometheus keeps local data for 15 days by default, so plan remote storage if you need a longer history.
Confirm the exact metric names, label sets and whether the endpoint is enabled by default in your Grafana version before writing alert rules against them.
Some plugins expose their own metrics, such as data source plugins that count query calls and record how long those calls take. These metrics show which data source or panel plugin is slow or failing, something the core server metrics cannot isolate. Plugin maintainers differ, since some plugins come from Grafana Labs, others from the community and others from your own team. You need to confirm which installed plugins emit metrics at all, because that varies by plugin and version.
- What it answers: Which data source is returning errors or high latency, separate from Grafana's own request handling.
- Who uses it: Teams that run data source plugins for logs, traces and metrics backends, and on-call engineers triaging failing panels.
- What it does not cover: The health of the backend system itself, which needs that system's own monitoring.
- Inventory: Record each installed plugin and its version, since metric names and availability can change between releases.
- Cardinality: Watch for plugin labels with many unique values, which can inflate Prometheus storage and slow queries.
Before building dashboards, check each plugin's documentation for the metric names it exposes, how they are labeled and whether the scrape configuration needs to change.
Tracing records an HTTP API call as a span with timing for each step inside the request. Traces answer why one request was slow, which aggregate metrics cannot show. OpenTelemetry is a vendor-neutral framework for generating and exporting telemetry, but it does not store or display data, so you need a backend to receive the traces. Confirm how to enable tracing for Grafana's own API in your version, along with the exporter settings that apply to your environment.
- What it answers: Where time goes inside a single API call, such as a dashboard query path or a data source proxy request.
- Who uses it: Platform engineers investigating slow dashboards, and support staff matching a user report to a specific request.
- Collector placement: The OpenTelemetry Collector can run as an agent beside Grafana or as a gateway that receives OTLP from several sources, then processes and exports the data to your backend.
- Volume: Trace volume grows with user activity, so decide sampling and retention before you turn tracing on.
- What it does not cover: Trends across all requests, which metrics show more clearly, or the content of query results.
If the Collector receives traces from outside the host, confirm how its receivers are authenticated and encrypted before opening the port.
Server logs record startup, configuration loading, errors and request-level events from the Grafana process. They answer what failed, when it failed and which component reported it, such as a database connection error, a plugin that did not load or a failed sign-in. Logs are often the strongest evidence after an incident, but their volume depends on the log level you choose and on the number of users. Confirm the log format, output path and level settings for your version before you design collection.
- What it answers: What failed on a specific node, at what time and in which component.
- Who uses it: Operators during incidents, and security staff reviewing authentication failures and configuration changes.
- Shipping: Decide whether logs are read from the host by an agent or written to a central platform, and document the path.
- Retention: Keep logs long enough to meet your incident and audit requirements, and limit who can read them.
- What it does not cover: Numeric trends or latency percentiles, which belong in metrics, and the internal path of one request, which belongs in traces.
Test that an alert built on a known error message fires in a staging instance, because log text can change between versions.
Configuration options and when each fits
This table compares the four documented ways to watch a self-hosted Grafana server: scraping its metrics endpoint, adding plugin metrics to Prometheus, sending HTTP API traces to a Collector, and sending traces straight to a backend. Read each row for the team it suits, what it asks of that team and the trade-off it carries. Confirm the exact config keys against your Grafana version before you change anything.
| Option | When it fits | What the team needs | Main trade-off |
|---|---|---|---|
| Scrape /metrics with a Prometheus server | You run Prometheus already and want Grafana server health, request and database metrics alongside other services | A Prometheus server, a scrape job for each Grafana node and a token or network rule protecting the endpoint | Prometheus is a single node store, so you plan retention and any remote storage yourself |
| Add scrape config for plugin metrics | Your custom or third-party plugins expose their own metrics and you want them in the same dashboards | Plugin metrics endpoints confirmed for each plugin, extra scrape jobs and a review of label cardinality | Each plugin adds series and a maintenance item every time it is upgraded |
| Trace HTTP API calls with OTLP to a Collector | You want per-request timing for the Grafana API and a shared gateway that filters or routes telemetry | A running OpenTelemetry Collector, OTLP receiver reachability from Grafana, and an agreed sampling and retention policy | Another component to run and secure, and trace data is only useful if a backend stores and displays it |
| Send traces directly to a tracing backend | You have one backend and no need for a Collector to process or route data | Backend endpoint and credentials managed in Grafana configuration, plus a plan for rotating them | Fewer moving parts, but changing backends later means reconfiguring every Grafana node |
| Monitor alerts on Grafana health | You need to know about slow queries, failing data sources or a stalled metrics scrape | Alert rules in Prometheus or Grafana Alerting and a named owner for each notification route | Thresholds need tuning against your own traffic, so early alerts can be noisy |
How to set up Grafana metrics scraping and OTLP tracing
This sequence turns on Grafana's own metrics endpoint, scrapes it with Prometheus, sends HTTP API traces through an OpenTelemetry Collector, and ends with dashboards and alerts for slow queries and failing data sources. Each step closes with a check so the team knows the signal is flowing before it relies on it.
Enable the metrics endpoint on a test node
On a non-production Grafana node, set the metrics options in the
[metrics]section ofgrafana.ini(or the matching environment variables), restart the service, and request the/metricspath from a host on the same network as the planned Prometheus server. Confirm the response is plain-text Prometheus exposition format and that it includesgrafana_series such as build information. If the endpoint is unreachable or returns nothing, check that the metrics setting is enabled and that the reverse proxy is not intercepting the path. Confirm exact option names against the version you run, since the installation and configuration reference is the authority for your release.Decide how the metrics endpoint is protected
Before the scrape job is written, the Grafana team and the security owner agree on how Prometheus authenticates. Grafana's security hardening guidance recommends requiring authentication on the metrics endpoint, so the usual choice is a dedicated credential for the scraper or a network path that only the Prometheus server can reach, combined with a proxy rule that keeps
/metricsoff public routes. Record who owns the credential, where it is stored, and how it is rotated. The step is done when a request without credentials is refused and a request with the agreed credential succeeds on the test node.Add the Prometheus scrape job
The Prometheus administrator adds a scrape job to
prometheus.ymlthat points at the Grafana node or load balancer address, sets the metrics path, applies the agreed authentication, and labels the job so Grafana series are easy to filter. Where Grafana runs as several active-active nodes, each node is listed as its own target so per-instance series stay distinct. Reload the Prometheus configuration and confirm the new job appears in the configuration as loaded without errors. The step closes when the configuration validates and no reload error is logged.Verify targets are up and series appear
In the Prometheus interface, open the target status page and confirm every Grafana node shows an up state with a recent last-scrape time. Then run a simple query against a Grafana series, such as the build information metric, and check that it returns a result for each node. If a target is down, compare the scrape error with the authentication and network decisions made in the earlier steps. Keep in mind that Prometheus stores its own data on a single node by default, so plan retention for these series as part of the same design.
Enable OTLP tracing toward a Collector
Grafana can export traces for its HTTP API calls over OTLP. In the tracing section of the Grafana configuration, the team sets the exporter to OTLP and points it at an OpenTelemetry Collector endpoint the team operates, using the Collector's receiver address. The Collector is configured with an OTLP receiver and a trace exporter to the chosen backend, and the Collector configuration best practices guidance should guide how its endpoints are secured. Restart Grafana and load a few dashboards to generate traffic. The step is done when the Collector logs show received trace data without export errors.
Confirm spans reach the tracing backend
Open the tracing backend that receives the Collector output and search for recent traces from the Grafana service name. Check that spans for API requests carry the service name and the operations you triggered, and that the timestamps match your test activity. If nothing arrives, look at the Collector pipeline for a dropped or misconfigured exporter, and at the Grafana tracing settings for a wrong endpoint or protocol. Record the Collector version used, because the OpenTelemetry Collector release line changes often and the configuration may need adjusting when you upgrade it.
Build dashboards and alerts for slow queries and failures
Using the scraped series, the operations team builds a dashboard that shows request latency for Grafana's HTTP API, the rate of error responses, and the metrics that indicate data source query time and failures. Alert rules are then written for sustained latency above a threshold agreed with the users, and for a rising failure rate on each data source, with notifications routed to the owning team. Test each rule by triggering a known condition, such as pointing a non-critical panel at an unavailable data source, and confirm the alert fires and reaches the intended contact point. Set silences and routing before handover so that test alerts do not page the on-call rota.
How to protect the metrics endpoint from exposure
The metrics endpoint and the OTLP trace path both reveal how your Grafana and its telemetry pipeline behave, so they need the same discipline as the admin interface. The cards below cover Grafana's own hardening controls, the network and credential decisions around the Prometheus scrape, and the authentication options on the OpenTelemetry Collector.
Require authentication on /metrics
Grafana's hardening guidance recommends requiring authentication for the metrics endpoint, so an anonymous visitor cannot read internal counters about users, data sources and alerting. Confirm the setting in your grafana.ini or environment overrides, then request /metrics without credentials and check that it is refused. Treat a successful anonymous read as a finding, not a convenience.
Hide the version from anonymous users
The hardening guidance also recommends hiding the Grafana version from unauthenticated users. A visible version lets an outsider match your instance to known vulnerabilities before any login attempt. Check the login page, the health endpoint and HTTP response headers from an unauthenticated client, and record what they disclose so your patch process can account for it.
Keep the endpoint off public listeners
Serve metrics from an internal interface or a separate listener that the public load balancer and reverse proxy never route to. Ask your platform team which paths the ingress forwards and confirm that /metrics is not among them. A dashboard that is reachable from the internet but protected by login still leaks metadata if a proxy rule changes later.
Allow only Prometheus through the network
Restrict the Grafana metrics port with a network allowlist that admits only the Prometheus server or the scrape collector. Prometheus should reach Grafana on the internal network, and nothing else should. Test the rule from an uninvolved host in the same subnet, since allowlists often cover whole ranges that were never meant to scrape Grafana.
Manage the scrape credential deliberately
When authentication is required, Prometheus must present a token or basic credential in its scrape configuration. Store it in a secret file that Prometheus reads, give it read-only permission, and set a rotation date with the team that owns Prometheus. Plan the rotation so a new credential is loaded before the old one is revoked, or scrapes will fail silently until someone notices the gap.
Authenticate OTLP receivers and exporters
The OpenTelemetry Collector can authenticate incoming connections on receivers, for example with OIDC through an authenticator extension, and add credentials to outgoing exporter requests with extensions such as oauth2client. Apply these to the path that carries Grafana traces. Also follow the Collector configuration best practices, and confirm which receivers bind to all interfaces by default before you expose any of them.
Which dashboards and alerts catch slow queries and failing data sources
The signals that reveal trouble in a self-hosted Grafana are rising latency on HTTP API calls, errors returned by data source queries, saturation of the database connection pool and alert rule evaluation falling behind its schedule. Each one points at a different layer, so each needs its own panel and its own alert. Latency and error rates for API routes come from the Grafana /metrics endpoint, which you should scrape from Prometheus with authentication enabled as the security hardening guidance recommends. Pool saturation shows up as open connections sitting at the configured maximum, and a sustained backlog in alert evaluation shows that rule groups are taking longer than their interval. Set thresholds from a baseline you record during normal business hours rather than from a number copied from another environment, and confirm the exact metric names against the version you run, since they change between releases.
Build alert rules in Prometheus for conditions that must page someone, such as sustained high API latency, a data source returning errors for several minutes, or evaluation lag that will cause missed alerts. Use a for duration so brief spikes do not fire. Prometheus sends firing alerts to Alertmanager, which groups related alerts so that a database outage does not produce one notification per dashboard. Route by a severity or team label, add inhibition so a data source outage suppresses the downstream latency alerts, and create silences with an expiry for planned upgrades. Point Prometheus at every Alertmanager instance rather than a load balancer, and plan a cluster if the paging path must survive a node failure. Prometheus values reliability over precision, so it is not suitable as the system of record for exact per-request counts. Local storage is a single node that is not clustered or replicated, and the default retention is 15 days. Decide retention by time or size and whether a remote write target will hold longer history, and test restores of that target before you depend on it.
What monitoring Grafana costs and what the work needs from your team
What does it cost to monitor Grafana with Prometheus?
Prometheus itself has no license fee, since it is Apache-2.0 licensed and there is no paid enterprise edition of the project. The cost comes from four places: the infrastructure that runs the Prometheus server, the storage that holds its data, the implementation work to scrape and alert on Grafana correctly, and the people who run it afterward. Infrastructure and storage recur every month and grow with the number of targets, the scrape interval and the retention period. Implementation is usually one-time, while on-call, upgrades and alert tuning recur. When you ask for a quote, ask the partner to separate these components, state the retention and target count the estimate assumes, and name who maintains the server after handover.
How much storage does a Prometheus server need for Grafana metrics?
Storage depends on how much data you keep and how long you keep it. Prometheus writes its local time series database in two-hour blocks with a write-ahead log, and its default retention is 15 days, which you can configure by time or size. Local storage is not clustered or replicated, so the disk plan is part of the design, not an afterthought. If you need longer history, remote write can send samples to external long-term storage. Confirm the retention you need for capacity reviews and audits, then size disks and backups for that number rather than the default.
How long does it take to set up Grafana monitoring?
Duration depends on the scope you agree in writing before work starts. The main drivers are how many Grafana instances need scraping, whether you run one server or a highly available pair, whether you include plugin metrics and tracing, how many dashboards and alert rules you build, and how many teams must approve access to the metrics endpoint. A basic scrape and a few alerts take much less time than a tracing pipeline with an OTLP path, authentication and tuned dashboards. Ask for a timeline tied to deliverables and list the environment dependencies, such as network paths, certificates and the Alertmanager target, so the schedule reflects them.
What does the work need from our team?
Your team provides the environment and the decisions. Expect to supply a Prometheus server or a place to run one, network access from Prometheus to each Grafana /metrics endpoint, a credential for scraping, and write access or a change process for the Grafana configuration. If you add tracing, you also need an OpenTelemetry Collector to receive OTLP data, plus someone who can review the Collector configuration. Name an owner for alert routing, since Alertmanager needs receivers your on-call process can use. Ask the partner to list these inputs in the scope document so nothing is discovered in the middle of the build.
Can our own team set up Grafana monitoring without a partner?
Yes, if your team already runs Prometheus and understands PromQL, service discovery and alert routing. Grafana exposes metrics that you can scrape, and the documentation covers the endpoint and its authentication. The harder parts are protecting the endpoint, deciding what to trace, operating the Collector, and building dashboards that catch slow queries and failing data sources without flooding on-call. A partner is most useful when the team lacks this experience, has a deadline, or wants a reviewed design. Whichever path you choose, assign a named owner for the Prometheus server, because it does not replicate its own data.
What happens to alerts during Grafana HA and upgrades?
In a highly available setup with two or more active-active Grafana servers, alerting needs additional configuration to avoid duplicate notifications, so do not assume the default behaves as you want. Grafana Live also has documented limitations in HA. During upgrades, a Grafana restart can pause dashboards and alert evaluation on that node, so plan a rolling sequence and check that notifications still arrive. Keep Prometheus alert rules for Grafana health separate from Grafana's own alerting where possible, so a Grafana outage still produces a signal. Confirm the exact HA alerting settings with the current Grafana high availability documentation for your version before you design the routing.
Does Prometheus or OpenTelemetry carry a license cost for tracing Grafana API calls?
OpenTelemetry and its Collector are open source. The Collector repository is Apache-2.0 licensed, and OpenTelemetry is not an observability backend, so you still choose and pay for storage and visualization. Tracing costs therefore come from the backend you pick, the volume of spans you keep, the Collector hosts and the people who maintain the pipeline. Some Collector components have mixed stability levels, so check each component's README before you depend on it. Ask for a quote that states the backend, expected span volume and the sampling approach, because those determine most of the ongoing spend.
Limits to confirm before you commit to this design
This design depends on metric names, tracing options, component stability and support windows that change between releases. Each card below is a question to answer with Grafana, the Prometheus and OpenTelemetry projects, or your own environment before the design is fixed.
Grafana metric names for your version
Confirm which metrics your running Grafana version exposes on its /metrics endpoint, including the HTTP request, database and alerting families you plan to alert on. Grafana releases ship on a monthly patch cadence and minor releases every other month, so names and labels can shift between versions. Ask whether any metric is marked as deprecated in the release notes for your target version, and check the exact names against a live scrape before writing alert rules that depend on them.
Grafana tracing options in use
Confirm which tracing settings your Grafana version supports for sending HTTP API spans over OTLP, including the protocol, the endpoint format and whether the exporter is configured in the grafana.ini file or through environment variables. Also ask whether trace context propagates from the load balancer and whether spans cover data source queries or only inbound API calls. Test with one sample request before you wire traces into dashboards.
Collector component stability levels
Each OpenTelemetry Collector component documents its own stability level in its README, and the mix across receivers, processors and exporters is not uniform. Check every component in your pipeline, such as the OTLP receiver, batch processor and the exporter for your trace backend, and record whether it is stable, beta or development. Ask your distribution vendor or internal platform team which contrib components they accept for production and who owns upgrades when a component changes.
Prometheus LTS support windows
Confirm which Prometheus line you will run and when its support ends before you commit to an upgrade cadence. The current LTS line is 3.13, released July 1, 2026 and supported until July 31, 2027, while non-LTS minor releases stop receiving bug fixes after six weeks. Ask whether your team will move to the next LTS line before 3.13 reaches end of support, and who signs off on that change.
Alertmanager cluster setup
Confirm how your Alertmanager instances will form a cluster using the --cluster flags, how many members you need for your notification path, and that Prometheus is configured to send alerts to every Alertmanager instance rather than through a load balancer. Ask who tests failover, how silences replicate across members and whether a Grafana-managed alerting path will also send duplicate notifications. Write down the answers before you route pages through this setup.
Per-node metrics behind Grafana HA
Grafana high availability runs two or more active-active servers behind a load balancer, so each node exposes its own /metrics series. Confirm whether your Prometheus scrape configuration reaches every node individually, whether dashboards aggregate by instance or by job, and how a node that restarts resets its counters. Ask whether your scrape jobs will also collect the metrics endpoint through the load balancer, which would hide individual nodes from view.
Official sources for Grafana, Prometheus and OpenTelemetry monitoring
Start with the Grafana security hardening documentation before you expose monitoring for the Grafana server itself. It covers the controls that matter most for this topic: requiring authentication on the metrics endpoint, hiding the version from unauthenticated users, and setting secure and SameSite cookies. Read it alongside your own reverse proxy configuration, because a proxy that forwards /metrics to the public internet undoes the protection the Grafana setting provides. Confirm the exact configuration keys against the version you run, since the hardening guidance changes between releases.
The Prometheus security model is the reference for deciding who can reach scrape targets and the Prometheus server. It explains that Prometheus itself has no built-in user authentication, so network segmentation, a reverse proxy or a TLS-terminating gateway must carry that responsibility. Pair it with the Alertmanager documentation when you design alert routing. Alertmanager's cluster flags and the guidance to point Prometheus at every Alertmanager instance, rather than through a load balancer, decide whether a single node failure silences your alerts about Grafana.
For the tracing path, read the Collector configuration best practices before you accept any receiver or exporter defaults. The Collector can authenticate incoming connections with authenticator extensions and attach credentials to outgoing exporter requests, so an OTLP endpoint that carries Grafana's HTTP API traces should not be left open. Ask your platform team to confirm which Collector distribution and version they will pin, and check each component's stability level in its README before relying on it in production.
How DeployOpen scopes and supports Grafana monitoring
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure, including the monitoring that watches Grafana itself. For this topic we scope the Prometheus scrape path, endpoint protection, OTLP tracing and the alerts that catch slow queries and failing data sources, then hand the running setup to your team.
How we scope
- Engagements are fixed-scope, with the metrics endpoints, scrape targets and tracing paths agreed in writing before work starts.
- The written scope lists which Grafana servers, plugin metrics and data sources are covered, and which are excluded.
- Scope records where Prometheus, Alertmanager and any OpenTelemetry Collector run, and who owns each component.
How we build
- We configure authenticated access to the metrics endpoint and document the scrape credentials and network paths used.
- We set up OTLP export from Grafana's HTTP API traces to a Collector pipeline in your environment, and test it end to end.
- We build dashboards and alert rules for slow queries and failing data sources, and check they fire against controlled test conditions.
How we hand over
- You receive configuration files, scrape and alert rule definitions, dashboards and a runbook describing each alert's response.
- Documentation covers retention settings, storage sizing assumptions and what your team must patch or rotate over time.
- We walk your engineers through restarting components, adding targets and reading traces, so operations do not depend on us.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer, scoped separately from the build.
- Retainer work can cover Prometheus and Collector version upgrades, reviewing alert noise and adjusting thresholds.
- Ask any partner for a written quote that separates one-time build effort from recurring support, storage and infrastructure costs.
