Prometheus and Grafana dashboards and alerting

A dashboard is only as durable as the PromQL behind its panels. When Grafana is rebuilt, reinstalled or moved to a new cluster, the panels should show the same meaning they did before, and that depends on query design more than on the Grafana configuration. The first check is the rate window. Functions such as rate() and increase() need a range vector long enough to contain several samples from each series. Compare the range in every query with the scrape interval set for its job in the Prometheus configuration, and keep the Grafana minimum interval consistent with both. A range shorter than the scrape interval produces gaps or empty panels that look like outages. Confirm the scrape intervals in your own environment before you fix any window.

Expensive queries should be moved into recording rules. A query that aggregates thousands of series across a long time range can be slow to render every time someone opens a dashboard. A recording rule evaluates that expression on a schedule and stores the result as a new series, so panels read a small precomputed result. Give recorded series a consistent name, such as the level:metric:operations convention in the Prometheus documentation, so readers can tell raw metrics from derived ones. Use the Grafana query inspector to find slow panels, then decide which expressions earn a recording rule. Recording rules live in Prometheus rule files, so they belong in version control with the dashboards that depend on them.

Panels break most often when a target is renamed or replaced. If a job label, instance name or service label changes, a panel filtering on the old value goes blank while the new series appears in a different panel. Series from the old name remain queryable for the retention period, which is 15 days by default and configurable by time or size, so a dashboard can show a confusing mix of old and new data during that window. Filter on stable labels where possible, and use relabeling to keep a consistent label set across deployments. Before a planned rename, list the panels and alerts that match on the affected label, then check each one after the change instead of waiting for someone to report an empty graph.

Grafana provisioning keeps the platform rebuildable. At startup, Grafana loads data sources, dashboards, plugins and alerting resources from YAML or JSON files placed in its provisioning directories, so the configuration can be restored from files rather than recreated by hand. Store those files in Git, review changes through the normal merge process, and deploy them with the same pipeline that deploys Grafana. Grafana also documents further as-code options, including Terraform and Git Sync, which suit teams that already manage infrastructure that way. Confirm which provisioning features your Grafana version supports and whether provisioned dashboards can still be edited in the UI. In a high availability setup, every node needs the same provisioning files.

Prometheus values reliability over precision, so dashboards should not be presented as exact billing or audit records. The Prometheus documentation says the system is not a good choice where 100% accuracy is required, such as per-request billing. Rate calculations, extrapolation over gaps and scrape timing all introduce small differences between what a panel shows and what actually happened. Label dashboards and their descriptions as operational views, and state the scrape interval and rate windows in the description so readers understand the resolution. If a team needs exact counts for charging or compliance evidence, that requirement should be handled by a system designed for it, and the decision should be recorded in the scope agreed before implementation.

The first decision for the team is which system evaluates alert rules and sends notifications. Prometheus can evaluate alert rules itself and pass firing alerts to Alertmanager, which handles grouping, inhibition and silences before notifying anyone. Grafana can also define and evaluate alert rules against the Prometheus data source and send notifications through its own contact points and notification policies. Both paths can page people, but they do not offer the same routing model, so the choice shapes how every later alert is written, reviewed and silenced.

Operating the two options carries different availability work. Alertmanager is designed to run as a cluster through its --cluster flags, and Prometheus must be configured to send alerts to every Alertmanager instance rather than to a load balancer in front of them. Grafana high availability uses two or more active-active servers behind a load balancer sharing a MySQL or PostgreSQL database. The Grafana high availability documentation says alerting needs additional configuration to avoid duplicate notifications, and it lists limitations for Grafana Live in that setup. Teams choosing Grafana alerting should confirm these settings in their own environment before they rely on it for paging.

The team should write down which system owns each alert before building dashboards. A panel threshold, a Grafana alert rule and a PrometheusRule can all look similar on screen, yet each one can wake a different on-call engineer. A simple ownership register works well: for each alert, record the name, the evaluating system, the notification path, the silence owner and the runbook. Mixing both systems for the same signal is the most common way paging logic becomes split across two tools, with duplicate pages, silences that apply in one place but not the other, and no single view of what is firing.

Some organizations use Prometheus and Alertmanager for infrastructure and service paging, and reserve Grafana alerting for business or log-derived conditions that already live in another Grafana data source. That split can work, but only if the register states the boundary clearly. Teams should also check whether their existing on-call tool accepts Alertmanager webhooks or Grafana contact points directly, because that integration often decides which path is simpler to operate.

Ask the implementation partner, or the internal platform team, for a written recommendation covering the ownership register, the Alertmanager or Grafana high availability design, and the process for changing alert rules. Confirm the Grafana version and the Alertmanager release line you plan to run, and check their current documentation for clustering and notification behavior before finalizing the design.

The local Prometheus store is enough until its single node can no longer hold the retention, or survive the failures, your dashboards and audits require. The Prometheus storage documentation is direct about this: local storage is not clustered or replicated, so the server should be run like any other single-node database, with its disk capacity, backups and restore procedure treated as your responsibility. Default retention is 15 days, configurable by time or size, so a team that needs months of history or wants metrics to survive a host loss is usually at the point where external long-term storage makes sense. Remote write lets Prometheus send samples to that storage, and remote read lets Prometheus query data back from it. Decide which of these you need before you choose a product, because they change how much the local server keeps and how the query path is built.

The goal of the cutover is that Grafana users see no difference. Keep the Grafana Prometheus data source pointed at a stable name, such as a query front end or proxy address, and change only what sits behind it. Provision the data source and the dashboards as code so that the URL, the authentication settings and the dashboard JSON are reviewed in version control rather than edited by hand in the UI. Dashboards built on PromQL should not need rewriting, but the team must test them. Before cutover, run each important panel and alert expression against both the old and new endpoints over the same time range, and compare results for the PromQL functions in use, such as rate, histogram_quantile and aggregations with label grouping. Differences in staleness handling, range selection or rounding are common and should be understood before users notice them.

Verify label consistency before anything else. Remote storage keeps the series labels that arrive with each sample, so external labels, relabeling rules and target labels set on the original server must match what dashboards and alert rules expect. Check that job and instance values are identical, that any external labels used to identify a cluster or replica are set deliberately, and that dropped or renamed labels are not still referenced in queries. Verify retention separately on both sides: local retention on Prometheus, the retention policy of the long-term store, and any compaction or downsampling the chosen project applies. Confirm which window your audit and incident needs cover, and which store answers queries for that window. Also confirm where alert rules are evaluated. If rules run on the Prometheus server, Alertmanager receives alerts from there; if they run in a ruler or equivalent component of the long-term store, the evaluation source changes, and so may the timing and the labels attached to firing alerts. Only one place should evaluate each rule, or notifications will duplicate.

The specifics of Thanos and Mimir deployment are for your team to confirm against each project's current documentation. Component layout, object storage configuration, query and store gateways, compactor behavior, multi-tenancy settings, and supported remote write and query interfaces change between releases, so do not copy a configuration from an older example without checking it. Ask the implementation partner, or your own platform team, to write down which project and version was chosen, the reasons, and the sizing assumptions for ingestion rate and series count. Remote write adds load and a failure mode to the Prometheus server, so test what happens when the remote endpoint is slow or unavailable, including how much data is buffered and whether it is lost. The exact configuration values should come from the projects and be validated in a non-production copy of your data before any dashboard or alert is moved.

Plan the cutover as a reversible change. Run remote write in parallel for a period, compare the same panels and alerts across both stores, and keep the local server running until the long-term store has passed your verification list. Record the Grafana data source change, the Prometheus remote write settings, and the rule evaluation location in one change record so a rollback is a single known action. Involve the security owners as well: remote endpoints need TLS and authentication, and the storage bucket and query endpoints need access controls that match how your Grafana users are authorized. For cost, the software is open source, so spend comes from the infrastructure for ingestion, storage and query, from the people who build and operate the system, and from any vendor support you buy. Ask any partner for a written quote that separates one-time build work from recurring operation, and for the assumptions behind the storage and compute estimates.