Grafana high availability

In a Grafana high availability setup, every node runs the alerting engine, so each node can evaluate the same alert rule and send the same notification. The Grafana high availability documentation states that alerting needs additional configuration to avoid duplicate notifications, so treat this as a required build step rather than an optional tuning exercise. Before the first node goes live, confirm for your Grafana version which alerting settings the documentation requires, how the nodes find each other and share alert state, and which network ports must be open between them. Record those answers in the build plan so the team knows what must be configured and verified.

The main decision is how alert evaluation and notification are coordinated across nodes. Ask whether the team must keep one node responsible for each evaluation, or whether the nodes should share state so that each alert instance is notified once. These are different problems. Shared database access and load balancing let any node serve a user request, but they do not, by themselves, stop two nodes from notifying about the same alert. Assign an owner for each alerting setting, and keep those settings in the same infrastructure-as-code repository as the rest of the configuration so that a later change does not quietly reintroduce duplicates. Rule design, contact points and notification policies are covered in the Grafana alerting sibling page.

Test the behavior before relying on it. Create a test alert rule with a condition that you can trigger on demand, and route it to a test contact point that logs every delivery. Trigger the alert and count the notifications per alert instance. Repeat the test with the alert evaluated by each node, and again after restarting one node, so you see the behavior across the full lifecycle. Check resolved notifications as well as firing ones. Illustrative example: if three nodes are running and one firing alert produces three deliveries, the coordination is not working, and the configuration must be corrected before production use. Run these tests in a non-production environment that matches the production database and load balancer layout.

Plan for a node failure explicitly. When a node stops, the remaining nodes should continue to evaluate and notify, but the documentation and your own tests must confirm how quickly responsibilities move and whether an alert can be delayed or duplicated during that transition. Test this by stopping a node while an alert is firing and recording what reaches the contact point. Decide whether your on-call process can tolerate a short delay or an extra notification during failover, and document that expectation for responders. Grafana Live also has limitations in high availability, which apply to live updates in the UI. Those limits are listed in the decisions table on this page.

The most useful thing a high availability test plan can do is prove that a failure produces a predictable, visible and reversible outcome. For Grafana, that means two or more active-active servers behind a load balancer, sharing a MySQL 8.0 or later or PostgreSQL 12 or later database, with database-backed auth tokens so a user's session survives a move to another node. Each test below names the action, the expected behavior and the evidence to keep. Run them in a staging environment that matches production topology, and record results before any change window is approved.

Test 1: stop one node during active sessions. Start a set of browser sessions and a scripted dashboard refresh loop against the load balancer address, then stop the Grafana service on one node. Expected behavior: sessions routed to the stopped node reconnect through the load balancer, and users are not forced into a repeated sign-in. Keep the load balancer access log, the health check transitions, the Grafana server log from both nodes and a timestamped screen recording of one affected session. Also confirm that the stopped node's removal does not leave a stale alert or session record in the shared database.

Test 2: restart or fail over the database. Restart the database primary, or trigger your managed failover, while dashboards are loading. Expected behavior: Grafana returns errors during the window, then recovers without a manual restart of either node. Confirm that the Grafana database connection pool re-establishes and that dashboard saves made during the window are either committed or clearly rejected, never half-written. Keep the database failover event log, Grafana error logs with timestamps, and a before and after comparison of dashboard versions. Confirm the exact failover behavior of your database platform with its vendor documentation, since recovery timing depends on it.

Test 3: take the load balancer health check path offline. Block or change the path your load balancer checks so that one node is marked unhealthy, then restore it. Expected behavior: traffic drains from the unhealthy node and returns only after the health check passes. Verify that the health check endpoint does not expose version or sensitive detail to unauthenticated callers, in line with the security hardening guidance. Keep the load balancer state history and confirm no user received a sign-in prompt loop during the change.

Test 4: alerts fire once and recover. Create a test alert rule that fires on a synthetic condition, then check the contact point. Expected behavior: one notification per alert transition, one resolved notification when the condition clears, and no duplicates from the second node. The Grafana HA documentation states that alerting needs additional configuration to avoid duplicate notifications, so verify that configuration rather than assuming it. Keep the notification receiver logs, the alert state history and the exact time of each notification.

Test 5: provisioning drift after restart. Edit a provisioned dashboard or data source through the UI, then restart the nodes one at a time. Expected behavior: provisioned resources reload from their files at startup, and the UI edit is either overwritten or flagged according to your provisioning settings. Check that both nodes end with identical provisioned state. Stale provisioned dashboards are a sign of unsafe failure, so keep a diff of the file-defined and live objects from each node. Finally, restore a database backup into a separate environment and confirm it loads, shows the expected dashboards and data sources, and that a test query succeeds. Keep the restore runbook, the timing and the verification checklist signed by the operator who ran it.

What unsafe failure looks like. Treat these as failed tests even if the page loads: duplicate notifications for one alert, users looping between the sign-in page and a dashboard, nodes serving different provisioned dashboard versions, and restores that start but show empty folders or broken data sources. Detect them with the notification receiver counts, load balancer logs showing repeated authentication redirects, a periodic comparison of dashboard JSON across nodes, and a restore check that counts dashboards and data sources against the source inventory. Ask your team to define pass and fail thresholds before testing, and confirm with your auditor or assessor what evidence format they accept.

Go-live is the start of the operating work, not the end of the project. A two-node or larger Grafana high availability setup has more moving parts than a single server, and each part needs a named owner and a health signal. Monitor three layers separately: each Grafana node (process up, response time, error rate and memory), the shared database (connections, replication or failover state, storage growth and slow queries), and the load balancer pool (which backends are in service, health check failures and session distribution). A healthy Grafana node does not prove the database is healthy, and a healthy pool can hide a node that is quietly failing its health check. Confirm with your load balancer vendor which health check endpoint Grafana exposes and how your pool reacts when it fails.

The database holds the dashboards, data source definitions, users, teams, folders, alert state and the auth tokens that let a user reach any node. Back it up on a schedule that matches how often people change dashboards and alerts, and keep the backup retention policy written down. Equally important, the backups have to be restorable. Schedule restore tests into a separate environment, check that Grafana starts against the restored database, and confirm that a sample of dashboards, data sources and alert rules comes back intact. Your provisioning repository is the second recovery source. If dashboards, data sources and alert rules are defined as files or code, a rebuild from the repository can restore much of the configuration, but it will not restore UI edits that were never committed, so decide which changes must go through review.

Upgrades in an HA cluster should be rolling, with one node taken out of the load balancer pool, upgraded, checked and returned before the next node is touched. Read the release notes for every minor release you apply, not only for major releases, because breaking changes and deprecations are announced there. Grafana's upgrade guidance describes major releases once a year in April or May, minor releases every other month, and patch releases at least monthly. Each minor release is supported for 9 months, and the last minor release of a major version for 15 months. Use those windows to set your own patch and minor review cadence, and confirm the current support dates on the release pages before you plan. The upgrade sequence, rollback steps and plugin checks are covered in more depth in the Grafana upgrades and support topic.

Finally, the topology and the operating model must be written down before the first incident. Record the node inventory, load balancer configuration, database engine and version, backup locations, the provisioning repository, alerting contact points and which node or service sends each notification. Assign an owner for each runbook, including failover of the database, removing and adding a Grafana node, restoring from backup and emergency rollback. Name an owner for the cadence itself, the person who tracks new minor and patch releases and schedules the work. Review the runbooks after each upgrade and after each restore test, because the first real failover is usually when gaps in these documents appear.