Grafana Alerting HA and duplicate notifications

How Grafana Alerting high availability works and when teams need it

If one Grafana node fails, alert rules must keep evaluating and notifications must keep going out, yet a single incident should page the on-call engineer once. Grafana's high availability model meets the first need by running two or more active-active Grafana servers behind a load balancer, all sharing one MySQL or PostgreSQL database, with database-backed auth tokens so users can reach any node. The documentation is explicit that alerting needs additional configuration to avoid duplicate notifications, so the database and load balancer alone do not give you clean paging. Teams need this design when a missed page carries real operational or compliance cost, and when a single Grafana host would be an unacceptable point of failure for on-call.

Three decisions shape the design. The first is alert state sharing: each node must know which alert instances are already firing, pending or resolved, and Grafana uses a gossip cluster between nodes to share that state. The second is evaluation placement: every node can evaluate rules, or you can restrict evaluation to a single node to reduce load on your data sources, which trades some resilience for lower query volume. The third is the notifier path: decide which node sends notifications and how the cluster deduplicates them, because this is where duplicate pages usually originate. Each choice affects the others, so make them together before you size the database or write the load balancer configuration.

Before you commit, confirm with your team which failure you are designing for. A single-node Grafana with a backed-up database may be enough when an outage of minutes is tolerable and alerts also reach a second system such as Prometheus Alertmanager. Run-time settings for clustering and notifier behavior vary by Grafana version and by how you deploy, so check the current Grafana high availability documentation against the version you plan to run. Treat the setup as untested until you have stopped a node during a test alert and counted the notifications received.

Choosing memberlist gossip, Redis or single-node evaluation

This table compares three ways to run Grafana Alerting across several instances: memberlist gossip, a Redis-backed option and single-node evaluation. Read each row against your network paths, Redis licensing position and tolerance for an evaluation pause. Confirm the supported setup in current Grafana documentation before you choose, because these settings change between releases.

Option	What it fits	What it requires	Main trade-off
Memberlist gossip on port 9094	Grafana instances that can reach each other directly on a shared network	Network paths between every instance on port 9094, plus ha_advertise_address on Kubernetes	Peer addresses and firewall rules must stay correct as instances are added or replaced
Redis-backed coordination	Teams that want shared coordination outside instance-to-instance gossip	A Redis deployment with Sentinel, at least three Sentinel instances, and a licensing check for your Redis release	Adds a stateful service to operate, monitor and patch
Redis Open Source license by release line	Deciding which Redis version your organization is permitted to run	Redis 7.2.x and earlier under BSD-3-Clause; 7.4.x to 7.8.x under RSALv2 or SSPLv1; 8.0 and later also offer AGPLv3	Redis 7.4.x to 7.8.x uses licenses that are not OSI-approved, so legal review may be needed
Sentinel high availability for Redis	Redis deployments that are not Redis Cluster and must fail over automatically	At least three Sentinel instances, with a replica promoted to master during failover	Failover behavior must be tested, since Sentinel does not cover Redis Cluster
Single-node evaluation	Smaller estates where one instance can evaluate every alert rule	Only one instance evaluates rules, with a documented plan to restart or replace it	Evaluation stops if that node fails unless failover is in place; query load is lower
Duplicate notifications under any option	Every multi-instance design, whatever coordination method is chosen	The HA alerting settings your Grafana version documents for avoiding duplicates	Coordination alone does not prove no duplicates; test with a rule that fires on purpose

Why single-node evaluation reduces query load and what it gives up

In an HA Grafana Alerting cluster, every instance that evaluates a rule sends that rule's queries to the data source, so a rule evaluated on three instances is queried three times per interval. Restricting evaluation to one node removes that duplicated load, which matters most when the data source is an expensive log store or a shared SQL database. The other instances keep serving dashboards, receive alert state through the cluster, and take over evaluation only when needed. Before you enable this, measure how many rule evaluations per minute your current cluster actually runs against each data source, so you can judge whether the saving is worth the trade-off below.

The cost of this design is a single point of evaluation. While the evaluating node is down, no rule is evaluated, so a metric threshold can be crossed without any alert firing. Recovery therefore depends on failover behavior: how quickly the cluster detects the lost node, which instance takes over evaluation, and whether pending and firing state carries across. Treat failover time as a number you have measured, not one you have assumed. Test it by stopping the evaluating node during a rule that is known to fire, then confirm the alert still reaches the contact point and note the gap. Also confirm how the cluster selects a new evaluator and what happens to alerts that were already firing.

The exact settings that control single-node evaluation, failover timing and their defaults change between Grafana releases, so confirm them in the Grafana documentation for the version you run before you rely on them. Check the high availability setup guide and the alerting configuration reference, and record the failover result from your own test in the runbook for your on-call team. If your data source cannot tolerate a short gap in evaluation, or if a missed alert is unacceptable for a given rule set, plan monitoring of the evaluating node and a tested recovery procedure as part of the design, not as an afterthought.

Setting up Grafana Alerting across two or more instances

This sequence takes a shared production database and two or more Grafana nodes to a state where the nodes form one alerting cluster, see each other, and load the same alert rules and contact points. Each step closes with a check you can run before relying on the cluster for paging.

Confirm the shared production database
Your platform or database team confirms that every Grafana node will point at one MySQL 8.0 or later, or PostgreSQL 12 or later, database. SQLite does not scale for production, so it cannot serve as the shared store for a high availability setup. Check the database version, the connection limits and the network path from each planned node. The step is done when a test connection from every node succeeds and the database engine and version are recorded in your runbook.
Install identical Grafana versions on each node
The platform engineer installs the same Grafana release on every node, using the same package source, image tag or Helm chart version. Mismatched versions can disagree about alerting state and cluster messages, so pin the version in configuration management rather than letting each node update on its own. Confirm the result by comparing the reported version on each node's login page or in the server log at startup. The step is done when every node reports the same version string.
Point every node at the shared database and auth tokens
Each node's configuration sets the same database type, host and credentials, so all nodes read and write the same configuration and session data. Confirm that database-backed auth tokens are in use, so a user who logs in through one node can reach any other node behind the load balancer. Restart nodes one at a time and watch the logs for migration errors. The step is done when a dashboard created on one node is visible from the other node.
Configure cluster peers with memberlist or Redis
On each node, the platform engineer sets the alerting cluster settings so nodes can find each other. With memberlist, each node lists its peers and listens for gossip on port 9094 (the default for the alerting cluster listen address). Where your network policy does not allow gossip between nodes, configure Redis as the coordination channel instead, and confirm with your Grafana version's documentation which cluster options apply. Keep the peer list identical across nodes. The step is done when the configuration files on every node contain the same peer list or Redis endpoint.
Set the advertise address on Kubernetes
When Grafana runs in pods, the address a node sees for itself is often a pod-internal IP that other pods cannot reach through the peer address. Set the advertise address (ha_advertise_address) to the address other nodes can route to, typically the pod IP injected through the Kubernetes downward API, and confirm the Service or headless Service that peers use for discovery. Check the chart or manifest you use for the exact key and path. The step is done when each pod logs the advertise address it uses and that address matches the one in its peer configuration.
Verify that nodes see each other
Restart the nodes and check the alerting cluster status in the Grafana interface or through the cluster metrics your monitoring stack already collects. Each node should list every other node as a connected peer. If a node is missing, check the advertise address, the gossip port and firewall or network policy rules between nodes before changing anything else. The step is done when every node reports the full member count and the membership stays stable across a restart of one node.
Confirm rules and contact points match across nodes
Confirm that alert rules, contact points and notification policies come from one source for all nodes, for example file provisioning loaded at startup or a single shared configuration. Spot check the same rule and contact point on each node, then send a test notification from one contact point and record how many deliveries arrive at the receiving channel. Expect one delivery per notification. If you see two, treat it as a duplicate and check the cluster health and routing before going live. The step is done when a test alert reaches the channel exactly once and every node shows identical configuration.

Why duplicate notifications can still happen after clustering is on

Turning on clustering does not guarantee one notification per alert. Several causes survive a correct cluster setup, and each one has a check you can run. The first is mixed versions: if nodes in the same Grafana instance run different releases, they may not agree on how alert state is shared, so one node can keep firing while another believes the alert was already delivered. Confirm the version on every node with the Grafana server version shown in the UI or the health endpoint, and align them through your upgrade process. The exact behavior across versions must be confirmed against the release notes for the version in use.

The second cause is a node that cannot reach its peers. A node that loses gossip traffic on port 9094 or cannot talk to Redis may decide it is alone and send notifications independently. Check the cluster status in the alerting section of the Grafana UI and the cluster peer metrics on each node. Confirm that every node lists the expected number of peers. Network partitions create the same failure at a larger scale, so test a node restart and a blocked port in a staging environment before you rely on the cluster for paging.

The third and fourth causes sit outside the cluster. An external Alertmanager or a second Grafana instance that evaluates the same rules, and is not part of the cluster, sends its own notifications. Review every place that forwards Grafana-managed alerts and confirm that only one evaluator and one notifier handle each rule. Finally, a contact point that posts to a chat channel is called from each node, so a single alert may reach the channel once per node that sends it. Check the delivery history for a test alert and compare the message count with the number of nodes. Confirm the cluster and contact point behavior for your Grafana version with the official alerting documentation.

Security and operational controls for the alerting cluster

A highly available alerting cluster adds attack surface and failure modes that a single node does not have. The controls below cover the peer channel, the shared database, secrets, change access, health monitoring and version consistency, and each one ends in a check your operations team can run.

Protect the peer channel

Gossip traffic between Grafana instances on port 9094 decides which nodes belong to the alerting cluster. Restrict that port to the Grafana nodes only, using host firewalls, security groups or network policies, so no other workload can join or observe the cluster. On Kubernetes, confirm that a NetworkPolicy allows 9094 only between Grafana pods. The check is to attempt a connection to 9094 from a host outside the cluster and confirm it fails.

Secure the shared database

Every Grafana node reads and writes the same MySQL or PostgreSQL database, so its credentials and network path are part of the alerting design. Use a dedicated database user with rights limited to the Grafana schema, store the password in a secrets manager or a protected environment file, and require TLS between Grafana and the database. Confirm that the database accepts connections only from the Grafana subnet. Grafana's own high availability guidance is the starting reference for the database setup.

Encrypt contact point secrets

Contact points hold webhook URLs, API keys and integration tokens for Slack, Teams and PagerDuty. Keep these values out of provisioning files committed to Git and out of shared configuration templates, and supply them through environment variables or a secrets store at startup. Review who can export or view contact point settings in the UI. The check is to search your repositories and ConfigMaps for known webhook prefixes and token patterns and confirm nothing is found.

Restrict silences and rule edits

A silence or mute timing that a junior user creates can stop paging for an entire service, and an edited rule can change what fires. Limit editor and admin roles to the people who own alerting, and limit silence creation to on-call leads. Open-source Grafana access control is coarser than Enterprise, so confirm what your edition allows and record silence changes in your audit process. The check is to list every user with alerting write rights and compare that list with the on-call roster.

Monitor cluster health metrics

Duplicate notifications and missed alerts often start as a node that silently drops out of the cluster. Scrape each Grafana instance's metrics endpoint and alert on the number of cluster peers, failed notification attempts and evaluation errors. Keep the metrics endpoint behind authentication, as the security hardening docs recommend, so the cluster's internal state is not exposed. The check is to stop one node in a test environment and confirm that a peer-count alert fires.

Keep all nodes on one version

Nodes running different Grafana versions can disagree about alerting state and cluster messages during a rolling upgrade. Pin one image or package version across all instances, upgrade them in a planned window, and verify that every node reports the same version before and after the change. Grafana publishes patch releases at least monthly, so set a recurring review that schedules these updates rather than letting nodes drift. The check is to compare the version reported by each instance after every upgrade.

How Kubernetes and virtual machines change the cluster setup

In a Kubernetes deployment, each Grafana pod is a peer that the other pods must reach directly over the cluster network. The address a pod announces has to be one its peers can route to, so set ha_advertise_address to the pod's own IP and inject that value at startup through the downward API. Confirm the exact section and key names for your Grafana version before you template them.

Advertised address: Each pod should advertise its own address, never a value baked into the image or shared across replicas.
Peer port exposure: The gossip port (9094 in the default memberlist setup) must be reachable between Grafana pods through the Service definition and any NetworkPolicy you apply.
Peer discovery: A headless Service or an explicit peer list lets new pods find existing members. Check how your Helm chart or manifests populate that list.
Rollouts and restarts: A rolling update replaces pods one at a time, and a replacement pod rejoins with a new address. Test whether alert state held in memory is rebuilt or lost in your version.
Ephemeral storage: Pods start with no local state, so the shared database is what carries configuration across restarts.

After a rollout, confirm that every pod sees all peers and that a deleted pod rejoins without producing duplicate notifications.

Questions teams ask before running Grafana alerting in HA

What drives the cost of running Grafana alerting in high availability?

The cost depends mostly on how many instances and environments must be covered, whether they run on Kubernetes or virtual machines, and how much testing your change process requires. Cluster size changes the number of nodes to configure and verify. Kubernetes adds work on pod networking and the advertise address, while virtual machines need host firewall and address planning. The number of contact points, any existing Alertmanager you forward to, and the depth of failover testing all add scope. The one-time work is design, build and test. The recurring work is upgrades, since Grafana publishes minor releases every other month and patch releases at least monthly, and each change to rules or routing should be retested. Ask any partner for a quote that lists each of these items separately, with the environments and test cases included.

How long does it take to be confident that deduplication works?

Confidence comes from test coverage and change control, not from elapsed time alone. You need to show that a firing alert produces one notification when several instances evaluate it, that the notification still arrives when one instance stops, and that the behavior holds after a restart or upgrade. Each of those tests needs a defined procedure, a named observer and a record of the result. Teams that already have a staging cluster and a repeatable failover test usually reach that point sooner than teams starting from nothing. No duration is fixed by the product, so ask any partner to state the test cases and the approval steps in the written scope before the work starts.

Can our internal team set up Grafana alerting HA without a partner?

Yes, if the team can test failover and read cluster health metrics. The setup itself is a matter of configuring the database, memberlist gossip on port 9094 or Redis, and the advertise address on Kubernetes, then checking that every instance joins the cluster. The harder skill is proving behavior: stopping an instance during a live alert, confirming a single notification, and reading the cluster metrics to spot a split or a stalled peer. Teams without that experience often benefit from an outside review of the design and test plan. DeployOpen plans, implements and documents this work in infrastructure you control, so you can decide which parts to keep in house.

What does the work need from our team?

The work needs access and decisions from several people. Expect to provide:

Database access for a MySQL 8.0 or later or PostgreSQL 12 or later instance, since SQLite does not suit production or high availability.
Network rules that allow the gossip traffic on port 9094 between Grafana instances, or connectivity to Redis if you choose that path.
A test alert window where failover can be exercised and notifications can be watched.
Named owners for the Grafana platform, the on-call routing, the receiving contact points and the change approvals.

Agree these owners in writing before the schedule is set, so that test results have a decision-maker.

Do Grafana Enterprise features change the high availability model?

Grafana Enterprise is a commercial option that is not governed by the AGPLv3 license used for the open-source edition, and its feature set differs in areas such as authentication. The high availability guidance for open-source Grafana still applies to the alerting cluster, but you should not assume that any Enterprise feature changes how duplicate notifications are prevented. Confirm with Grafana Labs which features apply to your edition and version, and ask for that answer in writing before you select a design.

Why can duplicate notifications still happen after the cluster is configured?

Clustering reduces duplicates but does not guarantee zero. A notification can repeat when cluster peers cannot see each other, when a network partition splits the gossip group, when a node restarts during a notification cycle, or when a receiver or external Alertmanager is configured in a way that bypasses the cluster. Check the cluster health metrics, the advertise address and the peer list after each change, and run failover tests on each release. Treat any repeat as a defect to trace, not as expected behavior.

Should we run alert evaluation on a single node to reduce query load?

Running evaluation on one node can cut query load on your data sources, but it changes your failure behavior. If that node stops, evaluation stops until another instance takes over or an operator restores it, so you need monitoring and a tested recovery step. Decide this with your data source owners, because the trade-off depends on how many rules you run and how quickly alerts must fire. Document the chosen approach and test the recovery path before relying on it for on-call.

Limits to confirm in your environment before go-live

Grafana Alerting high availability depends on details that vary by version, cluster transport and data source behavior. The items below are the ones most likely to change how many notifications your team receives and which dashboards behave differently across nodes. Each should be verified in the vendor documentation or in a test environment that mirrors production.

Exact Grafana version and HA alerting behavior

Alerting high availability behavior changes between releases, so the version you deploy determines which settings apply. The Grafana high availability docs state that alerting needs additional configuration to avoid duplicate notifications, but the exact steps should be read against your installed version. Confirm the release line you plan to run, its support window, and the upgrade path to the next minor release. Record the version in your go-live checklist so the HA settings are tested against the same build you will operate.

Memberlist or Redis for cluster transport

Grafana instances must exchange alert state, and you must choose how they reach each other. Memberlist gossip on port 9094 needs that port open between every Grafana node, which may be blocked by network policy or a load balancer. Redis adds another stateful service with its own high availability, failover and license terms. Confirm which transport fits your network segmentation, and if you choose Redis, confirm the license and release line before adopting it.

Redis license and release line

If Redis is the chosen transport, the license and release line affect what you can deploy and how you can operate it. Redis 8.0 and later are offered under a license of your choice among RSALv2, SSPLv1 and AGPLv3, while Redis 7.4.x to 7.8.x use the dual RSALv2 or SSPLv1 license and Redis 7.2.x and earlier use BSD-3-Clause. Have legal review the license and confirm the exact release you will pin. Check the license FAQ for your situation, especially if you modify the code or offer it as a network service.

Grafana Live limits on shared dashboards

Grafana Live powers streaming updates in dashboards, and the high availability docs note limitations for it in a clustered setup. Teams that rely on live panels, streaming data or real-time views may see behavior differ when requests land on different nodes. Test your most important dashboards behind the load balancer, including sessions that switch nodes, and confirm whether live features are affected before you promise real-time views to operators.

Data source behavior under single node evaluation

Running alert evaluation on a single node reduces query load on your data sources, but it changes the timing and failure profile of every rule. If that node stops or is slow, evaluation pauses until another node takes over, so you need to know how long alerts can go unevaluated. Confirm that each data source tolerates the query pattern, rate limits and connection counts from one evaluating node, and test what happens when the evaluating node restarts during an incident.

Repeat and late deliveries from notification providers

Clustered alerting can still send duplicates when a notification is delivered but not acknowledged in time, or when a node retries after a timeout. Your external providers, such as paging, chat and ticketing systems, may also accept repeated or delayed messages differently. Ask each provider how it handles duplicate webhook calls, retries and out-of-order delivery, and whether it deduplicates on a key. Test a forced failover during an active alert and count what each responder actually receives.

How DeployOpen scopes and delivers Grafana alerting high availability

DeployOpen plans and runs self-hosted Grafana Alerting high availability inside your own infrastructure, starting from a written scope that names the instances, database, gossip or Redis layer and notification paths involved. We build and test the cluster, prove that each notification is sent once, and document the runbook your team needs to operate it after handover.

How we scope

Inventory of Grafana instances, shared database, load balancer and hosting platform, with the database engine confirmed for high availability use.
Choice between memberlist gossip on port 9094 and Redis for alert state sharing, recorded with the reasons and network requirements.
Written list of alert rules that will run on one node for evaluation, and of contact points that must not fire from every instance.

How we build

Configuration of the cluster peers, advertise addresses and Kubernetes networking so instances find each other reliably after restarts.
Duplicate notification tests that fire test alerts through each contact point and confirm one delivery per receiver.
Dashboards and alerts on cluster health and peer status, configured so a lost member is visible before it causes missed or repeated pages.

How we hand over

Runbook covering node replacement, rolling upgrades, failover checks and the steps to take when duplicates appear.
Configuration kept as code in your version control system, with the cluster settings and provisioned alerting resources documented.
Walkthrough with your on-call and platform engineers, with a clear list of which behaviors remain your responsibility after handover.

How we support

Fixed-scope engagements, with scope, deliverables and timelines agreed in writing before work starts.
Ongoing support, Grafana upgrades and continuous testing available as a retainer, scoped separately from the build.
Review of each Grafana release against your cluster configuration before you upgrade, so peer settings and alert behavior are checked first.

Grafana Alerting high availability and duplicate notifications