When an external Alertmanager with Grafana makes sense
Forwarding Grafana-managed alerts to an external Alertmanager means Grafana evaluates the rules, then hands the firing alerts to a Prometheus Alertmanager or a Mimir Alertmanager instead of routing them through its own notification policy tree. The shared layer then applies grouping, inhibition and silences for every source that sends alerts into it. This fits a team that already runs Prometheus Alertmanager or Mimir, has Prometheus rules and Grafana rules firing into different places, and wants one routing and silencing layer that on-call staff check in one location. The first decision is whether that consolidation is worth the extra component to operate. If most alerts originate in Grafana and the team has no existing Alertmanager, Grafana's built-in notification policies, contact points and silences usually cover the need with less to run.
The second decision is who owns routing. In a shared Alertmanager, label design, receivers and silences live in one configuration that several teams may touch, so a change to a route can affect alerts from sources the editing team does not own. Confirm who may edit the Alertmanager configuration, how changes are reviewed, and whether silences created in Grafana and in Prometheus tooling will be visible to the same responders. Also confirm which Grafana version and Alertmanager version your environment runs, because the external Alertmanager option and its related settings are documented per release and should be checked against the Grafana documentation before you plan around them.
Finally, plan for the case where the external Alertmanager is unreachable. Grafana then cannot deliver notifications through that path, so the shared layer becomes a dependency for every alert it receives. Decide whether a separate fallback route will exist, how the Alertmanager cluster will be made highly available, and who is notified when delivery fails. The key question to settle before the build is whether one routing layer will carry all paging alerts, or only the alerts that already belong to a shared Alertmanager.
Which forwarding option fits your alert estate
This table compares four ways to connect Grafana-managed alerts to an external Alertmanager or Mimir, across routing control, duplicate risk and operational ownership. Read each row as a decision: the further down the table you go, the more routing stays in one place, but the more the external system has to be operated and trusted. The main takeaway is that forwarding everything is the simplest start, while the other options trade simplicity for finer control.
| Option | Routing control | Duplicate and ownership risk | Choose it when |
|---|---|---|---|
| Forward all Grafana-managed alerts | Routing moves to the external Alertmanager, and Grafana policies no longer decide the destination | Low duplicate risk if Grafana sends only to the external receiver; ownership sits with the team running Alertmanager | One team owns all alert routing and wants a single shared receiver |
| Forward selected alerts through a dedicated contact point | Split routing: chosen alerts go to the external Alertmanager, others stay in Grafana | Moderate duplicate risk if the same alert matches two routes; test each policy path | Only some alerts belong in the shared system, such as platform-wide signals |
| Keep Grafana notification policies as the router | Full control stays in Grafana's policy tree, labels and contact points | Lowest duplicate risk; ownership stays with the Grafana team | Teams already run rules and routes in Grafana and do not need a shared Alertmanager |
| Use Prometheus or Mimir as the shared rule and routing source | Rules and routes are defined in the shared system, which becomes the single source of truth | Highest ownership load: the shared system must be patched, sized and monitored, and Grafana must be read-only for those rules | Many Prometheus-based teams need one governed rule set and receiver configuration |
What changes when Prometheus or Mimir owns the rules
When Prometheus or Mimir evaluates the rules, the rule definitions live in that system's ruler configuration, and Grafana reads them through the data source. Your team edits rules in version control, in Prometheus rule files or through the Mimir ruler, and Grafana shows them without the edit controls it offers for rules it manages itself. Changes that used to happen in the Grafana UI now belong in the pipeline that deploys the rule files.
- Rule edits move upstream. Change expressions, labels and
fordurations in the rule files or the Mimir ruler, not in Grafana. - Grafana cannot change rule state. Editing, pausing or deleting externally managed rules from Grafana is not available, so each runbook should name the owning repository.
- Routing is a separate decision. The external Alertmanager's own routing tree decides who is notified, as described in the Alertmanager documentation.
- Mixed ownership needs a boundary. If some rules stay in Grafana, document which system owns each rule group so two places never change the same alert.
Confirm which rule actions remain available in the Grafana version you run, and test a rule change end to end before relying on it.
For responders, the main change is where you act on a page. Alert details may show the rule source, but acknowledging, silencing or changing an alert depends on what the external Alertmanager and Grafana expose in the version you run. A fix that used to take one click in the Grafana UI may now require a commit, a deploy or a trip to another interface.
- Silences may be created elsewhere. Confirm whether Grafana can create silences for alerts from an external Alertmanager, or whether silences must be created in the Alertmanager interface.
- Rule edits are slower. Raising a threshold or pausing evaluation goes through a commit and deployment, which takes longer than an in-UI change during an incident.
- Notification settings may not follow the alert. Contact point edits and mute timings configured in Grafana may not apply to alerts routed by the external Alertmanager unless the same configuration exists there.
- Runbooks should name the owner. Link each alert to the repository, team and Alertmanager where it can be silenced or changed.
Ask the platform team to demonstrate a silence on a test alert during your next handover, so the steps are known before an incident.
Reviewers need to know where the authoritative rule definitions, change history and notification records sit. When Prometheus or Mimir owns the rules, the evidence trail moves toward version control and the Alertmanager, and the Grafana view becomes a reflection of those systems rather than the place where changes are made.
- Source of truth. The rule definitions in the repository and the deployed ruler are the record of what is monitored, so Grafana screenshots alone are weak evidence.
- Change approval. Rule changes that pass code review and deployment approval create a change history that can be sampled.
- Silence records. Silences created in Alertmanager or Grafana need a record of who created them, when, and why. Confirm how each system stores this.
- Gaps to close. Document where the Grafana view stops being authoritative and which system holds the notification log.
DeployOpen prepares systems and evidence for assessments, but independent auditors decide whether that evidence is sufficient. Request a sample of rule changes and a silence record from the system that owns each rule, and confirm that the ownership map matches what the auditor is likely to sample.
Setting up forwarding to an external Alertmanager
This sequence connects Grafana-managed alerting to an Alertmanager cluster you already run, so that Grafana rules send notifications through the shared routing, grouping and silence model. Each step ends with a check, and the final step describes how to back out if duplicates or missing alerts appear.
Confirm the Alertmanager cluster and endpoints
The platform team confirms every Alertmanager instance, its URL and reachability from the Grafana server or pods, and whether the instances form a cluster using the cluster flags described in the Alertmanager docs. Prometheus should already list every instance, so Grafana should target the same set. Record the endpoints, the network path, and any TLS or authentication requirements. The step is done when each endpoint answers a basic request from the Grafana host and the cluster peers report as healthy.
Add the Alertmanager data source in Grafana
A Grafana administrator adds an Alertmanager data source in the Grafana UI or through file provisioning, choosing the implementation that matches your backend, such as Prometheus Alertmanager or Grafana Mimir. Enter the endpoint and any authentication settings, then save. Confirm the exact field names and implementation options for your Grafana version in the Grafana alerting documentation. The step is done when the data source test succeeds and the data source appears in the list of Alertmanagers Grafana can see.
Enable receive Grafana alerts
On the Alertmanager data source settings, the administrator turns on the option that lets this Alertmanager receive alerts from Grafana-managed rules. Only enable it on the data source that should receive Grafana alerts, and review whether the internal Grafana Alertmanager should keep sending notifications for the same rules during the trial. Save the change. The step is done when the setting shows as enabled and Grafana lists the external Alertmanager among its active alert receivers.
Create a test alert
An engineer creates a Grafana-managed alert rule with a clearly labeled test condition, such as a query that always breaches its threshold, and gives it labels that match an existing route on the external Alertmanager. Set a short evaluation interval so results arrive quickly. Label the rule as a test in its name and annotations so responders do not mistake it for a live incident. The step is done when Grafana shows the rule in a firing state.
Verify delivery and grouping
The engineer checks the Alertmanager UI or API to confirm the test alert arrived with the expected labels, that it matched the intended route, and that it was grouped with other alerts sharing the same grouping labels. Then confirm the receiver actually delivered the notification to the target channel or pager. Check that the same alert does not also appear through the internal Grafana path. The step is done when the alert shows once in the external Alertmanager, routes to the expected receiver, and the test is recorded.
Roll back if duplicates or missing alerts appear
If responders receive duplicate notifications, or expected alerts never reach the external Alertmanager, the administrator turns off receive Grafana alerts on the data source so that one delivery path remains. Resolve the test alert, remove any temporary silences, and review the endpoint, cluster membership and label matching before trying again. Keep the data source configuration in version control so the change can be reverted cleanly. The step is done when a test alert produces exactly one notification through the path that remains active.
Sending only some alerts through a contact point
Forwarding every Grafana-managed alert to a shared Alertmanager or Mimir makes that system responsible for all of them. The decisions below control which alerts leave Grafana, how they are selected, and what happens when a single alert could be handled in two places.
Select alerts by label
Decide which labels mark an alert as eligible for the external Alertmanager, such as a team, environment or a dedicated forwarding label set on each rule. The contact point and notification policy then match on that label, so new rules are included or excluded by design rather than by remembering to edit a routing list. Check that the label is set by the rule itself and cannot be dropped by a template or a provisioning file. Test one alert from each group before relying on the selection.
Separate critical from low-severity alerts
Paging alerts usually belong in the shared Alertmanager, where grouping, inhibition and silences already exist for that workload. Low-severity alerts that only post to chat may be better kept in Grafana's own routing. Agree the severity labels with the on-call owners first, then map each level to one destination. Confirm who can silence alerts in each system, since a silence in one place does not affect the other.
Keep log alerts apart from metric alerts
Alerts built on Loki log queries often have different volumes, labels and grouping needs than metric alerts. Sending them to the same Alertmanager can mix unrelated groups and make inhibition rules hard to reason about. Decide whether log alerts stay in Grafana or go to a separate receiver, and confirm that their label sets are stable enough to group on. Review how often these rules fire before adding them to a shared route.
Avoid double routing
An alert that matches both a forwarding contact point and a local route can notify the same team twice. Define one owner for each alert: either the external Alertmanager or Grafana's notification policy tree handles it. Write the matching rules so they do not overlap, and check the notification history after a test. Duplicate notifications also appear in high-availability setups, so this check should be part of the rollout plan.
Plan for alerts that stay in Grafana
Some alerts, such as those for dashboards or business metrics, may never need to leave Grafana. Keeping them local reduces the load on the shared system and keeps ownership with the team that wrote the rule. Document which alert types stay local, and review the list when new data sources are added. This also limits the size of the shared Alertmanager configuration that other teams depend on.
Questions teams ask before forwarding alerts externally
What drives the cost of forwarding Grafana alerts to an external Alertmanager?
The cost is driven by how many alert sources, teams and routing rules must be designed, plus the infrastructure and people who run the receiving side. Most of the one-time cost is planning and configuration: deciding which alerts leave Grafana, mapping labels to routes, and testing that each receiver gets the right pages. The recurring cost is the running Alertmanager, the storage behind it if you add Mimir, and the staff time to keep routes current. Prometheus itself is licensed under Apache-2.0 with no paid enterprise edition, so the spend is infrastructure, implementation and operations. Ask any partner to split the quote into design, build, testing and handover, and to state which parts recur.
How long does a typical external Alertmanager setup take?
Duration depends on scope rather than on a standard timeline. A single shared Alertmanager receiving alerts from one Grafana instance is a smaller job than several Grafana instances, a Prometheus fleet and a Mimir deployment that must agree on routing and read-only behavior. The schedule also grows with the number of teams, receivers such as paging and chat tools, and the testing needed before on-call relies on the routes. Ask for a timeline tied to a written scope with named deliverables, and check which tests are included in it.
What does the work need from our team?
You need owners for alert design, on-call routing and the receiving platform. Your team should supply the list of teams and escalation paths, the label conventions used on rules, access to the Grafana, Prometheus or Mimir environments, and a named approver for the notification policy. Someone must also decide which alerts stay in Grafana and which are sent to the external Alertmanager, because that choice changes who gets paged. Plan for at least one person who can run test alerts and confirm receipt on every channel.
Can our own engineers set this up without a partner?
Yes, if they have experience with Grafana Alerting, Alertmanager routing and the platform that will receive alerts. The configuration itself is not the hardest part. The risks are duplicate or missing notifications, rules that are edited in the UI after provisioning, and receivers that were never tested under failure. Teams without recent Alertmanager work often benefit from a fixed-scope review of the design before go-live. Whichever path you choose, document the routing tree and test each route before on-call depends on it.
What breaks most often when alerts are sent to an external Alertmanager?
The most common problems are label mismatches that send alerts to the wrong route, duplicate notifications when more than one sender reaches the same receiver, and silences or mute timings that exist in one system but not the other. Teams also find that Grafana's option to receive Grafana-managed alerts changes which features are available in the external system. Read-only limits in Prometheus can surprise teams that expected to edit rules there. Test each failure path, including an unreachable Alertmanager, before relying on the setup.
Does forwarding alerts externally change our security exposure?
It can. Each Alertmanager endpoint, webhook and receiver carries alert content, which may include hostnames, service names or customer context. Review who can read alert payloads, how the connection between Grafana and the receiver is authenticated and encrypted, and where logs of those notifications are kept. Prometheus's security model is documented as assuming a trusted network, so confirm what protection your network and proxies add. Include these checks in the design review rather than treating them as a later task.
How do we ask for a fixed-scope quote?
Send a written scope that lists the Grafana and Prometheus or Mimir versions in use, how many alert rules and teams are involved, which receivers must be reached, whether Grafana alerts will be received by the external Alertmanager, and which alerts must stay local. Ask for deliverables, acceptance tests, the assumptions behind each line item and what is one-time versus recurring. DeployOpen does not publish a price list, and most engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts. Ask the partner to confirm what is excluded, such as ongoing support or upgrades, which can be handled as a retainer.
How DeployOpen scopes and delivers Grafana alert forwarding
DeployOpen starts with a scoping pass over the alert estate before anyone changes a configuration. The first output is an inventory of which Grafana-managed rules exist, which of them already notify through Grafana's own notification policies, which teams own them, and which Prometheus or Mimir rules would be affected if Receive Grafana Alerts is switched on. The scope then records the target Alertmanager or Mimir tenant, the label set that will select forwarded alerts, and the decisions the client has to make, such as whether a shared Alertmanager is the right home for every team's alerts or only for a subset. Those decisions are written down because they determine how much of the work is configuration and how much is testing.
Once the scope is agreed in writing as a fixed-scope engagement, DeployOpen implements the forwarding path in the client's own infrastructure and tests it end to end. Testing covers sending representative firing and resolved alerts through each contact point, checking that routing and grouping behave as the client expects, and confirming that Prometheus or Mimir read-only limits do not block the intended workflow. The deliverables are the configuration, the test evidence and a runbook that explains who changes what. Items the client must provide, such as access to the Grafana instance, the target Alertmanager endpoints and a named owner for each team's alerts, are listed in the scope so that delays are visible early. Exact timelines and effort are set in the quote from that scope, not assumed in advance.
After handover, ongoing support is available as a retainer, which covers reviewing new alert rules for forwarding impact, applying Grafana and Alertmanager upgrades, and re-testing routes after changes to the notification policy tree. Whether a retainer is worth buying depends on how often the alert estate changes and who on the client side can absorb that work. Ask DeployOpen to state in the proposal which services are one-time and which recur, what the response expectations are for the retainer, and which tests are rerun after each upgrade. Confirm with your own security and operations teams where alert payloads may travel, because forwarded alerts can carry label and annotation content from your environment.
