How Grafana notification policies send each alert to the right team
A shared self-hosted Grafana often sends alerts for dozens of services, and pages either reach the wrong on-call group or land in one default contact point that nobody watches closely. The fix is a notification policy tree that reads the labels on each alert instance and decides who gets it. In Grafana Alerting, the policy tree starts at a root policy, which is the default route for anything no child policy matches. Each child policy matches on labels such as team, service or severity, sends the alert to a contact point, and can nest further policies for finer splits. Routing decisions are made from the labels the rule attaches to each alert, so the label design on your rules matters as much as the policy tree itself. Agree on a short, fixed label set (for example team, service, environment and severity) before anyone writes rules, and confirm in your own instance which labels your rule templates and folders already set.
Routing is separate from rule evaluation. Grafana still evaluates each rule on its schedule and records whether the condition is firing, and the notification policy only decides where a firing or resolved alert goes afterward. This separation means you can change who gets paged without editing or re-testing the rule, and it means a broken route does not stop evaluation. It also means a route can be silently wrong while the rule works perfectly, which is why routing needs its own review. Design routes deliberately once more than one team owns alerts on the same instance, once any alert is paging out of hours, or once a single default contact point starts receiving alerts that a specific team needs to act on quickly.
Before relying on a tree for on-call, test it. Use the policy preview in the Grafana Alerting interface with sample label sets that match real services, including alerts with missing labels, since those fall through to the root policy. Confirm that the grouping, timing and repeat settings on each policy match how your responders expect to be interrupted, and check that every label value a rule can produce has a destination. The official Grafana alerting documentation describes the current route options for your version, so confirm the exact fields and preview behavior there before you write the design down.
Routing options and when each one fits
This table compares the routing building blocks in Grafana Alerting: the default policy, label matchers, nested child policies, grouping, timing intervals and per-policy contact points. Read each row for the team structure it suits and the upkeep it adds, then confirm exact field names and defaults in the alerting documentation for your Grafana version.
| Building block | When it fits | Maintenance cost |
|---|---|---|
| Default policy | One on-call team or a small group that shares every alert | Low, but every unmatched alert lands here, so review it after each new rule set |
| Label matchers | Several teams that own alerts identified by a consistent team or service label | Moderate, since rule labels must be enforced and a missing or misspelled label silently misroutes alerts |
| Nested child policies | Teams with sub-owners, such as a platform team split into database and network groups | Moderate to high, because matcher order and inheritance need testing after every change |
| Grouping (group by labels) | Teams that get bursts of related alerts from one incident and want one notification | Low to moderate, but grouping on too many labels recreates the noise it was meant to remove |
| Timing intervals (group wait, group interval, repeat interval) | Services where first-page speed and reminder frequency differ by severity | Low, but values set per policy need a written owner so they do not drift between teams |
| Per-policy contact points | Routes that must reach a specific channel, such as a paging tool for critical alerts only | Moderate, since each contact point needs credentials, rotation and a test notification |
| Routing without label design | Small installs that accept manual triage while the team is still learning the alert set | Low at first, but it becomes costly as rule count grows and ownership questions multiply |
Label design on rules decides where each alert lands
In Grafana Alerting, a notification policy does not read the name of an alert rule. It matches the labels attached to each alert instance, so the label set is the routing contract between your rules and your on-call structure. Before you build a policy tree, agree on a small, fixed set of labels such as team, service, environment and severity, and make every rule carry them. A rule named "Checkout latency high" routes to the checkout team only if it has a team label with the right value. Without one, it falls through to the default policy, which is usually the wrong place for a page.
Most routing failures come from label hygiene rather than from policy logic. Inconsistent names such as team, Team and owner for the same concept split alerts across matchers that never fire. Labels added in some queries but not others produce alert instances that a matcher on the expected label never sees. High-cardinality labels, such as request IDs, pod names that change on every deploy or user identifiers, multiply alert instances and fragment grouping, which makes notifications noisy and hard to silence. Keep routing labels low in cardinality and put detail in annotations, where it helps responders without affecting matching.
A naming standard works only if it is enforced at the point of rule creation. Write the allowed keys and values down, decide who owns the list, and check it during review of rules created in the UI and in provisioning files. Then run a routing test with a sample alert for each team before relying on the policy for on-call. Ask your team to confirm how the default policy is used in your instance and to report any alert that lands there, since that count is a useful early signal that labels are drifting. The exact label behavior of your Grafana version should be confirmed against the current alerting documentation for your release.
Routing layouts by team and environment structure
A single on-call rotation receives most alerts, and the tree separates them by severity. The root policy sends everything to one default contact point, and child policies match a label such as severity=critical to paging and severity=warning to chat. Grouping is kept coarse so one incident produces one page rather than many. This layout suits a small platform team that owns most services and wants a short tree with few moving parts.
- Root policy: The default route targets the on-call paging contact point, so unlabeled alerts are never silently dropped.
- Severity children: Illustrative matchers such as
severity = criticalroute to a paging receiver, whileseverity = warningroutes to a team chat channel. - Shallow nesting: Two levels are usually enough, which keeps the tree readable during an incident.
- Shared grouping: Grouping by
alertnameandclusterlimits page volume, though it can merge unrelated alerts if labels are sparse. - Single maintainer: One team owns severity rules, so changes are quick but depend on that team's capacity.
The layout breaks down when services outnumber the people who understand them, because every new service adds severity decisions that only the shared team can judge. Check that every rule carries a severity label and that the default route is tested.
Each owning team gets its own branch in the tree, selected by a label such as team=payments or owner=identity. Rules carry the owner label, and the branch sends alerts to that team's contact points. Severity handling can sit underneath each branch, so a team controls both who is paged and how urgently. This layout fits organizations where service ownership is clear and teams run their own on-call.
- Ownership label: Illustrative values such as
team = paymentsmust be enforced on every rule, ideally through shared templates. - Team branch: Each branch holds its own contact points and optional child routes for severity or escalation.
- Nested severity: A team can split its branch with
severitymatchers without affecting other teams. - Independent timing: Teams set their own group wait and repeat interval values within their branch.
- Distributed maintenance: Teams edit their own branches, so a shared review step is needed to catch conflicts.
This layout strains when labels drift, for example when a rule uses team=Payments in one place and team=payments elsewhere, and the alert falls through to the default route. Ask who approves new branches and how unlabeled rules are caught before they reach production.
The top level of the tree splits on an environment label such as env=prod or env=staging. Production has its own branches, contact points and timing, while non-production routes to chat or a low-priority receiver. This keeps test noise away from on-call and lets staging rules change without touching production paging. It is often combined with one of the other layouts beneath each environment branch.
- Environment matcher: The first level uses an illustrative
envlabel, so every rule needs it set by the template or the data source. - Restricted paging: Only the production branch can reach paging receivers, which limits accidental pages from test systems.
- Lighter non-production policy: Staging routes can use longer repeat intervals and send to chat only.
- Separate review: Production changes go through a stricter approval step than staging changes.
- Duplicate trees: Maintaining two similar trees doubles the work unless both are generated from one source.
The risk is drift between the two trees, so a production fix that never reaches staging can hide a routing bug until an incident. Confirm whether your team keeps both trees in version control and how they are tested before production changes.
Build and test the notification policy tree in order
This sequence takes a Grafana alert from a rule label to a confirmed contact point, so each team receives its own alerts and nothing falls through to an unowned route. Each step ends with a check you can repeat before the policy tree is trusted for on-call.
Agree the label standard first
Before touching the policy tree, the platform team and the service owners agree which labels every alert rule must carry, such as team, severity and environment, and which values are allowed. Write the list down with one owner who approves new label values. The output is a short label reference that every rule author follows. You know it is done when a sample rule written from the reference carries all required labels and the reviewer signs off on the naming.
Confirm contact points and send test notifications
In the Grafana alerting contact points page, create or confirm one contact point for each receiving team and for the catch-all destination, then use the test action on each one. Check that the message arrives in the right Slack channel, mailbox or paging service, and that the message names the contact point. A route cannot be trusted if the destination itself is untested, so record which contact points passed.
Set the default policy as a safe catch-all
Edit the default root notification policy so it points to the catch-all contact point that a human reads during working hours, such as a shared operations channel. Leave grouping at the default values until the branches exist. Any alert without a matching child policy lands here instead of being dropped. You know it is done when a rule with no team label appears in that channel after you fire a test alert.
Add team branches from broad to specific
Under the root policy, add one child policy per team, matching on the team label from the standard. Where a team needs different handling for critical alerts, add a nested policy beneath that team's branch, matching on severity, and keep the more specific match below the broader one. Order matters because Grafana evaluates child policies in sequence, so review the tree from top to bottom. The check is a printed or exported view of the tree that matches the design document line for line.
Configure grouping and timing for each branch
For each team branch, set the group-by labels, group wait, group interval and repeat interval according to how that team wants to be interrupted. Keep grouping labels few, since too many labels create separate notifications for near-identical problems. Document the chosen values beside each branch, since changing them later changes how often responders are paged. You know it is done when the settings match the agreed table and the table is stored with the policy design.
Send test alerts with sample labels through each route
Create a test rule, or use a test alert tool that your Grafana version offers, with sample label sets for each team, each severity and one label set that matches no branch. Let the alert fire and open the alert instance view to see which policy matched and which contact point received the notification. Run the same tests after each policy edit. The output is a test matrix with the expected and actual route for every label set, and any mismatch must be fixed before the tree is used for paging.
Confirm the tree before on-call depends on it
Ask a member of each receiving team to fire one labeled test alert and confirm they receive it on the channel they will use while on call. Confirm that the catch-all still receives unlabeled alerts and that the escalation path for paging services works outside business hours. Store the test matrix and the final policy export in version control or your change record, so the next edit can be tested against the same cases. The sequence is complete when every team has confirmed its route and the matrix shows no unexpected matches.
Testing routes before on-call depends on them
A notification policy tree is only trustworthy once you have watched each route deliver to the right place. Start with the contact points themselves: send a test notification from every contact point in Grafana Alerting so you know the channel, credentials and destination work before any routing decision depends on them. Then test the tree itself. In the notification policy preview, enter a set of label combinations and confirm which policy each one matches, which contact point receives it and whether the grouping and timing values you set are the ones that apply. Treat this as a check of the tree's logic, not of delivery.
A route that passes with labels you typed by hand can still fail on real rules. Build the test set from the labels that current alert rules actually carry: export or copy the label sets from your rules, include the severity, team and service values exactly as they are spelled, and add the combinations that cause trouble, such as a rule that has no team label or one that uses a label name you renamed last quarter. Confirm that the default policy catches anything unmatched, because an alert that falls through to the root with no owner is the failure on-call will notice last. Run the same checks against a staging instance if your rules are provisioned from files.
Repeat this review after every label or team change, not only at initial setup. A new rule without the expected label, a renamed team or a nested policy added above an older one can silently shift alerts to another receiver. Keep a short written record of the test set and the date it passed, and ask your team to confirm which owner each label value maps to before they rely on the tree for paging. Whether your organization needs that record for an audit depends on the framework and its controls, so confirm the requirement with your assessor rather than assuming it.
Grouping and timing settings that control page volume
Grouping and timing settings decide how many notifications a responder receives for the same underlying problem. Each setting trades a faster first alert against fewer repeats, so change them together and test them against realistic alert bursts before on-call relies on them.
Group by labels
Group by labels decides which alert instances are bundled into one notification. Grouping by a broad set such as alertname and cluster sends one message for a fleet-wide failure, while grouping by instance sends one message per host. Choose labels that match how the owning team responds, and check that every rule carries them, because an alert missing a grouping label falls into a catch-all group that is hard to triage.
Group wait
Group wait is the delay before the first notification for a new group, so that related alerts arriving within the window can be combined. A longer wait reduces the number of initial messages during a cascading failure, but it delays the first page by the same amount. Confirm the value in your Grafana version's docs and set it lower for alerts where minutes of delay matter, such as customer-facing outages.
Group interval
Group interval controls how long Grafana waits before sending an update for a group that already has notifications and has changed, such as new alerts joining it. A short interval surfaces new members quickly but can produce a steady stream of updates during a flapping incident. A longer interval keeps the channel quieter at the cost of slower visibility into growth, so review it against how often your rules flap.
Repeat interval
Repeat interval sets how often Grafana resends a notification for an alert that is still firing and has not changed. Short repeats keep an unacknowledged page visible, but they multiply noise for long-running issues that a team already knows about. Pair the repeat interval with silences for known problems, and confirm in your version's docs how repeats interact with resolved notifications before you set it for paging contact points.
Mute timings and silences in routing
Mute timings suppress notifications on a schedule, such as outside business hours, while silences suppress them for a specific set of labels during a planned change. Both stop notifications without stopping rule evaluation, so the alert state is still recorded. Attach them to the right policy node, test that a matching alert is actually muted, and keep a record of who created each silence so that a suppressed page can be explained later.
Questions teams ask before routing alerts to on-call
How much effort does building a Grafana notification policy tree take?
The effort depends less on Grafana's configuration screens and more on the decisions behind them. Most of the time goes to agreeing which teams own which services, which severities page someone at night, and how alerts are grouped so one outage does not produce a flood of pages. Cost drivers include the number of teams and services, how many alert rules already exist without labels, how many contact points must be created and tested, and whether the tree needs to be reviewed by security or compliance owners. Ask for a written scope that lists the label standard, the number of routes, the contact points in use, the test cases you will run, and who signs off before the tree goes live.
Can our internal team build the routing tree without outside help?
Yes, many teams do, especially when one platform group already runs Grafana and understands the on-call rotations. The work is mostly configuration and decision-making, so the main question is capacity and experience with alert design. Internal teams often struggle with three things: getting service owners to agree on labels, testing routes without sending real pages, and keeping the tree documented after the first build. An outside partner can take on the design workshops, the test plan and the handover documentation under a fixed scope. Whichever route you choose, confirm who will own the tree after go-live.
What does the work need from our own teams?
The build depends on inputs only your teams can provide. You need a label standard that every alert rule uses, such as a team label, a service label and a severity label, along with a written definition of each value. You also need named contact points for each team, such as a chat channel, a paging service or an email address, and a named owner for each rotation who can confirm the escalation path. Ask for these inputs to be listed as dependencies in the written scope, with a date by which each owner must respond. Missing owners are the most common reason routing projects stall.
How long does a first working routing tree take to settle?
Timing is driven by decisions rather than a fixed duration. A first tree is usually ready once every team has agreed on its labels, the top-level route has a default receiver, and each severity that pages has a tested path to a human. Expect more time when ownership is unclear, when alerts lack labels, or when several teams must approve the same escalation. Ask a prospective partner to describe the decisions that gate each milestone, and ask your own teams to commit to named review sessions. Avoid committing to a calendar date before the label standard is approved.
Can notification policies and routes be provisioned as code?
Yes. Grafana provisioning loads alerting resources from YAML or JSON files at startup, and the Grafana provisioning documentation also lists further as-code options, including Terraform and Git Sync. Keeping the policy tree in version control lets you review changes, roll back a bad route and reproduce the same setup in a second environment. Confirm the exact file format and tooling against the current Grafana provisioning documentation for your version, and check whether anyone edits routes in the UI, since UI edits can drift from the files. Ask for the provisioning approach to be stated in the scope.
What changes when Grafana runs in a high availability setup?
High availability uses two or more active-active Grafana servers behind a load balancer that share a MySQL or PostgreSQL database. The Grafana high availability documentation notes that alerting needs additional configuration to avoid duplicate notifications, so the routing design must account for every node that evaluates and sends alerts. Your engineers should confirm the clustering settings for their version, test that a single alert produces one page when a node fails, and monitor cluster health. Ask the scope to include a failover test and a duplicate notification test, not only a configuration review.
What should we ask for in a written scope for routing work?
Ask for a document that names each deliverable and its acceptance test. It should list the label standard, the policy tree diagram, every contact point and its owner, the grouping and timing settings with the reasons for each value, the test cases for every route including a failed contact point, and the provisioning method. Also ask who performs each test, what evidence you will receive, how changes after go-live are requested and approved, and what documentation you keep. A clear scope makes costs comparable across quotes and shows which work remains with your team.
Who can change routes and how changes stay auditable
A notification policy tree decides who gets paged, so the people who can edit it and the way those edits are recorded matter as much as the routes themselves. The cards below cover access, secrets, change history, UI drift, team changes and endpoint reachability, which together make a route change something you can review, reverse and explain after an incident.
Restrict edit rights on policies
Only a small group should be able to change the notification policy tree, because one wrong matcher can send every critical page to a silent channel. In self-hosted Grafana, limit the Editor and Admin roles on the alerting area and confirm that viewers cannot open the policy editor. Ask who holds admin rights on the Grafana organization, since those users can change routing regardless of folder permissions.
Keep contact point secrets out of config
Webhook URLs, PagerDuty routing keys and Teams connector addresses act as credentials. Do not paste them into plain provisioning files committed to a repository. Reference them through environment variables or a secrets store, and confirm which fields Grafana encrypts at rest and which it returns to users with edit rights. Rotate any key that appears in a shared ticket or screenshot.
Record changes in version control
A change made in the UI leaves little trace beyond the instance's own history, so route changes should flow through a repository with reviewed commits. Store the notification policy file in Git, require a pull request for changes, and apply it with file provisioning or the Grafana Operator. The commit history then answers who changed a matcher, when, and why, which is the question an audit or post-incident review asks first.
Handle UI edits after provisioning
Once a policy tree is provisioned, edits made in the UI can drift from the repository and be overwritten on the next apply, or persist until someone notices. Decide upfront whether provisioned resources are locked in the UI, then test the behavior on your Grafana version. Schedule a periodic comparison between the live policy and the file in Git so drift is found by a check, not by a missed page.
Review routes after team changes
Routes that point to a team name go stale when people or teams move, and a matcher on a retired label can silently catch nothing. Tie route reviews to org changes, service ownership transfers and on-call rotation updates. Check that each leaf policy still maps to a live contact point with a current escalation owner, and remove routes for services that no longer exist.
Verify paging endpoints from Grafana
A correct route still fails if the Grafana server cannot reach the paging provider, often because of egress rules, proxies or a changed DNS name. Confirm outbound connectivity from every Grafana node, including each node in a high availability pair, to each paging and chat endpoint. Send a test notification through each contact point and record the result, so the failure is found during a change window rather than during an outage.
What to confirm in your environment before relying on routes
Before any route goes live for on-call, confirm how your own Grafana version evaluates matchers and grouping. Ask whether the label matchers on your notification policies behave the way your test cases assume, and whether the grouping fields you chose (such as grouping on the policy tree) are supported and applied as you expect on the exact release you run. Grafana ships minor releases every other month and patch releases at least monthly, so a route tested on one release should be retested after an upgrade. Check the release notes for your version, and rerun your route test after every upgrade window.
Confirm which database holds your Grafana configuration. The installation docs say SQLite is the default and does not scale for production, and that MySQL or PostgreSQL is required for production and high availability. Ask whether your instance is on one of those supported databases, what version it runs, and whether the same database is shared by every node. A route that works on a single SQLite instance can behave differently once several nodes share a database, so record the database type alongside your test results.
If you run more than one Grafana server, confirm how the cluster avoids duplicate notifications. The high availability docs note that alerting needs additional configuration for this, so ask who on your team owns that configuration and whether it has been tested with a real alert firing on every node. Then ask whether an external Alertmanager is involved, either as a forwarding target or as the place where some routes are actually decided. If one is, confirm which system owns each route, because a policy tree in Grafana and a tree in Alertmanager can both match the same alert and send it twice or not at all.
How DeployOpen scopes, builds and supports alert routing
DeployOpen treats notification policy design as a written scope with its own test plan, not a set of UI clicks. We plan the routing tree against your teams and on-call rota, build it in your self-hosted Grafana instance, and hand over documentation your team can maintain.
How we scope
- Map every team, service and on-call rota that must receive pages before any policy is drawn
- Agree the label schema that rules must carry, such as team, severity and service, in writing
- Define which alerts page, which go to chat and which are only recorded, with owners named
- Confirm contact point types and any external Alertmanager in the written scope
How we build
- Create the nested policy tree with default routing so no alert falls through unrouted
- Set group-by keys, group wait, group interval and repeat interval for each team's paging needs
- Write test alerts with sample labels and check each route reaches the intended contact point
- Review mute timings and silences as part of routing so maintenance does not reroute pages
How we hand over
- Document the policy tree, label conventions and the reasoning behind each timing setting
- Keep rules and policies in files or version control where your environment allows it
- Walk your on-call leads through a routing test and the steps to add a new team or service
- Record which settings were changed in the UI after delivery so drift is visible to your team
How we support
- Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts
- Ongoing support, upgrades and routing reviews are available as a retainer
- Re-test routes after Grafana upgrades, contact point changes or new services are added
- Confirm with your team how routing changes are approved before they reach on-call
