Grafana notification policies and alert routing
Building a notification policy tree that sends each alert to the right team: label design on rules, nested policies, grouping and timing settings, and testing routes before relying on them for on-call.
Grafana Alerting
Designing and running Grafana Alerting in a self-hosted instance: rules, notification routing, contact points, silences, high availability and external Alertmanagers.
Grafana contact points for Slack, Microsoft Teams and PagerDuty
Configuring the contact points most teams use: Slack and Microsoft Teams channels, PagerDuty and Opsgenie for paging, email and webhooks into ticketing, and which integrations Prometheus Alertmanager cannot send to.
Grafana silences and mute timings for maintenance windows
Stopping notifications without stopping evaluation: one-off silences for maintenance, recurring mute timings for business hours, who should be allowed to create them and how to keep a record for audits.
Customizing Grafana alert notification messages
Writing notification templates that tell responders what broke and where: titles, labels and annotations, links back to dashboards and runbooks, and keeping messages short enough for mobile paging.
Grafana Alerting high availability and duplicate notifications
Running Grafana Alerting across several instances: memberlist gossip on port 9094 or Redis, ha_advertise_address on Kubernetes, single node evaluation to cut query load, cluster health metrics and why duplicates can still happen.
Using an external Alertmanager with Grafana
Forwarding Grafana-managed alerts to Prometheus Alertmanager or Mimir: when a shared Alertmanager makes sense, enabling Receive Grafana Alerts, read-only limits with Prometheus, and sending only some alerts through a contact point.
Grafana alerts on Loki log queries
Alerting on log patterns and error rates from Loki with Grafana-managed rules: turning log queries into numeric conditions, per-label alert instances, keeping query cost down and routing log alerts separately from metric alerts.
Reducing Grafana alert noise and alert fatigue
Cutting the volume of pages from Grafana: grouping related alerts, notification timing, pending periods, silences and mute timings, routing low-severity alerts to chat instead of paging, and reviewing alerts nobody acts on.
Managing Grafana alert rules as code
Keeping alert rules, contact points and notification policies in version control with file provisioning, the Grafana Operator or Terraform, and handling rules edited in the UI after provisioning.
