Why customizing Grafana alert notification messages matters for on-call
An on-call engineer receives a page that says only that a rule fired, with no service name, no environment and no link. Notification templates fix this. They control the title, body and labels that Grafana Alerting sends to each contact point, so a responder can tell what broke, where it broke and how urgent it is before opening a laptop. Teams need custom templates once the default message stops being enough to act on, which usually happens as soon as several services, clusters or environments share one Grafana instance and one set of contact points.
Grafana builds each notification from Go templates that read the alert's labels, its *annotations* and the values returned by its queries. A useful page therefore names the failing system, the environment such as production or staging, and the severity, all drawn from labels set on the rule. It also carries a link to the dashboard panel that shows the problem and to the runbook that says what to do next. Annotations are the natural home for the runbook link and a short human summary, while labels should stay consistent across rules so that routing and templates can depend on them. Teams should agree on this label and annotation contract before writing any template.
Before you start, confirm which fields your current rules populate and which are blank, then check how the message renders on the mobile app, since a long body is often truncated on a pager screen. Ask who owns the label conventions, who reviews template changes and how a broken template is detected before it reaches a page at night. Tracking these decisions early keeps the templates short, consistent and useful when the alert fires for real.
Template options and when each fits
This table compares the four ways to shape Grafana alert text: default templates, named custom templates, per-contact-point overrides and rule annotations. Read each row for the maintenance effort, consistency risk and the team size where the option makes sense. The main takeaway is that shared named templates give multi-team organizations consistent messages, while annotations and overrides suit a small team or a single service.
| Option | Maintenance effort | Consistency and drift risk | When it fits |
|---|---|---|---|
| Default templates | Low, nothing to write or review | Consistent but generic, so responders may need to open the instance to find context | A small team with few rules that accepts the built-in message layout |
| Named custom templates reused across contact points | Moderate, one template file to version and review | Low drift, since every contact point that references the template gets the same layout | A multi-team organization that wants one message standard across chat, email and paging |
| Per-contact-point message overrides | Moderate to high, each receiver's text must be kept in step | High, a copy in one receiver can silently fall behind the shared layout | A single team sending to one channel that needs a different format for a specific tool |
| Annotations set on the rule | Low to moderate, edited alongside each rule | Medium, text varies by rule author, so confirm a shared annotation convention before rollout | A small team or service owner who wants runbook and dashboard links tied to one alert |
| Label and title conventions | Low once agreed, but needs an owner | Low if labels are standardized across rules, high if each team invents its own names | Organizations where routing and message text depend on the same label set |
| Mobile paging length | Ongoing review as alerts change | Long titles get truncated on pagers, so test each receiver's output | Any team paging from short notifications on phones |
| Version control of templates | Moderate, needs a review step | Low when templates live in Git with file provisioning, high when edited only in the UI | Organizations that must show who changed a message and when |
What a responder needs in the first lines of a page
A page preview on a phone shows only a few lines before the responder decides to act. The cards below list the elements that belong in that short title and first body lines, and what should move to the expanded body or annotations so the page stays readable under pressure.
What broke
The title should name the failing condition in plain words, such as "High error rate on checkout API", not the internal rule name. Responders decide in seconds whether to act, so the condition must be understandable without opening the rule. Put the query text, rule UID and evaluation details in the expanded body. Ask your on-call team to read titles from a phone and confirm each one tells them what is wrong.
Affected service or host
Include the service, cluster or host name in the title or first line, using the same names your responders use in chat and runbooks. Templates usually read these from alert labels such as service or instance. Confirm that every rule carries the label your routing depends on, because a missing label produces a page with no owner. Keep long host lists in the expanded body.
Environment
Production and staging alerts often share a rule, so the environment must appear early or a staging fault can wake someone at night. Use a label such as env and print it before the summary. Decide whether non-production alerts route to chat only. Check that the label is set on every rule, since a blank value is easy to miss in a template test.
Severity
Severity tells the responder whether to wake up, wait for working hours or ignore the page. Print it at the start of the title, for example as a severity label value, and tie it to the routing tree so that only critical alerts use paging contact points. Avoid encoding severity only in color, because mobile lock screens often drop it. Agree the severity scale with the on-call owners before writing the template.
Fired threshold and current value
The value and threshold let responders judge the size of the problem before opening a dashboard. Include the observed value and the limit in the body, for example as a short line derived from the reduced expression, so the number is visible without a click. Keep full query output and unit conversion details in the annotations. Test the formatting with real values, since raw decimals can be hard to read on a phone.
Dashboard and runbook links
Two links carry most of the first response: the dashboard that shows the trend and the runbook that says what to do. Put the runbook link where a responder sees it first, and keep both URLs in annotations such as dashboardURL and runbookURL so they are maintained in one place. Confirm that the links resolve from a phone and through your identity provider's sign-in flow, and that they point to the current dashboard after any rename.
Start time
The time the condition began tells responders whether it lines up with a deploy, a batch job or a maintenance window. Grafana records when an alert started firing, so the template can display that time rather than the notification send time, which changes on each repeat. Decide which time zone the team uses and state it in the body, so responders in different regions read the same moment.
How to build and test a notification template
This sequence takes a notification template from an agreed list of fields to a tested, versioned message that responders can read on a phone and act on. Each step ends with a check so the team knows when to move on.
Agree the fields each team needs
Before writing any template, the on-call lead and the responding teams agree on what a page must say: the service, the environment, the severity, what the alert measures, the threshold, and the first action to take. Write the list down as a short field specification that names each label and annotation the message will use. Reviewers from each responding team should sign off on it. The step is done when every team has confirmed the list and no one is asking for a field that the message cannot carry on a small screen.
Define labels and annotations on rules
Templates can only show data that the alert rules supply, so update each rule to set the labels used for routing and grouping and the annotations used for the message body. Typical annotations are a summary, a description and links to a dashboard and a runbook. Keep label values consistent across rules so the same field means the same thing everywhere. The check is to open a sample of rules in the Grafana UI or your provisioning files and confirm each one carries the agreed fields, with no empty required annotations.
Write the template in a test instance
Create the notification template in a non-production Grafana instance, not in the production contact point configuration. Use Go templating to define a reusable title and body, and reference the labels and annotations from the previous step. Keep the title short and put the most important fact first, since mobile lock screens and push previews often show only the opening words. The check is that the template saves without errors and that a preview of a firing alert shows every expected field populated.
Send test notifications to each contact point type
Route a test alert through every contact point type your teams use, such as email, Slack, Microsoft Teams, PagerDuty or a webhook. Each integration applies its own formatting and length handling, so the same template can look different in each one. Use the test notification function in Grafana and compare the received messages with the expected fields. The check is a written note per contact point confirming the title, severity and links arrived intact and that no field appeared as raw template syntax.
Check rendering on a mobile device
Open the test notifications on a phone using the same push app and account that on-call engineers carry. Look at the lock screen preview, the expanded notification and the link behavior. Confirm the first line states what broke and where, and that the critical action is visible without scrolling far. If the message is cut off, shorten the title or move detail into the annotation the runbook covers. The check is a short sign-off from at least one engineer who reads the page on a phone and confirms the message is actionable.
Promote the template through version control
Store the approved template and the rule changes in your version control system, then deploy them through file provisioning or your chosen as-code tooling rather than editing the production instance by hand. Review the change through the same merge process you use for other configuration. Confirm the deployed instance matches the repository after the change. The check is that a fresh deployment from the repository produces the same template, and that any later edits made in the UI are either captured back into the repository or reverted.
Confirm links resolve
Click every dashboard and runbook link in a production test notification, from a device that is outside the Grafana network if on-call engineers will use one. Confirm each link opens the right dashboard with the expected time range and labels, and that the runbook page is reachable without a separate login. Links that depend on the Grafana external URL setting or on a reverse proxy often break after a move, so test them after every promotion. The check is a list of tested links with the date tested and a named owner for any link that failed.
Keeping messages short enough for mobile paging
The first line of a page does most of the work on a phone. A responder often reads only the lock screen preview before deciding whether to unlock the device, open the incident, or wait. Put the service, the symptom and the environment in the opening words of the title, such as the service name, the failing condition and the cluster or region, and leave counts, timestamps and query text for later in the body. Paging apps do not all cut text the same way. Some show a short title and a clipped body, others collapse several lines into a preview, and some apply their own character limits before your message reaches the device. Do not assume a limit from a vendor blog post or from another tool. Confirm how each receiving tool in your environment truncates titles and bodies, including what happens to the first line of a multi-line body.
Every line you keep out of the page moves to a linked destination, so the trade-off is between what a responder needs to decide on the spot and what they need to diagnose. Keep the page to the facts that set severity and ownership: the rule or alert name, the affected labels, the current value against its threshold, and one link to the dashboard or runbook. Put the longer explanation, query output and remediation steps in the runbook behind the link. Be careful with links, though: a link that opens a generic dashboard forces the responder to search for the panel they need. Ask the owning team which panel and time range a responder should see first, then make the template link to exactly that view. Templates that include the alert's GeneratorURL or a stable runbook URL annotation work well for this, provided those fields are populated for every rule.
Test the message with the paging tool your team actually uses, not with the Grafana preview alone. Send a test notification from each contact point through the production route, then check the lock-screen preview, the expanded view and the incident record on a phone, not only on a desktop browser. Repeat the check with a grouped notification that contains several alerts, because grouped bodies are where truncation most often hides the important line. Record what each tool showed, and rerun the test after any change to the template, the grouping settings or the tool itself. Ask the platform team to confirm the receiving side's limits in writing when you configure the integration, and document them next to the template so the next author does not have to rediscover them during an incident.
Security and operational considerations for alert templates
Notification templates sit between your alert data and the third-party services that page people, so they inherit both the sensitivity of that data and the consequences of a broken template. The cards below pair each risk with one control you can put in place and check before templates reach on-call engineers.
Secrets and hostnames in messages
Annotations and labels often carry internal hostnames, IP addresses, query strings or tokens copied from a data source. Those values then leave your network inside the body of a message sent to a paging or chat provider. Before a template goes live, list every label and annotation it renders and confirm that none carries credentials or internal addressing. Keep the paging payload to the service name, severity and a short summary, and let responders open the full detail behind your single sign-on.
Who may edit templates
Templates run on every notification that matches a route, so an edit can change what thousands of pages say without anyone noticing. Decide which roles can create or change templates in the Grafana UI, and whether those changes should be made through version-controlled files instead. Ask who owns the on-call notification path and confirm that the people editing templates are the same people who answer the pages they write.
Reviewing template changes
A template change is a code change even when it is a single line of Go template syntax. Route template edits through the same review as other infrastructure changes, with a second reviewer who reads the rendered output for a firing and a resolved example. Store the reviewed templates in Git and deploy them with the same pipeline used for alert rules and contact points, so the running configuration can be traced to an approved commit.
Runbook links and internal documentation
A runbook link in an annotation is useful to a responder, but it also tells anyone who reads the message where your internal documentation lives. Some paging providers store message bodies and show them on mobile lock screens. Limit links to runbooks that are safe to reach from a phone, require sign-in on the wiki behind them, and confirm that the links resolve for on-call staff who have no prior access to the alerting system.
Template errors that suppress alerts
A template that fails to render can stop a notification from being sent, which turns a broken message into a silent alert. Test each template against sample alerts for firing, resolved and missing-label cases before deployment, and check the Grafana alerting error and notification metrics after every change. Confirm your escalation path still fires when a template is deliberately broken in a test environment, so you know the failure would be noticed.
Message exposure in shared channels
Chat channels often have wider membership than the team that owns an alert, and contractors or auditors may be members of channels they should not read. Check who belongs to each Slack, Teams or paging destination before adding detail to a template. Keep sensitive customer or system identifiers out of shared channels, and send the detailed variant of a message only to channels where the audience matches the data classification of the alert.
Questions teams ask before customizing alert messages
How much effort does customizing Grafana alert notification templates take?
The effort depends on how many teams receive alerts, how many contact point types they use, and how consistently alert rules are labeled and annotated. A single team paging into one channel with well-labeled rules is a small, bounded task. Multiple teams, several destinations such as chat, paging and ticketing, and rules with inconsistent labels multiply the number of templates and test cases. Ask any partner to break the work into template design, per-destination formatting, and test alerts, so each part is visible in a written scope before work starts.
What drives the cost of notification template work up or down?
Four factors move the cost most:
- The number of teams and routing paths that need different message content.
- The number of contact point types, since each destination formats titles, bodies and links differently.
- Labeling discipline on existing rules, including whether severity, service and owner labels are present.
- Whether runbooks and dashboards already exist with stable URLs that templates can link to.
Ask for a quote that lists each of these as a line item, along with what is one-time work and what is handled as ongoing support.
Can our own team write Grafana alert templates without outside help?
Yes, many teams do, especially when one engineer owns the alerting setup and can test changes in a non-production instance. Templates use Go templating against alert labels and annotations, so the work is learnable, but mistakes often surface only when a page fails to render on a phone. Internal teams usually need time to build test alerts and review messages across each contact point. If the team has no one who can own this over time, an outside partner can design the templates, document them and hand over a test procedure.
What does a partner need from our team to customize alert messages?
Plan to provide access and information before design starts:
- Read access to the Grafana instance, including the alerting configuration and existing contact points.
- Sample alert rules and recent notifications so message content reflects real alerts.
- Settings for the paging and chat tools, such as which fields they display and any length limits.
- The names and URLs of dashboards and runbooks that messages should link to.
- A named person who can approve message wording for each team.
Confirm which of these your security policy allows to be shared, and whether test notifications may reach live on-call channels.
How long does testing notification templates take?
Testing time depends on scope: the number of templates, the number of destinations, and how many alert scenarios must be checked. Each scenario, such as a firing alert, a resolved alert, a grouped alert with several instances, and a long label value, needs to be sent and inspected in every destination. Ask for the test plan to be written into the scope with the scenarios listed, and confirm who signs off on the results. Testing in a non-production contact point before touching on-call routing is a common requirement.
Do notification templates need updating after a Grafana upgrade?
They can. Grafana releases ship on a regular cadence, and the upgrade guidance describes monthly patch releases and yearly major releases. Template rendering and the fields available to templates can change between versions, so a message that worked before an upgrade may render differently afterward. Review templates as part of each upgrade, re-run the test alerts, and check the release notes for alerting changes. Plan this as recurring work, and confirm with any partner whether it is included in a support retainer.
Will customized messages cause our alerts to stop reaching on-call staff?
A template error can produce a blank or broken message, and in some setups it can cause a notification to fail. That risk is why templates should be tested against each live contact point type before rollout, and why a rollback plan matters. Keep the previous template version in version control, change one destination at a time, and confirm that a test page reaches the right responder. Templates change message content, but routing decisions are handled separately in notification policies.
How DeployOpen scopes and delivers Grafana alert message templates
DeployOpen starts with a written scope, because most of the effort in alert message work is agreeing on content with each receiving team before any template is written. The scope lists every alert rule and contact point in scope, the notification policy routes that send them to teams, and the receiving tools, such as chat channels, paging services and ticketing email. For each team, the scope records the agreed message content: the title line, the labels that identify the affected service and environment, the annotations that carry the summary and the dashboard and runbook links, and the length each channel can comfortably show on a phone. Most engagements are fixed-scope, so the deliverables, the number of templates and the acceptance checks are agreed in writing before work starts, and a quote follows from that written scope.
The build happens in your own Grafana instance, using the notification template editor or template files loaded through provisioning, so the templates live with the rest of your alerting configuration. Each receiving tool then gets a test send from a representative alert, and the reader checks that the title stays readable in a paging notification, that the links resolve for the on-call engineer, and that resolved notifications read correctly. Handover includes documentation of each template and its variables, and the templates are kept in version control so changes can be reviewed like other code. Ongoing support, template updates after new services or contact points are added, and periodic reviews are available as a retainer. Before you sign, ask the partner to show which template and provisioning files the handover will contain, and confirm which of your receiving tools you will test against and who signs off each test send.
