Why teams manage Grafana alert rules as code
Teams manage Grafana alert rules as code because alert rules, contact points and notification policies built by hand in the UI drift between environments and leave no review trail. A rule tuned in staging may never reach production, a routing change made during an incident may never be written down, and nobody can say who changed a threshold last quarter. Grafana supports file-based provisioning for alerting resources, so rules, contact points and policies can be defined in YAML or JSON files and loaded at startup. The provisioning documentation also describes further as-code options, including Terraform and Git Sync, which you can read in the Grafana provisioning docs.
The case for as-code grows with three conditions: more than one Grafana environment, an audit requirement for who changed alerting and when, and on-call routing that changes often. If the same rule set must exist in development, test and production, a versioned source is the only practical way to keep them aligned. If an auditor or incident review asks why a page went to a given team, the commit history and pull request approval answer that question. Confirm with your on-call lead which teams change routing most often, because that frequency determines how much review overhead the pipeline will add.
The open-source edition is self-hosted, so your team owns the sync pipeline end to end: where the files live, how changes are reviewed, how they reach each instance, and what happens when a sync fails. Decide before the first rule is exported who approves changes, which branch deploys to production, and whether a human can still edit in the UI. These choices, not the file format, determine whether the approach holds up over time. Plan for the pipeline to be operated like any other production service.
Choosing between file provisioning, Terraform and the Grafana Operator
File provisioning is the most direct route. Alerting resources are written as YAML files that Grafana loads when it starts, so the definitions live in a directory on the Grafana host or in a container image. The Grafana provisioning docs describe the file format for alert rules, contact points and notification policies, and that format is the starting point for any team choosing this path. Changes are rolled out by updating the files and restarting or reloading Grafana, which usually means your existing server or container deployment pipeline applies them.
- Where definitions live: YAML files in the provisioning directory, tracked in a Git repository and copied to each instance.
- Who applies them: Your deployment pipeline or configuration management, not individual users.
- What it needs: Comfort with Grafana's file layout and a process for restarting or reloading instances.
- Main trade-off: Simple and transparent, but it is weaker at detecting drift and at managing several instances with different settings.
- Edits in the UI: Provisioned resources are typically locked against UI edits, so confirm the behavior for your version.
Before you commit to this approach, check the file format and the reload behavior for your Grafana version in the provisioning documentation.
Terraform, and the OpenTofu fork, manage Grafana alerting through the Grafana provider. Rules, contact points and policies are declared as resources in HCL, stored as state, and applied with a plan-and-apply workflow. This suits teams that already manage cloud resources this way and want alerting in the same review and approval process. OpenTofu is a drop-in replacement for Terraform that preserves existing workflows and configurations, so teams on either tool can follow the same pattern.
- Where definitions live: HCL in a repository, with state in a remote backend.
- Who applies them: A CI pipeline or a platform team running plan and apply.
- What it needs: Working knowledge of Terraform or OpenTofu, state management and provider version pinning.
- Main trade-off: Plans show drift clearly and changes can be reviewed, but state adds operational work, and provider resources may lag new Grafana features.
- Secrets and state: State can contain sensitive values, so OpenTofu's state encryption options are worth evaluating.
Before relying on this path, confirm with the provider documentation that every alerting resource you need is supported for your Grafana version.
The Grafana Operator runs on Kubernetes and reconciles Grafana resources from custom resources you apply with your normal cluster tooling. Alerting definitions are stored as manifests, so they sit alongside the rest of your cluster configuration and are applied through the Kubernetes API. This fits teams that already run Grafana on Kubernetes and manage workloads declaratively. Because the Operator is a separate project, its supported Grafana versions and alerting resource coverage must be confirmed for your environment.
- Where definitions live: Custom resource manifests in your cluster configuration repository.
- Who applies them: The Operator, triggered by changes applied through your GitOps or CI tooling.
- What it needs: Kubernetes operations skills, Operator upgrades and RBAC for the custom resources.
- Main trade-off: Strong fit for cluster-native teams, but it adds a controller to run and ties alerting to Kubernetes availability.
- Ownership: Decide which namespace and team own alerting resources to avoid conflicting writers.
Confirm the Operator's supported Grafana versions and the alerting custom resources it offers before you design around them.
Git Sync connects a Grafana instance directly to a Git repository so that dashboards and related resources are read from and written back to version control. Grafana documents it as one of the further as-code options alongside Terraform. It is most natural for teams that want dashboard-centered review workflows and fewer external tools. Capabilities for alerting resources differ from dashboards, so the scope of what Git Sync covers must be checked before you plan around it.
- Where definitions live: A Git repository connected to the Grafana instance.
- Who applies them: Grafana itself pulls changes, and users may push changes from the interface.
- What it needs: Repository access for the instance, a branch and review convention, and a plan for credentials.
- Main trade-off: Close to the UI workflow, but two-way sync can blur which copy is authoritative.
- Scope: Confirm whether alert rules, contact points and notification policies are covered in your Grafana version.
Verify what Git Sync supports for alerting and how it handles conflicts before you choose it for alert rules.
Configuration options and when each fits
This table compares file provisioning, the Grafana Operator and Terraform for keeping alert rules, contact points and notification policies in version control. Read each row against your own runtime, rollback and secrets constraints, then pick the method your team can run without the others.
| Factor | File provisioning | Grafana Operator or Terraform |
|---|---|---|
| Runtime dependency | Files are loaded when Grafana starts, so no extra controller or pipeline is needed to read them | Requires a Kubernetes operator (Operator) or a Terraform run with state storage (Terraform) |
| Rollback method | Revert the file in Git and restart or reload Grafana to apply the earlier definition | Revert the commit and apply again, or restore prior state; confirm how each tool removes deleted resources |
| Secrets handling | Contact point secrets are referenced from files, so keep them in a secrets mechanism outside Git | Use the tool's variable or secret references, and encrypt state if it holds sensitive values |
| Drift detection | Limited: UI edits to provisioned resources need explicit handling, so confirm the behavior in your version | Plan runs show differences between code and live state, so drift appears before the next apply |
| Kubernetes fit | Works with Helm or ConfigMap mounts in a cluster, but is one more file pipeline to maintain | Native fit when Grafana is managed as a custom resource and the team already runs GitOps |
| Team skills | YAML and Git, with no new language to learn | Terraform or HCL and provider knowledge; OpenTofu is a drop-in alternative to Terraform |
| Multiple Grafana instances | Copy and manage files per instance, which grows with each environment | One code base can target several instances through provider configuration |
How to set up alert rules as code in a self-hosted Grafana
This sequence moves Grafana alert rules, contact points and notification policies from hand-edited state into version-controlled definitions that a pipeline applies and that can be checked against the running instance. The result is a repository your team can review, test and roll back, with the live instance confirmed to match it.
Inventory existing rules and contact points
Export or list every Grafana-managed alert rule, contact point, notification policy, mute timing and template in the current instance, and record which team owns each one and which were created in the UI. Include the folders and rule groups they sit in, since provisioning files reference these. Agree the list with the on-call owners before anything moves. The step is complete when you have a written inventory with an owner and a source (UI, file or API) for each item, and nobody reports an alert they expected but cannot find in the list.
Choose the as-code approach
Decide between file provisioning, the Grafana Operator for Kubernetes, and Terraform, using your deployment model and existing tooling. File provisioning loads YAML or JSON at startup and suits virtual machines and containers without an extra control plane. The Grafana Operator fits teams already reconciling resources through Kubernetes manifests. Terraform fits teams that manage other infrastructure with it and want one plan and apply workflow. The Grafana provisioning documentation lists Terraform and Git Sync as further options. Confirm the exact resource support and version compatibility for your chosen Grafana release with the tool's current documentation. The step is complete when the team has written down the chosen tool, the reason, and who maintains it.
Define the repository layout
Create a repository that separates alert rules, contact points and notification policies into their own directories, with one folder per environment such as development, staging and production. Keep shared templates and common labels in a location all environments can reference, and keep environment-specific values such as contact point targets in per-environment files. Add a README that states which directory the running instance reads and who approves changes to each directory. Use the same layout for rules and routing so a reviewer can see which alert reaches which team in one place. The step is complete when a reviewer can trace a single alert from its rule file to its notification policy and contact point without leaving the repository.
Store secrets outside the repository
Keep webhook URLs, API tokens, PagerDuty routing keys and SMTP credentials out of the repository. Reference them as environment variables or pull them from your secrets manager at deploy time, and document the variable names in the README. Confirm how your chosen tool expands these references, since provisioning files and Terraform handle substitution differently. Run a secret scan on the repository before the first commit and again in the pipeline. The step is complete when the scan reports no credentials, and a fresh clone with the environment variables set renders the same contact points as the running instance.
Validate definitions in a non-production instance
Load the repository into a non-production Grafana instance that uses the same release as production. Confirm that every rule group evaluates without errors, that each contact point accepts a test notification, and that a test alert follows the expected notification policy path to the right receiver. Include mute timings and templates in the test, since they change routing and message content. Record the Grafana version used, because provisioning behavior can differ between releases. The step is complete when the test alert reaches the intended receivers and the non-production instance shows no provisioning errors in its logs.
Enable a pipeline that applies changes
Add a pipeline that runs on each proposed change. It should lint and validate the definitions, run a plan or dry run where the tool supports it, require an approval from the owning team, and then apply the change to the non-production instance before production. Make the pipeline the only path that writes to production alerting, and restrict who can push to the protected branch. Keep the pipeline logs as a change record. The step is complete when a test change to a rule threshold moves through review and into the non-production instance without anyone editing that instance by hand.
Handle UI edits after provisioning
Decide how to treat rules and contact points that someone edits in the UI after they are provisioned. Provisioned resources are normally read-only in the UI, so confirm this behavior for your release and the tool you chose. Set a rule that any UI change is either exported back into the repository through a reviewed commit or reverted by the next apply, and tell the on-call team which applies. Check for resources that exist only in the UI using the inventory from the first step. The step is complete when the team has a written policy, every UI-only resource has been either committed or removed, and a deliberate UI edit is shown to be overwritten or flagged on the next run.
Verify rendered state against the running instance
After each production apply, compare the rendered definitions in the repository with the state the running instance reports. Export the live rules, contact points and notification policies and diff them against what the pipeline rendered, paying attention to label matchers, group intervals and receiver names. Run a scheduled drift check that reports differences to the owning team. Keep the diff output with the pipeline record. The step is complete when the diff is empty for a production apply, and the scheduled check has run without reporting drift that nobody has explained.
What happens when someone edits a provisioned rule in the UI
A rule provisioned from a file is managed from that file, so a change made in the Grafana UI is not a durable change. Whether the UI edit survives depends on how the resource was loaded and on the behavior of your Grafana version. With file provisioning, the YAML under the provisioning directory is read at startup and on reload, and the next apply or restart can replace what someone changed by hand. With the Grafana Operator or Terraform, the controller or state file reconciles the instance against the declared resources, so drift introduced in the UI is typically reverted on the next run, but the exact behavior depends on the tool version and how the resource was created. Before you rely on any of this, confirm with the Grafana provisioning documentation and in a test instance exactly which alerting resources are locked in the UI, which are editable, and what happens to an edit when the file changes or the process restarts.
The conflict risk is highest during an incident, when an on-call engineer changes a threshold, mutes a rule or adjusts a notification route to stop a page. Decide in advance who may make emergency UI changes, which resource types they may touch, and how long a change may live before it must be reconciled. A written policy might require the engineer to record the change in the incident ticket, notify the rule owner, and open a pull request within a fixed window. Silences and mute timings are usually the safer emergency tool because they stop notifications without changing the rule definition, which is a distinction worth testing in your version. Make sure the policy names an owner for each rule directory so that no one has to guess who approves the follow-up change.
Once the emergency is over, export the UI change back into the repository rather than simply reapplying the old file. Compare the exported definition with the committed one, keep the parts that reflect a real decision, and remove any incidental fields the UI added. Then run the normal review and apply path so the repository, not the running instance, remains the record. Schedule a periodic drift check that compares the live configuration with the repository, and treat any difference it reports as either an unrecorded change to commit or an unauthorized change to revert. Ask your implementation partner or platform team to document this export procedure as part of the runbook, including who runs it and where the evidence is stored.
Security and operational considerations for alert definitions in Git
Alert rules, contact points and routing policies in Git are code, and they also hold the credentials and routing logic that decide who gets paged. The cards below cover where secrets live, what the apply pipeline is allowed to change, how routing edits are reviewed, and how state and history are protected.
Keep webhook and paging secrets out of Git
Contact points often carry Slack webhook URLs, PagerDuty integration keys and SMTP passwords. Committing them in provisioning files exposes every person with repository read access to a paging credential. Reference secrets through environment variables or your secret manager at apply time, and check that the repository and its history contain no contact point values. If a key was ever committed, rotate it rather than relying on a later deletion.
Scope the apply pipeline's service account
The identity that applies alert changes can create, edit and delete rules and routes in production. Give it a dedicated service account with only the alerting permissions the repository manages, and separate it from the account used by humans and dashboards. Ask your platform team which role and organization the account is bound to, and confirm that it cannot change server settings or user access it does not need.
Review routing changes as paging changes
A small edit to a notification policy, a matcher or a mute timing can stop pages for an entire service without any rule changing. Require a second reviewer from the owning on-call team for changes under the policy tree, and run the route test for affected labels before merge. Treat a pull request that adds a mute timing the same way you treat a production change window.
Store Terraform state remotely with locking
If you manage alerting with Terraform or OpenTofu, the state file records every resource ID and can contain sensitive values. Keep it in a remote backend that supports locking so two pipeline runs cannot apply at once, and restrict read access to the pipeline and a small set of operators. Confirm the backend's own backup and restore process before you depend on it.
Encrypt state and plan files
OpenTofu can encrypt state and plan files at rest, for local storage and remote backends, using key providers such as AWS KMS, GCP KMS, OpenBao or Azure Vault. Plan files often contain the same sensitive values as state, so they need the same protection when pipelines upload them as artifacts. Decide which key provider your organization already governs and confirm who can decrypt, because losing that key means losing the state.
Use commit history as the change record
Git history shows who changed a rule, when, and which reviewer approved it, which is useful evidence for an audit trail. It only works if merges go through reviewed pull requests, direct pushes to the main branch are blocked, and the apply pipeline logs the commit it deployed. Ask your team to confirm that the pipeline records the commit SHA for each apply, so an alert change can be traced to the deployment that enabled it.
What changes for on-call and platform teams
Once alert rules, contact points and notification policies live in Git, the on-call team stops changing routes directly in the Grafana UI and starts proposing changes through a pull request. That shift matters most for the notification policy tree, because a small edit to a matcher or a group_wait value can silence a team's pages for an entire service. Responders who used to fix a routing gap in five minutes now need a reviewer, a CI check that validates the files, and a merge before anything changes. Platform engineers should agree on who approves policy changes, which labels a rule must carry before it can be merged, and how long a review may take before the on-call lead can escalate it. Write these rules down before the first migrated rule goes live, because reviewers without an agreed standard tend to either approve everything or block everything.
Every team also needs an emergency path, and it should be designed rather than improvised during an incident. Options include a break-glass account with edit rights that is used only under a documented trigger, a pre-approved fast-track label on the pull request, or a pipeline step that can be run manually against a named branch. Whichever you choose, the emergency path has to be tested in a non-production instance, and every use has to be reconciled back into version control afterward so the next sync does not undo the fix. Confirm how your chosen method interacts with provisioning, since a file-provisioned resource is generally managed from its source file, and check the current behavior in the Grafana provisioning documentation for the version you run before you rely on it.
The lag between a merged change and the running instance is the third thing to plan for. Provisioned files are read when Grafana starts, and whether a running instance picks up changed files without a restart, or whether a separate reload or Terraform apply is needed, depends on the method and the Grafana version in use. A responder who merges a silence or routing fix and then waits for a sync cycle during a live page will reasonably distrust the process. Ask the platform team to measure the delay from merge to effective routing in your environment, publish that figure to on-call, and add a post-deploy check that sends a test alert through the affected policy. Until that check exists, keep the UI path available for time-sensitive silences, and record every such change for later reconciliation.
Questions teams ask before managing Grafana alerts as code
How much effort does moving Grafana alerts into code take?
The effort depends mostly on how many alert rules, contact points and notification policies already exist, how many environments need the same configuration, and how far the live instance has drifted from what anyone documented. Each rule has to be exported, reviewed, rewritten into a provisioning file or Terraform resource, and checked against the instance it will run on. Teams with a small set of rules in one Grafana instance usually have a short, bounded job. Teams with hundreds of rules, several Grafana instances and years of untracked UI edits face a longer inventory and cleanup phase. Ask any partner for a written scope that lists rule counts, environments and the export method before agreeing to a fixed price.
Can our own team manage Grafana alerts as code without outside help?
Yes, many teams do this in-house, provided someone owns the Grafana configuration and has time to learn the chosen method. The work needs people who can read existing rules, write YAML or HCL, operate a Git workflow and reason about alert routing. A partner is most useful when the team lacks that time, when the instance has heavy UI drift, or when the migration must be finished before a deadline. Either way, the team keeps ownership of the repository and the rules afterward. Confirm who will review changes and who can approve production applies before the work starts.
What does our organization need to provide for alerts as code?
Plan for four things: a Git hosting platform with branch protection and review, a CI runner that can reach the Grafana API or run the provisioning checks, a secrets store for Grafana service account tokens and receiver credentials such as Slack or PagerDuty keys, and an account with enough Grafana administrator rights to read and apply alerting configuration. Secrets should never be committed in plain text. Provisioning files can reference environment variables, and Terraform state should be stored in an encrypted remote backend. Confirm your Grafana edition and access model first, since identity features differ between open source and Enterprise.
Should we use Terraform or file provisioning for Grafana alerts?
File provisioning is the simplest option when Grafana runs in a single environment and the team wants alerting defined in YAML or JSON files that load at startup. Terraform suits teams that already manage cloud and platform resources with it, need to promote the same configuration across environments, or want plan output for review. OpenTofu is a drop-in replacement for Terraform that keeps existing workflows and configurations, so it is a valid choice if your organization avoids the Terraform license. A Kubernetes-based Grafana Operator fits teams already running GitOps with Argo CD or similar tools. Choose based on your existing delivery pipeline, not on the tool alone.
What happens to alert rules that someone already created in the Grafana UI?
Rules created in the UI remain in the database and keep evaluating, but they are not managed by your code. Once a rule is provisioned from a file or Terraform, Grafana generally marks it as provisioned, and UI edits to those rules are restricted, so the repository becomes the source of truth. Rules that exist only in the UI need to be exported, reviewed and either brought into code or deleted on purpose. Expect to discover duplicates, stale rules and undocumented contact points during this step. Ask the team to freeze UI edits during migration and to record each decision in the repository.
What drives the timeline for managing Grafana alerts as code?
The timeline follows the same drivers as the effort: the number of rules and notification policies, the number of Grafana instances and environments, how much drift exists between the UI and any earlier exports, and how many people must review and approve changes. Access and approvals also add time, especially when secrets must be provisioned through a separate team. A short, well-defined inventory moves quickly, while unclear ownership of rules or receivers extends every step. Ask for a timeline built from a written inventory of your instance rather than from a generic estimate.
Will alerts as code prevent duplicate notifications in a high availability setup?
No, storing alerts in code does not by itself stop duplicate notifications. Grafana high availability runs several active servers that share a database, and alerting needs extra configuration to avoid duplicate sends. Provisioned rules and policies behave the same way across nodes, so the high availability design must be verified separately. Confirm the cluster settings for alerting in your Grafana version, and test one alert end to end before relying on it for on-call.
How DeployOpen works on this
DeployOpen plans, implements, documents and supports self-hosted open-source platforms inside your own infrastructure, so alert rules, contact points and notification policies for Grafana are scoped, built and handed over in your environment. Engagements are fixed-scope, with the work agreed in writing before it starts.
How we scope
- Inventory which alert rules, contact points and notification policies already exist and which are edited only in the UI today.
- Agree in writing whether the repository layout, provisioning files or Terraform modules will be the source of truth.
- Confirm the Grafana version, database backend and whether the instance runs in high availability before choosing a method.
How we build
- Export current alerting resources, convert them to provisioning files or Terraform configuration, and review each rule against the original.
- Set up a review and deployment path in your version control system so changes to alerting are reviewed before they reach production.
- Test routing and notification delivery in a non-production Grafana instance before any change reaches the on-call path.
How we hand over
- Document the repository structure, the deployment steps, the process for adding rules, and the rules for changes made in the UI.
- Walk your platform and on-call teams through the workflow, including how to roll back a bad alerting change.
- Hand over written runbooks for drift detection and for reconciling any alerting resource that was edited after provisioning.
How we support
- Ongoing support, Grafana upgrades and continuous testing are available as retainers, agreed separately from the initial scope.
- Review alerting changes after Grafana minor and major upgrades, since provisioning behavior can change between releases.
- Confirm with your team which support hours, response expectations and change windows apply before the retainer starts.
