To manage Grafana alerting as code, keep alert rules, contact points, and notification policies in version control, choose one provisioning method as the source of truth, and let Jenkins run checks in a fixed order: validate the configuration, produce a plan, review it, and only then apply. Each of those steps answers a different question, and none of them is a guarantee on its own. A passing syntax check does not show that a live Grafana instance will accept the change, and a Jenkins stage labeled “dry run” does not prove that nothing outside the pipeline was touched.
Pick one source of truth before writing any code
Grafana alerting resources can be managed three ways: Terraform, file provisioning, or the Alerting provisioning HTTP API. Each one marks resources differently and allows different kinds of edits later, so mixing them is the most common way to end up with drift. The Grafana provisioning overview is the starting point for choosing between them: Grafana provisioning overview.
As an Amazon Associate I earn from qualifying purchases.
| Method | Best fit | Edit behavior in the UI | Grafana Cloud |
|---|---|---|---|
| Terraform (Grafana provider) | Teams that already review infrastructure changes as plans and want a diff before every apply | Provisioned resources are read-only in the UI by default; the provider’s disable_provenance setting allows UI edits, as described in the Terraform guide |
Documented as usable with Grafana Alerting through the provider; confirm the exact support matrix for your stack |
| File provisioning (YAML or JSON) | Self-managed Grafana deployments where configuration files are deployed alongside the server | File-provisioned resources cannot be edited in the UI | Not available, according to the file provisioning guide |
| Alerting provisioning HTTP API | Programmatic management from scripts or services | Depends on how the resource was created; use the export endpoints when you need provisioning-format output | Not stated for every endpoint; check the export guide against your edition |
Whichever you choose, do not let a second method write to the same objects. A rule created in the UI and later declared in Terraform, or a policy tree pushed from files while a Terraform configuration also owns it, will produce conflicts that are hard to read from a plan alone.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallWhat you are actually managing
Three kinds of objects make up most alerting configuration, and each has its own review concerns.
#1 Best Overall
Alert rules
An alert rule defines the queries and conditions that decide whether something is firing, how often it is evaluated, and optional labels and annotations. It also sets how errors and missing data are handled. Those error and no-data settings are easy to overlook in review, yet they decide whether a broken data source pages someone or stays silent. Rule behavior is described in the alert rules documentation.
Contact points
A contact point defines where notifications go, such as a chat channel, an email address, or a webhook. Credentials and endpoints for these integrations are the parts most likely to be sensitive, so they are also the parts most likely to be pasted into the wrong file. The contact points documentation covers the object itself.
Notification policies
Notification policies route alerts to contact points. The policy tree is a single resource, which changes how you review it. Provisioning it replaces the whole tree, not one branch, so a change that looks small in a diff can remove routing you did not intend to touch.
Rank #2
Managing rules and contact points with Terraform
Grafana’s Terraform guide describes the provider as support for creating, managing, and maintaining the whole Alerting stack as code. In the provider, alert rules are declared as grafana_rule_group, contact points as grafana_contact_point, message templates as grafana_message_template, notification policies as grafana_notification_policy, and mute timings as grafana_mute_timing. The documented setup needs a running Grafana instance, Terraform, and a service-account token for provider access.
The documented flow runs in this order:
- Configure provider authentication with a service-account token, read from a secret store or a protected pipeline variable rather than from a file in the repository.
- Define the resources yourself, or export existing ones from Grafana and then take ownership of them. Exports are covered in the next section.
- Run
terraform initin the configuration directory to download the provider and set up the backend. - Run
terraform validateto check the configuration. - Run
terraform plan -out=tfplanto save the proposed change. Review the plan output orterraform show tfplan. - Run
terraform apply tfplanonly after the plan has been reviewed and approved under your team’s rules.
Two details matter for review. First, a plan is only as current as the state and provider reads it uses, so a plan produced against the wrong workspace or backend can look correct and still be wrong. Second, the saved plan file is the artifact to apply. Re-running the plan just before apply means you may be approving something different from what was reviewed.
Exporting existing alerting resources without losing control
Most teams do not start from an empty repository. They start from a Grafana instance that already has rules and contact points. The export options differ by purpose, and mixing them up is a frequent source of broken files.
- The Grafana UI can export resources as Terraform, YAML, or JSON.
- The standard Alerting HTTP API returns JSON that is generally not compatible with file or Terraform provisioning.
- Dedicated export endpoints return provisioning formats. Use these when you need output you can commit and load.
The export guide is the reference for the endpoint details: Grafana export guide. Export one resource type at a time, and do not generate a complete notification policy tree from a partial view of the current policies. Because provisioning replaces that tree, an incomplete export becomes an accidental deletion of routing.
File provisioning under provisioning/alerting
File provisioning places YAML or JSON files under provisioning/alerting and applies them when Grafana starts or when the Admin API reload is triggered. The documented result is strict: file-provisioned resources cannot be edited in the UI, so the files are the only place to change them. This suits a team that treats the Grafana server as disposable and redeploys it from a repository. It does not suit a team that expects operators to tweak thresholds during an incident, and it is unavailable in Grafana Cloud. See the file provisioning guide for the file layout and reload behavior.
Validation, planning, and applying are three different checks
Teams often use “test” for everything, which hides what each step can and cannot prove. HashiCorp’s reference for the validate command describes it plainly: “The terraform validate command validates the configuration files in a directory.” The same reference adds that validation “does not validate remote services, such as remote state or provider APIs.” Source: HashiCorp validate reference.
| Step | What it establishes | What it does not establish | Changes Grafana? |
|---|---|---|---|
Formatting and syntax checks (terraform fmt -check, terraform validate) |
The files are well formed and internally consistent within the configuration | That remote state is reachable, that the Grafana API will accept the objects, or that a credential has the right permissions | No |
Plan (terraform plan -out=tfplan) |
What Terraform would create, change, or destroy for this run, against the state and provider reads it can make | That the resulting state is correct for every downstream consumer, or that the pipeline itself has no side effects | Not to Grafana objects, but it does contact the backend and provider |
Apply (terraform apply tfplan) |
Executes the reviewed plan | Nothing further; it is the step that changes configuration | Yes |
The practical consequence is that a green validate stage tells you the code is coherent, not that Grafana will accept it. The plan is where you catch wrong targets, unexpected deletions, and changes to the notification policy tree.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Building a cautious Jenkins pipeline
Jenkins is the orchestrator here. It runs commands, passes artifacts between stages, and can hold a stage until someone approves it. The pipeline syntax reference covers the declarative and scripted forms: Jenkins Pipeline syntax, and the Jenkins getting started guide covers creating the pipeline job itself.
A sequence that keeps the three concepts separate looks like this:
- Check out the configuration from a reviewed branch or tag, not from a working copy on a build agent.
- Run
terraform initwithout any apply step in the same stage. - Run formatting and validation checks, failing the build on any error.
- Produce a plan for the intended environment and workspace, and save it with
-out. - Archive the plan output and show it to the approver.
- Gate the apply stage on your project’s approval policy, and apply only the saved plan file.
pipeline {
agent any
stages {
stage('Format and validate') {
steps {
sh 'terraform init -input=false'
sh 'terraform fmt -check -recursive'
sh 'terraform validate'
}
}
stage('Plan') {
steps {
sh 'terraform plan -input=false -out=tfplan'
archiveArtifacts artifacts: 'tfplan'
}
}
}
}The sketch above shows the ordering and the separation between checks and planning. It does not include credentials handling, backend configuration, or an approval gate, because those depend on your environment and Jenkins setup. The plugins you use for credentials, approvals, and artifact handling vary by installation, and their behavior should be confirmed against your own Jenkins version before you rely on it.
What a “dry run” stage does and does not prove
A stage named “dry run” usually means the pipeline does not call an apply command. That is useful, but it is not a guarantee. The stage may still run terraform init, which can download providers and configure a backend; it may still read state and call the Grafana API during a plan; and any custom script in the same stage can write files, send notifications, or change other systems. Before you describe a stage as a dry run to your team, list every command it executes and what each one touches.
Credentials and secrets
Provider tokens and contact point endpoints are secrets. Store them in your secret manager or Jenkins credential store, inject them only into the steps that need them, and keep them out of committed files and build logs. Redact command output that could echo a token or webhook URL. This is general operational practice; the Grafana guide shows where the token is configured but does not define how your pipeline should store it.
Recommended Free Tools
Version checks before you copy any of this
Grafana’s documentation links point to the latest channel, so the page you read today may describe different behavior from the one you deploy against. Confirm your Grafana edition and version, your Terraform provider version, and your Terraform CLI version before writing configuration. Check the Grafana guides for the edition-specific limitations, particularly the file provisioning restriction in Grafana Cloud, and test provisioning in a non-production stack first.
The primary references for this workflow are the Grafana Terraform provisioning guide, the Grafana export guide, and the HashiCorp validate reference linked above.
The Bottom Line
Manage each alerting object through exactly one method, let Terraform or file provisioning produce a reviewable diff, and treat validation, planning, and applying as separate gates. Use the Jenkins stages to enforce that order, and do not call any stage a dry run until you have listed what it actually executes.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
