How Airflow alerts to Slack, PagerDuty and Opsgenie reach the on-call team
Airflow sends failure and deadline alerts through separately installed provider packages, with notifiers attached to tasks, DAGs or Deadline Alerts. The Slack, PagerDuty and Opsgenie providers are distinct packages (apache-airflow-providers-slack, apache-airflow-providers-pagerduty and apache-airflow-providers-opsgenie), so each channel is a dependency you add to the image or environment, pin and upgrade on its own schedule. Nothing reaches an on-call tool until the package is present in every component that runs callbacks, which in Airflow 3 includes the workers and the DAG processor, not only the scheduler. Confirm the installed set in each image during change review.
The Slack provider is the basis for Slack routing. Its notifications cover the Slack API and the Slack Incoming Webhook service, and its built-in notifier, SlackWebhookNotifier, can be passed to a Deadline Alert callback. A webhook URL is a credential: it belongs in an Airflow connection or your secrets backend, not in DAG source. For PagerDuty and Opsgenie, confirm with each provider's package page which integration key or API credential each notifier expects, which Airflow connection type holds it, and whether your organization's routing service, escalation policy and region match the endpoint you configure. Those answers decide who gets woken up, so get them from the on-call owner before go-live.
Before wiring any channel, decide which failures page a human and which only post to a team channel. A single failed task on a non-critical DAG rarely justifies a page, while a missed deadline on a finance or regulatory feed usually does. Write that mapping down per DAG tier, then test each route end to end with a deliberately failing task in a non-production environment. Confirm the secrets path, the outbound network rule from your workers to the provider endpoint, and that a test alert arrives at the right escalation target.
What you need before wiring alerts to on-call tools
Alert routing depends on three layers: a supported Airflow 3 platform, the provider packages that carry alerts out of it, and the accounts and credentials on the Slack, PagerDuty and Opsgenie side. Settle each layer before writing any callback, because a missing key or an unsupported provider version usually shows up as silent alert loss.
Supported Airflow 3 baseline
Start from the current Airflow 3 patch release, 3.3.2, and confirm it is the line your team will run. Airflow 2 reached end of life on April 22, 2026, so an alerting build on Airflow 2 would receive no fixes. Check that your Python version falls in the 3.10 to 3.14 range and your Kubernetes version, if you use it, falls in the 1.30 to 1.35 range listed in the prerequisites.
Production metadata database
Run Airflow on PostgreSQL or MySQL, not SQLite, because the production guide warns that the default SQLite backend can lead to data loss. MariaDB is not supported as the metadata database. Alert deduplication and retry history rely on task instance state, so a database you can back up and restore is part of the alerting design, not a separate concern.
Provider packages installed separately
The Slack, PagerDuty and Opsgenie providers are separate packages, so install each one by name in the same environment as Airflow, for example through your constraints-based image build. Slack release 9.11.1 requires Airflow 2.11.0 or later. Before you pin any provider, confirm its compatibility with your Airflow 3 version in the providers reference.
Slack incoming webhook or API token
Decide whether Slack messages go through an Incoming Webhook URL or the Slack API with a bot token, since the Slack provider covers both services. Create the webhook or app in the workspace that owns the on-call channel, scope it to that channel, and store the value as an Airflow connection or in your secrets backend. Confirm with your Slack administrator who can rotate it.
PagerDuty integration key
Create a service in PagerDuty for the Airflow platform or for each owning data team, and generate an Events API integration key on that service. Each key routes alerts to one escalation policy, so the number of keys is an ownership decision. Ask your PagerDuty administrator which escalation policies apply to pipeline failures and who can rotate the key.
Opsgenie API key and team routing
Create an API integration in Opsgenie for the owning team and record its API key, along with the region your Opsgenie account uses. Map each Airflow alert source to a team so alerts reach the right rotation. Confirm the Opsgenie provider version you plan to use supports your account setup, and keep the key in the same secrets store as the other on-call credentials.
How to configure Slack, PagerDuty and Opsgenie alerts in order
This sequence moves Airflow failure and missed-deadline alerts from a single shared callback to the on-call tools your team already watches, with each route verified before anyone depends on it. Each step ends with a check you can see, so a gap shows up in testing rather than during an outage.
Install each provider package separately
The platform engineer installs apache-airflow-providers-slack, apache-airflow-providers-pagerduty and apache-airflow-providers-opsgenie only for the destinations you use, pinned in the same constraints file as the rest of your Airflow image. Each provider is a separate package, so a missing one is a clear install error rather than a silent gap. Done when the scheduler, worker and API server images all list the packages and an import of each provider module succeeds inside the running containers.
Store destination credentials outside DAG code
The platform engineer creates the Slack webhook or Slack connection, and the PagerDuty or Opsgenie integration key, as Airflow connections backed by your approved secrets store, so DAG files only reference connection IDs. Ask your security team which store and access model applies before creating them. Done when a DAG parse shows no secret values and a connection test or a read of the connection from a task succeeds under the service identity that runs it.
Define one shared alert callback
The data platform team writes a single on_failure_callback function in a shared module that builds the message from the task context: DAG ID, task ID, run ID, try number, logical date and a link to the task log in the Airflow UI. It sends to the chosen providers through their notifiers, and critical routes are selected by a tag or parameter rather than by copying code. Done when the callback runs in a unit test with a mocked task context and produces the expected payload for a sample failure.
Attach the callback at DAG and task level
The team sets the callback in default_args so every task in a DAG inherits it, then overrides it on critical tasks that page the on-call tool while lower-priority tasks post only to a channel. Retries should be configured so the failure callback fires after the final attempt, not on each retry, which keeps routine transient errors out of the on-call queue. Done when a DAG parse shows the callback on every task and a review of the retry settings confirms the intended trigger point.
Add Deadline Alerts with warning and escalation tiers
For pipelines with a business deadline, the team adds Deadline Alerts using a built-in notifier such as SlackWebhookNotifier, and passes a list to the deadline parameter so a first warning goes to a channel and a later escalation goes to the paging tool. The deadline values come from your service-level expectations for each pipeline, which the owning team should confirm in writing. Done when the DAG shows both deadline entries in the UI and a test run that is intentionally slow produces the warning message in the channel.
Test each route with a deliberate failure
The platform team runs a dedicated test DAG with a task that raises an error on purpose, plus a short deadline that will be missed, and confirms each route separately. The Slack message should appear in the intended channel, and the PagerDuty or Opsgenie test incident should appear in the correct service or team with the Airflow log link working. Done when each destination shows the expected message or incident, and a team member outside the build confirms the link opens the right task log.
Confirm alert content and retry behavior with owners
Before go-live, the on-call owners review sample messages and check that each alert carries enough context to act on without opening every log, and that retries do not produce duplicate pages. They also confirm who is paged outside working hours and how a test incident is resolved in the on-call tool. Done when the owners sign off in your change record and a repeated test run produces one page per final failure, not one per attempt.
Which notifier and deadline pattern fits each alert type
Task-level notifiers attach to a single task, usually through an on_failure_callback or a notifier passed to the task. They run when that task ends in a failed state, so they carry the most precise context: the DAG ID, task ID, run ID and a link to the task log. Use them where a failure has a known owner and a known response, such as a load into a regulated table or a step that feeds downstream consumers.
- Scope: Attach notifiers to the tasks that matter to consumers and leave low-value tasks alone, so the channel stays readable.
- Ownership: Map each DAG or task group to a channel or on-call rotation so the message reaches the team that can act on it.
- Retry behavior: Use the failure callback rather than a retry callback for paging decisions, so you alert once after retries are exhausted. Confirm this behavior in your Airflow version.
- Message content: Include the DAG ID, run ID, task ID, logical date and log link in the body, so responders do not start by searching the UI.
- Secrets: Reference webhook URLs and integration keys through Airflow connections, not literals in DAG code.
Before relying on these notifiers, run a test DAG that fails on purpose with retries enabled and confirm exactly one notification arrives per exhausted failure.
DAG-level callbacks are set in the DAG arguments and fire on the outcome of the whole run instead of one task. They suit pipelines with many tasks, where a team wants one summary per run rather than one message per failed task. The trade-off is less detail in the message, so the callback should point responders to the run and its failed tasks.
- One message per run: A failed run produces a single notification, which keeps a 40-task DAG from flooding a channel when several tasks fail together.
- Summary content: Include the failed task IDs, the run state and a link to the run page so the responder can drill down.
- Blind spots: A run that never starts, or that stalls while still running, does not produce a failure outcome, so a DAG callback alone will miss it.
- Shared definition: Define the callback once in a common module and apply it through default arguments, so every DAG reports in the same format.
- Ownership tags: Use DAG tags or a lookup table to route each DAG's summary to the right channel.
After deployment, stop a test run mid-way and confirm which callback fires. If a stalled run produces no alert, cover it with the deadline pattern in the next tab.
Deadline Alerts address late work rather than failed work. You set a deadline by which a run or task should finish, and a callback fires if it has not. Built-in notifiers such as SlackWebhookNotifier can serve as that callback, which removes the need for custom code in simple cases. This pattern catches the stalled or slow run that a failure callback never sees.
- Late versus failed: A late run may still succeed later, so treat a missed deadline as its own signal and keep it separate from failure alerts.
- Tiered list: Passing a list to the
deadlineparameter lets you define a warning first and an escalation later, for example a chat warning followed by a page if the run is still incomplete. - Deadlines from consumers: Set deadlines from the times that data consumers have agreed to, and record the agreement in the DAG documentation.
- Webhook setup:
SlackWebhookNotifierposts to a Slack incoming webhook configured through the Slack provider, so confirm the connection and where the webhook secret is stored. - Version check: Confirm that Deadline Alerts are available in the Airflow version you plan to run, and check the deadline documentation for its current parameters.
To verify tiers, run a test DAG that intentionally sleeps past each threshold and confirm that the warning and the escalation arrive in order.
Chat channels suit alerts someone should see during working hours and act on soon. PagerDuty and Opsgenie suit failures that need an acknowledged response at any hour, with escalation if nobody responds. The decision depends on the consumer impact of the pipeline, not on how often it fails. Confirm the exact notifier class names for PagerDuty and Opsgenie in the provider documentation before you build, since the names used in your code must match the installed package.
- Page criteria: Reserve paging for pipelines with a defined consumer impact, such as regulated reporting, customer-facing data or payment-related loads.
- Chat criteria: Send exhausted-retry failures for non-critical DAGs and deadline warnings to Slack, where the team can triage them.
- Escalation path: Use the tiered deadline pattern so the warning goes to chat and the escalation goes to paging.
- Provider installation: Each provider is installed separately, so install the PagerDuty or Opsgenie package and store the integration or API key in an Airflow connection.
- Routing review: Keep routing rules in version control so changes to on-call ownership go through review.
Test each path with a sample event and confirm that acknowledging the page in the paging tool closes the incident as expected.
How to stop retries and reruns from flooding the on-call channel
The main lever is deciding when a callback fires. A task configured with several retries can fail, retry and fail again, and a callback attached to the failure path may run on each failed attempt or only once the retries are exhausted, depending on how the callback is wired and which Airflow version you run. Confirm this timing in your version before you write any alert rule, because an alert that fires per attempt on a task with five retries produces five pages for one problem. A DAG rerun across many logical dates has the same issue at a larger scale: clearing or backfilling a month of runs can generate a page for each date unless the routing is designed for it.
Grouping and deduplication then do most of the remaining work. Slack messages can be grouped by DAG and run in one channel thread so that a single incident has one visible history. In PagerDuty and Opsgenie, the alert payload should carry a stable deduplication key built from the DAG ID and the failing task or deadline, so that repeated sends for the same underlying failure update one incident instead of opening new ones. Test this with a deliberately failing task and a manual rerun before going live, and check what the provider sends as the key in each case. Your provider versions and their parameters should be confirmed against the provider documentation for your Airflow release.
Reserve paging for conditions that need a human at night: a missed deadline on a production DAG, or a failed run of a DAG that feeds a regulated report or customer-facing process. Route routine failures of development or non-critical DAGs to Slack only, and let the retry policy absorb transient errors before anyone is notified. Agree this split with the data owners in writing, since they will decide which DAGs are production. The on-call runbook should state which alerts page, which post to chat, and who is expected to act on each, so that the paging policy can be reviewed as the DAG estate changes.
What each alert should contain and link back to
This table maps each field an alert should carry to where it comes from in Airflow and why on-call needs it. Read it as a checklist for the message template: a field that cannot be filled from the run context is a gap to close before alerts go live. The main takeaway is that the run link and the owner field decide whether an engineer can start triage within minutes of the page.
| Field | Source in Airflow | Why the on-call engineer needs it |
|---|---|---|
| DAG ID and task ID | Task context passed to the callback (dag_id, task_id) | Identifies the failing pipeline step without opening the UI |
| Run ID and logical date | Run context (run_id, logical date) | Pins the alert to one data interval so reruns are not confused with the original failure |
| Try number | Task instance try number at callback time | Shows whether this is the first failure or the last retry before the task gives up |
| Environment | Deployment label set in the alert template, such as dev, staging or production | Stops non-production noise from paging the production rotation |
| Owning team and on-call route | Owner from DAG metadata, mapped to a PagerDuty service or Opsgenie team | Confirm each owner value matches a live rotation in PagerDuty or Opsgenie, or the page goes to nobody |
| Airflow log and run URL | Link to the task instance in the API server or UI, built from the run context | The run link is the starting point for triage: logs, status and rerun options sit behind it |
| Missed deadline definition | The deadline definition attached to the DAG that was missed | Tells the reader which promise was broken and how late the run was, so severity is clear |
| Runbook link | Static link stored per DAG or team in the alert template | Gives the next action for this failure type; confirm each DAG has a runbook before go-live |
How to keep webhook keys, integration keys and alert content safe
Alert integrations carry credentials that can post into your incident channels or page your engineers, and alert text often reflects the data a task touched. The cards below cover where keys should live, who can change alert destinations, what must stay out of messages, and how keys are rotated.
Keep webhook and routing keys out of DAG files
Slack incoming webhook URLs, PagerDuty routing keys and Opsgenie API keys should be stored in an Airflow connection or a secrets backend, then referenced by connection ID from the DAG. A key written into a DAG file lands in source control, in code review history and in every copy of the repository. Ask your team which connection IDs each alert uses and confirm that no DAG file contains a literal webhook URL or key.
Confirm how secrets are masked
Before relying on connection values staying hidden, confirm with your chosen secrets backend and your exact Airflow version how connection fields are masked in task logs, the UI and API responses. Test by deliberately printing a connection field in a throwaway DAG and checking the task log. Masking behavior can differ between a database-backed connection and one fetched from an external backend, so test both if you use either.
Limit who can change alert destinations
Any DAG author who can edit an alert callback can redirect failure messages to a channel they control or send them to a stranger's on-call rotation. Decide who may create or edit connections in the Airflow UI and API, and separate that permission from DAG authoring. Review the access model for your deployment, since role and permission settings depend on the auth manager you run. Confirm this with your identity configuration before go-live.
Keep regulated data out of alert text
Exception messages, SQL error text and XCom values can contain customer identifiers, account numbers or health information, and a failed task often prints them. Slack channels and paging tools are usually retained and copied into other systems. Build alert messages from DAG ID, task ID, run ID, try number and a link, and have callbacks truncate or omit exception text unless your data classification allows it.
Rotate integration keys without missed pages
Rotating a Slack webhook, PagerDuty routing key or Opsgenie API key means updating the stored connection and then confirming a test alert arrives. Plan the rotation window so that no failure fires during the switch, and record who performed it. Confirm with each provider how old keys are revoked, because some services let a key be replaced without interruption and others do not. Write the rotation steps down as a runbook owned by the platform team.
Point alerts to logs, not to data
A good alert links to the Airflow task log and the run page, where access is already governed, rather than copying output into the message. Check that the links resolve only for users who already have access to the DAG, because a link posted in a broad channel exposes the URL to everyone in it. Confirm your web base URL and authentication setup so the link is reachable from the on-call channel without opening new access paths.
Why alerts go missing or arrive as floods
This table lists the failures that most often leave Airflow alerts silent or flood the on-call channel, with the cause and fix for each. Read it as a checklist to run through when a notifier fires late, never fires, or repeats itself after a single failure.
| Failure | Likely cause | Fix |
|---|---|---|
| Failure alert never arrives | The notifier is attached to the DAG default arguments or to some tasks, but not to the tasks that actually fail | Attach the callback to every task that can fail, or set it once in default_args for the DAG and confirm it on a deliberately failing test task |
| ImportError for the Slack, PagerDuty or Opsgenie provider | The provider package is installed in the web or API image but not in the scheduler, DAG processor or worker image that runs the callback | Install each provider package separately in every image that executes tasks or callbacks, and pin it with the same constraints file used for Airflow itself |
| Callback runs but no message is sent | The connection ID in the notifier does not match the connection ID stored in Airflow, or the connection was created in a different environment | Compare the connection ID string in code with the one in the Airflow connection list for that environment, and fail fast in CI when a referenced connection is missing |
| Connection timeouts to Slack, PagerDuty or Opsgenie | Egress from worker nodes is blocked by firewall rules, proxies or a restricted network segment | Allow outbound HTTPS from the workers that run callbacks to the endpoints your organization uses, or route through the approved proxy, and test from the same node pool |
| Several alerts for one failure | Retries fire the failure callback on each attempt, so a task with three retries can send several notices | Decide whether alerts should fire on the first failure or only after retries are exhausted, and gate the notifier on the final try number or the retry state |
| Deadline alert never fires | The DAG run finishes or stalls before the deadline check is evaluated, or the run is not reaching the point where Airflow checks the deadline | Confirm the deadline is set on the task or DAG that represents the business expectation, test with a short deadline on a run that is intentionally slow, and check that the scheduler is running |
| Deadline alert fires once with no follow-up | Only a single notifier is passed, so there is no earlier warning before the escalation | Pass a list of deadline alerts to get a warning first and an escalation later, with each tier sent to a suitable channel |
| Alert arrives with no way to investigate | The message contains only the DAG ID or task name, without the run ID, try number or log location | Include the DAG run link, task instance link, try number, logical date and environment name in every message so the on-call engineer can open the run directly |
Questions buyers ask about Airflow on-call alerting
How much more effort is a PagerDuty or Opsgenie route than a basic Slack route?
A basic Slack route is mostly configuration: a Slack connection, an incoming webhook or API token, and a notifier attached to tasks or DAGs. A PagerDuty or Opsgenie route adds work that Slack does not need: service or integration keys, escalation policies, schedules that match your rotation, severity mapping, and a decision about which failures page a person at night and which only post to a channel. The Slack provider is a separate package installed with pip, and the PagerDuty and Opsgenie providers are separate packages too, so each route adds its own dependency, credential and test. Ask your on-call owner to define the paging rules before the build starts, because that conversation usually takes longer than the code.
What drives the cost and duration of Airflow alerting work?
The main drivers are the number of DAGs and the number of distinct owners, because each team usually wants its own channel, severity and escalation path. Next come on-call rules, including time zones, follow-the-sun coverage and the difference between a failed task and a missed deadline. Deadline design matters: a single deadline is simple, while tiered alerts that warn first and then escalate need more thought about timing and recipients. Secrets handling adds work when webhook URLs and integration keys must stay out of DAG code and sit in your secrets store. Testing adds work when alerts must be proven against a staging workspace before they reach a real on-call rotation. Each of these is a scope item you can price, and a written scope lets you compare quotes on the same basis.
Does the cost of alerting work recur after go-live?
Most of the build cost is one-time: designing routes, writing notifiers, configuring credentials and testing. Recurring costs come from changes that follow the platform, such as new DAGs that need routing, rotation changes, provider upgrades that must be tested against Airflow 3, and periodic review of alert volume. Your own team carries the recurring work unless you buy ongoing support. Ask any provider to separate the one-time scope from the support terms so you can see what is still your responsibility after handover.
Can our team set up Airflow alerts to Slack and PagerDuty without a partner?
Yes, if you have an engineer who already runs Airflow and someone who owns your on-call policy. The provider packages are documented, and a basic Slack notifier is a small change. What usually slows internal teams is not the code but agreeing on severity, deduplication, quiet hours and which owner receives each alert. A partner can help by writing the routing design, testing the failure paths and documenting the runbook, which is the same work regardless of who performs it. DeployOpen implements and documents these integrations inside your own infrastructure under a written scope, so the decision comes down to whether your team has the time and the on-call design skills.
What do our on-call and platform owners need to provide?
On the on-call side, you need named rotations, escalation contacts, severity definitions, and an agreed list of failures that should wake someone up. On the platform side, you need access to create the Slack app or webhook, the PagerDuty service or Opsgenie integration, and a place to store credentials, along with a staging Airflow environment to test in. You also need to confirm where alert links should point, such as the Airflow task log, the DAG run page and your runbook. Without these inputs, the work stalls at the decisions stage, so confirm them in the scoping meeting rather than during the build.
Do the Apache Airflow maintainers provide these alerting integrations?
The Airflow project is released under the Apache License 2.0 and provides the core scheduler, the provider packages and their documentation. The Slack, PagerDuty and Opsgenie integrations are provider packages published alongside Airflow, and the project documents how to install and use them. The project does not offer paid implementation, custom routing design or operational support for your environment. For those, you need your own team or a services partner. Check the provider package page for the minimum Airflow version each package supports, since that affects your upgrade plan.
What should each alert link back to so on-call engineers can act quickly?
Each alert should carry enough context to decide whether to act without opening the Airflow UI first. The useful links are the failed task instance log, the DAG run page, the DAG owner or team, the expected completion time for missed deadlines, and a runbook for that pipeline. Keep the alert text short and avoid including data values or secrets in the message body. Confirm with your security owner which fields may appear in a chat channel, since shared channels often have wider membership than the on-call rotation.
How DeployOpen scopes and delivers Airflow alerting
DeployOpen plans, implements, documents and supports self-hosted open-source platforms inside your own infrastructure, including Apache Airflow alert routing to Slack, PagerDuty and Opsgenie. We agree the scope in writing first, then build the notifiers, deadline alerts and runbook links against your on-call rules.
How we scope
- We list which DAGs, task failures and missed deadlines must reach on-call, and which should stay in a log channel.
- We confirm which of Slack, PagerDuty and Opsgenie your organization already uses, and where each provider's credentials will be stored.
- We agree the escalation tiers, retry suppression rules and link targets in a written, fixed-scope document before work starts.
How we build
- We configure provider packages and connections on your Airflow 3 environment, keeping secrets out of DAG code and out of environment variables.
- We add notifiers on tasks and Deadline Alerts, then test failure, retry and missed-deadline paths against non-production DAGs.
- We check that each alert links to the right run, log view and runbook, and that noisy retry states do not page anyone.
How we hand over
- You receive the configuration, DAG changes, connection inventory and a record of what each alert means and who acts on it.
- We walk your on-call and platform staff through test alerts, escalation paths and how to silence or adjust a rule safely.
- We document how to add a new DAG to alerting so future changes follow the same pattern.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer, including reviews after Airflow or provider package upgrades.
- Retainer work can cover tuning of alert thresholds as schedules change and new DAGs are added.
- Ask any provider about the notifier and provider versions your Airflow release supports before you agree the retainer scope.
