Why Airflow 3 teams move from SLAs to Deadline Alerts
A team running scheduled pipelines on Airflow 3 should know when a DAG run is going to finish late while there is still time to act, not after the window has already closed. Deadline Alerts are the Airflow 3 mechanism for that job. They replace SLAs, which Airflow 3 removes. Teams upgrading from Airflow 2 need to find every SLA definition in their DAG code and move it to a Deadline Alert before the upgrade, as the Upgrading to Airflow 3 guide requires. Treat that as a migration task with an owner, not a cleanup item for later.
A Deadline Alert is built from three parts: a reference point, an interval and a callback. The reference is the moment the deadline is measured from, such as when the run was queued, its logical date, a fixed datetime, or the average runtime of earlier runs of the DAG. The interval sets how far after that reference the deadline falls. The callback is what runs when the deadline is missed, for example a notifier that posts to a chat channel. Deadline Alerts were introduced in Airflow 3.1 and are still marked experimental, so confirm the current status and API in the release you plan to run before you build production alerting on them.
Before you choose a reference, decide who must act on a late run and how much warning they need. Average runtime suits pipelines whose duration is stable and well understood, while a fixed datetime suits a hard publishing cutoff. Confirm with your own run history which reference matches each DAG. Also confirm where the callback executes, because that determines whether your triggerer and workers must be sized and monitored for alerting, which the sections on async and sync callbacks and infrastructure requirements cover in more detail.
Choosing a deadline reference: queued time, logical date, fixed time or average runtime
Every Deadline Alert measures its interval from a reference point, so the reference you pick decides when the clock starts. The four options differ in what they measure, and the right one depends on how your schedule and data availability behave. Confirm each option's exact behavior in the deadline docs before you rely on it.
Queued time for on-demand runs
DAGRUN_QUEUED_AT starts the clock when the DAG run is queued. It measures how long a run takes from entering the queue to finishing, so it captures scheduler and executor delay as well as task time. This suits manually triggered or event-driven runs that have no schedule-based start point. Check whether queue delays from a busy scheduler or full pools should count against the deadline, because a late alert may reflect capacity rather than a slow pipeline.
Logical date for scheduled windows
DAGRUN_LOGICAL_DATE anchors the reference to the logical date of the run, which represents the data interval the run processes. It fits pipelines that must finish processing a period by a fixed offset after that period's close, such as a daily load that must complete before a business-day report. Confirm how your schedule and timetable set the logical date, since cron and timetable choices change which date the alert uses.
Fixed time for hard cutoffs
FIXED_DATETIME uses one absolute point in time, independent of when the run was queued or how long it has taken. It fits external commitments such as a regulatory filing, a market data cutoff or a partner file pickup. Because the deadline does not move with the run, a backfill or late run can trigger the alert immediately. Decide in advance which runs should be checked against a fixed cutoff and which should not.
Average runtime for drift detection
AVERAGE_RUNTIME compares the current run against the historical average duration of that DAG. It does not depend on a calendar time, so it flags runs that are slower than usual even when the schedule still leaves room. It helps catch gradual slowdowns in growing datasets or degraded sources. It needs enough run history to produce a meaningful average, so confirm how it behaves for new DAGs and after major code changes.
Async versus sync callbacks and where each runs
Async callbacks are the mode Deadline Alerts uses from the start, and they run in the triggerer rather than on a worker. When a deadline's reference and interval are reached, the alert is handled by a trigger, and the triggerer process is what executes the callback code. Because the callback does not occupy a task slot, it does not compete with your DAG tasks for executor capacity.
- What must be running: a triggerer process must be up and connected to the metadata database, or async alerts will not fire.
- Failure mode: if the triggerer is down, deadline alerts are missed silently from the DAG's point of view, so the DAG run looks healthy while the alert never arrives.
- Scaling: deferrable work and deadline callbacks share the triggerer, so capacity planning should include both workloads.
- Notifier fit: built-in notifiers such as SlackWebhookNotifier are the usual choice for async callbacks, since they make an outbound HTTPS call without needing task resources.
- Network: the triggerer host needs outbound access to the webhook or messaging endpoint, which may require an egress rule in restricted networks.
Before go-live, confirm with your platform team that the triggerer runs under the same supervision as the scheduler and that its health is alerted on.
Sync callbacks were added in Airflow 3.2 and run in the executor, so the callback is dispatched and executed the same way a task would be. This suits callbacks that need the worker environment, such as internal libraries, network paths or credentials that only workers hold. The trade-off is that the callback now depends on executor capacity and on the worker environment being healthy.
- What must be running: a working executor with available slots, plus the scheduler that dispatches the callback.
- Blocked executor effect: if the executor queue is saturated or workers are down, the alert waits behind other work and may arrive late, which defeats the purpose of a deadline.
- Dependency surface: the callback inherits worker images, libraries and secrets, so a broken worker build can break alerting as well as tasks.
- Version check: sync callbacks require Airflow 3.2 or later. Confirm the version in your environment before choosing this mode.
- Testing: exercise the callback on the same executor and image your production DAGs use, not only on a local run.
Check that the executor has headroom for alert callbacks during peak schedules and that the alert path is tested separately from task failures.
The choice between modes is mostly about which runtime you want the alert to depend on. Async callbacks keep alerting independent of worker load, but they depend on the triggerer. Sync callbacks keep the callback in the same environment as your tasks, but they depend on executor health. Neither mode is a substitute for monitoring the scheduler and the components the alert depends on.
- Independence: pick async when alerts must still arrive during worker saturation.
- Environment access: pick sync when the callback must reach systems only workers can reach.
- Tiered alerts: a warning and an escalation can use different modes if the warning is low stakes and the escalation must use a different path.
- Dual paths: for critical pipelines, some teams keep a second, independent signal, such as a scheduler or metrics alert, so a single component failure does not hide a late run.
- Documentation: record the chosen mode per DAG so on-call staff know where to look when an alert is missing.
Ask your team which failure you would rather tolerate, a delayed alert or a silent one, and design the second signal around that answer.
Operating deadline alerts means treating the triggerer and executor as dependencies of the alerting path, not only of task execution. Routine checks should confirm that each component is running, that its logs are reachable, and that a test alert arrives within the time your on-call process expects. Health checks on the triggerer are easy to overlook because it often runs as a small process beside the scheduler.
- Component health: monitor the triggerer and the scheduler as separate services with their own restart policies.
- Test alerts: run a deliberately late test DAG on a schedule, so the alert path is exercised regularly.
- Credentials: rotate Slack or webhook secrets through the same process used for other connections, and confirm the callback still resolves them.
- Upgrades: re-test callbacks after each Airflow upgrade, since the callback runtime changed between modes.
- Ownership: name the team responsible for the triggerer and executor so alert gaps have an owner.
Before relying on alerts for regulated or customer-facing pipelines, confirm the test results and the monitoring coverage with your platform and on-call leads, and document the agreed behavior.
Deadline Alert parameters and when each setting fits
This table compares the main Deadline Alert settings: the reference point, the interval, the callback type, the notifier and the tiered list form. Read each row for the situation it suits and the trade-off it carries, then combine rows to match how your team responds to late runs.
| Setting | When it fits | Trade-off to weigh |
|---|---|---|
| Reference: DAGRUN_QUEUED_AT | Alert on runs that wait too long before starting, such as when pools or executor capacity are saturated. | Measures queue delay only, so a run that starts on time but runs slowly will not trigger it. |
| Reference: DAGRUN_LOGICAL_DATE | Protect a data delivery that must finish by a fixed point after the data interval the run represents. | Anchors to the logical date, so backfills and manual runs with older logical dates may alert immediately. |
| Reference: FIXED_DATETIME | Enforce a hard clock deadline, such as a daily report that must be published by a set time. | Does not adapt to run duration, so a legitimately slow day still breaches the deadline. |
| Reference: AVERAGE_RUNTIME | Catch runs that drift well beyond their normal duration without hand-tuning a fixed time for each DAG. | Depends on history; new or changing DAGs have little baseline, so confirm how the average is calculated in your version. |
| Interval | Set the time after the reference point that counts as late, based on the downstream consumer's real tolerance. | Too short creates noise on normal variance; too long delays the warning past the point where action helps. |
| Async callback (Triggerer) | Send notifications that do not need worker slots, such as a Slack message on a late run. | Needs a running triggerer and the deadline must be waiting in the triggerer; confirm the triggerer is highly available before relying on it. |
| Sync callback (executor, added in Airflow 3.2) | Run callback logic that must execute on an executor worker, such as code that needs worker-side network access. | Uses executor capacity and depends on worker availability; confirm the Airflow version supports it before choosing it. |
| Notifier such as SlackWebhookNotifier | Post a standard message to a team channel without writing custom callback code. | Message content is limited to what the notifier supports; add ownership context in the text so responders know what to do. |
| List passed to the deadline parameter | Send an early warning and then an escalation for the same run, such as a team alert followed by an on-call page. | Each tier adds a callback to maintain and test; the escalation tier needs its own clear owner and runbook. |
Setting up a Deadline Alert in your Airflow 3 environment
This sequence takes a team from confirming that its Airflow version and components support Deadline Alerts to proving that a late DAG run produces the alert it should. Each step ends with a check you can run before the alert is relied on for operations.
Confirm the Airflow version and experimental status
Check that the environment runs Airflow 3.1 or later, since Deadline Alerts were introduced in that release. Confirm the running version with the scheduler, API server and workers, not only the image tag in your repository. Note that the feature is marked experimental, so record that in your change request and agree with the platform owner how you will handle behavior changes in later releases. The step is done when every component reports a 3.1 or later version and the owner has signed off on experimental use.
Confirm the triggerer is running and sized
Async deadline callbacks run in the triggerer, so a DAG alert will not fire if no triggerer is active. Confirm that at least one triggerer process is running, that it is healthy in your monitoring, and that it has the capacity for the number of deadlines you plan to watch. Ask your platform team how the triggerer is restarted and whether it runs in a separate failure domain from the scheduler. The step is done when the triggerer reports healthy and you have a documented owner for restarts.
Confirm executor capacity for sync callbacks
Sync callbacks, added in Airflow 3.2, run in the executor rather than the triggerer. If you plan to use them, confirm that the executor you run (for example, Celery or Kubernetes) can start a short callback task at the moment a deadline is missed, even when other tasks are queued. Check the queue or pool that callback tasks use so they are not blocked behind long-running work. The step is done when you have confirmed the 3.2 or later version and tested that a callback task can start while the cluster is under normal load.
Define the deadline on the DAG
Choose a reference point for the deadline, such as the queued time of the run, its logical date, a fixed datetime, or the average runtime of previous runs, and set an interval from that reference. Add the deadline parameter to the DAG definition with the reference, interval and callback. Keep the deadline in version control with the rest of the DAG so reviewers can see it. The step is done when the DAG parses without errors in the DAG processor and the deadline appears in the DAG's view in the UI.
Attach the notifier or custom callback
Attach a notifier such as SlackWebhookNotifier, or write a custom callback that sends the alert to your incident system. Store the Slack webhook in your secrets backend rather than in DAG code, and confirm with your security team which channel receives production alerts. If you need tiered alerts, pass a list to the deadline parameter so that a warning is followed by an escalation, and confirm the interval for each tier. The step is done when a test message reaches the intended channel from a manual test call.
Test with a controlled late run
Run a test DAG that is designed to miss its deadline, for example by using a short interval and a task that sleeps longer than the deadline. Confirm that the callback fires in the expected component (triggerer for async, executor for sync) and that the alert reaches the channel. Check the task and scheduler logs for the callback's start and completion. Record the result, and remove or disable the test DAG before it runs in shared environments. The step is done when the alert fires once for the controlled late run and the evidence is saved with the change record.
Document the alert and set review points
Record which DAGs have deadlines, the reference and interval used, the callback destination, and the component that runs each callback. Name the team that responds to alerts and how they are routed. Schedule a review after each Airflow upgrade, because the feature is experimental and its behavior may change. Ask your vendor or the project documentation for any open issues before you rely on the alert in production. The step is done when the register is stored with the DAG repository and the review date is on the owner's calendar.
Tiered warnings and escalations for late DAG runs
Passing a list to the deadline parameter is how you turn a single late-run alert into a warning followed by an escalation. The first tier should reach the people who can still act before the run misses its target, usually the DAG owner or the data team's channel, and the later tier should reach the on-call rotation only if the run is still not finished. Each entry in the list is a separate Deadline Alert with its own reference, interval and callback, so the order of the intervals decides the order in which people are paged. Before you write the list, confirm the current parameter shape and tier behavior in the Deadline alerts documentation, since the feature is marked experimental in Airflow 3.1 and its callback options have changed around the 3.2 release.
Set interval ordering so that the warning fires with enough time left for a human to intervene, and the escalation fires only when that intervention has clearly failed. Keep the escalation interval well separated from the warning, so a single slow stage does not generate two pages within minutes. Test the chosen intervals against a DAG's real runtime history, not against the schedule interval, and confirm with your on-call lead how many pages per run is acceptable. A common mistake is to attach the same on-call-facing Slack notifier to every tier, which trains responders to mute the channel. Illustrative example: a warning goes to the owning team's channel, and only a run still unfinished at the later interval posts to the incident channel.
Security and operational considerations for alert callbacks
A deadline alert is only useful if its callback is trusted, its payload is appropriately scoped, its definitions are reviewed like any other code, and someone notices when the alert path itself fails. The cards below cover those four concerns for Airflow 3 Deadline Alerts and the notifiers that send them.
Webhook secret handling for notifiers
Slack and other webhook notifiers rely on a URL that works as a credential, so anyone who reads the DAG file or the task logs can post to your channel. Keep webhook URLs in an Airflow connection or a supported secrets backend rather than in DAG source. Ask who can read connections in your deployment, and rotate the webhook if it has ever been pasted into code, a ticket or a chat message.
What alert payloads expose
A deadline callback typically identifies the DAG, the run and the timing involved, and the message body is sent to a channel that may have many members. Review the text your callback builds before it reaches a shared channel, and avoid putting record contents, customer identifiers or data values into the message. Ask the security team which channels are approved for pipeline metadata in your environment.
Who can change deadline definitions
Deadline Alerts are declared in DAG code, so changing a reference, interval or callback is a code change. Route changes to deadline definitions through the same pull request review and branch protection used for other DAG code, and require review from the owning data team. A silently loosened or removed alert is a control gap, so treat deletions as seriously as additions.
Scope of alert recipients
Tiered alerts escalate from a warning to a broader audience, so the escalation target determines who learns about sensitive pipeline activity. Confirm which people or groups receive each tier and who is accountable for acting on it. Document the recipients as part of the alerting design so that staff changes do not leave an escalation pointing at a channel nobody watches.
Monitoring the alert path
An alert that fails silently looks the same as a pipeline that is on time. Check that the triggerer or worker that runs the callback is healthy, that its logs record send failures, and that a test deadline fires end to end on a schedule. Confirm with your platform team how callback failures reach an on-call rotation, since the alert system cannot report its own outage.
Questions teams ask before replacing SLAs with Deadline Alerts
What does the migration effort depend on?
The effort depends mostly on how many DAGs carry SLAs today, how many distinct alert rules those DAGs need, and how much logic sits behind each notification. Teams with a small set of critical pipelines usually have a short list to rebuild. Teams that set SLAs on every task across hundreds of DAGs face a larger review, because each DAG needs a chosen reference, an interval, a callback and an owner for the alert. Also count the upgrade work around it: SLAs are removed in Airflow 3, so any DAG that depends on them must change before the upgrade lands. Ask for an inventory of existing SLA definitions as the first deliverable, since it sets the scope for everything else.
Can our in-house team replace SLAs with Deadline Alerts themselves?
Yes, a team with Airflow 3 experience and access to its own DAG repository can make the change without outside help, provided it has the time and the skills to test callbacks. The work is mostly configuration and code review, not new infrastructure. The skills that matter are knowing which reference fits each pipeline, writing callbacks that handle failure cleanly, and checking that alerts fire in a staging environment before production. Teams that lack this experience, or that are also running an upgrade from Airflow 2, often choose to bring in a partner for the review. Compare the internal estimate against a written, fixed scope before deciding.
What does the work need from our team?
Expect to supply four things: time from the engineers who own the DAGs, access to the triggerer and worker environments, credentials for the notification channels, and a reviewer who signs off on each alert rule. Deadline Alerts with async callbacks run in the triggerer, so the triggerer must be deployed, running and reachable before the alerts work. Sync callbacks run in the executor. Notifier credentials, such as a Slack webhook, should be stored in your secrets backend rather than in DAG code. Ask the partner to name exactly which roles must review each change, so that alert ownership is clear after handover.
How long does the change take with many DAGs and complex callbacks?
Duration scales with the number of DAGs, the number of alert tiers per DAG and the complexity of each callback. Simple notifier callbacks on a few pipelines can be configured and tested quickly. Custom callbacks that query other systems, route alerts by team, or escalate through several channels need design, testing and review for each variant. Tiered alerts with a warning followed by an escalation multiply the test cases. No duration figure is given here because it depends on your environment. Ask for a timeline in the written scope that lists the DAG count, the callback types and the test environments included.
Does experimental status change how we should plan?
Yes, plan for it. Deadline Alerts were introduced in Airflow 3.1 and are marked experimental in the project documentation, so their behavior, parameters or callback options may change in later releases. Sync callbacks were added in 3.2. Pin your Airflow version, keep alert logic in version control and test each upgrade against your alert rules before rolling it out. Keep a fallback monitoring check, such as a separate task or external job that reports late runs, until your team has confirmed the alerts behave as expected in production. Review the Deadline Alerts documentation before each planned upgrade.
Which reference should we choose for each DAG?
The reference decides when the clock starts, so it should match what the business means by late. The supported reference types are DAGRUN_QUEUED_AT, DAGRUN_LOGICAL_DATE, FIXED_DATETIME and AVERAGE_RUNTIME. Use a queue-based or logical-date reference when the deadline is tied to when data should be ready. Use AVERAGE_RUNTIME when a run should be flagged because it is taking longer than usual. Document the choice for each DAG, because a wrong reference produces alerts that are either too late to act on or too noisy to trust.
Does this change affect our security or compliance evidence?
It can, because alert routing and notifier credentials become part of your operational controls. Record which DAGs have deadlines, who owns each alert, which channels receive them and how changes are approved. Store notifier secrets in your approved secrets backend and restrict who can edit DAG code that contains alert logic. DeployOpen prepares systems, controls and evidence for compliance assessments, but it does not issue audit reports or certifications. Independent auditors and certification bodies make those determinations, so confirm your evidence requirements with them before the change.
Official Airflow documentation for Deadline Alerts and upgrades
Start with the Deadline alerts how-to before writing any configuration. It sets out the reference types (DAGRUN_QUEUED_AT, DAGRUN_LOGICAL_DATE, FIXED_DATETIME and AVERAGE_RUNTIME), the interval and callback parameters, and how a list passed to the deadline parameter produces tiered warnings. Implementation teams should read it alongside their own run history, because the choice of reference decides whether an alert measures queue delay, schedule drift or runtime overrun. Confirm in the page which callbacks run in the triggerer and which run in the executor, since that decides where the triggerer and worker capacity must be provisioned and monitored.
Read the Upgrading to Airflow 3 guide next if the DAGs still use SLAs. It records that SLAs are removed in Airflow 3 and replaced by Deadline Alerts, so every sla argument must be translated before cutover. The same guide lists the other removals that affect alerting work, including SubDAGs, the SequentialExecutor and the --subdir flag. It also states the minimum upgrade starting point and recommends checking DAG code with the Ruff AIR301 and AIR302 rules, which is the fastest way to find SLA usage across a large code base.
Plan a short review with the DAG authors and the on-call team after reading both pages. Ask the reviewer to confirm three things: which Airflow version the platform runs, whether the Deadline Alerts feature is available and stable enough for the organization's risk tolerance, and who owns each callback destination. Because Deadline Alerts are marked experimental in the documentation, record the decision and the rollback path in the change record. Where the team cannot confirm a behavior from the documentation, test it in a non-production environment with a deliberately slow DAG before relying on it for production escalation.
How DeployOpen scopes and delivers Deadline Alert work
DeployOpen plans, implements, documents and supports self-hosted open-source platforms inside your own infrastructure, and Deadline Alert work follows the same path. We agree the scope in writing first, then build and test the alerting path in your environment, including the triggerer and worker requirements, before handing it over.
How we scope
- Inventory which DAGs need a finish-time expectation and which reference each one should use
- Confirm the Airflow version, since Deadline Alerts are new in 3.1 and marked experimental
- Agree in writing which callbacks, notifiers and escalation tiers are in and out of scope
- Engagements are fixed-scope, with deliverables and timelines agreed before work starts
How we build
- Define each deadline with its reference, interval and callback in version-controlled DAG code
- Choose async or sync callbacks based on what the triggerer and executor can reach in your network
- Configure the Slack notifier or other agreed callbacks with secrets kept out of DAG files
- Test late-run and on-time scenarios in a non-production environment before enabling alerts
How we hand over
- Document each deadline, its reference, owner, alert route and escalation tier in a runbook
- Verify that the triggerer and workers run where the alert paths require them
- Walk your operators through how to read an alert, silence it and tune an interval
- Record the configuration and test evidence as part of the delivered scope
How we support
- Ongoing support, upgrades and continuous testing are available as retainers
- Review alert behavior after Airflow upgrades, since Deadline Alerts are an evolving feature
- Ask a quote to separate one-time build work from any recurring support you need
- Confirm with your team which support hours and response expectations apply before signing
