Airflow deferrable operators and triggerer

When Airflow deferrable operators free worker capacity

Deferrable operators stop a waiting task from holding a worker slot. A classic sensor polls for a file, a partition or a external job from start to finish while occupying a worker, so a pool of workers can fill up with tasks that do nothing but check a condition. A deferrable operator instead hands the waiting logic to the triggerer, a separate Airflow component that runs many lightweight triggers in one asyncio event loop, and releases the worker until the trigger fires. Your scheduled work then competes for slots only when it actually runs.

The gain matters most in three situations: many long waits that overlap, such as hours-long upstream jobs or partitions that land late; shared worker pools where idle sensors block tasks from other teams; and clusters where each idle slot has a real cost, whether that is cloud compute, Kubernetes node capacity or licensed infrastructure. In these cases, a team should count how many slots are occupied by tasks in a waiting state before choosing a design. The triggerer becomes a new dependency, so its capacity, its database load from trigger state and its restart behavior belong in the same plan as the workers it frees.

Exact behavior differs by operator, provider package version and Airflow release, so confirm which operators in your provider set support deferral and which settings control polling intervals before you rely on them. The Apache Airflow documentation and the release notes for your target version are the reference for these details, and your own test DAGs should prove the slot savings you expect.

Which waiting pattern fits each task

This table compares the three ways a team usually handles a waiting step in Airflow: a deferrable operator or sensor, a classic sensor that holds a worker slot, and a custom polling task. Read each row as a trade-off between worker capacity, triggerer load and how much provider-specific behavior you must verify before relying on it. The main takeaway is that deferral only pays off when the specific operator documents deferrable support and your triggerer is sized and monitored for the load.

Pattern	When it fits	Cost in worker slots and triggerer load	What to confirm in provider documentation
Deferrable operator or sensor	Long waits on an external event, such as a file landing, a job finishing or an API status changing, where the provider ships a trigger	Worker slot is released while waiting. The triggerer holds the wait as a lightweight async process, so its load grows with the number of open waits	Confirm the operator or sensor lists deferrable mode, the trigger class it uses, and any extra connection or permission needs
Classic sensor in reschedule mode	Waits that are short or infrequent, or where no trigger exists for the target system yet	Worker slot is released between pokes but is occupied during each check. Frequent pokes add scheduler and worker overhead without a triggerer	Confirm the poke interval, timeout and soft-fail behavior, and whether the sensor is marked as deprecated in favor of a deferrable version
Classic sensor in poke mode	Very short waits where the check completes quickly and holding a slot costs less than a reschedule cycle	Worker slot stays occupied for the full wait. Long waits can exhaust pool and concurrency limits and block other tasks	Confirm pool assignment and concurrency limits, since a held slot counts against them for the whole wait
Custom polling task	Logic a provider does not cover, such as a proprietary status endpoint with unusual authentication or retry rules	Cost depends on how the poll is written. A loop inside a task holds a slot, while a custom trigger moves the wait to the triggerer but needs its own tests and ownership	Confirm who owns the code, how it handles credentials through connections, and whether it must run inside the triggerer with async-safe I/O

How deferred tasks move through the triggerer

A deferred task gives up its worker slot and hands the waiting work to a trigger, which is a small async piece of Python that the triggerer component runs. When an operator defers, the task instance records its state in the metadata database, releases its worker, and waits in a deferred state. The triggerer picks up the trigger, evaluates the condition it watches (a file landing, a remote job finishing, a timer elapsing), and when the condition is met it fires an event. The scheduler then resumes the task on a worker at the method the operator named when it deferred. Platform teams should treat the trigger as the only part of the wait that stays alive, so its code, its external connections and its dependencies are what need to be reviewed and sized.

The triggerer runs many triggers concurrently inside one process using asyncio, so a single triggerer can hold a large number of waiting tasks without a process per wait. That is the source of the capacity gain, and also its main constraint: a trigger that blocks the event loop with synchronous calls slows every other trigger in the same process. Trigger code should use async clients and should avoid heavy work in its run loop. Confirm for your Airflow version how the triggerer is started, how many instances you can run, and how it claims triggers across instances, because the exact distribution behavior and the required database access are version-specific and are described in the Airflow executor documentation and related core pages.

Between deferral and resumption, the state lives in the metadata database, not in the triggerer's memory. The trigger definition, its serialized arguments and the task instance's next-method details are stored there, so a deferred task can survive a worker restart or an intentional drain. If the triggerer restarts, triggers it had loaded are reloaded from the database and resume watching their conditions, but any event that fired during the gap is handled when the trigger is running again. The practical risk is duplicate or delayed resumption and triggers that were written to assume a single process. Before go-live, test a triggerer kill during a live deferred run, confirm in your environment that the task resumes once, and check how long a missed event can take to surface. Make the metadata database highly available first, since it is the state store for every deferred task.

Setting up the triggerer and deferrable tasks

This sequence takes a self-hosted Airflow 3 environment from version and database checks to a running triggerer, one converted sensor in a non-production DAG, and evidence that deferred tasks resume and release their worker slots. Each step ends with a check you can record before moving on.

Confirm the Airflow 3 target and supported versions
Your platform or data engineering lead confirms the Airflow 3 patch release to target, the Python version, and the metadata database version against the prerequisites for that release. Supported Python versions, PostgreSQL and MySQL versions, and Kubernetes versions are listed there, and MariaDB and SQLite are not suitable for the metadata database in production. Confirm whether your current Airflow 2 installation is on a supported line, because the version support table shows Airflow 2 reaching end of life, after which it gets no fixes. Done when you have a written version matrix that the team has signed off, and when the target database is reachable from every component that will connect to it.
Confirm the upgrade path if coming from Airflow 2
If you are upgrading rather than installing fresh, the upgrade guide requires being on Airflow 2.7 or later and recommends moving to the latest 2.x release first. Run the Ruff AIR301 and AIR302 rules over your DAG code and record every finding, because task code can no longer use Airflow database sessions or models directly and SubDAGs have been removed in favor of TaskGroups. Done when the lint report has no unresolved findings for the DAGs that will carry the sensor conversion, and when those findings have owners.
Deploy the triggerer as its own process
The platform engineer starts the triggerer as a separate component next to the scheduler, API server and DAG processor, all connected to the same metadata database and the same DAG delivery source. In Airflow 3 the DAG processor already runs separately, so the triggerer should sit alongside it rather than inside the scheduler process. Use the same Airflow version and configuration across these components, and keep the triggerer on a host or pod that can reach the systems your deferrable operators will poll. Done when the triggerer appears as a running component in your process supervisor or Kubernetes deployment, its logs show a clean start, and it registers a heartbeat in the metadata database.
Verify the triggerer has a live heartbeat
Before any DAG depends on it, check that the triggerer is healthy from the outside. Confirm that the Airflow health endpoint reports the triggerer as alive, and that the component appears in the UI or CLI status output. Confirm the component can open outbound connections to the endpoints your triggers will call, such as an API, a queue or a database, from the same network zone. Done when two consecutive health checks pass over a period your team chooses, and when a test connection from the triggerer host succeeds to every endpoint in the first deferrable use case.
Convert one sensor in a non-production DAG
A DAG author picks one existing sensor that spends worker time waiting, such as a file or partition sensor, and switches it to its deferrable form in a non-production environment. Keep the schedule, timeout and downstream dependencies unchanged so the only variable is the execution mode. Record the sensor's worker slot usage before the change, since you will compare against it. Done when the DAG parses without errors in the DAG processor, the converted task is visible in the UI with a deferred state, and the reviewer has approved the change in version control.
Trigger a run and watch the deferred state
Trigger the test DAG manually and observe the converted task move into a deferred state. Confirm that the triggerer is the component holding the wait, and that the worker process for that task is released instead of polling. Check the triggerer logs for the trigger registration and the condition it is watching. Done when the task shows deferred in the UI, the worker slot count drops for that task, and the triggerer log names the trigger for the run.
Verify resumption and slot release
Create the condition the sensor waits for, then confirm the task fires, resumes on a worker and completes, and that downstream tasks start. Also test failover by restarting the triggerer during a deferred wait, then confirm that the trigger is picked up again and the task still completes. Record the before and after worker slot figures from the earlier step. Done when the task completes after the condition is met, downstream tasks run in order, the restart test produces no lost or duplicate run, and the slot comparison is written into the change record for the production decision.

Running the triggerer with failover across instances

Running more than one triggerer changes how deferred tasks are held, woken and recovered. The cards below cover the decisions that connect trigger distribution, scheduler and database concurrency, orchestrator health checks and the failover tests to run before production.

Trigger distribution across instances

Each deferred task hands a trigger to the triggerer, which runs the async wait outside a worker slot. With several triggerers, decide whether work spreads across instances or whether one instance holds most triggers. Confirm in the Airflow documentation for your version how triggers are assigned and whether two instances can ever run the same trigger. Check the number of triggers each instance holds under a realistic load of deferred tasks before you size the fleet.

Scheduler concurrency and row-level locks

The scheduler supports running more than one instance at the same time, coordinating through database row-level locks. Triggerers also write to the metadata database when triggers fire and tasks resume, so the database must handle the added concurrent writes. Confirm the lock behavior and connection limits for your version, and size the connection pool for every scheduler, triggerer, API server and DAG processor together.

Metadata database support

Production Airflow needs PostgreSQL or MySQL as the metadata database, because SQLite is for testing only and MariaDB is not supported. Confirm that your chosen database version is on the prerequisites list for the Airflow 3 release you run, and that the database has its own high availability and backups. A triggerer failover is only as reliable as the database it writes trigger state to.

Health checks and restarts in your orchestrator

Your orchestrator decides when a triggerer is considered dead and how quickly it is replaced. Define a liveness check that confirms the process is working, not only that it started, and set restart policies so a crashed triggerer returns without manual action. On Kubernetes, use a deployment or stateful set with resource requests that leave room for trigger load. Document which signal triggers the alert and who responds.

Testing a triggerer failover

Test failover before production by stopping one triggerer while deferred tasks are waiting, then confirm that the remaining instance resumes or continues those triggers without duplicate or lost events. Repeat the test with the database under load and with a restart of the whole fleet. Record how long deferred tasks stayed waiting after the stop, and whether any task failed or ran twice. Use the results to set alert thresholds.

Confirming triggerer high availability for your version

Triggerer high availability behavior must be confirmed in the Airflow documentation for the version you run before you design around it. Ask whether the triggerer has documented support for multiple instances, how trigger ownership is recovered after a crash, and which settings control that recovery. Keep the answers in your runbook so upgrades that change this behavior are caught during the upgrade test cycle.

Scaling the triggerer by environment

On virtual machines, the triggerer is a process you place on a host and supervise with your own service manager. Its placement follows the same logic as the scheduler and workers: give it hosts that can reach the metadata database and the systems its triggers call. Triggers write state through the metadata database, so the database must be on the supported list, which is PostgreSQL 14 to 18 or MySQL 8.0, 8.4 or an Innovation release. MariaDB is not supported as the metadata database, and SQLite should not be used beyond testing.

Placement. Run at least two triggerer processes on separate hosts so one host failure does not stop deferred tasks from resuming. Confirm in the documentation for your version how multiple instances share trigger work.
Memory. Start each host from the recommended minimum of 4 GB of RAM, then add headroom if the API server, scheduler or DAG processor share the same host.
Capacity signals. Watch the number of deferred task instances, the delay between deferral and resume, triggerer log errors and database connection counts.
Image and upgrades. Without containers, each upgrade updates the Python environment with the constraints file on every host. Follow the upgrade order in the Airflow 3 upgrade guide so triggerers and the scheduler do not run mismatched versions.
Process supervision. Use systemd or your equivalent to restart a failed triggerer and alert when a process exits.

Before adding a second instance, check that every host runs a Python version from 3.10 to 3.14 and that the database version appears on the prerequisites list.

What deferrable operators cost to adopt and run

How much effort does it take to convert existing sensors to deferrable operators?

The effort depends on how many sensors and long polls exist, how many of them have a deferrable equivalent in the provider package you already run, and how much custom polling logic sits inside each one. A sensor that waits on a standard condition is usually a small change, while a sensor with bespoke retry, timeout or callback code needs its own trigger design and tests. No conversion effort figure is supplied for this topic, so ask any partner to inventory your sensors first and quote per category. Confirm during that inventory which provider versions you run, because deferrable support varies by package.

Can our platform team convert the sensors and run the triggerer on its own?

Often yes, if the team already operates Airflow 3 in its own infrastructure and has someone comfortable with the metadata database, container deployment and provider packages. The harder parts are usually the triggerer's placement and scaling, the testing of each converted DAG against real waiting times, and the change control around a fleet of DAGs that all depend on the same pattern. Ask whether your team has capacity to own those steps after go-live. Airflow's own docs on the triggerer and deferrable operators are the right starting point for that self-assessment.

What drives the cost of this work up or down?

The main drivers are:

The number of waiting tasks and how long they typically wait, since this sets how much worker capacity is at stake and how many triggers the triggerer must hold.
Provider coverage, meaning whether each external system has a deferrable operator or trigger, or whether custom triggers must be written.
Database and triggerer sizing, which depends on trigger volume and the PostgreSQL or MySQL instance behind the metadata database.
Testing, including staging runs that prove failover and restart behavior.

Each driver has a different cost owner, so ask for each line separately in a quote.

What parts of this work are one-time and what recurs after go-live?

One-time work includes the sensor inventory, conversion of existing DAGs, custom trigger development where needed, triggerer deployment and the initial failover test. Recurring work includes monitoring the triggerer, applying Airflow patch releases, keeping provider packages compatible with the Airflow version, and re-testing when new DAGs use the pattern. Airflow 2 reached end of life on April 22, 2026, and the support policy means only current versions receive fixes, so budget for regular upgrades rather than a single project. Ask any partner to separate these two groups in the quote.

What do we need to provide before the work starts?

Plan to provide access to the Airflow environment and its metadata database in a non-production setting, a representative set of DAGs that use sensors or long polls, and test data or stub endpoints for the external systems those tasks wait on. You will also need a staging environment that matches production's executor and deployment pattern closely enough to prove triggerer failover. Confirm with your security team how access is granted, since triggers often hold connection details. The partner should list these items in the written scope before work begins.

How do we ask for a fixed-scope quote for this work?

Most DeployOpen engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts, and the quote follows from that written scope. To get a comparable quote from any partner, provide the sensor inventory, the provider packages and Airflow version in use, the executor and deployment platform, the number of environments, and the testing and failover requirements. Ask for deliverables by name, the acceptance criteria for each, what is excluded, and which recurring items are priced separately, such as retainers for upgrades and monitoring.

Security and operational risks of the triggerer

The triggerer holds credentials, opens network paths to external systems and writes to the metadata database, so it carries risk that a worker-only design does not. The cards below cover the areas to review before the triggerer runs production deferrable work, starting with what it can reach and ending with how you will upgrade it.

Credentials used by triggers

Each trigger that polls an external system needs credentials, and those credentials live where the triggerer process can read them. Confirm which Airflow connections the triggerer resolves, which secrets backend supplies them and who can read that backend. Scope each credential to the single read the trigger performs, since a compromised triggerer exposes every credential it can resolve. Rotate them on the same schedule as worker credentials.

Network paths to polled sources

Triggers open outbound connections to the systems they watch, such as APIs, message brokers or file stores. Because the triggerer often runs in a different subnet from workers, its firewall rules can differ from theirs. Document each destination host and port, allow only those egress paths and confirm that TLS verification is enabled for each one. Review these rules whenever a new deferrable operator is added.

Metadata database access and backups

The triggerer reads and writes trigger and task state in the metadata database, so its database role needs the same protection as the scheduler's. Create a dedicated database account with only the privileges the triggerer needs and keep it off shared admin roles. Include the trigger tables in your point-in-time recovery plan and test a restore, because a restored database with missing trigger rows leaves deferred tasks stranded.

Logging without leaking values

Trigger activity appears in triggerer logs, and a poorly written trigger can print request URLs, headers or payloads that contain tokens or personal data. Review the log level used in production and confirm that Airflow's secret masking covers values your triggers pass around. Set a retention period for triggerer logs that matches your data policy, and test a sample trigger to see exactly what gets written.

Monitoring the triggerer process

A triggerer that stops or falls behind does not fail loudly. Deferred tasks simply wait, and sensors that should have resumed stay idle. Monitor the process itself with a liveness check, alert on a gap in heartbeats from each instance, and watch the count of deferred tasks for growth that does not clear. Confirm that your alerting reaches the team that owns the pipelines, not only the platform team.

Upgrade path for the triggerer

Airflow 3 changed how its components are deployed, and Airflow 2 reached end of life on April 22, 2026, so it no longer receives fixes. Plan the triggerer as part of the wider move to Airflow 3, with the API server and DAG processor deployed as separate components. Confirm your target version against the upgrade guide before any triggerer change, and test trigger behavior in staging with the same versions you will run in production.

Limits to confirm in your environment

Deferrable operators move waiting out of worker slots and into the triggerer, which shifts load and failure modes to new places. The cards below cover the checks that decide whether that shift helps your pipelines, so confirm each one against the Airflow version and executor you plan to run.

Deferrable forms by provider

Not every sensor or operator in your stack has a deferrable form, and availability depends on the provider package version as well as the Airflow version. List the sensors and long-polling operators your DAGs use, then check each provider's release notes for a deferrable mode. Where none exists, a blocking worker slot remains for that task, so plan the capacity for it separately.

Trigger behavior under restarts

A deferred task depends on its trigger running in the triggerer. Test what happens to waiting tasks when a triggerer pod or process restarts, is drained for an upgrade, or loses its database connection. Confirm that triggers resume without duplicate firing or missed events for the external conditions your workloads wait on, such as file arrivals or job completions.

Database load from deferred tasks

Each deferred task stores its state in the metadata database, and many simultaneous waits add rows, writes and polling traffic. Estimate the number of tasks that could be deferred at the same time during your peak windows, then load test the metadata database with that volume. Include database cleanup and backup timing in the same test, since large deferred state can slow both.

Executor compatibility

Deferral interacts with the executor that runs your tasks, and each executor has its own worker lifecycle. Confirm how the executor you choose releases and resumes a task after deferral, and whether combining several executors in one deployment changes that behavior. Run a representative DAG on the same executor and configuration you will use in production before you size the cluster.

Features that depend on the triggerer

Some Airflow features rely on the triggerer, so a triggerer outage can affect more than deferred operators. Deadline Alerts is one example that needs confirmation for your version. List every feature your pipelines use, check its documentation for triggerer requirements, and decide how alerting behaves if the triggerer stops.

Custom operators that poll in code

Custom operators that loop with sleep calls inside execute() still hold a worker slot and will not benefit from deferral. Review internal operators and hooks that poll external APIs, and decide whether to rewrite them around a trigger class. Budget engineering and test time for each rewrite, and confirm that the trigger code follows the same retry and idempotency rules as the operator it replaces.

How DeployOpen scopes and delivers triggerer work

DeployOpen plans, implements, documents and supports self-hosted open-source platforms inside your own infrastructure. For Airflow deferrable operators and the triggerer, we start with a written scope that names which waiting tasks move to deferrable operators and where the triggerer runs.

How we scope

Inventory sensors and long polls in your DAGs and record which ones hold worker slots during waits.
Agree in writing which operators move to deferrable versions and which stay as they are.
Define triggerer placement, instance count and failover expectations for your network and database setup.

How we build

Convert selected sensors and polls to deferrable operators and test resume behavior after each trigger fires.
Run triggerer instances on your infrastructure and verify that a surviving instance takes over trigger work after a failure.
Check worker slot usage before and after the change, using metrics from your own Airflow environment.

How we hand over

Deliver the DAG changes, triggerer configuration, deployment manifests and a test record for each converted task.
Document the runbook for restarting the triggerer, scaling it and reading its logs when a trigger stalls.
Walk your platform and data engineering teams through the design decisions, trade-offs and open items before handover.

How we support

Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
Ongoing support, Airflow upgrades and continuous testing are available as a retainer.
Confirm with your team which Airflow version and metadata database you run, since the triggerer setup depends on them.