Apache Airflow asset-aware scheduling
Replacing time-based schedules with asset-driven triggers: defining assets by URI, producer and consumer DAGs, AND and OR conditions, asset aliases, and the partitioning added in Airflow 3.3.
Apache Airflow features
Pages on specific Airflow scheduling and execution features that change how pipelines are designed and operated, based on the Airflow core documentation and release notes.
Apache Airflow deferrable operators and triggerer
Cutting worker capacity spent on waiting tasks by moving sensors and long polls to deferrable operators, plus running and scaling the triggerer component with failover across instances.
Apache Airflow DAG bundles and DAG versioning
Deploying DAGs from Git with GitDagBundle so every run records its commit and keeps the same code mid-run, compared with local and object storage bundles that always run the latest code, and the rerun setting that decides which version a retry uses.
Apache Airflow Edge Executor for remote workers
Running Airflow tasks on workers in other data centers, clouds or restricted networks with the Edge provider, where workers reach the central site over HTTPS only, and combining EdgeExecutor with Celery or Kubernetes executors.
Apache Airflow human-in-the-loop approvals
Adding approval gates and manual input steps to pipelines with the human-in-the-loop operators introduced in Airflow 3.1, and where they fit for data releases, change control and review steps.
Apache Airflow dynamic task mapping
Creating tasks at run time from upstream results with expand and partial: mapped task groups, the 1,024 default max_map_length, limiting parallel copies with max_active_tis_per_dag, and patterns that replace generated DAGs or loops.
Apache Airflow Deadline Alerts and replacing SLAs
Alerting when a DAG run will finish late with Deadline Alerts, the Airflow 3 replacement for SLAs: reference types including average runtime, async versus sync callbacks, Slack notifiers, tiered warnings, and the triggerer and worker requirements.
Apache Airflow backfills in Airflow 3
Reprocessing history with scheduler-managed backfills: starting them from the UI, CLI or API, choosing reprocess behavior for missing, failed or all runs, backfill concurrency, running backwards and protecting downstream systems from load.
Apache Airflow event-driven scheduling with asset watchers
Starting DAGs from messages instead of clocks: AssetWatcher with BaseEventTrigger triggers for Kafka, SQS, Pub/Sub and Azure Service Bus, idempotent consumers, acknowledgement timeouts, shared polling in 3.3, and conditions that cause endless reruns.
Apache Airflow language Task SDKs for Java and Go
Writing Airflow task logic in Java or Go with the language Task SDKs added in 3.3: how they build on the Task Execution API, when they beat containerized tasks, and what stays in Python DAG definitions.
Apache Airflow cron schedules and timetables in Airflow 3
How Airflow 3 interprets cron schedules: CronTriggerTimetable versus CronDataIntervalTimetable, catchup off by default, logical dates on manual runs, start_date rules, and choosing the right timetable before an upgrade changes run timing.
Apache Airflow UI plugins with React and FastAPI
Extending the Airflow 3 UI and API: the React plugin system from 3.1, external views, FastAPI apps and middlewares, migrating Flask AppBuilder plugins, and keeping plugins compatible across upgrades.
