When Apache Airflow asset-aware scheduling replaces time-based schedules
Time-based schedules fail downstream teams in two common ways. A reporting DAG set to start at 06:00 may begin before the upstream load has finished, so it reads stale or partial data. The same DAG may also run every night even when nothing upstream changed, which spends compute and clutters run history with empty results. Asset-aware scheduling addresses both problems by letting a consumer DAG run when the data it depends on has been updated. Before changing anything, confirm when your upstream loads usually finish and how often their output actually changes. Those two facts show whether the change will remove real waste or only move the timing problem.
In Airflow 3, an *asset* is a named reference to a piece of data, identified by a URI and declared in DAG code. A producer DAG lists the asset in the outlets of the task that writes the data, so a successful run of that task records an update. A consumer DAG lists the asset in its schedule, and it runs once the asset is updated. Several assets can be combined with AND conditions (run when every input has updated) or OR conditions (run when any input has updated). Asset aliases let a DAG refer to a set of assets by a stable name, which helps when the exact outputs change over time. Your team should agree on a URI naming convention and assign an owner to each producer before the first DAG is converted.
Adopt asset-driven scheduling when a consumer's correctness depends on an upstream finishing, when several sources feed one output, or when unchanged data triggers costly reruns. Keep cron-style schedules for work that is tied to a clock, such as a report promised at a fixed hour or a process driven by an external calendar. Also keep them where producers cannot signal completion reliably. The cost of the change is mostly engineering and review time, because Airflow is licensed under Apache-2.0 and the project offers no paid edition. Ask your partner or internal team to map each DAG to one of these two patterns and to document what a consumer should do when a producer fails or is late.
Defining assets by URI, alias and condition
An asset is identified by its URI, and Airflow matches producers and consumers on that identifier. Choosing the URI is the first design decision because it becomes a contract between teams. A consumer that points at a slightly different string never runs, even when the underlying data is the same. Build URIs around the stable dataset, such as a warehouse table path, an object storage prefix or a topic name, so the identifier survives changes to the DAG that produces it.
- Stable naming: Base the URI on the dataset's location so that renaming or splitting the producing DAG does not orphan its consumers.
- Named owner: Record which team publishes each URI so that consumers know whom to contact before anyone changes it.
- Shared formatting rules: Agree on casing, separators and trailing characters in a written convention, since two strings that look alike are still two different assets.
- Scheme consistency: Use the same scheme for the same system in every DAG, so producers and consumers describe one dataset in one way.
- Catalog entry: Keep the list of URIs in your data catalog or repository so new DAGs reuse existing identifiers instead of inventing near duplicates.
Before a consumer depends on a URI, confirm in the Airflow UI that the asset appears once with the expected producer and consumer attached.
Some producers create outputs whose names are unknown when the DAG is parsed, such as one file per customer or one partition per source system. An asset alias gives those outputs a stable name that consumers can subscribe to. The producing task declares its outlets through the alias and attaches the concrete asset once the run knows the value. The design choice is between subscribing to the alias, which reacts to any member of a family, and subscribing to a concrete URI, which reacts only to one specific dataset.
- Stable subscription point: Consumers depend on the alias name, so producers can add new concrete outputs without editing downstream schedules.
- Run-time attachment: The concrete asset is attached inside the run, so the set of triggering assets can differ from one run to the next.
- Broad triggering: An alias fires its consumers for any attached member, which can produce more runs than the team intended, so check the trigger behavior with a test.
- Alias naming: Treat alias names as contracts, since a renamed alias silently stops matching the consumers that depend on it.
- Traceability: Record which concrete assets each alias produced in each run, so that auditors can trace a downstream run back to its inputs.
Test the alias with a run that attaches no asset, one asset and several assets, and confirm how the consumer behaves in each case.
A consumer with several asset dependencies needs a rule for when it runs. An OR condition starts the consumer when any listed asset updates. An AND condition waits until every listed asset has been updated before the consumer starts. The choice depends on whether the consumer's output is valid after a single update. Independent sources usually suit OR, while sources that must be consistent with one another usually suit AND.
- OR for independent inputs: Each update triggers a run, so a frequently refreshed upstream can drive frequent consumer runs and their load.
- AND for joined inputs: The consumer waits for all sources, which prevents a join between a fresh table and a stale one.
- Stalled partner risk: Under AND, a source that stops updating blocks the consumer, so monitor each input and decide how long a gap is acceptable.
- Readable nesting: Combining AND and OR in one expression is possible, but keep it simple enough that a reviewer can follow it during code review.
- Reset behavior: Confirm how AND state resets after a run, since this determines whether updates that arrive during a run count toward the next one.
Write one test per condition: a single input updated, all inputs updated, and one input updated twice, then compare the run counts with what the design expects.
Most unmatched assets trace back to strings that differ by a single character, and the scheduler will not warn that a consumer is waiting on an identifier nobody publishes. These mistakes often appear during promotion from development to production, during copy-and-paste DAG development and during refactoring. A quick review of the URI list before merging catches most of them.
- Environment segments: A URI that includes a development or production segment in one DAG but not in another will never match across environments.
- Case and separators: Upper and lower case, underscores and hyphens each produce a distinct identifier.
- Invisible characters: A trailing slash or stray whitespace is enough to break a match, and it is easy to miss in a code review.
- Renamed URIs: Renaming a producer's DAG or task should leave the asset unchanged, but changing the URI itself orphans every consumer that subscribed to the old one.
- Copied DAGs: Duplicating a DAG and editing only part of its code can leave the copy publishing to the original dataset, which corrupts the original's lineage.
Before merging, search the repository for each URI string, compare the producer and consumer spellings and confirm that both sides appear in the asset view of your Airflow version.
Which asset scheduling options fit which pipeline shape
This table compares the asset scheduling options available for Airflow DAGs, from URI-defined assets to aliases, AND and OR conditions and time-based schedules kept alongside them. Read each row against the shape of your pipelines, then check the confirmation column before a design is approved. The main takeaway is that the cheapest option to build is not always the one that stays easy to operate as dependencies multiply.
| Option | When it fits | Design effort | What to confirm before adoption |
|---|---|---|---|
| URI-defined assets | One producer DAG feeds one or a few consumer DAGs and each dataset has a stable, unique identifier | Low. Agree on a URI naming convention and assign an owner per asset | Confirm the URI scheme is unique across teams and that producers only update an asset after data is committed |
| Asset aliases | Consumers need to reference a group of assets whose members change over time, or the producer cannot be named in advance | Moderate. Aliases add an indirection layer that teams must document | Confirm which DAGs own each alias and how alias changes are reviewed before a release |
| AND conditions | A consumer must wait until several upstream assets have all updated, such as a daily join across independent feeds | Moderate. Each input's freshness and timing must be understood, including what happens when one feed is late | Confirm the behavior when one input updates often and another rarely, since the consumer waits for the slowest input |
| OR conditions | A consumer should run when any of several equivalent sources updates, such as redundant feeds or regional copies | Moderate. Duplicate triggers must be handled so the consumer does not run twice for the same business data | Confirm the consumer is idempotent and how it identifies which source triggered the run |
| Nested AND and OR conditions | Complex dependencies combine required and alternative inputs in one schedule | High. Logic is harder to read, test and explain to auditors | Confirm a diagram or test exists for each condition tree and that operators can see why a run started |
| Time-based schedules kept alongside assets | Reporting deadlines, regulatory cutoffs or business calendars must run at fixed times regardless of data arrival | Low to moderate. Two trigger types must be reconciled so late data and fixed deadlines do not conflict | Confirm the business owner accepts a run that uses partial or stale inputs when the clock fires first |
| Asset-driven DAGs replacing cron chains | Multi-stage pipelines currently chained by fixed start times that drift when upstream jobs run late | Moderate to high. Existing timing assumptions must be mapped before the change | Confirm the cutover plan and whether run timing changes for downstream users, per the upgrade guidance for schedules |
How to set up producer and consumer DAGs step by step
This sequence takes a team from an Airflow 3 environment to a producer DAG that publishes an asset and a consumer DAG that runs when that asset updates. Each step names who does the work and the check that confirms it is finished. Most early problems surface at the asset URI and the first test run.
Confirm the Airflow 3 version and metadata database
Platform engineering confirms the target is Airflow 3, using the current patch release 3.3.2 as the reference point, and checks that Python, Kubernetes (if used) and the metadata database fall within the listed prerequisites. The supported backends are PostgreSQL 14 to 18 and MySQL 8.0, 8.4 or Innovation releases. SQLite should not be used in production, and MariaDB is not supported as the metadata database. Confirm the exact support matrix on the prerequisites page before committing. Done when a test environment connects to the chosen database and starts without errors.
Confirm the scheduler and DAG processor layout
Platform engineering checks that the scheduler and the DAG processor run as separate components, which Airflow 3 requires, and that the API server is running alongside them. Asset-driven runs depend on the scheduler seeing asset events written by producer tasks, so all three components must be healthy in the test environment. Run more than one scheduler only after the team has tested that setup. Done when all components report healthy and a scheduler restart in the test environment does not lose run state.
Define the producer DAG and its asset
The data engineering team writes the producer DAG and declares the asset it updates by URI on the task that produces the data. Choose a URI that names the dataset in a stable, descriptive way, because every consumer will reference the same string. Record the owner of the URI in your data catalog or runbook. Done when the producer DAG parses, the asset appears under the expected name in the Airflow UI, and code review confirms no other DAG declares the same URI with a different meaning.
Define the consumer DAG and its condition
Write the consumer DAG so its schedule is driven by the asset. For a single asset, the consumer runs on each update. For several assets, an AND condition waits until every listed asset has updated, and an OR condition runs when any one of them updates. Test the condition against business expectations, because an AND condition will keep waiting if one producer stops publishing. Done when the consumer DAG parses and the UI shows the asset condition on its schedule.
Deliver both DAGs through the chosen method
Platform engineering deploys both DAG files through the delivery method selected for the environment, such as Git-based bundles, a shared volume or an image build. Both DAGs must reach the same DAG processor, because a consumer cannot be scheduled on an asset that its environment has not parsed. Confirm that both DAGs carry the same tested version. Done when the DAG list shows both DAGs, no import errors appear, and the commit or image tag matches the change record.
Trigger a test run of the producer
Operators trigger the producer DAG manually in the test environment, using sample input so the run is safe to repeat. Watch the task that declares the asset outlet complete successfully, then check the asset view for a new update event. The consumer should create a run from that event. If it does not, compare the URI in the consumer with the producer's URI character for character. Done when the event is recorded for the asset and the consumer run appears in its history.
Check scheduler behavior and run history
Review the scheduler logs and the run history for both DAGs to confirm the consumer starts once per qualifying event and not repeatedly. For an AND condition, update only one asset and confirm the consumer waits for the other. Record which events produced which consumer runs, since auditors and on-call engineers will rely on that trail. Done when the logs show no scheduler errors, the consumer run count matches the expected events, and the results are written up for the change record.
How partitioning added in Airflow 3.3 changes asset design
The partitioning added in Airflow 3.3 lets producers and consumers of an asset work with smaller slices of it, such as one business date's load or one region's extract, so a downstream DAG can be triggered by the slice that finished. Treat this as a change to asset identity. Once partitions exist, the partition becomes part of what a producer reports and what a consumer waits for, so an asset defined only by its URI may no longer describe the data your consumers need. For example, in an illustrative nightly sales load that writes one partition per business date, a reporting DAG could run as soon as the day it depends on is complete, without waiting for the full history to be rebuilt. The current patch release is 3.3.2, so plan against that line.
Before you rely on the feature, confirm the details in the Airflow 3.3 release notes and the code history in the apache/airflow repository on GitHub. Check the exact way producers declare partitions and consumers select them, how partition keys are formatted, and whether partitioned assets combine correctly with AND and OR conditions and asset aliases. Also confirm how the scheduler handles a partition that never arrives, how manual runs and backfills interact with partition keys, and which known issues are fixed in the patch release you intend to run. If your team is still on Airflow 2, adopting partitions is a separate step: the upgrade guide requires Airflow 2.7 or later before moving to Airflow 3, so sequence the upgrade first.
Partition behavior has to be validated in your own DAGs, not inferred from the documentation alone. Build a staging pair with a producer that emits at least two partitions and a consumer that must run once per partition. Then test late partitions, missing partitions, duplicate emissions, out-of-order arrival, task retries and a manual run, and record which partition triggered each consumer run in the UI and logs. Decide early whether existing URI-only assets stay unpartitioned, which teams own partition naming in data contracts, and how consumers migrate in stages. Those decisions shape the design more than the feature syntax does, and they are where most rework tends to appear.
Security and operational risks of asset-driven scheduling
Asset-driven scheduling moves the trigger from a clock to a dependency graph, so the risks shift from timing to who can declare dependencies, how many links the scheduler evaluates, and what the metadata database must hold. The cards below cover the decisions to make before producer and consumer DAGs go into production.
DAG author trust boundary
In Airflow, any DAG author who can deploy code can declare an asset that another team's DAG consumes. A new producer can therefore start downstream pipelines that the consuming team never reviewed. Decide who may publish DAGs to a shared environment, and whether asset URIs need a naming convention or review so that one team cannot claim another team's asset. Check this against your DAG delivery process and the trust model in your security hardening plan.
Scheduler load from asset links
Each asset condition adds evaluation work to the scheduler, and AND and OR conditions across many producers and consumers multiply it. Airflow supports more than one scheduler at once using database row-level locks, but that does not remove the cost of evaluating a large graph. Ask which DAGs will consume each asset, how often producers update it, and whether the scheduler capacity you plan for includes these links.
Metadata database sizing
Asset events, run records and task instances all accumulate in the metadata database, which backs the scheduler, API server and asset state. The production guidance is clear that SQLite is for testing only and can lose data in production, and MariaDB is not a supported metadata database. Plan PostgreSQL or MySQL with capacity and retention set for the event volume you expect, and confirm the database cleanup procedure before go-live.
Access control on triggering runs
An asset update can start a downstream run without any person clicking a button, so the question shifts from who runs a DAG to who can produce the event that runs it. Airflow 3 uses the Simple auth manager by default and allows only one auth manager at a time. Confirm which roles may trigger, pause or backfill DAGs in your chosen auth manager, and whether producers can update assets from outside the DAG author group.
Auditability of asset events
When a consumer runs, the audit question is which asset update caused it and which DAG run produced that update. Asset events and run history must be retained long enough to answer that question for a review. Define which fields you need in logs and how they reach your log platform, and test that you can reconstruct a downstream run from its upstream events during a dry run before you rely on the trail for change control.
Upgrade risk from Airflow 2
Airflow 2 entered limited maintenance and reached end of life on April 22, 2026, so teams still on it are running without fixes. The Airflow 3 upgrade guide requires moving to 2.7 or later first and recommends checking DAG code with Ruff AIR301 and AIR302 rules. Asset-driven designs often depend on scheduling behavior that changes between versions, so test asset triggers and run timing on a staging copy before cutover.
What asset-aware scheduling costs and what it needs from your team
How much effort does migrating from cron to asset-aware scheduling take?
The effort depends mainly on how many schedules you have, how many of them depend on each other, and how much of your logic still relies on clock times. Asset-aware scheduling replaces a fixed time with a dependency on an asset, so each producer DAG must publish to a known asset URI and each consumer DAG must declare the asset it waits on. The main drivers are the number of cross-job dependencies, the number of sleeps and file-watch scripts you can retire, and the number of teams who must agree on asset names. One-time work includes design workshops, DAG rewrites and parallel-run testing. Recurring work includes asset naming governance and each Airflow upgrade. Ask any quote to list every schedule, its upstream and downstream jobs, and the asset URIs it will publish or consume.
Can our internal team migrate to asset-aware scheduling without a partner?
Yes, a team with the right experience can do this internally, and the work is more design than code. The Airflow project is released under the Apache-2.0 license and has no paid edition, so the software itself carries no license fee. The cost is people, infrastructure and time. Answer three questions honestly before you decide: has anyone run Airflow 3 in production, has the team built producer and consumer DAGs before, and is there capacity to test timing changes against live data. A partner is useful when those skills are missing or when the migration must finish by a fixed date. DeployOpen plans and implements in your own infrastructure, and most engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
What does asset-aware scheduling need from our engineers and platform team?
Your platform team must provide a supported runtime and your data engineers must own the DAG logic and asset names. On the platform side, the Airflow 3.3.2 prerequisites list Python 3.10 to 3.14, PostgreSQL 14 to 18 or MySQL 8.0, 8.4 and Innovation releases, and Kubernetes 1.30 to 1.35 if you run on Kubernetes. MariaDB is not supported as a metadata database, SQLite should not be used in production, and at least 4 GB of RAM is recommended. Airflow 3 runs the API server and DAG processor as separate components, and it supports more than one scheduler at once, so your platform team must plan for each one. On the engineering side, expect owners for each asset, review of how producers and consumers signal changes, and a test environment that mirrors production.
How do upgrades from Airflow 2 affect run timing?
An upgrade can change when runs start, so verify timing before you cut over. Airflow 2 entered limited maintenance on October 22, 2025 and reaches end of life on April 22, 2026, after which it receives no fixes. Airflow 3 requires that you already run Airflow 2.7 or later, and the upgrade guide recommends moving to the latest 2.x release first. Timing risk comes from how schedules are interpreted, how logical dates are set for manual runs, and whether catchup behaves as your team expects. Plan a parallel run where Airflow 2 and Airflow 3 process the same inputs, compare run start times and outputs, and keep the old schedules paused until the comparison passes. Review your cron and timetable settings before the upgrade so that changes are known rather than discovered in production.
How do we check DAG code with Ruff AIR301 and AIR302 rules?
Run Ruff with the AIR301 and AIR302 rules over your DAG folder and treat every finding as backlog. The Airflow 3 upgrade guide recommends this check, and it is one of the fastest ways to see which code will break after the move. Some changes cannot be caught by a linter. Task code can no longer use Airflow database sessions or models directly and must use the REST API or the Task SDK. SubDAGs have been removed in favor of TaskGroups. Asset-aware DAGs also need their producer and consumer logic reviewed by hand. Add the Ruff check to CI so new code does not reintroduce removed patterns, then run each migrated DAG in a staging environment against realistic copies of its inputs. A clean lint result is a starting point, not evidence that the pipeline works.
Does running Apache Airflow itself carry a license cost?
Apache Airflow is an Apache Software Foundation project released under the Apache License 2.0, and the project offers no paid edition. Your costs therefore come from the infrastructure you run it on, the people who build and operate it, and any support you buy. Recurring costs include compute and storage for the scheduler, workers, API server, DAG processor and metadata database, plus upgrades, security patching and on-call coverage. One-time costs include design, DAG migration and testing. Costs rise with the number of DAGs, the number of concurrent tasks and the number of environments you keep. When you request a quote, ask for each cost component to be listed separately, including which items recur and which are one-time, so you can compare an internal build with a partner-supported build on the same basis.
What does a DeployOpen engagement cover for asset-aware scheduling?
A DeployOpen engagement covers the planning, implementation and documentation of the asset-aware design inside infrastructure you control. Typical work includes mapping existing schedules and their dependencies, designing asset URIs and ownership, writing or rewriting producer and consumer DAGs, running Ruff checks and staging tests, planning the parallel run, and documenting the result for your operators. Engagements are usually fixed-scope, so the scope, deliverables and timelines are agreed in writing before work starts, and a quote follows from that written scope. Ongoing upgrades, support and continuous testing are available as retainers. DeployOpen does not issue audit reports or certifications. If you need compliance evidence, DeployOpen can prepare systems and records, and an independent auditor makes the assessment.
How DeployOpen scopes and delivers asset-aware scheduling
Asset-aware scheduling is scoped as a fixed-scope engagement, with the scope, deliverables and timeline agreed in writing before work starts. For this topic, the written scope should list which pipelines move from time-based schedules to asset triggers, which producer and consumer DAGs are in scope, which asset URIs and aliases are created, and which dependent pipelines stay on cron. Ask for each of these to be named individually, because the number of assets and the number of consumers are what drive the design and test effort. DeployOpen does not publish a price list, so a quote follows from that written scope.
Asset design starts with the client's data owners, not with the DAG code. Before any Asset is defined, the team needs agreement on what each URI identifies, who owns the producing job, which downstream teams consume it, and whether a dataset is updated in place or by partition. Those answers decide whether a consumer should wait for an AND condition across several assets or an OR condition on any one of them, and whether an asset alias is needed to keep a consumer stable while the underlying URI changes. The partitioning options available in Airflow 3.3 should also be checked against how the producers actually publish data, and that check should be recorded in the design document. Confirm the exact asset and partitioning behavior for your Airflow version against the Airflow documentation before committing to a design.
Implementation runs inside your own infrastructure, on the Airflow deployment you already operate or plan to build, with the metadata database, scheduler, DAG delivery and secrets handling kept under your control. Delivery usually covers the producer and consumer DAG changes, tests that simulate asset events, a staging run that confirms triggers fire in the expected order, and a rollback plan for each consumer that moves off its time-based schedule. Support after handover is optional. A retainer can cover monitoring of asset event volume, review of new consumers, and upgrades as Airflow releases change asset behavior. Ask for the retainer terms in writing as a separate line item so the one-time build and the recurring support are priced and reviewed independently.
