Why removed context variables block an Airflow 3 upgrade
If your team runs Airflow 2 and your DAGs read execution_date, prev_ds or related keys from the task context, those DAGs will not run cleanly on Airflow 3 until you replace them. Airflow 3 removes the context keys execution_date, prev_execution_date, prev_ds, next_ds, tomorrow_ds and yesterday_ds, and the upgrade guide warns that unreplaced uses cause DAG errors. The supported path is to move to Airflow 2.7 or later first, ideally the latest 2.x line, run the Ruff AIR checks against your DAG code, fix what they flag, and only then upgrade the core. Treat the context-key replacement as a code change that is tested on 2.x, not as a surprise discovered during the cutover.
The timing matters. Airflow 3 first shipped on April 22, 2025, and its current patch release is 3.3.2. Airflow 2 entered limited maintenance on October 22, 2025 and reached end of life on April 22, 2026, after which it receives no fixes. A team that starts the context-variable audit now is working on a line that still gets security and critical bug fixes only, so the work has to finish on a schedule your security team can accept. Confirm with your platform team which Airflow 2 patch you run in production and whether any managed service you use still offers it, since that determines how long you can stay on 2.x while you migrate.
Before you change anything, search every DAG repository, shared library and plugin for the six removed keys, including Jinja templates in operator arguments, because templated fields are rendered at runtime and a grep of Python files alone can miss them. Ask each DAG owner whether their logic depends on the old meaning of the date, since logical_date and data interval values do not always match the old keys one for one. Then run Ruff with the AIR rules on the repository, review every automatic rewrite before you accept it, and record the findings so the upgrade has evidence of what was changed and tested.
Which context keys change and what each one needs
This table lists each context key Airflow 3 removes, what happens to DAGs that still read it, and the action to take before upgrading. Read each row as a checklist item for your DAG repository and its templates. The main takeaway is that any removed key still referenced when Airflow 3 runs causes DAG errors, so find every use before the upgrade window.
| Context key | Impact in Airflow 3 | Action before upgrading |
|---|---|---|
| execution_date | Removed. Any task, template or callable that reads it causes DAG errors. | Replace with logical_date. Confirm the exact mapping in the upgrade guide. |
| prev_execution_date | Removed. Logic that compares the current run with the prior run breaks. | Rewrite the lookup using the previous run date context the guide names, then test the result. |
| prev_ds | Removed. Date-string templates for the prior run stop rendering. | Replace with the guide's equivalent for the previous run and compare the rendered value with 2.x output. |
| next_ds | Removed. Date-string templates for the following run stop rendering. | Replace with the guide's equivalent and confirm which logical date or data interval it derives from. |
| tomorrow_ds | Removed. Often used in SQL and file paths that point to the next day. | Derive the date in code from logical_date or the data interval, then check the output. |
| yesterday_ds | Removed. Often used in partition paths and queries for the prior day. | Derive the date from logical_date or the data interval and verify partition names in a test run. |
| Templated fields | Jinja templates in operator fields, params and SQL or template files are easy to miss. | Search Python code, external .sql and template files, and operator params for every removed key. |
| Ruff AIR rules | AIR301 and AIR302 flag breaking changes. Confirm which rules cover these keys. Requires Ruff 0.13.1 or later. | Run the AIR checks, review each finding, and apply --fix or --unsafe-fixes only after review. |
| Staging run | Static checks miss keys assembled dynamically at run time. | Run each DAG in a staging Airflow 3 environment and compare rendered values for a sample run. |
How to find, replace and verify context key usage
This sequence moves DAG code from removed Airflow 2 context keys to their Airflow 3 equivalents in an order that keeps each change reviewable. It ends with DAGs running on a staging scheduler and a rollback point recorded before anything reaches production.
Inventory removed keys in DAG code
The DAG owners and the platform team search the whole DAG repository for execution_date, prev_ds, prev_execution_date, next_ds, tomorrow_ds and yesterday_ds. The search should cover Jinja templates in operator arguments, custom operators, callbacks, SQL and shell files, and any helper modules that build context dictionaries. Done means every hit is listed in a tracking file with its DAG, task and owner, including hits in files that are rarely changed.
Review catchup and cron timetable settings
Before any rewrite, the platform team checks the values of catchup_by_default and create_cron_data_intervals in the Airflow configuration. Airflow 3 defaults both to False, so bare cron strings use CronTriggerTimetable. If the current environment sets catchup_by_default to True, keep it as True until after the upgrade, because changing it afterwards skips one scheduled run. Done means the current values are recorded for each environment and any DAG that depends on the old timetable behavior is flagged for review.
Run Ruff AIR rules in report mode
The DAG owners run Ruff version 0.13.1 or later with the AIR301, AIR302, AIR311 and AIR312 rules enabled, without any fix flags. AIR301 and AIR302 flag breaking changes, while AIR311 and AIR312 flag recommended updates that do not break the DAG. Done means the report is saved as a baseline, and each finding is matched against the inventory from the first step so that no context key is handled only by the linter.
Apply safe fixes and review the diff
The DAG owners run Ruff with --fix only, which applies the automatic rewrites the rules consider safe, and then read every changed line in a pull request. Changes to templated fields, operator arguments and any logic that uses the data interval must be reviewed by the DAG owner, not accepted from the tool alone. Done means the diff has been approved, the Ruff report is clean for the rules in scope, and the review notes are attached to the change.
Rewrite remaining templated fields manually
Where a key remains, usually inside custom operators, templated strings, or dynamic task generation, the DAG owners replace it by hand. The usual mapping is execution_date to logical_date, while previous, next and relative date values should be rebuilt from the data interval start and end values, which the team must confirm for each schedule. Ruff's --unsafe-fixes option may propose rewrites for these cases, but it should be used only on a branch and only after each suggested change is read and tested. Done means a search for the removed keys returns no results outside documented exceptions.
Run DAGs on a staging scheduler
The platform team deploys the rewritten DAGs to a staging Airflow 3 environment with its own scheduler, DAG processor and API server. The DAG owners trigger representative runs, including a backfill for a short interval, and compare output tables, row counts and logs with the results from the current production run. Done means each DAG has a recorded successful run on staging and any differences in output have been explained.
Set a rollback point before promotion
Before the production upgrade, the team records the Git commit or tag that holds the Airflow 2 DAG code, takes a consistent backup of the metadata database, and writes down how to restore both the DAG code and the database. The owner of each DAG confirms that the rollback steps work in staging. Done means a named rollback point exists for every environment, and the upgrade is scheduled only after the rollback steps have been signed off.
Replacement patterns by situation
Direct references to execution_date inside Python callables are the most visible case. They appear in PythonOperator and @task functions when a callable reads context["execution_date"] or takes it as a keyword argument. Airflow 3 removes the key, so the DAG fails when the callable runs rather than at parse time, which can hide the problem until a scheduled run reaches that task.
- Search the whole repository: include helper modules, shared libraries and callables passed by string name, not only the DAG files.
- Check keyword arguments: a callable that declares
execution_dateas a parameter depends on the same removed key. - Confirm the replacement: the upgrade guide names the successor for each removed key, so confirm the mapping there before editing code.
- Review time arithmetic: code that adds or subtracts a timedelta from the old value may depend on which timestamp the replacement represents.
- Test with a manual run: trigger the task in a non-production environment to surface failures that static search misses.
Confirm each mapping against the Airflow 3 upgrade guide and your own tests, not against a copied snippet.
Templated fields are evaluated at runtime, so a removed key inside {{ }} will not show up in Python parsing. Templates using prev_ds, next_ds, yesterday_ds and tomorrow_ds are common in SQL strings, file paths, bash commands and API payloads. A missing key in a template typically fails the task instance, and the error may only appear in the task log.
- Scan every templated field: check
sql,bash_command,op_kwargs,paramsand any field your operators list in theirtemplate_fields. - Scan external files: SQL files and Jinja templates loaded from a template search path are easy to overlook.
- Watch date-shifted logic:
prev_dsandnext_dsoften encode a window boundary, so a replacement may change which dates a query selects. - Check macros and user-defined helpers: custom macros that call removed keys internally will fail even when the template looks clean.
- Confirm the template context: verify the available template variables for your Airflow 3 version in the upgrade guide before rewriting expressions.
Run a rendered-template test for each affected task so you can compare output before and after the change.
Custom operators and hooks are the most likely place for removed keys to hide. A team may have written an operator years ago, placed it in a shared package and forgotten that its template_fields or execute method reads a removed key. Ruff rules catch many direct references, but they cannot judge how a custom class consumes the context at runtime.
- Inventory shared packages: list every operator, sensor and hook your DAGs import, including those maintained by other teams.
- Read `template_fields` and `execute`: confirm which context keys each class reads and whether it passes them to another object.
- Check provider operators separately: upgrade provider packages to versions that match your Airflow 3 target, then review their documentation for context changes.
- Assign an owner: each shared package needs a named owner who signs off on the change and its tests.
- Version the package: pin the updated package so DAG teams can roll forward on a known release.
Confirm compatibility for each custom class in a staging environment running the target Airflow version.
Cron-scheduled DAGs need a separate check because the replacement context keys depend on how the schedule is defined. In Airflow 3, catchup_by_default and create_cron_data_intervals are both False, so bare cron strings use CronTriggerTimetable. Teams that rely on data interval timetables, and expect the logical date and interval boundaries to behave as before, must decide how to keep that behavior.
- Check the configuration: record the current values of
catchup_by_defaultandcreate_cron_data_intervalsin each environment. - Set the flag before upgrading if needed: the upgrade guide describes setting
create_cron_data_intervalstoTruebefore the upgrade to keepCronDataIntervalTimetable. - Avoid changing it after the upgrade: the upgrade guide warns that changing it afterwards skips one scheduled run.
- Review each DAG's schedule argument: identify bare cron strings, explicit timetables and dataset or asset schedules.
- Plan the first run: confirm with the upgrade guide which run occurs first after the change, and communicate it to data consumers.
Confirm the timetable behavior in the upgrade guide and in a staging scheduler before the production cutover.
What to prepare before changing any DAG
Context key replacements touch every DAG that reads a removed variable, so the preparation work decides how much of the codebase you can change safely. These cards cover the inventory, the rollback point, the test environment, the expected outputs and the people who sign off on each pipeline.
Inventory every DAG repository and plugin
List each repository, shared module, custom operator and plugin that defines or imports DAGs, including code that builds DAGs dynamically from config files. Search every location for execution_date, prev_ds, prev_execution_date, next_ds, tomorrow_ds and yesterday_ds, both in Python and in Jinja templates. A search of Python files alone misses templated fields, so include .sql, .j2 and YAML config. Record which file, DAG and task owns each hit.
Back up the metadata database first
Take a consistent backup of the Airflow metadata database before any DAG change or upgrade step, and confirm you can restore it into a scratch database. Context key edits do not alter the database, but a failed upgrade later in the same window depends on this point of recovery. Store the backup outside the cluster that runs the scheduler. The metadata database migration page covers the backup procedure, history trimming and rollback steps in detail.
Build a staging environment that matches production
Run a staging deployment with the same Airflow version, executor, worker image, connections and variables as production, so a rewritten DAG fails or passes for the same reasons it would in production. A scheduler-only test misses problems in workers, where the context dictionary is actually built. Confirm that staging uses the same DAG delivery method as production, so the replaced code is tested the way it will be deployed.
Capture test runs with known outputs
Pick representative DAG runs for each pipeline and record their outputs before the change: row counts, file names, partition paths and downstream table values. After replacing execution_date with logical_date or the data interval fields, rerun the same logical period in staging and compare results. Illustrative example: a daily load that wrote 2025-06-01 partitions should still write the same partition after the change. Any drift points to a wrong date boundary, not a code style issue.
Name an owner for each pipeline
Assign one accountable owner and one backup for every DAG, including who signs off on the replaced date logic and who receives alerts if a run fails after the upgrade. Owners know which downstream consumers depend on a pipeline's date boundaries, which is information the code does not show. Keep the list in version control next to the DAG inventory so it stays current. Confirm each owner has time reserved for the staging comparison before the production change window is set.
How Ruff AIR rules find removed keys in your DAG code
Ruff's AIR rules give you a fast first pass over DAG code before you touch the Airflow 3 upgrade. AIR301 and AIR302 flag breaking changes, which are constructs that will fail or behave differently on Airflow 3 if they stay unchanged. AIR311 and AIR312 flag recommended updates that are not breaking, so the code keeps running but should move to the new names. The upgrade guide recommends Ruff 0.13.1 or later for this work, so pin that version in your CI image and in the developer tooling your team uses, and confirm the rule codes that cover each removed key in the release you install. Run the check against every DAG folder, plugin directory and shared module, not only the main DAG repository, because removed keys often appear in helper code that a DAG imports.
A static check reads the source as written, so it has blind spots that matter for context variables. If a key name is assembled at runtime, for example an f-string or a variable that combines a prefix and a suffix, the linter sees no literal execution_date and reports nothing. The same applies to lookups such as context[key] where key comes from configuration, a database row or a dictionary, and to keys passed through **kwargs into helper functions. Templated fields add another gap. A value like {{ prev_ds }} stored in a YAML file, a SQL template, a Variable or a string built in another module is ordinary text to the linter unless it sits in Python code the tool parses. The --fix option only rewrites patterns the rule recognizes, so it will not repair these cases.
Treat a clean Ruff run as a floor, not as proof that the upgrade is safe. Search the repository, configuration and template files for each of the six removed key names, and review every dynamic context lookup by hand. Then run the DAGs in a staging environment on Airflow 3 and compare rendered task instances and the logical dates they produce with your current production runs, so a silently wrong date shows up before cutover. Confirm the exact behavior with the upgrade guide to Airflow 3 and your team's own test results, and record what each review and test covered as evidence for change management.
What makes context key migration take longer
This table lists the factors that stretch the work of replacing removed context keys, and the direction each one pushes the effort. Read it to judge where your own estate sits and which factors to scope before quoting or scheduling the upgrade. The main takeaway is that the count of DAGs matters less than shared code, dynamic templates and the parity checks you must run before cutover.
| Driver | Why it increases effort | How to keep it down |
|---|---|---|
| Number of DAGs and repositories | Each DAG file and repository needs its own search, Ruff run, review and test pass, so effort grows with the count of places removed keys appear. | Run Ruff AIR checks across all repositories in one pass and group fixes by owning team. |
| Jinja templates in shared libraries | A removed key inside a shared macro or helper breaks every DAG that imports it, and the call sites are hard to find with a simple text search. | Fix shared libraries first, then pin and retest the consuming DAGs against them. |
| Templated fields built from strings | Keys assembled at runtime, such as concatenated template fragments in parameters, may slip past static checks and only fail when a run executes. | Render templates in tests for representative runs and review any dynamic template construction by hand. |
| Custom operators and plugins | Operators that read context internally or pass keys to a callable need code changes, not just template edits, and they may be maintained by a separate team. | Inventory custom operators early and assign an owner for each before the upgrade window opens. |
| Timetable and catchup settings | Changing catchup or cron data interval behavior alters which runs are scheduled, so date logic must be rewritten alongside key replacement and tested against the new timetable. | Decide the timetable approach per DAG before editing code, and confirm the catchup setting the upgrade guide describes. |
| Pipelines that must match outputs before cutover | Each pipeline whose results must be reconciled against the old run adds comparison runs, data sign-off and rework when dates differ by one interval. | Limit parity testing to pipelines with regulatory or downstream financial impact and document the agreed tolerance. |
| Missing ownership and tests | Where DAGs have no tests or no named owner, each change needs manual discovery of intent and expected output, which slows review. | Add a small run-level test for each date-sensitive DAG before editing its context usage. |
Questions teams ask before changing context keys
Can our team replace removed context keys without an outside partner?
Many teams can do this internally if they already own their DAG repository, run tests against it and have someone who understands the scheduling logic. Start by searching the whole repository for the six removed keys: execution_date, prev_execution_date, prev_ds, next_ds, tomorrow_ds and yesterday_ds. Then check templated fields, Jinja strings, macros, custom operators and shared helper modules, because these often hide uses outside the DAG files. Internal teams tend to struggle when many DAGs share code, when no staging environment exists or when nobody has time to review run-date changes. If you bring in a partner, ask for a written scope that names the repositories, the number of DAGs reviewed and the test evidence you will receive.
What drives the cost and effort of replacing execution_date?
The biggest driver is how many DAGs and shared modules depend on the removed keys, followed by how complex their scheduling is. Work typically falls into five parts: inventory, rewriting to logical_date or data interval equivalents, reviewing templated fields and custom code, testing against realistic run history and documenting the result. The rewrite itself is usually the smallest part, while review and testing take the most time. Most of the cost is one-time, but any new DAG that copies old patterns will need the same work later, so recurring cost depends on your code review discipline. When requesting a quote, ask who performs each step, which DAGs are in scope, what test runs are included and whether reruns after later changes are covered. No duration figure is supplied for this work, so confirm estimates against your own DAG count.
What do our data engineers and platform team need to provide?
Data engineers need to explain the business meaning of each run date, because logical_date and data interval values are not interchangeable with the old keys in every case. They should review each rewritten DAG, confirm which downstream tables or files depend on run dates and approve the test results. The platform team needs to provide a staging Airflow environment where the rewritten DAGs can run, access to representative run history, the CI pipeline that will execute the Ruff checks and a change window for the production upgrade. Both teams should agree on an owner for the final sign-off. Without these inputs, the rewrite can pass syntax checks while still producing different data windows. Ask your partner, or your internal team, to list these responsibilities in the written scope before work starts.
Are Ruff fixes safe to apply automatically to our DAGs?
Ruff can find most direct uses of the removed keys, but its fixes are not a full substitute for review. The upgrade guide recommends Ruff 0.13.1 or later with the AIR rules. AIR301 and AIR302 flag breaking changes, while AIR311 and AIR312 flag recommended updates that do not break DAGs on their own. Running Ruff with --fix applies safe automatic rewrites, and --unsafe-fixes applies rewrites that may change behavior, so treat that second option as a proposal to review line by line. Ruff will not reliably detect keys assembled at runtime, values passed through configuration files or logic in code outside the checked paths. Commit the changes in a branch, review the diff, and run your DAG tests in staging before merging. Read the Airflow 3 upgrade guide for the full rule list.
How does the catchup setting change scheduled runs after the upgrade?
In Airflow 3, catchup_by_default and create_cron_data_intervals both default to False, so bare cron strings use CronTriggerTimetable. If you set catchup_by_default to True before upgrading, the schedules keep CronDataIntervalTimetable, which matches how Airflow 2 handled cron schedules. Changing the setting after the upgrade skips one scheduled run, which can leave a gap in your data. Decide the value before the upgrade, record it in configuration, and confirm it in staging by comparing the next scheduled run times for each cron DAG. Tell downstream consumers that a run may be skipped if you change the setting later. Because the effect touches production data windows, include this check in the written test plan rather than leaving it to the final cutover.
Should we upgrade to Airflow 2.11 before moving to Airflow 3?
Usually yes, because the upgrade guide requires Airflow 2.7 or later before upgrading and recommends moving to the latest 2.x release first. Airflow 2.11 is the last 2.x line in the supplied version table, with the current patch at 2.11.2. Running Ruff AIR checks and installing the standard provider on 2.11 can reduce surprises during the move to Airflow 3. However, Airflow 2 entered limited maintenance on October 22, 2025 and reaches end of life on April 22, 2026, after which it receives no fixes. Use 2.11 as a stepping stone, not a long-term home. Check the current date against that schedule, confirm whether your managed service still offers 2.11, and set a target date for the Airflow 3 cutover in your change plan.
How DeployOpen scopes and delivers the Airflow 3 DAG migration
DeployOpen treats removed context variables as a code inventory problem first, then a controlled rewrite across your DAG repository, tested against your own schedules. Work is planned around the Airflow 3 upgrade guide and the Ruff AIR checks, and every change is documented so your team can run the migration after handover.
How we scope
- Inventory of DAG files and shared libraries that reference execution_date, prev_ds, next_ds, tomorrow_ds, yesterday_ds or prev_execution_date.
- Review of templated fields, Jinja strings, macros and custom operators that read removed context keys indirectly.
- Written scope listing repositories covered, DAG counts, schedules to verify and the environments used for testing. Engagements are fixed-scope.
How we build
- Replace removed keys with logical_date and data interval equivalents, aligning each rewrite with the schedule semantics of the original DAG.
- Run Ruff with the AIR301 and AIR302 rules to flag breaking changes, and review AIR311 and AIR312 recommendations before applying any automatic fix.
- Check catchup_by_default and create_cron_data_intervals settings so cron schedules keep the same run behavior after the move to Airflow 3.
How we hand over
- Before and after comparison of DAG run dates in staging, confirming that rewritten DAGs produce the intended data intervals and partitions.
- Written record of each replaced context key, the chosen replacement, the Ruff rule that flagged it and any manual exceptions reviewed.
- Ruff check added to your CI pipeline so new DAG code that uses removed context keys fails review before it reaches production.
How we support
- Support for follow-up fixes found after the first production run, scoped in writing like the original work.
- Ongoing upgrades, testing and documentation are available as a retainer, including later Airflow 3 patch and minor releases.
- Clear list of what your team should confirm in its own environment, such as schedules, data owners and downstream consumers of run dates.
