Apache Airflow dynamic task mapping: expand and partial

Apache Airflow dynamic task mapping: when run-time task creation replaces generated DAGs

Apache Airflow dynamic task mapping lets a DAG create its tasks at run time, after an upstream task has produced the list of work items. Teams usually reach for it when the number of files, partitions, customers or API pages is unknown until a previous task finishes. Before this feature, the common workarounds were a Python loop that builds a fixed set of tasks at parse time, or a generator script that writes one DAG file per item. Both break down when the item count changes between runs, because the DAG shape is fixed when the file is parsed. According to the Apache Airflow dynamic task mapping documentation, expand() creates one task instance for each element of an input, and partial() sets the arguments that stay the same across those copies. Together they let a single operator definition fan out over a list that exists only once the run is underway.

The decision to adopt the feature is mostly about where the work list comes from and how many copies the scheduler must handle. If the list is produced by an upstream task and can vary from run to run, mapping is the natural fit. If the list is static and small, a plain loop may be easier to read and test. The feature does not change which executor runs the tasks, so the choice of executor and worker capacity still needs to be made separately. This section introduces the concepts only. The rest of the page covers the configuration you need to set, the default limit on a single expansion, how to cap parallel copies, and the pattern for mapped task groups, so that you can judge whether mapping fits your pipelines before you change any DAG code.

Before you commit, confirm with your own DAG inventory which pipelines have run-time item counts and how large those lists can grow. Check the Airflow version you plan to run, because the docs and behavior of mapping settings are tied to the release, and review the limits described later on this page against your largest expected input.

Which mapping pattern fits which pipeline shape

The simplest mapping keeps one task and passes a list to expand() on a single argument, such as a list of file paths or partition identifiers. Airflow creates one task instance per list item at run time, and the length of the list is only known after the upstream task returns. This fits ingestion steps where every item runs the same code with the same settings, and where a failed item can be retried without touching the others.

Single varying argument: Only the argument named in expand() changes between copies, so the task definition stays short and easy to review.
Length known at run time: The count of copies comes from upstream output, so the Airflow UI shows the expanded instances only after the list exists.
Shared run: All copies belong to one DAG run, so one run view shows progress, failures and logs for every item.
Per-item retries: A failed copy can be retried on its own, which suits long lists where a single failure should not rerun everything.
Limit applies globally: max_active_tis_per_dag caps how many copies run at once across all active runs, so a large list can still overload a shared database or API if left unlimited.

Before you adopt this pattern, confirm the list length your upstream step can produce and compare it with the max_map_length default of 1,024 in your own environment.

When most arguments stay the same and one changes, use partial() to set the fixed values and expand() for the varying argument. The fixed arguments are defined once, so every copy receives the same connection name, target schema or timeout. This fits a load step that writes many tables to one warehouse, or a notification step that sends one message template to many recipients.

Fixed values in one place: partial() holds the shared settings, which keeps copies consistent and reduces copy-paste errors.
One mapped argument: Only the argument passed to expand() varies, which makes the mapping easy to explain to auditors and reviewers.
Shared connection pressure: All copies usually hit the same system, so the concurrency limit should be set with that system's capacity in mind.
Argument types matter: Confirm in the Airflow documentation which arguments can be mapped and which must stay fixed for your operator version.
Change control: A change to a fixed argument alters every copy in the next run, so review partial() settings as carefully as the code.

Check which operator arguments accept mapping in your installed provider version before you design around them.

When each copy needs several arguments that vary together, build a list of dictionaries and pass it to expand_kwargs(). Each dictionary supplies the keyword arguments for one task instance, so a table name, source path and load mode can change as a set. This fits configuration-driven loads where each entry describes a complete job, such as one source system with its own schema, credentials reference and schedule window.

Coupled arguments: Values that must stay together, such as a path and its matching format, travel in one dictionary and cannot drift apart.
Config as data: The list often comes from a file or an upstream query, so new jobs can be added without new DAG code.
Validation before the run: Malformed dictionaries fail when the mapped task runs, so validate the list in an upstream step or in tests.
Secrets stay references: Pass connection IDs or secret names rather than values, so the dictionaries can be logged and reviewed safely.
Review ownership: Whoever edits the source of the list changes production behavior, so define who can approve those changes.

Confirm the exact expand_kwargs() input format for your Airflow 3 release in the task mapping documentation.

Decorating a function with @task_group and calling expand() or expand_kwargs() on it repeats an entire sequence of tasks for each input. The group can contain extract, transform and load steps that run together per item, with dependencies defined inside the group. This fits pipelines where each partition or source needs the full chain, not just one step. Mapping inside a mapped group is not permitted, so the nesting must be planned in advance.

Whole chains repeat: Each copy of the group runs its internal tasks in order, so one item's failure does not block the others.
Group-level dependencies: Upstream and downstream links attach to the group as a whole, which keeps the outer DAG readable.
Larger run graphs: Every internal task multiplies by the group's input count, so the UI and scheduler load grow quickly with long lists.
No nested mapping: Plan the structure so that a group does not contain another mapped step, since this is not supported.
Concurrency still applies: Limits such as max_active_tis_per_dag count the internal task instances, so size the limit for the whole chain.

Before choosing this pattern, have your team estimate how many task instances one run will create and check it against your scheduler and database capacity.

Configuration options that control mapped task behavior

This table compares the settings and authoring choices that shape how many mapped tasks exist and how many run at once. Read each row by asking what it limits, what the default is, and what you give up by changing it. The main takeaway is that the expansion cap and the concurrency cap solve different problems, so raising one does not change the other.

Option	What it controls	Default and when to change it	Trade-off of raising or lowering it
[core] max_map_length	Maximum number of mapped task instances created from a single expand() call	1,024 by default (Airflow task mapping docs). Change it only when a legitimate input list is larger and splitting the work is not practical.	Raising it lets one task fan out further but increases scheduler and metadata database load per run. Lowering it makes oversized inputs fail early.
max_active_tis_per_dag	How many mapped task instances of a DAG can run at once across all active runs	Not set by default in this table. Set it when a wide fan-out would overwhelm a shared database, API or downstream system.	Lowering it protects downstream systems but lengthens run duration. Raising it speeds runs but can starve other tasks and other DAG runs of slots.
partial() arguments	Fixed arguments shared by every mapped copy of a task	Used whenever some inputs stay constant across copies, such as a target environment or a connection ID.	Keeps the mapped list short and readable. Passing values that vary per item to partial() instead of expand() removes the per-copy variation you need.
expand() with a single iterable	Creates one mapped copy per item of one keyword argument	Use when each copy needs one varying value, such as a file name or partition date from an upstream result.	Simplest form and easiest to debug. Does not work when each copy needs several values that must stay paired.
expand_kwargs()	Creates one mapped copy per dictionary in a list, supplying several keyword arguments per copy	Use when each copy needs multiple paired values, such as a source path with its matching target table.	Keeps related values together, but the input must be built as a list of dictionaries upstream, which adds a transformation step to maintain.
Mapped task groups (expand on @task_group)	Repeats a whole group of tasks once per input, creating a mapped group	Use when a sequence of steps must run as a unit for each item, such as extract, validate and load for one source.	Keeps a per-item pipeline together. Task mapping nested inside a mapped task group is not permitted, so the group design must be flat at that level.

Mapped task groups and their nesting rule

Dynamic task mapping can also apply to a whole @task_group function, so one group of related tasks is generated once per upstream item. The design work lies in what each copy receives, how the copies connect to downstream steps, and where the nesting restriction forces a different structure.

Building a mapped task group

Decorate a function with @task_group and call expand() or expand_kwargs() on it, the same way you would map a single task. Each item from the upstream result creates a full copy of the group, with its inner tasks named and tracked per copy. Check the group's inner task count before mapping, because every copy multiplies the total tasks in the run and counts against the default max_map_length of 1,024 per expansion.

Passing fixed arguments with partial

Use partial() before expand() to set arguments that stay the same across every copy of the group, such as a target schema, a connection ID or a run-level config value. Only the arguments you expand vary between copies. Keep fixed values out of the mapped input so a reviewer can see which parameter drives the fan-out.

Dependencies across mapped groups

Wire a mapped group like any other task: an upstream task can feed expand(), and a downstream task can depend on the whole group. Confirm in the UI how the group collapses and expands so operators know which copy failed. Decide whether downstream steps need every copy to succeed or can proceed on partial results, since that choice changes how failures and reruns behave.

Limiting parallel copies

Set max_active_tis_per_dag on the DAG to cap how many mapped copies run at once across all active runs, which protects databases, APIs and worker pools from a large fan-out. Confirm the value with the team that owns the downstream system, and review it when several DAGs share the same external resource.

Nesting restriction for mapped groups

Task mapping nested inside a mapped task group is not permitted. A mapped group cannot contain another mapping step, so a two-level fan-out of groups inside groups will fail to define. Check any group that might later gain a dynamic step during design review, because the restriction shows up only when the DAG is parsed.

Designing around the nesting limit

When two levels of fan-out are needed, flatten the work: build one list of combined inputs upstream, for example the product of two lists, and use a single expand_kwargs() over it. Alternatively, split the work into separate DAGs or a grouped structure that triggers the inner work. Test the chosen layout against realistic input sizes before production.

How to set up a mapped DAG and verify it

This sequence takes a DAG from an upstream task that returns a list to a mapped downstream task that runs at a controlled parallelism, with a check at each step so you know the mapping behaves as expected before production use.

Confirm the Airflow version and upstream output
Confirm that the target environment runs Airflow 3 and that the upstream task can return a list, dictionary list or other collection that describes the work to fan out. Your team should decide what one list item represents, such as a file, a partition, a customer account or a date range, and confirm that the item can be serialized as a task argument. Check that the upstream task is idempotent, because a retry will produce the same list again. The step is done when you can name the item type and show a sample output from the upstream task in a development environment.
Write the upstream task that produces the list
Write the task that generates the list using the @task decorator, keeping its return value small and deterministic. Avoid returning very large payloads, since task results pass through Airflow metadata. If the list may be empty, decide in advance how the DAG behaves in that case. The check is a manual test run of the upstream task alone, confirming that the returned list has the expected length and item shape in the task logs and the XCom view.
Add expand and partial to the downstream task
Call the downstream task with expand() on the upstream output, or expand_kwargs() when each mapped instance needs several different arguments. Pass arguments that stay the same for every instance through partial(), so that only the varying values are mapped. Confirm the parameter names match the function signature, because mismatches fail at parse time. The check is that the DAG parses without errors in the DAG list and that the graph view shows a mapped task with a placeholder for the expansion count.
Set the parallel limit for the downstream system
Set max_active_tis_per_dag on the mapped task to match the concurrency the downstream system can handle, such as a database connection limit or an API rate limit. This setting applies across all active runs of the DAG, so review how many runs can overlap before you set it. Confirm the value with the owners of the downstream system rather than setting it from the default. The check is a configuration review that records the chosen value next to the owner who approved it.
Check max map length against the largest input
Compare the largest list you expect with the [core] max_map_length setting, which caps a single expansion at 1,024 tasks by default. Confirm the value in your environment's configuration, since it can be changed. If the largest expected list is close to the limit, split the input into batches or use a mapped task group pattern rather than raising the cap without a reason. The check is a written comparison of the expected maximum against the configured limit, reviewed by the team that owns the DAG.
Test in staging with realistic input size
Run the DAG in a staging environment that matches production Airflow configuration, using an input list of a size close to the largest expected production run. Include failure cases, such as one mapped instance that raises an error, to confirm that retries and downstream dependencies behave as intended. Your team should also confirm that the metadata database and workers handle the number of task instances created. The check is a completed run where the mapped instance count in the grid and task instance views matches the length of the upstream list.
Confirm parallelism under production-like load
Trigger a run under load similar to production and observe how many mapped instances run at the same time in the UI and in your monitoring tools. Verify that the observed concurrency does not exceed the max_active_tis_per_dag value and that the downstream system stays within its agreed limits. Record the run duration and any queueing so the team can compare future runs. The step is done when the concurrency observed matches the configured limit and the owner of the downstream system confirms the load is acceptable.

Security and operational risks of run-time fan-out

Dynamic task mapping lets one DAG create many task copies from data it has not seen when the DAG was written. That flexibility moves risk from design time to run time, where volume, credentials, logs and retries are harder to predict. The cards below pair each risk with a control to decide on before the DAG reaches production.

Unbounded fan-out overloading systems

A mapped task creates one copy per item in its upstream list, so a larger than expected input can flood a source API, database or file share. Set [core] max_map_length deliberately, since it defaults to 1,024 tasks per expansion, and cap concurrency with max_active_tis_per_dag. Confirm the target system's rate limits and connection pool size against those values before scheduling the DAG.

Shared credentials across mapped copies

Every mapped copy of a task resolves the same connection IDs and variables, so a broad connection grants each copy the same reach. Give mapped tasks a dedicated connection with the narrowest read or write scope the job needs, and check who can edit that connection in the metadata database or secrets backend. Ask the team to document which connections each mapped task touches.

Log volume from many instances

Each mapped copy writes its own task instance log, so a fan-out of hundreds of copies multiplies log storage and makes search slower. Decide how long logs are kept, where they are shipped, and which fields identify a copy. Log only the input identifier and outcome, not full payloads, which may carry personal or regulated data. Confirm the remote logging setup and retention policy with your platform team.

Retries of individual copies

Retries apply to each mapped task instance, so one failing item can retry while the rest of the expansion finishes. This is useful for transient errors but can hide a systematic fault that repeats across hundreds of inputs. Set retry counts and delays per task, and alert when the retry rate for a mapped task passes a threshold your team agrees on. Make each copy idempotent so a retry does not duplicate writes.

Audit trail of inputs per copy

Auditors often ask which input each copy processed, when, and with what result. Mapped task instances are indexed by their map index, so the record links each index to an input only if the DAG writes that mapping explicitly. Have each copy record its input key, outcome and run ID in an audit table or log stream that your evidence process retains. Confirm the format with the auditors before the first run.

DAG authors as a trust boundary

Anyone who can write or change a DAG can decide what a mapped task calls, which connections it uses and how many copies it creates. Treat DAG code as privileged change, not as configuration: review it through the same change process as application code, and limit who can deploy to the DAG bundle. The security hardening guidance covers the trust model in more depth.

Questions teams ask before building dynamic task mapping

Can our team build dynamic task mapping without outside help?

Yes, if your engineers already write Airflow DAGs and can test them in a non-production environment. Dynamic task mapping is an authoring pattern inside DAG code, using expand() and partial(), so it does not need a separate platform. The harder questions are whether the team has time to design the upstream data shapes, set limits on parallel copies, and build tests for runs that create hundreds of tasks. An outside partner usually helps when several pipelines need the same pattern or when the team has no time to build a shared approach. Either way, the Apache-2.0 licensed project itself has no paid edition to buy.

What drives the effort for a dynamic task mapping project?

Effort depends on four things: how many pipelines need mapping, how much the upstream output varies, how much the downstream systems can absorb, and how thoroughly the work must be tested. Pipelines that loop over a stable list are simple. Pipelines where the number of items changes every run, or where one mapped task feeds a shared database, need more design. Each expansion is capped at 1,024 tasks by default through the [core] max_map_length setting, so teams with large inputs must decide whether to batch items or raise that limit. Ask any partner to list the pipelines in scope and to state how each one's input size is measured.

What does the work need from our team?

The work needs access to a running Airflow environment, a staging environment that matches production closely enough to test real behavior, and representative input data. Engineers must be available to explain upstream outputs, owners of downstream systems must say what load they can take, and someone must be able to approve changes to the DAG code. Without representative inputs, tests tend to check only the happy path and miss the empty-list and very large-list cases. Confirm these access needs in writing before the scope is agreed.

How long does a dynamic task mapping engagement take?

Duration depends on scope rather than on the feature itself. A single DAG that replaces one loop is a small change, while a set of pipelines that share limits, testing, and monitoring is a larger program. The written scope should list each pipeline, the upstream and downstream dependencies, the testing required, and the staging steps, and the timeline should follow from that list. DeployOpen does not publish fixed durations, so ask for a timeline tied to the agreed deliverables.

Do our Airflow 2 DAGs need changes before we use task mapping?

Usually yes, because the upgrade path comes first. Airflow 2 reached end of life on April 22, 2026, so it gets no fixes, and the Airflow 3 upgrade guide requires being on Airflow 2.7 or later before upgrading. The guide recommends moving to the latest 2.x release first and checking DAG code with the Ruff AIR301 and AIR302 rules. In Airflow 3, task code cannot use database sessions or models directly. Plan the upgrade and the mapping work together so mapped DAGs are not rewritten twice.

How is the cost structured in a fixed-scope quote?

A fixed-scope quote usually has a one-time cost for design, build, testing, and handover, based on the pipelines and deliverables listed in the written scope. Recurring costs are separate and appear only if you choose ongoing support, upgrades, or continuous testing as a retainer. Your own internal time, the Airflow infrastructure, and any downstream system changes are paid for by you. Ask for the quote to list each pipeline, each deliverable, what is excluded, and which costs recur.

What should we test before running mapped tasks in production?

Test the empty input, a single item, a typical input, and an input near the max_map_length limit. Confirm that max_active_tis_per_dag keeps parallel copies within what downstream databases and APIs can handle, and that mapped task groups do not nest task mapping inside them, which Airflow does not permit. Check that retries and partial failures behave as expected, and that logs make it clear which mapped item failed. Ask your partner or team to document these cases in the test plan.

What changes in pipeline design once mapping is in place

Once mapping is in place, the main design change is that one DAG file describes the work and the number of tasks is decided at run time from upstream output. Teams that previously kept a generated DAG file per table, customer or partition, or a Python loop that declared tasks at parse time, can replace that with a single DAG where expand() creates one task copy per item the upstream step returns. The DAG file becomes smaller and easier to review because the logic lives once, and a change to the processing step applies to every copy on the next run. Reviewers can read one definition instead of comparing dozens of near-identical files. The cost is that the graph shown in the UI no longer shows every copy in advance, so reviewers have to reason about the upstream function that produces the list.

What gets harder is debugging. A failure may affect one mapped copy out of many, and the question becomes which input caused it and whether the other copies succeeded. Before adopting mapping, decide how each copy will carry its input identity into logs, task instance names and alerts, and whether the upstream list is persisted so a rerun reproduces the same set. Also confirm the expansion ceiling set by [core] max_map_length (1,024 by default) against your largest realistic list, and make sure an oversized list fails loudly rather than silently truncating. Check how max_active_tis_per_dag interacts with your on-call expectations, since a limit that is too tight makes a healthy run look stalled.

For observability per copy, treat each mapped task as a unit you can filter, measure and retry. Ask the team that owns the platform to confirm which Airflow UI views, task log paths and metrics your version exposes for mapped task instances, because these details change between releases and should be verified in your environment rather than assumed. Keep the item identifier in the task output or in a structured log line, and agree on a naming convention for downstream reports. Test a mapped DAG with a small and an intentionally failing input before production, and document what a partial success means for the data consumers who depend on it.

How DeployOpen works on this

DeployOpen plans, implements, tests, documents and supports self-hosted Apache Airflow inside your own infrastructure. For dynamic task mapping, we start from the pipelines that need run-time fan-out, agree the expansion limits and parallelism controls in writing, and deliver the DAG code with its operating notes.

How we scope

Inventory the DAGs that currently generate tasks with loops or generated files, and record the upstream results that drive each expansion.
Agree in writing which pipelines move to expand() and partial(), and which stay as static tasks.
Set the expected maximum fan-out per expansion against the default max_map_length of 1,024, and document any change to it.

How we build

Write mapped and mapped task group DAGs using expand(), expand_kwargs() and partial(), with fixed arguments kept separate from mapped inputs.
Set max_active_tis_per_dag on each mapped task so parallel copies match the capacity of your executor and database.
Test mapped DAGs in a staging environment with empty, single and maximum-size upstream outputs before promotion.

How we hand over

Deliver the DAG code, configuration decisions and a short runbook covering expansion limits, failure handling and how to rerun mapped tasks.
Document the nesting rule that prevents task mapping inside a mapped task group, and the patterns used to avoid it.
Walk your engineers through the handover so they can extend mapped pipelines without breaking the agreed limits.

How we support

Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
Ongoing support, upgrades and continuous testing are available as a retainer after handover.
Ask any quote to itemize scoping, build, testing and handover work separately, so you can see which parts recur.

Apache Airflow dynamic task mapping