What moving from Luigi to Apache Airflow involves
The work is a port of task logic plus a new operating model. Your Luigi tasks have to be rewritten as Airflow DAGs, and the scheduling, run history and retry handling that Luigi's central scheduler and cron jobs provided today have to move to a platform that owns them. The effort depends on three things: how many pipelines you run, how heavily those pipelines depend on Luigi's output targets, and how much of the scheduling still sits outside Luigi in crontabs or wrapper scripts. Luigi is an Apache-2.0 Python module for batch pipelines, maintained by Spotify's data team, and it resolves dependencies between tasks and offers a web interface for its central scheduler. Airflow does those jobs too, but it expects the schedule, the retries and the history to live in Airflow itself, so each of those responsibilities needs an explicit owner after the move.
The target should be Airflow 3. According to the Airflow version support table, Airflow 2 reached end of life on April 22, 2026, after which it gets no fixes, so a port that lands on Airflow 2 starts with an unsupported platform. Airflow 3 was first released on April 22, 2025, and its current patch release is 3.3.2. Confirm the version you install against the version life cycle page before you plan the rollout, because patch numbers move. Also check which Python and database versions your Luigi environment already uses, since the Airflow 3.3.2 prerequisites set the supported ranges and a Luigi host that runs an older interpreter will need a new runtime. Decide early whether the team will build the Airflow platform itself or ask a partner to deploy and harden it, because that choice changes who carries the operating work after cutover.
Which Luigi concepts map cleanly to Airflow and which do not
This table pairs each Luigi concept with its closest Airflow 3 equivalent and notes where the move changes behavior. Read the rows marked for confirmation as questions to test on your own pipelines, not as settled equivalences, since Luigi's target-based logic does not carry over on its own.
| Luigi capability | Airflow 3 equivalent | Migration notes |
|---|---|---|
| Task class with run() | Python task (@task) or operator inside a DAG | Port each run() body into a task. Code that reads Airflow database sessions or models directly must move to the REST API or Task SDK in Airflow 3. |
| requires() dependencies | Task dependencies set with >> or inferred from data passing | Dependencies become explicit edges in the DAG. Confirm that every requires() edge is captured, including ones built dynamically at runtime. |
| output() targets | Assets, or an explicit completion check in the task | Target existence checks do not transfer automatically. Design the completion signal per task and confirm it matches what downstream tasks expect. |
| Target-based idempotence (skip if output exists) | No direct equivalent; design per task | Luigi skips work when a target exists. Airflow reruns tasks on retry or clear unless you build a check. Design and test idempotence for each task before cutover. |
| Parameters | DAG params and run configuration | Confirm how each Luigi parameter maps to a run-time value, and check that defaults and overrides behave the same in your scheduled runs. |
| Central scheduler and web interface | Scheduler, DAG processor and API server as separate components | Luigi's central scheduler tracks task state. Airflow splits this across components that you must deploy and monitor separately. |
| Scheduling left to cron | Timetables and schedule expressions in the DAG | Luigi usually runs under cron. Airflow owns the schedule, so retire the cron entry once the DAG is verified. Confirm the timetable matches the old run times. |
| External task waits | Sensors or deferrable waits | Confirm each wait's timeout and poke behavior, since a missed signal in Luigi may have blocked silently. |
| Central history and retries | Run history in the metadata database and task retries | Confirm retry counts and delays match your Luigi settings, and decide how long run history is kept. |
Where Luigi leaves scheduling to cron and what Airflow changes
Luigi resolves dependencies inside a run, but it does not decide when a run happens. In many Luigi estates, a crontab entry on a server or a scheduled job in a configuration tool calls the pipeline entry point at fixed times. Each cron line is a separate decision: which hour, which arguments, which host. Retries across runs, the record of what ran yesterday, and alerts when a night's run failed usually live in cron output, log files or the central scheduler's web interface, which shows task state but not the schedule itself. Before you migrate, inventory every cron line and wrapper script that starts Luigi, including the arguments and date parameters they pass. Those lines are the real schedule of the pipeline, and they must be reproduced in Airflow, not assumed from the Luigi code.
In Airflow, the schedule moves into the DAG definition, so the timing, the catchup behavior and the retries are versioned with the code. Airflow 3 changed the timetable defaults: catchup_by_default and create_cron_data_intervals are both False, so a bare cron string uses CronTriggerTimetable. If your Luigi-era schedules depend on data intervals, set create_cron_data_intervals to True in the configuration before you upgrade, which keeps CronDataIntervalTimetable. Changing the setting after the upgrade skips one scheduled run, so decide this with the data owners before cutover and record the decision. Official guidance is in the Upgrading to Airflow 3 guide.
The metadata database carries this history, so its choice is an architecture decision, not a default. Airflow supports running more than one scheduler at the same time for performance and resiliency, coordinated by database row-level locks, and that design depends on a production database. The production guidance recommends PostgreSQL or MySQL, and SQLite is for testing only because it can lead to data loss. Confirm the database platform, backup approach and scheduler count with your platform team before the first DAG goes live, because changing them later means rebuilding the run history you are trying to keep.
Which documented paths help port Luigi tasks
No established importer converts Luigi task definitions into Airflow DAGs, so the port is manual work supported by a small set of documented resources. Each card below covers one resource, what the team still writes by hand, and what to confirm against its own pipelines before committing to a conversion approach.
Luigi source repository
The spotify/luigi repository is where the team reads the actual task logic, requires() chains, output() targets and parameter handling that the port must reproduce. Most of the work is mapping each target to an Airflow dependency or asset and each parameter to a DAG run configuration. Nothing in the repository translates these for you, so every task's behavior has to be confirmed by reading the code and running it.
Airflow 3 upgrade guide
The upgrade guide lists the breaking changes that affect any DAG written for Airflow 2 and any new DAG written against Airflow 3. It also covers timetable behavior, including how catchup_by_default and create_cron_data_intervals change scheduled runs. The team still decides which Luigi schedules become cron-based timetables and confirms the first run after cutover against its own calendars.
Ruff AIR301 and AIR302 rules
The upgrade guide recommends checking DAG code with the Ruff AIR301 and AIR302 rules, which flag constructs that changed or were removed in Airflow 3. These rules catch import and API issues in code the team writes during the port, but they do not know Luigi semantics. The team still reviews each flagged change and writes the tests that confirm each task produces the same outputs.
Task SDK and REST API
In Airflow 3, task code cannot use database sessions or models directly and must go through the Task SDK or the REST API. Luigi tasks that read or write shared state in the same process need to be rewritten to use these interfaces. The team must design this boundary per task, including where credentials come from, and confirm that each ported task still behaves correctly under the new isolation model.
Airflow executor documentation
The executor documentation explains the options for running tasks, including LocalExecutor, CeleryExecutor and KubernetesExecutor, and that several executors can run in one deployment since Airflow 2.10.0. Luigi workers that ran on fixed machines often map to a specific executor choice. The team still decides where each ported task runs, sizes the workers and confirms that tasks needing local files or specific hosts are placed correctly.
Official Helm chart
The community Helm chart uses the official Docker image and is the documented route for Kubernetes installs. It gives the platform team a tested starting point for the scheduler, API server, DAG processor and workers. The team still writes its own values, supplies its metadata database, sets secrets handling and decides how DAG files are delivered before any ported task runs in that cluster.
How to port Luigi pipelines in batches with a parallel run
This sequence moves Luigi task pipelines to self-hosted Airflow one pipeline at a time, with each output checked against the Luigi result before the old path is retired.
Inventory every Luigi task and consumer
The data engineering lead and the owners of each downstream consumer list every Luigi task, the targets each task writes (files, database tables, or other outputs), the requires chains between tasks, and the cron entries or shell wrappers that start the central scheduler. Record who reads each target and how often. The inventory is done when every target has a named producer, a named consumer, and a known trigger, and the team can draw the dependency graph without gaps.
Stand up a supported Airflow 3 environment
The platform team installs Airflow 3 in infrastructure you control, using a Python version and PostgreSQL major release listed in the prerequisites, and chooses an executor that fits your workload. If an Airflow 2 install already exists, it is first moved to the latest 2.x release, because the upgrade guide requires being on 2.7 or later before upgrading. Confirm the metadata database is PostgreSQL or MySQL rather than SQLite, that the DAG processor and API server run as separate components, and that a test DAG runs end to end before any Luigi work is ported.
Port one low-risk pipeline and compare outputs
Choose a pipeline with few consumers and idempotent outputs. The data engineer rewrites its tasks as Airflow tasks, maps each Luigi target to an explicit output location or Airflow asset, and replaces the Luigi schedule with an Airflow schedule. Run the Luigi and Airflow versions against the same input for the same interval, then compare row counts, checksums or schema of the outputs. The step is complete when the outputs match or each difference is explained and accepted by the consumer owner.
Run Luigi and Airflow in parallel
For each ported pipeline, record in a shared register which system owns the production output, which system writes a shadow copy, and who gets alerted on failure. Both systems run on the agreed schedule while consumers keep reading from the Luigi output. Check that the Airflow run history, retries and alerts behave as expected, and confirm that the shadow output is never read by production consumers. Keep this phase until the owners of the affected consumers sign off on the comparison.
Batch the remaining pipelines by dependency
Group the remaining Luigi tasks into batches that respect upstream and downstream links, so a batch never depends on a task that still runs only in Luigi. The team ports each batch with the same comparison and parallel-run checks used for the pilot. Confirm for each batch whether schedules that Luigi left to cron now need an explicit Airflow timetable, and check the catchup setting and timetable behavior described in the Airflow 3 upgrade guide before enabling the schedule.
Cut over pipeline by pipeline
For each pipeline, the owner switches the production output to the Airflow run, disables the Luigi task for that pipeline, and records the cutover time and the consumers notified. The old Luigi task definition is kept in version control for rollback. The cutover for a pipeline is complete when its consumers read from Airflow output for a full scheduled cycle with no open incidents and the rollback path has been tested.
Decommission the Luigi scheduler and cron entries
Once every pipeline has cut over, the team disables the cron entries that started Luigi, stops the central scheduler and its web interface, and removes the Luigi worker hosts from monitoring. Archive the Luigi repository and the last run logs according to your retention policy. Confirm that no cron entry or script still calls Luigi, and that the Airflow run history, alerts and backups cover everything that the central Luigi scheduler previously recorded.
What drives the cost and duration of a Luigi port
This table lists the factors that change the effort of moving Luigi pipelines to Airflow, why each one matters, and what to ask a partner to state in a quote. Use it to compare scope across vendors, since DeployOpen quotes follow a written, fixed scope and no duration figure applies to this migration without that basis.
| Driver | Why it moves effort | What to ask for in a quote |
|---|---|---|
| Number and complexity of pipelines | Each Luigi task graph with many requires branches, dynamic yields or custom parameters needs its own DAG design and testing. | A task inventory with counts per pipeline and a rating for dynamic dependencies. |
| Tasks that use Luigi targets as completion signals | Luigi decides completion by checking whether output targets exist, so each target must become an explicit asset, sensor or idempotency rule in Airflow. | A list of targets per task and the mapping chosen for each (asset, sensor, or rewritten logic). |
| Scheduling spread across cron hosts | Luigi leaves triggering to cron, so the trigger jobs on each host must be found, translated to timetables and retired one by one. | The cron host inventory and the plan for timetables, including how catchup behavior is set. |
| Data stores and secrets the tasks touch | Connections to databases, object storage and APIs must be rebuilt as Airflow connections and a secrets backend, and each one needs testing. | The list of stores, credential sources and the secrets backend the client will run. |
| Python dependency isolation | Luigi tasks often share one virtualenv, so pinned packages may conflict once tasks run as separate Airflow units. | A dependency audit and the isolation approach for each conflicting package set. |
| Parallel-run length and comparison rigor | Running Luigi and Airflow side by side means comparing outputs and run history, which takes longer when results must match exactly. | The comparison method, the parallel-run period and the criteria for cutover, stated before work starts. |
| Number of downstream consumers | Every consumer of a Luigi output needs notice, a changed path or a compatibility shim, and testing against the new output. | A list of consumers and the interface each one reads, plus who signs off on cutover. |
| Airflow 2 starting version | Airflow 3 requires starting from Airflow 2.7 or later, so an older Airflow 2 base needs an intermediate upgrade before the port. | The current Airflow version in use and whether the upgrade is in or out of scope. |
What usually goes wrong in a Luigi to Airflow move
Most Luigi to Airflow problems come from running two schedulers at once, trusting the wrong completion signal, or changing configuration at the wrong moment. Each card below names one risk and the check that catches it before it reaches production consumers.
Duplicate runs during parallel operation
While both systems are live, a job scheduled in Luigi (often by cron) and again in Airflow will write the same outputs twice. Disable each schedule in the old system as the matching DAG is enabled, and keep a written list that maps every cron entry or Luigi trigger to one owner. Before each cutover, query Airflow's run history for the task's outputs and confirm that exactly one system produced each partition.
Silent output differences
Luigi treats a task as complete when its target exists, so a task that writes a partial file can look finished. Airflow has no equivalent default, so a DAG can succeed while producing different rows. Compare row counts, checksums or schema for a sample of partitions from the old and new pipelines over the same input window, and make the completion check in Airflow explicit before you retire the Luigi task.
Timetable change skipping a run
In Airflow 3, catchup_by_default is False and create_cron_data_intervals is False, so bare cron strings use CronTriggerTimetable. If you set create_cron_data_intervals to True before upgrading, the schedule keeps CronDataIntervalTimetable. Changing that setting after the upgrade skips one scheduled run. Decide the value once, record it in the upgrade change ticket, and confirm the next scheduled run dates in the UI before enabling each DAG.
Task code using removed interfaces
Luigi tasks often call database connections or ORM models directly. In Airflow 3, task code cannot use Airflow database sessions or models and must go through the REST API or Task SDK. Run the Ruff AIR301 and AIR302 rules over the ported DAG code in CI, and fail the build on findings. Treat any task that still imports database models as unported, even if it runs in a test environment.
SQLite or unsupported database in production
Teams often start Airflow with the default SQLite backend because it works in a laptop test. The production deployment guide warns that SQLite is for testing only and can lose data in production. MariaDB is also not supported as the metadata database. Confirm that the metadata database is PostgreSQL or MySQL, with a supported version from the prerequisites list, before the first production DAG is enabled.
Luigi retired before consumers confirmed
Other teams may read Luigi outputs, query its central scheduler history, or depend on its web interface for alerts. Shutting down the Luigi scheduler before those consumers are identified breaks them without warning. Build a consumer list from output paths and access logs, get each owner to confirm the new source, and keep the Luigi scheduler read-only for a defined period before decommissioning it.
What buyers ask before moving Luigi pipelines to Airflow
How much does a Luigi to Airflow port cost?
The cost depends on how many tasks, targets and schedules you have, how much custom Python sits inside each task, and how many external systems the pipelines touch. A quote should cover a written inventory of Luigi tasks, the target and dependency mapping, the ported DAGs, tests, a parallel run and cutover, and the handover to your team. Ask whether the quote is fixed-scope, which deliverables are included, and what you own afterwards. Airflow itself is licensed under Apache-2.0 with no paid edition from the project, so the spend is on infrastructure, people and services rather than software licenses.
How long does it take to move Luigi pipelines?
Duration is set mostly by the number of pipelines, how tangled their dependencies are, and how many outputs downstream consumers rely on. Pipelines that read and write well-defined targets port faster than tasks that depend on local files, shared state or undocumented side effects. Plan for time to build a parallel run, because outputs must be compared before anything is switched off. Ask the provider to state the phases, the pipelines in each phase and the exit criteria for each one.
Can our own team port Luigi pipelines without a partner?
Yes, if the team already knows Python, has operated a scheduler, and can run Airflow 3 with a production database and separate components. Luigi users usually need to learn how Airflow models dependencies, scheduling and state, and how Airflow 3 changed task code, since tasks can no longer use Airflow database sessions or models directly. Assess the skills honestly: Python testing, container or virtual machine operations, database administration and access control. A partner adds value when the team lacks time or when a compliance review needs independent evidence of the change.
Is there an automated importer that converts Luigi tasks to DAGs?
Confirm this before planning. Airflow's upgrade documentation covers moving between Airflow versions, including the Ruff AIR301 and AIR302 checks for DAG code, but it does not describe a conversion path from Luigi. Treat any Luigi-to-Airflow converter as something to evaluate on your own code, not as a given. Expect manual translation of task logic, target handling and scheduling, with automated checks used for linting and regression comparison.
How should we handle Luigi targets, and should they become Airflow assets?
Luigi targets, such as files or database rows that signal a task is complete, map to the outputs you want Airflow to track. Decide case by case whether an output should become an asset that triggers downstream DAGs, or stay an internal step. Assets fit outputs that other teams or pipelines consume. Keep the target check idempotent so reruns do not duplicate writes. Document which outputs are authoritative, because this decision also determines what the parallel run compares.
What must our team provide before a Luigi port starts?
Provide a complete inventory of Luigi tasks, their parameters, schedules, and the cron or scheduler calls that start them, since Luigi leaves scheduling to outside tools. Give the provider read and write access to the source code, the target storage and a non-production environment. Name an owner for each output who can confirm correctness and approve cutover. Share the data access rules and any classification constraints, and state which alerting and on-call process receives failures.
How does a parallel run prevent double processing?
Run the Airflow DAG in a mode that writes to separate staging targets or compares outputs, while Luigi continues to own production writes. Disable the Luigi schedule for the production path only after the outputs match over agreed periods. Airflow's scheduler supports more than one instance using database row-level locks, but that protects Airflow itself, not your external systems. Make sure no two systems write the same production target at the same time, and record the cutover date for each pipeline.
How DeployOpen scopes and delivers a Luigi to Airflow migration
DeployOpen plans, implements, hardens, documents and supports self-hosted Airflow inside your own infrastructure, including the move of Luigi task pipelines onto it. We start from an inventory of your Luigi tasks, targets and schedules, then port them in agreed stages so each stage can be checked before the next begins.
How we scope
- We inventory Luigi tasks, their requires chains, output targets and any cron entries that start them.
- The written scope lists which pipelines move, which stay on Luigi during coexistence, and the order of porting.
- Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts; a quote follows from that scope.
How we build
- We map Luigi targets and requires to Airflow dependencies, assets and task outputs, and document each decision.
- We replace cron-driven starts with Airflow 3 schedules, then test catchup and timetable behavior on a non-production instance.
- We run ported tasks alongside Luigi, comparing outputs and run history before any Luigi task is switched off.
How we hand over
- You receive the DAG code, configuration, and documentation of the mapping from each Luigi task to its Airflow equivalent.
- We document how retries, alerts and run history now live in Airflow, so your operators know where to look.
- We walk your platform and data teams through the runbook for retiring the Luigi scheduler and worker hosts.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer after handover.
- A retainer can cover Airflow version upgrades, including the Airflow 3 upgrade checks, once your pipelines are in production.
- We can review any new pipeline you add later so it follows the conventions set during the migration.
