Migrate from Oozie to Airflow without losing schedules or lineage
If your Hadoop workflows still run on an Oozie server, plan to move them to Apache Airflow now, because Oozie was retired to the Apache Attic in February 2025 and its project resources now remain available only read-only. The move is a redesign, not a file conversion. Each Oozie workflow becomes a DAG, each coordinator's time-based or data-based trigger becomes a timetable or an asset-driven schedule, and each bundle becomes a set of related DAGs that you start, pause and monitor as a group. Effort depends mostly on how many coordinators, bundles and custom action types you run, since each custom action needs a replacement operator or a justified re-implementation. Apache Airflow is an Apache Software Foundation project released under the Apache License 2.0, with no paid edition from the project itself, so your costs are the infrastructure you run and the people or partner who build and operate it. Start new work on Airflow 3, the current major line, rather than targeting Airflow 2.
Before you write any DAG, build an inventory that a platform owner can sign off. For every workflow, record its schedule, input datasets, the Hadoop actions it calls (Spark, Hive, HDFS, Pig, Shell, SSH), its retry and timeout settings, its SLA and who receives alerts. Then decide per workflow whether it moves, is retired, or is replaced by a managed service. Airflow's providers package list covers Spark, Hive, HDFS, Pig, Impala and SSH, so most standard actions have a mapped operator, but custom actions and Oozie EL expressions in parameters usually need hand translation and testing. Confirm with your Hadoop distribution vendor which Spark and Hive versions remain supported on your cluster, because the Spark versioning policy sets maintenance windows per release line, and your Airflow operators must match the client versions you run. Keep the Oozie server reachable until every coordinator has produced matching output in a parallel run, then decommission it and archive its database and job history for audit.
Which Oozie concepts map to Airflow and which need redesign
This table compares each Oozie concept with its nearest Airflow equivalent and marks where the move is a redesign rather than a translation. Read the middle column for the likely target and the right column for what your team must design or confirm before converting. The main takeaway is that workflows and Hadoop actions convert fairly directly, while bundles and Oozie-specific EL behavior need deliberate design.
| Oozie concept | Airflow target | Migration note |
|---|---|---|
| Workflow (DAG of actions) | DAG with tasks and dependencies | Map each action to one task and confirm error transitions and retry behavior are rebuilt as task retries and trigger rules. |
| Coordinator (time or data triggers) | Timetable or asset-triggered scheduling | Translate frequency and cron-style timing to timetables and replace dataset-availability waits with assets where upstream producers are in Airflow. |
| Bundle (grouped coordinators) | Grouped DAGs or separate deployments | No one-to-one equivalent. Decide whether groups become a set of DAGs, a parent DAG with TaskGroups, or independent deployments, and document the choice. |
| Hadoop actions (Spark, Hive, HDFS, Pig, Impala, Livy) | Provider operators from the Airflow providers packages | Provider packages exist for these systems. Confirm operator parameters and connection settings against the providers reference before assuming parity. |
| SSH action | SSH provider operator | Works for remote commands, but scripts that assumed Oozie's launcher environment, local files or credentials need explicit rework. |
| Oozie EL expressions and job properties | Jinja templating, DAG params and Airflow variables | EL functions have no direct copy. Rewrite each expression as Jinja or Python and move environment values into variables or connections. |
| Oozie sub-workflow and fork/join | TaskGroups or dependency branching | Validate that branch and join semantics match the original, especially when some branches are allowed to fail. |
| Oozie server and its database | Scheduler, API server, DAG processor and metadata database | Airflow 3 splits these components. Plan their hosting, sizing and backups, then retire the Oozie server only after parallel runs match. |
Which documented paths help move Oozie definitions
Reading each Oozie workflow.xml, coordinator and bundle, then writing an equivalent DAG, gives the team the most control over the result. Engineers read each action, decide what it should become in Airflow 3, and write Python that can be parsed, tested and reviewed. This suits workflows with branching logic, custom Java actions or business rules that the team wants to examine before copying.
- Map control flow first: Decision, fork and join nodes become branches and task dependencies, and error paths become retries or failure callbacks.
- Translate coordinators into schedules: Frequency and time zone settings move into the DAG schedule, and data-availability waits move into sensors or assets.
- Keep task code clear of metadata access: Airflow 3 task code cannot use database sessions or models directly, so any logic that read Oozie state must go through the REST API or Task SDK.
- Rebuild nested workflows as TaskGroups: SubDAGs are removed in Airflow 3, so nested sub-workflows need TaskGroups instead.
Check each rewritten DAG against the Oozie run history for the same input window before switching its schedule.
Airflow's provider packages include operators for Spark, Hive, HDFS, Livy, Pig, Impala and SSH, so most Oozie actions have a natural target. The mapping is by function rather than by one-to-one action name, so each operator's parameters still have to be rebuilt from the original action definition. The providers packages reference lists the operators and the packages that contain them.
- Spark and Livy: Spark submissions usually move to the Spark provider, while Livy suits teams that already submit work through a REST session. Confirm which fits your cluster.
- Hive and Impala: Query actions become operators that carry the SQL and connection details, with credentials kept in Airflow connections.
- HDFS and Pig: File operations and Pig scripts have their own packages, so confirm each operator's arguments for the version you install.
- SSH: Shell actions that ran on cluster nodes can often run through the SSH provider, but local file paths and environment variables still need review.
- Version pinning: Pin each provider package to a version tested with your Airflow 3 release, because operator names and arguments change between releases.
Confirm operator names and versions in the providers reference before writing the first DAG.
Oozie was retired in February 2025 and moved to the Apache Attic, where its website, source repository, issue tracker and downloads remain available read-only in the Apache Attic record for Oozie. Nothing there receives fixes, so the archive serves as evidence of how the old system behaved. That matters most when a workflow depends on behavior that the XML never documents.
- Action semantics: Read the source to confirm how actions such as the file system, sub-workflow and decision nodes behave when your XML relies on defaults.
- Issue history: Past issues often explain edge cases in timeouts, retry counts and property inheritance from bundles.
- Property defaults: Compare default configuration values so migrated tasks do not change behavior without anyone noticing.
- Release copies: Keep a copy of the exact Oozie release you ran so the team can reproduce old behavior in a test environment.
- Evidence trail: Record which archived sources informed each mapping decision so engineers and auditors can trace it later.
Save the archived references and the Oozie version in the migration record before decommissioning begins.
We cannot confirm an automated converter from Oozie XML to Airflow DAGs, so treat any such tool as unproven until your team tests it. Converters that exist usually handle common actions well and skip the custom parts of a workflow, so the test should focus on the workflows that look most unusual. Output that looks correct on a diagram may still differ in retries, timeouts or data paths.
- Search before assuming: Check the Airflow project, its providers and community channels for any tool that claims Oozie conversion, and note when it was last updated.
- Test on one representative workflow: Run a candidate on a single workflow with real dependencies, then compare the generated DAG with the original actions.
- List unsupported actions: Note custom actions, Java main classes and nested bundles that the tool does not handle, since these will need manual work.
- Review generated code: Generated DAGs still need Airflow 3 checks, including the Ruff AIR301 and AIR302 rules and the removal of direct database access from tasks.
- Get manual scope in writing: Ask any vendor or partner which workflows they will rewrite by hand and what they will test.
Choose the conversion method only after one tested workflow has been compared with its original run history.
How the Oozie to Airflow cutover is sequenced
The sequence moves Oozie workloads to Airflow one coordinator at a time, with each phase producing evidence that the next phase depends on. Oozie is retired, so every phase keeps a working fallback until its replacement has matched real output.
Inventory every Oozie artifact
Your team lists every workflow, coordinator, bundle, property file, shared library and external dependency, including the Hadoop, Hive, Spark and SSH actions each one calls. Record the owner, schedule, input and output datasets, and the downstream consumers of each output. Done means every artifact has a row in the inventory and every external dependency has a named owner who has confirmed it.
Stand up self-hosted Airflow 3
DeployOpen or your platform team installs Airflow 3 in your infrastructure with PostgreSQL or MySQL as the metadata database, since SQLite is intended for testing only. The DAG processor runs as its own component alongside the API server and scheduler, and the executor is chosen to match your Spark, Hive and SSH targets. Done means a test DAG runs end to end, logs are retained, and backups of the metadata database are verified.
Pilot one low-risk workflow
Pick a workflow with few dependencies, a stable schedule and an output that is easy to compare, then translate it into a DAG using the Airflow 3 timetable and asset model. Run it against a non-production copy of its inputs. Done means the DAG produces output that the owning team accepts and the translation decisions are written down for reuse on later workflows.
Run Oozie and Airflow in parallel
Both schedulers run the same workflows against separate output locations while a comparison job checks row counts, checksums and schema on each run. If Airflow can run more than one scheduler, add explicit guards such as a single writer for each target table or a separate output path, so that parallel runs cannot write duplicate data to production. Done means several consecutive runs match, and every mismatch has a recorded cause.
Cut over coordinator by coordinator
Move one coordinator at a time: disable its Oozie schedule, enable the matching DAG in Airflow, and confirm the first production run lands in the same place. Coordinators that feed other coordinators or bundles move together so no consumer waits on a retired producer. Done means each cutover has a signed-off run record, a rollback step that was tested, and the Oozie job confirmed as stopped.
Decommission the Oozie server
When every coordinator and bundle has run successfully on Airflow for the agreed period, stop the Oozie server and remove its scheduled jobs from the cluster. Archive the Oozie metadata database, workflow definitions and logs in read-only storage that your retention policy allows. Done means no Oozie process or scheduled job remains, the archive can be restored for audit, and the inventory shows every artifact as retired or migrated.
What makes an Oozie migration longer or shorter
This table compares the factors that lengthen or shorten an Oozie-to-Airflow move, and marks whether each one mainly affects the one-time migration or the ongoing run. Use it to shape the questions you put to a partner before asking for a quote. The main takeaway is that custom Java actions, Kerberos ties and downstream data contracts usually drive the effort more than the raw count of workflows.
| Driver | Why it changes the effort | Effect on the ongoing run |
|---|---|---|
| Coordinators and bundles | Each coordinator carries its own frequency, time zone, data dependencies and concurrency settings that must be re-expressed as timetables, assets or sensors. | One-time mapping work, but the resulting schedule count shapes scheduler load afterward. |
| Custom action types and Java main classes | Custom actions have no direct Airflow equivalent and usually need a rewrite as operators, scripts or containers, plus tests for their behavior. | Ongoing: each rewritten operator becomes code your team maintains and upgrades with Airflow. |
| Oozie EL and property inheritance | Expression language functions and job.properties layering must be traced and made explicit in Python, templates or Airflow variables. | One-time analysis, but poorly translated parameters can cause silent differences in run behavior. |
| Data contracts with downstream consumers | Output paths, table names, partition layouts and completion signals that consumers rely on must be preserved or formally changed. | Ongoing: consumers may depend on timing and naming that Airflow runs differently. |
| Kerberos and Hadoop security integration | Keytabs, delegation tokens, impersonation and cluster-specific connections must be rebuilt for Airflow workers and the target platform. | Ongoing: credential rotation and connection management become Airflow operational tasks. |
| Target Spark and Hive versions | Moving to a newer Spark or Hive line can change SQL behavior, configuration defaults and supported Scala or Python versions. | Ongoing: upgrade cadence for Spark and Hive providers joins the Airflow upgrade cycle. |
| History and SLA reporting | Retaining Oozie job history or reproducing SLA alerts requires an archive plan and mapped alerting rules. | One-time export and mapping, plus ongoing retention of the archive for audit needs. |
What usually goes wrong in Oozie to Airflow moves
Most Oozie to Airflow problems do not appear in the DAG code itself. They come from running two schedulers side by side, from timing assumptions that were never written down, and from retiring the old server before its history is kept. Each card below names one risk and the check that catches it early.
Duplicate runs during parallel operation
While Oozie and Airflow both run the same logical job, a dataset can be written twice or a downstream load can fire on each copy. Assign each workflow a single owner for every window of time, and disable the Oozie coordinator before enabling the matching DAG. Check that outputs are idempotent or partitioned by run date, so a repeated run overwrites rather than appends.
Implicit timing and time zone drift
Oozie coordinators often use a frequency and a configured time zone that teams rarely review. Airflow timetables interpret cron expressions, data intervals and time zones differently, so a daily job can shift by hours or process a different data window. Compare the first five scheduled and data interval timestamps from each system before cutover, and record the time zone used by each side.
Direct database access in task code
Oozie action code frequently reads or writes metadata tables directly. In Airflow 3, task code cannot use Airflow database sessions or models and must go through the REST API or the Task SDK. Search the migrated scripts for imports of Airflow models and session calls, and confirm the Ruff AIR301 and AIR302 checks pass before any DAG is scheduled.
Spark and Hive version drift
Oozie jobs often pin Spark, Hive and Hadoop client versions that were chosen for the old cluster. Replacement operators call whichever client exists in the Airflow worker image or the remote cluster, so a job that ran on one Spark line may fail or change results on another. Pin the client versions in the worker image, test each job against the target Spark release, and confirm the provider package versions in your constraints file.
Unmapped bundle behavior
An Oozie bundle starts and stops a group of coordinators together, often with kick-off rules that teams never documented. Airflow has no one-to-one bundle object, so the grouping, shared start conditions and pause behavior must be rebuilt through DAG dependencies, assets or a deliberate operational runbook. Inventory every bundle and write down what starts together, what pauses together, and what should happen after a failed upstream step.
Decommissioning before archiving history
Shutting down the Oozie server removes the job history, action logs and coordinator state that auditors and data owners may still need. Oozie itself is retired and its code remains available read-only through the Apache Attic, so the archive is your responsibility. Export workflow definitions, run history and logs to storage you control, record who approved retirement, and keep the archive for the retention period your compliance team sets.
Questions teams ask before moving off Oozie
What does an Oozie to Airflow migration cost?
The cost depends mainly on how many workflows, coordinators and bundles you run and how much logic each one carries. Simple linear workflows that call one Hive script are quick to translate. Workflows with fork and join branches, decision nodes, custom Java actions, or sub-workflows need design work and testing. Cost has a one-time part (inventory, translation, Spark and Hive rework, parallel runs, decommissioning) and a recurring part (running Airflow, its database, workers and upgrades). Ask any partner for a quote built from a written scope that lists the workflow inventory, the action types in use and the test approach. A quote without those inputs is not comparable.
How long does an Oozie migration take?
Duration is driven by the inventory, not by a fixed number. The main factors are the count of workflows and coordinators, how many depend on Hadoop actions that must be replaced, how many depend on each other, whether data is tested against production-like volumes, and how many owners must sign off on each cutover. Teams that move workflows in waves by business domain usually run Oozie and Airflow side by side until each wave is verified. Ask for a plan with phases and exit criteria, and confirm which dependencies on upstream teams could hold up a wave.
What must our team provide before the migration starts?
Three inputs decide whether the work can start on schedule:
- A workflow inventory: every workflow, coordinator and bundle, its schedule, its inputs and outputs, its action types, and the job.properties it uses.
- Access: read access to the Oozie server, its database and job history, plus Hadoop, Hive and Spark access in a test environment with representative data.
- Owners and test data: a named owner per workflow who can confirm expected results, and sample data or row-count and checksum baselines for validation.
Also confirm which secrets, Kerberos principals and service accounts each job uses, since these need a new home in the target environment.
Can our internal team do the migration without a partner?
Yes, if the team has Airflow 3 experience, knows the Hadoop stack the workflows call, and has the time to run a parallel period. Most in-house teams struggle with three areas: translating Oozie's XML control flow into DAG design, replacing Hadoop actions with Spark, Hive and SSH operators, and proving equal results during coexistence. A partner adds value when the inventory is large or when the same people must also run production. Either way, decide who owns each workflow after cutover, because that ownership is the part that most often lapses.
Is Airflow 2 still an option for a new Oozie replacement?
Not for a new build. Airflow 2 entered limited maintenance on October 22, 2025 and reached end of life on April 22, 2026, after which it gets no fixes. Airflow 3 was first released on April 22, 2025, and its current patch release is 3.3.2. Airflow 3 changes the architecture: the webserver is replaced by an API server, the DAG processor runs separately, and task code cannot use database sessions directly. Confirm the prerequisites for 3.3.2 in the official documentation before you pick Python, database and Kubernetes versions.
How do we keep Oozie job history for audit?
Oozie was retired in February 2025 and moved to the Apache Attic, where its website, source repository, issue tracker and downloads remain available read-only. Before you decommission the Oozie server, export what auditors need: workflow and coordinator definitions from your version control, job run history and logs from the Oozie database, and a record of who approved each cutover. Store the export in a location with retention and access controls that match your policy. Ask your compliance team which records they must keep and for how long, and confirm the format they accept.
What breaks most often when we move from Oozie?
Most failures come from assumptions that were never written down. Common examples are coordinator time zones and dataset availability rules that do not map directly to Airflow timetables and assets, Hadoop actions that relied on a specific cluster path or Kerberos ticket, and downstream jobs that read output before a late upstream run finished. Test each workflow against real data volumes, compare outputs after each wave, and confirm that alerting reaches the same people before the Oozie job is disabled.
Retiring the Oozie server and keeping its records
Do not switch off the Oozie server until you have confirmed that nothing is still running or waiting on it. Start by disabling every coordinator so no new materialized runs are created, then check for jobs in a pending, suspended or running state. A coordinator that is paused but still holds materialized actions can resume unexpectedly if someone restarts the server, so record the final job list and the state of each entry before shutdown. Confirm the same with the owners of downstream consumers, since a file drop or table refresh that Oozie used to trigger may now have no producer.
Export what you will need later before the server goes away. Job history, action logs and the Oozie database contents are the records auditors and support teams ask for when a past run is questioned. Archive them to storage with retention and access controls that match your data classification, and keep a list of which export covers which date range. Store the workflow XML, coordinator and bundle definitions, job.properties files and any shared library or configuration directories in version control, so the exact logic behind each historical run can still be retrieved after the cluster changes. Record the commit or tag that corresponds to the last production deployment.
Revoke access last, and keep the Apache Attic copy in its proper role. Remove service accounts, Kerberos principals, keytabs and any stored credentials that Oozie used to submit jobs to Hadoop, Hive or Spark, and rotate secrets that the server held. The Apache Attic entry for Oozie gives you the project website, source repository and downloads in read-only form, which is useful as reference when someone asks how an action behaved. It does not run jobs, receive fixes or provide support, so do not treat it as a fallback. Confirm the retirement steps with your security and records teams before you decommission the host.
How DeployOpen scopes and delivers an Oozie migration
DeployOpen is a Chainscore Labs company that plans, implements, integrates, hardens, documents and supports self-hosted open-source platforms in your own infrastructure. The cards below show how an Oozie migration moves from a written scope to a delivered, supported Airflow and Spark estate.
Written scope before any work
Each engagement starts with a scope document that lists the Oozie workflows, coordinators and bundles in or out of the move, the target Airflow and Spark versions, the environments, and the acceptance tests. Deliverables and timelines are agreed in writing before work starts. Check that the scope names which jobs are retired rather than ported, since unlisted jobs are the most common source of change requests.
Quote follows the scope
DeployOpen does not publish a price list. A quote is produced from the written scope, so the inventory of Oozie jobs, the number of Hadoop action types to replace, and the validation work drive the figure. Ask for the quote to separate one-time migration work from any recurring support, and to state what the client team must provide, such as Hadoop access, job logs and test data.
Delivery inside your infrastructure
Work is carried out in the client's own cloud accounts, data centers or air-gapped networks. The Airflow environment, Spark connectivity, secrets and the Oozie decommissioning all run on systems the client controls. Confirm early where build artifacts, container images and Airflow connections will be stored, and who holds the administrative credentials at handover.
Hardening and documentation
Delivery includes hardening the Airflow and Spark components to the client's standards and documenting the resulting configuration, runbooks and job ownership. The documentation records each DAG, its schedule, its dependencies and its failure handling, so operators can take over without the migration team. Ask who signs off the hardening baseline and where the documentation will live.
Retainers after cutover
Ongoing support, upgrades and continuous testing are available as retainers. After cutover, the most useful retainer covers Airflow upgrades, Spark maintenance releases and periodic checks that migrated DAGs still match business schedules. Ask for the response expectations, upgrade coverage and what is excluded before signing, because these terms vary with the scope agreed.
Compliance evidence, not certification
DeployOpen prepares systems, controls and evidence for compliance assessments, including records of job scheduling, change management and access to the new platform. It does not issue audit reports, attestations or certifications, which independent auditors and certification bodies provide. Confirm with your auditor which evidence they need from the Oozie retirement and the Airflow replacement before the cutover date is fixed.
