OpenMetadata and Apache Airflow integration

The decision here is where the code that reads Airflow state lives, who runs it, and which credentials it holds. Three routes are commonly considered. The REST API route uses a script or scheduled job outside the ingestion path that pushes metadata into OpenMetadata through its REST endpoints or the Python SDK. The in-Airflow DAG route schedules ingestion as a DAG inside the Airflow environment, so it runs with Airflow's workers, Python dependencies and connections. The direct database route points an ingestion workflow at the Airflow metadata database and reads run and task records from it. Before choosing, your team must confirm in the current OpenMetadata connector documentation which of these is a supported, documented method for your Airflow version.

Each route shifts ownership differently. With the REST API route, your team owns the code end to end: you write, test, version and redeploy the job, and you maintain the mapping between Airflow objects and OpenMetadata entities. Credentials for the OpenMetadata API sit wherever the job runs, so they should live in your secrets backend and be scoped to the minimum needed. OpenMetadata provides REST APIs, a Python SDK, TypeScript packages and Java libraries, so the language your team already operates is a practical factor. With the in-Airflow DAG route, the code lives in your DAG repository and shares Airflow's release, scheduler and worker lifecycle. Its OpenMetadata credentials are held in Airflow connections or the configured secrets backend, so anyone with Airflow administrator access may be able to read them.

The database route carries the most sensitive trade-off. It requires a database account with network access to the Airflow metadata database, which is the store the scheduler uses to record DAG and task state. That is a privileged path into the system that runs your orchestrator, so security teams will want a read-only account, a narrow network path and a documented owner. The table layout belongs to Airflow, not to a published integration interface, so the integration is coupled to Airflow's internal schema and may need changes after an Airflow upgrade. Airflow's production guidance recommends PostgreSQL or MySQL for this database, and the Airflow 3.3.2 prerequisites list supported versions for each. Confirm your database version against those prerequisites before you rely on this route.

Upgrade exposure and version support should drive the final choice as much as ownership does. Confirm which Airflow line you run: the Airflow version support table shows Airflow 2 reached end of life on April 22, 2026, and Airflow 3 was first released on April 22, 2025, with 3.3.2 as the current patch. An Airflow 2 estate needs an upgrade decision before any integration design. Also plan for OpenMetadata 2.0, whose breaking changes include rebuilding custom ingestion images for Python 3.12. Any method that embeds code in ingestion images or in Airflow will need that rebuild. Write these answers down for each candidate route, then review them with your security, platform and data teams before committing to one method.

If OpenMetadata's ingestion framework runs as Airflow DAGs, the ingestion images that execute those workflows become part of your platform upgrade, not only part of the catalog. OpenMetadata 2.0 moved its ingestion images to Python 3.12 with Apache Airflow 3.3.0, so any team running these workflows on its own Airflow must plan around that pairing before upgrading. This section covers that case only. It differs from cataloging your business DAGs, where Airflow is the source system and OpenMetadata reads its metadata. Here, OpenMetadata's ingestion code is the workload, so your team owns its runtime, dependencies and credentials. Confirm the supported Airflow and Python combination in the OpenMetadata 2.0 release notes before you choose a target version.

Custom ingestion images are the first consequence. Any image your team built on an earlier Python base must be rebuilt for Python 3.12 before you upgrade to 2.0, and the rebuild should include only the connector packages you actually run. Build the image in CI, run one ingestion workflow per connector against a non-production source, and tag each image with the OpenMetadata version it was tested against. Great Expectations is a separate gate. The 2.0 release notes list migrating Great Expectations to 1.3 as a step to complete before upgrading. List every ingestion workflow that publishes Great Expectations results, because those workflows can fail in ways that are easy to miss if the migration is skipped. Assign an owner to each item on this list.

Two decisions follow. First, choose the Airflow environment that hosts OpenMetadata's ingestion. A dedicated Airflow instance keeps catalog credentials and upgrade timing separate from data pipelines owned by other teams, while the shared business instance reduces the number of environments to operate. Second, decide how source and database credentials reach the ingestion tasks. Use your organization's secrets backend, and avoid storing these credentials as Airflow variables in the metadata database. Confirm that the chosen secrets approach works with the auth manager and access model in your Airflow environment. Airflow's production guidance recommends PostgreSQL or MySQL as the metadata database and treats SQLite as suitable for testing only. Record both decisions, their owners and the review date in your platform documentation.

Version risk is the third consideration. Airflow 2 reached end of life on April 22, 2026, after which it receives no fixes, and its earlier limited-maintenance period covered only security and critical bug fixes. If OpenMetadata ingestion still runs on Airflow 2, it needs a dated plan to move to a supported Airflow 3 release. Airflow 3 was first released on April 22, 2025, and the current patch listed in the version support table is 3.3.2. Check the Airflow upgrading guide for changes that affect the DAG code, operators and connections your ingestion uses. Complete this move before the OpenMetadata 2.0 upgrade, and record who approves the cutover.

Write the acceptance checks into the scope before any connector is configured. A statement such as "the integration works" cannot be tested, but a list of named checks can be signed off by the platform team, the data owners and the security reviewer. For the Airflow integration, the core set is short: compare a sample of DAGs and their tasks in OpenMetadata against the Airflow UI, confirm that each sampled DAG has an owner and a description, and record a pass or fail result for each item. Choose the sample deliberately, including at least one DAG owned by each team that will use the catalog, so the check covers more than the easiest case. Ask the implementer to attach the evidence to the acceptance record.

Task status needs two runs, not one. First, let Airflow execute a scheduled run of a sampled DAG and confirm that each task's state in OpenMetadata matches the state shown in the Airflow UI once the run finishes. Second, deliberately fail one task, for example by pointing it at a missing input in a test environment, and confirm that OpenMetadata records the failed state and the run that produced it. A successful run alone proves little, because a catalog can display stale success states without any error appearing. Record the Airflow version, the ingestion method (REST API, in-Airflow DAG or direct database connection) and the schedule used for the test. Each of these choices affects how and when status reaches the catalog, so confirm the timing against your own environment.

Lineage needs an end-to-end trace for one pipeline. Pick a DAG that reads from a known source table, writes to a known target table and has a clear owner. Starting from the target table in OpenMetadata, follow the lineage back through the pipeline to the source, and confirm that each edge matches the SQL or operator logic in the DAG. Confirm which lineage paths your chosen method produces and whether column-level lineage is in scope, then test each path on its own. Next, run the ingestion a second time without changing anything. Count the pipeline, task, table and lineage entities before and after, and confirm that their identifiers stay the same. Duplicate entities after a re-run usually point to unstable naming, so treat any increase as a failed check.

Keep profiling, freshness checks and data quality tests out of this acceptance set. They are separate configurations in OpenMetadata, each with its own schedule, scope and failure behavior, so a clean lineage trace does not show that a profiler runs or that a freshness rule fires. Verify each one with its own test: run the profiler on a sampled table and compare the result with a known row count, trigger a freshness check on a table with a known update time, and fail a data quality test on purpose to confirm the alert. OpenMetadata 2.0 made Dynamic Sampling the profiler default, so confirm the sampling settings your team expects before reading profile results. Finally, confirm that the OpenMetadata and Airflow versions used in testing match the versions in the production plan.