Apache Airflow data lineage with OpenLineage

How Apache Airflow data lineage with OpenLineage answers who depends on what

When a report or model looks wrong, the first questions are which DAGs, tasks and datasets feed it, and which run changed the data. A team running scheduled pipelines needs those answers without reading DAG code by hand. The OpenLineage provider for Apache Airflow addresses this by emitting run and dataset events from Airflow tasks to a transport you configure, which a lineage backend can then store and display. The provider does not build the graph itself, and it does not decide what your backend shows.

Coverage is the first thing to confirm. Lineage is only as complete as the operators and hooks each task uses, since the provider can only report inputs and outputs that those components expose. A task that runs arbitrary Python or shells out to a tool may appear as a run with no dataset edges. Review a representative set of DAGs in a staging environment and compare the emitted events with what your analysts expect to see. Confirm current operator support against the provider documentation for the exact provider version you plan to install, because coverage changes between releases.

The lineage backend is a separate system that your organization chooses, hosts and operates. Storage, retention, access control, upgrades and backups for that system sit outside Airflow, and they carry their own cost and ownership. Before enabling emission, decide which backend will receive events, who can read lineage that may reveal table and column names, and who is responsible when the transport drops events. The Airflow provider can be turned on quickly, but a lineage program is only useful when someone owns the backend and the review process that uses it.

What a lineage setup needs before configuration starts

Lineage events are only as useful as the platform emitting them and the backend receiving them. Confirm the Airflow baseline first, then settle the backend, network paths and naming ownership, since each one shapes the configuration that follows.

Airflow 3.3.2 platform baseline

Plan the lineage work on Airflow 3.3.2, the current patch release, and confirm that your deployment matches the supported Python range of 3.10 to 3.14. Lineage depends on the same scheduler, workers and API server you already run, so a version mismatch between components will surface as missing or partial events. Record the exact Airflow and provider versions in the change record before you touch configuration.

Metadata database and Kubernetes versions

The metadata database must be PostgreSQL 14 to 18 or MySQL 8.0, 8.4 or an Innovation release, and MariaDB and SQLite are not supported for production use. If Airflow runs on Kubernetes, confirm the cluster is on a version from 1.30 to 1.35. Lineage run state is stored alongside task state, so a database that cannot be backed up or sized properly puts traceability at risk along with the scheduler itself.

Chosen OpenLineage-compatible backend

Choose the lineage backend before configuration starts, either one your team already operates or one you will deploy in your own infrastructure. Confirm that the backend accepts the OpenLineage version emitted by your pinned provider, because a backend that lags behind the provider will drop or misread events without an obvious error. Decide who runs its upgrades and retention settings.

Outbound network paths to the backend

Both the scheduler and the workers may need to send events to the backend, so map each source component to the backend endpoint and open only that path. Check proxies, egress policies and TLS certificate trust for the internal CA in air-gapped or restricted networks. Test connectivity from a worker pod or host, not only from your workstation, because the components that emit events are the ones that need the route.

Provider version pinning

Pin the OpenLineage provider to a version compatible with your Airflow release, and confirm the supported OpenLineage version against the provider documentation and the backend's own compatibility notes. Pinning through your constraints file keeps workers, scheduler and API server on the same release. Treat provider upgrades as change-controlled events, since they can alter emitted facets and dataset names.

Namespace and naming owner

Assign one owner for the OpenLineage namespace and the dataset naming conventions before the first DAG emits events. Names such as warehouse tables, file paths and Kafka topics must match across teams, or the lineage graph will show the same asset as several unrelated nodes. Document the convention in a shared location and review it whenever a new source or connection is added.

What lineage covers in a DAG and what it leaves out

Lineage coverage is only as complete as the operators, hooks and SQL that your tasks use. The OpenLineage provider reports what each operator or extractor can see, so the first step is to inventory your DAGs by operator type and mark which ones have lineage support in the OpenLineage provider documentation for the Airflow and provider versions you have pinned. Do not promise full coverage to data owners until that inventory exists. Ask the platform team to confirm the exact supported operators and extractors for the pinned version rather than relying on memory or on older blog posts, because support changes between provider releases.

Task-level runs and dataset relationships are only as complete as what each operator reports. A SQL operator that exposes its statement may yield input and output tables, while a generic operator may report the run but no datasets at all. Custom Python tasks, file transfers that call a library directly, and tasks that write through a client Airflow does not recognize will usually appear as runs without meaningful dataset edges. Where a custom task reads or writes data that matters for audit or impact analysis, plan an explicit lineage declaration in the code, and treat that declaration as part of the DAG's change review.

Before you rely on the lineage backend for any control or impact decision, run a short coverage review in a staging environment. Choose representative DAGs for each operator family and each team, run them with the OpenLineage transport pointed at a non-production backend, and compare the emitted runs and datasets against what the pipeline actually reads and writes. Record the gaps as a list with owners, and decide which gaps are acceptable, which need code changes and which need a documented exclusion. Repeat the review after provider upgrades, since a version change can add or remove reported datasets.

Choosing a lineage backend by team situation

Your organization already runs a service that accepts OpenLineage events, so the work is mostly connecting Airflow to it. The OpenLineage provider sends events from Airflow tasks, and the decisions that matter are which transport the provider uses, how events reach the endpoint and who is allowed to write to it. Most of the effort moves into verification: confirming that endpoint, credentials and event volume match what the backend was built to receive.

Transport fit: Confirm which transports the provider supports and which one your backend accepts before writing any configuration.
Network path: Scheduler and worker hosts or pods need outbound routes to the endpoint, and any proxy must allow the request method and payload size the backend expects.
Access control: Give Airflow a dedicated service identity that can only write lineage events, and keep its credentials in your secrets store, out of DAG code.
Version alignment: Compare the OpenLineage version the backend documents with the version the provider emits, because field changes can silently drop facets.
Operations: The backend team usually owns retention, upgrades and capacity, so agree in advance who responds when event delivery fails.

Before committing, check the supported transports, endpoint format and documented OpenLineage version in the backend's own documentation and in the provider documentation.

How to configure OpenLineage emission in Airflow 3

This sequence takes a team from an Airflow 3 deployment with no lineage to verified OpenLineage events in a lineage backend, with each environment tested before the configuration reaches production.

Confirm the provider version matches Airflow
The platform engineer checks the Airflow 3.3.2 or later release line you run against the OpenLineage provider's compatibility notes in its documentation, then pins the provider version in your Airflow image or constraints file so every component installs the same build. Record the pinned version in your change ticket. The check is done when a fresh image build installs the pinned provider and the Airflow environment lists it with that exact version.
Configure transport and backend endpoint
The platform engineer sets the transport and the backend endpoint using the settings the OpenLineage provider documentation defines for your version, through your secrets and configuration management rather than in DAG files. Confirm the exact option names in the provider documentation before writing them, and make sure the scheduler and every worker or task pod receive the same settings. The check is done when a test task on each executor type starts without configuration errors and the scheduler logs show the emitter initialized.
Set namespace and job naming per environment
The data platform lead agrees a naming convention that separates development, staging and production, usually by putting the environment into the lineage namespace, and documents how job and dataset names are derived from DAG and task IDs. Apply the namespace through environment-specific configuration so a single DAG file produces distinct names in each environment. The check is done when the same DAG run in staging and in a development environment shows different namespaces and no production names appear in non-production events.
Enable emission in staging
The platform team turns on emission in the staging deployment only, using the same Helm values, image and provider pin planned for production so that staging tests the real configuration. Keep a rollback path that disables the emitter without redeploying DAG code. The check is done when a manual DAG trigger in staging produces start and completion events in the logs or network trace from the scheduler and task components, with no errors from the emitter.
Run representative DAGs in staging
The data engineering team runs a set of representative DAGs in staging, covering a simple Python task, a SQL task, a task group, a failed run and a retried run, because each produces different lineage events. Compare what the backend shows against what the pipeline reads and writes, and note any task types that emit no dataset information. The check is done when the backend shows runs for every test DAG, failures are marked as failed, and each dataset the tests read or write appears with the expected name and namespace.
Inspect events in the lineage backend
The platform engineer and a data owner open the lineage backend and confirm that jobs, runs, datasets and their relationships match the staging test plan, checking timestamps, run identifiers and namespace values against the Airflow UI. Confirm with the backend vendor how it handles duplicate or out-of-order events and how long it retains them. The check is done when a sample of runs traces back to the originating DAG run ID in Airflow without manual correction.
Promote configuration through change control
The team submits the pinned provider version, configuration and namespace values through the organization's normal change process, with the staging evidence attached and a named approver for the production window. Deploy to production using the same artifact that passed staging, and keep the rollback switch available for the first production runs. The check is done when production DAG runs appear in the backend under the production namespace and the change record links to the staging test results.

How Airflow objects map to lineage objects

This table pairs each Airflow object with the lineage object the OpenLineage provider emits for it, the naming choice the team must make, and who should sign it off. Read it before the first production run, because namespace and dataset names become the keys that lineage history is stored under. Changing them later splits one dataset's history into two.

Airflow object	Lineage object	Naming decision and owner
DAG	Job grouping of related tasks	Choose a stable DAG ID convention that matches team ownership. Owner: data platform lead with pipeline owners.
Task instance	Run of a job	Confirm which task instances emit events and whether mapped tasks are named per index. Owner: pipeline engineering lead.
DAG run ID	Run identity	Confirm the run ID format is unique per execution and survives retries and manual triggers. Owner: platform engineering.
Operator input table or path	Input dataset	Agree the namespace and table naming form, such as warehouse, schema and table. Owner: data governance lead.
Operator output table or path	Output dataset	Agree how temporary and staging tables are named or excluded so they do not clutter the graph. Owner: data governance lead.
Airflow connection	Namespace source	Map each connection to one namespace per environment, so development and production lineage stay apart. Owner: security and platform owner.
Environment (dev, staging, prod)	Namespace or job prefix	Decide whether environments share a namespace. Owner: platform engineering with compliance.
Renamed DAG or dataset	Separate job or dataset	Plan an alias or migration rule before renaming, since renames create new lineage nodes. Owner: data governance lead.

Running lineage emission as a production service

Lineage emission needs a named owner before it goes live. Someone on the platform team should watch emission health as part of normal Airflow operations, because a lineage gap is silent by default: the DAG still succeeds while the backend stops receiving events. Transport failures usually surface in the scheduler and task logs rather than in the Airflow UI, so the alerting plan should include searches for OpenLineage client errors in those logs, not only task failures. Set the policy for emission failure explicitly. In most teams lineage should not block pipeline success, so a lost event becomes a monitored gap, not a failed load. Confirm in the documentation of the pinned provider exactly how it behaves when the transport cannot deliver an event, and test that behavior against a deliberately unreachable backend before production.

Retries and reruns create repeated and superseded run events by design. Each attempt can emit its own start and completion events, and a backfill or cleared task can re-describe a run that already has lineage. The backend must be able to tell a retry from a new run and to show the latest state for each run identifier, so decide which view analysts will use and test it with a failed-then-retried task. Storage is the other recurring cost. Event volume grows with the number of tasks, retries and dataset facets, so size the backend for the highest expected run frequency, set a retention period that matches your audit and data governance requirements, and schedule purges with a documented owner. Confirm the purge method your chosen backend supports and whether deleted events can still be recovered from backups.

Provider versions move with Airflow upgrades, and emitted event shape can change with them. Treat the OpenLineage provider and its client as pinned dependencies in the same constraints file used for the rest of the environment, and record which versions were tested against which backend release. Before an Airflow upgrade, rerun a representative set of DAGs in staging and compare the emitted events with a recorded baseline, checking namespaces, job names, dataset identifiers and facets. Any change to the transport configuration, such as the endpoint, the API key source or the namespace, should go through the same change record as a DAG release, because a silent change in namespace can split one pipeline into two lineage graphs.

Securing lineage emission and the data it carries

Lineage events describe what your pipelines read and write, so they can expose more than most teams expect. The cards below cover the credentials that send events, the network path they travel, the content of the events themselves, and who can view the result.

Keep backend credentials out of DAG code

Store the lineage backend endpoint and any API key in your team's secrets backend and reference them through Airflow connections or configuration, not in DAG files or task parameters. DAG code is read by many authors and often lands in version control, so a key placed there is effectively shared with everyone who can open the repository. Check that rendered task logs and the Airflow UI do not print the value during emission errors.

Scope emission credentials to write-only access

Where the lineage backend supports it, issue a credential that can only post events, not read other teams' lineage or change backend settings. A leaked write-only key lets an attacker pollute the lineage graph but not read it. Confirm the scope model with your backend vendor, since OpenLineage transports and backends differ in what their tokens can do.

Restrict worker egress to the backend

Allow outbound traffic from Airflow workers and the triggerer only to the lineage backend endpoint, and block general internet access from the same subnets. Without this, a compromised task or a malicious DAG can send data anywhere under the cover of normal pipeline traffic. Test the rule with a failing emission so you know whether a blocked event fails quietly or stops the task.

Review what event facets contain

Facets can carry schema details, job source code, query text and run arguments. Literal values inside SQL, such as customer identifiers in WHERE clauses, can end up in the event, and connection details can appear in job metadata. Review sample events from each operator your teams use, and configure filtering or disable facets that include row data, query literals or connection strings before production rollout.

Map lineage viewer access to DAG roles

Lineage shows dataset names, job names and schedules, which can reveal the structure of sensitive domains even when no data is shown. Grant lineage viewer access through the same roles and groups that govern DAG access in Airflow, so a team that cannot see a DAG cannot see its lineage either. Confirm how your chosen viewer authenticates users and whether its groups can be mapped from your identity provider.

Keep the backend inside your enclave

When policy requires it, run the lineage backend in your own network or an air-gapped enclave, with no outbound connection to a vendor-hosted service. Confirm that every component, including storage, search indexes and any UI assets, can run without external downloads. Record where lineage data is stored and for how long so it can be covered by your retention and data handling rules.

Why lineage events go missing or look wrong

This table pairs common OpenLineage failures in Airflow with their likely causes and a check the team can run to confirm each one. Read it when events are absent, mislabeled, duplicated or unlinked in the lineage backend. Most problems trace to network path, configuration drift between environments, or naming conventions that were never agreed.

Symptom	Likely cause	Fix and check to run
No events reach the lineage backend	Workers or scheduler cannot reach the collector endpoint, or the transport URL points at the wrong host or path	Confirm the transport endpoint from the scheduler and a worker pod or host, then send a test request with curl from the same network segment and check the backend's ingestion logs
Events arrive with the wrong namespace	The namespace setting was copied from a development environment, or was never set per environment, so staging and production runs share one label	Set the namespace per environment in configuration, then compare run events from one DAG in each environment and confirm the namespace field differs
Datasets do not link across DAGs	Two DAGs refer to the same table with different names, such as a schema-qualified path in one and a bare table name in another, or different casing or account identifiers	Agree a naming convention for dataset URIs, list the URIs each DAG emits, and check that the backend shows one dataset node with both producing and consuming runs
Duplicate runs after retries or clears	Task retries and manual clears emit new start and completion events under the same run identifier or a new one, and the backend counts each as a separate run	Review how the backend groups events by run identifier, test a retry and a clear on a non-production DAG, and confirm the duplicate-handling rule your backend applies
Lineage missing for a custom Python task	The task does not use an operator the provider instruments, so no inputs or outputs are declared to OpenLineage	Declare inputs and outputs explicitly on the task or through the supported extraction hooks, then run the task once and confirm the event lists the expected datasets
Events fail or differ after a provider change	The OpenLineage provider version does not match the installed Airflow version, or an upgrade changed the instrumented operators	Compare the installed Airflow and provider versions against the provider documentation in the same environment, pin them together in your constraints file, and rerun a sample DAG after each change

Questions teams ask before setting up lineage

What does setting up OpenLineage with Airflow cost, and what drives the price?

The cost depends mostly on the scope written into the quote, since no price list applies. The main drivers are operator coverage (how many of your tasks use providers that emit lineage automatically and how many need custom extraction), the number of DAGs and datasets to review, the metadata database and Airflow version you run, and the security review your organization requires before data leaves a task. Whether a lineage backend must be built or already exists also changes the work considerably. Ask for a quote that separates one-time work (configuration, extractor fixes, review, documentation) from recurring work (upgrades, backend operations, new DAGs). Fixed-scope quotes are agreed in writing before work starts.

How long does a first working lineage setup take?

A first working setup usually takes less time than the full rollout, because it covers a small set of representative DAGs that prove events reach the backend with the right job, run and dataset names. The duration is set by the size of the DAG inventory, how many operators lack built-in lineage support, whether the backend already exists, and how quickly network and security approvals arrive. Teams that already run Airflow 3 on a supported version typically move faster than teams that first need an upgrade. Confirm the duration against a written scope and a named sample of DAGs, not against the total number of pipelines.

What does our team need to provide before the work starts?

Four inputs remove most of the delay. First, a DAG inventory with owners, schedules, operators and the systems each pipeline reads and writes. Second, a named backend owner who can accept events, set retention and approve access. Third, network approvals for Airflow workers, the scheduler or the component that sends events to reach the lineage backend, including any proxy or certificate rules. Fourth, naming decisions for namespaces, job names and dataset identifiers, since lineage graphs are only as readable as those names. Agree on who signs off each input so the schedule does not stall on a single reviewer.

Can an internal team set up OpenLineage alone, and where does outside help add value?

An internal team with Airflow and data engineering experience can complete a setup alone, particularly for a small number of DAGs that use well-supported operators. Outside help usually adds value in three places: reviewing the operator coverage gap across the full inventory, designing naming and namespace conventions that hold across teams, and testing that events remain correct through upgrades and failure paths such as retries and partial runs. Help is also useful when the security review requires evidence of what metadata leaves the environment. Decide early which of these you will keep in-house so ownership after handover is clear.

What breaks most often in OpenLineage emission, and how do we check it?

The most common failures are missing or inconsistent job and dataset names across DAGs, operators that do not emit lineage and need custom extraction, events dropped because the transport cannot reach the backend, and duplicate or incomplete runs after retries or manual clears. Check each path with a test DAG that reads and writes known datasets, then compare the events in the backend against what the DAG actually did. Repeat the check after each Airflow upgrade and each provider upgrade, since extractor behavior can change between versions. Keep the check in your change process so it runs whenever DAG code changes.

Does OpenLineage help with audit evidence for compliance?

Lineage supports audit evidence by showing which jobs ran, which datasets they read and wrote, and when those runs happened, which helps answer questions about data origin and change. It does not by itself prove that a control operates. Auditors look at the scheduling, change management, access and logging controls around the pipelines, and lineage is one input to that evidence. DeployOpen prepares systems, controls and evidence for compliance assessments, but independent auditors and certification bodies issue audit reports and certifications. Confirm with your auditor which evidence they expect and how long event history must be retained.

Does lineage emission send sensitive data out of our environment?

Lineage events carry metadata such as job names, dataset identifiers, schemas and run times, so the question is what fields appear in those identifiers and in facets. Review which tasks, connection names and table names end up in events before enabling emission to a backend outside your trust boundary. Keep the backend inside infrastructure you control where your policy requires it, and limit who can read event history. Record the decision in the security review so the scope of data exposure is documented.

How DeployOpen scopes and delivers Airflow lineage

DeployOpen plans and implements OpenLineage emission for the Airflow 3 environment you run in your own infrastructure. Engagements are fixed-scope, so the lineage backend, provider configuration, verification and handover are agreed in writing before build work starts.

How we scope

Inventory which DAGs, operators and external systems must appear in the lineage graph and which can stay out of scope.
Confirm the target lineage backend, where it runs, and who in your organization owns its data and access.
Agree in writing which Airflow version, executors and metadata database the lineage work will be built and tested against.

How we build

Install and configure the OpenLineage provider and transport settings in your Airflow environment, with configuration kept under version control.
Test lineage events for representative DAG runs, including failed and retried tasks, and compare emitted datasets with the expected graph.
Check that transport endpoints, credentials and network paths work from scheduler and worker components in your networks.

How we hand over

Deliver a written runbook covering configuration keys, event flow, known gaps in operator coverage and the owners of each component.
Walk your platform and data engineering teams through verifying lineage for a new DAG before it goes to production.
Record the verification steps and configuration baseline so your auditors or internal reviewers can trace how lineage was set up.

How we support

Ongoing support, upgrades and continuous testing are available as retainers, including checks after Airflow upgrades.
Support scope, response expectations and upgrade work are set in the written agreement rather than assumed.
Ask any support provider to confirm how they will retest lineage after provider package or Airflow version changes.