OpenMetadata data lineage setup guide

How to set up OpenMetadata data lineage without guessing at the gaps

OpenMetadata builds lineage from several sources: query logs, pipeline connectors, OpenLineage events, and manual or API-driven edges. None of them covers everything, so the setup work is about deciding which source answers which question in your estate. If your team needs to trace a column back to its origin or find every dashboard affected by a schema change, you need to know which source produced each edge and whether that source is complete for the path in question. OpenMetadata supports table, column, dashboard, pipeline and metric lineage, and each of those types can draw on a different mix of sources.

Enabling a connector is the easy part. Lineage quality depends on the warehouse you run, the orchestrator that moves data between systems, and the SQL patterns your analysts and engineers write. Query log parsing can miss relationships created through dynamic SQL, temporary objects or statements the parser does not resolve. Pipeline connectors only see the tasks and dependencies the orchestrator exposes. Coverage therefore has to be checked per source and per lineage type, against assets your teams already know depend on each other. Ask the connector owner which SQL dialects and orchestrator constructs are handled, then confirm those answers against your own pipelines before you rely on the graph.

Start with a short list of known chains: one column that feeds a regulated report, one dashboard that business users depend on, one pipeline that writes a critical table and one metric definition. For each, write down the expected upstream and downstream path before ingestion runs. After ingestion, compare the graph against that list. Missing edges usually fall into a few groups: the query never reached the query log, the parser could not resolve the statement, the orchestrator did not emit an OpenLineage event, or the relationship lives in a tool with no connector. Each group needs a different fix, so sorting the gaps this way is the first step toward closing them.

Gaps you cannot fix at the source are closed with manual or API-driven edges. Treat these as governed records: name an owner for each edge, record why it exists and review it whenever the upstream pipeline changes. Automated extraction will keep adding edges over time, so decide in advance what happens when an automated edge appears for a path that already has a manual one. Ask your implementation partner or platform team to document that rule before the first production run, and confirm the exact behavior in the OpenMetadata version you deploy.

Which lineage sources and settings OpenMetadata uses

OpenMetadata builds lineage from several sources at once: connectors that read warehouse query logs and pipeline definitions, OpenLineage events emitted at run time, and edges that people or code add by hand. Each source sees a different part of the data flow, so the setup decision is less about turning features on and more about knowing which edges each source can and cannot prove.

Database and warehouse connectors

Warehouse and database connectors read table and column metadata, and where the platform supports it, query log extraction to infer table-to-table and column-to-column edges from SQL that ran. The query log only shows statements the warehouse recorded, so SQL run outside logged paths, such as ad hoc sessions with logging disabled or retention shorter than your reporting cycle, will not appear. Confirm in the connector documentation for your version which query history sources, time windows and lineage flags it supports.

Pipeline and orchestration connectors

Pipeline connectors pull task and job definitions from orchestrators so that lineage can link pipelines to the tables they read and write. They capture declared inputs and outputs, not the logic inside each task, so a transformation that reads a table in code the connector cannot parse will leave its edges missing. Check which orchestrator versions and task types the connector parses, and whether it needs credentials with read access to DAG or job metadata.

OpenLineage event integration

OpenLineage events let running jobs report their own inputs and outputs to OpenMetadata at execution time, which catches transformations that static connectors miss. Coverage depends on each job emitting events through an integration that your team has installed and configured, so jobs without instrumentation remain invisible. Confirm the event transport, the namespace convention used for your data sources, and which OpenLineage facets your version maps to table and column lineage.

Lineage UI for manual edges

The lineage editor in the OpenMetadata interface lets stewards draw upstream and downstream edges between tables, columns, dashboards and pipelines that automated sources cannot prove. Manual edges are useful for transformations inside opaque tools such as spreadsheets, low-code platforms or vendor applications, but they do not update when the underlying SQL changes. Record who owns each manual edge and review them after major pipeline changes, and confirm in the documentation for your version how manual and automated edges are distinguished.

REST API and Python SDK for programmatic edges

The REST API and Python SDK let engineers create or update lineage edges from their own scripts, for example when a transformation runs in a service that no connector reads. Programmatic edges are only as accurate as the code that writes them, so they need tests and an owner, and a failed job can leave stale edges in place. Confirm the exact lineage endpoints, the authentication method your bot identity uses, and whether the SDK version you install matches the server version you run.

Which lineage source to use for each data path

This table compares four ways to get lineage into OpenMetadata: query log extraction, pipeline connector lineage, OpenLineage events from jobs, and manual or API-created edges. Read each row to see when an option fits, what it covers and what it asks of your team to run. The main takeaway is that most estate need a mix, with manual edges kept as the smallest and most-reviewed layer.

Lineage source	When it fits	What it covers	What it costs to operate
Query log extraction	Warehouses with query history you can read	Table and column edges from SQL that ran	Read access to query logs, and checks for lineage missed in dynamic SQL
Pipeline connector lineage	Orchestrators and transformation tools with a supported connector	Pipeline-to-table edges that the connector can parse	Source credentials in the secrets store, and connector upgrades with each release
OpenLineage events	Jobs that already emit or can be instrumented to emit events	Runtime edges from jobs, including steps outside the connector's reach	Emitter changes in pipeline code, and an endpoint that receives events
Manual or API-created edges	Known flows that no automated source captures, such as file drops or spreadsheet loads	Any edge you can name and document, at table or column level	Owner review on each change, and cleanup when the source system is retired

Why automated lineage misses tables and columns

Automated lineage in OpenMetadata is only as complete as the evidence the connectors can see. Most gaps come from five sources: query logs that are truncated or expire before extraction runs, SQL that the parser does not resolve, cross-database or cross-schema references, transformations that live in BI tools or notebooks, and jobs that never emit an event. Before you trust a lineage graph for impact analysis or audit evidence, decide which of these paths feed your catalog and where each one stops. Query-log based lineage depends on how much history the warehouse keeps and how often the ingestion pipeline reads it. If the usage or query log window is short, a monthly dbt run or a quarterly reporting job may never appear, and the edges it created quietly disappear from the view. Confirm the retention period of the query history in each warehouse, set the lineage ingestion schedule to run well inside that window, and record the retention you rely on so auditors can see the limit.

Parsing is the second weak point. Generated SQL from ORMs, dynamic SQL built with string concatenation, temporary tables, and vendor-specific syntax can all produce statements the parser cannot resolve to source and target objects. Unquoted or mixed-case identifiers and column aliases that are reused across CTEs are common failure points. Cross-database and cross-schema references add a further problem when the connector is scoped to one database or when the service name used in a fully qualified reference does not match the service registered in OpenMetadata. In both cases the edge is either missing or attached to the wrong node. Confirm which SQL dialects and reference forms the lineage parser supports for your OpenMetadata version and warehouse in the vendor documentation, and check connector scope settings against every database your critical jobs touch.

Lineage also stops at the edge of the warehouse. Calculations in Looker, Power BI, Tableau or similar tools, spreadsheet-style logic in notebooks, and Python transformations that move data through DataFrames often create no query record that the warehouse logs. The dashboard or notebook then appears with no upstream table, or the chain breaks at the BI layer. Jobs that run outside the connectors you have onboarded, such as scripts on a server or a managed service without an event emitter, have the same effect. OpenLineage events can close some of these gaps, but only for tools that emit them and only where the event payload includes the dataset names OpenMetadata can match. Inventory the jobs behind your top ten dashboards and tag each one as extracted, event-emitting or manual.

These gaps show up in three recognizable ways: a table with a known upstream source that has no incoming edge, a column that exists in a downstream model but has no column-level link back to its origin, and a dashboard with no source table at all. Orphaned columns are often the most damaging because they look complete at the table level. To find them, pick a short list of critical tables, such as those behind regulatory reports, revenue metrics or customer master data, and trace each one by hand against the pipeline code and BI definitions. Compare the traced path with the lineage graph in OpenMetadata, record every missing edge, and classify its cause. Repeating this sampling after each connector or ingestion change shows whether coverage is improving or drifting.

Whatever the sampling shows, the output should be a written gap register that names the asset, the missing edge, the cause and an owner. Some gaps can be closed with connector configuration or an OpenLineage integration, while others need manual lineage entries that the data owner maintains and reviews. Decide early which gaps are acceptable for which use cases, because a lineage graph used for operational troubleshooting can tolerate more gaps than one used to answer an audit question about where a regulated figure came from. Treat the register as a living control, reviewed on the same schedule as your connector and ingestion changes.

Configuring lineage in OpenMetadata step by step

This sequence takes an OpenMetadata instance from a confirmed deployment to table and column lineage that matches how data moves through your platform. Each step ends with a check you can run, so gaps surface in one domain before the rest of the estate is connected.

Confirm the deployment target and service connections
Your platform team confirms where OpenMetadata runs, whether on Kubernetes, bare metal or a Docker trial, and that the database and search services meet the documented prerequisites. For bare metal, that means Java 21, MySQL 8.0.42 or later or PostgreSQL 15 or later, and Elasticsearch 9.x or OpenSearch 3.x. Confirm that each warehouse and orchestrator in the first domain is reachable from the ingestion host over the network path you intend to use. Done when the server responds to requests from the ingestion host and the connection test to the first source succeeds in the connector setup screen.
Create least-privilege read accounts for each source
A source administrator creates a dedicated service account for each warehouse and orchestrator. The account needs read access to metadata, the tables in scope and the query history views that lineage extraction depends on. Avoid reusing administrator or personal credentials. Store the secrets in your approved secret manager and reference them from the connector through that store. Confirm the exact grants each connector needs in its documentation, because the location of query history differs by warehouse. Done when a test query run as that account returns rows from the query history view and the connector test passes without elevated roles.
Run metadata ingestion for the first domain
Create the ingestion service for each source in OpenMetadata, choosing from the connector catalog, which lists more than 130 connectors across warehouses, BI tools, pipelines and ML platforms. Limit the scope to the databases and schemas in the first domain only. Run metadata ingestion once so that tables, owners and pipelines appear in the catalog. Done when the run log shows a table count that matches what the source owner expects, with no unresolved errors, and the domain's tables appear under the correct service in the explorer.
Enable query log and pipeline lineage per connector
In each connector's lineage settings, turn on query log lineage for warehouses that record query history, and pipeline lineage for orchestrators whose jobs read and write tables. Set any date range or lookback option the connector provides to the period your domain reviewers need. Confirm the query history retention on the source side, because lineage cannot reach back past what the warehouse keeps. Run the lineage workflow for this domain only. Done when a sampled table in the domain shows its expected upstream table in the lineage view, and the run log reports no unresolved parsing failures.
Configure OpenLineage emission from pipelines
For pipelines that no connector can read, such as custom jobs or streaming applications, configure them to emit OpenLineage events to the OpenMetadata endpoint. Pipeline owners set the transport and endpoint, and the OpenMetadata administrator issues the credential the emitter uses. Confirm the emitter library version against the OpenLineage documentation for your runtime. Send one test run from a single job with a known input and output. Done when that job run appears in OpenMetadata with the expected input and output tables linked, and the server logs show no rejected or unauthenticated events from the emitter.
Review table and column edges in one domain
A domain steward and a data engineer compare the generated table and column edges against what they know of the domain's pipelines. Select a small set of representative tables, including ones with derived columns and complex SQL, and trace each column back to its sources. Record each gap with its likely cause, such as missing query history, unsupported SQL syntax or a pipeline that does not emit events. Done when the reviewers sign off on the sampled tables and every known gap has a recorded cause and an owner.
Fill remaining gaps with manual or API lineage
Where automated extraction misses an edge, such as a spreadsheet-fed table or a job outside the supported connectors, add the lineage in the UI or through the REST API. Scripted updates can use the Python SDK or the other supported client libraries. Tag each manual edge with an owner and a reason so reviewers can revisit it later. Done when each manual edge appears in the lineage view with its owner and reason, and you have confirmed after a re-ingestion run that manual edges persist, as the connector and API documentation describes.
Extend ingestion and lineage to remaining sources
Repeat the connector, lineage and review steps for each remaining domain, one domain at a time, so every domain has a reviewed baseline before the next one starts. Schedule recurring ingestion and lineage runs once the first domain is stable, and set alerts on failed runs so data owners see them quickly. After each platform upgrade, review parsing failures again, since extraction behavior can change between versions. Done when every in-scope source has a scheduled run, each domain has a signed-off review, and the most recent run logs show no unresolved failures.

How to test that lineage is complete and fails safely

Lineage is only useful if you can show it is complete for the assets that matter and that it fails loudly when extraction breaks. Start by choosing a short list of business-critical tables and dashboards, ideally the ones that feed regulatory reports, executive metrics or customer-facing figures. For each one, trace the path end to end: from the dashboard back through the semantic or BI layer, through the warehouse models, and into the source tables. Then compare that traced path against the transformation code in your repository, such as dbt models, SQL jobs or orchestrated pipeline definitions. Record every missing edge, every extra edge and every edge that points to the wrong column or table in a simple register. This manual walk is slow, but it is the only way to know what automated extraction from query logs, connectors or OpenLineage events actually captured.

Column-level lineage needs its own checks because table-level edges can look correct while the metric logic is wrong. Pick the key metrics your finance, risk or operations teams rely on and confirm that each metric's source columns appear in OpenMetadata with the correct transformation path. Pay particular attention to derived columns, joins, case statements and aggregations, since parsers commonly lose these. Confirm which connector or ingestion path produced each edge. Where a parser cannot resolve dynamic SQL, macros or stored procedures, expect gaps and treat them as findings rather than assuming coverage. Ask the team that owns each pipeline to confirm the trace, so the person who knows the logic signs off on it.

Failure behavior is the second half of the test. Deliberately break ingestion in a non-production environment: revoke or expire the credential the metadata ingestion uses, stop the source connection, or make a pipeline run fail. Then confirm that OpenMetadata records the run as failed and that the failure is visible to the people who receive pipeline alerts. A run that reports success while lineage stops updating is the dangerous case, because the catalog will look current while it is stale. Also check the timestamps on lineage edges so you can tell when each edge was last confirmed. Verify the same behavior for OpenLineage events if you use them, since a missing event stream is easy to overlook.

Next, test propagation. Change an upstream table in a controlled way, for example by adding a column or renaming a field in a test schema, run the next scheduled ingestion, and check that the change appears in the downstream views and dashboards that depend on it. Confirm that the expected edges still exist after the run and that nothing was silently dropped. Repeat the check after a connector or OpenMetadata version change, because parser behavior can shift between releases. Document the results with the date, the version tested and the people who ran the test.

Some gaps will be accepted rather than fixed. Record each accepted gap in a register with the asset affected, the reason it cannot be automated, the compensating control (such as a manual edge or a documented note), the business owner who accepts the risk and the data steward who maintains the entry. Set a review date for each gap and tie re-review to the next verification cycle. The business owner should approve acceptance, not the platform team alone, because the decision is about business risk. Ask your implementation partner or internal team to confirm which gaps are structural (for example, dynamic SQL) and which are configuration issues that can be closed.

Lineage setup on Kubernetes, bare metal and Docker trials

Kubernetes is the deployment model OpenMetadata documents for production, with AWS EKS, Azure AKS, Google GKE and on-premises clusters all covered in the project's deployment documentation. For lineage, the cluster is the place where ingestion runs on a schedule, so the cluster design decides whether query logs and pipeline metadata arrive reliably.

Cluster target: Confirm which of EKS, AKS, GKE or an on-premises distribution you will run, and whether your platform team already operates it.
Ingestion scheduling: Plan which ingestion workflows run for query history, pipeline connectors and OpenLineage sources, and how each schedule is monitored for failures.
Network paths per cluster: Each cluster needs a route to every warehouse, orchestrator and lineage source it reads from. Private endpoints, peering and egress rules often differ between clusters, so map them one cluster at a time.
Secrets and credentials: Store connector credentials in your cluster's secret mechanism and limit which namespaces can read them.
Sizing for volume: Ask the vendor or your own tests to confirm server, database and search sizing against your expected metadata volume and query log size.

Before go-live, confirm that a scheduled ingestion run completes against each source and that the lineage edges it produces match a sample of queries you know to be correct.

What drives lineage effort and cost in your data stack

This table lists the factors that most change how much work OpenMetadata lineage setup takes, with a reduction approach for each. Read each row against your own estimate of warehouses, orchestrators and BI tools to see which factors will dominate your scope. The main takeaway is that scoping lineage to critical domains first and standardizing SQL patterns usually does more to control effort than adding more connectors.

Effort driver	What increases effort	How to reduce it
Warehouses and query log volume	Many warehouses or large query history retention, which lengthens extraction runs and produces more candidate edges to review	Start with the warehouses that feed critical domains, limit query log lookback to a window your owners can review, and confirm log retention settings with your warehouse team
Orchestrators and pipelines needing OpenLineage	Several orchestrators, each with custom operators or pipelines that do not emit OpenLineage events, so their lineage must be instrumented or inferred separately	Inventory pipelines by domain, instrument the highest-value ones first, and confirm which of your orchestrator versions support the OpenLineage integration you plan to use
SQL dialect variety and dynamic SQL	Mixed dialects, stored procedures, and SQL built at runtime from string concatenation, which parsers may not resolve to the final table and column names	Standardize on fewer dialects where possible, move generated SQL to documented templates, and test parsing on a sample of representative queries before committing to a full scope
BI tool count	Each dashboard tool adds its own connector setup, permissions, and model-to-table mapping, and some dashboards depend on semantic layers that need manual confirmation	Onboard the BI tool that serves the most executive or regulatory reporting first, and document which semantic layer objects map to which warehouse tables
Manually curated edges	Lineage that automation cannot see, such as spreadsheet feeds, manual file uploads, or business rules applied outside the database, all requiring someone to add and maintain edges by hand	Record every manual edge with an owner and a review date, limit manual edges to a documented list, and replace recurring ones with pipeline instrumentation where practical
Environments (dev, test, production)	Each environment needing its own connector configuration, service names, and validation, which doubles or triples checks if every environment is treated equally	Treat production as the lineage source of truth, use lower environments to test connector and parser changes, and confirm naming conventions so environments do not create duplicate assets
Compliance expectations for lineage evidence	Requirements to show that lineage is complete, current, and traceable for a named report, model, or regulatory data set, which need evidence such as exports, run logs, and review records	Agree the evidence format with your auditor or compliance lead before build, scope the regulated reports first, and keep run logs and review sign-offs in a retained location

Securing lineage credentials and query log access

Lineage extraction reads more than table definitions. It reads query history, pipeline run metadata and connection secrets, so the credentials and network paths behind those reads need the same scrutiny as the data itself. The cards below cover the decisions that matter most before the first connector runs. Identity controls for the OpenMetadata user interface are covered in the sibling hardening and identity pages.

Read-only accounts per warehouse

Lineage needs query history and table metadata, not write access or access to table contents. Create a dedicated read-only account on each warehouse for lineage extraction, granted only the system views that expose query logs and catalog metadata. Keep one account per source so that revoking a single connector never breaks others, and confirm with each warehouse vendor which system views the account needs for query history.

Scoped access to query logs

Query logs often contain literal values such as customer identifiers, email addresses or account numbers embedded in WHERE clauses. Decide who can read lineage-related query text inside OpenMetadata and in the ingestion logs, and whether literals should be masked or excluded before storage. Confirm with your security team whether the connector options you plan to use can filter or truncate query text, and test that output with sample queries from a sensitive schema.

Connector and OpenLineage secrets in your vault

Connector passwords, private keys and OpenLineage endpoint credentials should live in the team's existing secrets manager, not in ingestion configuration files or the metadata database in plain text. Check how each connector accepts credentials, whether it reads from an external secrets store directly or needs a value injected at runtime, and who can rotate those values. Confirm the supported options for each connector before you design the rotation process.

Least-privilege bot and API tokens

Pipelines that push lineage events or run ingestion authenticate to OpenMetadata with bot or API tokens. Give each pipeline its own bot identity with only the permissions it needs to create or update lineage for its assets, and avoid sharing a single administrative token across jobs. Record which token belongs to which pipeline, set an expiry or rotation schedule, and test that a revoked token stops ingestion without affecting other workflows.

Network paths from ingestion runners

Ingestion runners need outbound paths to every warehouse, BI tool and pipeline orchestrator they read from, and OpenLineage events need a path back to the OpenMetadata server. Map each required source and port, allow only those destinations from the runner subnet, and keep runners out of general corporate network segments where possible. Confirm with your network team whether private endpoints or allow-listed egress addresses are available for each source.

Access review for lineage credentials

Credentials for lineage extraction accumulate over time as connectors are added and staff change roles. Schedule a periodic review that lists every read-only warehouse account, bot token and secret reference, names an owner for each, and confirms it is still in use. Remove accounts for retired sources promptly, and record each review so that auditors or internal reviewers can trace who approved continued access.

Operating lineage after go-live and through upgrades

Once lineage is live, the work shifts from building the graph to watching the pipelines that feed it. Monitor four signals. First, ingestion run status for each connector that produces lineage, including query log extraction and pipeline connectors. Second, connector errors in the run logs, which often show a changed permission or a renamed source before anyone notices a gap. Third, lineage freshness on critical tables, measured against a recency threshold your data owners agree to. Fourth, OpenLineage event delivery failures from the jobs that emit events, since a broken emitter fails silently from the catalog's point of view. Route each signal to the team that owns that connector. Before you build dashboards or alerts, confirm which run-status and failure metrics your deployed version exposes.

Back up two stores and one set of files. The metadata database, either MySQL or PostgreSQL, holds the lineage edges, ownership and configuration, so it is the primary recovery point. The search index, Elasticsearch or OpenSearch, serves the graph views and search results. Confirm with the vendor documentation for your version whether the index can be rebuilt from the database, and time that rebuild during a restore test rather than assuming it. Keep manual lineage mapping files and any scripts that create lineage edges in version control, with review on change, so manual gaps can be replayed after a rebuild. Store backups outside the cluster that runs them, and test restores on a non-production copy on a schedule your operations and compliance teams agree to.

Three documents keep lineage explainable to auditors and to new owners. The source inventory lists every query log source, pipeline connector and event emitter, with its owner, the team holding its credentials, its refresh schedule and the version of any custom ingestion image it uses. The accepted gaps register lists lineage that automated extraction cannot see, such as dynamic SQL, spreadsheet transformations and hand-built transforms outside the emitting tools. Each entry needs a reason, the person who accepted the gap, and a review date. The owner register names a primary and a backup owner for each domain, along with the person who decides whether a disputed lineage edge is corrected. Store these in a location the governance team can review without asking the platform team for access.

OpenMetadata releases arrive regularly, so plan to retest lineage connectors after every upgrade, not only after major versions. OpenMetadata 2.0.0 was released on August 24, 2026, and maintenance releases 2.0.1 to 2.0.5 followed at roughly one to two week intervals. For each upgrade, run every lineage connector against a staging source, then compare edge counts, column mappings and freshness against a baseline snapshot taken before the change. Breaking changes in the 2.0 release notes may affect pipelines that emit lineage. These include moving embedding credentials into an llmConfiguration block, a new Databricks auth format, rebuilding custom ingestion images for Python 3.12, and migrating Great Expectations to 1.3 before upgrading. Check each of these against your pipelines. The full upgrade process is covered in the OpenMetadata upgrades and support topic.

What buyers ask before bringing in help with lineage

How much does an OpenMetadata lineage setup engagement cost?

The cost follows from the written scope, and no fixed figure applies to every environment. The main cost components are the people doing discovery, connector configuration, gap modeling, testing and handover, plus any infrastructure the client provisions for OpenMetadata and its ingestion workers. Costs rise with the number of sources, the number of domains, the number of pipelines that need OpenLineage events, and the amount of manual lineage to model. Ask for a quote that separates one-time work (discovery, initial connector setup, the pilot domain, documentation) from recurring items such as ongoing support, upgrades or a retainer for continuous maintenance. Ask who pays for each infrastructure element, since database, search and compute resources run in the client's own environment and carry their own recurring cost.

How long does OpenMetadata lineage setup take to complete?

Duration depends mainly on how many sources and business domains are in scope, and on how quickly the client can provide access and owner decisions. A single warehouse with one orchestrator is a very different job from several warehouses, BI tools and pipeline platforms spread across multiple domains. Because the total can shift as discovery uncovers gaps, ask for phased milestones instead of one end date. A typical phasing separates discovery and access, a pilot domain with table and column lineage checked by its owners, and then rollout to remaining domains in agreed batches. Each milestone should name its deliverable, the person who accepts it and the evidence used to accept it, so schedule slips are visible early.

What does a lineage project need from our own teams?

The work needs access and people more than tooling. Expect to provide read access to warehouses, orchestrators and their run metadata, and access to query history so table and column relationships can be derived from actual SQL. Pipeline owners must confirm how jobs are scheduled, which jobs feed which tables, and where transformations live. Domain experts are needed to validate lineage for their own data, correct inferred relationships and decide what counts as an acceptable gap. Platform or security staff should approve service accounts, network paths to sources and where the OpenMetadata deployment runs. Ask the partner to list each access request, its owner and the date it is needed, because delayed access is one of the most common reasons lineage projects stall.

Can our own engineers set up OpenMetadata lineage without a partner?

Yes, a capable internal team can set up lineage alone, provided it has the connector documentation, a clear pilot domain and time from the people who know the pipelines. The connector setup itself is usually the more predictable part, because the documentation describes how to configure each supported source and how to schedule ingestion. The hardest part is the gap review: deciding which lineage edges automated extraction cannot see, such as dynamic SQL, file-based handoffs, manual spreadsheet steps or jobs in tools without an integration, and then documenting or modeling them correctly. Teams that keep a gap register, assign each gap an owner and re-test after each change tend to finish with fewer surprises. A partner is most useful for that gap review and for an independent check of the pilot before scaling.

Does OpenLineage require changes to every data pipeline?

No, OpenLineage does not require every pipeline to change, but it does require each pipeline that should emit events to be covered by an integration or instrumentation. OpenLineage events are produced by the tools that run the jobs, so the practical question is which orchestrators, transformation tools and processing engines in your estate have an integration, and which version of that integration your team runs. Pipelines on supported platforms often need configuration rather than code changes, such as pointing events at the OpenMetadata endpoint and setting namespaces. Pipelines outside those integrations may need a small instrumentation step, or they stay covered only by query log extraction or manual lineage. Confirm the integration list and version support against your own stack before estimating effort.

Can OpenMetadata lineage be exported for an audit?

Lineage can be retrieved programmatically, because OpenMetadata provides REST APIs, a Python SDK, TypeScript packages and Java libraries. Those interfaces can be used to pull table, column, dashboard and pipeline lineage on a schedule and write it to files, a data store or an evidence folder that an assessor can review. The format of that export is a design decision, so agree it with your auditor or compliance lead before building it. Include the extraction date, the OpenMetadata version, the source connectors, and any manual lineage with its owner and approval date, since assessors usually ask how lineage was produced as well as what it shows. DeployOpen prepares systems, controls and evidence for compliance assessments, but independent auditors and certification bodies issue the actual audit conclusions.

Which lineage gaps does automated extraction usually miss?

Automated extraction tends to miss relationships that never appear as a clean, parseable statement in logs or job definitions. Common examples include SQL generated dynamically at run time, stored logic that is only partly captured, files passed between systems outside a supported integration, reports built by hand in BI tools, and transformations executed in notebooks or ad hoc scripts. Column-level lineage is especially exposed, because expressions that combine many columns or use functions the parser does not fully understand may be simplified or dropped. The reliable approach is to measure coverage on the pilot domain, list every table with no upstream or downstream edge that the owner expects, and classify each gap as fix in the pipeline, model manually, or accept with a documented reason. Re-run that review after each major source or orchestrator change.

What should a written scope for lineage work include?

A useful written scope names the sources and connectors in scope, the domains and their owners, the lineage types required (table, column, dashboard, pipeline, or metric), and the OpenLineage integrations to configure. It should also state the deployment target and the environment where the work runs, the access requests the client must fulfill with owners and dates, and the acceptance criteria for the pilot and each later phase. Ask for the gap review method, the handling of manual lineage, the documentation delivered at handover, and how audit evidence will be produced. Finally, confirm what is excluded, such as upgrades, custom connectors, data quality tests or identity integration, because those are often sold as separate items. Fixed-scope agreements work best when every assumption behind the quote is written down and reviewed before work starts.

How DeployOpen works on this

DeployOpen approaches OpenMetadata data lineage setup as a fixed-scope engagement agreed in writing before work starts. We plan, build, test, document and hand over lineage inside your own infrastructure, then offer ongoing support as a retainer if you want it.

How we scope

Written scope names the source systems, query logs and pipelines whose lineage will be extracted.
Scope lists which table and column lineage paths must be complete and which may remain manual.
Scope records the deployment target, whether Kubernetes or bare metal, and the database and search services.
Scope defines the OpenLineage producers, if any, that will emit events into OpenMetadata.

How we build

We configure metadata ingestion and query log extraction for each warehouse and BI source in scope.
We wire pipeline connectors and OpenLineage events so job-level lineage links to the right tables.
We fill gaps automation misses with lineage added through the REST API or Python SDK, reviewed before release.
We test column lineage against known transformations and confirm dashboards and metrics trace back correctly.

How we hand over

Handover includes a runbook covering each ingestion schedule, connector credential and its owner.
Documentation records every manual lineage edge, its source, its reason and who approves changes.
Your team receives a walkthrough of how to validate lineage after a schema change or pipeline update.
Handover confirms which lineage tasks your data owners perform and which ones remain with the platform team.

How we support

Ongoing support covers connector and lineage regressions after OpenMetadata maintenance releases.
Support can include rebuilding custom ingestion images when the Python or Airflow runtime changes.
Retainer scope, response expectations and upgrade coverage are agreed in writing, and you should confirm them in the quote.
Confirm with your team how lineage gaps are reported and who approves fixes before the engagement starts.

OpenMetadata data lineage setup