How do you catalog Kafka topics in OpenMetadata?
OpenMetadata brings Kafka topics, their schemas and Kafka Connect pipelines into the catalog through a connector, a REST API or the Python SDK, and the path you use depends on what the release you run documents for your version. Its documented scope covers lineage, glossaries, ownership and data quality features, so a topic can carry the same owner, domain and glossary terms as a warehouse table. Do not assume the exact Kafka and Redpanda entity coverage. Confirm it against the connector documentation for your release before you commit to a design. Without this work, streaming data sits outside the catalog, and topics often have no named owner and no lineage to the consumers and sinks that depend on them.
Apache Kafka is licensed under the Apache License 2.0, and the Apache Software Foundation project offers no paid edition, so there is no Kafka licence line in your budget. The cost sits in the platform work: deploying and running the OpenMetadata services, creating an ingestion service account, configuring the connection to the brokers and any schema registry, scheduling ingestion, and mapping topics to owners and domains. OpenMetadata is also Apache-2.0 licensed and maintained by Collate, which offers commercial enterprise capabilities and premium support as separate, optional products. When you request a quote, ask the provider to separate one-time work, such as the initial connector setup and owner mapping, from recurring work, such as upgrades, connector maintenance and ongoing ingestion monitoring.
The main design decision is how the catalog reads cluster metadata. Kafka authenticates clients with SSL or SASL, and the documented SASL mechanisms are GSSAPI, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER. Confirm which of these the connector supports for your version, and create a dedicated principal for ingestion instead of reusing an administrator credential. In KRaft clusters, the StandardAuthorizer stores ACLs in the cluster metadata log, and resources without ACLs are limited to super users unless allow.everyone.if.no.acl.found is set to true. Grant the ingestion principal describe access to topics and consumer groups as your security policy allows, keep the default restriction in place unless your policy says otherwise, and test the connection from the ingestion host before you schedule a run.
Agree on an accountable owner for each topic family before the first full run, because the connector can populate fields but cannot decide who is responsible. Application teams often create topics without a review step, so schedule ingestion often enough that new topics appear, and assign someone to triage unowned topics each cycle. Check the Kafka version and metadata mode you run, since Kafka 4.0 supports only KRaft and ZooKeeper-based clusters must migrate before upgrading. Ask the connector documentation how it captures Kafka Connect pipelines and whether topic lineage comes from connector configuration or from OpenLineage events. Test both on one cluster before you extend the catalog to the rest of your estate.
What must be in place before you connect Kafka to OpenMetadata
Cataloging Kafka or Redpanda topics and Kafka Connect pipelines in OpenMetadata depends on four groups of prerequisites: the OpenMetadata server, the Kafka cluster, the security setup and the network path. Confirm each group in writing before the ingestion runner is configured. Redpanda needs its own compatibility check.
OpenMetadata server runtime and sizing
Bare metal installs need Java 21, MySQL 8.0.42 or later or PostgreSQL 15 or later for metadata storage, and Elasticsearch 9.x or OpenSearch 3.x for search. The documented minimum for the server is 2 vCPUs, 4 GB RAM and 20 GB storage, and the database and search services each need at least 2 vCPUs, 2 GB RAM and 30 GB storage. Treat these as floors. Confirm sizing against the number of topics and schemas you expect to catalog and the profiling workload you plan to run.
Supported Kafka releases and KRaft mode
Apache Kafka 4.1.2, 4.2.2 and 4.3.1 are the supported 4.x releases listed on the Kafka downloads page. Kafka 4.0 and later run only in KRaft mode, so a ZooKeeper-based cluster must migrate before you plan the connection. Kafka's upgrade notes also recommend moving KRaft clusters older than 3.3.x to 3.9.x before moving to 4.x. Record the exact broker version now, because the behavior you test against depends on it.
Java 17 for brokers and Connect
Since Kafka 4.0, brokers, Kafka Connect and Kafka tools require Java 17, while clients and Kafka Streams applications need at least Java 11. If you catalog pipelines from Kafka Connect, the Connect workers that expose the REST endpoint must also run Java 17. Check the runtime on every Connect host, not only on the brokers. Ask the platform team to confirm the JDK version in each environment before you configure the runner.
Authentication mechanisms and ACLs
Kafka authenticates clients, brokers and tools with SSL or SASL. The documented SASL mechanisms are GSSAPI (Kerberos), PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER. Choose the mechanism the cluster already offers for the ingestion principal, then grant that principal read access to the topics and consumer groups it must describe. In KRaft clusters, resources without ACLs are limited to super users unless allow.everyone.if.no.acl.found is set to true, so confirm the principal is explicitly authorized.
Network path and certificate chain
The ingestion runner must reach every advertised broker address and, for Kafka Connect, the Connect REST endpoint. Brokers that advertise internal hostnames often fail from outside the cluster network, so test name resolution from the runner itself. If TLS is enabled, the runner needs the CA chain and any client certificate its trust store requires. Firewall rules, proxies and certificate expiry are the most common blockers.
Redpanda compatibility check
Confirm compatibility with OpenMetadata for your specific Redpanda version and connector method before you plan the integration. Do not assume a Kafka connector works unchanged against Redpanda. Check any schema registry or Kafka Connect dependency separately, and run a test connection and a sample ingestion in a non-production environment first.
Choosing between a messaging connector, the REST API and the Python SDK
Start by checking whether OpenMetadata documents a messaging or Kafka Connect connector for the exact release you run. Open the connector list in the release documentation, then read the stated scope and stage for that connector. Confirm which objects it reads, such as topics, schemas and Connect pipelines, and whether it reports pipeline status. If the documented connector covers the topics and schemas you need, use it for those objects and avoid writing custom code for them. Record the release you checked, because connector scope can change between versions and the list is the only reliable reference for what is supported.
If the documented scope does not cover the metadata you need, such as Connect pipeline status or custom ownership fields, the next option is the REST API or the Python SDK. OpenMetadata documents both for programmatic access, along with TypeScript packages and Java libraries. Treat this path as custom code. Your team or a partner must own the logic, credentials, scheduling, error handling and tests for each OpenMetadata release. Before choosing it, write down the entities, fields and update frequency you need. Illustrative example: a team that needs an owner field on each topic could run a scheduled job after the connector run that sets that field through the API and logs every change.
Decide who maintains the ingestion code before the first upgrade, not after. OpenMetadata 2.0 moved ingestion images to Python 3.12 with Apache Airflow 3.3.0, and it lists rebuilding custom ingestion images for Python 3.12 as a breaking change. Custom Python code bound to an older image needs to be rebuilt and retested before the upgrade. The 2.0 notes also require migrating Great Expectations to 1.3 before upgrading. Name one owner for image builds, one for reading each release note and one for sign-off. Maintenance releases from 2.0.1 to 2.0.5 arrived about every one to two weeks, so the owner needs a recurring review slot, not a one-time handover.
Both paths depend on the Kafka side. Kafka 4.x requires Java 17 for brokers, Kafka Connect and tools, and at least Java 11 for clients, so confirm the runtime your connector or custom job will use. Access depends on the authentication you configure, either SSL or one of the documented SASL mechanisms, and on KRaft ACLs. Without ACLs, resources are restricted to super users unless the cluster allows otherwise. Ask your Kafka administrator to create a dedicated read-only service account for ingestion, and confirm its permissions for topics, schemas and Connect information. Document the chosen path, the code owner and the service account owner in the runbook before go-live.
Configuration sequence for the Kafka connection
This sequence connects a Kafka cluster, its optional schema registry and Kafka Connect to OpenMetadata in an order that keeps access narrow and results verifiable. Each step ends with a check, so the catalog matches the cluster before owners, glossary terms and lineage are added.
Create a read-only Kafka principal
The team that owns the Kafka cluster creates a dedicated principal for OpenMetadata and grants it only the ACLs needed to describe topics and read schemas. In KRaft clusters, the StandardAuthorizer stores these ACLs in the cluster metadata log. Confirm how the cluster treats resources without ACLs, because if allow.everyone.if.no.acl.found is set to true, unmatched resources may be readable by other principals. The step is done when a test client authenticated as this principal can list topics and fetch their metadata, but cannot produce or consume messages on sensitive topics.
Configure the documented authentication mechanism
Choose SSL or one SASL mechanism the cluster already documents, such as SCRAM-SHA-512 or OAUTHBEARER, and mirror it in the OpenMetadata connection settings. Kafka's documented SASL mechanisms are GSSAPI, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER. OpenMetadata does not run multiple authentication mechanisms at once, so keep one method only. Confirm the exact fields for your chosen method in the connector documentation for your OpenMetadata version. The check is a successful test connection with the dedicated principal and no authentication errors on the service page.
Register the messaging service and ingest metadata
Create a messaging service in OpenMetadata that points to the cluster's bootstrap servers, then run ingestion with metadata extraction only. Leave profiling and sampling off for the first pass so the run stays focused on topics and schemas. The check is that the topic count in OpenMetadata matches the topic count your Kafka administrator reports for the same cluster, after accounting for internal topics such as the consumer offsets topic if your filter excludes them. Record any differences and their causes before moving on.
Add schema registry access if used
If producers and consumers rely on a schema registry, add its endpoint and a read-only credential to the service configuration, using an authentication approach that matches your broker setup. Confirm which registry types your connector version supports, because support can differ between releases. The check is that a sample topic shows its schema version and fields in OpenMetadata. If the team does not use a registry, record that decision in the change record so the step closes explicitly.
Register Kafka Connect as a pipeline service
Register Kafka Connect as a separate pipeline service using the REST endpoint of the Connect cluster and a read-only credential. Connect clusters on Kafka 4.0 or later require Java 17, which matters for any self-managed host you use. Confirm which connector stage applies to your Connect version and deployment, and check the OpenMetadata documentation for the scope of Connect metadata it extracts. The check is that each expected connector appears as a pipeline with its status, and no connector listed by the Connect REST API is missing.
Assign owners and glossary terms to topics
Owners and glossary terms are decisions for people, so have the owning domain teams assign them in OpenMetadata using the domains and glossary agreed earlier. Settle the rule for which team owns each topic before any bulk editing starts. If you have many topics, bulk updates can go through the OpenMetadata API, but confirm the request format for your version first. The check is that a sample of topics from each domain shows an owner on the topic page, and each glossary term resolves to its approved definition.
Run lineage ingestion where a source is documented
Lineage between topics and downstream tables, pipelines or dashboards depends on a lineage source your OpenMetadata version documents. OpenMetadata supports OpenLineage event integration and lineage across several asset types, but confirm which Kafka-related lineage path your deployment supports before you plan around it. If no source is documented for your streaming flows, record that lineage stays manual. The check is that one producer-to-consumer path your team knows well appears as an edge between a topic and its downstream asset, with the direction correct.
Schedule recurring ingestion and review failures
Once the first runs are clean, schedule metadata ingestion at an interval the domain owners agree on, and run it under a service account the cluster team controls. Set a regular review of failed runs and of topics that still have no owner, so each drift item has a named person. The check is that the scheduled run completes, and that topics added or removed since the previous run match an approved change ticket on the Kafka side.
How Kafka objects map to OpenMetadata entities
This table lists each Kafka and Kafka Connect object, the OpenMetadata entity or field it populates, who sets that value, and whether the mapping is confirmed in the connector documentation. Check each row against the connector documentation for the OpenMetadata release you run before you commit to a design. Ownership, glossary terms and identity mapping are decisions you make in OpenMetadata and record, not values Kafka supplies.
| Kafka object | OpenMetadata entity or field | Who sets the value | Confirmed in connector docs? |
|---|---|---|---|
| Topics | Topic entity under a Kafka messaging service, with name and cluster reference | Ingestion sets name and service; steward adds description | Confirm the field list for your release |
| Partitions | Partition-related fields on the Topic entity | Ingestion | Confirm which partition fields are ingested |
| Consumer groups | No dedicated entity assumed; decide whether to track in a custom property or outside the catalog | API script if tracked; otherwise not set | Not confirmed; verify before design |
| Schemas | Message schema field on the Topic entity, from a connected schema registry | Ingestion where the registry is supported; API script otherwise | Confirm supported registry types in your release |
| Kafka Connect connectors | Pipeline entity under a Kafka Connect pipeline service | Ingestion | Confirm the Connect connector and version support |
| Connector tasks | Task entries within the Pipeline entity, with status where the connector reports it | Ingestion | Confirm task-level detail in the connector docs |
| Topic-to-connector relationships | Lineage edges between Topic and Pipeline entities | Ingestion where emitted; API script for gaps | Confirm lineage coverage for Connect sources and sinks |
| Ownership, glossary terms and tags | Owner, glossary term and tag fields on Topic and Pipeline entities | Steward in OpenMetadata; Kafka ACLs do not supply these values | Not applicable; set and maintain in OpenMetadata |
| Kafka principals to OpenMetadata users | User and team assignments in OpenMetadata | Platform owner decides and records the mapping | Not automatic; record the decision before onboarding |
Securing the connection between Kafka and the catalog
The catalog reads topic and schema metadata from Kafka through a dedicated principal, so each control below limits what that principal can reach and who can change what it finds. Transport encryption and authentication protect the connection, authorization limits what is visible to the ingestion account, and OpenMetadata roles and SSO govern who edits the catalog.
Encrypt ingestion traffic with TLS
Kafka encrypts traffic between clients, brokers and tools with SSL/TLS, but security is optional, so a cluster can run unsecured or mix secured and unsecured listeners. Configure the ingestion runner to connect only to a secured listener, and confirm that the advertised broker endpoints use the same protocol. Check that broker certificates match the hostnames the runner uses, and confirm which certificate authority the runner must trust before the first ingestion run.
Choose a SASL mechanism and store its secret
Kafka documents GSSAPI (Kerberos), PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER as its SASL mechanisms. Pick the one your identity estate can operate and support over time, and confirm it is enabled on the listener the runner uses. Store the principal's password, keytab or token in your approved secrets store, not in ingestion configuration files, and confirm how a rotated secret reaches the runner.
Grant read-only ACLs to the ingestion principal
Give the ingestion principal only the operations its connector needs on the topics, consumer groups and schemas it catalogs, confirmed against the connector's documentation. In KRaft clusters the StandardAuthorizer stores ACLs in the cluster metadata log, so grants are made through cluster admin tooling and should go through your change process. Verify the grants with a test ingestion run and check the broker logs for authorization errors.
Keep allow.everyone.if.no.acl.found disabled
Resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Enabling it would let any authenticated principal reach unprotected topics, which undoes the limits set for the catalog principal. Confirm the setting is not true on any broker, and list each topic that needs access so it receives a deliberate ACL the catalog can use.
Limit ownership and glossary edits to stewards
OpenMetadata role assignments decide who can change ownership and glossary terms on catalogued Kafka topics and schemas. Limit those edits to stewards, and keep the ingestion account out of any role that can edit metadata by hand. Review the role definitions in the version you run, since role names and permissions can change between releases, and test the setup with a non-steward account before go-live.
Review SSO and plan for basic authentication
OpenMetadata documents SSO with Auth0, Azure, Google, Okta, Amazon Cognito, OneLogin, Keycloak and custom OIDC providers. Choose the provider your workforce already uses and confirm it meets your sign-in policy. OpenMetadata does not support combining SSO with basic authentication, so confirm how the ingestion service authenticates under the chosen mode before switching off password logins.
How to verify the catalog matches the cluster
The verification test plan is what shows the catalog matches the cluster, so run it once after setup and again after every OpenMetadata upgrade and every Apache Kafka upgrade. Start by listing topic names from the cluster using the same principal the OpenMetadata ingestion will use, then compare that list with the topic entities OpenMetadata holds for the service. Check both directions. Topics missing from the catalog usually point to filter rules or permissions that block the ingestion principal. Topics in the catalog that no longer exist in Kafka point to stale entities or to a filter that was not applied. Decide in advance how internal topics are handled, and record each comparison with the date, the cluster identifier and the ingestion run ID so the result can be traced later.
Next, create a test topic under a name prefix reserved for catalog tests, then run the next scheduled ingestion or trigger one manually. Confirm the topic appears with the expected name, partition count and key configuration. For schema checks, pick one sample topic that has a registered schema, publish a new schema version, run ingestion again, and confirm OpenMetadata shows the new version while keeping the earlier one visible. Confirm in the connector documentation for your OpenMetadata release which schema source it reads and whether it reads schema versions at all, because this behavior is a decision to check, not an assumption. Remove the test topic afterward and confirm how the catalog handles the deletion on the next run.
Verify one Kafka Connect pipeline end to end. Choose a connector whose source and sink topics you already know, then confirm the pipeline entity exists, its status is captured, and it links to the topics it reads or writes. Lineage should be tested only where the documented scope supports it. Confirm which connector types and which OpenLineage event sources your release supports before writing expected results. Next, test that owners and glossary terms survive re-ingestion. Assign an owner and a glossary term to the test topic and the sample pipeline, run ingestion again, and check that both assignments remain. Also confirm how the connector treats metadata edited in the UI, since some ingestion behavior overwrites manual changes.
Test the ingestion principal's permissions directly. The principal should be able to describe topics and read the metadata the connector needs, but it should not be able to produce messages, change topic configuration or delete topics. Attempt each of those actions with the principal's credentials and record the denial. In KRaft clusters, ACLs are stored in the cluster metadata log, and resources without ACLs are limited to super users unless allow.everyone.if.no.acl.found is set to true. Confirm that setting in the test cluster, so the grants you reviewed are the only grants that apply. Also confirm the SASL mechanism or TLS configuration the principal uses, since the connection method is part of the test result.
Lineage and data quality checks should run against known sample pipelines, where the correct answer is already documented. Build one small flow with a known source, a known topic and a known consumer, then check that the lineage shown matches it. Run one data quality test that is designed to pass and one designed to fail on a sample topic, and confirm both results appear as expected. Any lineage that the documented scope cannot produce should be recorded as a gap, with the reason and the team that accepts it. Do not infer edges from topic naming patterns or from consumer group names, because an inferred edge looks correct in the catalog until a change in the pipeline breaks it without notice.
Common failures when Kafka metadata does not reach OpenMetadata
This table lists common reasons Kafka topics, schemas or Connect pipelines fail to appear in OpenMetadata, with the likely cause and first fix for each. Read each row from symptom to cause, then confirm the fix against the connector and release documentation for your version before changing configuration. The main takeaway is that most failures trace to authentication, Kafka ACLs, Connect setup or an upgrade step.
| Symptom | Likely cause | Fix |
|---|---|---|
| Ingestion fails authentication | Wrong SASL mechanism, expired credentials, or a TLS trust chain the ingestion runner does not have | Match the mechanism the broker lists, rotate credentials, and install the cluster CA on the runner. Confirm settings in the release docs. |
| Topics missing from the catalog | ACLs too narrow for the ingestion principal, or a super-user-only default blocking reads | Grant the ingestion principal describe and read access to the topics in scope. Without ACLs, only super users can read by default. |
| Kafka Connect pipelines absent | Wrong Connect REST endpoint, Connect on an unsupported Java runtime, or a connector stage that does not cover the feature | Point the connector at the Connect REST API the cluster serves. Kafka 4.0 and later Connect requires Java 17. Check the connector's documented scope. |
| Ingestion images fail after upgrade | OpenMetadata 2.0 moved ingestion images to Python 3.12, so custom images built on older Python break | Rebuild custom ingestion images on Python 3.12 before upgrading. Confirm the required base image in the 2.0 release notes. |
| Duplicate authentication error | SSO and basic authentication configured together, which OpenMetadata does not support | Choose one authentication mechanism per deployment and remove the other from the server configuration. Confirm in the security docs. |
| Slow ingestion or UI on search and database | Database or search service at minimum size, or undersized for your topic and schema volume | Size above the documented minimums of 2 vCPUs, 2 GB RAM and 30 GB storage per service, based on your topic volume. Confirm in docs. |
Questions teams ask before connecting Kafka to OpenMetadata
How much effort does a Kafka to OpenMetadata integration take?
Effort depends mostly on scope and on how much of that scope a documented connector covers. Scope drivers: the number of Kafka clusters, topics and Kafka Connect pipelines to catalog, and whether each one maps to a documented OpenMetadata connector or needs custom code against the REST API or SDKs. Security drivers: the authentication model on each cluster, the service principal OpenMetadata uses and the ACLs that principal needs. Governance drivers: how many topics need named owners, glossary terms and classifications before the catalog is useful. These multiply across environments, so ask for the effort to be broken down by cluster, connector and governance work.
Can our team connect Kafka to OpenMetadata without a partner?
Yes, if your team already has three things: Kafka administration skills, operating experience with OpenMetadata (deployment, upgrades and ingestion runs), and a stewardship process that assigns owners and keeps glossary terms current. A common gap is the third, and the catalog stalls there. A partner adds value by deciding which topics and pipelines to catalog first, configuring connectors and the service principal with minimum ACLs, testing ingestion against your clusters, setting up scheduled runs and handing over runbooks your team can operate. DeployOpen plans, implements, documents and supports self-hosted platforms inside your own infrastructure, so operation stays with your team after handover.
How long does a Kafka and OpenMetadata integration take?
Duration depends on the scope written into the engagement, so no fixed figure applies here. The main factors are the number of clusters and environments, how many topics and Connect pipelines are in scope, whether connectors cover them or custom code is needed, the security model you must match and how quickly owners and glossary terms can be agreed. Most DeployOpen engagements are fixed-scope, so the timeline should appear in a written plan that lists phases, deliverables, client dependencies such as cluster access and test environments, and acceptance tests. Ask any provider which dependencies could move the dates and how delays from data owners will be handled.
What must we provide before the Kafka integration starts?
Before work starts, the client usually needs to provide the following. Read access to the Kafka clusters in scope, including broker endpoints and the authentication details the integration will use. A test environment with a non-production cluster or a representative topic set where ingestion can be validated. Named data owners for the topics, schemas and Connect pipelines, who can confirm meaning and approve glossary terms. A decision on the Kafka principal and secrets process, meaning which identity OpenMetadata uses, where its credentials are stored, how they are rotated and who approves changes. Ask the provider to list these as written dependencies so gaps show up early.
Is Redpanda supported as a source in OpenMetadata?
Confirm this before you commit. Redpanda implements the Kafka API, but OpenMetadata support depends on the specific connector and the Redpanda version you run. Check the OpenMetadata connector documentation for the Kafka and schema registry sources and their listed scope, then test against your exact Redpanda version in a non-production environment. Verify which metadata is read (topics, schemas and configuration), how authentication is handled and whether Kafka Connect pipelines are covered if you run Connect alongside Redpanda. If a documented connector does not cover a feature you need, the work becomes custom development, which changes the effort and should be written into the scope.
Does DeployOpen issue audit or compliance attestations for the catalog?
No. DeployOpen does not issue audit reports, attestations or certifications. It prepares the evidence an independent assessor will ask for, such as documented architecture, configuration baselines, the ACL and principal inventory, ownership and glossary assignments, change records for connector runs and upgrades, and access review notes. Your auditor or certification body decides what satisfies a given control. Ask your assessor early which records they expect, and design the catalog's ownership and lineage process so it produces those records as routine output.
Which Kafka security settings must we decide before connecting OpenMetadata?
Decide the security model before the first connection. Kafka uses SSL/TLS to encrypt traffic and SASL to authenticate clients, with GSSAPI (Kerberos), PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER as the documented SASL mechanisms. Kafka security is optional, so confirm that every listener OpenMetadata reaches is secured as your policy requires. Create a dedicated principal with only the ACLs ingestion needs. In KRaft clusters, the StandardAuthorizer restricts resources without ACLs to super users unless allow.everyone.if.no.acl.found is set to true, so leave that setting off unless you have reviewed the exposure. Store the principal's credentials in your secrets manager and plan how they will be rotated.
Which Kafka and Java versions work with this integration?
Match the Kafka version to a supported release line and confirm the Java runtime. The Kafka downloads page lists 4.3.1, 4.2.2 and 4.1.2 as supported releases, and older versions are archived. Since Kafka 4.0, brokers, Kafka Connect and tools require Java 17, while clients and Kafka Streams applications require at least Java 11. Kafka 4.0 supports only KRaft mode, so ZooKeeper-based clusters must migrate to KRaft before upgrading, and KRaft clusters older than the 3.3.x line should first move to 3.9.x. On the OpenMetadata side, version 2.0 moved ingestion images to Python 3.12, so custom ingestion images must be rebuilt. Confirm the combination against both vendors' current documentation before planning.
How DeployOpen scopes and delivers the Kafka integration
DeployOpen scopes, builds and hands over the OpenMetadata and Kafka integration inside infrastructure you control, starting from a written scope that names the clusters, topics and Kafka Connect pipelines in play. Each stage produces named deliverables your team can review before the next one begins. Engagements are fixed-scope, and ongoing support is available as a retainer.
How we scope
- The written scope lists which Kafka or Redpanda clusters, topic families and schema registries are cataloged.
- It names the Kafka Connect pipelines to model and the owner team for each one.
- Open items such as connector coverage and version compatibility are confirmed against current release notes before build starts.
How we build
- Connectivity uses the SSL or SASL settings your brokers already enforce.
- Topic and schema metadata is mapped to OpenMetadata domains, owners, glossary terms and classifications.
- Lineage between Kafka Connect pipelines and downstream tables is traced through OpenLineage events where your stack emits them.
How we hand over
- Handover includes the configuration as code, a runbook and the ownership model for each catalogued topic.
- Your team receives verification results showing which topics, schemas and pipelines appear with owners and lineage.
- Access control for the catalog and the Kafka principals used for ingestion is documented for your security team.
How we support
- Ongoing support is available as a retainer for OpenMetadata upgrades, connector changes and new topics.
- Kafka version upgrades are planned as their own work, checked against the upgrade notes for each release.
- Catalog changes are reviewed against the OpenMetadata release notes before each upgrade is applied.
