Change data capture with open source event streaming

Why change data capture with open source event streaming fits regulated data estates

When batch extraction is too slow, too heavy on source databases, or blind to deletes and intermediate changes, Debezium on Kafka Connect, self-hosted, is the usual open source route to row-level change capture. Debezium is Apache 2.0 licensed, gets its durability and fault tolerance from Kafka and Kafka Connect, and runs each connector as a Kafka Connect job. The connector install is the easy part. The design work sits in source database permissions, the initial snapshot, how schema changes flow downstream, and how long change topics are retained, and those decisions shape both the load on production systems and the evidence an assessor will later ask for. Start with the source inventory. For each operational database, confirm which Debezium connector is listed as stable for that engine in the Debezium connector documentation, and which are incubating or unsupported for your version. Ask the database team which replication or log settings they will allow, what the connector account may read, and who approves a snapshot on a busy production table. Agree before go-live how schema changes will be handled, since a column rename upstream can break consumers that were never told. Decide retention per topic in writing, because change history is often the only record of who changed what and when. DeployOpen plans, installs, hardens and operates this pipeline inside your own cloud accounts or data centers, with the source permissions, snapshot runbooks, schema change procedure and retention settings documented as part of the handover. DeployOpen prepares systems, controls and evidence for compliance assessments, but it does not issue audit reports or certifications; those come from independent auditors and certification bodies. Ask any implementation partner to show which logs, access records and configuration exports they will hand you for the assessor, and who will own them after the project closes.

Requirements to agree before choosing a CDC design

This table lists the requirements to settle before building change data capture on Debezium and Kafka Connect, with the source of each and what it means for the build. Read each row as a decision to confirm with your database owners, security team and platform team before a connector is deployed. The main takeaway is that source access, transport security and topic access are set per connector and per cluster, so they need agreement before the first topic is created.

Requirement	Where it comes from	What it means for the build
Source access and least privilege	Debezium connector documentation lists stable connectors per database; each connector needs its own database account	Create a dedicated replication account with read access to the captured tables and the change log only. Confirm required grants for each database with the vendor documentation and your DBAs.
Log settings and retention on source databases	Each database engine exposes its change log differently; Debezium depends on it	Confirm log mode, retention window and whether logs survive backups with your DBAs. Short retention can cause a connector to lose its position after an outage.
Encryption between connectors, brokers and clients	Kafka security overview: SSL/TLS is optional and must be configured per listener	Enable TLS on every broker listener used by Kafka Connect and consumers. Avoid mixed secured and unsecured listeners for CDC traffic that carries customer or account data.
Authentication with SASL or SSL	Kafka security overview: documented SASL mechanisms are GSSAPI, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER	Pick one mechanism the identity platform supports and give the Connect worker its own principal. Confirm credential storage and rotation with your security team.
Topic access through ACLs	Kafka authorization documentation: in KRaft clusters, resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true	Grant the Connect principal write access to its change topics and offset, config and status topics only. Keep allow.everyone.if.no.acl.found at its default unless you have a documented reason to change it.
Supported Kafka release line and Java runtime	Kafka downloads page lists 4.3.1, 4.2.2 and 4.1.2 as supported; upgrade notes require Java 17 for brokers and Kafka Connect since 4.0	Standardize on a supported release line and a Java 17 runtime for brokers and Connect workers. Confirm the patch cadence and upgrade window with your platform team and the Debezium version you plan to run.
Cross-site resilience through MirrorMaker 2	Kafka geo-replication documentation: MirrorMaker 2 is built on Kafka Connect and replicates topics, consumer group offsets and ACLs	Needed only if change streams must survive a site loss. Agree which topics replicate, where offsets are translated and how failover is tested before going live.

Which open source platforms fit CDC and why

Change data capture pipelines usually combine a connector that reads the source database log, a broker or runtime that carries the changes, and a scheduler for the loads that stay batch. The cards below cover the components a team is most likely to choose between, and the decisions each one forces.

Debezium connectors on Kafka Connect

Debezium is Apache-2.0 licensed and reads row-level changes from the database log, then publishes them as events. Its durability and fault tolerance come from Kafka and Kafka Connect, and each connector is deployed to the Kafka Connect service. Check that your source database and version are on the stable connector list, and confirm how the connector will be upgraded alongside the Kafka Connect workers. This is the default choice when Kafka is already the event backbone.

Debezium Server without Kafka

Debezium Server runs the same connectors as a standalone process and needs no Kafka cluster. A team might pick it when changes feed a single downstream system, such as a cache, search index or object store, and running a broker would add more operating work than the use case justifies. You give up Kafka's replay, multiple consumer groups and the retention model that many downstream teams rely on, so confirm those needs before choosing it.

Apache Kafka and Kafka Connect

Apache Kafka and Kafka Connect are released under Apache-2.0, and there is no paid edition from the project itself. The supported releases are 4.3.1, 4.2.2 and 4.1.2, and Kafka Connect ships with each Kafka release. Since Kafka 4.0 the cluster runs only in KRaft mode and requires Java 17 for brokers and Connect, so ZooKeeper-based clusters must migrate first. Plan the Kafka upgrade path and the Connect worker rollout as one change.

Apache NiFi for visual lineage

Apache NiFi is an Apache-2.0 dataflow system with a browser-based interface for designing, controlling and monitoring flows. Its data provenance records lineage from beginning to end, which helps teams that need to show where each change came from and where it went. NiFi can read database changes, but it is a different operating model from Debezium, with its own cluster and flow lifecycle. Check whether visual design is a real requirement or a preference before adding a second streaming stack.

Apache Airflow or Dagster for batch loads

Apache Airflow is Apache-2.0 licensed, and Dagster is Apache-2.0 licensed with 1.13.26 published on October 9, 2026. Both schedule and monitor periodic work, such as full reloads, reconciliation jobs and tables that do not justify continuous capture. Keep them alongside CDC rather than replacing it. Confirm which pipelines still run on a schedule, and who owns reconciliation between the batch extracts and the streamed change history.

Which databases Debezium supports and what changes per engine

Debezium lists stable source connectors for MongoDB, MariaDB, MySQL, PostgreSQL, SQL Server, Oracle, Db2 and Cassandra. Stable means the connector is documented as production-ready for the engines it covers, but it does not mean every version, hosting option or configuration behaves the same way. Each connector reads the database's own change log, such as the binary log or write-ahead log, and runs as a connector on Kafka Connect, so its durability comes from Kafka and Kafka Connect.

Log access settings: Most engines need change logging enabled and retained long enough for the connector to catch up after an outage. Confirm the log format, retention period and whether the engine emits full row images.
Privileges: The connector account typically needs replication or log-reading rights plus read access to the captured tables. Ask the database owner for the exact grant set for your engine version.
Managed-service restrictions: Some cloud-hosted databases restrict logical decoding, replication slots or log access. Confirm with the provider whether the settings Debezium needs are exposed on your tier.
Supported versions: Check the connector documentation for the engine versions tested with your Debezium release, and confirm your exact server version before planning.

Before committing, ask the database owner to confirm log retention, the connector account grants and any provider restrictions in writing for each engine in scope.

Where CDC beats batch extraction and where it does not

CDC wins when consumers need changes soon after they happen, when the business needs to see deletes and intermediate updates, and when repeated full-table reads are putting load on the source. A nightly or hourly extract reads whole tables or large date-filtered slices, so it sees only the state at the moment of the read. A row that was inserted and then updated twice between runs appears once, and a deleted row often disappears without a trace unless the extract was built to detect it. Debezium on Kafka Connect reads the database transaction log instead (the binlog in MySQL, the write-ahead log in PostgreSQL), so every insert, update and delete is emitted as an event in commit order. Ask the source team which tables need delete semantics, and confirm with the connector documentation for your database version how deletes and tombstone records are represented before downstream consumers depend on them.

Batch remains the better choice in several common cases. If the source does not expose a transaction log that you are allowed to read, or the database administrators will not enable the required logging settings, CDC cannot start, and a query-based or file-based extract is the realistic option. If change volume is low and the table is small, a scheduled query costs less to build and operate than a connector, a Kafka cluster and the monitoring that goes with them. The same applies when consumers only need a daily snapshot for reporting, reconciliation or a model retrained once a day. Streaming a value that nobody reads until tomorrow adds operating burden without adding value. Before you commit, confirm the log retention window on the source, the expected change rate per table, and which consumers truly need low latency. A batch job that you can explain in one sentence is often the right answer for the table that answers that sentence.

How a CDC rollout runs from pilot to handover

This sequence moves a change data capture feed from source assessment to a production connector that the client's own team can restart, re-snapshot and recover without DeployOpen on call. Each step closes with a named artifact that the client keeps.

Assess each source database and design its privileges
DeployOpen and the client's database administrators inventory every source in scope, including the database engine and version, whether the connector type is stable or incubating for that engine, how binary logging or replication slots are configured, and which tables carry personal or regulated data. The team then designs a dedicated read-only account per source with only the grants the connector needs for snapshots and log reading. The output is a source assessment record and a privilege design per database, reviewed by the database owner before any change is made.
Prove the connector on a non-production copy
A pilot connector runs against a non-production copy of one source, with Debezium deployed to Kafka Connect. The team runs an initial snapshot, measures its effect on the copy, then makes a controlled set of inserts, updates and deletes and checks that each change arrives in the expected topic with the expected key and payload. Results are recorded against the test script. The pilot is complete when the snapshot and the controlled change test both pass and the observed load on the copy is documented.
Set the cluster and security baseline
The team confirms a supported Apache Kafka release running on KRaft, with Java 17 for brokers and Kafka Connect, and enables TLS between clients, brokers and Connect. SASL is configured with a mechanism the client's identity standards accept, and the StandardAuthorizer ACLs are written so each connector principal can read its source topics and write only to its own topics and offset storage. Since resources without ACLs are restricted to super users by default, the team verifies the allow.everyone.if.no.acl.found setting. The output is an ACL map, a TLS and SASL configuration record and a version record, signed off by the client's security team.
Onboard production connectors with lag and offset monitoring
Production connectors are deployed from versioned configuration files with secrets held outside the configuration record. Monitoring is set up for connector and task status, consumer lag, offset progress and error rates, with alerts routed to the client's on-call team. The team confirms that a connector failure produces an alert and that the lag dashboard shows the expected baseline after the initial snapshot completes. The output is a connector configuration record for each source and a monitoring runbook that names the owner of each alert.
Write and rehearse schema change and snapshot runbooks
DeployOpen documents the steps for adding a column, changing a type, renaming a table and dropping a field, including the order of changes on the source and the schema registry or compatibility rules that apply on the topic. A second runbook covers when and how to trigger a new snapshot, for example after a connector has been offline longer than the source retains change logs. Both runbooks are rehearsed on the pilot copy with the client's database and platform staff performing the steps. The output is two tested runbooks with recorded rehearsal results.
Hand over with restart, re-snapshot and recovery procedures
The handover covers the full operating record: the source assessment, privilege design, ACL map, connector configurations, monitoring and alert ownership, and the schema change and snapshot runbooks. The client's operators perform a supervised restart of one connector and a supervised recovery drill, and DeployOpen confirms the steps match what was documented. Support scope after handover is agreed in writing, and any ongoing upgrades or testing run under a separate retainer. The output is a handover pack and a signed acceptance record.

Snapshots, schema changes and source load: the risks that shape the design

Debezium's risks cluster around three moments: the first snapshot, every later change to a source table, and the long life of the connectors and the Kafka platform beneath them. Each card names a decision to make before go-live and a check to run against your own databases, because the actual load and effects depend on your engine, volumes and configuration.

Plan the initial snapshot

When a connector starts for the first time, it typically reads the existing tables before it streams live changes, and that read competes with your production workload. Choose the window, the tables and the read approach with your DBA team, and confirm the expected load on each source engine in a test environment that mirrors production volume. Ask who can pause the snapshot if replication lag or lock contention appears, and what happens to the connector state if it is interrupted.

Govern schema changes

A column rename, type change or dropped field in a source table changes the event records that every consumer reads. Agree a compatibility policy before the first connector goes live: which changes are additive, which need a new topic version, and who signs off. Back each policy with automated tests that replay a representative change against the connector and the downstream consumers, and confirm how your schema registry, if you use one, enforces compatibility.

Monitor connector state and restarts

Kafka Connect manages connector offsets automatically and scales with Kafka's group management protocol, so connector developers do not write offset commits. Operators still own the outcome. Alert on connector and task state, watch lag between the source log position and the emitted events, and rehearse a restart so you know what replays after a failure. Confirm how your team will detect a task stuck in a failed state, since a silent stall can go unnoticed until downstream data is stale.

Control upgrade drift

Kafka 4.x supports only KRaft mode, so ZooKeeper-based clusters must migrate before upgrading, and brokers, Kafka Connect and tools require Java 17. Rolling upgrades are supported from each release in the past year, the last three releases, to the latest version, so a cluster that falls further behind needs a staged path. Record the Kafka, Kafka Connect, Java and Debezium versions per environment, and confirm which Debezium release supports your chosen Kafka line before scheduling each upgrade.

Evidence auditors ask for and which component produces it

This table maps each evidence item an assessor is likely to request for a Debezium on Kafka Connect deployment to the component that produces it. Read it as a collection checklist: DeployOpen prepares and organizes this evidence, but the independent auditor decides whether it is sufficient for the control objective being tested.

Evidence item	What it shows	Component that produces it
ACL definitions and their export from the KRaft metadata log	Which principals may read, write or manage each topic, connector and group, and that no resource is open by default	Kafka StandardAuthorizer, exported with Kafka admin tooling on a scheduled basis
TLS and SASL configuration for brokers and Kafka Connect	Which listeners require encryption and which SASL mechanism each client and worker uses	Broker and Kafka Connect worker configuration files, plus listener settings captured from the running cluster
Connector configuration records and change history	Which tables each connector captures, with what options, and who changed them and when	Kafka Connect REST API connector config, stored under version control or a change management system
Source privilege grants for each CDC account	That the database account used by each connector holds only the read and replication privileges it needs	Source database grant queries run during onboarding and on a recurring review, recorded with the reviewer's name
Offset and lag monitoring logs	That connectors kept running, how far behind the source they were, and whether gaps or restarts occurred	Kafka Connect status endpoints and JMX metrics exported to the monitoring stack, retained according to the agreed log policy
Snapshot run records	When each initial or re-snapshot ran, which tables it covered, and whether it completed	Debezium connector logs and snapshot status fields exposed through the Kafka Connect REST API
Schema change approvals	That changes to captured table structures were reviewed before they reached the stream, and how consumers were told	Change tickets linked to schema registry or topic configuration history; the source database DDL audit trail where enabled
Upgrade records showing the Kafka and Java versions in use	Which Kafka, Kafka Connect, Debezium and Java versions ran in each period, and that they were supported releases	Deployment manifests and change records; the Kafka downloads and upgrade notes used to plan each move, kept with the approval

What buyers ask before hiring help with change data capture

What drives the cost and effort of a change data capture project?

The main drivers are the number of source databases, how often their schemas change, the volume of the initial snapshot, how many consumers depend on the streams, and how much security and disaster recovery scope applies. Each source adds connector configuration, testing and monitoring. Schema churn adds recurring review work, because every change to a table can affect downstream consumers. Snapshot volume determines how long the initial load runs and how much source load it creates. Consumer count drives topic design, access control and support load. Ask for a quote that itemizes these drivers per source system, separates one-time build work from recurring operations, and states which environments are covered. DeployOpen does not publish a price list; a quote follows from the written scope.

Can our team set up change data capture without outside help?

A team with Kafka and database experience can often run one well-understood source on its own, using Debezium on Kafka Connect as documented. The difficulty rises with several database engines, strict compliance scope, and tight change windows on production systems. Kafka Connect manages connector offsets automatically, which removes one common source of error, but connector setup, TLS and ACLs, snapshot planning, monitoring and upgrades still need skilled owners. Assess your team against the specific engines, the security evidence you must produce, and who will be on call. An outside partner can take on the build, documentation and handover so that your staff can run the platform afterward.

How long does a change data capture pilot take?

The timeline depends on the number and type of source databases, the snapshot volume, the security and disaster recovery requirements, and how quickly the data owner and DBA teams can approve access and change windows. No fixed duration applies across these cases, so treat any single number you see as unverified. Ask for a dated plan in the written scope that lists milestones, dependencies on your teams, and the acceptance tests for each source. Most engagements are fixed-scope, which means the schedule is agreed in writing before work begins.

What do we need to provide before a change data capture project starts?

Plan for four things before work begins. First, DBA access to create replication roles, read the transaction log configuration and review connector settings. Second, network paths from Kafka Connect to each source database and to the Kafka brokers, including firewall rules approved by your network team. Third, a named data owner who can decide which tables are in scope, how sensitive fields are handled and which consumers may read them. Fourth, agreed change windows for snapshots, connector restarts and upgrades on production databases. Missing any of these usually delays the pilot more than the technical work does.

Does Debezium work with our managed database service?

Debezium lists stable source connectors for MongoDB, MariaDB, MySQL, PostgreSQL, SQL Server, Oracle, Db2 and Cassandra, plus incubating connectors for other engines. Whether a connector works with a specific managed offering depends on the vendor's support for the replication settings the connector needs, such as access to the transaction log and replication privileges. Confirm this with your database vendor and check the Debezium connector documentation for the exact version and requirements of your engine. Ask us to verify the connector and configuration against your managed service in a short technical review before committing to a scope.

What does a change data capture platform need after go-live?

After go-live, the recurring work includes Kafka and Kafka Connect upgrades, connector version updates, schema change handling, monitoring, and on-call response for lag or failed connectors. Kafka 4.x runs only in KRaft mode, and the project's supported releases are listed on its downloads page, so plan upgrade windows around those releases. Schema changes in source tables need a review process so downstream consumers are not broken silently. Ask any partner for the upgrade path, the on-call model, the runbook for connector failures, and whether a retainer covers ongoing support and upgrades.

How does change data capture affect our source databases?

Log-based connectors read the database transaction log rather than querying tables, which typically lowers read load compared with repeated batch queries. The initial snapshot is the exception, because it reads existing rows and can add significant load on a busy system. Confirm the expected load with your DBA team, run snapshots in agreed change windows, and check retention of transaction logs, since a stopped connector can cause logs to accumulate on the source. Test these effects in a non-production copy before enabling the connector on production.

Who is responsible for security and compliance evidence in a change data capture project?

The organization owns its compliance outcome. DeployOpen prepares systems, controls and evidence for compliance assessments, but it does not issue audit reports, attestations or certifications; independent auditors and certification bodies do. For a streaming pipeline built on Kafka, the evidence typically covers TLS between clients and brokers, SASL authentication, ACLs in the KRaft authorizer, and log retention. Ask your partner which controls they will configure, which evidence they will hand over, and which items your compliance lead must confirm with the assessor.

What to ask an implementation partner about CDC

Start with the source databases, because that is where CDC causes the most damage when it is planned badly. Ask the partner to show, for each database, the exact privileges the Debezium connector user will hold, how it will enable the log that feeds change capture, and when the initial snapshot will run. A partner that proposes one shared administrator account for every source, or that cannot say when the snapshot will read the tables, has not thought about locks, log retention or load on production. Ask for the written snapshot plan per source, including the window agreed with the database owner and what happens if the snapshot is interrupted. Confirm the exact database versions and connector releases against the Debezium connector documentation before scope is signed.

Next, ask how schema changes are tested end to end. A good answer names a test that adds a column, renames a field, changes a type and drops a table on a staging source, then follows each change through the connector, the topic, the schema registry if one is used, and a downstream consumer. Ask how the partner sets ACLs for each topic and consumer group, whether TLS is enforced on every listener by default rather than left optional, and how the allow.everyone.if.no.acl.found setting is kept at its secure value. Ask how Kafka and Kafka Connect upgrades are handled, including the Java 17 requirement for brokers and Connect, the supported upgrade path in the Kafka upgrade notes, and whether connector plugins are re-tested after each change. The answers that should worry you are "we will handle that later" and a plan that treats security as an add-on.

Finally, ask for proof of recovery and handover before any production source is connected. Ask whether the partner has tested restart of Kafka Connect and each connector, and a forced re-snapshot, and whether it measured how long consumers take to catch up. Then ask for the runbooks, the handover list and the evidence list in the written scope, with named owners and the format of each item. Ask the partner to show examples of these documents from earlier work, or to explain why it cannot. A partner that cannot put the runbooks and evidence list in writing before work starts is asking you to accept the risk of an unknown deliverable.

How DeployOpen scopes and runs CDC in your infrastructure

DeployOpen plans, implements, hardens, documents and supports self-hosted open-source streaming in your own cloud accounts, data centers or air-gapped networks. For change data capture, that means agreeing in writing which source databases, topics and consumers are in scope, then building Debezium on Kafka Connect inside your environment and handing over the running pipeline with its runbooks.

How we scope

Inventory the source databases, their versions and the connectors each one needs, such as MySQL, PostgreSQL or SQL Server
Agree the tables, snapshot approach and expected load on each source database before any connector is configured
Define schema change handling, topic naming and which downstream consumers depend on each captured stream

How we build

Deploy Kafka Connect in distributed mode on your Kafka cluster, with each Debezium connector registered through its REST API
Configure source database accounts with the minimum privileges each connector requires, and secure connections with TLS
Set ACLs for connector topics and test snapshot, failover and schema change behavior in a non-production environment first

How we hand over

Deliver connector configurations, runbooks for restarts, offset reset and snapshot re-runs, and an architecture record
Walk your operations and database teams through monitoring of connector lag, task failures and source log retention
Document the evidence your assessors may sample, including access controls, encryption settings and change history

How we support

Offer ongoing support, upgrades and continuous testing as a retainer after the fixed-scope engagement ends
Plan Debezium and Kafka upgrades against the version compatibility you confirm for your environment
Review connector behavior after source database upgrades or new schema changes before they reach production consumers