Connecting Apache Kafka and MongoDB with the MongoDB Kafka Connector
A team that needs data moving between Kafka topics and MongoDB collections should plan for the official MongoDB Kafka Connector running on Kafka Connect workers that you operate. The connector ships as two halves: a source connector that reads MongoDB change streams and publishes change events to Kafka topics, and a sink connector that consumes Kafka topics and writes documents into MongoDB collections. Choose the direction per pipeline before anything else, because it determines which side owns the ordering, the offsets and the failure handling. Confirm the exact connector version, its supported Kafka Connect and MongoDB server versions, and its license terms in the connector's own documentation and repository before you pin it in your plugin image, since this section does not assume any specific release.
The connector does not replace the rest of your platform. It does not create or size the Kafka cluster, which is covered by the broker deployment and sizing sections, and it does not decide how documents should be modeled, indexed or validated in MongoDB. Change streams also depend on the MongoDB deployment: they require a replica set or sharded cluster rather than a standalone server, so a standalone instance is a blocker to resolve early. Connect itself adds a second system to run, with its own worker group, offsets, configuration and upgrade cycle. Ask who owns each of those layers, including which team answers when a connector task fails or when a change stream falls behind the oplog retention window.
What the MongoDB connector needs before configuration starts
The MongoDB Kafka Connector runs on Connect workers, so three systems must line up before any connector is configured: the Kafka and Connect side, the MongoDB deployment and the network path between them. Confirm each item below with the owners of those systems, because a gap in one usually surfaces as a failed connector start rather than a clear error.
Kafka release line and runtime
Confirm which supported Kafka release line your cluster runs and that your Connect workers use Java 17. Since Kafka 4.0, brokers, Kafka Connect and tools require Java 17, so a Connect worker on an older JDK will not start the plugin correctly. Check the supported releases on the downloads page before choosing a target, and confirm that your cluster runs in KRaft mode, since Kafka 4.0 supports only KRaft.
Connect cluster sizing
Ask the platform team how many Connect workers the connectors will run on, how much memory each worker gets, and whether the cluster is shared with other connectors. Sizing depends on the number of tasks, the document size and the throughput of each pipeline, so confirm these numbers with a load test in your own environment rather than assuming a default. Record the worker count and task limits you expect before you write any connector configuration.
Replica set for change streams
The source connector reads MongoDB change streams, and change streams require a replica set or a sharded cluster. A standalone mongod will not provide them, so confirm that the target deployment is a replica set and that the oplog window is long enough to cover the longest expected Connect outage. Ask the database team how the oplog is sized and monitored before the source connector goes live.
MongoDB version and connector compatibility
Confirm the MongoDB server version your deployment runs and check it against the compatibility table for the MongoDB Kafka Connector release you plan to install. Pin the connector version in your plugin image so that a later plugin update cannot change behavior unexpectedly. Also confirm which Kafka version the connector release is tested against, because the Connect runtime and the connector must both be supported.
Database user and role scope
Create a dedicated MongoDB user for each connector, with only the roles it needs. A source connector reads from the source database and its change streams, while a sink connector writes to the target collections, so separate credentials keep the two directions auditable. Ask the database team to confirm the exact role set against the connector documentation and to store the credentials in your secrets manager, not in connector configuration files.
Network and TLS paths
Map every network path between Connect workers, the Kafka brokers and MongoDB nodes, including any load balancer, firewall or VPN. Decide where TLS terminates, confirm that the Connect workers trust the MongoDB certificate chain, and confirm that broker listeners use the TLS and SASL settings your security team approved. A blocked port or an untrusted certificate often looks like a timeout, so test each path from a worker host before you configure connectors.
How to configure the MongoDB source and sink connectors in order
This sequence takes you from a MongoDB replica set and a scoped database user to two running connectors on Kafka Connect, with a verified document path in each direction.
Confirm the replica set and connector user
Your MongoDB deployment must run as a replica set, because the source connector reads change streams and change streams are not available on a standalone server. Ask the database team to confirm the replica set members, the connection string that Connect will use, and a dedicated user with read access to the source database and write access to the sink collections, with no broader roles. Confirm that TLS is on for the connection, since the connector config will reference the certificate chain. Done when a test connection from a Connect worker host succeeds with that user and the replica set reports a primary.
Install the connector plugin on Connect workers
Your platform team installs the MongoDB Kafka Connector plugin on every Connect worker through the plugin path that the workers scan, so each worker loads the same version. Check the connector's release notes and compatibility with your Kafka and Connect version before choosing a build, and confirm the license terms with your legal team. Store the connection string and passwords in your secrets mechanism rather than in connector JSON. Done when the worker's plugin list includes both the MongoDB source and sink connector classes on every worker in the group.
Create the source connector with change stream settings
Create the source connector with a JSON config that sets the connection string, the database and collection to watch, and a topic naming rule so that each collection maps to a predictable Kafka topic. Decide the pipeline filter and whether to include full document updates, since these settings determine what each change event contains. Set the output format for keys and values to match the converters your consumers expect. Done when the connector status reports RUNNING for the connector and its task, and inserting a test document into the watched collection produces a record on the expected topic.
Verify the source path end to end
Insert, update and delete one test document in the watched collection and read the resulting records from the topic with a console consumer. Confirm the key, the payload shape and the operation type match what downstream consumers will parse, and that the record appears in order. Record the change stream resume information the connector stores, so you know where it restarts after a worker restart. Done when all three operations appear on the topic and a worker restart does not replay or drop the test change.
Create the sink connector with its write model
Create the sink connector pointing at the target database and collection, reading from the topic that carries your records. Choose the write model deliberately: insert-only writes suit append-style data, while replace-one or update-one models with a document key suit records that change over time. Define how the sink derives the document identifier so that reprocessing does not create duplicates. Done when the sink status reports RUNNING and the status of its task shows no failures.
Set converters and error handling for the sink
Set the key and value converters so that the sink reads the exact format the topic was written with, and reconcile these with any schema registry you operate. Set the error tolerance and dead letter settings that your team has agreed for records that fail validation or fail to write. Confirm that a deliberately malformed test record is routed as configured rather than stopping the task. Done when the malformed record appears where you expect and valid records keep flowing.
Test the sink path and the round trip
Produce a test record to the source topic and confirm that it lands in the target collection with the expected document identifier and fields. Then run a round trip: write a document into the source collection, confirm it reaches the sink target, and compare the two documents field by field. Keep the test documents and the connector JSON under version control so that changes are reviewed. Done when the round trip matches and both connector statuses stay RUNNING through the test.
Choosing change stream scope and sink write models
The first decision is how much of MongoDB the source connector watches: one collection, every collection in a database, or the whole deployment. Change streams read the replica set oplog, so the scope choice sits on top of oplog behavior. The oplog is capped, and once a resume token points to an event that has aged out, the stream cannot continue from that position and the pipeline needs a new snapshot. Before you choose a scope, confirm the oplog size and retention window your deployment provides under peak write load, and ask the team that owns the cluster how long a Connect outage it must survive. Narrowing with database, collection and a pipeline filter reduces the events Connect must process, but it does not extend how long the oplog keeps them. Update events need a decision too. By default they carry only the changed fields, so set change.stream.full.document if consumers need the whole document, knowing the lookup returns the current state, which may already include later writes.
Sink write models decide what happens when a record arrives twice, which Connect can cause after a worker restart or a failed offset commit. An insert strategy fails on the second delivery if the _id already exists, so the task either stops or routes the record to error handling. A replace strategy keyed on a business key or _id produces the same result on each delivery, which makes it a common choice for idempotent sinks. Update strategies that set fields are also repeatable for the same values, but they are order sensitive, and an upsert can recreate a document whose delete arrived earlier in the stream. The document ID source matters as much as the write model. If the connector generates a new ObjectId for each delivery, redelivery creates duplicates regardless of strategy, so derive the _id from the Kafka record key or a stable field. Confirm the exact write model class names and delete handling keys against the connector version you deploy.
Settle these choices in writing before the build, then test them. Replay one topic partition into a test collection, restart a worker mid-batch, and compare document counts and contents. Check that a delete followed by a late update does not bring a record back under your chosen strategy. Record the oplog retention figure you confirmed next to the scope decision, because the recovery plan for an expired resume token depends on it. Ask for these tests explicitly in any fixed-scope quote so they are priced and scheduled with the build.
How MongoDB documents map to Kafka records and back
This table compares how a MongoDB document, its _id, and its change event envelope become Kafka topic names, record keys, and record values on the source side, and how sink settings turn Kafka records back into documents. Read each row against the connector release you plan to run, because field naming, envelope shape, and converter behavior are the settings most likely to change between versions.
| Element | Source connector (MongoDB to Kafka) | Sink connector (Kafka to MongoDB) |
|---|---|---|
| Topic name | Derived from database and collection names through the topic naming settings, with an optional prefix and a configurable separator | Read from the topic list or topic regex in the sink configuration, so the topic name is an input rather than an output |
| Record key | Built from the document _id by default, and can be shaped with a key output format or a key projection; confirm the key format options for your release | Used by the sink to build the document identifier when the document write model relies on it, so key choice directly affects upserts and deletes |
| Document _id | Carried through into the change event document key, which keeps the original identifier for downstream consumers | Kept as the primary key of the target document; if the record key and value disagree, the sink write model decides which value wins |
| Change event envelope | Change stream events arrive with fields such as operation type, full document, namespace, and resume token; confirm which fields are emitted for your release and settings | Not produced on the sink side unless a transform or write model reconstructs the document from the envelope, so the envelope must be unwrapped deliberately |
| Full document on updates | Update events may carry only the changed fields unless full document lookup is enabled, so downstream consumers can see partial data | Partial update records must be applied with an update write model rather than a full replace, or fields will be lost |
| Converters (JSON or Avro) | Key and value converters determine serialized form; JSON handling of BSON types such as dates, decimals, and binary depends on the converter and output format settings | Input converters must match the producer format exactly; a schema registry adds a dependency that must be provisioned and secured |
| Schema and type conversion | BSON types with no direct equivalent may be converted to strings or nested structures, so numeric precision and timestamps can change shape | Schema-less JSON loses type information that Avro or Protobuf would keep, so numeric and date fields may need explicit casting in the sink |
| Deletes and tombstones | Delete events are emitted as change events, and downstream compacted topics depend on tombstone handling that you must confirm | Delete handling depends on the write model and delete-on-null settings, so a null value may or may not remove the target document |
Securing the connector between Kafka and MongoDB
The connector sits between two systems that each hold sensitive data, so its security depends on the links between them. The cards below cover the transport, credentials, database roles and Kafka topic permissions that each need an explicit decision before the first connector starts.
TLS on the MongoDB connection
Encrypt the connection from Connect workers to MongoDB so that credentials and documents never cross the network in clear text. Confirm that the MongoDB deployment presents a certificate that the worker's trust store accepts, and that the connection string enables TLS explicitly rather than relying on a default. Failure mode: a connection string without TLS works in testing and then sends production documents unencrypted across any network segment the workers share with other hosts.
Encrypted and authenticated Connect-to-Kafka traffic
Choose SSL or one of the documented SASL mechanisms for the link between Connect workers and the Kafka brokers, and disable unsecured listeners for this path. The security overview lists GSSAPI, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER as SASL options, and mixing secured and unsecured listeners is allowed, so confirm that the worker's listener is the secured one. Failure mode: a worker configured against a plaintext listener can read and write the data topics without any identity attached to the traffic.
Least-privilege MongoDB users for source and sink
Create separate MongoDB users for the source connector and the sink connector, and grant each only the roles its job requires. The source reads change streams from the collections it watches, while the sink needs write access only to the target collections it populates. Failure mode: one shared administrative user used by both connectors can drop collections or change indexes if a connector configuration is misdirected, which turns a pipeline bug into a database incident.
Connection string secrets outside configuration
Keep MongoDB connection strings and Kafka client credentials out of connector JSON, worker configuration files and version control. Use the secret handling supported by your Connect deployment, such as externalized configuration providers, and confirm that the value is not echoed in the REST API responses or in worker logs. Failure mode: a connector configuration pasted into a ticket or a repository exposes the database password, and rotating it then requires updating every connector that uses it.
ACLs on Connect internal and data topics
In a KRaft cluster, the StandardAuthorizer stores ACLs in the metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Grant the Connect worker principal the access it needs to its internal configuration, offset and status topics, and grant the source and sink principals only the topics they read or write. Failure mode: a sink principal with broad write access to the Connect internal topics can corrupt the offset and config state that every connector on the cluster depends on.
Principal separation across connectors
Run each connector under its own Kafka principal and its own MongoDB user, so that a compromised or misconfigured pipeline has access only to its own topics and collections. Review the ACL list for overlap when a second connector is added to the same worker group. Failure mode: connectors sharing one principal make audit logs unable to show which pipeline wrote a given record, and revoking access for one pipeline also breaks the others.
Common MongoDB connector failures and how to fix them
This table lists the failures teams most often hit when running the MongoDB Kafka Connector source and sink connectors, with the likely cause and the first fix to try. Read each row as a triage starting point, then confirm the behavior against the connector and MongoDB versions you run, since settings and error messages vary between releases.
| Failure | Likely cause | First fix to try |
|---|---|---|
| Source connector fails to start on MongoDB | The source uses change streams, which require a replica set or sharded cluster. A standalone mongod does not provide the oplog that change streams read. | Confirm the target deployment is a replica set (a single-node replica set is acceptable for test work), then restart the connector and check the task status. |
| Source resumes from the wrong point after a restart | The stored resume token or Connect offset is missing, stale, or no longer valid because the oplog rolled over or the collection was dropped and recreated. | Check the connector offset topic and the oplog window on MongoDB. If the token has expired, restart from the current time or a known position and plan a backfill for the gap. |
| Duplicate documents appear in the sink after retries | Retries or task restarts re-deliver records, and the sink write model inserts rather than replaces. Without a stable document key, each delivery creates a new document. | Configure the sink with a replace or update write model keyed on a stable field such as the Kafka record key, so re-delivered records overwrite the same document. |
| Deserialization errors on sink or source records | The configured key or value converter does not match the bytes on the topic, for example JSON with schemas on one side and plain JSON or Avro on the other. | Compare the producer's serializer with the connector's key.converter and value.converter settings, then align them. Use a dead letter queue for bad records while you fix the mismatch. |
| Authentication failure when connecting to MongoDB | The connection string names the wrong authentication database, the user lacks roles on the target database, or credentials are not passed through the worker configuration. | Verify the user, password and authSource with a MongoDB shell client from the worker host, then confirm the user holds the read or write roles the connector needs. |
| TLS handshake error between the worker and MongoDB | The worker does not trust the MongoDB server certificate chain, the hostname does not match the certificate, or the worker is using a different TLS setting than the cluster requires. | Import the CA chain into the truststore the worker JVM uses, confirm the certificate names match the hosts in the connection string, and test the same connection string from the worker host. |
| Connector plugin not found on workers | The connector JAR is not under a directory listed in plugin.path, the worker was not restarted after installation, or the plugin was installed on some workers but not all. | Check plugin.path on every worker, confirm the connector directory is present on each one, restart the workers, and query the Connect REST plugin list to confirm the class is visible. |
| Sink or source task fails after a worker restart with no data loss signal | The connector is configured without a clear error policy, so the task stops on a single failure and the rest of the pipeline stalls unnoticed. | Check task status through the Connect REST API, review worker logs for the first exception, and decide on error handling settings before restarting the task. |
Questions teams ask before building the MongoDB connector
What drives the effort and cost of building the MongoDB connector in-house versus with a partner?
The main drivers are scope, environments, write semantics, testing and operations. Scope covers how many collections, topics and directions (source, sink or both) the pipeline needs. Environments cover how many Kafka, Connect and MongoDB stages must be built and kept in step. Write semantics decide how the sink handles inserts, upserts, updates and deletes, and how it treats duplicate or out-of-order events. Testing covers replay, failure injection and schema change tests. Operations covers monitoring, credential rotation, plugin upgrades and on-call. An in-house team pays for these with its own staff time. A partner quote should itemize each of these areas so you can see what is included and what you would still own.
How long does a scoped MongoDB connector build usually take?
Duration depends on the scope, the number of environments and how many data flows are involved, so no fixed figure applies. Ask any provider, including DeployOpen, to state the duration in the written scope along with the assumptions behind it, such as how many collections are in play, whether change streams need new replica set configuration, and who approves the test data. Timelines should also name the milestones where the pipeline is first verified end to end and where it is handed over for operation. Treat any timeline that is not tied to a written scope as a guess.
What does the work need from our team before it starts?
The build needs access and information in four areas. For MongoDB, you need a replica set (change streams do not run on a standalone server) and a dedicated user with the roles that the source and sink require, granted on the databases in scope. For Kafka, you need credentials and ACLs that let Connect create or use the topics, internal topics and consumer groups it needs, plus access to the Connect worker environment. You also need representative data samples with their schemas, including nested documents and fields that change type over time. Finally, you need named owners for the data, the target collections and the approval of test results. Confirm the exact role names and the minimum permissions in the MongoDB documentation for your server version.
Can our team build and run the MongoDB connector alone?
A team with working Kafka Connect and MongoDB experience can build a simple one-way pipeline alone. The harder parts are the write model decisions, the handling of failed records and schema drift, the security review of stored credentials, and the upgrade path for the connector and Connect workers. These areas need someone who has tested failure cases, not only a working happy path. A useful self-check is whether your team can replay a window of events, show that no record was lost or duplicated, and explain how the pipeline behaves when MongoDB or Kafka is unavailable. If any of these are unclear, a scoped partner engagement can cover the gap and leave your team with runbooks.
Is the MongoDB Kafka Connector open source, and what license applies?
The MongoDB Kafka Connector is published by MongoDB, and you should confirm its current license directly in the project repository before you depend on it, since licensing terms can change between releases. Check the license file and the package metadata for the version you plan to deploy, and record the result in your third-party software inventory. Also check the licenses of any other connector plugins in the same Connect worker, because each plugin carries its own terms. Licensing review is part of the scope of any build, and a provider should be able to show you the license check as a deliverable.
What breaks most often in MongoDB and Kafka pipelines?
The most common problems are structural rather than network failures. Change streams need a replica set and resume tokens that must be stored and kept within the oplog retention window, so a long outage can make a resume impossible. Sinks that use the wrong write model can create duplicates or overwrite newer documents. Schema drift in nested documents can break converters or downstream consumers. Connector credentials stored in plain configuration can leak through logs or dashboards. Each of these should be covered by a test in the build, with a written decision on how the pipeline recovers, and a runbook for operators.
Which security controls should a MongoDB connector build include?
A build should cover encrypted connections between Connect, Kafka and MongoDB, least-privilege MongoDB roles for the source and sink users, Kafka ACLs limited to the topics and groups the connector needs, and secrets management so credentials are not stored in plain text in connector configuration. Kafka itself supports SSL/TLS and SASL mechanisms, and the Kafka security overview describes how these are configured. Access to the Connect REST interface and the worker configuration should also be restricted. Document which controls are in place and who reviews them, because an assessor will ask for that evidence rather than a description.
Testing the pipeline and running it after go-live
Before go-live, the team should prove the pipeline with replayed sample documents in both directions. For the source connector, insert a known set of documents into a test collection on a replica set, confirm each one appears on the target topic with the expected key and payload, and check that change events for updates and deletes arrive in the order the database produced them. For the sink connector, produce a known set of records to a test topic, including a deliberate malformed record, and confirm the documents land in the target collection with the write model you chose. Compare document counts and sample field values, not just totals, because a count match can hide a mapped field that was silently dropped. Repeat the sink test with the same records twice to see whether the write model produces duplicates or converges to one document per key.
Next, test failure behavior on purpose. Stop the Connect worker during a batch, restart it, and confirm that the source resumes from its stored offset or resume token without a gap, and that the sink does not write a second copy of records it already committed. Stop and restart the MongoDB replica set member or change the primary during a load test to confirm the connector reconnects. Check that duplicate handling and ordering match what your consumers assume: a sink with a unique key tolerates replays, while a sink that appends does not. Read the connector and task status through the Connect REST API after each test and alert on any task in a FAILED state, since a failed task does not restart itself unless your restart policy does so. Record the expected behavior for each test so the runbook has an acceptance baseline.
After go-live, pin the exact connector version and the Kafka Connect runtime version in your build, and treat each upgrade as a tested change. Kafka's release plan says rolling upgrades are supported from each release in the past year (the last three releases) to the latest version, and the downloads page lists 4.3.1, 4.2.2 and 4.1.2 as supported at the time of writing, so confirm the current list before choosing a target. Upgrade a staging cluster first, replay the test set above on the new versions, and only then roll production workers one at a time. Check the MongoDB connector's release notes and compatibility matrix for the Kafka Connect version it was built against, because a connector pinned to an older plugin can fail after a Connect upgrade. Keep the previous plugin artifact available so you can roll back the connector without rebuilding the image.
How DeployOpen scopes and delivers the MongoDB connector
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure, including the Kafka and MongoDB connector work described on this page. Engagements start from a written scope with agreed deliverables and timelines, so the connector design, testing and handover are settled before build work begins.
How we scope
- Written scope lists the MongoDB clusters, replica set topology and Kafka topics in each data flow
- Source and sink directions are defined, with the change stream or write model chosen per collection
- Converter, schema and error handling choices are recorded as decisions before any build starts
- Quotes follow from the written scope, and engagements are agreed as fixed scope
How we build
- Connect workers are deployed in your infrastructure with the MongoDB Kafka Connector plugin installed from a trusted source
- Source connectors are configured against a replica set, with resume tokens and topic mapping tested
- Sink connectors are tested with the chosen write model, idempotent keys and duplicate handling
- Credentials, TLS and database roles are configured to match your access model and security policy
How we hand over
- Connector configuration, topic maps and converter settings are documented as versioned files
- Test results cover normal flow, malformed records, restart behavior and replica set failover
- Runbooks describe connector status checks, restarts, offset handling and plugin upgrades
- Your team receives knowledge transfer on the deployed design and the decisions behind it
How we support
- Ongoing support, upgrades and continuous testing are available as retainers after handover
- Connector and plugin upgrades are planned against the Kafka and MongoDB versions you run
- Retainer scope and response expectations are written down before support begins
- Confirm with DeployOpen which MongoDB and Kafka versions are in scope for your engagement
