Why Kafka Connect deployment needs its own plan
Kafka Connect in distributed mode is the supported way to run database, object storage and SaaS pipelines as a managed group of workers, and it deserves its own plan separate from the broker cluster. When several teams need data moving between Kafka and outside systems, the decision is not only whether to run Connect but who owns its workers, configuration and connector lifecycle. Connect runs as its own process group with its own configuration files, plugin path and REST API, so it fails, scales, secures and upgrades on a different timetable from the brokers. Since Kafka 4.0, Kafka Connect requires Java 17, so any host still on an older runtime needs a change before the group moves to 4.x. MirrorMaker 2 is built on Kafka Connect, so the same worker discipline carries over to cross-cluster replication.
The first hard part is plugin version coupling. Connector plugins are jars placed on the worker plugin path, and each is built against a specific Kafka Connect API and often a specific client library. A connector that loads on one worker version may fail after an upgrade, and mixed plugin versions across workers in the same group can produce behavior that is hard to reproduce. Treat the plugin set as a versioned artifact. Record each connector's version, its source, the Kafka release it was tested against and a checksum for every jar. Confirm with each connector vendor or project which Kafka releases it supports before you change the group.
The second is that each connector fails differently. A database source connector can stall after a schema change or lose its position if the source log is purged before it reads, while an object storage sink can write partial files or duplicate records when a task restarts. Connect offers error tolerance and dead letter queue settings, but the right choice depends on the connector and the downstream system. Decide per connector which errors are retried, which are routed aside and which stop the task. Write down who is paged for a failed task and what the recovery step is before the first pipeline goes live.
The third is credential sprawl. Each connector needs credentials for its source or sink, and those values often end up in connector configurations submitted through the REST API, in worker files or in the group's internal topics. Kafka authenticates clients and tools with SSL or SASL, so the Connect worker needs its own principal with ACLs scoped to the topics it reads, writes and uses internally. The REST API also needs authentication and encryption before it is reachable from anywhere beyond the management network. Inventory every secret, decide where each one lives and agree with your security team how rotation will happen without redeploying workers.
The fourth is upgrade compatibility. Rolling upgrades are supported from each release in the last three releases to the latest version, but a Connect group also depends on its connector plugins, the Java 17 runtime and any custom converters. Before scheduling a move, list every connector in use, confirm its compatibility with the target release and run a full pipeline on a non-production group. Check the Kafka downloads page for the currently supported releases, since that list changes as new releases ship and archived versions no longer receive bugfixes.
Workers, connectors, tasks and the REST API that controls them
A Connect cluster is a set of worker processes that share a group identifier, store their state in Kafka topics, load connector plugins from a plugin path, and expose a REST API for lifecycle control. Each card below is a component you must understand and configure deliberately before you build anything on top of it.
Distributed workers and the group identifier
Workers started with the same group identifier form one Connect cluster and share connector and task work between them. Any worker that joins or leaves triggers a rebalance, so the operator must decide how many workers run, where they run, and how they are replaced during maintenance. Check that every worker in a cluster carries the same group identifier and that no two clusters accidentally share one.
Internal topics for configuration, offsets and status
Distributed mode stores connector configuration, source connector offsets and connector and task status in internal Kafka topics rather than on local disk. These topics are part of your cluster's durable state, so their names, partition counts, replication and retention must be reviewed before go-live. Confirm the exact topic names and replication settings for your target Kafka version in the Kafka 4.3 documentation, and make sure the Connect cluster's own replication settings match your broker fault tolerance.
Connector plugins and the plugin path
Connector plugins are JARs for sources, sinks, converters and transformations that workers load from the directories listed in the plugin path. Every worker must see the same plugin set at the same versions, or a connector will start on one node and fail on another. Plan a controlled process for adding, pinning and removing plugins, and record the plugin version alongside each connector so that upgrades can be tested against known builds.
Connectors and the tasks they split into
A connector is the logical unit you create and configure, and it divides its work into tasks that run across workers. The maximum task count sets an upper bound on parallelism, but the number of tasks that actually run depends on the connector, the source or sink, and the partitions or objects it reads. Review task distribution after each rebalance, and size the worker count and task limits together so that no worker is idle or overloaded.
Converters and the serialization format
Converters translate records between Kafka's byte format and the structure a connector expects, so they determine how keys and values are serialized. Key and value converters can differ, and a mismatch with downstream consumers or schema registries causes data that appears valid but cannot be read. Set converters explicitly in the worker and in each connector configuration, and agree on the format with every team that consumes the topic.
The REST API for lifecycle control
The Connect REST API creates, updates, pauses, resumes, restarts and deletes connectors, and returns the status of connectors and their tasks. It is the operational interface most teams use for automation and incident response, so access to it is effectively administrative control over your data pipelines. Decide which listener the API binds to, how it is authenticated and encrypted, and which people or automation accounts may call it, before connectors reach production.
Which Connect topology and connector sourcing fit your integrations
This table compares the main Kafka Connect topology and connector sourcing choices and the operational cost each one carries. Read each row for the failure, upgrade or pipeline burden it adds, then match the option to your teams, security zones and source systems. The main takeaway is that sharing workers and sourcing custom plugins save setup effort but couple failures and upgrades across teams.
| Option | When it fits | Operational cost |
|---|---|---|
| One shared Connect cluster | Few teams, similar data sensitivity, and a single platform owner who can coordinate restarts and plugin changes | A failing connector or worker can affect every team's pipelines, and one plugin upgrade touches all workloads at once |
| Separate worker groups per domain or zone | Teams with different owners, change calendars or regulated data that must stay within a defined network segment | More clusters to size, patch, monitor and secure, and more REST API endpoints to protect |
| Upstream connector plugins | Common sources and sinks where the Apache Kafka project or the connector's own community publishes maintained plugins | Limited to the features and support terms of the upstream release, so confirm the plugin's license and maintenance status before adoption |
| Vendor-supplied plugins | A specific commercial system where the vendor certifies the connector against your Kafka and Connect versions | Licensing terms, vendor release timing and support scope set upgrade pace, so confirm them in writing before rollout |
| Custom-built connectors | A proprietary source or sink with no maintained plugin, and a team willing to own the code | Build pipelines, tests, version pinning and ongoing fixes for each Kafka upgrade become a permanent internal responsibility |
| Self-managed workers on virtual machines | Teams with established configuration management, fixed hosts and direct control over network paths to databases | Host patching, JVM upgrades and rolling restarts are handled by your team, and scaling requires provisioning new machines |
| Containers on an orchestrator | Teams already running an orchestrator that can handle rolling restarts, resource limits and secrets injection | Image build pipelines, orchestrator expertise and careful handling of plugin paths and storage add operational work |
| CDC-style source connectors from production databases | Near-real-time change capture where the database team accepts log reading and replication privileges | Requires database log retention, replication slot or log monitoring, and coordination with DBAs during schema changes and failovers |
| Batch extraction from source systems | Reporting or lake loads where some latency is acceptable and the source team wants minimal load on production | Missed windows, duplicate handling on retries and full-reload planning fall to your team, so confirm extraction windows with the source owner |
Kafka Connect deployment steps from workers to first connector
This sequence takes a team from an empty environment to a working distributed Kafka Connect cluster running one source and one sink connector. Each step ends with a check the team can observe, so problems surface before any production data moves.
Confirm Java and Kafka versions
The platform team confirms that every Connect worker runs Java 17, the runtime Kafka's upgrade notes require for Connect since Kafka 4.0, and that the brokers run the Kafka release chosen for the rollout. Check the supported releases on the downloads page and confirm the exact requirements for that release in its upgrade notes. The step closes when the Java version and the broker version are recorded in the runbook and match the target release on every host.
Create internal topics and broker access
The Kafka administrator creates the topics Connect uses for its own state: configuration, offsets and status. Confirm the replication factor and partition settings for these topics against the documentation for the chosen release, and grant the Connect service principal the ACLs it needs to read and write them and the connector topics. The check is a successful topic describe on each internal topic and a test produce and consume using the Connect principal.
Configure distributed workers
The engineer sets up each worker in distributed mode with the same group identifier, the same internal topic names and a plugin path that holds only the connector plugins the team has approved. Workers that share a group identifier join one cluster, so a mismatch shows up as a separate group. Start the workers and call the REST API root and the worker list endpoint. The check is that every worker appears in the group with the expected version and that the plugin path is visible in the worker's connector plugin list.
Add TLS and SASL for worker traffic
The security engineer enables SSL/TLS and a SASL mechanism for worker-to-broker traffic, using the client settings prefixed for the Connect producer and consumer. Pick a mechanism that matches the cluster's identity model, such as SCRAM or OAUTHBEARER, and confirm the setting names for the chosen release. Test with a single worker first. The check is that the worker reaches the broker over the secured listener, with no authentication errors in its log and the expected principal visible in the broker audit trail.
Deploy and verify connector plugins
The engineer copies each approved connector plugin into the plugin path on every worker, using the same version on all hosts. A rolling restart loads the plugins, and the team then queries the connector plugins endpoint through the REST API to confirm each class is listed. Any plugin that fails to load usually shows a missing dependency or a Java version mismatch in the worker log. The check is that every expected plugin appears in the list on every worker.
Externalize credentials from connector JSON
The engineer moves database passwords, storage keys and API tokens out of connector JSON and into a secrets source read through a Connect config provider, such as the file provider, with references in the connector configuration. Confirm the provider class names and the allowed-path settings for the release in use. Store the secrets with restricted file permissions or in the organization's secret manager. The check is that a connector config fetched through the REST API shows the references rather than the secret values.
Configure error handling and dead letter topic
The engineer sets the error tolerance for each connector and, for sink connectors, a dead letter topic that receives records failing conversion or transformation. Choose tolerance settings deliberately, because a permissive setting can hide bad data. Confirm in the release documentation which error settings apply to source and sink connectors, since dead letter support is documented for sink connectors. The check is a deliberately malformed test record that lands in the dead letter topic with its error headers, and the connector stays in the RUNNING state.
Run first connectors and test restarts
The team deploys one low-risk source connector and one low-risk sink connector in a non-production environment, using test data. After both are running, stop one worker and let the cluster rebalance. Then call the connector status endpoint and check the task states and assigned workers. For the sink, compare the consumer group offsets for the connector against the topic end offsets, and for the source, confirm that new records appear once after the restart. The step closes when tasks are redistributed, no records are duplicated or lost in the test, and the results are documented as the baseline for production.
What drives Kafka Connect deployment effort and cost
This table lists the factors that change how much work a Kafka Connect deployment takes, who typically pays for that work, and whether the cost is one-time or recurring. Use it to scope internal versus partner work and to frame the questions you ask in a quote. The main takeaway is that connector count, custom code and security requirements usually matter more than the Connect cluster itself.
| Driver | Why it changes effort or cost | How to keep it down |
|---|---|---|
| Number of source and sink systems | Each database, object store or SaaS endpoint needs its own connector configuration, credentials, network path and failure testing. Effort grows with each distinct system, not with each topic. | Standardize on a short list of supported connectors. Confirm the vendor and version of each target before scoping. Retire systems that no longer need a pipeline. |
| Connectors needing custom code or an unsupported plugin | Custom connectors require development, unit and integration tests, packaging and a plan to rebuild them on each Connect or Java upgrade. Plugin vendors may charge licenses or support fees, and that cost recurs. | Prefer maintained connectors with a clear license. Ask whether a plugin vendor offers support terms. Budget for custom code to be retested on every Kafka upgrade. |
| Volume and topic count per connector | Higher throughput and more topics raise task counts, worker memory, error rates and the time needed for load and failover testing. Sizing is usually confirmed with load tests, not estimates. | Measure real throughput in a pre-production run. Group low-volume flows onto shared connectors where the design allows. Ask the partner to show the sizing method. |
| Number of environments and separate worker groups | Each environment and each isolated worker group needs its own configuration, secrets, monitoring and upgrade cycle. Separate groups are often needed for data-classification or team boundaries. | Use one configuration template with environment variables. Justify each extra worker group in writing. Automate deployment so that environments stay consistent. |
| Security requirements such as mutual TLS, Kerberos or OAuth | Kafka supports SSL and SASL mechanisms including GSSAPI (Kerberos) and OAUTHBEARER. Connectors that reach databases or SaaS APIs need their own credential handling, certificate rotation and secret storage. Setup is largely one-time, rotation recurs. | Choose one authentication model per connector class. Automate certificate and secret rotation. Confirm which auth methods each connector supports before design. |
| Compliance evidence required | Assessors may ask for configuration baselines, access reviews of the REST API, change records and proof of error handling and monitoring. Preparing this evidence is internal or partner work, and auditor fees sit outside DeployOpen's scope. | Agree the control list before build. Keep configuration in version control so changes are traceable. Ask your auditor which evidence format they accept. |
| Number of teams creating connectors through the REST API | Each team that can create or change connectors adds review load, access control design and risk of misconfiguration. Access to the REST API must be restricted and logged to prevent unapproved data flows. | Limit REST API access to a small group with a change process. Provide approved connector templates. Review connector inventory on a fixed schedule. |
How Connect deployment differs on virtual machines, containers and air-gapped networks
A single-node worker is the simplest way to exercise a connector before anything reaches shared infrastructure. It usually runs in standard mode on one host, with a small set of internal topics, and it is the place to confirm that a connector class loads, that its configuration validates and that records flow end to end. Because nothing is shared, the controls that matter in production are often relaxed here, which is exactly why teams must not copy this setup forward unchanged.
- Plugins: Connector JARs are usually copied straight into a local plugin path. Nobody reviews them formally, so the team should record the exact artifact version and checksum they tested.
- Scaling and replacement: Replacing the worker means stopping it and starting a new one. There is no rebalancing to validate, and offsets and configuration must be kept so the test can be repeated.
- Secrets: Credentials are often typed into local property files. Use throwaway credentials and never reuse production secrets in this environment.
- TLS certificates: Plaintext listeners or self-signed certificates are common. Note which connections were not encrypted so they are rebuilt properly for later stages.
- Connector installs: Installs happen from a developer machine with internet access, which hides any dependency that the production network cannot reach.
Before promoting a connector, confirm that the artifact, its transitive dependencies and its configuration are recorded so the same build can be reproduced in the next environment.
A distributed group runs several workers on virtual machines that share group ID and internal topics, so tasks are spread and rebalanced across them. This is the model most teams use for production database, object storage and SaaS integrations. Plugin delivery, certificate handling and worker replacement must be designed as repeatable operations, because each change touches every node in the group.
- Plugins: Plugin directories must be identical on every worker. Configuration management or an image pipeline should place the same reviewed artifacts on each host, and a checksum check should run at startup or in the deployment job.
- Scaling and replacement: New workers join the group and tasks rebalance. Replace nodes one at a time and watch rebalancing and task status before continuing, so the group keeps enough capacity during the change.
- Secrets: Store credentials in a secrets manager or a restricted vault and render them into worker configuration at deploy time. Avoid putting them in shared templates or version control.
- TLS certificates: Rotate certificates for the REST API and broker connections with a runbook that updates the truststore and keystore on each node, then restarts workers in sequence. Confirm that all workers present the new certificate before finishing.
- Connector installs: Installs are made through the REST API or configuration management. Confirm that every worker can resolve the plugin, since a connector that loads on one node and not another causes intermittent task failures.
Before going live, confirm how the team will detect a worker that is running an outdated plugin set, and who approves each artifact change.
On containers or an orchestrator, workers are defined as images and deployment specifications rather than as long-lived hosts. Plugins are usually baked into images or mounted from a controlled volume, and replacement happens by rolling out new pods or containers. The same distributed group rules apply, but image provenance, secret injection and certificate rotation are handled by the platform, so the team must confirm exactly which of these the platform provides.
- Plugins: Build a dedicated image that contains only approved connector artifacts. Tag images with the artifact versions and keep the build record so an image can be traced back to its sources.
- Scaling and replacement: Scaling changes the replica count, and rolling updates replace workers. Set disruption limits so the group keeps enough workers during node maintenance.
- Secrets: Inject credentials through the orchestrator's secret mechanism or an external secrets operator. Confirm whether secrets appear in environment variables, which can be exposed through inspection tools, or as mounted files.
- TLS certificates: Use a certificate manager or the platform's rotation tooling. Confirm whether workers reload certificates without a restart, because many Connect setups still need a rolling restart to pick up new keystores.
- Connector installs: Connectors are installed by changing the image or plugin volume, then calling the REST API to create or update connector configurations. Keep connector definitions in version control and apply them through a pipeline.
Confirm with your platform team which parts of secret handling and certificate rotation the platform supports natively, and which the Connect deployment must handle itself.
On an air-gapped network, workers cannot download plugins, dependency packages or container base images from public sources at deploy time. Every artifact must be approved, transferred through the approved channel and mirrored into an internal repository before a worker can use it. This turns connector installation into a supply chain process, and it applies to Kafka Connect plugins, their transitive dependencies and any image layers that include them.
- Plugins: Approved plugin artifacts must be mirrored into an internal artifact repository, with records of who approved each version, its checksum and the source it came from. Workers should be configured to read only from that internal location.
- Scaling and replacement: New workers need the same mirrored artifacts and configuration. Replacement is slower when every image or package must travel through a transfer process, so plan replacement capacity in advance.
- Secrets: Use the internal secrets infrastructure, since external secret services are usually unreachable. Confirm how credentials are rotated without internet access.
- TLS certificates: Certificates must come from an internal certificate authority. Confirm how revocation is checked and how rotated certificates are distributed to every worker.
- Connector installs: Installs use the internal mirror only. Test every connector in a staging enclave against the mirrored set, so a missing dependency is found before it reaches production.
Before approving a connector for this network, confirm that every transitive dependency is present in the mirror and that the approval record names the artifact version being deployed.
Securing the Kafka Connect REST API and worker credentials
Kafka Connect exposes more than the brokers do: a REST API that changes what data moves where, and worker credentials that reach your topics. The cards below cover each access surface, how Kafka's default ACL posture affects connectors, and what to confirm in your Connect version before go-live.
REST API as a control plane
The Connect REST API can create, modify, pause, restart and delete connectors, so anyone who reaches it can redirect data or drop a pipeline. Place the listener behind TLS and an authentication layer you configure yourself, and restrict network access to the operators and automation that need it. Check which listener settings and authentication options your Connect version supports, and test that unauthenticated requests are rejected before any connector is deployed.
Worker credentials to brokers
Each Connect worker authenticates to the brokers with its own principal, and that principal is what the authorizer checks on every topic read and write. Scope it with ACLs to only the topics, consumer groups and internal Connect topics that the workers need. Avoid sharing one broad principal across unrelated connectors, because a single compromised worker would then reach every pipeline.
Default ACL posture
In KRaft clusters, the StandardAuthorizer stores ACLs in the metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Confirm that this setting is false in your environment, then grant each connector principal explicit access. Connectors that fail with authorization errors after go-live usually point to a missing grant rather than a broken connector.
Connector secrets from an external store
Database passwords, cloud storage keys and SaaS tokens should not sit in connector JSON or in plain worker properties files. Use externalized configuration so that the worker resolves secret references from an external secrets store at runtime. Ask which config providers your Connect version and plugins support, and confirm who can read the secrets store and how rotation reaches running connectors.
Transport security and SASL mechanisms
The Kafka security overview lists SSL for encryption and the SASL mechanisms GSSAPI, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER for authentication. Confirm which of these applies to the Connect REST listener and to worker-to-broker connections in your version, because the REST listener and the Kafka client settings are configured separately. Choose the mechanism your identity systems can support, and keep plaintext listeners out of the path.
Verifying access before go-live
Build a short test plan that covers each access surface: a rejected REST call without credentials, a worker denied access to a topic outside its grant, and a connector that starts only with resolved secrets. Record the principals, ACL entries and config provider settings used in the test so the configuration can be reproduced after an upgrade or a worker rebuild. Re-run the checks whenever a connector, plugin or Connect version changes.
How to test that connectors deliver data and fail safely
Test each Kafka Connect deployment against four groups of checks before it carries production data: functional delivery, failure handling, rebalancing and REST API security. Run them in a staging environment that mirrors production topology, with the same Connect worker configuration, plugin versions and target systems. The aim is to show, with evidence you can hand to your security and data owners, that records reach the target with the expected schema and count, that bad input does not stop the pipeline, and that unauthenticated or unauthorized callers cannot change connectors. Write the results down as you go, because the same plan becomes your regression suite for later upgrades.
Start with functional checks. Produce a known set of records to the source topic or database, including records with every field type your schema uses, and confirm the target holds the same count with the expected schema. For sink connectors, compare record counts and a sample of field values at the target. For source connectors, confirm that new rows or objects appear on the topic and that restarting the connector does not replay or skip records beyond what the connector's documentation promises. Keep the expected results in version control so a reviewer can see what was checked.
Next, run failure tests. Stop a worker process while the connector is running and confirm that its tasks move to surviving workers and resume. Break the target connection by stopping the database, bucket endpoint or SaaS sandbox credentials for a short period, then restore it and check that the connector retries and recovers. Inject one malformed record into the input topic and confirm that, when error tolerance and the dead letter queue are configured, the bad record lands in the dead letter topic with error context in its headers, and that healthy records before and after it keep flowing. Verify the dead letter topic's retention and access settings as part of this test, since a failure path that cannot be read or retained is not useful during an incident.
Then test rebalancing. Add a worker to a running group and remove one, watching the task assignment in the REST status output and the group's logs. Confirm that tasks redistribute and that data neither duplicates nor disappears. Duplicate and loss behavior depends on each connector. Some sink connectors rely on idempotent writes or upsert keys, and some source connectors store offsets differently, so exactly-once delivery and offset semantics are a per-connector property, not a property of Kafka Connect itself. Read each connector's documentation, record which guarantee it claims and under which settings, and design your tests to check that claim, not a general one.
Finally, test the security boundary. Call the Connect REST API without credentials and confirm the request is refused. Then authenticate with a principal that has no ACLs for the relevant topics and groups, and confirm that connector creation, status reads and the topics the connector touches are refused where your configuration says they should be. Confirm that the connector principal can only reach the topics and consumer groups it needs. Ask your platform team to confirm which authorization settings apply to Connect internal topics in your build, since the defaults for resources without ACLs depend on configuration. Treat any accepted request that should have been refused as a blocking defect.
Kafka Connect deployment cost, effort and what you must provide
What does a Kafka Connect deployment cost and what drives the fee?
The fee follows from a written scope, and DeployOpen does not publish a price list. The main drivers are the number and type of connectors, whether custom plugins must be built and tested, how deeply Connect is tied to your identity and TLS setup, how many environments need the same build, and how much handover documentation and operator training you want. One-time work covers design, build, testing and cutover. Recurring costs depend on the infrastructure you run and on any support retainer you add later. Ask for a fixed-scope quote that lists each connector, each plugin, each environment and each handover deliverable as a separate line, so you can see what changes the price if the scope moves.
How long does it take to deploy Kafka Connect for our systems?
Duration depends on the factors that set it, not on a fixed number. The biggest are the number of source and sink systems, whether their owners can grant access and test credentials on time, whether any connector needs custom code, how much security integration is required, and how many environments must be promoted through change control. Delays usually come from waiting on access, approvals and test data rather than from the Connect configuration itself. Ask for a timeline in the written quote that names each dependency, the party responsible for it and the change windows you must respect.
Can our internal team run Kafka Connect without outside help?
Yes, if the team has the right skills and enough capacity to own the platform after go-live. Since Kafka 4.0, Connect requires Java 17, so the team needs a working Java runtime and the ability to patch it. Engineers need Kafka administration skills, including topics, consumer groups, ACLs, rolling upgrades and the Connect REST API. They also need working knowledge of each target system, such as database change capture, object storage formats and SaaS API limits. Teams with one or two people often use outside help for the initial build and then keep the runbooks in-house. Assess your skills against each connector in your inventory before deciding.
What must our organization provide before a Kafka Connect engagement starts?
You need to provide access and information that only your organization holds. This includes access to the Kafka cluster and the Connect REST endpoints in your environment, a connector inventory with source and sink systems, expected volumes and current versions, and a credentials policy that states where secrets are stored, who can read them and how they rotate. You also need test systems that resemble production for each source and sink, network paths between workers and targets, named owners for each connector and a change approval process. Confirm these inputs in the scoping stage, because missing test systems and credentials are common causes of schedule slippage.
Does the Apache-2.0 license mean Kafka Connect has no software cost?
Apache Kafka is released under the Apache License 2.0, and the Apache Software Foundation project does not offer a paid edition, so there is no license fee for the Kafka software itself. Recurring costs remain. They include compute and storage for workers and brokers, network traffic between systems, monitoring, backup, and staff time for upgrades. The project aims for three releases a year and supports rolling upgrades from the last three releases, so upgrade work recurs. Third-party connector plugins may carry their own licenses, so check the license of each one in your inventory before you assume it is free to use.
Can DeployOpen take over an existing Kafka Connect deployment?
DeployOpen can take over an existing deployment after reviewing it, and takeover is scoped in writing before work starts. The review usually covers the Kafka and Connect versions, the Java runtime, worker configuration, each connector and its error handling, plugin packaging, REST API security, backups and runbooks. Older KRaft clusters may need an upgrade path first, and clusters still based on ZooKeeper must migrate to KRaft before moving to Kafka 4.x. Ask for a written takeover assessment and a list of the gaps it finds, with the work needed to close each one, before you agree to a fixed-scope handover.
What to monitor, back up, upgrade and document after go-live
The operating routine after handover starts with knowing whether every connector and task is running. Poll the Kafka Connect REST API for connector and task state on each worker, and alert on any task in a FAILED state or any connector that does not return to RUNNING after a restart. Track consumer lag for each sink connector's consumer group, because a connector can report healthy while it falls behind the topic it reads. Also watch the dead letter topic named in errors.deadletterqueue.topic.name on each sink connector. Steady growth there usually points to a conversion, schema or data problem that will not clear on its own, so the alert should reach someone who can read the error headers and decide whether to replay or discard the records.
Back up the connector definitions and the Connect internal topics, then test that a restore works. Connector configuration files, including the JSON or infrastructure-as-code used to create connectors, belong in version control, with credentials referenced from a secrets store and never stored in the file. The internal topics named by config.storage.topic, offset.storage.topic and status.storage.topic hold connector configuration, source offsets and task state, so a rebuilt cluster depends on them. Confirm the replication factor and retention or cleanup settings for these topics with your Kafka platform team. Run restore tests in a separate environment, record how long recovery takes, and check that sink connectors resume from their committed offsets with the duplicate and gap behavior your delivery guarantees allow.
Plan upgrades against the Kafka release cadence and the rolling upgrade rule. The project aims for three releases a year, and the supported releases listed on the downloads page are 4.3.1, 4.2.2 and 4.1.2. Rolling upgrades are supported from each of the last three releases to the latest version, so a cluster on an older release needs an intermediate upgrade first. Before each upgrade, confirm with every plugin vendor, or in your own build records, that each custom and third-party connector plugin supports the target Kafka version, then retest each plugin in a staging Connect cluster. Kafka 4.x requires Java 17 for Connect, so check the JVM on every worker host and container image. Custom plugins must be retested on every Kafka upgrade, even when the plugin code has not changed.
Document the connector inventory well enough that someone other than the original builders can restart a failed pipeline. For each connector, record its name, business owner, technical owner, plugin class and version, the Kafka version it was last tested against, the topics it reads or writes, and where its credentials are held, such as a secrets manager reference, without copying secret values into the document. Attach runbooks for worker restarts, task restarts and offset resets. Each runbook should name the REST call or command used, who approves a reset on a production sink, and how the result is confirmed. Review the inventory at every Kafka upgrade and after each change of owner, so the record stays accurate when it is needed most.
How DeployOpen scopes, builds and hands over Kafka Connect work
DeployOpen plans, builds, documents and supports Apache Kafka Connect inside your own cloud accounts, data centers or air-gapped networks. Each engagement starts from a written, fixed scope, and the work is handed over with runbooks your team can operate. Deliverables are documented so your own engineers can run and extend the connectors after the project closes.
How we scope
- The written scope lists the Connect worker cluster, each connector and its source or sink system.
- Engagements are fixed-scope, so deliverables and timelines are agreed in writing before any build work starts.
- Scoping records the Java 17 runtime requirement for Connect on Kafka 4.x and the target Kafka release.
How we build
- Workers are deployed in distributed mode in your infrastructure, with connector plugins installed from reviewed artifacts.
- Error handling, dead letter queue topics and converter settings are configured and tested against sample records.
- REST API access is protected with TLS and authentication and limited to trusted networks.
How we hand over
- Handover includes runbooks for connector restarts, offset resets, worker scaling and plugin rollouts.
- Monitoring guidance covers connector and task state, consumer lag on sink connectors and worker health.
- Configuration, plugin versions and test evidence are documented so your team can repeat the deployment.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer after handover.
- Connector and plugin compatibility is rechecked when the Kafka release or Connect worker version changes.
- Security testing of the REST API and worker configuration can be scheduled alongside upgrades.
