Running Kafka Connect on Kubernetes with Strimzi: when your team needs it
Choose Strimzi when your platform team already runs self-hosted Kafka on Kubernetes and needs to move data into lakehouses, databases or search without building its own deployment tooling. Strimzi runs Connect workers and manages connectors as Kubernetes custom resources, so connector configuration, worker replica counts and connector lifecycle can live in version-controlled manifests next to the broker definitions. That model fits better than running Connect on virtual machines when your delivery pipelines, access reviews and rollback habits already revolve around Kubernetes. Kafka is licensed under the Apache License 2.0 with no paid edition from the project itself, but Connect requires Java 17 since Kafka 4.0, so every plugin image must be built on a Java 17 base.
The trade-off is that Strimzi adds a control layer your team must operate. Kubernetes ships roughly three releases a year, and the version skew policy requires kube-apiserver to be upgraded one minor version at a time, with kubelets allowed to lag up to three minor versions behind. Kafka, Strimzi, Connect workers and the Kubernetes version therefore need to be checked together before each upgrade. Kubernetes also stores Secrets unencrypted in etcd by default, so connector credentials need encryption at rest or an external store, such as the Secrets Store CSI Driver, before they reach the cluster. Virtual machines may suit better if your team has limited Kubernetes operations depth, or if connectors depend on host-level network paths or local plugin directories that are awkward to express in pods.
Before you commit, confirm three things against the Strimzi documentation and your own environment: which Strimzi release supports the Kafka 4.x version you plan to run, how plugin images will be built, scanned and promoted through your pipeline, and which connectors need features that depend on the Connect version you choose. Ask for a written scope that names the owner of the image pipeline, the Kubernetes upgrade calendar and the credential rotation process for each connector. Those answers decide whether Connect on Kubernetes becomes a stable platform capability or a source of recurring operational work for your team.
Building the Connect image with connector plugins
Every connector a worker runs has to be in its image before the pod starts, so the image build is where plugin choice, provenance and version pinning actually happen. The cards below cover the build options, the checks to run before a plugin is baked in, and the decisions that make later upgrades predictable.
Custom image from the Strimzi base
Strimzi's documented pattern is to start from a Kafka Connect base image and add connector JARs in a new image layer, then reference that image in the Connect custom resource. Each connector version becomes a new image tag, so a plugin change is a reviewed build rather than an edit on a running worker. Confirm the base image tag matches the Kafka version your Strimzi release supports before you build. Plugin layout and discovery rules belong to the plugin management page and should be checked against the Strimzi and Kafka docs.
Pulling plugins from artifact sources
Plugins can be fetched from a Maven repository, an artifact store or a vendor download during the build, or Strimzi can pull them declaratively when the image is built by its tooling. Each option changes who controls the bytes and how reproducible the build is. Ask whether your build can run without internet access, and whether a mirror of every plugin artifact is kept in your own registry.
Verifying provenance before baking in
Before a connector JAR enters the image, record its source, publisher and license, and check the checksum against the value the publisher supplies. Connector code runs inside your Connect workers with access to the credentials and data those pipelines use, so an unknown JAR is a supply chain risk. Treat plugin approval as a security review with a named owner, not a convenience step in the pipeline.
Pinning versions and digests
Pin every plugin to an exact version and reference the image by digest rather than a floating tag such as latest. A pinned build lets you reproduce the worker image during an incident and roll back to the previous image without guessing which connector code was running. Record the Kafka version, the Strimzi version and each plugin version together in the build manifest so an upgrade can be planned as one change.
Checking compatibility before the build
A connector built for one Kafka client line may not behave the same on another, and Kafka's own upgrade rules limit which broker and Connect versions can run together. Check each plugin's supported Kafka and Java requirements against the target image. Kafka 4.0 and later require Java 17 for Connect, so confirm that the base image and every plugin run on that runtime before you promote the build.
Testing the image before rollout
Start the new image in a non-production namespace with the same connectors, converters and secrets references, then confirm each plugin is discovered and each connector reaches the running state. Keep a short list of checks: plugin load errors in worker logs, task failures, and a test record flowing through each pipeline. Do not promote an image that has not run each connector class it contains.
Which Connect deployment options fit which team
A single team that owns its pipelines and its Kubernetes namespace usually benefits from the smallest operational surface. One KafkaConnect resource can run the connectors that team needs, with a plugin image built from a short list of reviewed connectors. Connector definitions and the image live in the same repository, so they change together and roll back together.
- Replicas: Start with a small worker count and raise it when task lag or repeated rebalancing becomes visible, since workers share the connector tasks.
- Plugin image: Build one image that contains only the plugins in use, pinned to exact versions, so a rollback means redeploying a prior image tag.
- Configuration: Set Connect internal topic names and replication factors explicitly in
spec.configinstead of relying on broker defaults. - Resources: Set requests and limits from observed heap use and task load, not from values copied from another cluster.
- Credentials: Reference a Kubernetes Secret through
spec.externalConfigurationso passwords stay out of the custom resource manifest.
Confirm that the Strimzi version you run supports these field names, which Kafka version it bundles, and that the Connect runtime meets the Java 17 requirement that applies to Kafka 4.0 and later.
When several teams share Connect workers, the platform team usually owns the KafkaConnect resource and the plugin image, while application teams own their KafkaConnector resources. The design question becomes how to give teams self-service without letting one team change worker settings that affect everyone. Separate clusters per team, or one shared cluster with strict namespace boundaries, are both workable, and the choice depends on how much isolation each team needs.
- Plugin governance: The platform team approves each connector plugin and version before it enters the shared image, so no team installs unvetted code.
- Role separation: Use Kubernetes RBAC so teams can create and change
KafkaConnectorresources in their own namespaces but cannot edit theKafkaConnectresource. - Resource quotas: Apply namespace quotas and agree on worker capacity so one heavy connector does not starve the others.
- Naming conventions: Require a team prefix for connector names and internal topics so ownership is visible in logs and metrics.
- Upgrade coordination: Schedule Connect upgrades with every consuming team, because a worker restart pauses all of its connectors.
Confirm whether your Strimzi operator watches all namespaces or specific ones, since that setting decides which teams can manage resources through the operator.
In regulated environments, the Connect cluster often sits in a segment with no direct internet access. Images, operator components and connector artifacts must come from a registry you control, and every plugin needs an owner, a license check and a recorded approval. Access to Kafka and to any external system the connectors reach should be explicit and logged, so the evidence an assessor asks for can be produced from the platform itself.
- Image sourcing: Build the plugin image in a controlled pipeline and push it to an internal registry, so workers never pull code at runtime.
- Transport security: Connect to Kafka over SSL/TLS with SASL authentication, and grant the Connect principal only the ACLs its connectors need.
- Secret handling: Kubernetes stores Secrets unencrypted in etcd by default, so enable encryption at rest and restrict list and watch access with least-privilege RBAC. External secret stores are an option when your policy requires them.
- Pod hardening: Apply the Pod Security Standards level your policy requires and restrict privilege escalation for Connect pods.
- Evidence: Keep build records, plugin approvals and configuration history in version control so changes can be traced.
Good practices for Kubernetes Secrets explains the options. Confirm with your assessor which evidence they expect, and confirm the Strimzi procedure for mirroring its operator and Kafka images into a disconnected registry.
Setting up Kafka Connect and your first connector
This sequence takes a Strimzi-managed Kafka Connect cluster from version checks to a running connector that writes records to a topic. Each step ends with a check, so a failure is caught before the next layer depends on it.
Confirm Kafka and Kubernetes versions
Before installing anything, check the Kafka downloads page for the currently supported releases and confirm the Strimzi release you plan to use supports the Kafka version you choose. Check the Kubernetes releases page as well, because the three most recent minor releases are maintained and older clusters need an upgrade plan first. Record the Java runtime your Connect image will use, since Kafka 4.0 and later require Java 17 for Connect. The step is done when you have a written version matrix for Kubernetes, Strimzi, Kafka and the JDK, and your platform team has signed off on it.
Build and publish the plugin image
Create a Connect image that extends the Strimzi Kafka base image and adds the connector plugin directories under the plugin path. Use only connector archives whose license and provenance your team has reviewed, and pin each plugin to an exact version. Push the image to a registry your cluster can pull from, and scan it with your standard container tooling. The check is that the image pulls successfully and that listing the plugin path inside a throwaway container shows each connector's directory and JAR files.
Store broker credentials as Kubernetes secrets
Create the Kubernetes Secrets that hold the TLS certificate authority, the Connect user credentials for SCRAM or mTLS and any connector-specific passwords. Keep these out of manifests and Git, and restrict read access with least-privilege RBAC, because Secrets are stored unencrypted in etcd unless encryption at rest is configured. Confirm with your cluster administrator whether encryption at rest or an external store through the Secrets Store CSI Driver is in place. The check is that each Secret exists in the target namespace and that only the Connect service account and operators can read it.
Deploy the KafkaConnect resource with TLS and authentication
Apply a KafkaConnect custom resource that points at your image, sets the bootstrap servers, enables TLS on the listener and references the authentication Secret. Set the replica count and the Connect internal topic replication factors so they match the broker count in your cluster. Strimzi reconciles the resource into a Connect deployment, and the operator status reports whether the workers reached the ready condition. The check is that the KafkaConnect status shows ready, that the worker pods are running, and that the worker logs show a successful connection to the brokers without authentication or certificate errors.
Create the KafkaConnector resource
Apply a KafkaConnector custom resource that names the connector class, the target topics, the converter settings and any transforms or error tolerance you have decided on, and set the tasks.max value deliberately. Reference secrets for credentials rather than writing them into the connector config. The Strimzi operator submits the configuration to the Connect REST API, so the connector appears in the KafkaConnector status without a manual REST call. The check is that the connector status reports the expected connector class and that no configuration validation errors appear.
Verify worker status and task state
Read the KafkaConnector status and the Connect worker status to confirm the connector and each of its tasks are in the running state. Check that tasks are distributed across the worker pods, and if a task has failed, read its trace in the status and the worker logs before restarting anything. Confirm that the connector is not stuck in a rebalance loop. The check is that the connector and all of its tasks report running on the expected number of workers for a stable period you define in your test plan.
Confirm topic output and record a baseline
Produce a small set of test records with a known structure, or use records already in the source system, and read the destination topic with a consumer that uses TLS and your test credentials. Compare the record count, key format and value schema against the connector design. Save the connector manifest, image digest and the output sample as the baseline for later upgrades. The check is that every expected record appears once in the topic in the expected format, and the saved baseline is stored in version control for the next change window.
Managing connectors as custom resources
Declaring each connector as a KafkaConnector resource makes the connector's class, tasksMax, configuration and desired state into versioned manifests that Git history and code review can govern. The alternative is changing connectors by calling the Connect REST API by hand, which leaves no reliable record of what is running. With the resource model, the Strimzi operator watches these objects and reconciles the live connector to match them, so the manifest in the repository is the change record. Teams can keep one connector definition and supply environment-specific values such as topic names and secret references, and they can roll back by reverting a commit. Separate the rights to create KafkaConnector objects in a namespace from the rights to change the KafkaConnect cluster itself, because a connector change and a worker change carry different risk. Confirm in the Strimzi documentation for your chosen version how the operator handles connectors that exist in the cluster but have no matching manifest.
Configuration and state changes are reconciled in place, but they are not free. Changing a connector's configuration, pausing it or resuming it is applied to the running connector, and the operator reports the outcome in the resource's status. Changing tasksMax is different because Kafka Connect redistributes tasks across workers, and that rebalance briefly stops processing for the affected connector. Treat the status conditions as the verification step: after each apply, check that the connector and its tasks report a running state, and do not assume that a successful commit means successful reconciliation. Confirm for your version whether the operator retries a failed reconcile on its own, and what it reports when a connector configuration is rejected by Connect validation.
Some changes require a restart, and restarts are where cost and risk concentrate. Upgrading a connector plugin usually means building a new image, and changing worker-level settings in the KafkaConnect resource can cause the Connect pods to roll. Each restart triggers a rebalance and can cause tasks to resume from their last committed offsets, so sinks may reprocess a small window of records. Make sure the target systems handle duplicates through idempotent writes, or confirm that exactly-once support applies to your connectors. Sequence image rebuilds and connector configuration changes into planned windows, and rehearse the same change in a non-production cluster. Confirm the restart behavior for your Strimzi and Kafka versions before scheduling production changes, because operator behavior has changed between releases.
Handling connector credentials with Kubernetes Secrets
Connect workers and connectors need database passwords, cloud keys and TLS material, and on Kubernetes those values usually live in Secrets. The cards below cover how Strimzi references them, where Secrets are stored, how to protect them at rest and in the API, and when an external store is the better home.
Referencing Secrets from worker and connector config
Strimzi resources can pull values from Kubernetes Secrets and ConfigMaps instead of embedding them in the KafkaConnect or KafkaConnector manifest. Keep passwords, access keys and truststore passwords out of the Git-managed custom resource itself. Check that each referenced Secret sits in the same namespace as the Connect cluster and that secret names are stable across environments, since a renamed Secret silently breaks a restart.
Secrets are unencrypted in etcd by default
A Kubernetes Secret is stored unencrypted in etcd unless the cluster administrator enables encryption at rest. Anyone who can read the etcd data or its backups can read every connector password on the cluster. Before you place database or cloud credentials in Secrets, ask the platform team whether encryption at rest is configured for the API server and whether etcd snapshots are encrypted and access-controlled.
Encryption at rest for the API server
Enabling encryption at rest means the API server encrypts Secret data before writing it to etcd, using a key the cluster holds. Confirm which provider the distribution uses, how the encryption key is rotated and whether key material sits in a KMS or on the control plane nodes. Rotation procedures should be tested on a non-production cluster, because a bad key change can make existing Secrets unreadable.
Least-privilege RBAC on list and watch
Verbs such as list and watch on Secrets return every Secret in a namespace, not one named Secret, so granting them to a broad service account exposes all Connect credentials at once. Give the Strimzi operator and the Connect pods only the access they need, and limit human access to Secrets to a small group. Review role bindings whenever a new connector namespace is added.
When the Secrets Store CSI Driver fits better
The Secrets Store CSI Driver mounts values from an external store, such as a cloud secrets manager or a self-hosted vault, into pods at runtime. This fits when security policy forbids storing credential values in the cluster at all, or when rotation must happen in the external store without redeploying Connect. Confirm that the store has a provider for your platform and that Strimzi can consume the mounted values for the fields you need.
Rotation and restart planning
Changing a Secret does not automatically update values that Connect already loaded into a running worker. Plan credential rotation as a documented step: update the Secret, roll the Connect pods or restart affected connector tasks, and verify that each connector reconnects. Record which connectors depend on each credential so that a rotation does not surprise the owners of unrelated pipelines.
Scaling Connect workers and tasks
Scaling a Connect cluster on Strimzi is mostly a question of tasks, not workers. A connector divides its work into tasks, and each task runs on one worker at a time, so adding worker pods only helps when there are enough tasks to spread across them. For a sink connector reading a Kafka topic, the practical ceiling is the partition count of the source topics: Kafka assigns each partition to at most one consumer in a group, so a sink connector with tasks.max set above the number of partitions leaves the extra tasks idle. For a source connector, the limit depends on how the connector splits its input, such as tables or file sets. Check the connector's documentation for how it partitions work before you raise tasks.max in the KafkaConnector resource.
The second limit sits outside Kafka. A database, object store or search cluster accepts writes or reads at a rate set by its own capacity, connection limits and load from other clients. Adding tasks and workers past that point moves the bottleneck to the target system, increases connection counts and can trigger throttling or lock contention. Capacity figures should come from your own load tests against the real source or sink, using representative message sizes and transforms, and measured consumer lag and task status during those tests. Ask the connector vendor and your database or search team which throughput limits and connection caps apply, and write those numbers into the test plan rather than assuming them.
Scaling workers alone has clear limits. Strimzi manages the Connect worker replicas through the KafkaConnect resource, and the connector tasks are rebalanced across the workers in the group. Adding replicas without changing tasks.max or the partition count gives no extra parallelism and can cause short rebalance pauses that increase lag. Treat worker replica count as a resilience setting: enough workers that a pod failure does not stop the pipeline, and changes to tasks as the throughput setting. Review resource requests and limits for each worker, since tasks that are added without memory headroom will fail under load in ways that look like connector errors.
Upgrading Kafka, Strimzi and Kubernetes without stopping pipelines
Connect pipelines run on three moving parts: the Kubernetes cluster, the Strimzi operator and the Kafka release underneath. Each has its own compatibility rules, so the order of changes matters as much as the changes themselves. The cards below cover the sequence, the version constraints and the checks to run after every step.
Upgrade the control plane first
Kubernetes requires the kube-apiserver to move one minor version at a time and to go before the kube-controller-manager, kube-scheduler and kubelets. Plan the control plane upgrade as its own change window, then confirm the API is healthy before touching worker nodes. Check the Strimzi release notes for the Kubernetes versions it supports before you start, because the operator may need to move first in some cases. Confirm this with your Strimzi version rather than assuming it.
Keep kubelets within skew
Worker nodes may run kubelets up to three minor versions older than the API server. That gives room to roll node pools gradually, but it is not a reason to leave nodes behind indefinitely. Drain Connect worker pods through a PodDisruptionBudget so tasks rebalance instead of failing together. Track which node pool runs which kubelet version during the change.
Move Kafka in rolling steps
Kafka supports rolling upgrades from the last three releases to the latest. The supported releases on the downloads page are 4.3.1, 4.2.2 and 4.1.2, so a cluster on an older line needs intermediate steps. Strimzi manages the broker rollout, but your team should confirm the target Kafka version in the Kafka custom resource and watch under-replicated partitions during each restart. Pin the version in your change record so the rollout is traceable.
Java 17 for Connect workers
Since Kafka 4.0, Kafka Connect requires Java 17. Any custom connector image built on an older base image or a plugin compiled for an older runtime will fail after the move. Check the base image in your Dockerfile and the Java target of every plugin jar before upgrading the Connect cluster. Treat the base image as a versioned artifact so you can roll back to a known-good build.
Re-check plugins after every change
A connector that loaded on the previous Kafka release can still break after a Kafka, Strimzi or base image change because of altered class loading, removed APIs or a different Java runtime. After each change, confirm each connector reports RUNNING, that its tasks are assigned and that a test record flows end to end. Keep a manifest of plugin versions per image so you can compare builds. Confirm connector-specific compatibility with each vendor, since the Kafka version alone does not decide it.
Sequence and roll back each step
Run one change at a time: Kubernetes control plane, node pools, Strimzi operator, then Kafka and Connect. Take a snapshot of connector configurations and offsets before each step, and record which connectors use exactly-once settings, since those may need additional steps. Define the rollback point for each stage in advance, for example the previous image tag and operator version. Test the full sequence in a non-production cluster first.
Questions teams ask before running Connect on Strimzi
Who does the work when running Kafka Connect on Strimzi?
The work splits between the platform team, the data team that owns each pipeline, and any partner you hire for the build. The platform team owns the Kubernetes cluster, the Strimzi operator, the Connect image pipeline, network policy and secrets handling. Data engineers own connector configuration, topic contracts and error routing. A partner such as DeployOpen can plan, build and document this under a written, fixed scope, and ongoing support is available as a retainer. Confirm early which team carries each item, because unowned items such as image rebuilds after a connector release are the most common gaps.
What drives the cost of running Connect on Strimzi?
Cost follows the number of connectors, the number of source and sink systems you must reach, and how much custom plugin work is needed. Other drivers are the Kubernetes platform you already run, the secrets store you choose, how many environments you maintain, and how strict your security and compliance evidence must be. One-time costs include design, image build, and first deployment. Recurring costs include Kubernetes capacity, operator and connector upgrades, monitoring, and on-call time. Ask any quote to itemize these components and to state which items are included in the fixed scope.
How long does it take to get a first working connector?
The time depends mostly on whether the Kubernetes platform and the target system are already available, and on how many decisions remain open. A team with a working cluster, a container registry and a reachable target database can usually reach a first test connector faster than a team building the platform at the same time. Connector plugin selection and license checks also add time when the plugin is not already approved. Ask for a timeline tied to written milestones in your scope document, and confirm the dependencies on your side, such as network access and credential issuance.
Can an in-house team run Connect on Strimzi without a partner?
Yes, if the team already operates Kubernetes and has at least one engineer comfortable with Kafka concepts such as topics, offsets and consumer groups. The skills that matter most are Kubernetes operations, container image builds, secrets management, and monitoring of connector status and task failures. Teams with no prior Kafka or Kubernetes operations experience usually need training or outside help for the first build. Review your staffing against the on-call load of running connectors that touch production data, and confirm that someone owns each upgrade before you begin.
How do Strimzi and Kafka upgrades affect running pipelines?
Upgrades can interrupt connector tasks, so plan them as rolling changes with a test environment first. Kafka supports rolling upgrades from each release in the past year, the last three releases, to the latest version, according to the Kafka time-based release plan, and Kafka 4.x runs only in KRaft mode. Kubernetes upgrades follow the version skew policy, which requires the API server to move one minor version at a time before other control plane components and kubelets. Check connector plugin compatibility with each Connect version, pause sensitive connectors where needed, and verify offsets after each step.
When should a team choose a different deployment model for Connect?
Choose a different model when you do not run Kubernetes, when a managed service already meets your data residency and compliance needs, or when the number of connectors is small enough that a single managed or VM-based Connect worker is simpler to operate. Kubernetes adds operational surface, including the operator, the image pipeline and cluster upgrades, so the cost is justified when you already run many workloads on the platform. Also compare against a managed Connect offering if your licensing review rules out a connector you need. Document the decision with its drivers so it can be revisited when the connector count changes.
What should a quote for Connect on Strimzi include?
A useful quote lists the scope of each deliverable, the connectors and target systems in scope, the Kubernetes environments covered, and the handover items such as runbooks and configuration documentation. It should also state assumptions about your platform, such as the cluster version and secrets store, and say what is excluded, for example ongoing connector upgrades or new plugin development. DeployOpen agrees scope, deliverables and timelines in writing before work starts and does not publish a price list. Ask which items recur after go-live and how they are priced.
How DeployOpen scopes and supports Strimzi-based Connect
Every Strimzi-based Connect engagement starts with a written scope that names the clusters, namespaces and deliverables before any work begins. For a Connect build, that scope usually lists the Kafka cluster the workers attach to, the Connect cluster resources and their replica counts, the container image pipeline that bakes connector plugins into the image, the KafkaConnector resources to be created, the Secrets and RBAC bindings for credentials, and the runbooks handed over at the end. Most engagements are fixed-scope, so the scope document also records what is out of scope, such as connector source-side database changes, downstream lakehouse design or Kubernetes platform work. Ask any partner, including DeployOpen, to show the exact deliverable list and the acceptance checks for each item before you sign.
The work happens inside your infrastructure. DeployOpen plans and implements against your own Kubernetes clusters, your own container registry, your own Kafka brokers and your own secret store, so connector credentials and data never move to a DeployOpen environment. Access is granted under your identity controls and removed when the engagement closes. Written scope also covers who holds the Strimzi operator version pin, who approves image rebuilds and who owns the upgrade calendar after handover.
For the period after go-live, DeployOpen offers retainers that cover ongoing upgrades, connector version reviews, plugin image rebuilds and support, as an alternative to a one-time build. A retainer should be priced against a written list of recurring tasks, such as applying Strimzi and Kafka version changes in staging before production, re-running connector smoke tests and reviewing connector licenses when plugins change. DeployOpen does not hold certifications or partner status with Strimzi, the CNCF or any connector vendor for this work, so ask for the references and runbook samples that show the method instead.
