Apache Kafka customization services

What Kafka customization covers

This service covers the custom code you need to run beside a self-hosted Apache Kafka cluster: Kafka Connect source and sink connectors, Kafka Streams applications, serializers and deserializers, and authorizer or authentication plugins. Kafka itself is released under the Apache License 2.0, so none of this code comes from a paid vendor edition. Someone on your side or a partner has to write it, test it, package it and keep it working.

Teams usually start this work for one of three reasons. A source system or sink has no existing connector that meets your data and security requirements. A business rule, such as enrichment, windowed aggregation or fraud scoring, must run inside the stream rather than in each consuming service. Or your organization needs an identity or access model that the built-in SASL mechanisms and StandardAuthorizer ACLs cannot express without a custom plugin. Each trigger should be written down with its owner before design begins.

The direct answer is that many organizations build the first connector or Streams application in-house, then bring in a partner when the code has to ship at scale or the team lacks Kafka internals experience. Either way, customization is maintained code, not a one-time install. Connectors and plugins depend on Kafka client and server interfaces, so each Kafka upgrade, including moves between the supported 4.x releases, can break them. Confirm with your team which plugin APIs you will depend on and who owns regression testing after each upgrade.

What the work includes

Each engagement produces code, tests, packaging and documentation that your team can build, run and upgrade without us. The scope of any plugin or authorizer code is agreed in writing before work starts, because it runs inside broker or worker processes.

Connector and Streams code

We build the custom Kafka Connect connector, Kafka Streams application, serializer or transform you scoped, with design notes covering topic keys, partitioning, state stores, error handling and delivery semantics. The notes record why each choice was made so your engineers can change it later without guessing.

Unit and integration tests

Tests run against a Kafka test setup that mirrors your target version and client configuration, not only mocks. They cover serialization round trips, failure and retry paths, rebalancing and restart behavior, so regressions show up before a cluster upgrade does.

Runtime packaging

Connect plugins are packaged for the plugin path your workers load from, and Streams applications are built as deployable artifacts with pinned dependencies. Packaging targets Java 17 for Connect and brokers, and Java 11 or later for clients and Streams applications, as the Kafka upgrade notes require.

Broker-side plugin review

Authorizer and authentication plugins load into broker processes, so a fault can affect the whole cluster. We agree the interface, permitted dependencies, failure behavior and review steps in writing before coding, and record who approves each change to that code.

Configuration documentation

We document every configuration key the code reads, its default, its allowed values and the effect of changing it. Topic names, consumer group IDs, security settings and connector options are listed in one place so operators can audit and reproduce the deployment.

Upgrade test plan

The plan lists the Kafka versions you intend to run and the rolling upgrade path between them, along with the tests to repeat at each step. It also states which client and connector versions you must confirm for compatibility, since rolling upgrades are supported from the last three releases to the latest version.

Handover to your team

Your engineers receive the source repository, build pipeline, runbooks and a walkthrough of the design notes and tests. The handover records who owns each component after delivery, so maintenance and future changes stay with people inside your organization.

What drives cost and duration

This table lists the main effort drivers in Kafka customization, why each one raises work, and one way to keep it down.

Driver	Why it raises effort	One way to keep it down
Source and sink systems	Each system needs its own connector configuration, authentication, schema mapping, error handling and test fixtures.	Reuse maintained connectors and limit custom code to the gaps they cannot cover.
Delivery semantics and error handling	Exactly-once or at-least-once behavior must be designed, tested under failure and documented for operators.	Agree required semantics per flow first, and route bad records to a dead-letter topic with alerting.
State and windowing in Streams	Stateful operators, changelog topics and window retention add state stores to size, back up and migrate.	Keep topologies stateless where possible and set window retention to the actual business need.
Serialization and schema evolution	Each format change needs compatibility rules, versioned consumers and tests against old and new payloads.	Choose one format and one compatibility mode early, and enforce both in CI with a schema registry.
Authentication or authorization model extended	Custom authenticators or authorizers affect every client and all ACLs, so a change has a wide blast radius.	Extend the documented pluggable authorization interface rather than replacing the StandardAuthorizer.
Environments and runtime versions	Brokers, Kafka Connect and tools need Java 17 from Kafka 4.0, while clients and Streams apps need at least Java 11.	Pin each plugin to tested Kafka and Java versions and build one artifact per supported runtime line.
Compliance evidence	Assessors expect change records, test results, access reviews and ACL exports covering custom code paths.	Build versioned configuration, reproducible packaging and audit logging into the build from the start.

Project phases and outputs

The sequence takes a Kafka customization from agreed requirements to tested, deployed artifacts with runbooks, and the schedule follows the written scope, so ask for the agreed timeline in the quote.

Confirm requirements and target Kafka version
DeployOpen and your team confirm the integration or policy requirement, the systems on each side, expected data volumes and the target Kafka release from the supported versions listed by the project. The Java runtime each component needs is confirmed at this point. Output: a written requirements note and a compatibility statement naming the versions to build against.
Design topics, schemas and plugin interfaces
The design covers topic names, partitioning and retention, schema format and compatibility rules, the interfaces each connector, Streams application, serializer or authorizer plugin will implement, and how failures are retried, dead-lettered or alerted. Output: a design document reviewed by your architecture and security teams before any code is written.
Build components with code review
Developers implement the connectors, Streams applications and plugins against the approved design, with configuration externalized and secrets kept out of source code. Each change goes through peer review and automated static checks in your repository. Output: reviewed source, build files and a change log that maps code to design decisions.
Test and rehearse the upgrade
Testing covers unit logic, integration against a running broker, schema evolution, restart and failure behavior, and security settings such as TLS and SASL. The same artifacts are then run against a supported Kafka release, and a rolling upgrade is rehearsed from an earlier release in the supported range. Output: test reports and an upgrade rehearsal log with every issue resolved or accepted.
Package and deploy to staging
Artifacts are built with versioned, repeatable builds, stored in your artifact repository and deployed to staging using the same configuration pattern as production. Staging enforces the production authentication and authorization model, including ACLs with deny-by-default. Output: a checksummed artifact set and a staging sign-off record.
Deploy to production with rollback ready
After staging sign-off, the artifacts are promoted to production in a change window agreed with your operations team. Monitoring on consumer lag, task failures and dead-letter topics is active during the change, and the rollback path is rehearsed beforehand. Output: a production change record and a completed set of post-deployment checks.
Hand over runbooks and ownership
DeployOpen walks your operations and development teams through the system and provides runbooks for deployment, restart, scaling, failure recovery and the upgrade procedure. The written scope should state how code, repository access and credentials transfer to you. Output: runbooks, architecture notes and a recorded handover session.

What to keep in-house and what to outsource

This table splits Kafka customization work into parts teams usually keep and parts they often outsource, with the skills each part needs, so you can see where your own staff must sit.

Work part	Ownership	Skills it needs	Why it sits there
Choosing which integrations matter	Keep in-house	Data ownership knowledge, source system familiarity, Kafka Connect basics	Only your teams know which datasets, consumers and business events carry value and risk.
Owning business rules	Keep in-house	Domain modeling, data governance, subject matter expertise	Rules for transformations, filtering and enrichment encode decisions that must stay under your control.
Reviewing and approving code	Keep in-house	Java code review, Kafka internals, security review of authorizer code	Approval protects production topics and ACLs, so a named internal reviewer should sign off every release.
Operating the connectors	Keep in-house	Kafka Connect REST operations, monitoring, incident response, on-call runbooks	Day-to-day operation runs against your clusters, credentials and alerting, which your operators must own.
Plugin and Streams code design	Often outsourced	Kafka Streams topology design, Kafka internals, Java 17 build tooling, data modeling	Designing custom serializers, authorizers and topologies takes deep client-side expertise that many teams build only occasionally.
Test harnesses	Often outsourced	Embedded broker testing, Java test frameworks, Kafka Streams test utilities, fixture data design	Repeatable tests for topologies and plugins are specialized work that can be built once and handed over.
Packaging	Often outsourced	Java build tooling, dependency management, artifact signing, Kafka Connect plugin layout	Packaging for Connect and broker classpaths must match your runtime, and mistakes surface only at deploy time.
Upgrade rehearsal	Often outsourced	Kafka upgrade notes, client compatibility testing, rolling upgrade planning, Java runtime versions	Rehearsing plugin and client behavior across supported Kafka releases needs a staging cluster and a repeatable checklist.

What usually goes wrong

Most failures in custom Kafka work come from build-time decisions that surface later, during upgrades, broker failures or access changes. Each card names one failure and the check that prevents it.

Code tied to internal interfaces

Connectors and plugins often import classes from Kafka's internal packages, and those classes can change between releases without notice to custom code. Limit custom code to the public Connect, Streams and client APIs, list every Kafka class it imports, and review that list against the upgrade notes before each release move.

Plugins that pass unit tests only

Unit tests rarely exercise consumer group rebalancing, Connect task restarts or partition leader changes, which is where custom connectors and Streams apps usually fail. Run integration tests on a multi-broker cluster while stopping a broker and forcing a rebalance, then confirm offsets, state stores and output records remain correct.

Serializers that break existing consumers

A changed serializer can write payloads that existing consumers cannot read, even when every producer write succeeds. Set the schema compatibility mode for each topic, test new writers against readers on the current and previous schema versions, and record the serializer version used for each topic so rollbacks stay possible.

Deny-by-default authorizer lockouts

With StandardAuthorizer in KRaft mode, a resource without an ACL is limited to super users unless allow.everyone.if.no.acl.found is set to true. Custom authorizer logic that follows this pattern can block applications that worked before the change. Inventory every principal and topic, grant ACLs before enforcement starts, and test denial paths in staging.

No upgrade tests for custom code

Kafka supports rolling upgrades from each of the last three releases to the latest version, but custom code only has that coverage if a test suite exercises it. Run connectors, Streams apps and plugins against each target release and its required Java runtime, and treat a failing combination as a release blocker. Confirm the target is still a supported release, since bugfix releases come only for supported lines.

No named owner after handover

Custom components often outlive the team that wrote them. Without a named owner, nobody decides when to upgrade a plugin, rotate its credentials or confirm which topics depend on it. Record an owner, a dependency inventory and a runbook for each component at handover, and review that list at every Kafka upgrade.

How the work changes by starting point

If you already run Kafka Connect in distributed mode, the main question is how custom connectors join a running fleet. Each change must be tested against the connector version and the plugin path your workers load.

Build: Compile against the Connect API version your workers run, and match the Java runtime. Since Kafka 4.0, Connect requires Java 17.
Testing: Run the connector against production-like topics and configurations, including worker restarts and task rebalances.
Packaging: Ship the plugin in the directory the workers scan, and check for conflicting transitive libraries already present on the workers.
Delivery: Roll new plugins to one worker at a time, and confirm through the REST API that tasks report healthy after each step.
Confirm: Which Java version your workers run, and whether connector configurations are kept in version control.

What you need to provide

Customization moves quickly when the inputs are settled before code is written. The most useful thing you can provide is a named owner for each source and target system, someone who can confirm field meanings, expected volumes and what a bad record looks like. Without that person, connector mapping and Streams topology decisions get made by guesswork, and those decisions are the most common source of rework.

Provide sample payloads and schemas for every topic and external system in scope, including the Schema Registry subjects if you use one. Include real edge cases such as nulls, late or duplicate events, and out-of-order updates. Also state the failure requirements in writing: which records may be dropped, which must go to a dead letter topic, and how long reprocessing may take. Unclear failure rules are a frequent cause of rework, because error handling is hard to retrofit after tests are written.

Give the team access to a non-production Kafka cluster that matches production in version, listener configuration, authentication and ACL model. Confirm the current Kafka and Java versions in both environments, since the Kafka 4.x line requires Java 17 for brokers and Kafka Connect, while clients and Kafka Streams applications need at least Java 11. Describe the security requirements for connectors and plugins: the SASL mechanism in use, TLS settings, and whether authorization is handled by StandardAuthorizer ACLs or an external authorizer.

Share the code repository, branching model, and the build pipeline that will package the artifacts, along with any artifact repository and signing rules your organization requires. Finally, name one reviewer who owns the code after handover and has authority to approve changes, upgrades and dependency updates. Without a named owner, custom plugins tend to drift behind Kafka releases and stop surviving upgrades.

Frequently asked questions

How is the cost of Kafka customization estimated?

Cost follows from the written scope: the number of connectors, Streams applications, serializers and plugins, the source and target systems they touch, and the testing and packaging you require. What the quote should cover: build work, tests against your Kafka version, packaging, handover documentation and any retainer for later upgrades. Most engagements are fixed-scope, so deliverables and timelines are agreed in writing before work starts. Kafka is licensed under Apache-2.0 with no paid edition from the project, so your main costs are infrastructure, your team's time and the implementation work.

How long does a Kafka customization project usually take?

Duration depends mainly on how many components are in scope and how complex each one is. The main drivers are the number of connectors or Streams topologies, the APIs and schemas of the systems they connect, the security model (TLS, SASL and authorizer integration), and how much automated testing must exist before handover. Late decisions on data contracts and delayed access approvals in your environment extend the schedule. DeployOpen does not publish a standard timeline. Ask for phases and dates in the written scope, and check them against your change calendar.

Can our own engineers build Kafka Connect connectors and Streams apps?

Yes, if the team has the skills and capacity to own the code after handover. Connector work needs Java, the Kafka Connect API, offset and error handling, and knowledge of the source or sink system. Kafka Streams applications need Java and an understanding of state stores, repartitioning and processing guarantees. Since Kafka 4.0, brokers, Kafka Connect and tools require Java 17, while clients and Streams applications need at least Java 11, so your build toolchain must match. Plan for on-call ownership, dependency updates and test coverage.

What happens to custom Kafka code when Kafka is upgraded?

Custom code must be checked at each upgrade, and it needs changes only when the APIs or configuration it depends on change. Kafka 4.0 supports only KRaft mode, so ZooKeeper-based clusters must migrate to KRaft before upgrading to 4.x. Rolling upgrades are supported from each of the last three releases to the latest version. The project aims for three releases a year, one every four months. Keep a compatibility test suite for each target version, pin client and connector dependencies, and review the upgrade notes for changed APIs before planning the cutover.

Can custom authorizer or authentication plugins be security reviewed before deployment?

Yes, and the review should happen before the plugin reaches a production broker. Kafka authorization is pluggable and can integrate with external authorization services. A plugin that replaces or extends the StandardAuthorizer sits on the trust path for every request, so review how it fails, how it treats resources without ACLs (which are restricted to super users unless allow.everyone.if.no.acl.found is set to true) and how it logs decisions. Ask for source review, dependency scanning and a test run against your TLS and SASL configuration, and confirm the review method in the quote.

What must our team provide before a Kafka customization project starts?

Provide the Kafka version and deployment model you run, access to a non-production cluster, and the source and target details for each connector or Streams application. Include sample payloads, schemas and data contracts, the security model in use (TLS, SASL mechanism and ACL approach), and your Java build standards. Name the owners who review code and approve changes, and state your change-control and test environment rules. Access requests sit outside the build team's control, so start them early.

How DeployOpen delivers this service

DeployOpen plans, builds, tests, documents and supports custom Kafka Connect connectors, Kafka Streams applications, serializers and authorizer or authentication plugins inside your own infrastructure. Each customization is scoped against the Kafka release you run or plan to run, so that the code survives planned upgrades.

How we scope

Confirm the target Kafka release line and the Java runtime your brokers, Connect workers and client applications will use
Record each connector, Streams application or plugin, with its data sources, sinks, message formats and owners
Agree in writing which plugin interfaces, serializers and authentication or authorization hooks the work will touch
Define acceptance tests, packaging format, deployment targets and handover items before build begins
Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts

How we build

Implement connectors and Streams topologies against the released Kafka APIs, keeping project code separate from Kafka internals
Build serializers and authorizer or authentication plugins with configuration that your security team can review
Test with unit tests, integration tests against a Kafka cluster you control, and failure cases such as broker restarts and rebalances
Package artifacts with pinned dependency versions and reproducible build steps stored in your source control
Run compatibility checks against the client and Connect versions your applications actually use

How we hand over

Deliver source code, build pipeline definitions, plugin artifacts and deployment instructions for your environment
Document configuration keys, topic and ACL requirements, expected error behavior and a runbook for operators
Provide an upgrade test checklist covering Kafka version changes, Java runtime changes and plugin rebuilds
Walk your engineers through the code and operations with recorded or written knowledge transfer
Confirm which items your team owns after handover and which items remain under a support agreement

How we support

Offer ongoing support, upgrades and continuous testing as a retainer after the fixed-scope delivery
Re-test custom plugins and connectors when you plan a move to a newer supported Kafka release
Review plugin and connector code when authentication, authorization or serialization requirements change
Confirm with your team the response expectations and service levels to write into any retainer, since none are assumed here
Ask for a written quote based on the agreed scope, which will list the components priced and the recurring items

Apache Kafka customization: connectors, Streams apps and plugins