Kafka Connect single message transforms

When Kafka Connect single message transforms are enough

Single message transforms (SMTs) cover per-record changes made inside Kafka Connect, before a record is written to a sink or after it is read from a source. If your pipeline needs a field renamed, a topic routed to a different name, a value masked or a little cleanup applied, an SMT is usually the right first place to put that logic. Each transform works on one record at a time, with no state carried between records and no joins across topics. The licensing question is settled: Apache Kafka is released under the Apache License 2.0 and has no paid edition from the project itself, and the transforms ship with Kafka Connect. That means the cost of using them is mostly configuration effort, review and testing, not a license fee.

Typical cases are narrow and mechanical. Teams use RegexRouter or TimestampRouter to shape topic names, ReplaceField to drop or rename fields, MaskField to blank sensitive values before they reach a lakehouse or search index, InsertField to add metadata such as a source name, and Cast or ValueToKey to align types and keys with the sink. Predicates let you apply a transform only to records that match a condition, which keeps one connector from needing several copies. Before relying on any of these, confirm the transform names, options and predicate behavior for your Kafka version in the Connect documentation, because options change between releases. Chains of transforms also run in order, so test the combined output on realistic records, including nulls and malformed values.

The boundary comes when the logic needs context beyond one record. Enrichment from a lookup table, aggregation over a time window, deduplication across records, conditional logic that depends on earlier events, or complex parsing of nested payloads usually belongs in a stream processor such as Kafka Streams or another application built on the Kafka clients. Those cases also raise testing and ownership questions that a connector configuration does not handle well. A useful check is to ask whether a change could be expressed as a settings block that a reviewer can read in a few minutes. If it cannot, plan the logic in a processor and keep the SMT chain limited to simple shaping. Also confirm which SMTs your connector plugin version supports, since some connectors ship their own transforms.

Which included transform fits which change

This table pairs each common reshaping need inside Connect with the built-in transform or predicate that usually fits it and the caution that matters most. Read each row as a first check: if the change needs logic beyond a single record, the caution column points toward a stream processor instead. The full, version-specific list of transforms and predicates is in the Kafka 4.3 documentation.

Change needed	Typical fit	Main caution
Drop or keep fields	ReplaceField to include or exclude named fields before the record leaves the connector	Removing a field a downstream consumer or primary key depends on breaks the contract silently
Rename fields	ReplaceField with a rename mapping for simple name changes on flat records	Renames apply per field name, so nested structures and repeated names need explicit testing
Extract key or value field	ValueToKey or ExtractField to lift one field into the record key or pull out a single value	Changing the key alters partitioning and compaction behavior for the topic
Route records to other topics	TopicRegex and TopicReplace to rename destination topics by pattern	Regex mistakes can send records to unexpected topics, so validate patterns against real topic names
Convert timestamps	TimestampConverter to change timestamp formats between epoch and string representations	Time zone and precision assumptions differ by source, so confirm units with a sample record
Handle headers	HeaderFrom or InsertHeader to copy or add metadata used for routing or tracing	Headers are visible to every consumer, so avoid placing sensitive values in them
Mask or replace values	MaskField to replace a field value with a null or empty placeholder	Masking is not encryption or tokenization, and the original value may still exist in upstream systems or logs
Apply transforms selectively	Predicates such as TopicNameMatches or HasHeaderKey to limit a transform to matching records	Each predicate adds a condition to test, so unmatched records pass through unchanged and need separate review
Multi-record or stateful logic	Usually not a transform fit, since SMTs act on one record at a time	Joins, aggregations and enrichment across records belong in a stream processor such as Kafka Streams

Deciding between an SMT and a stream processor

Single message transforms run inside Connect workers on one record at a time, which makes them cheap to operate but limited in what they can see. The cards below sort common transformation needs by whether they stay stateless, need other records, or carry rules that outgrow a config file, and they flag the cost of maintaining custom transform code.

Stateless per-record changes

Renaming a field, masking a value, adding a static header or dropping a null is the work SMTs were built for. Each record is handled in isolation, so the transform chain adds little operational weight. Check that every change can be expressed with the included transforms and predicates before you consider anything custom. Confirm the exact transform names and options for your Kafka release in the official documentation, since configuration keys differ across versions.

Joins, windows and aggregation

Anything that needs another record, such as enriching an order with a customer row, counting events per hour or deduplicating over a time window, needs state that spans records. An SMT cannot hold that state across records. Move this logic to a stream processor such as Kafka Streams, Flink or Spark, and keep the Connect pipeline limited to moving data in and out.

Schema-heavy restructuring

Flattening nested structs, reshaping arrays into separate rows or rebuilding a payload to match a target table can work as SMTs when the shape is fixed and simple. It becomes fragile when several schema versions arrive on one topic or when field types must change. Test each schema version against the chain, and move the restructuring upstream or downstream if the mapping grows past a few predictable steps.

Complex conditional business rules

Predicates can select records by topic, key or header, which covers routing by simple criteria. Rules with many branches, lookups against reference data or thresholds that change over time are hard to read and review as chained configuration. Treat such rules as application logic, with its own tests and release process, rather than as an ever-growing list of predicate and transform pairs.

Custom transform code is your code

A transform outside the included set is a Java class that implements the Connect transformation interface. Your team must write it, unit test it against the schemas it handles, package it as a plugin, place it on each worker's plugin path and roll it out with every connector restart. Ask who owns those tests, how versions are pinned and how a broken transform is rolled back before you choose this path.

Build cost of a processing job

A Kafka Streams or Flink job adds a separate deployment, state store sizing, checkpointing and monitoring. Those costs are real and recurring, so they should be weighed against the maintenance of a custom SMT. The fair comparison is the total effort for both paths over the life of the pipeline, including who patches, tests and on-calls each one. Ask any partner for that breakdown in writing as part of the scope.

Setting up transforms on a connector

This sequence takes a connector from a chosen transform design to a tested configuration, with checks at the sink and a confirmed error path. Each step ends with evidence you can review before the change reaches production.

Confirm connector and Connect versions
Record the Kafka Connect worker version, the connector plugin name and version, and the Java runtime on each worker. Confirm in the connector's own documentation that it supports the transforms and predicates you plan to use, because some connectors add their own transform classes or restrict which fields can be changed. Since Kafka 4.0, Connect workers require Java 17, so check that as well. Done means you have a short written record of each version and a yes or no on support from the connector documentation.
Define the required output shape
Write down the exact record shape the sink expects: field names, nesting, key format, topic names and any routing rule that sends records to different destinations. List each change as an input field, the operation applied to it, and the output field. This document is the reference for every later check. Done means a reviewer can read the list and predict the output of a sample record without opening the connector configuration.
Write transform aliases and predicates
In the connector configuration, declare each transform under a named alias with its class and settings, and list the aliases in the transforms key in the order they run. Add predicates only where a transform must apply to a subset of records, and attach each one to its transform with the predicate key. Keep the configuration in version control so changes are reviewed. Done means the configuration validates against the Connect REST API without errors and the alias order matches the sequence you wrote in the previous step.
Test on a non-production topic
Create a non-production topic and produce a sample of real records, with sensitive values masked if the data requires it. Point a test connector at that topic with the new configuration and capture the output on the destination. Run the same sample through the production configuration if one exists to compare results. Done means every record in the sample matches the output shape from step two, and any mismatch is documented with the input record that caused it.
Check converter and schema output at the sink
Read the output with the same converter and schema settings the sink uses in production, and inspect key and value schemas. Confirm that field types survive the transform, that optional fields are null where expected rather than missing, and that schema changes are compatible with what the destination already holds. If you use a schema registry, check the registered schema version after the test run. Done means the sink accepts every test record and the schema versions are the ones you expected.
Confirm error behavior for failing records
Feed deliberately malformed records through the transform, such as a missing field or an unexpected type, and observe what the connector does. Check the errors tolerance setting, whether failed records stop the task or are skipped, and whether error context is logged or routed to a dead letter queue topic. Confirm that the behavior matches the operational policy your team has agreed. Done means you have a recorded result for each malformed case and a documented decision on whether to stop, skip or route each one.
Promote the configuration with a rollback plan
Apply the reviewed configuration to the production connector, ideally through the same deployment path used for other connector changes. Keep the previous configuration available so you can restore it if the first production records differ from the test output. Watch the sink output and error metrics for the first records after the change. Done means production output matches the test results, and the rollback steps are written down and tested at least once in a non-production environment.

Security and operational risks of transform logic

Transforms run inside Connect workers, so they touch sensitive data, credentials and the topics they write to. The cards below cover the controls that keep transform logic from becoming a security or operations gap, from masking and secrets to access rights and change control.

Masking in transit only

An SMT such as MaskField changes values as records pass through Connect, but the original data may still sit in the source topic, in earlier offsets or in upstream databases. Masking that only applies in transit does not protect data at rest. Check which topics hold unmasked values, who can consume them, and whether retention on those topics matches your data classification. Treat the masked output as the only copy you are willing to store downstream.

Credentials kept out of configs

Connector configurations often carry database passwords, API tokens and keys that a transform or converter also needs. Store these in an external secrets provider and reference them from the connector config rather than writing values inline. Review whether any transform config, such as a regex or replacement value, embeds a secret that would appear in REST API responses or logs. Confirm which secret provider your Connect version supports before you depend on it.

Authentication with SSL or SASL

Kafka clients, brokers and tools can authenticate with SSL or SASL, and the documented SASL mechanisms include GSSAPI, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER. Each Connect worker and connector should use its own identity so that actions taken through a transform can be traced to a principal. Confirm that your workers do not share one account across unrelated pipelines.

Least-privilege ACLs on output topics

In KRaft clusters, ACLs are stored in the cluster metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Grant each connector write access only to the topics its transforms produce, and read access only to its source topics. Route any sensitive output to a separate topic with narrower consumer groups. Verify the ACL set after each new route or renamed topic, because a transform that writes to an unexpected topic will fail or succeed depending on what was granted.

Auditing config changes

A transform chain can be changed through the Connect REST API or a configuration file, and a single edited field can change which records are dropped, renamed or exposed. Keep connector configs in version control, require review for changes to transform and predicate settings, and record who applied each change. Compare the running config with the repository on a schedule so that drift is found quickly.

Restarts after transform changes

Changing transform or predicate settings usually requires reconfiguring the connector, and some changes require restarting its tasks before the new logic takes effect. During that window, records may be processed under the old rules or delayed while tasks restart. Plan changes in a maintenance window, confirm which tasks picked up the new config, and check the output topic for records that match the old shape. Ask your Connect team how they verify this for each connector type.

Writing custom transforms versus using a stream processor

A custom single message transform is a Java class that implements the Connect transformation interface and is packaged as a connector plugin. It runs inside the Connect worker on each record, before the record reaches the sink or after it leaves the source, so it suits per-record reshaping such as renaming a field, masking a value or deriving a key. Effort sits in the code, the packaging and the compatibility checks that follow each Kafka release.

Testing scope: Unit tests cover the transform logic, but you also need tests against the Connect converters and the schema types your connectors emit, including null values and tombstones.
Plugin packaging: The jar and its dependencies must be placed in a plugin.path directory with a layout the worker can discover, and each dependency version adds to the review burden.
Upgrade compatibility: Every Kafka upgrade requires retesting the transform against the new Connect runtime, since internal interfaces and the Java 17 requirement for Connect hosts can affect the build.
Ownership: Someone on your team owns the code for the life of the pipeline, including security review of any code you run in the worker.
Scope limit: A transform sees one record at a time, so logic that needs history or joins across topics does not belong here.

Confirm the transform's test coverage and the owner of its release process before you approve it for production.

Limits to confirm in your environment

Single message transforms depend on the Kafka version you run, the converters you choose and the connector that applies them. The cards below list what to confirm for your exact stack before you commit transform logic to a pipeline.

Transforms and predicates in your version

The set of included transforms and predicates is tied to the Kafka release you deploy, so the catalog you read in one version may differ from another. Confirm the exact transform class names, their configuration keys and any predicates you plan to use against the release your Connect workers run. Check the upgrade notes for the version you are moving to, since changes between releases can affect pipelines that already run. Record the verified list in your build documentation so reviewers can trace each transform to a version.

Behavior with your converters and registry

A transform sees the record as the converter presents it, so the same field change can behave differently with JSON, Avro or Protobuf and with or without a schema registry. Test each transform on sample records that use your real converter settings and schema evolution cases, including nulls, nested structures and missing fields. Confirm how the transform changes the schema, because a sink that depends on a fixed schema can fail after an edit that looked harmless in the value.

Overlap with connector transform support

Many connectors ship their own transform or routing options, and some apply logic that overlaps with the Connect-level transforms. Confirm which layer owns each change, whether the connector documentation supports the same transform chain, and what order the transforms run in when both are configured. Duplicate or conflicting logic across layers is hard to debug later, so keep one layer responsible for each change and document the choice in the connector configuration.

Throughput under production load

Each transform runs per record inside the worker task, so the cost adds up at high message rates and large payloads. Load test the full chain with records sized like production traffic and measure task CPU, throughput and consumer lag on the worker nodes. Include the cost of regex matching or predicates in the test, since a pattern that is cheap on one sample can become expensive at volume. Re-run the test after changing partition counts or task counts.

Rolling out changes to running connectors

Editing a transform on a running connector changes how new records are written, and it may not change records already processed or in flight. Confirm how your deployment tooling applies configuration updates, whether tasks restart, and how offsets and sink output behave during the change. Plan a rollback path that restores the previous configuration, and check that downstream consumers can accept both the old and new record shapes during the transition.

Questions teams ask before using SMTs

Do single message transforms cost extra to use?

The transforms that ship with Kafka Connect carry no license fee, because Apache Kafka is released under the Apache License 2.0 and the project offers no paid edition. The costs sit elsewhere: engineering time to design the transform chain, test it against real record shapes, and keep the configuration under version control. Connector plugins that bring their own transforms may carry different licenses, so check those separately. Ask any partner to separate one-time build work from recurring work such as upgrades and changes to upstream schemas, since those are priced differently in a quote.

How long does a typical SMT setup take, and what drives it?

The duration depends mainly on how many transforms run in sequence, how varied the incoming record shapes are, and how much of the logic can be expressed as configuration. A chain that renames fields, routes topics by pattern and masks one column is usually a short exercise. Timing grows when records carry nested or schema-less payloads, when predicates must select among many topics, or when a custom transform is required. Ask for the timeline to be tied to a written list of record types and target behavior, and confirm the test data the team will supply.

What must our team provide to a partner before the work starts?

A partner needs representative records from each source topic, including edge cases such as nulls, missing keys and nested structures. It also needs the target schema or downstream contract, the routing rules, the fields that must never leave the cluster, and the Connect version and plugin list running today. Access to a non-production Connect cluster, the team's naming conventions and a named owner for sign-off are equally important. Without these inputs, the transform design becomes guesswork and testing takes longer.

Can our own engineers build and run SMTs without outside help?

Yes, many teams do. The built-in transforms are configured in connector JSON and need no custom code, so a team comfortable with Connect can handle simple chains. Outside help is most useful when the logic needs a custom Java transform, when several connectors must share a consistent convention, or when the team wants a tested pattern and documentation before handing the platform over. DeployOpen plans, implements and documents these setups in your infrastructure under a written scope, and can support them through a retainer afterward if you choose.

When should transformation logic move into a stream processor?

Move logic out of Connect when a transform needs state, joins across topics, windowed aggregation, or multi-step branching that a single record cannot express. Transforms run one record at a time inside a connector, so they cannot enrich a record from another topic or compute running totals. Kafka Streams or another stream processor is the better place for those rules, and it can publish a clean topic that the connector then consumes. Keep simple renames, masks and topic routing in the connector, since moving them adds a service to operate.

How are custom transforms upgraded across Kafka releases?

Custom transforms are compiled against the Kafka Connect API, so each Kafka upgrade needs a check that the transform still builds, loads and behaves the same. Upgrade notes for each release describe Connect API and plugin loading changes, and the time-based release plan states that rolling upgrades are supported from the last three releases to the latest version. Confirm the target version in the Kafka downloads page, rebuild the plugin, run it against recorded sample records and keep the old and new builds available for rollback. Built-in transforms follow the same upgrade test but need less rework.

What breaks most often when transforms are misconfigured?

The most common failures come from assumptions about record shape. A transform that expects a struct fails on a schema-less string, a field rename silently drops a value when the source name differs by case, and a predicate that matches too broadly sends records to the wrong topic. Errors are often hidden until a downstream consumer rejects data. Test each transform chain against negative samples, check the error handling settings for the connector, and log the output of a dry run before enabling the task in production.

Do SMTs affect what data is protected or exposed?

They can. Masking, dropping and renaming fields inside Connect decide what reaches the destination, so a misplaced transform can expose personal data or break a retention rule. Treat the transform configuration as part of your data access design: review it with the data owner, limit who can change connector configuration, and record the change history. Compliance teams will ask for this evidence, and DeployOpen prepares configurations and records for review, though it does not issue audit reports or certifications.

How DeployOpen scopes and delivers Connect transform work

DeployOpen plans, implements, customizes, documents and supports self-hosted open-source platforms inside your own infrastructure, so transform work for Kafka Connect is scoped and built where your Connect workers already run. Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.

How we scope

Inventory each connector, its source and sink, and the record shapes it produces and consumes.
List every required transform and predicate, with the field changes, routing rules and null handling each one needs.
Mark which logic stays in Connect and which moves to a stream processor, and record the reasons in the written scope.

How we build

Write transform configuration as versioned files so each change to a connector can be reviewed and rolled back.
Test each transform chain against representative records, including malformed, null and schema-changed samples.
Confirm the Kafka Connect and Java versions in your environment before choosing transforms, since requirements differ by release.

How we hand over

Deliver a document for each connector listing its transform chain, predicates, error behavior and expected output schema.
Provide test records, expected outputs and the commands used to verify a connector after deployment.
Walk your platform and data teams through the configuration so they can change transforms without rebuilding the pipeline.

How we support

Offer ongoing support, upgrades and continuous testing as a retainer for transforms that change as sources evolve.
Review transform changes when Kafka releases or connector plugins are upgraded, based on the written scope.
Confirm with your team which support hours, response paths and upgrade windows apply before the retainer starts.