When the Apache Camel Kafka Connector is the right connector to use
Use the Apache Camel Kafka Connector when you need Kafka connected to a system that has no dedicated Kafka Connect connector, or only a connector you cannot license, support or run. The Camel Kafka Connector packages Apache Camel components as Kafka Connect source and sink connectors, so one plugin family can move data between Kafka and systems such as RabbitMQ, JMS brokers, MQTT and Amazon S3. Before you commit, check the current component catalog and the release you plan to run in the Camel documentation, because the set of supported components changes between releases. Confirm the exact catalog and version in that documentation and in your own environment rather than relying on a count or list from a third party.
The decision is mostly about where the integration logic should live. A dedicated connector usually exposes the options that matter for its target system, such as delivery semantics, schema handling and error behavior, and it is maintained by the team that knows that system best. A Camel-based connector is a good fit when the target protocol is well covered by Camel components and your team already runs Camel routes, or when a single connector family covers several systems you need to reach. Ask the connector's maintainers which delivery guarantees, offset handling and failure modes apply to your specific source or sink, and compare that answer against the Camel component's documented behavior. Test a representative message flow before you treat the choice as settled.
Treat the Camel connector as a plugin you must license, version and verify like any other Connect plugin. Each connector ships with its own Camel component versions and dependencies, so check the license of the connector and of each underlying component, and pin the versions you test. Plan a staging run that covers a poison message, a broker outage and a restart of the Connect worker, and confirm that offsets, retries and dead letter routing behave as your design expects. Compare the connector's logs and metrics with what your operations team already monitors in the Connect cluster before moving to production.
What you need before installing Camel connectors on Kafka Connect
A Camel Kafka connector runs inside Kafka Connect, so the Kafka release, the Java runtime, the plugin layout and the external system's access path all have to line up before the first connector starts. The cards below cover the decisions to make and the checks to run on each side of the connection.
A supported Kafka release
Choose one of the supported Apache Kafka releases listed on the downloads page: 4.1.2, 4.2.2 or 4.3.1. Older releases are archived and receive no fixes. Confirm in the Camel Kafka connector release notes that the connector version you plan to use is tested against the Kafka version you pick, since the Camel project does not guarantee compatibility with every Kafka release. Plan the Kafka version first, then choose the connector release to match it.
Java 17 on Connect workers
Since Kafka 4.0, brokers, Kafka Connect and tools require Java 17. Camel connector plugins run inside the Connect worker JVM, so every worker that loads them must run Java 17 or later, and the image or host used for each worker must be built with that runtime. Check the runtime on every worker node and container image, not only on the brokers, and record the JVM version in your build pipeline so it stays visible during upgrades.
KRaft mode for the cluster
Confirm the cluster runs in KRaft mode before adding Connect. Kafka 4.0 removed ZooKeeper mode, and the upgrade notes require ZooKeeper-based clusters to migrate to KRaft first. Connect stores its own configuration, offsets and status in internal topics, so these must use a replication factor that survives a broker loss. Decide these replication settings and the Connect group ID before the first worker starts, because changing them later is disruptive.
Plugin path access for workers
Camel connector JARs must be placed in a directory listed in the worker's plugin.path setting, with each connector in its own subdirectory so its dependencies do not collide with other plugins. Confirm that the worker user can read those directories, that the same plugins are present on every worker in the group, and that the path is not writable by application users. Only install connector code your team has reviewed and approved.
External endpoint, credentials and trust
Gather the endpoint address, port, virtual host or path, and the service account the connector will use, such as a RabbitMQ user, JMS broker credentials or an S3 access key. Store the secrets in your secrets manager or a Connect config provider, not in connector JSON committed to version control. Obtain the CA certificate chain the external system presents, and test TLS from a worker host with the same trust store the worker will use.
Network paths from workers
Connect workers open outbound connections to the external system and to the Kafka brokers. Map each path, including the source or sink port, DNS names, proxies and egress rules in Kubernetes network policies or cloud security groups. Test each path from the same host or pod that will run the worker, since a laptop test does not prove that the worker can reach the system. Record the owner of each firewall rule so changes are reviewed before a connector fails in production.
How to configure a Camel source or sink step by step
This sequence takes a Camel Kafka connector from selection to a verified running pipeline inside Kafka Connect. Each step closes with a check, so you know the plugin loaded, the connector has access, and data moves in the expected direction before the pipeline is treated as ready.
Select the Camel connector class and version
Identify the Camel Kafka connector you need (for example, the RabbitMQ, JMS, MQTT or S3 source or sink) and confirm its connector class name and version in the Camel Kafka connector catalog. Check that the Camel version it bundles is compatible with your Kafka Connect runtime, and confirm the connector's license terms before you depend on it. Done when you have a written record of the class name, version, and the Kafka release it was tested against.
Install the plugin into plugin.path
Unpack the connector archive into a dedicated directory listed in the Connect worker's
plugin.pathsetting, keeping each connector in its own folder so its dependencies do not collide with other plugins. Restart the workers so they scan the new path. Done when the connector class appears in the worker's list of installed connector plugins when you query the Connect REST API.Create a dedicated Kafka principal and ACLs
Create a separate Kafka principal for this connector, using the SASL mechanism your cluster is configured for, and grant it only the topic, consumer group and cluster permissions the connector needs. In KRaft clusters, ACLs are stored in the cluster metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true, so confirm that setting before you test. Done when a client authenticated as the new principal can read or write its designated topic and is denied on others.
Store endpoint credentials outside connector JSON
Place the external system's passwords, keys and tokens in a secrets store or a Connect config provider, and reference them from the connector configuration instead of pasting values into the JSON. Confirm with your platform team which secrets mechanism your Connect deployment supports. Done when the connector configuration contains only references, and a search of the configuration file and the REST response finds no plaintext credentials.
Create the source or sink connector
Submit the connector configuration through the Connect REST API, setting the connector class, the topic mapping for a sink or the target topic for a source, the converters, the error handling settings, and the principal's client properties. Keep the first version minimal and add transforms later. Done when the create request returns successfully and the connector appears in the list of connectors on the worker.
Verify connector and task state
Query the connector status endpoint and confirm the connector state and every task state report RUNNING. If a task is FAILED, read its trace field, fix the cause (usually a credential, network, or ACL issue), and restart the task rather than recreating the connector. Done when the connector and all tasks report RUNNING and the worker logs show no repeated authentication or authorization errors.
Send and read a test message
For a sink, publish a known test record to the input topic and confirm it arrives at the external system with the expected content. For a source, produce or trigger a test event in the external system and read the topic with a consumer running under an approved principal. Done when the test record appears in the expected place with the expected payload, and the test is recorded so you can repeat it after upgrades.
Camel connector or a dedicated connector: how to decide
When a connector built for the specific system exists, is maintained by its project or vendor, and carries a license your organization accepts, it is usually the first choice. Its documentation, defaults and error handling are written for that one system, and its maintainers are the first point of contact for defects. Confirm that the project is still active before you commit, because a connector with no recent releases can become a long-term risk.
- Support path: Confirm who answers defects, where security reports go and whether a vendor offers paid support for the connector.
- Licensing review: Read the license file in the connector's repository and check whether any features sit behind a separate commercial license.
- Error behavior: Test malformed records, timeouts and retries with your own data instead of relying on the defaults.
- Ownership: Assign who upgrades the plugin, tracks vulnerabilities and rebuilds the Connect image when Kafka moves to a new release.
Before adoption, check the license, the date of the latest release and whether the connector supports the Kafka Connect version you plan to run.
Apache Camel Kafka connectors give one configuration pattern for many systems that have no dedicated connector, including messaging brokers, file systems and object stores. You gain breadth, but each underlying Camel component has its own configuration model, maturity level and maintenance pace. Treat the Camel connector as a third-party plugin that must pass the same checks as any other connector, and check the component behind it as well as the wrapper.
- Breadth versus depth: A component may cover the protocol but expose fewer tuning options, delivery guarantees or schema handling than a dedicated connector.
- Licensing review: Confirm the license of the Camel Kafka connector and of each Camel component it uses, since both carry obligations.
- Error behavior: Verify how failed messages are acknowledged or redelivered, because the source system's delivery semantics may differ from Kafka's offset model.
- Versioning: Pin the connector to a version you have tested and confirm it matches your Kafka Connect runtime and the Camel release it was built against.
- Ownership: Your team usually carries the upgrade and vulnerability work unless a vendor contract says otherwise.
Run the connector in a staging environment with injected failures before production use, and record the delivery behavior you observed.
When no maintained connector and no suitable Camel component fits, the choice is between a custom Kafka Connect connector and an application or stream processor that produces to or consumes from topics directly. A custom connector gives you Connect's worker lifecycle, offset tracking and scaling, but your organization then owns the code for its whole life. A stream processor is often simpler when the requirement is transformation or routing rather than moving data between systems.
- Connector versus application: Choose a custom connector when you need Connect's offset management and distributed workers, and an application when the logic is business processing.
- Licensing review: Your own code avoids third-party terms, but every library it bundles still needs a license review.
- Error behavior: You define retry, dead letter and idempotence behavior yourself, so document it and test it before go-live.
- Ownership: Plan for build, testing, security review, documentation and maintenance by named engineers, not a one-time project.
- Trusted code: Install only plugins you have built or reviewed, and keep them in your plugin path under change control.
Before writing new plugin code, check whether an existing connector combined with Kafka transforms or a stream processor meets the requirement.
How Camel connector configuration differs from other Connect plugins
The Camel Kafka connectors do not follow the flat key and value style that most other Kafka Connect plugins use. Each connector is configured around a Camel endpoint URI, which names the external system and carries its connection options, alongside camel.* properties that control how the connector behaves inside Connect. The practical consequence is that a reviewer reading a connector JSON file has to understand two layers: the endpoint string that addresses the external queue, topic or bucket, and the connector-level settings that decide how records are converted and delivered. Before you copy a configuration from an example, confirm the exact endpoint syntax and every camel.* key against the property reference for the connector version you have installed, because these names differ between connectors and between releases. Do not assume a key from one Camel connector applies to another.
Mapping is the second point to plan. A Camel source turns each message it consumes from the external system into a Camel exchange, then into a Kafka record, and a sink reverses that path. Headers, bodies and message identifiers may land in different parts of the Kafka record than you expect, and the choice of key and value converter determines whether the body is stored as raw bytes, a string or structured data. Test one representative message end to end in a non-production cluster and inspect the resulting record, including headers, before you write downstream consumers or schema rules against it. Ask the implementer to document which converter is used for keys and values and why, because a converter change later can break every consumer that reads the topic. Secrets need a deliberate design from the start. Broker credentials, certificates and tokens for the external system should be referenced through Kafka Connect config providers, so the worker resolves them at runtime instead of storing them in connector JSON that gets submitted through the REST API, committed to version control or shown in a support ticket. Confirm which config provider implementations ship with your Connect distribution and which ones your secret store requires, and check that the Camel connector you select accepts placeholders in the fields that hold passwords and keys. Set file permissions on the secret source and rotate credentials through the same path, not by editing running connector definitions by hand.
How Camel message fields map to Kafka records
This table traces where each field of a message from a Camel-supported source lands in a Kafka record: value, key, headers, topic and timestamp. Read each row with the converter column, because the converter choice often decides whether a field arrives intact. Confirm every row against the documentation for your connector version before you build on it.
| Source field | Kafka record target | Converter or schema choice | Confirm in connector documentation |
|---|---|---|---|
| Message body | Record value | Byte array or string converter for raw payloads; JSON or Avro converter for structured bodies | Which converter the source expects and whether it passes the body through unchanged |
| Message headers | Record headers | String values are the most portable; confirm how the header converter serializes them | Which headers are copied and whether Camel-internal headers are filtered out |
| Routing key or queue name | Record key or topic selection | String key converter is typical; key may be unset if the source does not provide one | Whether the routing key becomes the key, a header or a topic selector |
| Source destination (exchange, queue, subject) | Target Kafka topic | Not applicable; topic naming is a configuration choice | Whether the topic is fixed per connector or derived per message, and the naming template |
| Message timestamp | Record timestamp | Not applicable; timestamp type is set on the topic | Whether the source sets the record timestamp from the message or uses ingestion time |
| Message identifier or correlation ID | Record key or header | Depends on the chosen key and header converters | Whether the source exposes a message ID and whether redelivery can produce duplicate records |
| Content type or encoding header | Value converter selection | Use to pick a converter only if the connector documents that behavior | Whether the connector reads the content type header or ignores it |
| Illustrative: JMS text message with a correlation ID | Value as string, correlation ID as header | String converter for the value, string converter for the header | Whether the JMS message type maps to a string value without transformation |
Securing the connector identity, credentials and data path
A Camel Kafka connector runs as a Kafka Connect worker that holds credentials for both the Kafka cluster and the external system it reads from or writes to. The controls below cover who the worker is on Kafka, how its secrets are stored, how the external connection is trusted, and who can change what runs.
Dedicated Kafka principal per worker
Give each Connect cluster its own Kafka principal rather than sharing an admin or application identity. Grant that principal only the topic, consumer group and Connect internal topic permissions its connectors need. In KRaft clusters, ACLs live in the metadata log, and resources without ACLs are limited to super users unless allow.everyone.if.no.acl.found is enabled, so confirm that setting stays false in production. Check the grants with a test connector before you go live.
SASL or TLS on the listener
Kafka supports SSL for encryption and SASL for authentication, and security is optional, so a cluster can run with unsecured or mixed listeners. Confirm that the listener the Connect worker uses requires authentication and encrypts traffic, and that no plaintext listener is reachable from the Connect network. Choose a SASL mechanism your identity stack supports, such as SCRAM or OAUTHBEARER, and record the choice in the design document.
Secret handling for broker and endpoint credentials
Connector configurations carry broker passwords, keystore paths and endpoint credentials. Keep these out of connector JSON stored in version control and out of plain REST responses. Use an external configuration provider or a secrets manager that the worker resolves at runtime, and rotate credentials on a schedule your team can execute. Ask the connector maintainer which configuration fields are treated as sensitive and whether they are masked in status output.
TLS trust for the external system
Each Camel component connects to a system outside Kafka, such as a JMS broker, RabbitMQ, an MQTT broker or an S3 endpoint. Decide which certificate authority the worker trusts for that endpoint, and keep that trust store separate from the Kafka keystore. Verify hostname validation is on and that certificate expiry is monitored, because a failed handshake will stop the connector and may look like a Kafka problem.
Restricting connector changes through REST
The Kafka Connect REST API can create, pause, restart and delete connectors, and it accepts configuration that points the worker at new endpoints. Place the REST interface on an internal network, require authentication in front of it, and limit write access to a small group who manage connectors through reviewed change requests. Log every create and update call so that a changed endpoint or credential can be traced to a person.
Limiting what a connector can reach
A connector can read or write any topic its principal may access, so the Kafka ACLs define the blast radius of a misconfigured or compromised connector. Review topic patterns in connector configuration, since a broad regular expression can match topics that were never intended for that pipeline. Confirm that sink connectors cannot write to control or internal topics owned by other teams.
Common Camel connector failures and how to fix them
This table lists the failures teams most often hit when running Camel Kafka connectors, with the likely cause and a first fix for each. Read it by matching the symptom you see in Connect to the row, then working through the cause before changing configuration. Confirm version-specific behavior against the connector release notes for your deployed version.
| Failure | Likely cause | First fix |
|---|---|---|
| Plugin not discovered after install | Connector JAR or its dependencies are not under a plugin.path directory, or the worker was not restarted | Check the plugin.path layout on every worker, restart workers, then confirm the connector class appears in the plugin list |
| Connector stuck in FAILED or restarting | Bad configuration value, unreachable external endpoint or an unhandled exception in the task | Read the task trace in the Connect REST status output, correct the failing key, then restart the task rather than recreating the connector |
| Authentication or TLS error to the external system | Wrong credentials, expired certificate, missing truststore entry or wrong endpoint scheme | Test the endpoint from a worker host with the same credentials and truststore, then update the secret the connector reads |
| ACL denial on Kafka topics | Connector principal lacks Read, Write or Describe on the topic, or the consumer group | Grant the minimum operations the connector needs on its topics and group, and confirm the principal in the authorizer logs |
| Converter mismatch | Source produces one format (for example JSON or Avro) while the sink expects another | Set key and value converters to match the data, and confirm schema settings on both sides of the pipeline |
| Duplicate messages after restarts | Offsets were not committed or the external write is not idempotent | Check offset commit history, use idempotent writes or upserts where supported, and test a restart in staging |
| Version drift after a Kafka upgrade | Connector build targets an older Connect API or Java runtime than the upgraded cluster | Confirm the connector version and Java runtime against the Kafka upgrade notes, then upgrade connectors in a staged rollout |
Questions buyers ask about Camel Kafka connectors
What drives the cost of a Camel Kafka connector project?
The main cost driver is the number of endpoints you need to connect and how well each one is understood. Each source or sink has its own configuration, authentication model, error behavior and message format, so the work grows with the number of systems and the number of distinct flows. Costs split into one-time work (requirements, configuration design, testing, deployment and handover) and recurring work (patching, version upgrades, monitoring and support). If you engage DeployOpen, the written scope sets the price, and it should name the endpoints, flows, environments and deliverables. DeployOpen does not publish a price list. Ask any partner to itemize one-time and recurring effort separately so you can compare it with your own team's time.
How long does a first Camel Kafka connector take to reach production?
The answer depends on the endpoint, the access path and how quickly the owners of each system can approve changes, so no single duration applies. A simple, well-documented endpoint in a test environment can move quickly. A first connector usually takes longer because the team also has to set up Kafka Connect workers, plugin packaging, secrets handling, monitoring and a rollback path. Plan separate time for network approvals, credential issuance and a load or failure test before production. Ask for a timeline with milestones in the written scope, and check which of those milestones depend on other teams.
What does our team need to supply before a connector project starts?
Before work begins, the team should supply four things for each endpoint. First, the endpoint details, such as hostnames, ports, queue or topic names, file paths or bucket names, and the message format. Second, credentials or a secure way to issue them, with a rotation owner. Third, network access from the Connect workers to the endpoint, including firewall rules and DNS. Fourth, named owners on both sides who can answer questions about message meaning, error handling and change windows. Missing owners are one of the most common causes of delay, because connector behavior often depends on details only the system owner knows.
Can our internal team build and run Camel Kafka connectors alone?
A capable internal team can do this work if it has Kafka Connect experience, Java or JVM operations skills, and access to the endpoints it must integrate. The skills most often missing are Connect worker operations, plugin packaging and version management, and testing failure paths such as broker outages or malformed messages. Ask whether someone on the team will own the connector after handover, since a connector without an owner tends to drift out of date. A partner can help with design, build and handover, and the written scope should name who runs each later step. Decide which work stays internal before you choose a partner.
How do we handle version upgrades for Camel Kafka connectors?
Treat the connector, the Camel components it uses, and the Kafka Connect runtime as three versioned items that must be tested together. Kafka itself has its own upgrade rules: the Kafka downloads page lists the supported releases, and the time-based release plan states that rolling upgrades are supported from each release in the past year to the latest version. Confirm the Camel Kafka Connector release notes for each upgrade, and keep a staging environment that matches production. Record which connector version runs on each cluster, and plan a rollback path before each change. Pin versions in source control so upgrades are deliberate rather than accidental.
Do Camel Kafka connectors support exactly-once delivery?
Support depends on the specific connector and its configuration, so confirm it for each connector you plan to use before you design around it. Do not assume that a Camel source or sink gives exactly-once behavior because Kafka Connect offers it in general. Check the connector's documentation for its delivery guarantees, and check what Kafka Connect requires from the cluster and the sink. Our exactly-once delivery topic explains the read_committed setting for sinks, the requirements for exactly-once source support and the connectors that can use it. Design duplicate handling for any endpoint that cannot give you that guarantee.
What should we check before a Camel Kafka connector goes to production?
A production check should cover several areas. Confirm that the connector runs on a supported Kafka release and a supported Java version, and that the plugin version is pinned. Check that credentials are stored in a secrets manager, that TLS is enabled to the endpoint where the endpoint supports it, and that Connect worker access is limited. Test failure paths such as an endpoint outage, a malformed message and a broker restart, and confirm that the error handling route works. Set up monitoring for connector status, task failures and consumer lag, and document who responds to alerts. Review the results with the endpoint owner before go-live.
Which parts of a Camel Kafka connector are security-sensitive?
The most sensitive parts are the credentials the connector holds, the network path it opens into the endpoint, and the data it moves. Camel connectors often carry credentials for queues, file stores and message brokers, so store them in a secrets system, limit who can read Connect configuration, and rotate them on a schedule your security team sets. Restrict the Connect workers to the network paths they need. Decide how messages containing personal or regulated data are masked or excluded before they reach topics. Confirm with your security team which of these controls apply to your environment before you approve the design.
Production readiness checks before a Camel connector goes live
Before a Camel Kafka connector carries production traffic, run a tested restart and rebalance on a staging copy of the Connect cluster. Stop a worker mid-stream, restart the connector, and confirm that tasks are reassigned, offsets resume without a gap, and the sink or source reports RUNNING on every task. Repeat the exercise against the JMS, RabbitMQ, MQTT or S3 endpoint going down and coming back, because the broker-side behavior (acknowledgements, redelivery, prefetch) differs by system and determines whether duplicates or losses appear. Confirm the expected behavior with the owners of the source or target system, not only with the connector documentation.
Set up lag and error monitoring before the first production run, not after the first incident. For sinks, track consumer group lag on the topics the connector reads. For sources, track the rate of records produced against the rate the external system emits. Alert on connector and task state changes and on error counts from the Connect worker logs. Confirm which error-handling settings the connector exposes, such as errors.tolerance and the dead letter queue topic, and decide whether a bad record should stop the task or be routed aside. Also run a load test at expected message rates, including peak bursts, and confirm that lag recovers within the window your consumers can tolerate.
Pin the connector version in your deployment manifests and record the Kafka release it was tested against. The Kafka downloads page lists 4.3.1, 4.2.2 and 4.1.2 as supported releases, and the Kafka time-based release plan supports rolling upgrades from the last three releases to the latest. Run the upgrade test against each supported Kafka release you may move to, and confirm the connector's compatibility statement for that release. Review the connector's license before go-live, because Camel components carry their own license terms and dependencies may add obligations. Finally, write a runbook for replaying or rerouting failed records, naming the owner, the source of truth for offsets, and the steps to reprocess without creating duplicates in the target system.
How DeployOpen scopes and delivers Camel Kafka connector work
DeployOpen approaches Apache Camel Kafka connectors as a scoped engagement: first confirming the source or sink, the Camel component and the Kafka Connect runtime it will run in. Work is agreed in writing before build starts, then delivered and documented inside your own infrastructure.
How we scope
- Confirm the source or sink system, its protocol and the Camel component that reads or writes it
- Record which Kafka Connect runtime, Kafka version and Java version the connector must run on
- Agree in writing which topics, message formats, error handling and credentials are in and out of scope
- Define acceptance tests and the production readiness checks the connector must pass before go-live
How we build
- Configure the connector in your environment with explicit settings for each endpoint and converter
- Pin the connector and Camel versions and document how plugins are packaged and loaded into Connect
- Test failure modes such as broker outages, malformed messages and endpoint restarts against agreed criteria
- Review connector licensing and transitive dependencies before the plugin is approved for your environment
How we hand over
- Deliver configuration files, deployment manifests and a runbook describing each connector's expected behavior
- Document the upgrade path for the connector, Camel and Kafka Connect, including the rollback steps
- Walk your platform and integration teams through monitoring, error topics and day-to-day troubleshooting
- Provide a record of test results and open risks so your team can make the next decision independently
How we support
- Fixed-scope engagements end at the agreed handover, with any follow-on work scoped separately in writing
- Ongoing support, upgrades and continuous testing are available as a retainer if your team wants them
- Retainer scope can cover connector version updates, Camel security fixes and periodic readiness reviews
- Ask any provider to confirm which connector versions and Kafka releases their support covers before signing
