ClickHouse Kafka Connect Sink setup

How the ClickHouse Kafka Connect Sink delivers Kafka topics

The official ClickHouse Kafka Connect Sink moves data from a Kafka topic into a ClickHouse table, and it is the alternative to the Kafka table engine. It is distributed under the Apache 2.0 License and runs inside Kafka Connect, the part of Apache Kafka that streams data between Kafka and other systems through connectors. The connector does not run inside the ClickHouse server, so the work of consuming, batching and retrying sits in the Connect worker process that your team operates alongside the brokers. The connector needs Kafka Connect 2.7 or later on Java 11 or later, and it refuses to start against ClickHouse servers older than 23.3. Check these three requirements against the Kafka and ClickHouse versions you already run before you plan the rest of the build.

For a team already running Kafka, the practical consequence is that the Connect cluster becomes a new component to size, secure and monitor. Kafka 4.x brokers, Kafka Connect and tools require Java 17, so a Connect host that is current with your brokers may already meet the requirement, while an older Connect deployment may not. Connect also manages connector offsets on its own, which shifts the delivery guarantees you need to confirm to the connector configuration and the ClickHouse side. Read the exactly-once options and the deduplication behavior in the connector documentation, and confirm them against a test topic before you point production traffic at the sink. The connector documentation is the ClickHouse Kafka Connect Sink reference and should be the source for any version pairing you finalize.

What you need before installing the sink

The sink depends on three layers agreeing: a Kafka Connect runtime with its Java version, a ClickHouse server at or above the minimum the connector accepts, and a Kafka cluster your Connect workers can reach securely. Check each layer against the supported release lines before you pick versions, because the connector refuses to start against older servers.

Kafka Connect and Java versions

The sink needs Kafka Connect 2.7 or later running on Java 11 or later. If you deploy Kafka 4.x, Connect and the other tools require Java 17, while clients and Kafka Streams applications need at least Java 11. Pick the Java runtime for the Connect workers, not the producers, and confirm it against the Kafka upgrade notes before you build the worker image.

Supported Kafka releases

The Kafka downloads page lists 4.3.1, 4.2.2 and 4.1.2 as supported releases, and Kafka Connect ships with each Kafka release. Choose one of these lines for the Connect runtime so you receive bugfix releases. Plan the Connect version and the broker version together, since rolling upgrades are supported from the last three releases to the latest version.

ClickHouse server version floor

The connector refuses to start against ClickHouse servers older than 23.3, so this is a hard floor rather than a recommendation. Above that floor, choose a line from the ClickHouse security policy and release pages: the supported lines get security updates, and LTS lines are the usual choice for slower upgrade cycles. Confirm the current supported versions yourself before you select one, because support lists change.

ClickHouse account and grants

Create a dedicated ClickHouse user for the connector with insert rights only on the target database and table. Do not reuse the default user, which starts with all permissions, and do not let the sink create tables it does not need. Record who owns the grant and rotate its password through your secrets process, since the connector configuration will hold it.

Network path and TLS certificates

Confirm that Connect workers can reach the ClickHouse HTTP(S) port and every Kafka broker listener, and that firewalls allow both directions your chosen mode requires. Decide where certificates come from, whether an internal CA or a public one, and make sure the trust store used by Connect includes the CA for both ClickHouse and Kafka. Test the handshake from the worker host before you deploy the connector.

KRaft cluster and Kafka authentication

Kafka 4.x runs only in KRaft mode, so a ZooKeeper-based cluster must migrate before the Connect runtime can target it. Confirm that the Connect principal has the topic and group permissions it needs under your authorizer, since resources without ACLs are restricted to super users by default. Agree the SASL mechanism and TLS settings with the Kafka owner before you write the worker configuration.

How to configure the sink step by step

This sequence takes a self-hosted cluster from version checks to a verified pipeline that moves messages from a Kafka topic into a MergeTree table through the official Kafka Connect Sink. Each step ends with a check you can run before moving on.

Confirm versions and Java runtime
Confirm that your ClickHouse server is version 23.3 or later, because the connector refuses to start against older servers. Confirm that Kafka Connect is 2.7 or later running on Java 11 or later. If you run Apache Kafka 4.x, Connect and its tools need Java 17, and the cluster must already be KRaft-based. Record the exact versions in your change record, and confirm the ClickHouse version against the supported list in the security policy before you choose a target. Done means you have a written version matrix for ClickHouse, Kafka, Connect and Java, and every component meets the minimums.
Create a dedicated ClickHouse ingest user
Create a separate ClickHouse user that the connector uses, with INSERT rights on the target database only and no access to other databases. Keep the default user restricted, as the access control documentation recommends, and avoid reusing an administrator account. Set authentication and network restrictions according to your security baseline. Done means you can log in as the ingest user, insert a test row into the target database, and get an access-denied error when you try to write to any other database.
Create the target MergeTree table
Create a MergeTree table whose columns match the fields in the topic payload, including types and nullability. Choose the ORDER BY key with your query pattern in mind, since it cannot be changed later without rebuilding the table. On a replicated cluster, use the replicated variant of the engine and confirm the table name is the same on every node. Done means the table exists on every node you expect, and a manual INSERT of a sample row from the ingest user returns the expected row when you query it.
Install the connector plugin into Connect
Obtain the ClickHouse Kafka Connect Sink from its official distribution, which is licensed under Apache 2.0, and place it in a plugin path that your Connect workers load. Restart the workers so they scan the path. Do this on every Connect worker in distributed mode so any worker can run a task. Done means the Connect plugin list returns the ClickHouse sink class on each worker, and a worker log shows no plugin loading errors.
Create the connector through the REST API
Post a JSON connector configuration to the Connect REST interface, naming the source topic, the ClickHouse hostname, the database, the target table, and the ingest user credentials. Store the password in a secrets provider or file reference, not in plain text in the request history. Set the task count to match the topic partitions you plan to use. Done means the create call returns success, and a status call on the connector reports the connector and its tasks as RUNNING.
Verify with a test topic
Create a separate test topic, produce a small known set of messages with a fixed number of rows, and let the connector deliver them to a test table. Compare the row count in ClickHouse with the number of messages produced, and spot-check several field values against the source. Check the connector status again and read the Connect worker logs for insert errors. Done means the row counts match and the field values are correct, so you can point the connector at the production topic.
Confirm delivery semantics before production
Before you connect the production topic, review the connector's delivery and deduplication options in its documentation and test how it behaves on a forced restart and on a replayed batch. Compare the row count after the restart with the expected total. Record which guarantee you rely on and what happens to duplicates. Done means you have a documented test result showing that restarts do not create unexplained duplicate rows in your test table.

Mapping Kafka fields to ClickHouse columns

This table compares how Kafka record parts and payload fields map to ClickHouse columns when using the Kafka Connect Sink. Read each row as a decision to confirm against the connector documentation for your payload format before building the target table. The main takeaway is that key, timestamp and ordering choices are hard to change once data is loaded.

Kafka element	Mapping approach	What to confirm	Common failure
Record key	Map to a dedicated column when the key identifies an entity or deduplicates events	How the connector handles null keys and key converter type in your version	Null or inconsistent keys leave the column empty and break joins
Record value	Map payload fields to typed columns, one per field used in queries	Converter and schema settings for JSON or Avro in the connector documentation	Storing the whole payload as a string forces parsing at query time
Headers	Map only the headers you query, such as source or trace identifiers	Whether the connector exposes headers to the table mapping in your release	Assuming headers are available when the connector does not map them
Partition and offset	Store as columns when you need replay tracing or deduplication checks	Which metadata fields the connector writes by default	Offsets reset after topic recreation, so they are not stable identifiers
Nullable fields	Use Nullable types only for fields that can be absent in the source	Behavior when a field is missing or has a type mismatch	Wide Nullable columns slow queries and raise storage
Timestamp precision	Choose DateTime64 precision that matches the event time in the source	Units of the source timestamp (milliseconds or microseconds) and time zone handling	Mixed units across producers shift event times silently
ORDER BY key	Pick columns that match common filters, usually tenant and event time	Cardinality and the order you filter on most often in production queries	High-cardinality leading keys make merges and range scans costly
Schema changes	Plan new columns and type changes before producers evolve their schema	How your connector version handles added fields and schema registry evolution	Unplanned type changes cause rejected rows or dropped fields

Exactly-once options and what duplicates look like

The key decision in this section is whether your pipeline can tolerate duplicate rows, and where you will enforce uniqueness if it cannot. Kafka Connect manages connector offsets automatically and scales through Kafka's group management protocol, so the sink does not commit offsets through custom code. That automation covers the offset bookkeeping between Kafka and Connect, but it does not make the write to ClickHouse a single atomic step with the offset commit. A batch can reach ClickHouse, the worker can fail before the offset is recorded, and the same records are then delivered again after restart. Confirm the exact exactly-once settings, their supported combinations and their prerequisites in the ClickHouse Kafka Connect Sink documentation before you rely on any of them, because these options depend on the connector version you deploy.

Duplicates usually appear during consumer rebalances, worker restarts, connector task retries and replays after an offset reset or topic reprocessing. They also appear when a producer retries a send that the broker had already accepted. In ClickHouse, a duplicate is a second row with the same logical event identity, and it stays visible to queries until a background merge removes it or a query filters it out. Run a test that kills a sink task mid-batch and counts rows per event key afterward, so you measure the duplicate rate in your own environment rather than assume it.

A common design choice is to store the event key in a ReplacingMergeTree table, with the sorting key set to the columns that identify an event, and to use the version column to keep the latest copy. Treat this as a decision to validate, not a default. Deduplication happens during merges, so counts can be wrong until merges complete, and FINAL or a deduplicated view adds query cost. Test the table engine, sorting key and query pattern with representative data and replayed batches before production, and document the chosen approach so analysts know which queries need deduplication.

When the Connect Sink fits a self-hosted cluster and when the table engine does

The Connect Sink suits teams that want ingestion to run as its own service, separate from the database. Kafka Connect manages connector offsets automatically and scales through Kafka's group management protocol, so the team does not write offset commit logic. Connectors are administered through a REST interface, which fits teams that already automate Kafka operations through APIs or third-party tools. The sink also keeps ClickHouse focused on storage and queries, so a restart or upgrade of the ingestion layer does not touch the database nodes.

Offsets handled by the framework: Connect commits offsets for the team, which reduces custom code in the pipeline.
REST-driven operations: Connectors are created, paused and updated through the Connect API, so changes can be scripted and reviewed.
Separate failure domain: Connect worker failures surface in Connect logs and task status, not inside ClickHouse query paths.
Independent scaling: Connect workers can be added or moved without changing ClickHouse replicas.
Java and version prerequisites: Plan for Kafka Connect 2.7 or later on Java 11 or later, and a ClickHouse server version of at least 23.3.

Confirm with your platform team who will own Connect workers, their upgrades and their monitoring before you choose this path.

Securing the Kafka to ClickHouse path

The Kafka to ClickHouse path crosses three trust boundaries: the Kafka cluster, the Connect workers and the ClickHouse HTTP interface. Each card below covers one control on that path, and the set works only if all of them are in place together.

TLS on the ClickHouse HTTP interface

The Connect Sink writes to ClickHouse over its HTTP interface, so credentials and row data travel over that connection. Enable TLS on the HTTP port in your ClickHouse server configuration, issue the server certificate from your internal CA, and point the connector at the HTTPS endpoint. Confirm that the connector's trust store contains your CA chain, since a silently downgraded plain HTTP connection is easy to miss in testing.

Dedicated least-privilege ClickHouse user

Create a separate ClickHouse user for the connector with insert rights only on the target database and table, and no rights on system tables or other databases. The default user starts with all permissions, so disable it or restrict it before the sink goes live. Use roles and settings profiles so the grant set is reviewable as SQL, and confirm that schema changes are made by a different, human-controlled account.

Kafka SASL or SSL client authentication

Choose how the Connect worker authenticates to the brokers. The documented SASL mechanisms include SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER, alongside GSSAPI and PLAIN, and TLS can carry client certificates instead. Pick one mechanism per environment, avoid PLAIN without TLS, and confirm with your identity team whether an existing OAuth or Kerberos realm should be reused for service accounts.

Kafka ACLs under the StandardAuthorizer

In KRaft clusters, the StandardAuthorizer stores ACLs in the cluster metadata log. Grant the connector principal READ on the source topics, READ on its consumer group, and the group and topic permissions Connect needs for its internal offset and config topics. Keep allow.everyone.if.no.acl.found unset or false, because resources without ACLs are otherwise open to anyone, and test denials with a second principal before production.

Connector credentials in Connect configuration

The sink configuration holds the ClickHouse password and Kafka client secrets, and REST API responses can expose connector config to anyone with access to the endpoint. Load credentials from a secrets provider or environment variables rather than plain text in connector JSON, restrict the Connect REST interface to administrators, and rotate both the ClickHouse and Kafka secrets on a documented schedule. Check that worker logs do not print resolved configuration.

Common failures and how to fix them

This table lists the failures teams most often hit when running the ClickHouse Kafka Connect Sink, with the likely cause and the fix for each. Read it before go-live and again when a connector stops, lags or writes duplicates, and confirm exact error messages against your own logs and the connector documentation.

Failure	Likely cause	Fix
Connector refuses to start	The ClickHouse server is older than 23.3, which the connector does not accept.	Upgrade the target cluster to 23.3 or later, or plan a supported version. Check the current server version before deploying the connector.
Java version mismatch	The Connect worker runs on a Java release below the minimum required for the Kafka Connect version in use.	Use Java 11 or later for Kafka Connect 2.7 or later. If the worker runs Kafka 4.x, confirm it runs Java 17, as the Kafka upgrade notes require for Connect since 4.0.
Connector plugin not found	The connector JAR or its unpacked directory is not on the worker's plugin path, or the worker was not restarted after copying it.	Place the connector in a directory listed in the worker plugin.path setting, restart the worker and confirm the plugin appears in the REST API before creating a connector.
Schema mismatch between payload and table	Field names, types or nullability in the Kafka message do not match the ClickHouse table columns, so inserts fail or values land in the wrong columns.	Define the target table first, map every payload field explicitly, and test with sample messages from each producer version in a pre-production cluster.
Consumer lag keeps growing	Insert batches are too small or too slow, the target cluster is overloaded, or the task count is lower than the topic partition count.	Check connector task status and sink throughput, tune batch size and task count against the topic partitions, and confirm ClickHouse merges and inserts keep pace. Alert on consumer group lag.
Duplicate rows after restarts	Offsets were committed after a failure, so a batch was written to ClickHouse and then delivered again.	Confirm the exactly-once options documented for the connector and your target engine, use deduplication-friendly table design, and run a duplicate count query after each restart test.
Replicated target issues	A ClickHouse Keeper node is down or below quorum, a replica is lagging, or the connector writes to a table whose replicas disagree.	Keep at least three Keeper nodes on dedicated hosts as the replication guide recommends, check replica health and sync status, and point the connector at a healthy node or load balancer.

Questions teams ask before setting up the sink

What drives the cost of a Kafka Connect Sink setup for ClickHouse?

The main cost driver is how much design and testing the pipeline needs, not the connector itself, which is open source under Apache 2.0. Schema complexity matters most: flat event records map to a table quickly, while nested payloads, schema evolution and type conversions take longer to design and verify. The number of topics and their throughput add work for each mapping, and the delivery guarantee you require changes how much verification is needed. Security setup, covering SSL or SASL on Kafka and TLS and user access on ClickHouse, adds further effort. Ask any partner for a quote that lists each topic, its schema, the delivery expectations, the security model and the test plan as separate line items.

How long does a Kafka Connect Sink setup take?

Duration depends on the same factors as cost: the number of topics, schema complexity, the delivery guarantees required and how many environments must be validated. No duration is set by the connector documentation, so treat any timeline as something to confirm against your written scope. The work moves faster when Kafka and ClickHouse access is granted early, a non-production cluster with representative data is available, and data owners can answer questions about field meanings and retention. Delays usually come from waiting on access, unclear schemas or security approvals rather than from the Connect configuration itself.

What does the client need to provide for the work?

Expect to supply access and people, not just budget. You need a Kafka cluster with credentials that can read the target topics, a Kafka Connect runtime on Java 11 or later with Connect 2.7 or later, and a ClickHouse server running version 23.3 or later, since the connector refuses to start against older servers. A test environment with representative data and realistic message volumes is essential before production. Data owners should confirm field meanings, key formats and retention expectations. A named platform engineer should be available to approve network paths, certificates and service accounts. Confirm the exact Kafka and ClickHouse versions in your environment before scoping.

Can our own team set up the Kafka Connect Sink without a partner?

Yes, a team with Kafka and ClickHouse operations experience can build the sink in-house, because the connector is open source and documented. The questions to answer honestly are whether your team has run Kafka Connect in distributed mode, whether it can own the connector's configuration and upgrades, and whether it has time to build a restore-tested, monitored pipeline. A partner is most useful for a fixed-scope build with written deliverables, a verification plan and handover documentation. DeployOpen implements these pipelines inside your own infrastructure, so the decision is about capacity and skills, not about who controls the data.

What is the difference between the Kafka Connect Sink and the Kafka table engine?

The Kafka Connect Sink runs as a separate Connect service that consumes a topic and writes rows into a ClickHouse table, so it is managed through Connect's REST interface and its own deployment. The Kafka table engine runs inside ClickHouse and consumes topics through materialized views, which keeps the pipeline in one system but places consumption logic on the database servers. The Connect Sink suits teams that want ingestion operated and monitored alongside other Connect pipelines and that need stronger delivery control. The table engine suits small, simple pipelines managed by database administrators. Choose based on who will operate the pipeline and how strict your delivery requirements are.

What happens to the sink when Kafka or ClickHouse is upgraded?

Upgrades touch three layers: Kafka, Kafka Connect and ClickHouse. Kafka 4.x runs only in KRaft mode, and ZooKeeper-based clusters must migrate to KRaft before upgrading to 4.x. Kafka 4.x also requires Java 17 for Connect, so the runtime may need upgrading too. Upgrades of Kafka and the connector should be tested on a non-production copy before production. On the ClickHouse side, the connector refuses to start against servers older than 23.3, so a ClickHouse upgrade must keep that floor in mind. Check the changelog for each release, since minor backward-incompatible changes can occur, and confirm the connector version supports your target ClickHouse release.

Which delivery guarantees can the sink provide?

Delivery behavior should be decided before the build, because it shapes configuration, testing and monitoring. The sink's exactly-once options and their limits should be confirmed against the connector documentation for the version you deploy. Kafka Connect manages connector offsets automatically, which gives a consistent starting point, but the end-to-end outcome also depends on ClickHouse insert behavior and on how failed batches are retried. Ask for a written statement of which guarantee applies to each topic, how duplicates are detected in test, and how the team will count rows against the source. Any partner should describe this as a verification plan, not as a promise.

What security setup does the Kafka to ClickHouse path need?

Secure each hop separately. Kafka authenticates clients with SSL or SASL, using GSSAPI, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 or OAUTHBEARER, and encrypts traffic with TLS. The Connect worker needs its own service account with read access only to the required topics and consumer groups. On ClickHouse, create a dedicated user with insert rights to the target table and nothing more, and restrict the default user, which starts with all permissions. Keep credentials in your secrets store rather than in connector files. Confirm the authentication options your Kafka and ClickHouse versions support before setting the design.

How DeployOpen scopes and delivers a Kafka to ClickHouse sink

DeployOpen plans, builds, tests and documents the Kafka Connect Sink path into your self-hosted ClickHouse cluster, inside your own infrastructure. Each engagement starts from a written scope that fixes the topics, target tables, Connect runtime and acceptance tests before any build work begins.

How we scope

Inventory the Kafka topics, message formats and expected volumes that will feed each target table
Confirm the Kafka Connect and Java runtime, the ClickHouse server version and the Kafka release in use
Record the exactly-once or at-least-once delivery requirement and the authentication model for each system
Agree the written scope, deliverables and timeline in advance, since most engagements are fixed-scope

How we build

Create the ClickHouse target tables, ingestion users and roles, and restrict the default user before connector setup
Configure the sink connector, its topic-to-table mapping and TLS and SASL settings, managed through the Connect REST API
Test restarts, broker failover, schema changes and malformed records against a pre-production environment with representative data
Verify row counts and duplicate behavior end to end, and check consumer lag under expected load

How we hand over

Document the connector configuration, secrets handling and the reasoning behind the chosen delivery approach
Provide runbooks for connector restarts, offset reset, failed-record triage and ClickHouse version upgrades
Walk your operators through monitoring for connector status, lag and insert errors in their own tooling
Deliver configuration in your repositories so changes can be reviewed and rolled back by your team

How we support

Review connector behavior and changelog notes before any Kafka or ClickHouse upgrade in your environment
Ongoing support, upgrades and continuous testing are available as a retainer after handover
Run restore and replay drills on spare infrastructure so ingestion recovery is tested, not assumed
Confirm each support boundary, response expectations and included services in the written agreement