Kafka to Apache Iceberg sink connector: setup guide

How Kafka topics land in Iceberg tables with the Kafka Connect sink

If your team has Kafka topics that must be queryable in a lakehouse, the supported method is the Iceberg sink connector running on Kafka Connect, not a custom consumer you write and operate yourself. The sink reads records from topics, writes data files and commits them to an Iceberg table, so the platform team configures and runs a connector rather than maintaining bespoke commit logic. The rest of this page covers how that works in practice: catalog choices, the control topic that coordinates commits, exactly-once behavior, routing one stream into several tables and handling schema changes over time.

Apache Iceberg is an open table format for huge analytic tables. Its licensing is Apache License 2.0, and it is designed so that engines such as Spark, Trino, Flink, Presto, Hive and Impala can work with the same tables at the same time. Iceberg also provides schema evolution without rewriting tables, hidden partitioning, time travel to exact historical snapshots and rollback to an earlier table state. Those features matter for Kafka data because the table you land events in becomes a shared, versioned asset that several teams query, rather than a private copy owned by one pipeline. The Apache Iceberg website describes the format and its feature set in more detail.

Before you build, confirm three things with your own environment and the connector documentation: which Iceberg catalog your organization will run (the catalog holds table metadata and decides which engines can see the tables), which Kafka Connect and Iceberg versions you will pin together, and which engines must read the tables on day one. Those answers drive most of the later decisions. Treat the connector as an open-source component you must validate against your own catalog and storage, because this page makes no claim that any vendor certifies or supports it.

What you need before you configure the Iceberg sink

The Iceberg sink runs as a Kafka Connect connector, so the cluster, the Connect workers, the catalog, the object store and the network all have to be ready before any connector configuration is written. Each card below is a checkpoint that a failed deployment usually traces back to.

Kafka 4.x on KRaft

Run the source cluster on a supported Kafka 4.x release in KRaft mode, since Kafka 4.0 removed ZooKeeper mode. The downloads page lists 4.3.1, 4.2.2 and 4.1.2 as supported, so pick one of those and confirm the Connect plugin you plan to use supports it. Clusters still on ZooKeeper must migrate to KRaft first.

Java 17 for Connect workers

Since Kafka 4.0, Kafka Connect requires Java 17, so the worker image or host must run that runtime even if your producers run on older Java. Confirm the JDK in every worker container and in the base image used for Connect builds. Mismatched runtimes are a common reason workers fail to start after an upgrade.

Target Iceberg version and connector build

Iceberg's latest listed release is 1.12.0, but the sink's plugin jar is built against a specific Iceberg library version. Confirm with the connector project which Iceberg release its build bundles, and keep the table format version your query engines read in line with what the catalog and engines support. Do not mix Iceberg runtime jars on the same worker path.

Chosen catalog and its credentials

Decide which catalog tracks table metadata before deployment, such as a Hive metastore, AWS Glue, JDBC, REST or Nessie catalog. Create a dedicated service identity for the sink with only the namespace and table permissions it needs, and store its credentials in your secret manager rather than in connector JSON. Confirm that your query engines can use the same catalog.

Object storage access

Workers write Parquet data files and metadata into object storage, so the connector needs a bucket or prefix, write permission and a region or endpoint setting. Confirm lifecycle rules will not expire files that Iceberg still references, and confirm encryption and bucket policy requirements with your storage team before the first commit.

Network paths between workers, catalog and storage

Map the connections from each Connect worker to the Kafka brokers, the catalog endpoint and the object store, including DNS, TLS trust chains and any egress proxy or firewall rules. Test each path from the worker's own network namespace, not from a workstation, because a catalog reachable from your laptop is not necessarily reachable from the worker.

Kafka ACLs for the Connect principal

The Connect principal must read the source topics and the control topic that coordinates commits, and it needs group permissions for its consumer group. In KRaft clusters, resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set, so a missing grant will block the connector without an obvious error. Grant these before starting the connector.

Configuration sequence for the Iceberg sink connector

This sequence takes an empty environment to a running Iceberg sink that commits Kafka records to tables you can query. Each step closes with a check, so a failed step is caught before the next one depends on it.

Create the catalog and namespace
Your platform team creates the Iceberg catalog that the sink will commit to, plus a namespace to hold the tables. Choose the catalog type with your query engines in mind, because every engine that reads these tables must reach the same catalog. Confirm which catalog types your connector release supports in its documentation before you commit to one. Done when a test table can be created in the namespace from one engine and read back from a second engine.
Create the control topic and worker configuration
Provision the Kafka 4.x cluster that will host the control topic, and run the Connect workers on Java 17, which the upgrade notes require for Connect. Check the connector documentation to see whether it creates its control topic or expects you to create it. If you create it, set partition count, replication factor and retention deliberately, and give the worker's internal topics their own names and ACLs. Done when the worker REST API responds, the control topic exists with the settings you chose, and the worker reports no failed internal topic creation.
Install the connector plugin from a trusted build
Obtain the Iceberg sink connector from a source you can verify, such as a tagged release or a build your team produces from a reviewed tag. Check its checksum or signature against the published values before it enters your environment. Place the plugin directory under the worker's plugin path, then restart the workers in a controlled order. Install only the Iceberg runtime libraries that match your catalog and storage, because mismatched library versions can break commits. Done when the worker's plugin list shows the Iceberg sink and the workers start without plugin loading errors.
Define target tables and partition specs
Create the Iceberg tables before the sink writes to them, with schemas and partition specs agreed with the data owners. Iceberg supports hidden partitioning and schema evolution without rewriting tables, so partition on the columns your queries filter by. Avoid partitioning low-volume topics in ways that produce many small files. Confirm whether your connector release can create tables automatically and decide whether that behavior belongs in production. Done when the table metadata shows the intended schema and partition spec, and a query engine reports the table as empty with no snapshots.
Deploy the sink with topic-to-table mapping
Submit the connector configuration through the Connect REST API or your Kubernetes resource, mapping each source topic to its target table and setting the catalog options. Keep the configuration in version control and hold catalog credentials in your secrets store, outside the file. Check every configuration key against the connector documentation for your release before you submit it. Done when the connector and all of its tasks report RUNNING and the worker logs show a first commit attempt for each mapped topic.
Validate commits, snapshots and query results
Produce a known batch of test records to each mapped topic, then check three things. The connector's consumer group should show committed offsets that reach the produced end offsets. Each target table should list a new snapshot for each commit. A query through a second engine should return the expected row count and sample values for the batch. Record the snapshot IDs and offsets as a baseline, since Iceberg supports time travel to exact historical snapshots and rollback to an earlier state. Done when all three checks pass for every mapped topic.

How catalog choice and the control topic shape commit coordination

The Iceberg sink does not write a finished table in one step. Its tasks write data files, and a coordinator uses a control topic to agree on which files belong to each commit before the commit is made to the Iceberg catalog. Data becomes visible to readers only when that catalog commit succeeds, so the catalog and the control topic are both on the critical path. If the catalog is unreachable, or a control topic partition stalls or loses records, files may accumulate without becoming part of any snapshot, and consumers of the table will see the data late or not at all. Confirm the exact coordination behavior, including how commits are retried and how orphaned files are cleaned up, in the sink documentation for the release you plan to run.

The catalog choice decides who operates the most critical dependency and which engines must reach it. The options named in the Iceberg documentation are Hive metastore, AWS Glue, JDBC, REST, Nessie and Snowflake. A Hive metastore or JDBC catalog is typically run by your data platform team as a service you already own, with its own database backups and upgrades. Glue is a managed AWS service, and the sink's availability then depends on AWS IAM permissions and regional endpoints. A REST catalog is the most portable choice for multi-engine access, but its operator, and the API contract it implements, must be confirmed before you commit. Nessie and Snowflake bring their own versioning or hosting models, and Snowflake-managed tables are read by other engines through its Horizon Iceberg REST Catalog API. Every engine that reads the table, such as Trino, Spark, Athena or Flink, needs a supported path to the chosen catalog, so check that list before you pick one.

Before you build, confirm which catalog types the sink release you plan to deploy supports, and whether it supports the commit coordination you need for your topic count and partition layout. Ask the vendor or project for the catalog configuration keys, the required credentials and the behavior on catalog outage. Then set alerts on the control topic consumer lag, on failed commit attempts and on the age of the most recent snapshot, so a stalled commit is caught within your data freshness target rather than by a downstream report. Test a catalog failover and a broker restart during an active load in a non-production environment, and record how long tables take to show new data afterward.

Mapping Kafka records, schemas and table identifiers

This table maps each part of a Kafka record to its Iceberg counterpart: topic to namespace and table, value fields to columns, keys and timestamps to metadata fields, and converter choices to column types. Read each row as a decision point, and confirm the type rows against the connector release you plan to run before you finalize the table schema.

Source element	Iceberg target	Decision or check
Topic name	Namespace and table name	Choose a routing rule that maps each topic to one table. Confirm how the connector handles dots, hyphens and case in topic names.
Record value fields	Top-level columns	Nested structs can become struct columns. Decide early whether consumers need flattened columns, since later renames are harder to manage.
Record key	Key column or identifier field	Keys are often not part of the value. Decide whether the key is stored as a column, used for upserts, or dropped.
Record timestamp	Timestamp column	Choose between producer create time and log append time. Document the choice, since it drives partition pruning and time travel queries.
Schema registry (Avro, Protobuf, JSON Schema)	Column types and evolution	A registry gives stable field types across producers. Without one, the converter infers schemas, so confirm the inference behavior with the connector release.
Schemaless JSON	Column types from sampled or declared schema	Type inference can change between records. Declare types explicitly or validate upstream before data reaches the sink.
Logical types (dates, decimals, timestamps)	Iceberg date, decimal and timestamp types	Confirm the mapping for each logical type with the connector release, including precision, time zone handling and nullability.
Schema changes in the source	Added or renamed columns	Adding optional fields is usually simpler than renaming or changing types. Test each change type against the connector before production.

Routing one connector to several tables or one table per topic

This is the simplest model. Each Kafka topic maps to its own Iceberg table, and the sink writes that topic's records only into that table. Ownership is easy to read: the team that produces a topic owns the table, its schema and its retention. The cost is multiplied bookkeeping. Every new topic needs a table, a schema entry, and a connector configuration change, and each table carries its own metadata, snapshots and data files.

Configuration: the connector is told the topic-to-table mapping explicitly, so the mapping is visible in the connector config and easy to audit.
Partitioning: partition specs are set per table, so each table can use partitioning that fits its own query pattern.
Small files: low-volume topics produce few records per commit, so each table can accumulate many small data files unless compaction is scheduled for each one.
Adding tables: a new topic means a new table and a connector change, which is manageable for a few topics and grows as the topic count grows.
Check: confirm with your catalog and query engines how they discover new tables and how compaction jobs are scheduled per table.

Securing the Kafka to Iceberg path

The sink path crosses several trust boundaries: the Kafka cluster, the Connect workers, the Iceberg catalog and the object storage behind it. Each boundary needs its own identity, its own encryption and its own least-privilege grant, or one compromised component exposes the rest.

Client authentication with SSL or SASL

Kafka authenticates clients, brokers and tools with SSL or SASL, and security is optional, so a cluster can run with unsecured listeners unless you require otherwise. Give the Connect workers their own principal and choose a SASL mechanism your identity team can operate, such as SCRAM-SHA-512, OAUTHBEARER or GSSAPI (Kerberos). Check that no plaintext listener is reachable from the Connect network.

Per-principal ACLs on source and control topics

In KRaft clusters, the StandardAuthorizer stores ACLs in the metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Grant the connector principal read on each source topic and its consumer group, and write access on the control topic and its commit state. Keep allow.everyone.if.no.acl.found off so new topics stay closed until someone grants access.

TLS to the catalog and object storage

The connector sends metadata calls to the Iceberg catalog and data files to object storage, and both links should use TLS with certificates your team controls. Confirm that the chosen catalog and storage endpoints support TLS and how they validate client trust, and pin the internal certificate authority in the worker truststore. Test the handshake from the Connect worker host, not only from a workstation.

Least-privilege storage credentials

The connector writes Parquet files and commits table metadata, so its storage identity needs write access to the table prefixes it manages and nothing else. Scope bucket policies to specific warehouse paths, deny delete on data it does not own if your retention process does not require it, and separate catalog admin rights from the writer role. Ask the catalog and storage vendors which actions each commit path needs before you write the policy.

Secrets in Connect configuration

Connector configurations hold Kafka passwords, catalog tokens and storage keys, and the Connect REST API can display stored config values to anyone with access to it. Load credentials through a config provider backed by your secrets manager instead of inline JSON, restrict who can call the REST API, and rotate credentials on a schedule you document. Review worker logs and connector status output for leaked values before go-live.

Separating environments and principals

Development, test and production sinks should not share principals, catalog namespaces or storage prefixes, so a misconfigured test connector cannot write into a production table. Map one principal per connector and one catalog role per warehouse area, and record the mapping in your access review. Illustrative example: a staging connector that points at the production bucket is a common cause of accidental writes that ACLs alone will not catch.

Handling schema evolution without rewriting tables

Iceberg can add, rename and reorder columns through metadata changes without rewriting existing data files, so the table side of schema evolution is rarely the bottleneck. The harder work is on the Kafka side, because producers add or change fields on their own schedule and the Iceberg sink has to keep up without stopping the pipeline. Before going live, agree with the producing teams which changes are allowed to flow through automatically and which need a planned change window. The sink's schema handling depends on the converter and the Connect release you run, so confirm in the connector documentation which additions it creates as new columns and which type changes it rejects or routes to error handling.

Set the compatibility rules in your schema registry before the first topic is connected, not after the first incident. For Avro or Protobuf topics, backward compatibility lets consumers using the new schema read old data, and it is the usual default for a sink that writes to a long-lived table. Adding optional fields with defaults is typically safe under such rules, while removing a field, renaming it or changing its type is not. Write these rules down per topic, name the owner who approves exceptions, and make the registry reject incompatible registrations so that a breaking change fails in CI rather than in the Connect task log. Record which registry and compatibility mode each topic uses, because the sink can only apply what the registry allows.

Test every type change against the exact connector release and Iceberg version you plan to run in production. Widening a numeric type, changing a string to a timestamp or altering nested struct fields can behave differently across releases, and a failure may only appear after a task has already consumed records. Build a staging table fed by a replay of real topic data that includes the new shape, then confirm the table schema, the query results in your engine and the behavior of the error handling settings you chose. Plan a rollback for the schema registry and the table together, because a registry rollback alone will not undo a column Iceberg has already added.

Common failures when streaming Kafka into Iceberg

This table lists the failures most often seen when the Iceberg Kafka Connect sink runs, with the likely cause and a check you can run for each. Read it as a triage order: start with commit stalls and control topic lag, then work through access, storage and schema problems. The main takeaway is that most failures show up first as stalled commits or missing rows, so check the control topic and the catalog before changing connector settings.

Failure	Likely cause	Check or fix
Commits stalling	The catalog endpoint is unreachable or slow, so the coordinator cannot finalize a commit	Confirm the catalog responds from every Connect worker host, then review worker logs for repeated catalog connection errors before restarting the connector
Control topic lag	Workers are down, rebalancing, or a task failed, so commit requests and responses are not being processed	Check consumer lag on the control topic and the connector task states, then confirm all tasks are running and the worker group is stable
ACL denials on source or control topics	The Connect principal lacks read on source topics, or read and write on the control topic and its consumer group	Read the authorization errors in worker logs, then verify ACLs for the source topics, the control topic and the connector consumer group
Storage credential errors	The warehouse path credentials, object store keys or assumed roles are missing, expired or scoped to the wrong bucket	Test write and delete access to the table location with the same credentials the workers use, then rotate or correct the secret and restart tasks
Small-file growth	Commits happen too often or with too few records per batch, leaving many tiny data files	Compare commit interval and records per commit with file counts in the table metadata, then raise batch size or commit interval and schedule compaction
Schema mismatch errors	A record field changed type or was removed in a way the table schema does not allow, or schema evolution is disabled	Compare the producer schema with the current table schema, then decide whether to evolve the table or fix the producer before resuming the connector
Records not visible to readers	Readers use a snapshot older than the latest commit, a cached catalog view, or the connector has not committed yet	Query the table metadata for the latest snapshot and compare its timestamp with the last commit, then confirm the reader's catalog and refresh settings

Questions teams ask before streaming Kafka into Iceberg

Does exactly-once delivery hold end to end when streaming Kafka into Iceberg?

Exactly-once holds only when the Iceberg sink, the Kafka topics it reads and the catalog commit path are all configured for it, and each piece has to be verified separately. On the Kafka side, the sink reads with read_committed isolation so it never sees records from aborted transactions. The sink then commits data files to Iceberg through a coordinated commit, so a retried batch does not create duplicate rows. Ask for a test plan that injects worker failures and restarts during commits, then compares row counts and checksums in the target table against the source. Confirm the exact guarantees against the connector version you choose, since they depend on that version's documentation.

What does read_committed mean for my consumers and my producers?

read_committed tells a consumer to deliver only records from transactions that committed, and to skip records from aborted transactions. For the Iceberg sink, this means producers that write transactionally must have their transactions completed before the sink sees the data, so the sink can lag slightly behind a producer that is still mid-transaction. Producers that do not use transactions write records that are visible immediately, which means a failed producer retry can still create duplicates upstream. Teams should check which producers write transactionally and whether downstream reports can tolerate the added commit latency.

How much effort does an Iceberg sink build take, and what drives it?

Effort depends mostly on a few drivers: the catalog you choose, how often schemas change, how many topics map to how many tables, and how much security review your organization requires. A single topic with a stable schema and an existing catalog is a much smaller job than dozens of topics with evolving event payloads and a new catalog. Ask any partner, including DeployOpen, for a written scope that lists the topics in scope, the catalog and storage backends, the schema rules and the verification tests. Fixed scope is agreed in writing before work starts, so the quote follows from that document rather than from a generic estimate.

Which catalog should we choose for Iceberg tables?

The catalog holds the table metadata and the pointer to the current snapshot, so the choice affects every engine that reads the data. Common options named in the Trino Iceberg connector include the Hive metastore, AWS Glue, JDBC, REST, Nessie and Snowflake, selected with iceberg.catalog.type. Choose based on which query engines you already run, whether you need a REST catalog that several engines can share, and who operates the catalog service. Confirm each engine's support for your chosen catalog before committing, because engine coverage varies by version.

What must our team provide before the build starts?

The team must provide access to the Kafka cluster and the topics in scope, including the service identity and the ACLs that let the sink read them. You also need a catalog to operate or reach, object storage for Parquet data files, and credentials for both. Schema decisions come from your side too: which fields become table columns, how nested payloads are handled, which field defines partitioning, and who approves changes. Finally, name a contact who can approve security reviews and give access to a non-production environment for testing before production cutover.

Can our in-house team build this without a partner?

Yes, a capable platform team can build an Iceberg sink in-house, and the Iceberg and Kafka projects are both open source under the Apache License 2.0. The harder parts are rarely the first deployment. They are the failure tests, the catalog operations, the schema change process and the security evidence. Assess your team on Kafka Connect operations, Iceberg table maintenance and the time available for testing. A partner's value is the written scope and test evidence, and your team can use the same checklist to judge whether it can deliver those internally.

What recurring work follows after the sink is running?

Recurring work includes upgrading the connector plugin, tracking Kafka and Iceberg releases, and handling schema changes as producers evolve. Kafka's supported releases and upgrade path are published on its downloads page and in its time-based release plan, and Iceberg's latest release is listed on its releases page, so set a review cadence for both. Iceberg tables also need compaction and snapshot expiration to keep small files and metadata under control, and those jobs must be scheduled and monitored. Ask for a runbook that names who performs each task and how often.

How DeployOpen works on this

DeployOpen plans, builds, documents and supports the Iceberg sink integration inside your own infrastructure, with the Kafka cluster, Connect workers and lakehouse catalog all under your control. Each engagement starts from a written scope so the catalog choice, table layout and delivery guarantees are agreed before any connector is configured.

How we scope

Inventory the Kafka topics, record formats and expected volumes that will land in Iceberg tables
Agree the catalog choice, storage location and which query engines must read the tables
Define the table routing rules, schema evolution expectations and the dead letter handling for bad records
Record scope, deliverables and timelines in writing before work starts, as engagements are fixed-scope

How we build

Deploy Kafka Connect workers with the Iceberg sink plugin installed from a trusted, version-pinned source
Configure the control topic, commit coordination and read_committed settings for exactly-once delivery
Set up catalog credentials, storage access and network paths to your lakehouse under least-privilege ACLs
Test commits, failure recovery and schema changes against representative topics before go-live

How we hand over

Document connector configuration, table layout, catalog settings and the reasoning behind each decision
Provide runbooks for restarts, offset checks, snapshot expiration and rollback to an earlier table state
Walk your platform, data and security teams through monitoring signals and the handover checklist
Deliver configuration in your repositories so your team can review and change it independently

How we support

Plan Kafka, Connect and Iceberg library upgrades as a tested change with a rollback path
Review connector licensing and plugin versions whenever the sink or its dependencies change
Offer ongoing support, upgrades and continuous testing as a retainer after handover
Confirm with your team which evidence and access records your data governance program requires

Streaming Kafka topics into Apache Iceberg tables