Kafka becomes shared infrastructure quickly
Apache Kafka stores streams of records in partitioned, replicated topics. Producers write without knowing who will read, and each consumer group reads at its own pace from retained data. Kafka Connect moves data between Kafka and other systems, and Kafka Streams is a client library for stream processing. The project is licensed under Apache-2.0.
Typical uses include event distribution between services, change data capture, buffering for log and telemetry pipelines, and integration feeds read by several teams. Once a few teams depend on a cluster, topic naming, ownership, retention and schema contracts matter as much as broker count. Agree on them before the first production topic, because renaming and repartitioning later affect every producer and consumer.
Components to plan
Give each component an owner, a version and a recovery path.
Brokers
Store partition replicas on local disks and serve clients. Spread them across racks or zones and set broker.rack so replicas land in different failure domains.
KRaft controllers
A Raft quorum holding cluster metadata, typically three nodes (tolerating one failure) or five (tolerating two). Separate controllers from brokers in critical environments.
Kafka Connect
Worker clusters running source and sink connectors. Each connector plugin has its own version, license and failure modes, so keep an approved inventory.
Schema registry
Stores record schemas and enforces compatibility between producers and consumers. Apicurio Registry is one Apache-2.0 option.
MirrorMaker 2
Replicates topics, configurations and consumer group offsets between clusters. It runs on Kafka Connect and ships with Apache Kafka.
Plan partitions, replication and retention
These settings drive disk, network and consumer parallelism. Review them per topic with the owning team.
| Setting | What it controls | Plan for |
|---|---|---|
| Partition count | The limit on parallel consumers within one group. Ordering holds only within a partition. | Adding partitions later changes which partition a key maps to. |
| Replication factor and min.insync.replicas | How many copies exist and how many must confirm a write sent with acks=all. | Three replicas with two in sync required is a common starting point for important topics. |
| Retention | How long or how much data a topic keeps. | Disk is roughly ingest rate times retention times replication factor, plus headroom. |
| Cleanup policy | Delete removes old segments; compact keeps the latest record per key. | Compacted topics suit state and reference data; consumers must handle tombstones. |
| Tiered storage | Moves older segments to remote storage while recent data stays on brokers. | Production-ready since Kafka 3.9. Needs a remote storage plugin for your object store and testing of reads from old data. |
Move off ZooKeeper before upgrading to Kafka 4
Kafka 4.0, released in 2025, supports only KRaft mode; ZooKeeper mode has been removed. Clusters still on ZooKeeper must migrate to KRaft before upgrading to 4.x. Kafka 3.9 is the last release line with ZooKeeper mode and serves as the bridge: move to 3.9, migrate metadata into a KRaft controller quorum, finalize the migration, then upgrade. The 4.0 upgrade also requires software and metadata version 3.3.x or later, and older clusters may need an intermediate step before 3.9.
Check runtimes and clients at the same time. In Kafka 4.0, brokers, Connect and tools require Java 17, while clients and Kafka Streams require Java 11 or newer. Clients, including Streams and Connect, must be on version 2.1 or higher before brokers move to 4.0. An old client in a forgotten service can block the upgrade, so inventory them early.
Rehearse the migration on a staging cluster with production-like topics, ACLs and load. Set checkpoints with a rollback decision at each, since returning to ZooKeeper is no longer possible once the migration is finalized. Retire the ZooKeeper ensemble after the KRaft cluster has run through normal operations and a rolling restart.
Choose where brokers run
Dedicated hosts with local or block storage, managed through configuration management. This gives direct control over disks and network. Rolling restarts, certificate renewal and broker replacement need scripted runbooks.
The Strimzi Cluster Operator deploys and upgrades Kafka from custom resources, and the Topic and User Operators manage KafkaTopic and KafkaUser resources. It suits teams already running stateful workloads on Kubernetes.
Mirror distributions, images and connector plugins into an internal repository. Plan the certificate authority, schema registry and monitoring inside the boundary, and review new connector versions before import.
Secure listeners, identities and ACLs
Kafka carries data from many systems, so every client needs its own identity and only the access it uses.
| Layer | Approach | Check |
|---|---|---|
| Listeners | Separate listeners for clients, inter-broker traffic and controllers, each with TLS. | No plaintext listener is reachable outside the cluster network. |
| Authentication | Mutual TLS, SASL/SCRAM, SASL/GSSAPI for Kerberos, or SASL/OAUTHBEARER against an OpenID Connect provider such as Keycloak. | Each application has its own principal; shared credentials are retired. |
| Authorization | ACLs per principal, with prefixed ACLs for team topic namespaces. Access is denied when no ACL matches. | A test principal cannot reach topics outside its grant. |
| Data at rest | Kafka does not encrypt log segments itself, so use encrypted disks on brokers. | Encryption is confirmed on every broker and on remote storage for tiered data. |
| Administration | Restrict admin tools and Connect REST endpoints to operators. | Topic deletion and ACL changes are limited to named roles and logged. |
Plan recovery beyond in-cluster replication
Partition replication covers broker and disk failure inside one cluster. It does not help with a deleted topic, a producer writing bad data, or the loss of a site. Decide which topics need a second cluster, which need a copy in long-term storage through a sink connector, and which can be rebuilt from source systems.
MirrorMaker 2 replicates topics and translates consumer group offsets to a target cluster. By default, replicated topics carry the source cluster alias as a prefix, which affects how consumers find them after failover. Document how applications switch clusters, where they resume and how you fail back. Keep topic configurations, ACLs, schemas, connector configurations and Strimzi resources in version control as well.
Kafka operating ownership
The platform team runs the cluster; topic owners are responsible for what flows through it.
Platform
- Monitor under-replicated partitions, controller health, disk and request latency.
- Run rolling restarts, broker replacement and upgrades.
- Rebalance partitions when brokers change.
Topic owners
- Request topics with partitions, retention and schema rules set.
- Watch consumer lag and handle poison messages.
- Retire unused topics and consumer groups.
Security
- Issue principals per application and review ACLs.
- Rotate certificates and SASL credentials before expiry.
- Review logs for topic deletion and permission changes.
Integration
- Operate Connect workers and approve connector versions.
- Route failed records to dead letter topics with an owner.
- Run MirrorMaker 2 and test cross-site failover.
Kafka deployment questions
Do we still need ZooKeeper?
Kafka 4.x runs only in KRaft mode, so a new cluster does not need ZooKeeper. An existing ZooKeeper-based cluster needs a migration on Kafka 3.9 before it can move to 4.x.
Can brokers and controllers run on the same nodes?
Combined mode is convenient for development and small setups. The Kafka documentation does not recommend it for critical environments, because controllers are less isolated and cannot be restarted or scaled separately.
How many partitions should a topic have?
Enough for the consumer parallelism the topic needs over its life, with some headroom. Very high counts add metadata, open files and recovery time. Size from measured throughput per consumer.
Does licensing differ across the Kafka ecosystem?
Apache Kafka is Apache-2.0. Some surrounding components, including certain connectors, schema registries and management tools, use source-available or commercial licenses with restrictions. Check each license before adopting a component.
Scope and handover
DeployOpen plans the cluster layout and topic conventions with you, deploys Kafka in KRaft mode on virtual machines or Kubernetes, configures TLS and ACLs, connects client authentication to your identity provider where it fits, sets up monitoring and cross-site replication, migrates ZooKeeper-based clusters, carries out upgrades, documents runbooks and provides ongoing support if you want it. Scope depends on clusters, environments, connectors, client teams and whether a ZooKeeper migration is involved.
Handover includes the cluster and controller layout, listener and certificate design, principal and ACL model, topic register with owners and retention, approved connectors and client settings, failover runbooks with the latest test result, alert routes and the version plan.
