What a Kafka implementation involves
A self-hosted Apache Kafka rollout on a new build covers more than installing brokers. The core pieces are a KRaft controller quorum, the broker nodes that store and serve partitions, the topic and partition design that clients will depend on, retention settings that determine how long data is kept and at what storage cost, TLS and SASL access control on every listener, monitoring, and a handover that leaves your team able to operate the cluster. Each of these needs a named owner before work starts, because decisions in one area, such as partition count or listener layout, change the work in another.
Apache Kafka is licensed under the Apache License 2.0 and the Apache Software Foundation does not sell a paid edition of the project. The license fee is therefore zero, and the cost of a rollout is the engineering, infrastructure and support work around it: compute and storage for controllers and brokers, network capacity between sites, the time of the people who design and build the cluster, and any support agreement you choose to buy. Ask any partner to separate these components in a quote, and ask which ones are one-time and which recur each year.
Most rollouts are triggered by one of three situations: a new event backbone for several applications, a replacement for another messaging system, or a team that is building Kafka for the first time. Many organizations keep the platform choice and day-to-day operation in-house while they outsource the design and build. Confirm early whether your internal team has run Kafka in production, because that single fact changes how much design and handover work a partner should carry.
What the work includes
The exact list of workstreams is agreed in writing before work starts and follows your cluster size and requirements. Kafka Connect, MirrorMaker 2 and custom plugins are separate scope lines when they appear in your plan.
Architecture and sizing
Agree the cluster topology, the number of controllers and brokers, the racks or zones they occupy, and the throughput, retention and storage assumptions behind them. The deliverable is a sizing workbook that records each assumption so your team can rerun the numbers when traffic changes. Check that the throughput figures come from your actual producers and consumers, not from estimates.
KRaft cluster build
Controllers form the KRaft metadata quorum, and brokers carry client traffic. The build places both on the infrastructure you control, whether virtual machines, bare metal or Kubernetes. The deliverable is deployment code and a configuration baseline for every node, pinned to a supported Kafka release. Confirm node roles, storage layout and the Java 17 runtime with your platform team before provisioning.
Topic, partition and retention design
Each topic gets a named owner, a partition count, a replication factor, a min.insync.replicas value and a retention or compaction setting, all based on the consumers and ordering needs it serves. The deliverable is a topic catalog, a retention matrix, a naming convention and a provisioning process. Ask how partition counts will change later, because adding partitions alters key-to-partition mapping for keyed data.
Listener security and ACLs
Every listener used by clients, brokers and tools is configured for TLS. SASL mechanisms are chosen from the documented options, such as SCRAM, OAUTHBEARER or GSSAPI, and are connected to your identity systems where possible. Authorization runs through StandardAuthorizer, with a short list of super users. The deliverable is the listener configuration and an ACL matrix that maps each principal to its topics and consumer groups.
Monitoring and alerting
Metrics from brokers, controllers and clients flow into your monitoring stack. Alert rules cover under-replicated and offline partitions, controller quorum health, disk use and consumer lag. The deliverable is dashboards, alert rules and thresholds written for your environment, with notification routes confirmed by your on-call team. Check which metrics your current monitoring product already collects before choosing an exporter.
Recovery testing and handover
Recovery depends on replication, the cluster metadata and configuration exports, so each of these is included in the restore plan. The deliverable is a tested recovery procedure, runbooks for broker loss, controller replacement and rollback, and training sessions for the operators who will run the cluster. Define your recovery point and recovery time objectives before the test plan is written.
What drives cost and duration
Each row names one factor that changes the effort of a self-hosted Kafka rollout, why it moves the work, and how to keep it contained; the Kafka documentation gives no sizing figures for your workload, so sizing must come from your measured data.
| Driver | Why it changes effort | How to keep it down |
|---|---|---|
| Number of environments | Each environment needs its own KRaft controllers, brokers, certificates, ACLs and monitoring, and separate test data for upgrades. | Keep non-production clusters smaller and automate their build from the same configuration used in production. |
| Expected throughput and message size | Throughput and message size set broker count, disk and network capacity, and how much tuning of producer batching and compression is needed. | Measure real producer rates and payload sizes on a representative load test before choosing hardware. |
| Number of topics and partitions | Partition counts drive replication traffic, controller metadata load, consumer rebalancing and the ACL and naming design that must be maintained. | Agree a topic naming standard and partition policy up front, and add partitions only when measured consumer lag requires it. |
| Retention period and storage tier | Longer retention multiplies disk needs, affects recovery time after broker loss and may require tiered storage planning if your version and platform support it. | Set retention per topic from business need, and confirm storage options with your platform team before fixing disk sizes. |
| Number of producers and consumers and their client languages | Each client language has its own library, configuration and security setup, and each application needs compatibility testing against the chosen Kafka release. | Limit supported client libraries to a short approved list and publish a shared client configuration template. |
| Authentication model and identity integration | SCRAM, Kerberos (GSSAPI), OAUTHBEARER and mTLS differ in credential lifecycle, directory integration and principal mapping to ACLs, so each adds design and testing work. | Choose one primary mechanism for each client type and confirm which identity provider integrations your team already runs. |
| Number of listeners and network zones | Each listener needs its own TLS and authentication settings, advertised addresses and firewall rules, and each zone adds routing and testing. | Limit listeners to what clients need and document the advertised address rules for each zone. |
| Cross-site replication needs | MirrorMaker 2 adds a Connect-based deployment, replication topology decisions, offset translation for consumers and lag monitoring on each link. | Confirm which topics and consumer groups must replicate before designing the topology, and replicate only those. |
| Kafka Connect or Kafka Streams workloads | Connect workers and connector configurations, and Streams applications with their own state stores, add deployment, error handling and upgrade testing. | Start with the connectors you actually need, and confirm the Java runtime requirements for each workload before upgrading. |
| Compliance evidence requirements | Assessors often ask for documented configuration, access reviews, encryption settings and audit logs, which must be produced and kept current during the build. | Ask your assessor which evidence they expect and design logging and configuration management to produce it from the start. |
| Kubernetes or virtual machines | Running on Kubernetes adds operator choice, storage class design and pod networking work, while virtual machines add host configuration and patching work. | Pick the platform your operations team already runs and avoid introducing a second orchestration model for Kafka alone. |
Project phases and their outputs
The sequence moves from measured workload facts to a tested, secured and documented cluster that your team can operate.
Capture discovery and workloads
Interview owners of producing and consuming applications and record topics, message rates, record sizes, retention needs, latency expectations and the identity systems involved. The client receives a workload inventory and a list of open questions, both signed off before design starts.
Design architecture and size capacity
Design the KRaft controller and broker layout, listeners, replication factor, min.insync.replicas, partition plan and retention using the inventory. The client receives an architecture decision record and a sizing worksheet that the infrastructure team uses to provision servers, storage and network.
Build the non-production environment
Confirm the JVM standard before the build starts, because brokers and Kafka Connect require Java 17. Start from a supported release: at the time of writing, the downloads page lists 4.3.1, 4.2.2 and 4.1.2. The client receives a working non-production cluster, deployment code and a build record with versions.
Configure security and ACLs
Enable TLS on the listeners, select SASL mechanisms from the documented options, and define StandardAuthorizer ACLs in the metadata log under a deny-by-default posture. The client receives an ACL matrix, a super user list and security test results, with the allow.everyone.if.no.acl.found setting verified.
Test performance and failure scenarios
Run load tests at the sized throughput and fault tests such as broker loss, controller loss, network partition and client reconnects. Each test checks data durability and recovery against the targets written in the scope. The client receives a test report with measured results and a log of defects with their closure status.
Cut over to production
Agree a change window, a rollback plan and the order in which producers and consumers move to the production cluster. Monitoring and alerting go live before cutover. The client receives a go-live record and a post-cutover validation report confirming that traffic, offsets and ACLs behave as tested.
Hand over runbooks and monitoring
Write runbooks for broker replacement, rolling upgrades, topic and ACL requests, and recovery steps, then deliver dashboards and alert rules and train the platform team on them. The client receives the handover pack and a written acceptance sign-off that closes the fixed-scope engagement.
What usually goes wrong
Most Kafka failures are decided during design, before the first broker starts. Java and version mismatches across brokers, Kafka Connect and clients add another avoidable source of incidents.
Partitions sized from guesses
Partition counts chosen before anyone knows target throughput often end up too low for the number of consumers in a group, because each partition is read by only one consumer in a group at a time. Ask for the consumer parallelism target and the throughput estimate, size partitions against both, and record the basis. Increasing partitions later changes how keys map to partitions, so the count needs planning before go-live.
Retention without storage math
Retention left at default or habit-based values can fill broker disks and stall writes. Multiply the expected ingress rate by the retention period and the replication factor, then add headroom for rebalancing and broker recovery. Confirm per-topic retention settings and disk alerts with the team before production traffic starts.
Security added after clients connect
Enabling TLS or SASL on a cluster that already serves producers and consumers means changing listeners, client configuration and credentials in one coordinated window. Teams that defer this often find clients hard-coded to plaintext endpoints. Decide the listener layout, SASL mechanism and certificate lifecycle during design, and include client changes in the rollout plan.
Open access through default authorization
In KRaft clusters, resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Teams that set it to true to unblock a deployment leave topics open to every principal, while teams that run applications as super users bypass the ACLs they write. Decide the default policy, give each application its own principal, and reserve super user for administrators.
Failure behavior never tested
Replication factor and min.insync.replicas set to defaults are often never checked against broker or controller loss. Illustratively, with a replication factor of 3 and min.insync.replicas of 2, one broker can fail while producers using acks=all keep writing, but other combinations can block writes. Simulate broker and controller loss in a non-production cluster, record the outcome and runbook steps, and repeat after each upgrade.
Monitoring that stops at brokers
Dashboards showing broker CPU, disk and request rates can look healthy while consumers fall far behind. Consumer group lag is what reveals a stalled application, a slow sink or a partition that no consumer is reading. Ask for lag alerts per group and topic, along with alerts for under-replicated partitions and controller quorum health, before the cluster serves production traffic.
Recurring costs after go-live
This table lists each cost line of a self-hosted Kafka cluster with whether it is one-time or recurring and what drives it, so you can ask any partner to price every line separately in the quote.
| Cost line | One-time or recurring | What drives it |
|---|---|---|
| Broker and controller compute | Recurring | Broker count, partition count, KRaft controller quorum size and instance size |
| Storage and retention growth | Recurring | Ingest rate, retention period, replication factor and disk growth over time |
| Network and cross-site transfer | Recurring | Client traffic, replication between sites, MirrorMaker 2 links and cloud transfer charges |
| Backup storage | Recurring | Copies of topic data and cluster configuration, backup retention and restore test effort |
| Monitoring stack | One-time setup, then recurring | Metrics and log volume, dashboard and alert upkeep, and hosting of the monitoring tools |
| Internal platform staff time | Recurring | On-call coverage, topic and ACL requests, capacity reviews and client onboarding |
| Upgrades and security patching | Recurring | Three project releases a year, rolling upgrades from the last three releases, and OS and Java patching |
| Support agreement | Optional, recurring | Response terms, coverage hours and whether upgrade assistance is included |
| Project license fee | None from the project | Apache-2.0 has no paid edition from the project itself; commercial distributions are compared in the cost-of-ownership topic |
How the work differs by starting point
A new cluster on servers you control has the fewest inherited constraints, so the work centers on sizing, placement and operating procedures. Confirm the target release on the Kafka downloads page before hardware is ordered.
- Design: Decide whether controllers and brokers run on separate nodes or combined nodes, and size the controller quorum against the failure domains your data center actually has.
- Design: Standardize Java 17 on brokers and Kafka Connect hosts from the start, since Kafka 4.0 and later require it.
- Testing: Run load and broker-failure tests on hardware that matches the production profile, including disk and network throughput.
- Handover: Deliver runbooks for adding brokers, rotating certificates and changing topic defaults, each with a named owner.
Replication settings are covered in the high availability topic, and hardening controls are covered in the security hardening topic.
Running Kafka on Kubernetes adds a scheduler, storage layer and operator to the design, and each one changes how failures and upgrades behave. Agree early which team owns the cluster, the operator and the Kafka configuration.
- Design: Choose between a maintained operator and hand-written manifests, and document which Kafka settings the operator manages.
- Design: Use pod anti-affinity and topology spread so brokers are not scheduled onto the same node or zone. Confirm the storage class and volume behavior with the storage team.
- Testing: Test node drains, rolling restarts and volume reattachment under producer load, not only pod restarts.
- Handover: Write a sequence for cluster upgrades and Kafka upgrades that the platform team and the Kafka team both sign off on.
For authentication options inside the cluster, see the SSO and identity integration topic.
When hosts cannot reach public repositories, every artifact the cluster needs must be mirrored into your environment before installation. The mirror becomes part of the operating model and needs its own update process.
- Design: Inventory and mirror the Kafka distribution, container images, operating system packages, the Java runtime and any Connect connectors or plugins into internal registries and repositories.
- Design: Plan how new releases enter the network through an approved transfer process, and record which artifact versions each environment runs.
- Testing: Install and upgrade from the internal mirror with outbound access blocked, so any hidden dependency surfaces before production.
- Handover: Provide a mirror refresh procedure, checksum verification records and a bill of materials listing each artifact and its version.
Before mirroring third-party connectors, confirm their licenses and whether your policy allows them inside the network.
Joining an existing identity provider changes how clients authenticate and how service accounts are provisioned. Authorization still lives in Kafka, so the identity design must map cleanly onto ACLs.
- Design: Select the SASL mechanism that fits your provider, such as OAUTHBEARER with an OIDC provider, GSSAPI with Kerberos, or SCRAM, and decide where mutual TLS is required.
- Design: Define how identity provider principals map to Kafka principals, and keep that mapping in the same review as the ACL model.
- Testing: Test token expiry and renewal under sustained load, and confirm client libraries refresh credentials without dropping consumers.
- Handover: Document who approves new service accounts, how departures remove access, and which logs show failed authentication attempts.
The mechanism-by-mechanism detail sits in the SSO and identity integration topic.
Moving from another messaging platform makes the inventory of producers, consumers and delivery expectations the first deliverable. The cutover plan depends on which clients can change and which must keep running unchanged.
- Design: Inventory topics, producers, consumers, ordering needs, retention periods and replay requirements, then map each one to a Kafka pattern.
- Design: Decide whether the two platforms run in parallel for a period or whether consumers switch in a single window.
- Testing: Verify client compatibility, consumer offset behavior and message ordering, and rehearse the cutover at least once.
- Handover: Deliver a cutover runbook with explicit rollback criteria and a decommissioning plan for the old platform.
If the source is a ZooKeeper-based Kafka cluster or an older 3.x cluster, the migration topic covers the required upgrade path to KRaft and Kafka 4.
What you need to provide
A partner can only give a firm, fixed-scope quote once the inputs below are known. Before the build starts, name one accountable owner each for the Kafka platform, security and data. The platform owner decides how brokers, KRaft controllers and monitoring are run. The security owner signs off on listeners, authentication and ACL design. The data owner confirms which topics carry regulated or personal data and who may read them.
Next, provide infrastructure access and the surrounding services. That means virtual machines or Kubernetes namespaces, block storage sized for your retention targets, network paths between brokers, clients and any second site, DNS records, and an internal certificate authority that can issue server and client certificates. Also confirm the Java 17 runtime on broker hosts, since Kafka 4.x requires it, and whether your Java clients are on Java 11 or later.
Workload data drives partition counts, broker sizing and storage. Gather expected message rates and average and maximum message sizes, peak patterns with timestamps, the number of producers and consumer groups, and required retention by topic. List every producer, consumer and connector with its client language and library version, because client compatibility is a common source of rework. Record which identity mechanism you intend to use (for example OAUTHBEARER with your OIDC provider, Kerberos, SCRAM or mTLS) and which system is the source of truth for it.
Finally, provide data classification for each topic, any compliance evidence your assessors will ask for, the change windows your operations team allows, and a non-production test environment that resembles production in broker count and security settings. Any decision still open at quote time should be written into the scope as a named assumption with an owner and a decision date. Unresolved items are the most common cause of later change requests.
Questions buyers ask
Is Apache Kafka free to use, and what do we still pay for?
Apache Kafka is free to use under the Apache License 2.0, and the Apache Software Foundation does not sell a paid edition of the project. What you still pay for is the infrastructure it runs on, including broker and controller compute, storage sized for your retention settings, network transfer between sites, and the staff time to operate it. Commercial distributions and managed services add their own license or service fees, which should be compared as a separate line.
What drives the cost and timeline of a Kafka rollout?
The main drivers are the number of brokers and KRaft controllers, planned partition counts, retention periods, replication factor, and how many sites are involved. Kafka Connect and MirrorMaker 2 workloads, the identity systems you need to integrate, the security scope, and the monitoring and backup requirements also add work. No fixed duration is given here because these inputs vary widely between organizations. Ask for a written scope that lists each input before comparing timelines.
How should a Kafka implementation quote be structured?
A useful quote separates one-time build work from recurring support. Build lines usually cover architecture and sizing, KRaft controller and broker deployment, TLS and SASL with ACLs, monitoring, backup or recovery testing, documentation and handover. Recurring lines cover support, patching and upgrades on a supported release. Ask for Kafka Connect and MirrorMaker 2 as separate lines, since each adds workers, connectors or replication topology that change the effort. Every line should name its deliverables and what you must provide.
Can our own team run a Kafka rollout without outside help?
Yes, if your team has skills in Linux, networking, TLS and identity systems and time to learn Kafka operations. Kafka can be installed and run without a partner. Your team should expect to provide cloud or data center access, DNS and certificate authority access, identity system details, decisions on topic ownership and retention, on-call coverage and a test environment. Outside help is most useful for design decisions, security hardening and the first upgrade cycle, where mistakes are costly to reverse.
Does a new Kafka cluster need KRaft or ZooKeeper?
New clusters should use KRaft. Apache Kafka 4.0 supports only KRaft mode, and ZooKeeper mode has been removed, so ZooKeeper-based clusters must migrate to KRaft before upgrading to 4.x. The upgrade notes recommend that KRaft clusters older than 3.3.x first move to 3.9.x. Since Kafka 4.0, brokers, Kafka Connect and tools require Java 17, and clients and Kafka Streams applications require at least Java 11. Confirm these requirements against the upgrade notes for the release you plan to use.
What do we own at the end of a Kafka engagement?
You should own the infrastructure, configuration, certificates, credentials, deployment code, documentation and the Kafka data itself, all running in your accounts or data centers. Kafka is Apache-2.0 licensed, so no license transfer is needed. Ask the partner to confirm in writing which repositories, runbooks, dashboards, ACL definitions and test results are handed over, and who holds administrator access at the end. Ongoing support is a separate agreement if you want it.
How DeployOpen scopes and delivers Kafka work
DeployOpen plans, implements, hardens, documents and supports self-hosted Apache Kafka inside your own cloud accounts, data centers or air-gapped networks. Engagements are fixed-scope, so the phases, deliverables and cost components are agreed in writing before work starts.
How we scope
- The written scope lists the KRaft controller and broker layout, topic and partition plan, and retention targets agreed with your teams.
- Scope names the authentication mechanisms, ACL model and TLS coverage for each listener, so security work is priced and tested explicitly.
- Assumptions about client languages, Java runtimes and connected systems are recorded so changes to the quote can be traced.
How we build
- Clusters are built in KRaft mode on the release line you approve, with Java runtime and installation method confirmed against the upstream upgrade notes.
- Partition sizing, replication factor, min.insync.replicas and monitoring are configured and load-tested against the scope before go-live.
- Authorization is configured with the StandardAuthorizer and deny-by-default behavior, with super user access reviewed and documented.
How we hand over
- Handover includes configuration records, runbooks for broker and controller failure, and the documented upgrade path for your chosen release.
- Recovery steps are tested and written up, and your operators are walked through alerting, topic provisioning and ACL changes.
- Deliverables are listed in the written scope, so you can confirm what you receive before the engagement closes.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers, scoped separately from the implementation.
- Upgrades follow the upstream supported-release list and rolling upgrade rules, with compatibility checks for clients and connectors.
- Ask any partner to confirm in writing which releases, response commitments and upgrade tasks their support agreement covers.
