Apache Kafka hardware sizing starts with your workload, not a server model
Sizing decides how many broker and controller hosts you need, how much memory and disk each one carries, how much network capacity replication requires, and how many controllers form the metadata quorum. Those numbers should come from your message rates, retention periods, replication factor and failure tolerance, not from a server model picked off a vendor list. Apache Kafka is an Apache Software Foundation project released under the Apache License 2.0, so there is no paid edition to buy. Your cost centers are the hardware or cloud capacity you run, the storage behind it, and the labor to build, operate and upgrade the cluster. Ask your team to confirm the supported releases on the Kafka downloads page before fixing any host image or Java baseline, since the page lists 4.3.1, 4.2.2 and 4.1.2 as the supported releases today.
Start by writing down the workload: peak and average ingress in bytes per second, the number of partitions you expect, how long each topic must be retained, and the replication factor you will use. Those four inputs determine most of the storage and network budget. Kafka leans on the operating system page cache for recent reads, so memory on broker hosts is best treated as a buffer for the data consumers read most often, while the JVM heap stays modest. Disk throughput is usually the first bottleneck teams hit, so plan separate data drives per broker rather than one shared volume, and benchmark the actual drives with your own message size and acknowledgment settings. Controllers run the KRaft metadata quorum and carry far less data, so they can be sized much smaller than brokers, but the number of controllers you run decides how many failures the quorum survives.
Sizing brokers, controllers and data drives separately
Broker hosts store partition logs, serve producers and consumers, and replicate data to other brokers. Their requirements come mainly from data volume, the replication factor and how far consumers fall behind producers. Memory and disk usually decide most of the hardware cost.
- Page cache memory: Kafka depends on the operating system page cache to serve recent data to consumers. Size RAM so the recent write working set fits in cache, using the 30-second buffering rule of thumb as a starting point and confirming it with a test.
- Data drive throughput: Sustained read and write throughput across all partitions on a broker is usually the first bottleneck. Replication factor and consumer fan-out raise the load on each drive.
- CPU for compression and TLS: Producers may compress batches, brokers may decompress or recompress them, and TLS adds encryption work for each connection. Clusters with many clients or heavy TLS need more cores.
- JVM heap: The heap holds broker object state and is separate from page cache, so do not assign the heap all of the host memory. Brokers on Kafka 4.0 and later need Java 17 according to the upgrade notes.
Confirm these figures by replaying your peak producer rate with your real compression and TLS settings in a test environment.
Controllers store the cluster metadata log and elect the quorum leader, including partition leadership changes. They carry little message traffic, so their throughput demand is modest, but they are sensitive to disk latency and network stability. Their size follows metadata volume, meaning topics, partitions, configurations and ACLs, not message volume.
- Metadata growth: A platform with many small topics and partitions creates more metadata work than its message rate suggests. Track topic and partition counts when you plan controller capacity.
- Disk latency: Controllers write the metadata log synchronously, so a slow sync on a busy drive delays elections and metadata changes. A dedicated low-latency device is the safer choice where your hardware allows it.
- Quorum size: Use an odd number of controllers so a majority survives failures, and have your team confirm the count against the KRaft guidance for your release.
- Network stability: Controllers must reach each other reliably. Packet loss or long pauses can trigger leader changes that look like broker failures.
Test controller failover and metadata latency under load before you fix the host profile for production.
Keeping broker log data on drives separate from the operating system and the metadata directory stops I/O from competing and makes capacity planning easier to read. Several data drives can spread partition directories across devices, but each device that fails takes its partitions with it on that broker, so replication has to cover that loss.
- OS isolation: Place the operating system and packages on their own device so heavy log writes do not slow system services or package operations.
- Multiple log directories: Kafka can spread partitions across several directories listed in
log.dirs. Uneven placement can leave one drive busy while others sit idle, so check partition distribution after setup. - Capacity headroom: Size each drive for retained data, replicated copies and free space for rebalancing and growth. Ask your team or the vendor to confirm the headroom policy you will use.
- Drive failure behavior: A failed log directory takes its partitions offline on that broker. Confirm how your Kafka version handles this case and which replicas remain available during recovery.
- Filesystem and mount options: Filesystem choice and mount options affect throughput and latency, and they are covered in the Linux and filesystem tuning topic in this group.
Check per-drive throughput and partition distribution in a test cluster before you commit to a drive layout.
On small clusters, running controllers on the same hosts as brokers saves machines and simplifies operations. The cost is that broker load can slow metadata work and a single host failure removes more than one role. The right layout depends on how many hosts you can fund, the peak workload and how much failure isolation you need.
- Fewer hosts: Combined nodes reduce hardware and operating overhead, which suits a small test environment or a departmental cluster with modest traffic.
- Shared resource contention: Heavy producer or consumer load competes with controller threads, disk and memory on the same host, so metadata responsiveness can drop during traffic peaks.
- Failure scope: Losing one combined host removes a broker and a controller voter together, which narrows the margin the quorum has to survive additional failures.
- Growth path: Moving controllers onto dedicated hosts later is possible but adds a migration step. Plan that path before go-live, including how your deployment method handles separate role pools.
- Deployment model: Some Kubernetes operators and node pool designs can separate roles without adding physical machines, so confirm which options your environment supports.
Decide the layout after measuring peak load on a test cluster, and record the migration path in your plan.
How replication traffic changes the network plan
A broker's network load is the sum of four flows: producers writing in, consumers reading out, replicas copying every partition between brokers, and recovery traffic when a broker fails or partitions move. Size NICs and links for the combined peak, not for the average producer rate, because replication multiplies ingress before any client reads a byte.
Producer ingress
Producer traffic is the write rate that lands on partition leaders. Each leader receives its share of the topic's inbound bytes, so a few hot partitions can concentrate ingress on a single broker. Measure peak producer throughput per broker, including compression effects, and check whether your producers batch and compress before you set the NIC budget.
Consumer egress
Consumers read from leaders, so every additional consumer group adds outbound bytes equal to the data it reads. Fan-out from one topic to many groups can make egress larger than ingress on the same broker. List the consumer groups per topic and their catch-up behavior, since a lagging group that rereads history can saturate the link during recovery.
Inter-broker replication
Replication factor multiplies inbound traffic. Each follower fetches every partition it replicates, so a broker that leads data also receives copies from the rest of the cluster and sends copies out. Confirm the replication factor per topic, the number of partitions each broker hosts as leader and follower, and whether replicas and leaders share the same NICs.
Failover and rebalancing bursts
When a broker stops, its partitions move to other replicas, and those replicas must catch up and serve clients. Partition reassignments and broker restarts create short bursts that can exceed steady-state load several times over. Plan headroom for the largest broker loss you accept, and test a planned partition move at production volume before relying on the estimate.
NIC bandwidth and link redundancy
Confirm NIC speed against measured peak throughput, not nominal capacity, and leave room for TLS overhead and background traffic. Use bonded or redundant links on separate switches so one failed port does not turn into an under-replicated partition set. Verify the actual throughput with a load test on the production network path.
Cross-site links are separate
Traffic between sites follows its own budget, latency and failure profile, and it should not share the broker's intra-site NICs. MirrorMaker 2 and stretched clusters both place replication on WAN links, so the multi-site layout page covers those designs. Here, confirm only that local broker networks can carry local traffic at peak.
How many controllers to run and when each count fits
This table compares one, three and five KRaft controller layouts on quorum tolerance, write latency, operational overhead and cost. Read each row against the failure domains you actually have, because the quorum needs a majority of controllers to keep working and the right count follows from where those controllers can fail independently.
| Layout | Fit | Trade-off | What to confirm |
|---|---|---|---|
| One controller | Lab, proof of concept or throwaway test clusters with no availability expectations | No tolerance: losing the single controller stops metadata changes, and it offers no quorum protection | Whether any environment that holds real topics will ever run with a single controller |
| Three controllers | Most production clusters where three hosts or zones can fail independently | Tolerates one controller failure while a majority of two remains; adds hosts and some replication overhead on metadata writes | Whether the three controllers sit in three separate racks, hosts or availability zones rather than sharing one failure domain |
| Five controllers | Clusters that must keep a majority through two simultaneous controller losses, or that span three or more failure domains with spares | Tolerates two failures; more hosts to patch, monitor and keep certificates current, and more replication traffic per metadata write | Whether you have five distinct failure domains, and whether the team can run rolling maintenance on five controllers without losing quorum |
| Even counts (two or four) | Generally not a recommended layout | A majority still needs more than half, so an even count adds hosts without adding tolerance over the next odd count down | Whether any design proposal uses an even count to fit an existing host budget |
| Controllers colocated with brokers | Small clusters where dedicated hosts are not justified | Broker load, garbage collection pauses and disk pressure can affect controller responsiveness | Whether the Kafka documentation for your release and your deployment tooling support the layout you plan |
| Dedicated controllers | Larger clusters where broker scaling and maintenance should not touch metadata | Separate hosts and separate sizing; more inventory and more deployment steps | Controller sizing needs for your expected partition count and metadata churn, confirmed with your release notes and load tests |
Sizing sequence from workload to purchase order
This sequence turns business traffic expectations into a tested broker and controller layout that can be costed, quoted and written into a scope. Each step produces a figure or check that the next step depends on, so host counts are fixed only after measured behavior supports them.
Collect produce and consume rates
Gather peak and average produce and consume rates in megabytes and messages per second for each planned topic, along with message size distribution and the number of producers and consumer groups. Owners are the application teams and the platform team, who should agree on the peak window and the growth horizon in writing. The step is done when every topic has a documented rate profile and the peak-to-average ratio is recorded, because sizing built on averages alone usually underestimates bursts.
Set retention and replication factor
Record the retention period, the retention size limit and whether any topic uses compaction, then set the replication factor and minimum in-sync replicas for each class of data. Retention multiplied by sustained write rate and replication factor gives the raw storage requirement before headroom. Ask the data owners to confirm retention in their own compliance terms, since regulated data often has minimum or maximum retention rules. The step is done when storage totals per topic are calculated and reviewed by the owners.
Define failure domains
Map brokers and controllers to racks, hypervisors, availability zones or data halls, so that losing one domain does not remove enough replicas to break the minimum in-sync requirement. Decide how many domains the cluster spans and whether each one can hold a full set of replicas for the most critical topics. The output is a placement diagram that the infrastructure team signs off, and it becomes the basis for host counts in later steps.
Estimate memory and disk per broker
Calculate the data volume each broker must hold from the rate, retention and replication figures, then add headroom for rebalancing and for recovering a failed broker. Size memory with the page cache in mind, using the 30-second buffering rule of thumb as a starting point, and keep the JVM heap sized separately. Confirm the drive layout with the vendor or storage team: log directories on separate data drives, sustained write throughput under load, and capacity after failure. The step is done when a per-broker disk and memory table exists with the assumptions written beside each number.
Choose the controller count
Decide whether controllers run on dedicated hosts or share hosts with brokers, and select an odd number of controllers that matches the failure domains defined earlier. Confirm with the Kafka documentation for your target release how controller quorum is formed and what happens when a majority is lost. Controllers hold metadata rather than message data, so their disk and memory needs are usually modest, but their placement determines whether the cluster survives a domain outage. The step is done when the controller count and placement are written into the design.
Validate network capacity
Add the replication traffic, client produce and consume traffic, and catch-up traffic from a broker restart, then compare the total against the usable bandwidth of each host and each link between sites or racks. Include the cost of replication across failure domains, which can exceed client traffic for high replication factors. Ask the network team to measure real throughput between hosts rather than relying on nominal link speed. The step is done when peak network demand per broker is below the measured capacity with margin that the team has accepted.
Load test a pilot cluster
Build a pilot with the planned hardware, OS settings and Kafka configuration, then run sustained produce and consume load at the measured peak rates for long enough to observe disk flush behavior, garbage collection and replication lag. Simulate a broker failure and a controller failure during the test and record recovery time and whether consumers keep up. Compare results with the sizing model and adjust host counts or disk layout where the pilot falls short. The step is done when the pilot meets the agreed latency and lag targets under peak load and failure, and the results are archived with the design.
Issue the written scope or purchase order
Convert the validated design into a written scope that lists host counts, specifications, storage layout, network requirements, controller placement and the acceptance tests from the pilot. Provide the same document to every quoting party so the quotes are comparable, and ask each to state what is included for installation, configuration and support. The step is done when the scope is approved by the platform, security and infrastructure owners and a purchase order or fixed-scope agreement references it.
Security and operational considerations for host sizing
Host size is only half the design. Isolation, TLS overhead, patching, disk access, metadata backups and pressure monitoring all change what a broker or controller host needs to be. Check each one before the hardware order is final.
Isolate controller hosts
Controllers hold the KRaft metadata log for the whole cluster, so they should not share hosts or disks with busy brokers if you can avoid it. A noisy broker that saturates its disk or memory can delay controller votes and metadata updates. Decide whether controllers run on dedicated hosts or co-located nodes, and size the co-located case for the broker load it will also carry.
Budget CPU for TLS and SASL
Encryption and authentication add CPU work on every connection and every replicated byte path that is encrypted. Kafka's security overview describes SSL/TLS between clients, brokers and tools, plus SASL mechanisms such as SCRAM and OAUTHBEARER. Test your chosen listeners with realistic producer and consumer counts, and size CPU headroom from those results rather than from an unsecured benchmark.
Plan the patch cadence
Kafka releases on a planned schedule, and the OS and Java runtime get security fixes on their own timelines. Host sizing should leave room for rolling restarts, since each broker restart drains leadership and warms page cache again. Confirm your Java version against the upgrade notes and agree on how often you patch the OS, JVM and Kafka together.
Restrict access to host disks
Anyone with read access to a broker's log directories can read the message data on disk, whatever topic-level ACLs say. Limit SSH and sudo rights on Kafka hosts, set file ownership so only the Kafka service user can read data and metadata directories, and confirm that encryption at rest is handled by the disk or volume layer if your policy requires it.
Back up metadata directories
The metadata directories hold the KRaft cluster state, and losing them can make a controller unable to rejoin the quorum. Include metadata directories in host backups separately from topic data, and test a restore onto a spare host. Document which directory holds metadata on each node so recovery is not improvised during an outage.
Monitor disk and memory pressure
Disk throughput is usually the first limit, and running out of memory for page cache slows reads even when the JVM heap looks healthy. Alert on disk utilization, I/O wait, free page cache and swap activity, and on controller lag. Set thresholds from your own baseline during load tests, so the alerts fire before consumers fall behind.
Questions buyers ask about Kafka hardware sizing
How much does a Kafka hardware sizing engagement cost?
The fee depends on the scope agreed in writing: how many clusters and environments are covered, whether sizing includes storage, network and cloud instance choices, and how much telemetry your team must export before analysis starts. DeployOpen does not publish a price list, and a quote follows from the written scope. Most of the fee pays for discovery, modeling, validation and a written sizing report with its assumptions. Apache Kafka itself is licensed under Apache-2.0 with no paid edition from the project, so software licensing is not a line item. Hardware, cloud or data center spend is paid to your provider and is separate. Sizing is mostly one-time work, but workloads grow, so a re-sizing review recurs. Ask for each phase as a separate line and whether a follow-up review is included.
How long does Kafka hardware sizing take?
Duration is set by scope, and the signed scope is the source for dates. The main drivers are the number of clusters, whether historical broker metrics already exist and are easy to export, how many topics and producers must be modeled, and whether topology choices such as one cluster per site or a stretched layout are part of the work. A single cluster with clean metrics moves faster than a multi-site estate where throughput is unknown for many applications. Your team's availability matters too: someone must provide metric exports, confirm business peaks and grant access to any test environment used for validation. Ask for the phases, the review rounds included, and the dependencies on your team, so that delays on your side are visible early.
What information must we provide for Kafka sizing?
The sizing model depends on measured inputs more than on assumptions. Expect to provide:
- Throughput: average and peak bytes in and out per topic, taken from broker metrics that cover your busiest business cycle.
- Retention targets: time and size limits per topic, and whether any topics are compacted.
- Replication factor and the number of consumer groups reading each topic.
- Message size distribution, since large messages change memory and network behavior.
- Topology: number of sites, failure domains, latency between them and any existing hardware or instance types.
- A growth forecast for the next planning period, agreed with the teams that own each workload.
If some figures are missing, the report should state which assumptions replace them so you can check them later.
Can our own team size Kafka hardware internally?
Yes, if the team can measure production traffic, reason about the Kafka write path and run load tests against realistic data. Sizing needs knowledge of replication, page cache behavior, consumer fan-out and recovery traffic, plus the discipline to test assumptions instead of trusting a single calculation. An internal team keeps that knowledge in-house and can repeat the exercise as workloads change. The costs are staff time, test infrastructure and the risk of blind spots when the same people built the platform. A common middle path is for your team to collect and export the metrics while an outside reviewer challenges the model and the validation results. Decide which path fits your staffing and document the reasoning, so auditors and future engineers can follow it.
Why is disk throughput usually the bottleneck for Kafka brokers?
Kafka appends every message to a log on disk, and each replica follows its leader, so write I/O multiplies with the replication factor. Consumers that read data no longer in the page cache add more disk reads. Sequential writes are fast, but sustained mixed read and write load, plus partition reassignment and broker recovery, can saturate drives long before CPU or memory run out. Dedicated data drives per broker keep log I/O separate from the operating system and make the bottleneck measurable. Memory still matters because page cache serves recent reads. A common rule of thumb sizes page cache to hold about 30 seconds of write traffic, which is a starting point to validate against your own peaks rather than a guarantee.
How many KRaft controllers should we run?
KRaft controllers form a Raft quorum that stores cluster metadata, and the quorum keeps working only while a majority of its voters are available. For that reason, run an odd number of controllers. Three is the usual choice for a single production cluster, and five suits larger estates or multi-site designs where you must tolerate losing more voters at once. Dedicated controllers separate metadata work from broker load and reduce the blast radius of a broker failure, while combined mode is generally reserved for small test or development setups. Spread controllers across failure domains such as racks, zones or sites where the network allows it. Confirm the quorum and bootstrap guidance for your exact Kafka release before you finalize the design.
What network capacity do Kafka brokers need for replication?
Each produced byte travels from the leader to every follower, so inter-broker traffic is a multiple of ingest, on top of consumer egress and any traffic from mirroring tools. Size the broker network for the peak combined load, including recovery, reassignment and catch-up after an outage, because those events often exceed normal traffic. Dedicated network interfaces for replication are an option when the design calls for them, and they should be validated under load rather than assumed. Across sites, round-trip latency and link capacity change the picture, and socket buffer tuning becomes part of the design. Ask for the measured link speeds, switch capacity and WAN characteristics so the model reflects your real network.
How DeployOpen scopes and delivers Kafka sizing work
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure, including cloud accounts, data centers and air-gapped networks. For Apache Kafka hardware sizing, we turn your workload assumptions into a written sizing plan for broker and controller hosts, agreed before any hardware is ordered.
How we scope
- Engagements are fixed-scope, with sizing inputs, deliverables and timelines agreed in writing before work starts.
- We collect producer rates, message sizes, retention targets, replication factor and consumer fan-out to frame the sizing model.
- Scope states the number of controllers, broker pools, site count and the storage and network assumptions the plan will cover.
How we build
- We model memory for page cache, separate data volumes and disk throughput against the expected write and read load.
- We size network capacity for replication traffic, client traffic and any cross-site mirroring that the scope includes.
- We test the proposed layout against load profiles you supply and record where the first bottleneck is expected to appear.
How we hand over
- You receive a sizing document with assumptions, formulas, chosen host profiles and the reasoning behind each decision.
- Hardware and cloud instance requirements are listed so your procurement or platform team can order or request them directly.
- We walk your engineers through the model so they can rerun it when volumes, retention or release versions change.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer, including re-sizing when workloads grow.
- We can review sizing as Kafka releases change and confirm capacity assumptions before each planned upgrade.
- Retainer scope, response expectations and review cadence are defined in writing so your team knows what to expect.
