Sizing Kafka storage from throughput and retention starts with three numbers
A team moving a self-hosted Kafka cluster into production needs to know how much disk each broker requires before hardware is ordered. The core estimate is sustained ingest rate multiplied by retention period and by replication factor, and headroom is added on top of that result. The three numbers are the average bytes per second each topic receives at peak sustained load, how long the data must stay on disk, and how many replicas each partition keeps. Everything else in this section adjusts one of those inputs or the headroom around them. Measure the ingest rate from production-like traffic where you can, because averages from a quiet month understate what a busy day writes.
Apache Kafka is licensed under the Apache License 2.0 and the project does not sell a paid edition, so the costs you control are storage, hardware and the labor to run them. Storage is the line most sensitive to these three numbers, and it recurs: every extra day of retention or extra replica is paid for in disks in every broker, plus the backup and recovery capacity that holds copies of that data. The figures used on this page come from the Apache Kafka 4.3 documentation and the behavior described may differ in other releases. Confirm the defaults and retention behavior in the release you deploy before you finalize any disk plan, and record the release number next to the sizing worksheet so the estimate can be rechecked at upgrade time.
Which Kafka settings change how much disk you need
This table compares the Kafka settings that decide how much broker disk a topic consumes, what each one changes on disk, and where it fits. Read each row as a lever: retention and replication multiply your footprint, while cleanup policy and tiered storage change what is kept and where. Confirm default values in the Kafka 4.3 configuration reference before you size anything.
| Setting | What it changes on disk | When it fits and trade-off |
|---|---|---|
| retention.ms (time-based retention) | Sets how long closed log segments stay before deletion. Disk need grows roughly with retention time multiplied by ingest rate. | Fits when consumers or audits need a defined replay window. Longer windows raise disk need and recovery time for new brokers. |
| retention.bytes (size-based retention) | Caps the size of each partition log. Segments are removed once the limit is exceeded. | Fits when disk is fixed and a hard ceiling matters more than a time guarantee. Retention may be shorter than the time you intended during bursts. |
| cleanup.policy: delete | Removes whole old segments once retention limits are reached. Disk use follows the retention settings. | Fits event streams where each record is only useful for a window. Consumers must finish reading before data expires. |
| cleanup.policy: compact | Keeps the latest value per key and removes older values for the same key. Disk use depends on key cardinality and update rate, not elapsed time. | Fits changelog or state topics where only the latest state matters. Tombstones and delete.retention.ms affect what remains, so test with real keys. |
| cleanup.policy: delete,compact | Applies both rules: compaction keeps the latest value per key, and time or size limits also remove old segments. | Fits keyed topics that need a bounded history. It is harder to predict, so measure disk growth with production-like keys and traffic. |
| replication.factor (topic level) | Multiplies stored bytes by the number of replicas for each partition across brokers. | Fits production durability needs. A replication factor of 3 triples raw disk need compared with a single copy, so include it in every estimate. |
| Tiered storage (remote log storage) | Moves older log segments to remote storage, so local broker disk holds mainly recent data. Local retention and remote retention are configured separately. | Fits long retention with limited local disk. It adds a remote storage dependency, cost and operational testing. Confirm the setting names and support status for your release in the 4.3 reference. |
Kafka broker disk sizing step by step
This sequence turns measured ingest and retention needs into a per-broker disk figure that still covers retention after a reassignment or a broker loss. Each step ends with a check so the number can be defended to reviewers before hardware is bought or cloud volumes are provisioned.
Measure sustained and peak ingest per topic
Collect the compressed bytes per second written to each topic from production or pilot traffic, not from the size of the source payloads. Record a sustained average over a representative period and a peak over the busiest windows, separately for each topic. Your platform team and the application owners usually provide this data, and broker-side byte rate metrics are the most reliable source once a pilot is running. The step is done when every topic has a sustained and a peak figure with the measurement window recorded beside it. Check that the figures are compressed on-disk rates, since the broker stores batches as the producer compressed them.
Set retention per topic from business needs
For each topic, agree how long data must remain available for replay, reprocessing, audit or downstream backfill, and write that value down with its owner. Retention is set by time (retention.ms) or by size (retention.bytes), and the stricter limit wins, so record both if either applies. Topics that hold only current state may need compaction instead of time-based deletion, which is covered separately. The step is done when each topic has one retention value, a named owner and a reason. Check with the owners that no consumer expects data older than that value, because a retention decision that ignores replay needs is the most common cause of undersized clusters.
Multiply ingest by retention and replication factor
For each topic, multiply the sustained compressed write rate by the retention period in seconds to get the logical bytes held for one copy of the data. Then multiply by the replication factor, which is the number of replicas stored for each partition. Sum the results across topics to get the raw cluster requirement before overhead. Use the peak rate only in a separate scenario, because retention is a sum over time and the sustained rate usually governs the stored volume. The step is done when you have one raw total for the cluster and a worksheet showing each topic's contribution. Check the arithmetic by recomputing one topic by hand, and confirm that the replication factor matches the topic configuration rather than the cluster default you intended.
Add a measured margin for index and compaction overhead
Kafka stores index files and, for compacted topics, keeps superseded records until the cleaner runs, so stored bytes exceed the raw payload total. Do not guess this overhead. Load a pilot topic with production-like batches, let it run through several retention cycles, and compare the bytes on disk with the logical total from the previous step. Apply the observed ratio as a margin to the topics that behave similarly. The step is done when the margin is taken from a measurement and written next to its test conditions. Check that the pilot used the same compression codec, batch sizes and segment settings as production, because each of these changes the result.
Add headroom for reassignment and broker loss
Add capacity for two conditions. First, partition reassignment temporarily stores extra copies while data moves to new brokers, so leave room for the largest partition moves you expect to run. Second, when one broker fails or is removed, its replicas must be re-created elsewhere, so the surviving brokers must hold the data that was on the lost broker. Size the cluster so that, with one broker removed, the remaining disks still hold the full retention window at the planned replication factor, with an agreed reserve for the moves. The step is done when the headroom figure is written as a percentage or byte value with the scenario that produced it. Check it by subtracting one broker's disk capacity from the cluster total and confirming the result still covers the raw requirement plus margin.
Divide the total across brokers and disks
Divide the required capacity, including margin and headroom, across the number of brokers the design uses. Partitions are spread across brokers, so confirm that the expected partition placement produces roughly even disk use, because a few large topics can fill one broker before the average looks full. Decide whether each broker uses one large volume or several disks, and remember that a single disk failure removes that disk's partitions from the broker. The step is done when each broker has a per-disk capacity target and a placement check. Check the result against the cloud or hardware options available to you, and confirm the volume sizes and performance tiers with your infrastructure team or vendor before procurement.
Confirm the sizing against the plan and record it
Review the worksheet with the platform team, the topic owners and the people who run storage, and confirm that every input traces to a measurement or an agreed decision. Record the assumptions that would change the answer, including the compression codec, the replication factor, the retention values and the broker count, so that a later change to any of them triggers a new calculation. Write down the monitoring thresholds that will warn before disks approach the planned level. The step is done when the worksheet is signed off and the alerts for disk use and retention are defined. Check that a reviewer can rerun the calculation from the recorded inputs and reach the same figure.
Working through the storage formula with an illustrative cluster
The point of this section is to show how the formula turns a topic's traffic into a per-broker disk figure, using invented round numbers. Assume an illustrative topic receives 100 MB/s of producer traffic, which is measured after the producer has compressed each batch, and keeps data for 7 days with a replication factor of 3. Ingest multiplied by retention gives 100 MB/s times 604,800 seconds, about 60 TB of stored data per copy. Multiplying by the replication factor of 3 gives a raw total of roughly 181 TB across the cluster. Dividing that total across 6 brokers gives about 30 TB per broker before headroom. Add headroom for reassignment, broker failure and growth, and the per-broker target rises well above that base figure. The exact headroom percentage is a policy choice your team should set, not a value this example supplies.
The input to the formula must be the stored rate, not the raw producer rate. Compression changes the bytes written to the log, so if your producers send 100 MB/s uncompressed and the codec achieves a meaningful ratio on your payloads, the broker-side write rate is lower. Measure the rate on the broker with the metrics your monitoring already collects, using a representative payload sample and the codec you plan to run in production. Compression ratios vary widely by payload, so a ratio measured on synthetic data will mislead the sizing. Confirm the broker-side byte rate per topic before you fix any disk number.
Retention is set per partition, so adding partitions does not multiply disk need. A topic with 12 partitions and a topic with 48 partitions that receive the same ingest and keep the same retention store the same total data, because the ingest is divided among partitions rather than copied to each one. The replication factor is different: each partition keeps that many replicas, so the replication factor does multiply disk need. Check the replication.factor the topic actually uses, since a topic created with a lower value than the cluster default will not match the formula you wrote for it.
Headroom for reassignment, failures and growth
Broker disk sized to exactly the retained data leaves no room for the work a Kafka cluster does during change. The cards below cover the four events that consume spare capacity, how to measure each one, and what to confirm in the operations documentation before you fix a disk size.
Partition reassignment traffic
Moving partitions between brokers copies whole replicas, so the destination broker needs free disk for each incoming partition before the old copy is removed. Reassignment also competes with producers and consumers for network and disk I/O. Measure the size of the partitions you expect to move, then add that amount to the free space you plan to keep on each broker. Confirm in the operations documentation how throttling for reassignment is configured and how the old replicas are deleted after the move completes.
Replica recovery after a broker failure
When a broker fails, the partitions it led move to surviving replicas, which then take on the full write and read load. When the broker returns, it catches up by copying data it missed, and that catch-up uses disk on the returning broker while the rest of the cluster absorbs the extra load. Measure how much data each broker stores and how far the remaining brokers can grow before reaching your disk alert threshold. Confirm the failover and catch-up behavior for your Kafka version in the operations documentation.
Traffic spikes and backfill replays
A backfill that replays weeks of events, or a sudden burst from a producer, can write far more data in a short period than the steady-state rate. Retention is measured in time or bytes, so a replay can also push older segments out sooner than expected. Measure peak ingest over your busiest hour rather than the daily average, and record how long a replay of your largest topic would take. Confirm with your team which replays are planned and whether producers have quotas that cap their write rate.
Retention changes and growth
Raising the retention period of a topic does not free space at first. Every segment kept for the longer window stays on disk until it ages out, so the disk footprint grows gradually over the whole new retention period. Lowering retention frees space only after the expired segments are deleted. Measure current bytes per topic, project the new footprint for the full new window, and plan for the transition period. Confirm in the Kafka operations documentation how log retention settings are applied to existing data and when segment deletion runs.
Metadata and log segment overhead
Each partition holds segment files, index files and log directories, and the number of segments grows with data volume and with how often segments roll. Many small partitions can therefore use more disk and file handles than the raw data size suggests. Measure the number of partitions and replicas per broker, then compare the total file count and directory overhead on a test broker with a representative topic. Confirm the segment size and roll settings that apply to your topics, since these change how much overhead each partition carries.
Disk alerts before the failure point
Headroom only protects the cluster if someone is warned before it runs out. Set alerts on used and free disk per broker, tied to the free space your reassignment, failover and replay estimates require, rather than to a generic percentage. Test the alert by simulating a reassignment on a non-production cluster and observing how quickly free space drops. Confirm with your operations team who receives the alert, how long a response takes, and which runbook applies when a broker reaches the threshold.
When compacted topics change the math
A compacted topic keeps the latest value for each key instead of deleting records by age, so its disk need does not follow the ingest rate and a retention window in the same way as a time-based topic. What matters is the number of distinct keys and how often each key is updated. A topic with few keys and heavy churn stays small no matter how long it runs, while a topic with millions of keys that rarely change can grow large even at a modest ingest rate. Changelog topics from stream processors and offset topics from Kafka Connect are the usual cases, so their sizing should start from key cardinality, not from the throughput figure alone.
Deletion in a compacted topic happens through tombstones, which are records with a key and a null value. A tombstone tells compaction to remove earlier values for that key, and the tombstone itself is kept for the period set by delete.retention.ms before it is removed. Measure how many tombstones your workload produces per day, how long they remain, and whether consumers depend on seeing them. Retention behavior is governed by cleanup.policy, and a topic set to delete,compact applies both an age or size bound and key-based compaction, so the smaller of the two limits decides the actual footprint. Confirm the exact defaults and their behavior for your Kafka version in the Apache Kafka 4.3 documentation before you set them.
Compaction runs on closed log segments, so the active segment is not compacted and the on-disk size can sit above the steady-state estimate between cleaner runs. Run a representative workload with realistic key counts, update patterns and tombstone rates for long enough to see several compaction cycles, then record the log size per partition over time and compare it with the number of distinct keys. Segment size and the cleaner settings used in your environment should be included in the test. The result is a measured ratio of stored bytes to live keys, which you can apply to production key counts and add to the headroom you plan for reassignment and broker failure.
When tiered storage changes the sizing model
With local disk only, every retained byte lives on broker volumes, so the formula from the rest of this page applies in full: ingest rate multiplied by retention, multiplied by the replication factor, then increased for headroom. Broker disk is the main storage cost, and it is paid for in hardware or cloud volumes that you provision and expand yourself.
- Full retention on brokers: Every broker must hold its share of all partitions for the full retention period, plus replicas.
- Reassignment space: Partition moves temporarily store two copies of moved data, so keep free space for the largest partitions you expect to move.
- Failure headroom: Confirm how much disk remains on surviving brokers when one fails and its replicas are rebuilt elsewhere.
- Expansion lead time: Adding volumes or brokers takes time in your own environment, so plan growth before disks reach their alert threshold.
- Operating cost: Hardware, monitoring and capacity reviews recur every quarter, and nothing here is offloaded.
Before you commit to hardware, confirm the disk, filesystem and JVM settings that your deployment guidance and the Kafka documentation require for your chosen release.
With tiered storage, older log segments can move to a remote store while recent segments stay on local disk. Local broker disk then holds the hot portion of retention, and total retention is no longer limited by local capacity alone. The exact behavior, supported storage plugins and configuration keys must be taken from the Kafka 4.3 documentation for your release, and you should verify whether the feature is production-ready for your use before designing around it.
- Local hot window: Size local disk for the period consumers actually read from it, plus replicas and headroom.
- Remote capacity: Size the remote store separately, based on total retention and replication policy in that store.
- Plugin and configuration support: Confirm which remote storage implementations are supported, and which configurations are tested together, in the 4.3 documentation.
- Failure and recovery behavior: Confirm how reads, rebuilds and reassignments behave when segments are fetched from the remote store.
- Procurement: The remote store adds a recurring storage and request bill, plus a separate support contract or licensing terms if you use a vendor product.
Ask the implementation team to document which parts of the sizing model move to the remote store and which remain on brokers, and to test a reassignment and a broker loss before go-live.
Long-retention and replay-heavy workloads, such as event archives, audit trails or backfills for new consumers, keep large volumes available far longer than normal operations need. The sizing question shifts from peak throughput to total stored bytes and how often old data is read. Replay traffic can also compete with live consumers for broker I/O and network capacity.
- Total bytes over the retention window: Multiply ingest rate by the full retention period and replication factor, then check whether local disk or a remote store holds the bulk.
- Read pattern: Estimate how often historical reads happen, because cold reads from remote storage behave differently from reads on local disk.
- Replay isolation: Consider consumer quotas or dedicated read paths so backfills do not slow live consumers.
- Retention review: Confirm the retention policy with data owners, since legal or business retention periods often change after launch.
- Cost per retained byte: Storage cost grows with time as well as volume, so review the retention period as a recurring budget item.
Ask for a written retention schedule by topic, and confirm with the vendor or project documentation how replay reads are served once segments leave local disk.
What to confirm before you buy disks
Disk sizing rests on numbers that come from outside the formula: the settings your release actually enforces, the compressed rate your producers really send, and the hardware under your brokers. Each card below names who should confirm an item and where to check it before a purchase order goes out.
Retention and cleanup defaults
Confirm the default and valid ranges for retention.ms, retention.bytes, segment settings and cleanup.policy in the exact Kafka release you will run, because defaults and bounds change between releases. Platform engineering should read the broker and topic configuration reference for that version and record which values apply to new topics versus existing ones. A retention value you assume but do not verify will set the disk total for the whole cluster.
Measured compressed ingest rate
The ingest figure in the formula must be bytes written to disk after producer compression, not the raw application payload rate. Ask the team that owns the producers to measure bytes per second on the wire over a representative week, including peak hours. Compression ratio depends on payload type, so a rate measured with synthetic messages will understate or overstate the real total.
Disk and filesystem performance
Storage capacity is only half the purchase. Infrastructure engineers should benchmark sustained write throughput, fsync latency and rebuild behavior on the actual disks, RAID or volume layout and filesystem the brokers will use. Ask the storage vendor or cloud provider for the published limits of each volume class, then test under load with the broker's log directories in place.
Java 17 on every broker
Since Kafka 4.0, brokers and Kafka Connect require Java 17, so any host image that still ships an older runtime blocks the planned upgrade path. Platform engineering should confirm the JVM version on each broker image and note the requirement in the hardware plan, because a rebuilt image can change the runtime. The upgrade notes list the full set of requirements for each release.
Growth forecast and ownership
The retention multiplier only holds if the ingest rate stays put, so the product team that owns each event stream should give a written growth forecast for the next budget cycle. Ask them to name the topics that will grow, the expected onboarding of new producers and any retention changes planned by regulation or analytics. Without an owner for that forecast, the headroom figure becomes a guess that is revisited only after disks run short.
How much does Kafka storage sizing cost and what does it need from us?
What does Kafka storage sizing cost and what drives the fee?
The fee follows from the written scope, and DeployOpen does not publish a price list. The main drivers are the number of topics and clusters in scope, how many distinct retention rules apply, whether compacted topics need separate treatment, whether the sizing relies on measured production rates or estimates, and whether the output includes a reusable calculator and a review of headroom for failures and partition reassignment. Ask for a quote that lists each deliverable separately, so you can see what is one-time analysis and what would recur if you later add topics or change retention.
How long does Kafka storage sizing take?
Duration depends on the scope and the quality of the measurement data. A cluster with a small, well-documented topic list and reliable broker metrics takes far less time than a multi-cluster platform where ingest rates must be reconstructed from application logs. Before agreeing a timeline, confirm which deliverables are in the written scope, who will provide metrics access, and how many review rounds are included. DeployOpen commits to timelines only in the written scope agreed before work starts.
What does our team need to provide before sizing starts?
The work needs a topic inventory, the average and peak ingest rate per topic, the average and maximum message size, the replication factor you plan to use, and the retention period each business or legal requirement sets for each topic. It also helps to have broker disk capacity and current utilization, the expected growth over the planning period, and any compaction or tiered storage settings already in use. Where rates are not measured, ask for the producing applications' owners to confirm expected volumes, since the sizing is only as reliable as those inputs.
Can our internal team size Kafka storage without outside help?
Yes, a team that understands Kafka's log retention, replication and partition layout can do the arithmetic itself, and the core calculation is straightforward once inputs are known. Outside help tends to add value in collecting trustworthy rate data, modeling broker failure and partition reassignment headroom, deciding how compacted topics and tiered storage change the numbers, and documenting the result so it can be reviewed later. Confirm the exact retention and storage settings against the Kafka documentation for your release before relying on any model.
What recurs after the storage sizing is delivered?
The sizing is a point-in-time model, so the recurring work is tracking actual disk growth against it. Ingest rates change as products grow, retention requirements change with policy, and new topics arrive without a sizing review. Plan for someone to review disk utilization regularly, re-run the model when a topic or retention period changes, and revisit headroom before any broker is added, removed or upgraded. Ongoing support, upgrades and continuous testing are available as retainers if you prefer that the model is maintained by a partner.
Does DeployOpen issue a compliance attestation for our data retention?
No. DeployOpen does not issue audit reports, attestations or certifications. It can prepare the retention settings, sizing records and evidence that show how long data is kept and how storage was planned, so that your independent auditor or certification body can assess them. Whether a given retention period meets your regulatory obligations is a legal and compliance decision for your organization, and you should confirm the requirement with your compliance lead and auditor before fixing the numbers.
Will storage sizing change if we use tiered storage?
Yes, tiered storage changes which data occupies local broker disk and which sits in remote storage, so the local disk calculation must separate those tiers. Whether your release and deployment support tiered storage, and how it is configured, must be confirmed against the Kafka documentation for your version and your storage backend. The sizing should state the local retention, the total retention, and the assumptions about remote storage cost and availability, since each affects both capacity and recovery time.
How DeployOpen scopes and delivers Kafka storage sizing
DeployOpen treats storage sizing as a written, fixed-scope deliverable tied to your ingest rates, retention rules and replication settings. The output is a sizing model, a tested broker disk layout and runbooks your platform team can maintain after the engagement closes.
How we scope
- Collect ingest rates per topic, record sizes, peak-to-average ratios and the retention period each data owner requires.
- Confirm replication factor, compacted topic inventory and whether any topics will use tiered storage in your environment.
- Agree in writing which headroom targets, failure scenarios and growth horizons the sizing model must cover before work starts.
How we build
- Build a sizing workbook that calculates raw and replicated disk per broker, with formulas your team can edit as rates change.
- Test disk alerts, reassignment headroom and single-broker failure behavior against the sized layout in a non-production cluster.
- Document assumptions, including compression ratio and log segment overhead, so each figure can be traced back to a measured input.
How we hand over
- Deliver the sizing model, broker storage configuration, retention settings per topic and the disk alert thresholds derived from them.
- Provide a runbook for adding capacity, reassigning partitions and reviewing sizing when a new topic or retention change is requested.
- Walk your operations and capacity owners through the model so they can re-run it without DeployOpen involvement.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers if you want sizing reviewed as volumes grow.
- Re-check the model after each Kafka upgrade, since defaults and storage behavior can change between releases you adopt.
- Confirm with your own environment and the Kafka documentation which storage features your chosen release supports before relying on them.
