Why Kafka disk usage needs a plan before brokers fill up
Disk use on a Kafka broker is set by three things working together: the retention settings that decide how long data is kept, the segment behavior that decides when old data can actually be deleted, and the write rate of each topic. Plan these together, or the first sign of trouble will be a full disk rather than a chart. Teams usually need this work when they create new topics, when producers grow faster than forecast, when they add compacted topics whose size does not fall on a timer, or when a first incident shows that nobody knew how much space each topic was using.
Apache Kafka is licensed under the Apache License 2.0 and is an Apache Software Foundation project, so there is no paid edition from the project itself. That means the retention, segment and monitoring controls described here are open-source features you configure yourself. Costs come from the infrastructure you run the brokers on and from the people who size and operate them. Before you rely on any broker default, confirm the values your cluster is actually running with, because topic-level and broker-level settings can differ from what the platform team assumes.
A useful first step is an inventory: for each topic, record the write rate, the retention policy, whether it is compacted, and how much disk it occupies today. The kafka-log-dirs tool and per-topic size metrics from your monitoring stack can supply most of this. Decide who owns each number, and what the alert threshold is for each broker log directory. Teams that skip this step often discover that one fast-growing topic with a long retention period consumes the headroom that every other topic depends on, and that the only remaining fix is an emergency change under pressure.
Retention and segment settings and when each fits
This table compares the topic and broker settings that control how long Kafka keeps data, how it cuts log segments and how it cleans up old records. Read each row for what the setting controls and the kind of topic it suits, then confirm defaults and valid ranges in the 4.3 configuration reference before you set them. The main takeaway is that time, size and cleanup policy act together, so a topic can be limited by whichever limit is reached first.
| Setting | What it controls | When it fits |
|---|---|---|
| retention.ms (topic) | Maximum age of log data before eligible segments are deleted or compacted, set per topic | Time-bound event streams where consumers only need recent history, such as the last few days of clickstream events |
| retention.bytes (topic) | Maximum size of a partition's log before the oldest segments become eligible for removal | Size-capped buffers or topics with unpredictable volume, where disk budget matters more than age |
| cleanup.policy (topic) | Whether old data is deleted, compacted (kept as the latest value per key), or both | delete for event history, compact for changelog or state topics, and compact,delete when both key-based retention and an age limit are needed |
| segment.bytes (topic) | Size at which the active log segment rolls over to a new file | Smaller values let deletion and compaction reclaim space sooner at the cost of more files; larger values suit high-throughput topics with few partitions |
| segment.ms (topic) | Maximum time before the active segment rolls, even if it is not full | Low-volume topics whose data would otherwise sit in one open segment and never become eligible for time-based deletion or compaction |
| log.retention.hours, log.retention.minutes, log.retention.ms (broker) | Cluster-wide default for time-based retention when a topic does not set retention.ms | Platform-wide baseline that teams can override per topic; confirm which of the three keys takes precedence in the 4.3 reference |
| log.retention.bytes (broker) | Cluster-wide default size limit per partition when a topic does not set retention.bytes | A safety net across all topics, so a single misconfigured topic cannot fill the broker disks |
| log.segment.bytes and log.roll.ms (broker) | Cluster-wide defaults for segment size and roll time | Baseline segment behavior for new topics; change them only after checking how many segments your partition counts will create |
Why compacted topics never shrink to a fixed size
A compacted topic keeps the latest value for each key rather than a fixed window of history, so its size follows the number of distinct keys and how often they change. Each card below covers one part of that behavior that affects disk planning, alerting and the choice between compaction and time-based cleanup.
Latest value per key is kept
With cleanup.policy=compact, Kafka retains at least the most recent record for every key in a partition. Older values for the same key become eligible for removal during cleaning, but the newest one stays. Disk use therefore tracks how many distinct keys the topic holds, not how many days of messages arrived. A topic that receives many updates to a small key set stays small, while one with ever-new keys grows without a size ceiling. Confirm the key cardinality with the producing team before sizing the volume.
Active segment is never cleaned
Cleaning only applies to closed segments. The segment currently being written to is excluded, so a low-traffic compacted topic can hold duplicate keys for a long time before any of them are removed. Segment roll settings such as segment.bytes and segment.ms decide how quickly data becomes eligible. Smaller segments clean sooner but create more files and more cleaner work. Test with production-like write rates before assuming a compacted topic will reclaim space on a predictable schedule.
Tombstones linger before removal
A null-value record, called a tombstone, tells the cleaner to delete a key. The tombstone itself is kept for at least delete.retention.ms so that consumers can see the deletion, and only then is it removed. Until that point the key still occupies disk, which surprises teams that expect deletes to free space immediately. Check the topic's delete retention value against how long slow or offline consumers may take to read the tombstone, and include tombstone volume in your capacity model.
Cleaner threads and dirty ratio
Compaction runs on background cleaner threads, and a segment is cleaned only after its share of duplicate records passes the min.cleanable.dirty.ratio threshold. Too few cleaner threads or heavy write load can leave the log dirty for long periods, and the broker's disk fills while cleaning lags behind. Track the log cleaner metrics and the size of each compacted partition over time, not just the total for the broker. Ask the platform owner which cleaner settings were tested under peak load.
Compaction or time retention
Compaction keeps current state and suits changelog and reference data, but it does not bound the history a new consumer can replay. Time-based deletion bounds disk by age and suits event streams where the history has a limited business value. Setting cleanup.policy=compact,delete combines both, so old keys are removed by age as well as superseded by newer values. Before choosing, confirm which consumers rebuild state from the topic, how far back they need to read, and what the data owner says about retention obligations.
Checks before choosing cleanup policy
Before setting cleanup.policy=compact or compact,delete, document the expected distinct key count, the update rate per key, the tombstone pattern and the replay window each consumer needs. Estimate disk per broker with a growth ceiling that accounts for the active segment and the dirty backlog. Add a per-topic size metric and an alert on sustained growth, so the team learns about unbounded key sets early. Confirm these numbers with your own test environment, since they depend on your workload.
Setting per-topic size metrics and disk alerts
This sequence exposes per-topic and per-partition log size from each broker, records a baseline, and turns that data into alerts on free disk and topic growth. It ends with alerts proven to fire on a staging topic before they guard production.
Confirm the metric names for your exporter
Before writing any rule, the platform engineer opens the Kafka 4.3 metrics documentation and the documentation for the JMX exporter you plan to use, then lists the log size and disk metrics that each exposes. Names differ between exporters and between how the exporter maps MBean attributes, so the list is written down per exporter version. The step is done when every alert in later steps points to a metric name that appears in both documents, with the exact label set recorded.
Expose broker log size metrics through JMX
The platform team enables a JMX exporter on every broker and controller, using a rule file that maps the broker log size MBeans to Prometheus metrics with topic and partition labels. Per-partition labels can multiply series quickly on clusters with many partitions, so the team decides whether to keep partition granularity or aggregate to topic level before the scrape goes live. The step is done when a query against the broker returns a non-zero size for a topic known to hold data, and the series count per broker is recorded.
Verify scrapes on every broker
The team checks the scrape target list in the monitoring system and confirms that each broker reports the log size series, with no gaps after a broker restart. They also confirm that the disk free space metric for each log directory is present, since disk alerts depend on it. A broker that is missing from the results is investigated before any alert is written, because a silent gap looks like a healthy cluster. The step is done when every broker shows fresh samples for both metric families over a window the team chooses.
Record a per-topic size baseline
The operations team captures the current size of each topic and partition, along with its retention settings, and stores the results with the date in the team runbook. The baseline shows which topics are large, which grow steadily, and which are compacted and should hold steady. Growth rate alerts are set relative to this baseline, not to an arbitrary number. The step is done when the baseline table lists every topic, its size, its retention or cleanup policy, and the observed daily change, and the owner of each topic has reviewed it.
Set broker free space alerts
The platform team writes alert rules on the free space of each log directory, with thresholds agreed with the on-call team and the disk capacity owner. Two levels are usually useful: a warning that leaves room to add capacity or adjust retention, and a critical alert that fires while cleanup is still possible. The rules also cover the case where a broker is absent from the scrape, since a missing series should not clear an alert. The step is done when each broker has both rules, each rule has a documented owner and runbook link, and the thresholds are written in the runbook with the reasoning behind them.
Set growth rate alerts per topic
The team writes rules that compare the recent change in each topic's size against its baseline, so that a producer that suddenly writes far more than expected is caught before the disk fills. Rules for compacted topics are written separately, because their size is expected to stay flat while key cardinality is stable, and a sustained rise there signals a different problem. Each rule names the topic group it covers and the team that responds. The step is done when the growth rules cover every topic group in the baseline table and each group has a named responder.
Test alerts against a staging topic
In a staging cluster that matches production configuration, the team creates a test topic, writes data at a controlled rate until the growth rule and the warning rule fire, then stops the producer and confirms that the alerts resolve. They also stop one broker's exporter to confirm that the missing-target alert fires. Each test result is recorded with the time it fired and the time it resolved, so the team knows the alert path works end to end, including notification routing. The step is done when every rule has fired at least once in staging and its notification reached the named responder.
Review alerts after the first production period
After the rules run in production, the operations team reviews every alert that fired, separating true disk pressure from noise caused by normal batch loads or planned maintenance. They adjust thresholds in the runbook and in the rule files, and they record each change with its reason. The team also checks whether any topic has outgrown its retention plan and needs a change to retention or cleanup settings. The step is done when the threshold changes are committed to version control, and the runbook lists the review date and the owner of the next review.
Emergency steps when a broker runs out of disk
This sequence gets a full broker back to healthy operation by stopping the growth first, then reclaiming space through supported settings, and finally confirming that replicas have caught up. It is designed so that no log segment is ever removed by hand.
Confirm whether the broker is still running
Check the broker process and its logs before touching anything, because the next steps depend on it. If the broker has crashed on a full disk, it must be restarted with space freed first; if it is running but rejecting writes, you can use the live cluster tools. Whoever is on call records the state, the affected log directory and the free space reported by the operating system, so the rest of the team works from the same facts.
Stop or throttle producers to the affected topics
Reduce the write load on the full broker before reclaiming space, so that the disk does not keep filling while you work. Coordinate with the owning teams to pause or slow the producers writing to the largest topics on that broker, or to lower their send rate. The step is complete when the disk usage on the broker stops climbing across a short observation window, which the on-call engineer should confirm from the operating system's disk metrics rather than assume.
Identify the largest topics and partitions
Use the Kafka admin tooling and your size metrics to list the topics and partitions consuming the most space on the affected log directory. Compare each topic's retention settings with what its consumers actually need. The output is a ranked list of candidate topics with their current retention, compaction policy and replica placement, which becomes the basis for every decision in the following steps.
Do not delete segment files by hand
Never remove
.log,.indexor.timeindexfiles from the log directory withrmor similar tools. The broker tracks each segment in memory and in its recovery checkpoints, so manual deletion corrupts the log, can cause the broker to fail on restart and can make partitions unavailable. Every space recovery in this sequence goes through Kafka's own retention mechanism or through moving replicas with the supported tools.Shorten retention on the largest topics where safe
For each candidate topic, lower
retention.msorretention.bytesonly where downstream consumers have already read the data they need and the business owner agrees the loss of older history is acceptable. Apply the change with the admin client, then watch the broker log for the cleaner removing segments. Do not change compacted topics this way, because their size is governed by key cleanup rather than time. The step is complete when the log directory shows reclaimed space.Add disk or move log directories
If retention changes do not free enough space, add capacity to the broker or move partition replicas to brokers with room, using the supported reassignment process. Reassignment copies data over the network, so plan the throttle and confirm that the destination has enough free space before starting. The step is complete when the reassignment reports finished and the source directory no longer holds the moved replicas.
Verify the broker rejoins and replicas catch up
Restart the broker if it was stopped, then confirm it has registered with the cluster and that its partitions show as in sync with their leaders. Check the under-replicated partition count and the in-sync replica state for the affected topics, and keep producer throttles in place until they return to normal. Restore producer rates only after the cluster is healthy, and record the retention and placement changes made so they can be reviewed later.
Security and operational risks of retention changes
Retention is one of the few Kafka settings that deletes data on purpose, so a change to it can remove records that compliance, replay or a lagging consumer still needs. The cards below cover the risks to review before anyone shortens retention, adds headroom, or edits a topic's settings in production.
Retention can delete data you still need
Shortening retention.ms or lowering retention.bytes makes the broker delete closed log segments on its next cleanup pass, and that deletion is not undone by reverting the setting. Before a change, confirm with data owners which topics feed audit trails, regulatory retention periods, or downstream rebuilds, and check whether any of them depend on Kafka as the system of record. For those topics, the retention decision belongs in the data governance process, not in a disk alert response.
Lagging consumers lose unread records
A consumer group that is behind the log end can lose the records it has not read yet once retention removes them. The group then resumes at the earliest available offset or fails, depending on its auto.offset.reset setting, and the gap is silent unless someone is watching lag. Check per-group lag on every affected topic before shortening retention, and confirm each owning team has accepted the new window in writing.
Headroom for reassignment and recovery
Partition reassignments, replica catch-up after a broker restart and log directory moves all write new copies of data before old copies are removed. A disk that looks safe for steady-state traffic can fill during these operations, which turns a routine task into an outage. Keep free space above what a full replica copy of your largest partitions would need, and test that margin in a non-production cluster before relying on it.
Limit who can change retention
Retention and segment settings are topic configurations, so anyone with permission to alter topic configs can shorten them for every consumer of that topic. In KRaft clusters, the StandardAuthorizer keeps ACLs in the metadata log, and resources without ACLs are limited to super users unless allow.everyone.if.no.acl.found is enabled. Review which principals hold topic-config permissions, and restrict configuration changes to a small admin group that is separate from application service accounts.
Record every retention change
A retention change should leave a trail that says who made it, which topics changed, the old and new values, the reason, and the date. Kafka's own configuration history is limited, so store that record in your change management system or infrastructure-as-code repository, and tie it to a ticket that names the data owner who approved the change. This record is also the evidence auditors and incident responders ask for when data is missing.
Confirm changes before the cleanup runs
Retention changes apply on the broker's cleanup cycle, so the effect may appear later than the moment the command succeeded. Use kafka-configs.sh with a describe step before and after each change, then watch log size and the oldest offset per partition for the expected drop. Confirm the change on every topic that shares the same policy, and keep a rollback value ready in case a consumer reports missing data.
Questions teams ask before fixing Kafka disk usage
What drives the cost of a Kafka retention review?
The cost follows the number of topics, the number of brokers, and how many topics have settings that someone must justify. A review is mostly reading configuration, measuring how much data each topic writes per hour, and agreeing with topic owners how long each one must keep data. Clusters with many unowned topics take longer because owners have to be found before any decision can be made. Ask for the review scope in writing, listing which clusters, which topics and which deliverables are included, because that list is what the quote is built on.
Can our own team review Kafka retention without a partner?
Yes, if someone on the team can read broker and topic configuration, run the Kafka command-line tools against the cluster, and has authority to ask topic owners for retention decisions. The harder part is usually the business side: finding who owns each topic and what legal or operational retention applies to its data. A partner can add capacity and a repeatable method, but the decisions about what must be kept still belong to your organization. Check whether your team has time to do the inventory before deciding.
What does our team need to provide for a retention review?
Plan to supply three inputs: a topic inventory with owners, producer rates or write volumes per topic, and the retention requirements that apply to each data set. Retention requirements often come from data protection rules, replay needs of consumers, or contractual terms, so the people who know those should attend. Also provide read access to broker and topic configuration and to the metrics that show disk growth. Missing owners or requirements is the most common reason a review stalls.
How long does a Kafka retention review take?
Duration depends on cluster size, the number of topics, how many owners must be consulted and how many clusters are in scope. The timeline is set in the written scope, which should state the number of clusters, topics and review sessions included and the dates for each deliverable. Ask for the milestones and the client dependencies that could move them, such as owner availability or change windows on production brokers.
Will log compaction fix our growing Kafka disk usage?
Compaction only helps topics whose data has keys and whose value is a latest-state record, and it does not shrink the active segment or remove data that is still within the retained range. Topics without keys, event streams that need full history, and topics with frequent unique keys often keep growing. Compaction also needs its own cleaner capacity and can make brokers work harder. Check which topics use the compact cleanup policy, confirm whether consumers need full history, and measure per-topic size before deciding that compaction is the fix.
What should we do first when a Kafka broker is already full?
First, stop the data from growing: identify the topics writing the most data and confirm that producers are not stuck in a retry loop, then avoid restarting brokers before you know why the disk filled. Next, check whether any topic can have its retention reduced safely, after confirming with its owner that consumers have read the data they need. Do not delete log files by hand, because the broker tracks them. Work through the broker and partition moves with the Kafka tools and record every change, since a rushed fix can cause under-replicated partitions.
Does DeployOpen certify our Kafka storage for compliance?
No. DeployOpen can prepare the retention settings, the written retention decisions, and the evidence that shows how each topic is kept, which compliance assessors may review. Independent auditors and certification bodies decide whether the controls meet a framework. Ask any partner to confirm exactly which records they will hand over, and confirm with your assessor what evidence they expect before the review starts.
What to confirm in your environment before changing retention
Before you change any retention setting, confirm which Kafka release line your brokers run and whether the cluster uses KRaft. The downloads page lists 4.3.1, 4.2.2 and 4.1.2 as the supported releases, and anything older is archived, so a broker on an archived line is the first thing to fix. Confirm the mode too: a cluster still depending on ZooKeeper must migrate to KRaft before it can move to 4.x, and that migration changes how you plan any maintenance window. Read the Apache Kafka upgrade notes for your target release to see which behaviors and defaults apply to your version, and check them against your own configuration rather than assuming they match the examples you find online.
Next, get written retention requirements from the data owners for each topic. Ask how long each topic must remain readable, whether any topic must be kept by size instead of age, and whether any compacted topic holds data that must not be overwritten or deleted. Record which retention keys (such as retention.ms and retention.bytes) are set at the topic level and which are inherited from broker defaults, because a change at the broker level can silently affect topics whose owners never reviewed it. Also confirm the consumer lag each consumer group carries today. A group that is already far behind will lose unread records if retention shrinks, so the lag figure decides whether a shorter window is safe or whether the consumers must catch up first.
Finally, measure the disk headroom on every broker, not the cluster total. Compare current log directory usage with the capacity of each volume, and check whether the brokers share disks or use separate log directories. Ask the platform team how much growth the current producers generate per day, since retention only controls the ceiling. Then decide what the alert and emergency plan will be before you change anything. Confirm who receives disk alerts, who can run a reassignment or log directory change, and whether backups or mirrored clusters are relied on for recovery. Those answers determine how much headroom you need and whether retention changes alone will be enough.
How DeployOpen scopes and delivers Kafka retention and disk work
DeployOpen plans, implements, documents and supports self-hosted Apache Kafka inside your own infrastructure, including the retention settings, disk capacity alerts and recovery runbooks that keep brokers from filling up. Each engagement starts from a written scope that names the topics, retention rules and failure cases covered before any change is made to your cluster.
How we scope
- Inventory every topic, its retention by time and size, and whether it is compacted.
- Agree the per-topic size metrics and alert paths your operations team will receive.
- Record which log directories, disks and volumes the written scope includes and excludes.
How we build
- Set retention and segment settings per topic, then test their effect on disk use in a non-production cluster.
- Configure disk usage alerts and per-topic size dashboards in the monitoring stack you already run.
- Write and rehearse emergency steps for a broker running out of disk, including a dry run of each step.
How we hand over
- Deliver a runbook covering retention changes, compacted topic checks and disk recovery procedures.
- Document every configuration change, its owner and the reason it was made.
- Walk your platform and on-call teams through the alerts, runbooks and escalation paths.
How we support
- Offer ongoing support and upgrades as a retainer after the fixed-scope work is complete.
- Review retention settings and disk trends when topic volumes or teams change.
- Confirm with your team which support hours, response paths and upgrade windows apply before work starts.
