Kafka log compaction for changelog and state topics: when a team needs it
A team needs log compaction when a Kafka topic holds current state rather than history: consumer offsets, a derived table rebuilt from events, or configuration that services read on startup and expect to be the latest value per key. Set cleanup.policy to compact on such a topic and Kafka keeps at least the most recent record for each key in each partition, discarding older values for the same key during cleanup. A topic that should keep history for a fixed period uses delete instead, and delete,compact keeps both behaviors: records older than the retention window are deleted, and the remaining records are compacted by key. Choose the policy from the question the consumer asks. If the answer is "what is the latest value for this key," compaction fits. If the answer is "what happened in the last seven days," it does not.
Compaction is a background cleanup process, not an immediate guarantee. The log cleaner runs on segments that are no longer the active segment, so older values for a key can remain readable for a while after a newer value is written, and consumers that read from the start will see them. Consumers must therefore handle duplicates and older values safely, typically by letting the latest record for a key overwrite earlier state. Tombstones, records with a null value, mark a key for deletion, and delete.retention.ms controls how long those tombstones remain so that consumers who read the topic later still see the deletion. Confirm the cleaner and segment settings for your Kafka version in the Apache Kafka 4.3 documentation and test the timing against your own write rate before you rely on a deleted key disappearing.
How the log cleaner keeps the latest value per key
Compaction runs in the background on closed log segments, not on the segment currently being written. The cards below cover where the cleaner works, what it guarantees and what consumers will and will not see, so the design of a changelog or state topic matches what Kafka actually does.
Active and closed segments
Each partition log is split into segments. The active segment is the one the broker is still appending to, and the cleaner never compacts it. Only closed segments are candidates. This means a freshly written key can keep duplicate values for a while, and a topic with a large segment size may hold older duplicates longer than expected. Check your segment settings before you assume a compacted topic is already clean.
The log cleaner thread
A pool of background cleaner threads on each broker selects the dirtiest closed segments and rewrites them, keeping only the latest record for each key. The work consumes broker CPU, memory and disk I/O, so it competes with produce and fetch traffic. Ask for cleaner thread count and dirty ratio settings as part of your capacity review, and confirm the defaults for your release in the Kafka 4.3 configuration reference.
Head and tail of the log
The head is the cleaned portion of the log, where each key appears once. The tail is the uncleaned portion, where duplicates still exist. Consumers that start from the beginning read through the tail first, so a new consumer of a compacted topic may see several values for the same key before the cleaner catches up. Plan your bootstrap logic on the assumption that the tail exists.
Latest value per key is retained
The newest record for each key is always kept, even when it is older than the retention window that applies to deletes. A key only disappears when its latest record is a tombstone, meaning a null value, and that tombstone has aged past delete.retention.ms. Confirm this rule against the Kafka 4.3 log compaction documentation before depending on it for state recovery.
Offsets preserved, gaps visible
The cleaner removes older records but never renumbers the survivors, so offsets stay the same. A consumer reading a compacted partition will see offset gaps where removed records used to be. Code that assumes contiguous offsets, such as offset arithmetic to count messages, will report wrong results. Treat offsets as identifiers, not as record counts, and test this with a real consumer.
Compaction settings and when each one fits
This table compares the topic-level and broker-level settings that control how compacted topics are cleaned, and what each one trades off. Read each row as a question: what it controls, what it costs when you change it, and which workload usually needs it. Defaults are not listed here, so confirm the current values in the Kafka 4.3 documentation before you set them.
| Setting | What it controls | Trade-off when you change it | When it fits |
|---|---|---|---|
| cleanup.policy: compact | Keeps only the latest record per key in closed segments, so the topic grows with the number of distinct keys rather than with time. | Older values for a key are removed, so consumers cannot replay full history from the topic. | Changelog and state topics where only the latest value per key matters. |
| cleanup.policy: delete | Removes whole segments once they pass retention time or size limits. | Keys are not deduplicated, so a key can appear many times until its segment expires. | Event streams where every record matters for the retention window. |
| cleanup.policy: delete,compact | Applies key-based compaction and time or size-based deletion together. | Records are removed by age even if they are the latest for their key, so a quiet key can disappear. | Topics that need the latest state per key and a bound on how long stale keys remain. |
| delete.retention.ms | How long tombstone markers (null values) are kept so that consumers can see a key was deleted. | Shorter values free space sooner but can let slow consumers miss deletes. Longer values keep more markers on disk. | Any compacted topic read by consumers that rebuild state from the log. |
| min.cleanable.dirty.ratio | The share of a log that must be uncleaned before the cleaner selects it. | Lower values clean more often and use more broker I/O and CPU. Higher values save I/O but leave more duplicate keys on disk. | Write-heavy topics with frequent key overwrites, where disk use matters more than cleaner load. |
| min.compaction.lag.ms | The minimum time a record must stay uncompacted, so consumers have time to read it. | Longer lag keeps duplicates on disk for longer. Setting it too short can hide intermediate values from slow readers. | Topics with consumers that must observe most intermediate updates before they are removed. |
| max.compaction.lag.ms | The maximum time a record can remain uncompacted before the cleaner must process it. | A tighter bound forces more cleaning work and I/O. A loose bound can let disk use grow unexpectedly. | Compliance or storage limits where stale duplicates must not persist beyond a set time. |
| segment.ms and segment.bytes | When the active segment rolls, which controls which data the cleaner can process. | Small segments allow compaction to run sooner but create more files. Large segments delay cleaning of recent data. | Low-volume compacted topics that need timely cleanup, or high-volume topics where file count is a concern. |
| Broker log.cleaner settings | Number of cleaner threads, memory and I/O throttles used by the log cleaner across all compacted topics on a broker. | More threads or less throttling finish cleaning faster but compete with produce and fetch traffic. | Brokers that hold many compacted partitions and show a growing cleaner backlog in monitoring. |
Tombstones and delete.retention.ms: deleting a key safely
In a compacted topic, a record with a key and a null value is a tombstone. During compaction, the log cleaner removes older records for that key, and the tombstone itself stays in the log for a while so that consumers reading the log can see that the key was deleted. The tombstone is not removed immediately. Kafka keeps it for the period set by delete.retention.ms on the topic, after which the cleaner may remove it in a later pass. Once the tombstone is gone, a consumer that starts from the beginning of the log finds no record for that key at all, so it cannot tell that the key was ever deleted.
- Null value marks the delete: The record must have a non-null key and a null value. An empty byte array is a value, not a delete.
- Retention is per topic:
delete.retention.msapplies to the topic, so check it on every compacted topic you own. - Tombstones are not instant: The record remains readable until the retention period ends and a cleaning pass runs.
- Other records for the key are removed first: Older values for the same key can be dropped during compaction, while the tombstone is kept for the retention window.
- Check the effective value: Confirm the topic setting with your platform team rather than relying on the broker default.
Before you rely on a delete, confirm the topic's cleanup settings and the broker's cleaner configuration with the platform team.
Teams writing deletes need to send the tombstone with the same key and partitioning as the original records, or the delete will never reach the compacted state for that key. A tombstone sent to a different partition does not remove anything, because compaction works within a partition. The producer should also treat a delete as a normal, durable event and confirm the write before reporting success to the caller. The record's timing is set by the producer, so the producer team should understand how long the tombstone will remain visible to readers.
- Use the same key serialization: Byte-for-byte equal keys route to the same partition and match the earlier records.
- Send a null value, not an empty payload: Verify this in tests because some client wrappers convert null into an empty value.
- Wait for acknowledgement: Check the send result or the producer callback before marking the deletion as complete.
- Document the delete contract: State which fields in downstream systems will disappear and when.
- Test with a sample key: Produce a value, produce a tombstone, and inspect the topic with a console consumer to confirm the sequence.
Ask your platform team how long tombstones are kept on your topics and whether any downstream system requires a delete to remain visible for longer.
Consumers that rebuild state from a compacted topic depend on seeing every tombstone, or they must accept that some deletes may never appear. A consumer that reads from the beginning of the topic after a long outage, or that starts from a new group after the tombstone has been removed, can end up with stale keys in its local store. The risk is greatest for consumers that were offline longer than the delete retention period, because the cleaner may already have removed the tombstones they needed. Consumers should design their state store so that a missing delete is detected and handled, rather than assuming the log is a complete history.
- Handle null values explicitly: When a record has a null value, remove the key from local state instead of storing an empty object.
- Plan for late starts: A new consumer group reading from the beginning may not see deletes that were already cleaned.
- Compare with the source of truth: For critical state, reconcile against the system that produced the data at regular intervals.
- Track offsets carefully: Resetting offsets to an old position can replay values without their tombstones.
- Record the last successful rebuild: Keep evidence of when state was last rebuilt from a complete log.
Before deploying a rebuild path, confirm with your platform team the retention period for each compacted topic and the longest outage your consumers must survive.
The most common failure with tombstones is a retention setting that is shorter than the time a consumer can take to catch up. If delete.retention.ms expires before a slow or stopped consumer reads the tombstone, that consumer will never learn about the delete from the log. Retention must be set longer than the worst realistic lag, including maintenance windows, failed deployments, and the time needed to restore a consumer from backup. The number to choose depends on your own recovery procedures, not on a general rule, so the platform and application teams should agree on it together.
- Measure the slowest consumer: Use consumer lag metrics to find the longest time any group has fallen behind.
- Include recovery time: Add the time needed to restore or redeploy a consumer that holds state.
- Set retention for the worst case: Choose the topic value so it covers that worst case with margin.
- Review after changes: Revisit the setting when a new consumer group is added or when batch jobs are scheduled.
- Plan for retention cost: Longer retention keeps tombstones and cleaned records on disk for longer, so storage sizing must reflect it.
For each compacted topic, write down the slowest consumer, the longest expected outage, and the delete.retention.ms value that covers both.
Typical uses: Connect offsets, Streams changelogs and other state
Do Kafka Connect offset topics need log compaction?
Yes, Connect offset, config and status topics are normally created as compacted topics, so the latest value for each key is kept and older values are eventually removed. Connect workers read these topics on startup to recover where each connector left off. The consumer relies on the newest record per key being present. It must not assume the full history of offset commits is still there. Confirm the topic names and cleanup settings your Connect version creates, because they are set by the worker configuration and can differ from defaults in your environment.
What does a Kafka Streams changelog topic store?
A Streams changelog topic records every update to a state store so the store can be rebuilt after a failure or a task move. Streams creates these topics with compaction so that the latest value per key is enough to restore state. A restoring instance reads the changelog from the beginning and replays it into a local store. It should not assume that old intermediate values, or values for deleted keys, remain available once compaction and tombstones have run. Check the changelog configuration Streams applies on your target version before relying on retention for recovery time.
Can a consumer rely on seeing every event in a compacted topic?
No. A compacted topic keeps only the most recent record for each key after the cleaner runs, so consumers that read from the start may see only the latest state for a key and not each change. Records still in the active segment are not yet compacted, so duplicates of a key can appear for a while. Consumers should be written to treat each record as the current value for its key and should not use the topic as an audit trail. If you need full history, use a separate topic with the delete policy.
How do tombstones affect consumers of a compacted topic?
A tombstone is a record with a key and a null value, and it tells compaction to remove that key. Consumers that treat a null value as a delete will stop showing the key, which is the intended behavior for state. The tombstone itself stays in the topic for delete.retention.ms so that consumers reading slowly can still see the delete. After that period, the tombstone is removed and a consumer starting fresh will not learn that the key was ever deleted. Confirm how your application handles null values and the retention window you set.
Should a custom materialized view use a compacted topic?
A compacted topic suits a custom materialized view when each record is a full snapshot of one entity and the view only needs the current state. Examples include a customer profile or account balance keyed by identifier. Consumers rebuild the view by reading the topic from the beginning, so the topic must hold the complete latest state for every key. If a view needs event ordering, aggregates over time or a record of every change, a compacted topic is the wrong choice and a delete-policy topic or an event log is more suitable.
Who should own the settings for Connect and Streams topics?
The platform team usually owns broker-level defaults, while the application or Connect owners set topic-level settings through their worker or Streams configuration. Ask for a written list of every internal topic created, its cleanup policy, retention settings and replication factor. Review that list in each upgrade because internal topic defaults can change between releases. Make sure someone also owns the decision to delete or recreate these topics, since a wrong change can force a full restore or a connector offset reset.
What should I confirm about these topics before going live?
Confirm the cleanup policy, min.cleanable.dirty.ratio, delete.retention.ms and segment settings on each compacted topic, and check whether they match what Connect or Streams create on your exact Kafka version. Test a connector restart and a Streams state restore from a clean instance, and measure how long restoration takes. Also confirm that no one has changed the topic configuration manually. Record the results so the restore procedure is known before an incident.
Can DeployOpen review these internal topic settings for our cluster?
DeployOpen can review the Connect and Streams configuration, the topics they create and the restore behavior on a self-hosted cluster as part of a fixed-scope engagement. The scope, deliverables and timeline are agreed in writing before work starts, and a quote follows from that scope. Your own team can also check these settings, using the Kafka documentation for the release you run, if they have the time and the access to test restores.
Setting up a compacted topic and verifying it
This sequence takes a compacted topic from key design to a verified state, so the team knows the topic keeps the latest value per key and that a fresh consumer reads what the application expects. Each step ends with a check you can record before moving on.
Define the key design and record schema
The application owner decides which field forms the record key and confirms that every change to the same entity produces the same key. The team also fixes the value schema and the rule for deletes, so a tombstone (a record with a key and a null value) has a clear meaning. Done when the key, value schema and delete rule are written down and reviewed by the consuming team. If the key is not stable across producers, compaction will keep duplicate entities and the design must be fixed before the topic exists.
Choose the cleanup policy
Platform and data owners select
cleanup.policyfor the topic. Usecompactwhen only the latest value per key matters, anddelete,compactwhen the topic also needs time or size based removal of old segments. Confirm the choice against the consumer's needs, because a consumer that starts from the beginning of acompacttopic sees only the latest value per key, not the full history. Done when the policy is recorded with the reason for it.Set tombstone retention and cleaner lag settings
The owner sets
delete.retention.msso tombstones remain long enough for slow consumers to observe deletes before the cleaner removes them. The owner also reviews the cleaner settings that control how soon closed segments become eligible for compaction, such asmin.cleanable.dirty.ratioandsegment.ms, and records the values in the change request. Confirm the exact key names and defaults for your release in the Kafka documentation for your version before you set them. Done when each value is written down with the consumer lag it tolerates.Create the topic with explicit configs
The platform team creates the topic with partition count, replication factor and all compaction settings stated on the command line or in the infrastructure definition, instead of relying on broker defaults. Store the command or manifest in version control so the configuration can be reviewed and reproduced. Done when the topic exists and a describe command shows the cleanup policy and tombstone settings exactly as recorded in the previous steps.
Produce test keys, updates and tombstones
A test producer writes several keys, updates some of them with new values, and then sends a tombstone for at least one key so the delete path is exercised. Use a non-production environment and a sample of payloads that match the real schema. Record the offset and key of each produced record in a test log. Done when the test log matches the sequence you intended, including the tombstone for the deleted key.
Confirm cleaning with describe commands
The operator checks topic and log state with the admin tools for your release, looking at the topic configuration and the segment and cleaner information the tools expose. Compaction runs in the background on the broker, so this check may need to wait for the cleaner to act on closed segments. Confirm the commands and output fields against your Kafka version before running them in production. Done when the operator can show that older values for test keys have been removed from closed segments and that the tombstone is present while within its retention window.
Validate a fresh consumer reads the final state
A new consumer group with no committed offsets reads the topic from the earliest offset and builds a map from key to value, applying tombstones as deletes. The result is compared with the expected final state from the test plan. Repeat the read after the cleaner has run to confirm the outcome does not change. Done when the consumer output matches the expected state for every test key, and the team records the consumer group, the read date and the comparison result as evidence.
Security and operational risks of compacted topics
Compacted topics keep the latest value per key, but that behavior creates risks that a simple retention setting does not. The cards below cover who can write tombstones, how long personal data really persists, what cleaning costs in disk and CPU, and which signals show the cleaner is falling behind.
Restrict who writes tombstones
A tombstone is a record with a key and a null value, and it tells compaction to remove that key. Anyone with write access to a compacted topic can therefore delete state that other services depend on. Grant write permission on changelog and offset topics only to the service principals that own them, and review those grants in the KRaft ACL metadata. Confirm the rules against the authorization documentation for your Kafka version, including how the cluster treats resources that have no ACLs.
Personal data persists until compaction
Compaction does not delete a key on a schedule. An old value for a personal data key stays readable until the cleaner rewrites its segment and a tombstone is applied. Retention for such topics depends on segment roll timing, the active segment, and the delete retention period for tombstones. Ask your privacy and legal teams to define the maximum acceptable time a deleted person's record may remain in the log, then test that window against your cleaner settings rather than assuming it.
Tombstones have their own lifetime
Consumers that start after a tombstone has been removed will never see the delete marker, so they may hold stale state or assume a key was never deleted. Set the delete retention window long enough for your slowest consumer to read the tombstone, and document that window as part of the topic contract. Changing it on an existing topic affects which deletes remain visible, so treat it as a controlled change.
Cleaner disk and CPU cost
Compaction runs on brokers, so it competes with produce and fetch traffic for disk I/O and CPU. Large numbers of distinct keys, short segment roll intervals and high write rates all increase the cleaning work. Size broker disks for the uncompacted active data plus the space the cleaner needs to write new segments, and ask your platform team to measure cleaner CPU and I/O during a load test that reflects your key cardinality.
Watch cleaner health continuously
A cleaner thread that fails or falls behind lets the log grow without bound, and nothing in the client output shows it. Monitor the log cleaner metrics exposed by the brokers, the size of the log's dirty ratio, and disk usage trends for every compacted topic. Alert when cleaning stops making progress, and set a runbook that explains how to inspect the cleaner logs on a broker before disk pressure becomes an outage.
Protect compacted state from loss
Because a compacted topic holds only the latest value per key, a bad write or accidental tombstone is hard to reverse from the log alone. Keep replication factor and minimum in-sync replicas aligned with your durability requirements, and back up the source of truth for the state (for example, the upstream database or a snapshot of the application state store). Test a restore that rebuilds a compacted topic from that source, and record how long it takes.
How much effort compaction takes and what the work needs from your team
Can our own team configure log compaction without outside help?
Yes, for a single topic with simple keys, a team that already runs Kafka can set cleanup.policy to compact or delete,compact and monitor the result on its own. The effort rises when several teams share the cluster, when keys are complex, or when consumers must rebuild state from the log. Most of the work is in deciding which topics need compaction and how producers emit keys and tombstones, not in the broker setting itself. If your engineers have not run Kafka in production before, a scoped review of the design is the lower-risk step.
What drives the effort of a compaction project?
Four factors move the effort most: key design, state size, consumer rebuild time and testing. Key design decides whether the latest value per key is meaningful, and changing keys later means rewriting history. State size determines how long a new consumer takes to read a compacted topic from start to finish. Consumer rebuild time matters for Kafka Streams changelogs and Connect offset topics, where a restart may replay the full compacted log. Testing covers tombstone behavior, retention windows and failure recovery, and it usually takes more time than the configuration change.
How long does it take to validate compaction on a topic?
Validation time depends on how often the topic is written, how long delete.retention.ms is set, and how many consumers must be tested. Compaction runs asynchronously in the background, so a short test does not show what production will do over weeks. Plan a test window long enough to see cleaner runs, confirm that tombstones are removed after the retention window, and rehearse a consumer restart from the beginning of the log. Ask any partner for a written timeline that names these checks instead of a single duration.
What must our team provide before compaction work starts?
Three inputs make the scope concrete:
- A topic inventory listing each topic, its producers, its consumers and its current cleanup policy.
- Retention requirements, including how long each key's latest value must remain readable and how long deletions must be visible to downstream systems.
- An access model that states which teams may produce, consume, create and alter topics, based on the Kafka authorization model.
Without these, the design cannot confirm which topics should be compacted or what consumers depend on.
Which access controls affect how compacted topics are managed?
In KRaft clusters, the StandardAuthorizer stores ACLs in the cluster metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Compacted topics that hold offsets or state need tight write access, because a producer with the wrong key can overwrite or delete state for other consumers. Review who can alter topic configuration, since changing cleanup.policy on a live topic affects every consumer of that topic. Confirm these rules against the authorization documentation for your release before you finalize the access model.
Should we hire help for log compaction, and what should we ask for in a quote?
Hire help when the topics are shared across teams, when Streams or Connect state is business critical, or when your team lacks time to test restarts and tombstone behavior. DeployOpen plans, implements and documents this work in your own infrastructure, and most engagements are fixed-scope with scope, deliverables and timelines agreed in writing before work starts. Ask any provider for a written scope that lists topics in scope, the design review, the test plan, the handover documents and any ongoing support. DeployOpen does not publish prices, so a quote follows from that written scope.
What recurring work follows after compacted topics go live?
Compacted topics need periodic checks rather than constant attention. Teams should monitor the log cleaner, watch consumer lag on state topics and confirm that deleted keys disappear after the expected window. Any change to key format, retention or cleanup policy becomes a new change that must be tested against consumers. Ongoing support, upgrades and testing can be covered by a retainer, and the cost of that retainer should be compared with the internal hours your team would spend on the same checks.
Does compaction change how the platform meets compliance evidence needs?
Compaction can affect evidence because it removes older values for a key once cleanup runs. If your retention rules require keeping every historical event, use a separate topic with the delete policy for that purpose and keep compacted topics for latest state only. Document which topics hold personal or regulated data, how deletions propagate, and how the retention schedule is enforced. DeployOpen prepares systems, controls and evidence for compliance assessments, but independent auditors issue the actual assessment.
How DeployOpen scopes and supports Kafka compaction work
DeployOpen plans and implements compacted topics for changelog and state data inside the infrastructure you control, starting from the consumers that depend on them. Each engagement is fixed-scope, with the topic configuration, retention behavior and access model agreed in writing before any change reaches a production cluster.
How we scope
- Inventory every topic that needs compaction, including Connect offset and status topics and Streams changelog topics
- Record which consumers rebuild state from the log and what they expect to see after a tombstone
- Agree in writing the cleanup.policy values, delete.retention.ms settings and the Kafka version under test
- Confirm ACL requirements for topic creation and configuration changes, and who approves them
How we build
- Create or reconfigure topics with compaction settings, applied through reviewed configuration files rather than manual edits
- Test key design, tombstone handling and compaction timing against representative data in a non-production cluster
- Verify that consumers restart, replay the compacted log and reach the same state as before the change
- Document the producer keying rules and the disk sizing assumptions for compacted topics
How we hand over
- Deliver a runbook covering topic settings, tombstone conventions, replay procedures and rollback steps
- Provide test evidence showing consumer rebuild results on the compacted topics in scope
- Walk your platform and application teams through what compaction guarantees and what it does not guarantee
- Transfer configuration and documentation to your repositories so your team owns the running system
How we support
- Fixed-scope delivery ends at handover, and ongoing support is available as a retainer
- A retainer can cover configuration reviews when new changelog or state topics are added
- Support can include checks on compaction lag, disk growth and consumer behavior after Kafka upgrades
- Ask any provider to confirm how upgrade testing against your Kafka release is scoped and priced
