Why Kafka Streams state stores need planning before go-live
A stateful Kafka Streams application keeps part of its working data on the instance that processes it. Aggregations, joins, windowed counts and deduplication all read and write a state store, a local key-value structure that sits next to the processing threads. For persistent stores, Kafka Streams uses RocksDB by default, so the data lives on the local disk of each application instance rather than in memory alone. The application can be fast because reads and writes are local, but the instance that holds the store is the only place the current state exists at that moment.
Durability comes from a *changelog topic*. Every update to a store is also written to a changelog topic in Kafka, and that topic is the durable copy. When an instance is replaced, when a rebalance moves a task to another instance, or when a local disk is lost, the new owner rebuilds the store by replaying the changelog from the beginning or from its last checkpoint. The time this replay takes depends on the size of the changelog and the throughput of the cluster, so the restore path is part of the application's recovery plan, not an afterthought. Teams should plan this before go-live if their state grows with traffic, if their business tolerates only short outages, or if they run on Kubernetes where pods and volumes are replaced routinely.
Before go-live, confirm the store types the application uses, whether their changelogs are enabled, how many replicas of standby state the design needs, and how much disk each instance can dedicate to RocksDB. Measure the restore time from a realistic changelog size in a test environment, and decide what a rebalance during a large restore is allowed to cost. The sections that follow turn these questions into sizing, persistence and monitoring decisions.
How restore time after failures is determined
When an instance of a Kafka Streams application fails or its state is moved, the replacement does not start from a blank store. It rebuilds each local RocksDB store by replaying the store's changelog topic from the beginning of the state it needs, so restore duration is set by how much state must be replayed, how many changelog records that represents and how fast the new instance can read from Kafka and write to its local disk. Restore is not the same as the normal processing path. Records are applied to rebuild the store, and the application does not process new input for the tasks being restored, so the delay is visible to downstream consumers. Changelog volume depends on write rate, overwrite patterns and whether the changelog is compacted, so a store that is heavily updated can take far longer to restore than its final size suggests.
Standby replicas reduce this cost by keeping a warm copy of a task's state on another instance. Standbys consume the same changelog continuously, so when a task moves, the new owner usually needs to catch up only on the records written since the last sync rather than the full history. The trade-off is extra CPU, network and disk on the standby hosts, and the number of standbys is a sizing decision that belongs in the same plan as memory and disk. Check the standby settings in the Streams configuration for your version, and confirm how rack-aware assignment interacts with them in your topology, because a standby on the same failure domain as its active task protects less than it appears to.
Restore time has to be measured in your environment. Disk type, network path to the brokers, broker load during a failure, the number of tasks that move at once and the state size in your own test data all change the result, and vendor or community figures do not transfer directly. Run a controlled test that kills an instance under realistic input rates, record the time until each affected task reports running, and repeat it after you change the standby count or the store configuration. Watch the restore metrics exposed by Kafka Streams and the RocksDB metrics described on this page, and set your recovery objective from the measured numbers rather than from an assumed duration.
Which state store and tuning options fit which workload
This table compares persistent and in-memory stores, changelog retention, standby replicas, and RocksDB memory and cache settings for common Streams workloads. Read each row as a question to answer for your application, then confirm the default and valid values for your Kafka version before you set them.
| Option | What it trades off | When it fits | What to confirm |
|---|---|---|---|
| Persistent (RocksDB-backed) store | Keeps state on local disk and survives restarts, at the cost of disk I/O and RocksDB tuning | Large or unbounded key sets, or any state too big to hold comfortably in heap memory | Store type defaults and RocksDB configuration hooks for your Streams version |
| In-memory store | Fast lookups, but state must fit in memory and is rebuilt from the changelog after a restart | Small, bounded state where restore time from the changelog is acceptable | Heap sizing for your instance and the memory headroom your JVM needs |
| Changelog topic retention and compaction | Compaction keeps the latest value per key so restores stay bounded, while time-based retention can drop state that restore still needs | Compacted changelogs for key-value state; check retention before relying on time-based deletion | Cleanup policy that Streams sets for internal changelog topics in your version |
| Standby replica count | Each standby holds a warm copy of state, so failover is faster but storage and network use rise with each replica | Stateful services where restore time after a failure is a stated business requirement | Maximum standby setting, rack-aware placement support, and extra disk per instance |
| RocksDB block cache and memory settings | A larger cache improves read latency but competes with the JVM heap and off-heap memory on the same host | Read-heavy stores with a working set that fits in cache | Off-heap memory accounting and the cache and write buffer keys exposed by your release |
| RocksDB write buffer and compaction settings | Larger write buffers reduce write amplification but increase memory use and flush size | Write-heavy stores with frequent updates to the same keys | Compaction style and write buffer defaults for your RocksDB build |
| Disk sizing for state directories | State plus changelog overlap, RocksDB compaction temporarily needs extra space, and standbys add copies | Any persistent store, sized with headroom for compaction and standby copies | Per-instance state directory size under load, measured in a staging run with your data |
| Restore and rebalance behavior | Large state with long restores delays task reassignment, so restore time depends on state size and changelog volume | Services with frequent deploys, autoscaling or node replacement | Restore time measured on your data volumes, and whether standbys or warm-up settings reduce it |
What drives memory and disk sizing for RocksDB state
Memory and disk for a Kafka Streams application are set by how many state partitions each instance owns, how many keys those partitions hold, and how long changelogs are kept. RocksDB then adds its own memory and compaction overhead on top, so sizing has to cover both the data and the engine that stores it.
Partitions and tasks per instance
Each input partition becomes a task, and each stateful task owns its own set of RocksDB stores. Fewer instances mean each one holds more tasks, more open stores and more on-disk state directories. Adding instances spreads that load but adds standby replicas if you enable them, which doubles the state footprint for each standby. Check the task count per instance against the thread count and disk you plan to provide before choosing a partition count.
Key cardinality and value size
Store size grows with the number of distinct keys and the size of each serialized value, not with message throughput alone. A windowed aggregation keeps entries for every key in every open window, so cardinality multiplies with window count. Ask the application owners for the expected key count per partition at peak and the largest value size, and measure a representative sample before sizing disks.
Changelog retention and compaction
Each store writes a changelog topic that restores it after a failure, and that topic is usually compacted rather than time-deleted. Compaction keeps the latest value per key, so disk use tracks live key count plus the uncompacted tail, but tombstones from deletes and windowed stores with retention settings behave differently. Confirm the cleanup policy on every internal changelog and the retention set for windowed stores, since these settings decide how much history the broker stores.
Block cache and write buffers
RocksDB keeps frequently read blocks in a block cache and buffers writes in memtables before flushing them to disk. Both sit outside the JVM heap in native memory, so a container limit that covers only the heap will be exceeded under load. Kafka Streams lets you bound this memory through a custom RocksDB configuration setter, so the cache and write buffer sizes should be set per application and tested against the container limit the platform team enforces.
JVM heap versus off-heap memory
The JVM heap holds Kafka Streams and client objects, record caches and buffers, while RocksDB allocates native memory outside it. A large heap leaves less room for the native memory that RocksDB needs, and a small heap can cause garbage collection pauses under load. Budget the heap, the native memory and the operating system overhead as separate items in the container memory request so that a pod is not killed for memory it did not plan to use.
Compaction headroom on disk
RocksDB rewrites SST files during background compaction, which temporarily needs free space beyond the live data size. Disks that run close to full can stall compaction, slow writes and trigger restore delays after a restart. Plan disk capacity as live state plus changelog history plus a margin for compaction, and alert on free space well before that margin is used. Ask the storage team for the volume size and the expansion process, since a Kubernetes persistent volume may not resize without a restart.
Setting up stateful Streams apps on persistent storage
This sequence takes a Kafka Streams application from a version check to a verified stateful deployment, so that its RocksDB state survives restarts and can be rebuilt within a time you have measured. Each step ends with a check that tells the team whether to move on.
Confirm Kafka and Java versions
The platform engineering lead and the application team confirm the Kafka cluster version and the Java runtime for the Streams application. The Kafka upgrade notes state that Kafka Streams applications require at least Java 11, and that brokers require Java 17 from Kafka 4.0 onward. Check the supported releases on the Kafka downloads page and confirm the cluster is on a release that still receives fixes. The step is done when the team has written down the broker version, the client library version and the JVM version in the design record, and each one matches the upgrade notes for the chosen release.
Define store names and changelog settings
The application developers name every state store explicitly in the topology, so that the store name, its changelog topic and its restore behavior are stable across redeploys. Changing a store name or topology without a plan can strand state or trigger a full rebuild from the changelog. Review the changelog topic configuration with the platform team, including its replication factor and retention or cleanup policy, and confirm it is not left to broker defaults that nobody has reviewed. The step is done when a topology dump lists each named store with its changelog topic and the platform team has signed off on those topic settings.
Set the state directory on durable disk
The operations engineer sets
state.dirto a path on a volume that survives process restarts and pod rescheduling. In Kubernetes, this means a persistent volume bound to each stateful pod rather than the container's ephemeral filesystem. Confirm that each instance gets its own directory, that the volume has enough free space for the RocksDB files plus headroom, and that the volume is not shared between instances. The step is done when a test instance is restarted and the existing store directories are reused rather than recreated, which you can confirm in the application logs and the directory listing.Enable standby replicas and rack awareness
The application team sets
num.standby.replicasto a value agreed with the platform team, so that a warm copy of each store exists on another instance and failover does not start from an empty state. Where the cluster spans failure domains, the team also decides how rack-aware assignment should place standbys. Standbys consume disk and memory on other instances, so the sizing review in the earlier design step should account for them. The step is done when the deployed configuration shows the chosen standby count and the application logs report standby tasks being assigned to other instances.Run a restore test with realistic state
The team loads a copy of production-like data into a non-production environment, stops one instance, and starts a replacement on a clean volume. The restore is timed from the moment the replacement starts until the store reaches the end of its changelog and the application reports running. Use a state size that reflects production, because restore time depends on the data volume and not on the size of the test topic. Record the measured restore duration, the state size on disk and the peak memory. The step is done when the team has a recorded restore time that the operations owner accepts as the recovery expectation for this application.
Verify RocksDB and restore metrics before production
The operations team confirms that the Streams and RocksDB metrics are exported to the monitoring system and that dashboards and alerts exist for restore progress, store size, disk usage and rebalance activity. Confirm the metric names against the Kafka version in use, since they are documented per release. Test at least one alert by forcing a condition such as low disk space in the test environment. The step is done when every metric the runbook depends on shows data in the dashboard, and the alert test fires and clears as expected.
Keeping Streams state on Kubernetes persistent volumes
Kafka Streams keeps its RocksDB stores on the local disk of each application instance, so running it on Kubernetes means giving each instance a stable place to write that data. A StatefulSet with a volume claim template is the usual pattern, because each pod gets its own named PersistentVolumeClaim that stays attached to its ordinal identity across restarts. A Deployment with a shared claim does not fit this model, since two Streams instances writing to the same state directory would corrupt each other's stores. Configure the state directory (state.dir) to a path on the mounted volume, and keep one claim per instance.
- Stable identity: Each pod name maps to one claim, so a restarted pod can reopen the stores it previously wrote.
- Separate claims per replica: Scaling out creates new claims, and scaling in leaves old claims behind unless you delete them.
- Mount path: Point
state.dirat the mounted path and confirm nothing else writes to that directory. - Claim retention: Decide whether claims survive StatefulSet deletion, since that choice determines whether state is kept during a redeploy.
- Ordered rollout: StatefulSet updates roll pods one at a time, which limits how many instances rebalance at once.
Before production, confirm the claim template, mount path and retention policy against your Streams application's rebalance behavior in a test cluster.
RocksDB reads and writes frequently during processing and compaction, so the storage class behind each claim affects latency and how quickly stores recover. The Kubernetes side only exposes a requested size and class name; the actual throughput, latency and snapshot behavior come from the storage provider, whether that is a local disk provisioner, a block storage system or a software-defined storage layer. Local volumes are often faster but tie a pod to one node, while network-attached volumes move with the pod but add I/O latency. Choose the class with the Streams team, not only the platform team.
- Local versus network volumes: Local volumes give fast disk access but pin the pod to a node; network volumes allow reattachment elsewhere at a latency cost.
- Volume binding mode: Delayed binding lets the scheduler choose a node that matches the volume, which matters for local storage.
- Expansion support: Confirm whether the class allows online volume expansion before you size stores at launch.
- Reclaim policy: A Delete policy removes state with the claim; a Retain policy keeps the data but needs manual cleanup.
- Measured I/O: Benchmark random reads and writes under expected store sizes, not only sequential throughput.
Ask the storage provider for documented behavior on binding, expansion, reclaim and snapshots, and confirm each one in your own environment.
When a pod is rescheduled, the new instance either attaches to its original volume or starts with empty storage, depending on where the volume lives. If the volume follows the pod, the instance reopens existing stores and only replays the changelog records written since the last checkpoint. If the volume is tied to a failed node and cannot move, the instance has to rebuild its stores from the changelog topics on the brokers, which can take much longer for large state. Plan for both cases, because a node loss is a more common trigger for full restoration than a simple pod restart.
- Pod eviction: A pod evicted from a healthy node can often reattach a network volume, reducing restore work.
- Node failure: A failed node with a local volume leaves its state unreachable, so the pod restarts with empty storage.
- Standby replicas: Standby tasks keep warm copies of state on other instances, which shortens failover when configured.
- Rack or zone placement: Spreading instances across failure domains limits the chance that one zone holds all copies of a store.
- Disruption budgets: Pod disruption budgets slow voluntary evictions during node drains.
Test a node drain and a forced node loss in a staging cluster, then measure how long each instance takes to return to processing.
Reusing a persistent volume saves restore time but adds operational risk: the volume must match the application version, the state directory must be clean of stale files, and the volume must belong to the instance that wrote it. Restoring from the changelog is slower for large stores, but it rebuilds state from the source of truth in Kafka and avoids trusting a volume that may be damaged or out of date. Many teams reuse volumes for routine restarts and accept changelog restoration after node loss, storage faults or major application changes. The break-even point depends on store size, changelog throughput and how often each path occurs.
- Restore cost: Changelog restoration reads the store topic from the beginning or from the last valid position, so its cost grows with store size and retention.
- Stale state: Reusing a volume after an application change can mix incompatible state, so decide when to clear it.
- Topic retention: Changelog topics must retain enough history for the restore path you choose, and compacted topics suit this use.
- Clearing the directory: Define a runbook step that deletes local state safely when a restore must start clean.
- Recovery drills: Run each path during testing so the team knows the time and steps involved.
Confirm the changelog retention and cleanup policy for each store with the Streams team before deciding which path is the default.
Which RocksDB and Streams metrics to watch
Kafka Streams state problems usually show up first as slow restoration, rising cache misses or growing disk use, long before a task fails. The cards below group the signals worth alerting on, so each alert maps to a decision about capacity, tuning or a stuck instance.
State store restoration progress
Restoration determines how long a task stays unavailable after a restart or rebalance, so it is the first signal to alert on. Track restore progress per task and compare the time to restore with your recovery target. Look for restorations that stall rather than slowly finish, because a stalled restore usually points to a changelog topic problem, a throttled consumer or an instance short on disk. Confirm exact metric names against the Kafka 4.3 metrics documentation and your exporter.
Record cache hit and eviction rates
The Streams record cache sits in front of each state store and absorbs repeated writes before they reach RocksDB and the changelog. A falling hit rate or rising eviction rate means more reads and writes hit disk and more changelog traffic is produced. Watch these per store, because one heavily keyed store can hide behind healthy totals. Use the trend after a configuration or traffic change to decide whether the cache size needs revisiting.
RocksDB operation latency
Average and tail latency for get, put and iterator operations show whether the local store keeps up with the input rate. Latency that rises with traffic while CPU stays low often points to I/O limits on the volume holding state. Compare latency across instances running the same application, since one slow instance usually indicates a disk, noisy neighbor or uneven partition assignment rather than an application defect.
Compaction and write stall indicators
RocksDB compacts SST files in the background, and write stalls occur when compaction cannot keep up with incoming writes. Track pending compaction bytes, running compactions and stall counters together. Sustained stalls during peak load are a signal to review compaction settings, add I/O capacity or reduce the write amplification of the store. Treat any stall that correlates with processing lag as an alert, not a dashboard curiosity.
State directory disk usage
Each Streams instance keeps its state directory on local disk, and RocksDB files can grow faster than expected during restores, when old and new files briefly coexist. Alert on used and free space per volume with a margin that covers a full restore of the largest task on that instance. Also watch for state directories left behind by tasks that moved elsewhere, since they consume space without serving any task.
Changelog lag and standby freshness
Changelog topics feed restores and standby replicas, so their lag determines how current a recovery point is. Monitor the offset gap between a store and its changelog, and confirm standby replicas stay close to the active task. Lagging standbys mean a failover would restore more data than planned. Confirm the metric names for these views in the Kafka 4.3 metrics documentation before building dashboards on them.
Questions teams ask before running stateful Streams apps
How much effort does running stateful Kafka Streams apps take?
Effort depends mostly on how many applications keep local state, how large those stores grow, and how many environments you must keep in step. Each stateful application adds RocksDB stores, changelog topics that Kafka must retain and replicate, and a restore path you have to rehearse. The one-time work covers topology review, store naming, changelog and retention settings, sizing, and dashboards. The recurring work covers watching disk and memory, reviewing restore behavior after releases, and tuning when data volumes change. Ask any partner to break the quote down by application, by environment and by one-time versus recurring work, and to state which assumptions about state size and throughput it used.
Can an internal team run stateful Streams apps without outside help?
Yes, if the team already runs Kafka in production and has engineers comfortable with JVM tuning, disk management and Kubernetes storage. The hard parts are less about the Streams API and more about operations: understanding why a restore is slow, sizing volumes that are hard to resize later, and recognizing when a rebalance is moving large stores between instances. Teams without this experience often do well with a short design review and a runbook built with a partner, then take over operations. Confirm your team's skills against the Streams and RocksDB areas listed in the Kafka documentation before deciding, and check the upgrade notes for the Kafka release you plan to run.
How long does restoration testing take, and what drives it?
The duration of restoration testing is driven by total state size, the number of changelog records that must be replayed, the number of partitions, network and disk throughput, and how many standby replicas exist. A realistic test loads production-like volumes, kills an instance, and measures how long the replacement takes to rebuild each store before it serves traffic. Plan the test as a fixed step in the schedule, not as a side task, and repeat it after major changes to volume or topology. Ask for the test plan, the data volumes used, and the acceptance criteria in writing before work starts, because no general duration applies across workloads.
What does the work need from our team?
The work needs named owners for the Kafka cluster, the Streams applications, the Kubernetes storage layer and the observability stack. In practice, you will provide access to a non-production cluster with realistic topic configurations, sample or synthetic data at production-like shape, the current application topologies, and the storage classes available on your clusters. You will also need a decision on who approves changes to retention and changelog settings, since those affect cost and recovery time. Expect to supply metrics access for RocksDB and Streams so that sizing decisions rest on your own measurements rather than assumptions.
How do Kafka and Streams upgrades affect state stores?
Upgrades can change how stores and changelogs are read and written, so each upgrade needs a compatibility check before rollout. Kafka's upgrade notes list the supported paths between releases, and Streams applications have their own minimum client requirements. Plan a rolling upgrade in a non-production environment with the same store contents, confirm that restore still completes within your recovery target, and keep a rollback path that does not depend on discarding state you cannot rebuild quickly. Confirm the exact Java requirement and any state-related changes for your target release in the upgrade notes before scheduling the change.
Can we keep Streams state on persistent volumes in Kubernetes?
Yes, many teams do, and persistent volumes help avoid full restores after pod rescheduling, but the choice has trade-offs. The storage class must deliver consistent latency for RocksDB, volumes must survive node drains, and resizing or moving them can be slow or unsupported depending on the provider. Verify the storage class with a load test that resembles compaction and checkpoint activity, and confirm how your scheduler places pods relative to their volumes. Ask the vendor or platform team to confirm volume expansion and snapshot behavior for your environment.
Does DeployOpen handle the whole Streams state layer, and what does a scope cover?
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms in your own infrastructure, including Kafka and the applications that depend on it. A fixed-scope engagement would typically name the applications, the environments, the sizing and restore tests, the dashboards and the runbooks, each with written deliverables and timelines agreed before work starts. Ongoing support and upgrades are available as retainers. Quotes follow from the written scope, so bring your application inventory and target state-size estimates to the first conversation.
Limits to confirm in your environment before committing
A Streams application with local RocksDB state depends on the Kafka release line, the Java runtime, the disks under the state directory and the changelog retention you set. Confirm each of these before you size instances or commit to a design, because a mismatch in any one of them changes restore time, failure behavior and upgrade cost.
Supported Kafka release line
The downloads page lists 4.1.2, 4.2.2 and 4.3.1 as the supported releases, and earlier versions are archived. Pick the broker and client line together, since Streams state handling and client behavior are tied to the release you run. Confirm with your platform team which line your brokers run and whether your Streams applications link against a matching client version before you plan upgrades.
Java runtime for Streams apps
Kafka Streams applications need at least Java 11 on the client side, while brokers, Connect and tools need Java 17 from Kafka 4.0. Your Streams services may run on a different JVM than the brokers, so check the runtime image each application ships with. Also confirm the JVM heap and off-heap headroom, since RocksDB allocates native memory outside the heap.
Storage performance under state directories
Each Streams instance writes RocksDB files to its state directory, and restore and compaction speed depend on the disk under it. Measure sustained write and read throughput on the volumes you plan to use, including the persistent volume class in Kubernetes. Ask your storage team whether the volumes are local or network-attached, because network volumes can slow restores and add latency to every store read.
Changelog retention requirements
Each state store is backed by a changelog topic that restores it after a failure or rebalance. Its retention and cleanup settings decide how far back a restore can read and how much disk the topic uses on brokers. Agree with the data owners how long recovery must cover, and confirm that the changelog settings will not expire records a store still needs.
Version-specific state behavior
Restore behavior, standby handling and the metrics exposed for RocksDB can change between releases. Read the release notes for each Kafka line you plan to run before you rely on a specific restore time or alert threshold. Test a restore of a representative state store on the exact versions you will deploy, and record the results as your baseline.
Rebalance protocol compatibility
Streams applications can use the classic group protocol or the server-side rebalance protocol, and the two differ in how tasks move and how long a store is unavailable during reassignment. Confirm which protocol your brokers and applications support and whether mixed groups are allowed in your environment. Test a rolling restart under each protocol you plan to use, because task migration is when state restores are most likely to slow down.
How DeployOpen scopes and supports Kafka Streams state work
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure, including Kafka Streams applications that keep state in RocksDB-backed stores. Engagements start with a written, fixed scope that covers the stores, changelogs, volumes and metrics your applications depend on.
How we scope
- Inventory every Streams application, its state stores, and whether each store is key-value, windowed or session based.
- Record changelog topic settings, standby replica counts and expected restore behavior after an instance failure.
- Agree in writing which memory, disk, volume and RocksDB metric work is in scope, with deliverables and timelines fixed before work starts.
How we build
- Configure state directories, RocksDB settings and persistent volume claims in the Kubernetes or VM environment you operate.
- Test restore time by stopping instances under representative load and measuring recovery against your agreed targets.
- Wire RocksDB and Streams metrics into your monitoring stack with alert rules tied to disk pressure and restore lag.
How we hand over
- Provide a written runbook covering store layout, changelog retention, state directory cleanup and application reset steps.
- Document sizing assumptions, the tested failure scenarios and the results you can repeat in your own environment.
- Walk your platform and application teams through the configuration so they can operate and change it without DeployOpen.
How we support
- Engagements are fixed-scope, and ongoing support, upgrades and continuous testing are available as a retainer.
- Retainer work can cover Kafka version upgrades that affect Streams state, RocksDB tuning changes and new store additions.
- Confirm with us how support hours, response expectations and escalation paths will be written into your retainer before signing.
