When Kafka Streams interactive queries replace a separate read store
If a Kafka Streams application already holds the derived state another service needs, the first decision is whether that service should read the application's own state or receive a copy in a separate database. Interactive queries let code outside the topology read the local state stores of a running Streams instance through the state store API that the application exposes via its KafkaStreams object. Only materialized stores can be queried, so each store that serves reads must be named when the topology is built. Key-value, window and session stores each offer their own read methods. The catch is that every instance holds only the partitions it owns, so a lookup for one key succeeds only on the instance that hosts that key's partition. Finding that instance is a separate step, covered later on this page.
Choose interactive queries when the reader needs point lookups by key, the data is derived from topics the application already consumes, and a short staleness window is acceptable because values reflect what the topology has processed so far. The approach also suits teams that own both the stream processing and the read path, since state can be rebuilt from changelog topics and no second write path has to be kept in sync. A separate read store fits better when readers need ad hoc queries or joins across keys, retention longer than the changelog holds, several independent consumers with different access rules, or read load that should not compete with processing threads on the same instances. Confirm that your read volume and latency targets have been measured against the instances that will serve them.
Before committing, ask the owners of both services to confirm a short list: the store names and key types the reader needs, the exact lookup pattern per request, how many instances must stay reachable during a deployment, and how the caller should behave when a lookup returns a stale value or fails during a rebalance. Then prototype one lookup path in a non-production environment and measure how long reads fail or lag while an instance restarts, because that window determines whether the design meets the reader's needs. The remaining sections cover rebalance behavior, the RPC layer between instances and the discovery calls that route requests.
Which store types and query patterns fit your reads
A point lookup asks for one key, such as a customer ID or an order number, and returns its current value from the store. This is the pattern interactive queries handle best. A key-value store answers it directly, and a window store can answer it for one key within a time interval. Because each key's value lives on one partition, the request reaches one instance, provided the caller first determines which instance owns the key.
- Store types: Key-value stores, usually backed by RocksDB, answer
getcalls against local state. - Latency: A local read is fast, but a remote hop through your RPC layer adds a network round trip and serialization cost.
- Freshness: The value reflects records the instance has processed and written to the store, so it can trail the input topic by the processing lag at the moment of the read.
- Load: Each lookup is small, so high request rates are usually limited by the RPC layer and thread pool rather than the store itself.
- Metadata calls: Routing depends on a metadata lookup for each key, which must be refreshed after a rebalance moves partitions.
Check that your key serializer matches the one used in the topology, and measure end-to-end latency including the remote hop.
Range reads return entries between a lower and an upper bound, and scans iterate over a key prefix or a time window. Keys are spread across partitions, so a range read against one instance returns only the slice that instance holds. Window and session stores support time-bounded fetches, which suit event history such as activity for one customer over a period.
- Store types: Key-value stores support ranges over local keys, window stores support fetches by key and time interval, and session stores support lookups of sessions for a key.
- Partial results: A range query on one instance covers only that instance's keys, so the caller must query every instance and merge the results.
- Iterator handling: Iterators hold resources on the store and must be closed, since leaked iterators degrade the instance over time.
- Latency: Scan time grows with the number of entries in the range, so bound the range and return results in pages.
- Freshness: A scan reads a view that can lag changelog writes, and updates arriving during a long scan may or may not appear. Confirm the behavior for your version.
Decide early whether your query contract needs complete results, paged results or time-bounded results, because that choice determines the store type and how many instances you must reach.
Whole-store reads return or count everything in a store, for example to populate a dashboard or report an approximate entry count for monitoring. This is the most expensive pattern. Each instance holds a subset of the partitions, so a full read means every instance scans its local state, and a caller or aggregating service merges the results.
- Load: A full scan reads every local entry on every instance, competing with stream processing for CPU, memory and disk I/O.
- Memory: Collecting all results into one response can exhaust heap on the serving instance, so stream results where your framework allows it.
- Approximate counts: Count methods on stores return estimates, which suit monitoring and are unsuitable for reconciliation.
- Consistency: Each instance returns its own point in time, so the merged result is not one consistent snapshot across the application.
- Alternatives: If product views need full reads often, a materialized copy in an external database may serve better. Confirm this design with your data team.
Before exposing a whole-store read, confirm who may call it, how often, and whether a separate analytics store would meet the need with less risk to the stream processor.
How to set up queryable state and instance discovery
This sequence turns a Kafka Streams application's local state stores into a read path that any instance can answer for, by making each store queryable, advertising each instance's endpoint, routing requests to the owning instance and proving the result on a two-instance test. Confirm the Java 11 minimum for Streams applications against the Kafka upgrade notes before you start, since that is the floor for every instance in the sequence.
Confirm the Java runtime and application baseline
The engineering team checks that every Streams instance runs Java 11 or later, which is the minimum for Kafka Streams applications according to the Kafka 4.3 upgrade notes, and that the Kafka Streams dependency matches the brokers' supported release line. Brokers and tools have a higher requirement (Java 17 since Kafka 4.0), so confirm the broker runtime separately. You know this step is done when the application builds and starts in a test environment on the agreed JVM and the team has written down the dependency version it will use for every instance.
Enable queryable stores in the topology
Developers name each state store that should serve reads and create it with an explicit store name, using a materialized store supplier for the aggregation, table or key-value operation. Stores without a stable name cannot be looked up later, so the team records the store names and key and value serdes in the code review. The step is complete when a unit test builds the topology, starts the application against a test broker and can list the store through the Streams instance's local state query API.
Set application.server on each instance
Each instance sets the application.server configuration to a host and port that other instances can reach, and the value must differ per instance. Platform engineers usually inject it from the pod IP, the host name or a deployment variable, so it reflects the address the RPC layer actually listens on. The check is that each instance logs or exposes its own endpoint at startup and that the value appears in the group metadata when the application joins the consumer group.
Expose the RPC endpoint between instances
Developers add a small HTTP or gRPC service inside the application that accepts a read request for a store and key, and that either answers from the local store or forwards the request to the instance that owns the key. Network policy and TLS for this endpoint are set by the platform team, because the endpoint serves application data and is not protected by Kafka's broker security. The step is done when a request sent to one instance for a key owned by another instance reaches the right store, and an unauthenticated request from outside the service network is refused.
Implement key discovery with metadata lookup
The application calls the Streams metadata API to find which instance and partition hold a given key, using the store name and the key serde so the partitioner matches the one the topology uses. If the answer is the local instance, it reads locally, and if not, it forwards to the endpoint returned by the metadata. Teams should handle keys that are not yet present or that are moving during a rebalance by returning an explicit status instead of an empty result. The check is that the lookup returns the same instance from every entry point for the same key.
Handle rebalances during reads
While a rebalance is running, the metadata for a store may be incomplete or point to an instance that has just lost a partition, so the read path needs a defined response. Developers typically check the Streams application state before answering, return a retryable status when the store is not running, and bound the wait on the client side. Operators should test this by restarting one instance during a read load and confirming that the service returns retryable responses rather than stale or wrong answers. The step is complete when that test passes and the team has agreed the retry policy for callers.
Run a two-instance read checkpoint
Engineers start two instances of the application against a shared test cluster, write a known set of keys through the input topic, and send reads to both instances for every key. The checkpoint passes when each read returns the value the topology should have produced, regardless of which instance received the request, and when the owning instance is the one reported by discovery. Keep this test in the pipeline so that changes to the topology, the serdes or the endpoint configuration fail the build before they reach production.
How rebalances change what a query returns
An interactive query reads local state that is only as current as the task that owns it. Rebalances move partitions and state between instances, so the same key can return a stale value, a partial result, no result or an error depending on timing. The four cards below cover where those failures come from and what to build around them.
Stale metadata after a partition moves
Instances cache which peer holds which key. After a rebalance reassigns a partition, a cached answer can point at the former owner, which may no longer host that store. Check how your code refreshes metadata after a rebalance listener fires, and confirm that a lookup which finds no local store falls back to a fresh metadata request instead of returning an empty result. Test this by forcing a reassignment during a load test and watching which instance answers.
Reads during store restoration
A task that has just gained a partition must replay its changelog topic before its store reflects recent writes. A query routed to that instance during restoration can return older values or a not-yet-present key. Decide whether your service should return a clear unavailable response, serve from a standby replica, or accept lag for that key class. Confirm how your application reports the restoration state so callers can tell an empty answer from an unrestored one.
Missing keys while a task migrates
During a task migration, the partition has left one instance and not yet finished arriving at another, so a key can be absent from both views for a short period. A lookup that treats a miss as a definitive not-found will report false negatives. Design the handler to distinguish a key that does not exist from a key whose owner is currently in transition, and retry the owner lookup before answering. Standby replicas reduce the window, and you should confirm the setting in your environment.
Client retries during rebalances
A caller that retries a failed query may hit a different instance on the next attempt, and a retry during a rebalance can return data from a newer or older state than the first response. Retries also multiply load on the instance that is still restoring. Set a bounded retry count with backoff, include the instance or store version in responses where your API allows it, and make the caller treat repeated errors during a rebalance as a signal to wait rather than to hammer the endpoint. Confirm the retry policy in every client library your teams use.
Configuration options and when each fits
This table lists the settings that decide how interactive queries route and serve reads across Streams instances, with the situations where each fits and the trade-off it brings. Read each row as a decision: the setting you choose, when it makes sense, and what you give up. Confirm exact property names, defaults and behavior against the Kafka Streams documentation for your release before you build.
| Setting or choice | When it fits | Trade-off to weigh |
|---|---|---|
| application.server | Every instance that serves queries needs a reachable host and port so peers can forward requests for keys held elsewhere. Set it to an address the other instances can resolve, not a loopback or container-local name. | Misconfigured or unreachable addresses fail only at query time, so test cross-instance forwarding in staging. Behind NAT, load balancers or pod networks, the advertised address may differ from the bind address. |
| Local state store access only | Use when the application can answer from the instance that owns the partition, or when the caller already routes requests by key to the right instance. | Each instance sees only its own partitions, so a query for a key on another instance returns nothing until your code routes it. Callers need a routing layer or a retry path. |
| Remote lookup through the RPC layer | Use when clients send requests to any instance and expect a correct answer for any key. | Every query may take an extra network hop, and the forwarding layer becomes part of your latency and failure surface. Plan timeouts and error handling for the hop. |
| Standby replicas | Use when a restarted or rebalanced instance must rebuild a large store before queries for its partitions can succeed without a long gap. | Standbys consume extra memory, disk and network to keep copies of state up to date. They do not serve queries on their own unless your design reads them explicitly. |
| State store type and changelog settings | Choose persistent stores with changelog topics when state must survive restarts; choose in-memory stores only when a rebuild from the changelog is acceptable. | Persistent stores add disk I/O and sizing work. Changelog topics add broker storage and replication load and affect restore time after a failure. |
| RPC framework choice | Pick an RPC framework your team already operates, such as an HTTP server or a gRPC stack, and that supports the serialization and timeouts you need. Keep the interface small: one endpoint per query type is easier to test. | The framework adds dependencies, versioning work and its own security controls. Serialization of keys and values across instances must match the store types you expose. |
| Metadata refresh approach | Use the metadata lookup on each request for accuracy during frequent rebalances, or cache results briefly when partition ownership changes rarely. | Per-request lookups add load and latency. Cached metadata can point to a previous owner during a rebalance, so the caller must handle a not-found or retry response. |
| Rebalance handling in the query layer | Return a clear retriable response while an instance is restoring or the group is rebalancing, and check the instance state before answering. | Callers see more retries and temporary errors during deployments. Without this, a query can return an incomplete answer that looks valid. |
Securing the RPC layer between Streams instances
Interactive queries open a second network path next to the Kafka brokers: instances call each other over an RPC endpoint, and clients call that endpoint to read state. The cards below separate the controls that protect the underlying topics from the controls that must be built for the RPC layer itself, since Kafka's own security does not cover it.
Transport encryption on the broker path
Kafka encrypts traffic between clients, brokers and tools with SSL/TLS, and security is optional, so a cluster can run unsecured or mix secured and unsecured listeners. Streams instances reading changelog and source topics must use a secured listener, and the Streams configuration must match it. Check that no plaintext listener is reachable from the subnet where Streams instances run, because an unsecured listener leaves the state data in transit exposed.
SASL mechanisms for Streams clients
Kafka authenticates clients, brokers and tools with SSL or SASL. The documented SASL mechanisms are GSSAPI (Kerberos), PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER. Pick one mechanism per Streams application and give each application its own identity, so that access to internal and changelog topics can be traced to the service that owns the store. Confirm with your identity team how credentials are rotated for long-running Streams services.
Mutual TLS between instances
The RPC endpoint that serves local store reads is application code, so Kafka's client security does not protect it. Configure the application server to require TLS on every instance-to-instance call and, where your policy requires it, mutual TLS so each instance proves its identity with a certificate. Plan how certificates are issued, renewed and revoked during rolling restarts, since an expired certificate on one instance breaks metadata lookups for the keys it owns.
Client-facing authentication on the query endpoint
Callers that query state through the endpoint need their own authentication, separate from the broker credentials. Many teams place the endpoint behind an API gateway or an identity-aware proxy that validates tokens before a request reaches a Streams instance. Decide whether the query path accepts the same tokens as the rest of your services, and confirm that the proxy forwards only the requests the instance is meant to serve.
Pluggable authorization for store access
In KRaft clusters the StandardAuthorizer stores ACLs in the cluster metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Kafka authorization is pluggable and supports integration with external authorization services. Those ACLs cover topics, not your query endpoint, so define a separate policy that maps callers to the stores they may read, and document which team approves each change.
Per-store read permissions
Not every caller should read every store. A store may hold customer balances or session data, while another holds public counters. Write the permission model as a table of caller, store and allowed operation, and enforce it in the endpoint code before the store lookup runs. Test denied requests as well as allowed ones, and confirm that a denied key lookup does not reveal the partition or instance that holds it.
Questions buyers ask about Kafka Streams interactive queries
What drives the cost of building Kafka Streams interactive queries?
The cost depends mostly on how many stores you expose, how many instances must answer queries, how the lookup layer is secured, and how much of the surrounding service you already run. Apache Kafka and Kafka Streams are open source under the Apache License 2.0, so there is no license fee from the project itself. The paid work is engineering time: designing the key layout, building the metadata discovery and RPC layer between instances, testing behavior during rebalances, and documenting the result. Ask any partner for a fixed-scope quote that lists each store, each query path, the authentication model and the test cases for rebalances. Infrastructure for the instances and the Kafka cluster is a separate cost that recurs.
How long does it take to build interactive queries on Kafka Streams?
Duration follows the number of stores, query types and the depth of failure testing. A single store behind one lookup endpoint is a much smaller job than several stores with cross-instance routing, authentication and rebalance handling. No researched timeline is supplied for this work, so the figure should come from a written scope rather than a general estimate. Ask the partner to break the timeline into design of the state layout, implementation of discovery and the RPC layer, load and rebalance testing, and handover. Confirm which items are included and which depend on your team's review cycles.
What does the work need from our team?
Your team must provide the Kafka Streams topology and state store definitions, access to a non-production cluster, the instance deployment model (for example, containers or virtual machines), and a decision on how instances reach each other inside your network. Someone who owns the identity provider or certificate authority must be available for the RPC layer's authentication. You also need a named owner for the data model, because key design decides which queries are possible. Plan for time from your platform team to review network policies and to run the rebalance tests in an environment that resembles production.
Can an internal team build interactive queries without outside help?
Yes, if the team already runs Kafka Streams in production and has engineers who understand state stores, partition assignment and internal topics. The hard parts are discovering which instance holds a key during a rebalance, returning consistent answers while state is restoring, and securing the instance-to-instance call. Teams new to these topics often underestimate the test effort for rebalances and failures. A useful check is whether your engineers can explain what a query returns while a standby replica is still restoring. If not, a short design review with an outside partner can reduce the risk before build starts.
How does interactive query work fit with a Kafka Streams service in production?
Interactive queries run inside the same Kafka Streams application that processes the data, so they share its sizing, deployment and upgrade cycle. The service in production must already handle instance count, threads, standby replicas and state store restoration, because the query layer depends on all of them. Rebalances move partitions between instances, so a query can land on an instance that does not hold the key. Plan the production runbook together with the query design, and confirm monitoring covers restoration and lookup errors. The operating guidance for the stream processing service should be treated as a prerequisite, not a separate task.
What ongoing work follows after the interactive queries go live?
Ongoing work includes monitoring lookup latency and errors, reviewing rebalance behavior after each deployment, rotating certificates or tokens used by the RPC layer, and testing upgrades of Kafka and the Kafka Streams library. The Kafka upgrade notes say rolling upgrades are supported from each release in the past year to the latest version, so plan an upgrade test cycle for the query layer each time the cluster moves. Kafka's release plan aims for three releases a year. Ask a partner whether retainer support, upgrade work and continuous testing are included, and what the handover covers. Support and upgrades are available as retainers.
What security questions should we settle before building the RPC layer?
Decide how instances authenticate each other, who may call the query endpoint, and whether query results contain sensitive data that needs encryption in transit. Kafka supports SSL and SASL for clients, brokers and tools, but the RPC layer between your Streams instances is your own code, so its authentication is your design decision. Confirm which identity and certificate tools your organization already uses, and who approves exceptions. Treat the query endpoint as an internal API with the same review, logging and access control expected for other services that expose customer or financial data.
Does DeployOpen handle the whole Kafka Streams interactive query build?
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure, so it can scope the interactive query layer alongside the Kafka cluster and the stream processing service. Most engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts. DeployOpen does not publish a price list, so a quote follows from the written scope. Ongoing support and testing are available as retainers. DeployOpen does not issue audit reports or certifications for any compliance framework.
What a scoped interactive queries engagement covers
The written scope comes first. Before any build starts, DeployOpen and the client agree which Streams applications expose state, which stores are queryable (for example, key-value and windowed stores), which query shapes the service must answer, and which instances will host the RPC layer. The scope also states the expected behavior during rebalances, whether reads may return stale results while a standby takes over, and how far the work reaches into the surrounding Streams application. Most engagements are fixed-scope, so these decisions are recorded in the agreement and any change to them goes through a written change to scope. Ask for the list of stores and query endpoints in the quote, since that list is what drives the effort.
The build happens in your environment: your Kubernetes clusters, virtual machines or data centers, with your identity provider, TLS certificates and secrets store. DeployOpen does not run the Kafka cluster or the query service on its own infrastructure. Deliverables typically include the query and discovery code, the internal RPC configuration between instances, integration tests that exercise KafkaStreams.queryMetadataForKey style lookups and remote fetches during a forced rebalance, load tests against agreed read latencies, and runbooks covering instance replacement, application.server configuration, and the steps to diagnose a failed remote lookup. Confirm with your team which of these artifacts your security and operations reviewers expect in their own format.
When the engagement closes, the client owns the code, configuration and documentation. Ongoing support, upgrades of the Streams and Kafka client libraries, and periodic re-testing after rebalance-related changes are available as a retainer, which is priced separately from the fixed-scope build. Ask any partner to state in writing what the retainer covers, how quickly they respond to incidents, and how they handle library upgrades that change state store behavior.
How DeployOpen scopes and supports Kafka Streams query services
DeployOpen plans, builds, documents and supports Kafka Streams interactive query services inside your own infrastructure, on a self-hosted Apache Kafka cluster you control. Engagements are fixed-scope, so the query surface, state store design and instance-to-instance RPC are agreed in writing before build work starts.
How we scope
- Inventory which state stores need external reads, and the key types, value shapes and latency expectations for each.
- Decide which query endpoints, metadata lookups and error responses the written scope covers, and which are excluded.
- Confirm the Kafka Streams version, Java runtime and client authentication model that the query services will run on.
How we build
- Implement local store reads, key-to-instance discovery through the Streams metadata API, and a documented RPC layer between instances.
- Build handling for rebalances and restoring stores during queries, including retry and not-yet-ready responses.
- Test query behavior during rolling restarts, instance failure and partition reassignment in a non-production environment first.
How we hand over
- Deliver source code, configuration, deployment manifests and an architecture note describing how each instance answers a request.
- Document the runbook for adding instances, verifying store readiness and diagnosing stale or missing reads.
- Walk your engineers through the code and operations material so your team can maintain the service after handover.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer, covering Kafka Streams and dependency updates.
- Retest query endpoints after Kafka and Streams upgrades, since rebalance and metadata behavior can change between releases.
- Ask any support provider to confirm response times, escalation paths and the scope of testing in writing before you sign.
