Running Kafka Streams applications in production

Start with consumer group lag on the input topics, the rebalance rate of the application, and the state of its tasks, because these three signals show most failures before users notice them. Add processing latency and the error or skipped-record counts once those baselines are known. Set thresholds from your own traffic pattern in a test environment rather than copying values from another system. Check the metric names and tags against the Kafka Streams documentation for your exact version, since metric groups have changed across releases. Route alerts to the team that owns the application so that a lag spike leads to a named responder.

A stuck task shows up as a partition whose committed offset stops advancing while its end offset keeps growing, and the owning thread's processing rate drops to zero. Compare the task's position with the log end offset for each partition the instance owns. If the task keeps moving between instances, the cause is usually a rebalance loop rather than slow processing, so check the rebalance rate first. Thread state that stays in a restoring or partitions-revoked state for a long period also points to a problem. Capture thread dumps and the application logs at the same moment so you can tell a blocked processor from a slow external call.

Restore time is the interval during which a task replays its changelog topic into the local state store before it can process new records. Read it from the restore metrics for each store, together with the number of records restored and the changelog lag. Long restores usually come from large changelogs, frequent compaction gaps, or state stored on slow disks. Standby replicas shorten restores because a warm copy of the state already exists on another instance, which is why their effect should be measured in your environment. Record a baseline restore time before and after each change so the result is evidence rather than an impression.

The useful signals are write stalls, memtable and block cache hit behavior, compaction pending bytes, and the number of open files, along with the store's own size on disk. Rising stall counts or growing pending compaction bytes indicate that the disk or the memory budget cannot keep up with the write rate. Growing store size with flat record counts can signal that retention or cleanup settings do not match the workload. Expose these metrics through the JMX or metrics reporter you already run, and tie them to the same dashboard as lag so that a disk problem and a lag problem can be read together. Confirm metric names with the version of Kafka Streams you deploy.

The kafka-streams-groups.sh tool describes Streams groups and their members, which makes it useful for checking assignments and lag from the command line during an incident. It helps confirm which instance owns which partitions and whether any member is missing. Run it against the bootstrap servers with the same client configuration your application uses, including the security settings, so that access control does not hide the group. Scripts and output format can differ between Kafka releases, so verify the options with the help output of your installed version before writing runbooks around them. Keep a saved snapshot from a healthy period to compare against during an incident.

Frequent rebalances stop processing across the affected partitions, so a rising rebalance rate is one of the most important signals to track. Compare rebalance counts with deployments, scaling events, and network or pod restarts to find the trigger. Long session timeouts or slow processing inside the poll loop can make a healthy instance look dead, which causes repeated reassignment. If your group uses the server-side rebalance protocol, check the protocol-specific metrics for your Kafka version. Review the timing settings with your platform team, because changes to them alter failure detection as well as rebalance frequency.

Track processing latency per task or per processor node, the rate of records failing deserialization, and the number of records dropped or skipped by the error handlers you configure. Most applications should not skip silently, so confirm which handler your code uses and whether skipped records are logged or sent to a dead letter destination. Latency that grows while lag stays flat often points to a slow downstream dependency, such as a database write inside a processor. Errors that appear after a deployment usually reflect a serialization or schema change. Pair these counters with the application logs so each alert links to a specific record type or processor.

Your own team can run this monitoring if it already operates a metrics stack, can export Kafka and Streams metrics, and has an owner for alert tuning. The work is mostly configuration: exposing metrics, building dashboards, writing alert rules, and documenting runbooks for lag, rebalance, and restore events. A partner engagement is usually scoped in writing as a fixed set of dashboards, alerts, and runbooks for named applications, with the deliverables and timeline agreed before work starts. Ask any provider to list the metrics, thresholds, and evidence they will hand over, and to state who maintains the alerts after handover. Ongoing tuning and testing are available as retainers when the team wants them.

The cost follows the written scope: how many Streams applications are in play, how much state each one keeps, how many environments and failure scenarios must be tested, and how much compliance evidence the work must produce. Each application adds design, configuration, testing and runbook work, and stateful applications add restore and disk sizing work on top. Cluster ownership matters too, because a rollout on a cluster you already run costs less to integrate than one that also needs broker changes. Ask the quote to list each application separately with its state profile and environments.

Duration depends on scope rather than on a fixed template. A single stateless application with an existing cluster is a much smaller piece of work than several stateful applications that need standby replicas, rack-aware assignment, load testing and monitoring built in. Fixed-scope engagements agree timelines in writing before work starts, so the timeline should appear in the statement of work with its dependencies listed, such as access to test clusters, change approvals and the release window for your upgrade path. Ask for milestones per application rather than a single end date.

The work needs named people who know the business logic of each Streams application, people who run the Kafka cluster, and someone who owns the platform the applications run on, whether that is virtual machines or Kubernetes. Your infrastructure must provide persistent volumes for state stores, enough broker capacity for changelog and repartition topics, and a test environment that resembles production. Confirm access to these systems early, along with the Java runtime your organization approves. Kafka Streams applications require at least Java 11, and brokers on 4.x require Java 17, so the runtime version affects what can be deployed.

An internal team can do it alone if it already operates Kafka brokers, has Java developers who have built stateful stream processing, and has time to test failure scenarios such as instance loss, rebalances and state restoration. Teams without that experience usually find the hard parts are operational rather than coding: sizing threads and instances, setting up standby replicas, and deciding how to reset an application safely. A partner can cover part of this, such as design review, test harness work or runbooks, while your team keeps ownership. Decide which parts you want to own before requesting a quote.

One-time costs include design, application configuration, load and failure testing, monitoring setup and the initial runbook. Recurring costs include running the applications, monitoring and alert tuning, Kafka version upgrades, dependency updates, and rerunning tests after changes to topology or traffic. Retainers for ongoing support, upgrades and continuous testing are available from DeployOpen and are priced from the written scope. The quote should separate these categories so you can budget the run phase, and it should state what happens when the Kafka cluster or the Streams library changes version.

A rollout can produce the technical evidence that auditors and assessors typically ask for, such as documented configuration, access control settings, change records, test results and monitoring definitions. DeployOpen prepares systems, controls and evidence for compliance assessments, but it does not issue audit reports, attestations or certifications, and independent auditors make that determination. Ask the quote to name which control areas the evidence covers and in what format. Also confirm which regulations apply to the data flowing through the applications, because that decides how much evidence you need.

A useful written quote lists each Streams application with its input and output topics, state store profile, environments, and integrations. It should state the deliverables, such as configuration, tests, dashboards, alert rules and runbooks, and the acceptance criteria for each. It should name the Kafka and Java versions in scope, the upgrade assumptions, and the client responsibilities. Ask for the exclusions in writing, including anything about the Kafka cluster itself, so you know which work sits outside the agreed scope. Most engagements are fixed-scope, so the quote and the scope document should match.