Which Apache Kafka metrics to monitor and alert on first
A self-hosted KRaft cluster can be running fine and still have no alerts you would trust during an incident. A small set of broker, controller and consumer metrics catches most of the failures that hurt production: partitions that lose replicas, partitions with no leader, a controller quorum that has lost its footing, and consumers that fall further behind. The rest of the metrics Kafka exposes are for diagnosis once an alert has fired. Apache Kafka is licensed under the Apache License 2.0 and has no paid edition from the project, so the monitoring stack is yours to build from the JMX metrics each broker and controller exposes, using the collection and alerting tools your team already runs or chooses to adopt. Nobody else will set the thresholds for you, which is why the choice of metrics and the first thresholds belong to your platform team.
Settle the list before three moments: before the cluster takes production traffic, after the first incident where someone learned about a problem from an application team rather than from a dashboard, and before any Kafka upgrade. Before go-live, confirm that every alert in the first set has a named owner, a runbook and a test that proves it fires. After an incident, check whether the alert arrived late, arrived without context, or never arrived, and adjust. Before an upgrade, review the metric names against the release notes for the version you are moving to, because JMX names and dashboards can change between releases and a silent metric gap looks like a healthy cluster. The Kafka downloads page lists the supported releases, which is the right place to confirm the version you are running is still receiving fixes before you tune alerts against it.
Six metric groups that catch real Kafka problems
Most Kafka incidents show up first in a small set of signals: partition health, controller state, replica sync, request latency, KRaft quorum health and consumer lag. Each group points to a different failure, so alert on all six and confirm exact metric names against the Kafka release you run.
Under-replicated and offline partitions
Under-replicated partitions count partitions where one or more replicas have fallen behind the leader. Offline partitions have no leader at all, so producers and consumers cannot use them. Treat any offline partition as an immediate page and any sustained under-replicated count as a broker, disk or network problem to trace. Check which brokers own the affected partitions before restarting anything.
Active controller count
In a KRaft cluster exactly one active controller should exist at any time. A count of zero means no leader is elected for metadata changes, and a count above one during steady state suggests a split or misconfigured controller set. Watch this metric alongside controller election events in logs. Confirm the exact metric name and expected value for your release before writing the alert rule.
ISR shrinks and expands
The in-sync replica set changes when a follower falls behind or catches up. Frequent shrink events followed by expansions usually mean a slow disk, a network path with packet loss or a broker under heavy garbage collection. A single shrink during a planned restart is expected. Repeated shrinks on the same broker are the early warning that the next under-replicated alert will follow.
Request latency
Request latency metrics show how long produce, fetch and metadata requests spend in each stage, from queueing to local processing to remote waits. Rising queue time usually means request handler threads are saturated, while rising remote time points to replication or follower delays. Baseline these values during normal load so alerts reflect a change from your own traffic pattern, not a generic number.
KRaft quorum health
KRaft controllers replicate metadata through a Raft quorum. Quorum metrics show the current leader, the number of voters and how far each voter lags behind the leader. A voter that falls behind can reduce the cluster's tolerance for further failures before any partition shows a problem. Pair these metrics with the metadata quorum status output when you investigate a lagging controller.
Consumer lag growth
Consumer lag is the distance between the latest offset written to a partition and the offset a consumer group has committed. Lag that exists is often normal during bursts, so alert on lag that keeps growing across several evaluation windows for a group that should be keeping up. Track it per group and per topic so a stalled consumer does not hide inside a healthy total.
Starting alert thresholds and how to tune them
These alerts fire when the cluster is already losing availability or is close to losing data, so the on-call engineer should be paged immediately. The conditions are qualitative: any partition with no leader is serving no traffic, and any change in the number of active controllers away from exactly one points to a control-plane problem that can stall every topic. Because a healthy cluster should show none of these conditions, a single occurrence is enough to page, and the threshold is zero or a state change rather than a tuned value.
- Offline partitions above zero: Producers and consumers for those partitions fail until a leader is elected, so any nonzero count is urgent.
- Active controller count not equal to one: Zero means no controller can change metadata. More than one, reported over a sustained period, suggests a split or a misconfigured quorum.
- KRaft quorum loss or lost voters: A controller majority that cannot be reached blocks metadata changes, so page when the quorum state shows a missing majority.
- Sudden drop in healthy brokers: A broker leaving the cluster unexpectedly should page, especially when the cluster runs at its minimum replica count.
Before relying on these alerts, confirm the exact metric names and default behavior for your release in the Kafka 4.3 documentation, and test each one by stopping a broker in a non-production cluster.
These alerts show a problem that is not yet an outage but will become one if ignored, so they should open a ticket that someone works the same business day. Under-replicated partitions and ISR shrinks are the main examples. A brief spike is normal during a broker restart or a partition move, so the team should alert on a count that persists across a duration window, not on any single sample. Set that window from the cluster's own history of restarts and reassignments, and keep it longer than the normal duration of planned maintenance.
- Under-replicated partitions held over a window: A steady nonzero count means at least one follower is not keeping up, which reduces the margin before a leader failure becomes data risk.
- ISR shrink events: Repeated shrinks on the same brokers or topics point to slow disks, network trouble or overloaded followers.
- Request latency above your baseline: Rising produce or fetch latency at the 99th percentile often precedes a broker falling behind, so compare against a measured baseline rather than a published figure.
- Request handler or network idle capacity dropping: Sustained low idle time means brokers are close to saturation.
- Consumer lag growing on critical groups: Lag that exists during a catch-up is expected; lag that keeps climbing needs investigation.
For each alert, record the baseline and the window you chose, and confirm the metric definitions against the Kafka 4.3 documentation before setting thresholds.
Trend alerts catch slow changes that no single sample reveals, such as disks filling, partition counts drifting upward or consumers gradually falling behind. These signals rarely require a page, but a team that ignores them tends to meet the same failure during a busy period. Review them on a schedule, and set alerts on the rate of change or on the projected time to a limit, because a fixed absolute value tells you less than the slope.
- Log directory growth rate: A broker's disk usage climbing faster than retention predicts usually means a topic is receiving more data than planned.
- Partition count per broker: Gradual growth in partitions per broker raises recovery time and memory use, so track it against your sizing plan.
- Consumer lag trend: Compare lag over days, not minutes, to separate a consumer that is slowly falling behind from one that is simply catching up.
- Leadership imbalance: Leaders concentrated on a few brokers increase load on those brokers, which shows up as rising latency over time.
- Traffic against capacity: Bytes in and out per broker trending toward the level your hardware was sized for.
Review these trends at the cadence your capacity planning uses, and confirm the appropriate limits with your own sizing exercise.
How the metrics relate during a broker or controller failure
No single Kafka metric tells you what kind of failure you are facing, so the section of your alerting that matters most is how the signals line up. When a broker drops out, the first visible change is usually an ISR shrink: the partitions it led or followed lose a replica from their in-sync set, and the UnderReplicatedPartitions count rises on the brokers that still hold copies. If the lost broker led partitions, leadership moves to surviving replicas, and a short spike in leader changes appears alongside the shrink. A rise in under-replicated partitions by itself could also mean a slow disk, a network partition or a planned restart, so the reader should confirm which broker is missing before treating it as an outage.
A controller problem looks different. In a KRaft cluster, ActiveControllerCount should settle at exactly one across the controller quorum, and a change in which node holds the active role shows up as a brief period with no active controller followed by a new one. Quorum state metrics from the KRaft controllers tell you whether the remaining voters still form a majority. If the active controller count is wrong while broker-side partition metrics look normal, the problem is in the metadata layer, and you should check the quorum before touching partitions. Broker failures and controller changes often happen together, so an alert that fires on one should be reviewed against the other.
Slow followers are the hardest case because they rarely fail outright. A follower on a saturated disk or a congested link falls behind, drops out of the ISR, rejoins, and drops out again. This shows up as repeated ISR shrink and expand events on the same partitions, rising produce and fetch request latency on the affected leaders, and sometimes a lag that the consumer side sees before the broker side does. Frequent churn without a sustained under-replicated count is a signal to investigate, not a reason to page. Set alerts so that a shrink event alone creates a ticket, while a shrink combined with rising request latency or a non-zero under-replicated count creates a page. Confirm your thresholds against the metric names in your Kafka version and against the troubleshooting guidance for under-replicated partitions.
How to set up Kafka metric collection and alert rules
This sequence takes a Kafka cluster from an unmonitored state to a set of alert rules that have each been seen to fire under a controlled failure. The output is a verified collection path, a recorded baseline and alerts that route to people who can act on them.
Confirm the Kafka release and its support window
The platform team checks which Kafka release the cluster runs and whether it appears on the project's downloads page as supported, since bugfix releases are made only for supported lines. Metric names and JMX beans can change between releases, so alert rules must be written against the exact version in use. Record the version for brokers and controllers separately, because a rolling upgrade can leave them briefly mixed. The step is done when the team has a written version record and has confirmed it against the upgrade notes for that release.
Expose JMX metrics on controllers and brokers
The platform team enables JMX on every controller and broker, then confirms the metric endpoint responds on each node before any collector is pointed at it. Restrict the JMX port to the monitoring network, and if authentication is enabled, create a read-only monitoring identity rather than reusing an administrative one. Controllers must be included, because active controller count and KRaft quorum metrics live there and are invisible from brokers alone. The step is done when a manual read of a sample bean returns a value from each node.
Deploy the collection path
The platform team deploys a JMX exporter or equivalent agent beside each node, plus a scraper such as Prometheus, and a separate exporter for consumer group lag if the team wants it at the group level. Each target should carry labels for cluster, role and node so that a single alert can be scoped correctly. Confirm the scraper sees every controller and broker as an up target and that no series are dropped by relabeling. The step is done when every node appears in the scraper's target list with a recent scrape time.
Record a baseline under normal load
The platform team watches the chosen metrics over a representative period of normal production traffic and records typical values for under-replicated partitions, request latency percentiles, ISR shrinks, quorum lag and consumer lag per group. The baseline shows which values are routine for this cluster and which are genuinely unusual, so thresholds are not copied from a generic guide. Ask the operators who run the cluster which traffic peaks are expected, such as batch jobs or month-end loads, and note them. The step is done when the baseline is saved with a date, the cluster version and the load conditions observed.
Write alert rules with duration windows
The platform team writes one rule per signal, starting from the baseline and not from a fixed number. Each rule should include a duration window so that a brief spike during a rolling restart does not page anyone, while a sustained condition such as offline partitions or a missing active controller does. Use severity to separate conditions that need immediate action from those that need review in working hours. The step is done when each rule has a documented condition, window, severity and the reason it exists, and the rules load without errors in the scraper.
Route alerts to the people who act on them
The platform team connects alert severities to the on-call channel, the ticket queue or the owning team, and confirms that each receiver is reachable by someone with the access needed to respond. Include the runbook link or the name of the troubleshooting page in each alert so the responder knows the next step. Set repeat intervals so a persistent condition does not flood the channel. The step is done when a test notification reaches each receiver and the team has agreed who owns each severity level.
Test each alert with a controlled failure
In a non-production cluster that matches production settings as closely as the team can manage, the platform team creates a controlled condition for each alert, such as stopping one broker to trigger under-replicated partitions or slowing a follower to trigger ISR shrinks, then confirms the alert fires within its window and clears after recovery. Record the test, the observed time to fire and any rule that did not behave as written. Do not run these failure tests against production without a change record and a rollback plan. The step is done when every alert has a recorded test result and any failed rule has been corrected and retested.
Securing the metrics path and avoiding blind spots
Metrics endpoints, JMX and the scraping identity are part of the Kafka attack surface, and a monitoring stack that is unreachable or silently stale hides the very problems the alerts exist to catch. The cards below cover how to authenticate and restrict the metrics path, what should stay out of labels and logs, and how to know when monitoring itself has stopped working.
Authenticate the scrape path
Kafka clients, brokers and tools authenticate with SSL or SASL, and the same mechanisms apply to the identities your monitoring uses to read cluster state. Decide which listener or JMX port the exporter uses, whether it requires TLS, and which SASL mechanism its service account authenticates with. Confirm each option against the version you run, because JMX access and exporter setup differ between deployments. Leaving a JMX port open without authentication exposes internal cluster detail to anyone on the network.
Grant the monitoring identity only what it reads
In KRaft clusters, ACLs are stored in the cluster metadata log and enforced by the StandardAuthorizer. Resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true, so a scraper can be blocked by default and a broad grant is not needed to make it work. Create a dedicated principal, grant the describe and read operations its collectors actually call, and review those grants when the exporter changes. Avoid giving the monitoring account super user rights.
Keep sensitive data out of labels
Metric labels such as topic names, consumer group IDs and client IDs are useful for alerting, but they can reveal business or customer context when the naming scheme is loose. Review topic and group naming conventions before exporting per-topic series, and drop or hash labels that embed identifiers. Check that exporter and broker logs do not echo connection strings, SASL credentials or keystore paths during startup or errors. Ask the monitoring vendor or team where stored label values end up and who can query them.
Restrict who reads the dashboards
Dashboards and alert rules expose cluster topology, partition counts and lag for every team that can view them. Apply role-based access in Grafana or your alerting tool so that only the platform team can see broker-level detail and any per-tenant consumer groups. Store alert notification credentials in a secrets manager rather than in dashboard definitions, and review who can silence alerts, since silencing is itself a blind spot.
Alert when the monitoring goes quiet
An alert on under-replicated partitions is worthless if the exporter stopped scraping three hours earlier. Add a rule that fires when a broker or controller target disappears, when the scrape timestamp for a Kafka series goes stale, and when the exporter itself fails its health check. Test this by stopping the exporter in a non-production cluster and confirming that a page reaches the on-call channel. Treat a missing series as a condition to alert on, not as zero.
Review the metrics path after changes
Certificate rotation, SASL credential changes, a Kafka upgrade or a new exporter version can each break scraping without any Kafka error. Add the metrics path to your change checklist: confirm the scraper reconnects after a certificate rotation, confirm the ACLs still match the metrics the dashboards use, and confirm the dashboards still render the expected series after upgrade. Record the result so that the next on-call engineer can see when the path last worked.
Questions teams ask before setting up Kafka alerts
How much effort does it take to build Kafka alerting in-house?
Effort depends mostly on how many brokers and controllers you run, how many consumer groups need lag alerts, and whether metrics already flow into a monitoring stack. A small cluster with one team and existing dashboards needs far less work than a shared platform serving dozens of teams. Your engineers will spend time on three things: exposing JMX metrics from brokers and controllers, writing alert rules with thresholds that match your traffic, and testing that each alert fires and routes to someone who can act. Ask your team to estimate these three items separately, because the first is often done already and the third is the one teams skip.
What skills and access does the team need to set up these alerts?
You need someone who understands Kafka replication, since the key signals (under-replicated partitions, ISR shrinks, active controller count) only make sense with that background. You also need someone comfortable with your metrics pipeline, including the JMX exporter or equivalent, Prometheus rule files and Grafana dashboards. Access requirements are modest but real: read access to broker and controller metrics endpoints, permission to deploy exporters, and a way to query consumer group offsets. Plan for on-call engineers to review thresholds after the first few weeks, because a threshold that is quiet in staging can be noisy in production.
Do we need a partner to set up Kafka monitoring, or can we do it ourselves?
Many teams do it themselves, especially when they already operate Prometheus and Grafana and have Kafka experience in-house. A partner is useful when the team lacks replication expertise, when the platform must be reviewed before a compliance assessment, or when the work competes with a release schedule. DeployOpen plans, implements and documents self-hosted platforms in your own infrastructure, and most engagements are fixed-scope with scope and deliverables agreed in writing before work starts. Whichever path you choose, the monitoring design should be written down so the next engineer can operate it.
How long does it take to get useful Kafka alerts running?
The honest answer is that duration depends on the same drivers as effort: cluster size, the number of consumer groups, the state of your metrics pipeline and the team's Kafka experience. The first useful step is a short list of critical alerts covering offline partitions, under-replicated partitions and the active controller count, because these catch the most damaging failures. Consumer lag alerts usually take longer, since each group has its own normal pattern. Ask any provider or your own team to break the timeline into metrics collection, first alerts, threshold tuning and routing tests, and to state which steps depend on access approvals.
What drives the cost of Kafka monitoring up or down?
Four factors shape most of the cost. Cluster size affects how many metric endpoints you scrape and how many thresholds you must justify. The number of consumer groups multiplies lag alerting work, because each group needs an expected processing rate. An existing Prometheus and Grafana setup removes a large share of the platform work, while building it from scratch adds it. Team skills decide whether the work is done in days of focused effort or spread across months of learning. One-time costs include design, exporter setup, dashboards and threshold tuning. Recurring costs include reviewing alerts, updating rules after upgrades and keeping on-call runbooks current.
Can we monitor Kafka without a commercial console or vendor tool?
Yes. Apache Kafka is licensed under the Apache License 2.0 and the project itself offers no paid edition. Its JMX metrics can be exported and collected with open-source tools such as Prometheus and Grafana, and consumer lag can be read with the Kafka command-line tools and client metrics. The trade-off is that you assemble and maintain the pieces yourself. Confirm the exporter approach that fits your deployment with the documentation for your Kafka release, and check which metric names your version exposes before writing alert rules, since names can change between releases.
What should we ask for in a written scope for Kafka monitoring?
A written scope should name the Kafka version and deployment model, the number of brokers and controllers, the consumer groups to cover and the monitoring stack in use. It should list each alert with its trigger condition, severity, owner and notification route, and state which dashboards will be delivered. Ask for a runbook per alert, a test plan that shows each alert firing in a controlled way, and a handover session. Confirm what happens after delivery, including who tunes thresholds after upgrades and whether ongoing support is included or offered separately. Fixed-scope agreements make these points easier to compare across quotes.
Limits and what to confirm in your environment
The metric names, alert thresholds and agent setup in this guide depend on the Kafka release, the runtime and how the cluster is deployed. The cards below list what to confirm with the project, your vendor or your own test environment before alerts go live.
Metric names for your release
Metric names, MBean paths and availability change between Kafka releases, so the list in this guide is a starting point, not a fixed catalog. Check each metric you plan to alert on against the release you run, and confirm which controller-side metrics are exposed on brokers and which only appear on controllers. Record the names in your alert rules and dashboards so a later upgrade does not silently break an alert.
Supported versions and patch levels
The downloads page lists which releases receive bugfix updates, and only supported releases get them. Confirm that the version you deploy is on that list and that you have a plan for applying bugfix releases, since a metric fix or alerting correction may only ship in a newer patch. Ask your platform vendor or internal owner which release line they commit to patching.
Upgrade path and release cadence
The project plans three releases a year and supports rolling upgrades from releases within the past year to the latest version. This affects when a metric you rely on may be renamed or removed. Before you schedule an upgrade, confirm the supported upgrade path from your current release, retest your alert rules on the target release in a non-production cluster, and decide how far behind the latest release you intend to stay.
Java runtime for the metrics agent
If you export JMX metrics with a Java agent, the agent runs inside the broker or controller JVM and must match the Java version the broker requires. Brokers, Connect and tools require Java 17 from Kafka 4.0 onward. Confirm the exporter version supports your JVM, that its configuration file works with your release, and that agent overhead fits your heap and CPU budget. Check the exporter's own compatibility notes instead of assuming them.
Managed and operator-based exposure
Managed services and Kubernetes operators may expose a different set of metrics, use different endpoints or rename JMX values before they reach your monitoring system. Ask the service provider or operator maintainer which metrics are exported by default, whether KRaft controller and quorum metrics are included, and whether the operator's metric names match the upstream names used in this guide. Test each alert against real traffic in your environment before relying on it.
Security settings on metrics endpoints
Metrics endpoints can reveal topic names, client identities and cluster layout, so they need the same access controls as the rest of the cluster. Confirm whether your JMX or exporter endpoint uses TLS and authentication in your deployment, and who can scrape it. Verify these settings with your security team and the exporter documentation, because the defaults depend on how you configure the broker and the agent.
What changes for the on-call team after alerts are in place
The first change is the number of pages. Alerts that fire on a single sample of a metric, or on a value with no sense of what normal looks like for that cluster, wake people for conditions that clear on their own. Once you set a duration window for each alert, so a condition must hold for a set time before it pages, and compare lag and request latency against a baseline from your own traffic, most noise drops away. Under-replicated partitions during a broker restart and a short ISR shrink are the usual examples. Decide who owns tuning each threshold, and record the reason for every change so the next engineer can see why a number is what it is.
The second change is that every alert should link to a runbook that a tired engineer can follow at three in the morning. For each metric on this page, the runbook should state what the alert means, what to check first on the controller and brokers, which dashboard to open, and when to escalate. Write the runbook around the decision the on-call person must make, not around the metric definition. The detailed causes and recovery steps for under-replicated and offline partitions, for consumer lag that is growing, and for controller and metadata problems belong in the sibling topics on those subjects, so link to them from the runbook instead of copying their steps into it.
The third change is a review after each Kafka upgrade. Upgrade notes can change metric names, default settings and the behavior of controller and quorum signals, so an alert that worked on one release may go quiet or misfire on the next. Check the upgrade notes for the release you are moving to, confirm that every alert still resolves to a metric that exists, and replay the expected failure modes in a test cluster where possible. Confirm the supported upgrade path and release timing against the Apache Kafka downloads page before scheduling the change, and treat the post-upgrade alert review as part of the change window, not as follow-up work.
Official Apache Kafka sources for this page
Where can I confirm which Apache Kafka versions are currently supported?
The official Apache Kafka downloads page lists the supported releases, and it is the first place to check before you choose a version to monitor. At the time of the supplied reference, the page listed 4.3.1, 4.2.2 and 4.1.2 as supported, with all earlier versions archived. Check the page again before you build dashboards or alert rules, because the supported list changes as new releases ship and bugfix releases are made for supported lines only. Match your metric names and alert thresholds to the exact release your brokers and controllers run, and confirm any version-specific behavior in the release notes for that line. The project also states that it aims for three feature releases a year, which affects how often your monitoring baseline may need review.
Which Kafka upgrade notes should I read before changing monitoring for 4.3?
The Apache Kafka 4.3 upgrade notes are the reference to read before you change broker, controller or client settings that affect the metrics you alert on. They describe changes between releases, removed features and compatibility requirements that can rename or retire metrics or change default behavior. Read the notes for every version between your current release and your target, not only the latest one. Pay particular attention to KRaft-related changes, because quorum and controller metrics are part of the alert set in this guide. Record which metric names your dashboards depend on, then compare them against the notes so that a silent rename does not turn an alert into a permanent zero. Your team should confirm any change in your own environment with a test cluster first.
Where is the official guidance on Kafka authentication and encryption?
The Apache Kafka 4.3 security overview explains how clients, brokers and tools authenticate and how traffic is encrypted. It documents SSL/TLS for encryption and SASL mechanisms, including GSSAPI (Kerberos), PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER. It also states that security is optional, so a cluster can run unsecured or mix secured and unsecured listeners. For monitoring, this matters because authentication failures, TLS handshake errors and listener configuration can all surface as client errors or lag rather than as obvious broker faults. Review the overview before you decide which listeners to expose to your metrics pipeline and which credentials your exporters use. Confirm the chosen mechanism against your identity provider and your own security policy before rollout.
Does the official documentation give exact alert thresholds for Kafka?
The supplied official sources do not provide exact alert thresholds, so you should treat any number as a starting point to tune against your own baseline. The downloads page, upgrade notes and security overview define what to run and how to configure it, but they do not set limits for under-replicated partitions, request latency or consumer lag for your workload. Start with alerts on conditions that should never occur in a healthy cluster, such as offline partitions or a controller count other than one, then add trend-based alerts for rising lag and ISR shrinks once you have measured normal behavior. Ask your engineering team to document the baseline period, the traffic profile and the owner of each alert so the thresholds can be reviewed after upgrades.
How do I know if a Kafka upgrade will support rolling restarts?
The Kafka time-based release plan states that rolling upgrades are supported from each release in the past year, which means the last three releases, to the latest version. Use that plan together with the upgrade notes to confirm your starting version before you schedule a rolling upgrade. Monitoring is part of the upgrade check: during a rolling restart, expect temporary under-replicated partitions and ISR changes as brokers restart, and confirm they clear before moving to the next node. If your cluster is outside the supported upgrade window, the upgrade notes recommend an intermediate step for KRaft clusters, so plan for more than one hop. Confirm the exact path with your own change process and a rehearsal in a non-production environment.
Where can I find the official Kafka source code and issue history?
The apache/kafka repository on GitHub holds the source code for Apache Kafka, and it is the place to check when you need the exact behavior of a metric, configuration key or release change. Documentation describes intent, but the code and commit history show what a given version emits. When an alert fires on a metric that behaves unexpectedly, search the repository for the metric name and read the associated change before you assume a fault in your cluster. Pin the tag or branch that matches your running version so that you are not reading behavior from a different release. Your team should record the version and commit used during investigations so that the findings can be reproduced later.
What license applies to Apache Kafka, and is there a paid edition?
Apache Kafka is an Apache Software Foundation project released under the Apache License 2.0, identified as Apache-2.0 in SPDX. The Apache Software Foundation project itself does not offer a paid edition, so the software you deploy from the official sources is the open-source release. Licensing questions still matter for your organization because any commercial distribution, managed service or add-on tool you choose on top of Kafka carries its own terms. Confirm the license of each exporter, dashboard tool and operator you add to the monitoring stack, and ask your legal or procurement team to review them against your policy. This page does not provide legal advice, so treat the license summary as a starting point for your own review.
How DeployOpen scopes and delivers Kafka monitoring
The sequence below takes a Kafka cluster from an agreed list of metrics to dashboards, tested alerts and runbooks that your team can operate after handover. Each step ends with a check you can review before the next one starts.
Inventory the cluster and its monitoring sources
DeployOpen reviews the cluster layout with your platform team: the Kafka version, whether it runs in KRaft mode with dedicated controllers or combined nodes, the listeners and security settings, and which metrics paths already exist, such as JMX exporters or a Prometheus stack. The output is a short inventory of brokers, controllers, client applications and consumer groups that will be in scope. You confirm the inventory before any configuration changes are made.
Agree the written scope and deliverables
DeployOpen writes down which metrics, dashboards, alert rules, notification routes and runbooks will be delivered, along with the systems they touch and what your team must provide, such as access to the monitoring stack and named alert recipients. Most engagements are fixed-scope, so the document also states what is excluded, for example changes to application code or broker tuning. Work starts only after both sides sign off on this scope.
Expose and collect the metrics
Within your infrastructure, the team enables the Kafka metrics endpoints and the collection path agreed in scope, then confirms that each planned metric is returned for every broker and controller. Scrape intervals, label conventions and retention are set to match your existing monitoring standards. The check is a query for each metric family returning data from all nodes, with gaps documented.
Build dashboards for each failure domain
Dashboards are organized around the questions operators ask during an incident: is the cluster healthy, are partitions under-replicated, is there one active controller, how is request latency trending, and is consumer lag growing. DeployOpen builds these in the visualization tool you already run and labels each panel with the metric it reads. Sign-off is a review with your on-call engineers who confirm each panel answers its question.
Write and tune alert rules
Alert rules are created for the agreed set of metrics, starting from the thresholds in this guide and adjusted to your traffic patterns, replication factor and minimum in-sync replica settings. Each rule records its severity, the expected response and whether it pages or only notifies. The output is a rule file under version control with a review record that explains any threshold changed from the starting values.
Test alerts with controlled failures
In a non-production environment or an agreed maintenance window, DeployOpen induces controlled failures such as stopping a broker, slowing a follower or driving consumer lag on a test group, then checks that each alert fires, reaches the right recipient and clears after recovery. Any alert that does not fire or fires on normal behavior is adjusted and retested. Testing is complete when every rule in scope has a recorded result.
Document runbooks and hand over
For each alert, DeployOpen writes a runbook that states the likely causes, the first commands or dashboards to check, escalation paths and the recovery steps for common cases. Your team walks through the runbooks in a handover session and runs one drill from the documents. The handover is closed when your operators can resolve the test failures from the runbooks alone.
Agree ongoing support and upgrade cover
Once monitoring is live, thresholds usually need revisiting as traffic grows and Kafka releases change metric names or defaults. DeployOpen offers retainers for ongoing support, periodic threshold reviews, upgrade checks and continuous testing of alerts, with the scope and cadence written into the agreement. Ask for the retainer terms in writing so you know what is covered after handover.
