Why monitoring Kafka consumer lag starts with measuring the right signal
A common failure looks like this: every consumer process reports healthy, the group is up and processing, yet the data landing in downstream tables is hours old. The cause is usually that the team watched the wrong signal. Consumer lag is the distance between the newest offset written to a partition and the offset the consumer group has committed for that partition. It measures how much data is waiting, not whether consumers are alive. A consumer can be running, polling and committing offsets while still falling further behind, because the producers write faster than the group reads. Before you pick tools, confirm with your team which lag metric each consumer exposes and how the group's committed offsets are read, since those two facts decide what your alerts can see.
Total lag, the sum across all partitions of a topic or group, is the number most dashboards show first, and it misleads in two ways. A large total can hide one partition that has stopped moving while the others are current, which often points to a stuck consumer instance or a poison record on that partition. A small total can hide a group that is idle on some partitions and saturated on others after a rebalance. The Kafka documentation describes partitions as the unit of ordering and parallelism, so the per-partition view is the one that matches how work is actually divided. The records-lag-max client metric, which reports the largest lag among a consumer's assigned partitions, gives a quick per-client view, and kafka-consumer-groups.sh --describe shows committed offsets, log-end offsets and lag for each partition of a group. Use both to check whether a problem is one partition or the whole group.
Lag that exists is normal in many systems, since a batch consumer may be deliberately behind for part of each cycle. The more useful alert is lag that grows over a window, because growth means consumption is slower than production. Decide early which trend you will alert on, how long the window should be, and who owds the response for each topic. The Kafka docs do not give a universal threshold, so set the numbers from your own producer rate and business tolerance, and test them against a replayed load. Prometheus-based exporters can expose these values for trend queries, a topic covered in the sibling monitoring pages. Plan this decision before go-live, because retrofitting trend alerts onto a group that is already falling behind is harder than building them into the first release. Kafka itself is licensed under the Apache License 2.0, and the project does not offer a paid edition, so the cost of this monitoring is the engineering time and infrastructure your team chooses to run, not a license fee.
Where lag can be measured and what each source tells you
The consumer client exposes its own view of lag through its metrics. The records-lag-max metric is the largest gap, across all partitions the consumer is assigned, between the latest offset the broker reports and the position the consumer has fetched. It is published by the application's Kafka client library, so the application team owns it, and it reflects what that one consumer instance sees at that moment.
- Per-instance scope: Each consumer instance reports its own maximum, so a group with many instances needs each instance's metric collected and aggregated.
- Fetch position, not commit position: The metric measures fetched progress, so records fetched but not yet processed or committed can look caught up.
- Blind to stopped consumers: A crashed or hung instance stops emitting metrics, so the lag it leaves behind disappears from this view.
- Blind to unassigned partitions: Partitions with no active consumer produce no client metric at all.
- Metric exposure depends on your setup: Confirm how your application exports client metrics (for example, through JMX or your metrics library) before relying on it for alerts.
Check that every consumer instance exports this metric and that someone owns the alert when it stops appearing.
The kafka-consumer-groups command-line tool, shipped with Kafka, reads committed offsets and log end offsets from the cluster and reports lag per partition for a group. A platform or operations engineer runs it on demand, usually from a bastion host or a maintenance job, and it needs admin access to the cluster through a configured client properties file.
- Per-partition detail: The
--describeoutput lists the current offset, log end offset, lag, and the consumer ID for each partition, which shows which partitions are behind. - Committed offsets: It reads what the group has committed, so a consumer that processes but does not commit will look behind.
- Point-in-time snapshot: It returns the state at the moment of the call, with no history, so trend questions need repeated runs or another source.
- Authorization applies: The account running it needs describe permissions on the group and topics, which you should confirm against your ACL design.
- Manual by default: Nothing alerts unless you wrap it in a script or scheduled job.
Before using it in incident work, confirm the exact flags and output columns for your Kafka version in the official documentation.
A lag exporter polls the cluster for committed offsets and log end offsets on a schedule, then exposes lag per group, topic and partition as time series for Prometheus. The exporter is run by the platform team, usually as a separate service with its own credentials, and it gives the only view here that can answer whether lag is growing over time.
- History and trends: Repeated samples let you alert on lag that grows across several scrape intervals instead of a single high reading.
- Cluster-side view: It reads from the broker, so it still reports a group whose consumers have all stopped, which is the main gap in the client view.
- Cardinality cost: Per-partition series multiply quickly across many groups and topics, so decide which dimensions you keep.
- Credential and network scope: The exporter needs read access to group and offset data and a route to the brokers, which must be covered by your access model.
- Exporter choice matters: Confirm which exporter you use, which metrics it names and how it handles groups that have no committed offsets.
Before relying on this source for paging, verify it against a manual kafka-consumer-groups run on a known lagging group.
Configuration and tooling options and when each fits
This table compares the four ways teams measure consumer lag: client metrics, the Kafka CLI, an offset exporter and the choice between per-partition and aggregated series. Read each row for the situation it suits, what it costs in cardinality or access, and what to confirm against the Kafka 4.3 documentation before relying on it.
| Option | When it fits | Cost in cardinality or access | Confirm in Kafka 4.3 docs |
|---|---|---|---|
| Client-side records-lag-max | Teams that own the consumer code and want lag reported from inside the application, including during rebalances | Each consumer instance exports its own series, so pod churn adds series; no broker-side view of groups with no running consumers | Metric names and tags in the consumer metrics section, and whether your client library version exposes them |
| kafka-consumer-groups describe | Ad hoc checks during an incident, or a runbook step before a reset or partition move | Needs admin access to the cluster and a reachable bootstrap address; output is a point-in-time snapshot, not a trend | Command options for --describe, the bootstrap and client config flags, and output columns for LAG and CURRENT-OFFSET |
| Offset exporter for Prometheus | Central alerting across many groups where you want lag for groups that have no live consumers | Polls the cluster on an interval, so lag is sampled rather than exact; the number of groups and partitions sets series count | Admin client permissions the exporter needs, and which group and topic filters the chosen exporter supports |
| Per-partition lag series | Finding a stuck partition, a hot key or a single slow consumer instance | Highest series count, growing with partitions times groups; costly in long retention storage | Partition-level offset metrics the tool exposes and how labels are named in your version |
| Aggregated total lag series | Group-level alerts and dashboards where the question is whether the whole group is falling behind | Low series count, but hides a single stuck partition behind a healthy total | Whether your tool sums per group and the time window it uses for rate calculations |
| Growth-based alert rules | Alerting on lag that rises over time instead of any non-zero lag, which fits most steady workloads | Needs a rate or delta over a window, so alerts respond late to sudden spikes | Your chosen retention and scrape interval, and the lag baseline for each topic under normal load |
Setting up lag monitoring step by step
This sequence takes lag monitoring from an unknown set of consumer groups to a verified exporter feeding per-partition series and alerts that fire on growing lag. Each step ends with a check so that the metrics match what the Kafka tools report before anyone relies on them.
Inventory consumer groups and their owners
List every consumer group on the cluster with kafka-consumer-groups.sh --bootstrap-server and --list, then record the topics each group reads, the team that owns the application, and the business impact if that group falls behind. Ask owners to confirm which groups are live and which are retired, because stale groups produce alerts nobody can act on. The step is done when every active group has a named owner and a contact in a shared register. Check the register against the group list and resolve any group without an owner before continuing.
Choose the lag measurement sources
Decide which sources will report lag: the consumer client metric records-lag-max exposed through the application's own metrics, the broker-side view from kafka-consumer-groups describe, or a dedicated exporter that queries the cluster for committed offsets and log end offsets. Each source answers a different question, so most teams use the exporter for fleet-wide coverage and client metrics for per-application detail. Confirm with the platform team which option your Kafka distribution and monitoring stack support. The step is done when the sources are written down with the question each one answers.
Deploy the exporter with read-only access
Install the chosen exporter where it can reach the brokers, then create a dedicated Kafka principal for it with describe permissions on consumer groups and topics only. In a KRaft cluster with the StandardAuthorizer, ACLs live in the metadata log, and resources without an ACL are limited to super users unless allow.everyone.if.no.acl.found is set to true, so the exporter principal needs explicit ACLs. Confirm the authentication mechanism with your security team, since SASL and TLS settings depend on the cluster. The step is done when the exporter scrapes successfully and a test principal without the ACL is refused.
Define per-partition series with bounded labels
Configure the exporter and Prometheus so that lag is recorded per consumer group, topic and partition, while keeping label values to a fixed, known set. Use the group and topic names as labels, avoid client IDs or hostnames that change on every pod restart, and drop any series for retired groups. Partition-level detail is needed to find a single stuck partition, but total lag per group is the number most alerts should use. The step is done when a query returns one series per partition for a sample group and the label count stays stable across a restart.
Verify values against kafka-consumer-groups describe
Pick two or three consumer groups, run kafka-consumer-groups.sh --bootstrap-server --describe --group for each, and compare the CURRENT-OFFSET, LOG-END-OFFSET and LAG columns with the exporter's values for the same partitions. Run the comparison during active traffic and again during a quiet period, because committed offsets update on a commit interval and small differences are expected. The step is done when the exporter's lag matches the tool's output within that interval for every sampled partition. A mismatch usually points to a wrong bootstrap address, a stale group, or a label collision.
Add alerts on growing lag
Write alert rules that fire when lag increases over a window instead of when lag is merely present, for example by comparing the current total lag for a group with its value a few evaluation periods earlier, and by checking that the group's committed offsets are advancing. Pair each rule with the owner from step one and a runbook note that says what to check first. Agree the window and threshold values with the group owners rather than copying defaults, since a batch consumer and a real-time consumer have different tolerances. The step is done when a test alert fires in a staging environment, reaches the owner, and resolves after the group catches up.
Review the register on a fixed schedule
Set a recurring review of the consumer group register, the exporter's series count and the alert rules, so that new groups are added, retired groups are removed and noisy alerts are retuned. Record each change with the owner who approved it. The step is done when the review has a named owner, a calendar entry and a first completed record that matches the live group list.
Alerting on lag that grows instead of lag that exists
A steady backlog is not the alert. Many consumer groups carry lag by design: a batch job that reads in bursts every night will show total lag climbing for hours and then draining to near zero. A static threshold on total lag will page someone for that pattern every night. The signal worth alerting on is whether lag for a group is growing faster than the consumer can drain it, sustained over a window longer than the normal burst. Alert on the sign and duration of the change, not on the level alone. Start by recording records-lag-max from the consumer clients and the per-partition lag from kafka-consumer-groups --describe for each group over several normal cycles, so you know what "normal" looks like before any rule is written.
Build rules from each group's own baseline. Use a rate-of-change expression, for example the increase in total lag over a 15 or 30 minute window, combined with a for duration so a single sample does not fire. Pair it with a check that consumption is still happening, such as committed offsets advancing, because a group whose lag rises while offsets stall is a stalled consumer, while a group whose lag rises and offsets keep moving is undersized. Those two cases need different responses, so route them separately. Confirm the exact metric names and labels your exporter emits in your own environment before writing rules, since the JMX exporter and Kafka Exporter name them differently.
Suppress expected noise. Planned rebalances, partition moves and offset resets produce lag spikes that look like incidents. Use maintenance windows or silences tied to a change ticket, and require the rate condition to persist past the expected recovery time. Replays are harder: a deliberate reset to an earlier offset creates large lag by design, so label those groups or tag the change so the alert excludes them while the replay runs. Review silences after each event, because a silence left in place hides the next real stall.
Securing the lag monitoring path
Lag data is less sensitive than message payloads, but it still shows which topics, consumer groups and tenants exist and how busy they are. The controls below cover the credentials that read group state, the authorizer rules that limit them, the network path to exporters, and how long the metrics are kept.
Dedicated read-only monitoring identity
Give the lag exporter and any admin tooling its own principal rather than reusing an application or admin account. The Kafka security overview lists SASL mechanisms (GSSAPI, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER) and SSL for authenticating clients and tools, so pick the one your identity platform supports. Check that the monitoring principal can read group offsets and nothing else, and that its credentials rotate on the same schedule as other service accounts.
Encrypted client connections for monitors
Every monitoring connection should use TLS, including the exporter's connection to the brokers and the path from kafka-consumer-groups runs on jump hosts. The security overview notes that security is optional and clusters can mix secured and unsecured listeners, so confirm that the listener the exporter uses is the secured one. Ask your team whether any plaintext listener remains enabled for internal tooling and who can reach it.
Scoped group describe permissions
Under the StandardAuthorizer, ACLs live in the cluster metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Grant the monitoring principal describe access on the consumer groups and topics it reports on, and avoid a cluster-wide wildcard. Leave allow.everyone.if.no.acl.found unset, then test that the exporter still returns lag for a group it should see and fails for one it should not.
Exporter endpoints kept private
Lag exporters expose an HTTP metrics endpoint that lists group, topic and partition names. Bind it to an internal interface, restrict it at the network layer to the Prometheus servers, and do not publish it through a public load balancer or ingress. Check that the scrape path uses the same network controls as the rest of your monitoring stack, and record which hosts are allowed to scrape it.
Retention of lag metrics
Time series of per-partition lag reveal topic names, tenant or team identifiers and traffic patterns long after the topics are gone. Decide how long Prometheus or your long-term store keeps these series, and whether high-cardinality labels can be dropped or hashed before storage. Ask your data protection or compliance lead to confirm the retention period in writing, since it depends on the data classification of your topics and the rules that apply to your organization.
Access to dashboards and alert history
Grafana dashboards and alert notifications can expose the same group and topic names as the exporter. Limit dashboard folders that show per-tenant lag to the teams that own those consumers, and make sure alert messages do not paste message content or sensitive header values. Review who can export panel data or query the datasource directly, since those routes bypass the exporter's own controls.
Questions teams ask before monitoring Kafka consumer lag
How long does it take to set up Kafka consumer lag monitoring?
The duration depends on how many consumer groups, topics and clients need coverage, and on how much of the lag signal already exists. Teams that already run Prometheus and expose client metrics mostly spend time on label design, alert routing and agreeing on which groups matter. Teams starting from nothing also need to deploy an exporter, confirm network access to the brokers and tune alerts against real traffic. No fixed duration is set here; ask any partner for a timeline tied to a written list of groups, clients and alert channels.
Can an in-house platform team monitor consumer lag without a partner?
Yes, most platform teams can do this with the Kafka CLI, the Prometheus ecosystem and their existing alerting tools. The work is mostly about discipline: knowing which lag numbers are expected for each group, instrumenting application clients, and maintaining alert rules as topics change. A partner adds value when the team lacks Kafka operations experience, needs a repeatable pattern across many teams, or wants the setup documented and tested before go-live. The skills to check honestly are JMX or client metric exposure, Prometheus rule authoring and on-call ownership of consumer groups.
What does a partner need from our team to start?
A partner needs a short list of inputs before scoping. These usually include:
- An inventory of Kafka clusters, consumer groups and the owning application teams
- Access to the monitoring stack, such as Prometheus and Grafana, and the alert routing used by on-call staff
- Read access to cluster metadata and the ability to run the consumer group describe command against each cluster
- A list of applications whose clients can expose lag metrics, and any change approval process that applies to them
- Named owners who can confirm which lag values are expected for each group
The written scope should state which of these the client provides and which the partner delivers.
How does the cost of lag monitoring depend on partitions and consumer groups?
The cost is driven mostly by the amount of labeled data and the number of groups and clients to cover. Each partition and consumer group adds series to the metrics store and rows to the lag output, so large clusters with many short-lived groups need more careful label design and retention planning. Software licensing is not a cost here, because Apache Kafka is released under the Apache License 2.0 and the project offers no paid edition. The recurring costs are infrastructure for metrics storage, staff time for alert tuning, and any retainer for ongoing support. Ask a quote to separate one-time setup from recurring operations and to state how partition and group counts were assumed.
Does consumer lag monitoring replace broker metrics?
No. Consumer lag shows whether consumers are keeping up, but it does not show whether brokers are healthy. A cluster can show small lag while under-replicated partitions, an unstable controller quorum or rising request latency signal trouble that will soon affect consumers. Broker metrics explain why lag is growing, and lag explains whether the business impact has started. Teams should monitor both, with consumer lag alerts tied to the owners of each group and broker alerts tied to the platform team.
Should we alert on lag that exists or lag that keeps growing?
Alert on growth in most cases, and use a fixed lag threshold only as a secondary guard. A group with a steady backlog from a batch job may be healthy, while a group whose lag rises every few minutes is falling behind even if the total is still small. Growth alerts need a window long enough to ignore normal spikes, and they should be reviewed against each group's expected processing pattern. Confirm the exact thresholds and windows during testing with the owning team, because they depend on message rate, partition count and downstream recovery targets.
Does lag monitoring cover security and access control on the cluster?
Lag monitoring reads consumer group state, so the monitoring identity needs read access to the relevant groups and cluster metadata. Kafka authenticates clients and tools with SSL or SASL, and authorization in KRaft clusters is stored as ACLs in the cluster metadata log. Grant the monitoring identity only the describe permissions it needs, store its credentials in your secrets system, and review which consumers can see group details. Monitoring does not replace the security controls required for the cluster, which should be planned separately.
How DeployOpen scopes and delivers lag monitoring
DeployOpen approaches consumer lag monitoring as part of the Kafka platform it runs in your infrastructure, not as a standalone dashboard. We agree the scope in writing first, build the lag signals against your actual consumer groups and topics, and document how the team should respond when lag starts growing.
How we scope
- Inventory every consumer group, its client library and how each group reports lag today
- Agree which groups need paging alerts and which only need dashboards or reports
- Record the lag baseline for each critical group during normal operation before setting thresholds
- Confirm which clients expose records-lag-max and which need an exporter or describe command
How we build
- Configure lag collection from clients, kafka-consumer-groups output or a Prometheus exporter, as the scope defines
- Build per-partition and total lag views so hot partitions are visible next to group totals
- Write alert rules that fire on sustained growth in lag, not on a single high reading
- Test each alert against replayed or simulated slow-consumer conditions in a non-production environment
How we hand over
- Deliver the written configuration, alert rules, dashboards and collection scripts in your repositories
- Document what each alert means, who receives it and the first diagnostic steps to take
- Walk your operations team through the dashboards and a live alert exercise before handover
- Record the lag thresholds, their basis and the owner who approves changes to them
How we support
- Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts
- Ongoing support, Kafka upgrades and continuous testing are available as a retainer
- Review alert noise and threshold drift after Kafka or client library upgrades when a retainer is in place
- Ask your quote to separate the one-time build from any recurring operations or retainer work
