How KEDA autoscaling Kafka consumers on Kubernetes works
Consumer deployments that fall behind during traffic peaks and sit mostly idle otherwise are a common symptom of fixed replica counts. The KEDA Apache Kafka scaler scales on total consumer group lag, and its default lagThreshold is 10, meaning KEDA adds replicas when the backlog per replica exceeds that value. This page covers the planning decisions around that behavior for consumers you run on Kubernetes clusters you control, not a general KEDA installation tutorial. Those decisions are the threshold, the replica ceiling, the authentication method and the way scaling interacts with consumer group rebalancing.
The scaler works by reading the committed offsets of the consumer group and the latest offsets of each partition of the subscribed topics, then summing the lag. Because the input is total group lag, a topic with many partitions but a small backlog will still scale on the total, so the threshold should be set from the message rate each replica can clear, not from a generic number. The scaler does not exceed the partition count of the topic unless allowIdleConsumers is true, so a replica beyond the partition count will sit idle without that setting. Confirm the current behavior for your KEDA version in the KEDA Apache Kafka scaler documentation before you set the ceiling.
Authentication is a planning item, not an afterthought. The scaler supports SASL plaintext, SCRAM, GSSAPI and OAUTHBEARER, plus TLS, so the credentials KEDA uses must match the listener and the ACLs your brokers enforce. The scaler needs read access to the consumer group and topic metadata, which is a separate grant from the consumer's own permissions. Decide early which principal KEDA runs as, where its secret lives and who rotates it, because a wrong or expired credential stops scaling silently for the affected group.
What you need before scaling consumers with KEDA
Autoscaling consumers on lag only works when the cluster, the Kubernetes platform and the consumer groups are ready for it. The cards below cover what must be in place before KEDA changes a single replica count, and which settings to confirm first.
Kafka cluster in KRaft mode
Run the cluster on a supported Kafka release, 4.1.x, 4.2.x or 4.3.x, in KRaft mode. Check the downloads page for the current patch level before you pin a version, because bugfix releases are made only for supported lines. KEDA reads consumer group lag from the brokers, so a broker-side problem shows up as false scaling signals. Confirm that the team can read group offsets and lag from the cluster before you add any scaler.
Kubernetes cluster you control
KEDA runs as workloads inside a Kubernetes cluster you operate, so the control plane, node capacity and upgrade schedule must be under your team's control. Confirm the Kubernetes minor version against the version skew policy before installing KEDA, and check that the cluster has enough headroom for the maximum replica count you plan to allow. Scaling out without node capacity leaves pods pending and makes lag look worse, not better.
KEDA version and documentation match
Install a KEDA release and read the scaler documentation for that same version. Settings such as lagThreshold, allowIdleConsumers and authentication fields can change between releases, so a mismatch between the deployed controller and the docs you follow leads to silent misconfiguration. Record the KEDA version in your change tickets and confirm it with your platform team before every upgrade.
Consumer groups with stable offsets
Autoscaling assumes that each consumer group has committed offsets and that the group is stable. Confirm that consumers commit offsets with a defined reset policy, that group IDs are fixed in deployment manifests, and that retention is longer than the longest expected backlog. Groups with missing or expired offsets report misleading lag, and KEDA will scale on that number.
Network path and TLS certificates
The KEDA operator and the consumer pods must reach every advertised broker address, not only the bootstrap address. Map the listener ports, confirm the certificate chain that the brokers present, and decide where the CA bundle for the scaler will be stored. Kafka supports SSL and SASL, and the scaler supports TLS and several SASL mechanisms, so the authentication method chosen here must match the broker listener configuration.
Credentials stored as Kubernetes Secrets
The scaler needs SASL credentials or TLS material, and these must reach the KEDA trigger authentication without being written into manifests in plain text. Kubernetes Secrets are stored unencrypted in etcd by default, so confirm that encryption at rest is enabled and that list and watch access on Secrets is restricted to the components that need it. Use a dedicated Kafka principal for the scaler with read-only access to group lag.
Configure the KEDA Kafka scaler in order
This sequence takes a consumer deployment from a visible consumer group to a ScaledObject that scales replicas on lag, with a checked result at each step. Each check shows whether the next step is safe to start.
Confirm the consumer group and its lag
Your platform team or the application owner starts the consumer deployment with a fixed consumer group ID, then reads the group's offsets and lag from a Kafka admin client or the kafka-consumer-groups command. The group must exist and report lag per partition before you add any autoscaling. Record the partition count of the topic, because KEDA does not scale past it unless allowIdleConsumers is set. This step is done when the group appears in the broker's group list and lag values change as you produce messages.
Create the authentication secret
Create a Kubernetes Secret in the same namespace as the consumer deployment that holds the SASL username and password, the CA certificate or the client certificate, depending on the mechanism your brokers enforce. The KEDA TriggerAuthentication or Secret reference points to these keys rather than embedding values in the ScaledObject. Limit who can read the Secret with RBAC, because Kubernetes stores Secrets unencrypted in etcd unless encryption at rest is configured. This step is done when the Secret exists, its keys match the names your TriggerAuthentication expects, and the KEDA operator can read it.
Define the ScaledObject trigger
Write a ScaledObject that targets the consumer Deployment and sets the Kafka trigger fields: bootstrapServers, topic, consumerGroup and lagThreshold. The lagThreshold is the lag per replica that KEDA aims to hold, and it defaults to 10 if omitted, so set it deliberately for your message cost and latency target. Apply the manifest through your normal GitOps or deployment path. This step is done when the ScaledObject reports Ready and KEDA has created the HPA it manages.
Set replica bounds
Set minReplicaCount and maxReplicaCount in the ScaledObject. Keep the maximum at or below the topic's partition count unless you have decided to set allowIdleConsumers, because extra consumers in a group sit idle when there are no partitions left to assign. Keep the minimum at a level that lets the group keep consuming when lag is zero, and confirm that the cluster has capacity for the maximum. This step is done when kubectl shows the HPA with the expected minimum and maximum and no warnings about the replica range.
Decide on allowIdleConsumers
Choose whether consumers beyond the partition count should run. With allowIdleConsumers left false, the replica count stops at the partition count, which keeps spare pods from consuming resources. Setting it to true lets KEDA add idle consumers that hold no partition and only take over when a partition is reassigned, which can help with failover but costs compute. Record the decision and the reason in the platform runbook, and tie it to your partition plan. This step is done when the field value is set in the manifest and the choice is documented.
Verify scaling under a controlled produce load
Produce a known number of messages to the topic at a steady rate using a test producer, then watch lag in the consumer group and the replica count on the Deployment. Lag should rise, the HPA should add replicas until lag per replica approaches lagThreshold, and replicas should fall back to the minimum after the group drains. Also check for rebalance log lines and consumer restarts during scale-up, because frequent rebalances mean the scaling step is too aggressive or the consumers are slow to join. This step is done when the replica count tracks the lag you produced and the group returns to zero lag without repeated rebalances.
Choosing lagThreshold, the partition cap and allowIdleConsumers
On a topic that receives a few messages at a time, the KEDA Kafka scaler's default lagThreshold of 10 is a small number of messages. A short burst can push total consumer group lag above that value, add a replica, and then the backlog drains within seconds and the replica is removed again. The result is replica count that rises and falls with every burst, which adds startup cost to each consumer and can interrupt processing. Lag on a quiet topic is usually a handful of messages, so a threshold that looks conservative for a busy topic can be too sensitive for a quiet one.
- Measure the normal backlog first: Watch consumer group lag over several business cycles before setting any threshold.
- Raise lagThreshold above normal noise: Choose a value that a routine burst does not cross, then confirm it against the measured baseline.
- Pair the threshold with a cooldown: Use the scaler's cooldown and polling settings so brief lag spikes do not trigger repeated scale events.
- Set a floor for minReplicaCount: Keep at least one replica running if a delayed first message matters more than idle cost.
- Check consumer startup time: Slow starts make flapping more expensive, so the threshold and startup behavior should be reviewed together.
Before you go live, confirm the threshold against a replay or a load test that reproduces your real burst pattern, not an average.
On a topic with sustained high volume, total lag is a poor proxy for how much work each replica carries. The KEDA scaler divides lag by lagThreshold to decide how many replicas it needs, so the threshold effectively sets how much backlog one replica is expected to absorb. If the threshold is too high, lag grows before scaling responds and consumers fall behind a growing backlog. If it is too low, the deployment scales to the partition cap sooner than the workload requires. Tuning starts from measured per-replica throughput, not from the default.
- Measure per-replica throughput: Record how many messages one consumer processes per second under normal load, including downstream write limits.
- Derive the threshold from acceptable backlog: Choose a lag value that one replica can drain within your latency target.
- Watch downstream capacity: Scaling consumers can overload a database or API that they write to, so the cap may come from the downstream system rather than from Kafka.
- Compare lag growth with replica count: Confirm in your dashboards that lag stabilizes after scaling, rather than climbing while replicas are added.
- Plan for partition limits: If sustained throughput needs more replicas than there are partitions, the topic layout needs review, not a lower threshold.
Re-run the measurement after every change in message size, consumer code or partition count, since each one shifts the right threshold.
Kafka assigns each partition to at most one consumer in a group at a time. Without allowIdleConsumers, the KEDA Kafka scaler does not scale a deployment above the topic's partition count, so any replicas beyond that number would have no partition to read and would sit idle. With allowIdleConsumers set to true, KEDA can run more replicas than partitions. The extra replicas then stay idle, consuming cluster and node resources while they wait for a partition to be reassigned to them. Turning the option on therefore changes cost more than throughput, and it can change how often the group rebalances when pods start and stop.
- Know the cap before scaling: Check the partition count of each topic, since it is the upper bound of useful parallelism per consumer group.
- Leave allowIdleConsumers false by default: Keep idle replicas out of the deployment unless a failover or standby requirement justifies them.
- Account for idle pods in cost: Each idle replica consumes requests and memory, and the cost recurs for as long as it runs.
- Expect rebalances on membership change: Adding or removing consumers triggers partition reassignment, so frequent scaling events cause pauses in consumption.
- Use a static partition plan when possible: Choosing partition counts for expected peak parallelism avoids the need for idle replicas.
Confirm with your platform team whether partition counts can be increased, and note that adding partitions changes key-to-partition mapping for keyed topics.
Avoiding rebalance churn while scaling consumers
Every time KEDA adds or removes a consumer pod, the consumer group changes membership, and Kafka responds with a rebalance that pauses consumption while partitions are reassigned. Scaling is therefore not free for the group: a scale event that looks harmless on a dashboard can stop every consumer for a moment and replay some uncommitted messages afterward. The goal of this section is to make scaling changes rare and deliberate, so that the group spends most of its time processing rather than reassigning.
The first levers are the timing controls. The cooldownPeriod field on the ScaledObject governs how long KEDA waits after activity before scaling back to zero, and the behavior block of the HorizontalPodAutoscaler that KEDA creates controls how quickly replicas are added or removed through its scale-up and scale-down policies and stabilization windows. Lengthening the scale-down stabilization window and limiting how many pods can be removed in one step usually reduces churn more than any other single change. Confirm the exact field names and defaults against the KEDA version you run, since the HPA behavior API and ScaledObject fields evolve across releases.
Scaling on a short lag spike is the most common cause of repeated rebalances. A brief burst pushes lag above lagThreshold, a replica is added, the spike drains, and the replica is removed minutes later. Raising the stabilization windows, sizing lagThreshold against the normal backlog of your topics, and reviewing whether a sustained lag window is needed before scaling all reduce this loop. For clients that support it, a stable group.instance.id lets a restarted consumer rejoin without triggering a full rebalance, but it only helps where your client library and deployment supports static membership, so verify that before relying on it. Measure rebalance frequency before and after each change, using consumer group metrics or broker logs, so you can tell whether a tuning change actually reduced churn.
Mapping Kafka authentication settings to KEDA trigger fields
This table maps each Kafka authentication method to the KEDA trigger settings, the TriggerAuthentication secret keys and the matching consumer client settings. Read each row as one set that must agree across the scaler, the secret and the consumer. Confirm every field name against the KEDA 2.17 Apache Kafka scaler page before you write manifests.
| Auth method | KEDA trigger side | Secret references to set | Consumer client settings |
|---|---|---|---|
| SASL PLAIN | Set the SASL mechanism and enable SASL in the trigger metadata | Username and password keys in a TriggerAuthentication secret | security.protocol with SASL, sasl.mechanism set to PLAIN, credentials in sasl.jaas.config |
| SCRAM-SHA-256 or SCRAM-SHA-512 | Set the matching SCRAM mechanism in the trigger metadata | Username and password keys, with the same mechanism as the broker | sasl.mechanism matching the scaler, credentials in sasl.jaas.config |
| GSSAPI (Kerberos) | Set the GSSAPI mechanism; keytab or principal handling depends on the scaler version | Kerberos principal and keytab or ticket material, stored outside the manifest | sasl.mechanism set to GSSAPI, sasl.kerberos.service.name and a keytab or ticket cache |
| OAUTHBEARER | Set the OAUTHBEARER mechanism and the token endpoint or client details the scaler needs | Client ID and client secret, or the token material the scaler reads | sasl.mechanism set to OAUTHBEARER and the token callback or OIDC client settings |
| TLS (encryption and optional client certificates) | Enable TLS in the trigger metadata and point to the CA bundle | CA certificate, and client certificate and key if the broker requires mutual TLS | security.protocol with SSL or SASL_SSL, ssl.truststore or CA location, and client key material |
| Confirm before rollout | Check each field name and accepted value on the KEDA 2.17 scaler page | Confirm the secret key names the scaler reads for each method | Confirm the same mechanism and TLS settings on the broker listener and the consumer |
Securing scaler credentials and the Kafka listener
The KEDA scaler needs a Kafka identity to read lag, and that identity and its credentials become part of your cluster's attack surface. The cards below cover the Kafka ACLs the scaler should hold, where its credentials live in Kubernetes, who can read them, and how the listener is encrypted.
Scope the scaler's Kafka ACLs
Give the scaler principal only the access it needs to read lag: read on the consumer group it monitors and describe on the topics it inspects. Treat these as a starting point to confirm in your ACL design, not a fixed list. In KRaft clusters, the StandardAuthorizer stores ACLs in the metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Confirm that default before you rely on it, and check that the scaler principal cannot consume or produce on the topics.
Keep credentials out of manifests
Do not place SASL usernames, passwords, or certificate material directly in Deployment, ScaledObject, or TriggerAuthentication manifests stored in Git. Reference Kubernetes Secrets instead, and deliver them through your GitOps or secrets pipeline. Rotate the scaler's SCRAM or OAuth credentials on a schedule you document, and confirm that rotation does not require editing manifests in version control.
Encrypt Secrets at rest
Kubernetes Secrets are stored unencrypted in etcd by default. Configure encryption at rest for the API server's etcd data, or move the scaler credentials to an external store accessed through the Secrets Store CSI Driver. Ask your platform team which option applies to your clusters, and verify it by reading the stored object back from etcd after setup.
Restrict RBAC on trigger and secret objects
Limit get, list, and watch on Secret objects and on TriggerAuthentication resources to the service accounts and operators that need them. Anyone who can list Secrets in the namespace can usually read the scaler's Kafka password, so review bindings that grant broad read access. Apply the same least-privilege review to who can edit ScaledObject and TriggerAuthentication, because those objects control how the scaler authenticates.
Encrypt the Kafka listener
Enable TLS on the broker listener the scaler uses, and configure the KEDA trigger with the matching CA certificate. The Kafka security overview states that security is optional and clusters can mix secured and unsecured listeners, so an unsecured listener left open for convenience will still accept the scaler's traffic. Confirm which listeners are exposed to the cluster network, and disable plaintext listeners that the scaler does not need.
Common KEDA Kafka scaling failures and their causes
This table lists the failures teams most often hit when KEDA scales Kafka consumers, with the likely cause and fix for each. Read it top to bottom when scaling misbehaves, and start with the first check listed in the final row, because most symptoms trace back to a name mismatch, the partition count or the authentication settings.
| Failure | Likely cause | Fix and first check |
|---|---|---|
| Scaling never triggers | The topic or consumerGroup in the ScaledObject does not match the live consumer group, or total lag stays below lagThreshold | Confirm the group name and topic with the consumer group describe command, then compare total lag with lagThreshold (default 10) |
| Replicas stop at the partition count | KEDA does not scale above the number of partitions unless allowIdleConsumers is true | Check the partition count of the topic. Add partitions if you need more parallelism, or set allowIdleConsumers only if idle pods are acceptable |
| Scaler reports authentication errors | SASL mechanism, username, password or CA certificate in the TriggerAuthentication does not match the broker listener | Verify the listener's mechanism and TLS settings, then check that the secret referenced by TriggerAuthentication holds the expected keys |
| TLS handshake fails | CA certificate missing or wrong, or TLS not enabled in the trigger while the broker requires it | Confirm the broker listener uses SSL or SASL_SSL, and that the CA in the secret signed the broker certificate |
| Replicas flap up and down | lagThreshold is small relative to message rate, so each small burst or drain moves the desired replica count | Raise lagThreshold to a value that reflects processing time per message, and review the cooldownPeriod and polling interval |
| Repeated consumer rebalances during scale events | Each new or removed pod triggers a group rebalance that pauses consumption, especially with eager assignment | Use a cooperative assignor where your client supports it, and scale in steps instead of large jumps |
| Pods scale to zero while lag remains | minReplicaCount is 0 and the scaler does not see lag because of a wrong group, topic or credentials, so it reads no backlog | Test the scaler's view of lag first. Keep minReplicaCount above zero for topics where delay is unacceptable until the trigger is verified |
| Lag is high but consumers process slowly | Scaling responds to lag, not to processing speed, so a slow downstream dependency keeps lag high regardless of replicas | Profile per-message processing and downstream limits before raising maxReplicaCount |
Questions teams ask before scaling Kafka consumers with KEDA
What drives the effort and cost of autoscaling Kafka consumers with KEDA?
The main drivers are how ready the Kafka cluster already is, which authentication model it uses, how many consumer groups need scaling, and how much load testing the team needs before trusting the scaling rules. A cluster that already has planned partition counts, retention, ACLs and TLS needs less groundwork than one where those decisions are still open. Each consumer group adds its own trigger, lag threshold and cap to validate. Costs fall into discovery and design, Kubernetes and KEDA configuration, authentication wiring, load testing and handover. Setup work is largely one-time, while monitoring, KEDA upgrades and changes to partition counts recur. Ask any quote to list each cost component separately and to state that the scope is based on written requirements.
How long does this work take for different scopes?
Duration depends on scope, not on a fixed template. A single consumer group on a cluster with working authentication and monitoring is a small piece of work. Scaling many consumer groups across several clusters, or introducing SASL with TLS for the first time, adds discovery, testing and change control steps. Load testing windows often set the pace because the team needs to observe lag and rebalances under realistic traffic. DeployOpen does not publish durations; the timeline follows from the written scope, and you should ask for the milestones and the client dependencies in that scope before work starts.
What does the work need from our team?
The work needs named owners for the topics and consumer groups being scaled, decisions on the ACLs that KEDA's scaler identity will need to read lag, and a window for load testing against non-production or agreed production-like traffic. Your team also needs to confirm the Kubernetes cluster and Kafka cluster versions in use, who approves changes to consumer deployments, and where credentials are stored. Kafka's authorization model restricts resources without ACLs to super users unless configured otherwise, so ACL decisions are on the critical path. Ask the partner to list these inputs in the written scope so that dependencies are visible early.
Can our internal team do this without a partner?
Yes, many internal platform teams can do this alone, especially if they already run Kafka consumers on Kubernetes and understand consumer group lag. The areas that most often slow internal teams are authentication (SASL mechanisms and TLS trust chains), ACL design for a new scaler identity, and testing how consumer rebalances behave as replicas change. A partner is useful when the team lacks time for load testing, needs an independent review of the scaling rules, or must document the design for an audit. Compare the internal estimate for those activities with a quote built from the same written requirements.
How does KEDA fit alongside Strimzi or a plain Horizontal Pod Autoscaler?
KEDA adds an event-driven scaler that reads Kafka consumer group lag and drives the replica count of a Deployment, which a plain Horizontal Pod Autoscaler cannot read from Kafka on its own. Strimzi manages the Kafka cluster, topics and users as Kubernetes resources, so it complements KEDA rather than replacing it. Without KEDA, teams usually scale on CPU or memory, which may not reflect backlog. The engineering question is whether scaling should follow lag, and whether the cluster is managed by Strimzi or by another method. Confirm the KEDA version and its Kafka scaler documentation before choosing settings such as lagThreshold, the partition cap and allowIdleConsumers.
What are the risks of scaling consumers on lag?
The most common risks are scaling beyond the useful number of consumers, causing rebalance churn during scale events, and misconfigured authentication that stops scaling silently. The KEDA Kafka scaler does not exceed the partition count unless allowIdleConsumers is true, so extra replicas sit idle by default. Each scale-up or scale-down can trigger a consumer group rebalance, which pauses processing, so thresholds and cooldown behavior need testing. Scaling on lag also hides the cause of a backlog, such as a slow downstream system, so teams should monitor both lag and processing latency.
Which authentication methods can KEDA use with Kafka, and what should we check?
According to the KEDA Apache Kafka scaler documentation, the scaler supports SASL plaintext, SCRAM, GSSAPI and OAUTHBEARER, along with TLS. Choose the method that matches the Kafka listener configuration, and confirm that the scaler identity has the ACLs it needs to read consumer group lag. Kafka's security overview documents the SASL mechanisms the broker side accepts, so the scaler and broker settings must agree. Store credentials in a Kubernetes secret management approach you have approved, and rotate them through a documented process. Ask any partner to confirm the exact settings against the KEDA version you run.
How DeployOpen works on this
DeployOpen plans, implements, tests, documents and supports self-hosted open-source platforms inside your own infrastructure, including the Kafka consumer autoscaling setup described on this page. Engagements start with a written scope so the consumer group, the KEDA configuration and the cluster boundaries are agreed before any build work begins.
How we scope
- We confirm which consumer groups, topics and partition counts the KEDA ScaledObject will govern.
- We agree the lagThreshold, maximum replica count and allowIdleConsumers decision in writing before build starts.
- Engagements are fixed-scope, with deliverables and timelines agreed in writing and a quote that follows from that scope.
How we build
- We configure the KEDA Kafka scaler with SASL or TLS settings that match your brokers and secret handling.
- We test scale-out and scale-in against real lag patterns and check for rebalance churn during replica changes.
- We set up Kubernetes RBAC and Secret access for the scaler credentials, scoped to the namespaces you approve.
How we hand over
- You receive the manifests, configuration decisions and test results, with the reasoning for each threshold and cap.
- We document runbooks for lag alerts, scaler failures, broker authentication changes and manual scale overrides.
- We walk your platform and operations engineers through the setup so they can change thresholds with confidence.
How we support
- Ongoing support, KEDA and Kubernetes upgrades, and continuous testing are available as retainers.
- We review scaler behavior after partition changes, broker upgrades and changes to consumer processing time.
- We confirm current KEDA version compatibility with your cluster and Kafka release before any upgrade is scheduled.
