Kafka OAuth and OIDC authentication with Keycloak

Keycloak as the token issuer for Apache Kafka OAuth and OIDC authentication

A platform team that wants Kafka clients to sign in with central identities instead of shared passwords should plan for SASL/OAUTHBEARER, the supported method in which Keycloak issues tokens and Kafka brokers validate them. The Kafka security overview lists OAUTHBEARER among the documented SASL mechanisms, next to GSSAPI, PLAIN and the SCRAM variants. Keycloak supports OpenID Connect and OAuth 2.0, so it can act as the token issuer for clients and brokers alike, and the Kafka side only needs to trust that issuer.

The work splits into three parts. Keycloak authenticates the user or service account and issues a signed access token. Each broker checks the token's signature against the issuer's JWKS endpoint and validates claims such as the issuer, audience and expiry. Authorization then happens in Kafka itself: the principal derived from the validated token is matched against ACLs stored in the KRaft metadata log. A token that validates but matches no ACL is still denied, so the identity model in Keycloak and the ACL model in Kafka must be designed together rather than in sequence.

Before building, confirm three things with your own Keycloak administrators and the Kafka documentation for your target release: which claim becomes the Kafka principal, which realm and clients will be dedicated to Kafka, and whether the issuer's JWKS endpoint is reachable from every broker network segment. Decide early whether human users, service accounts or both will authenticate, because the lifetime and renewal of their tokens differ. The Apache Kafka security overview is the natural next read for the mechanism's configuration model.

What you need before enabling OAUTHBEARER on Kafka

OAUTHBEARER touches the Kafka release line, the Keycloak release, its database, certificates and the network between brokers and the issuer. Each of these must be settled before broker configuration starts, because a gap in any one of them shows up later as a failed login or a fragile setup.

Supported Kafka 4.x release line

Pick a release from the supported 4.x line listed on the Kafka downloads page, and treat older versions as archived. Brokers need Java 17 or later, so confirm the JVM on every broker host and in your image before you change listener configuration. Plan the upgrade path now if your cluster runs an older version, since rolling upgrades are supported only from recent releases to the latest version.

Client-side Java requirement

Every Java client and Kafka Streams application that will authenticate with OAUTHBEARER must run on a supported Java version. Clients and Kafka Streams applications need at least Java 11, which is lower than the broker requirement, so older application fleets may still qualify. Inventory each client runtime and its JVM, because the OAuth login module and token refresh behavior depend on the client library version you deploy.

Supported Keycloak release

Choose a Keycloak release from the supported line and confirm it against the project's release page before you pin the image. The current release at the time of writing is 26.8.1, but your organization may run a different supported version. Check that the OIDC discovery document, JWKS endpoint and token claims you plan to map are present in that version, and record the exact tag you deploy.

Production database for Keycloak

Keycloak must run on a production database from its supported list, such as PostgreSQL 14 to 18, MySQL, MariaDB, Oracle, Microsoft SQL Server or Azure SQL. The development file database is not suitable for production. If you run Keycloak in a multi-site or highly available design, confirm the database replicates synchronously, because both supported high availability designs depend on it.

TLS certificates for brokers and issuer

OAUTHBEARER login over the network should run on TLS, so each broker listener needs a certificate trusted by clients, and the Keycloak endpoint needs one trusted by brokers. Brokers fetch signing keys from the issuer, so the CA bundle used by the JVM on each broker must include the issuer's chain. Plan renewal and rotation for both sides, since an expired certificate on either path stops authentication.

Network path to the JWKS endpoint

Brokers must reach the issuer's JWKS endpoint over the network to validate token signatures, so open the route from every broker subnet, including any DNS and proxy path the JVM uses. Check that the issuer hostname resolves identically from brokers and from clients, and that egress rules permit the JWKS fetch during broker startup and key rotation. Test this path from a broker host before you change any listener.

Confirm the Keycloak version with the vendor

If you consume Keycloak through Red Hat build of Keycloak or another distribution, the supported version and release cadence may differ from the upstream project. Ask the vendor or distributor which release line they support, which database and Kafka client combinations they test, and how they handle breaking changes in minor releases. Get the answer in writing before you pin the version for production.

How to configure Keycloak and Kafka for OAUTHBEARER

This sequence takes a Keycloak realm and per-workload clients through to a Kafka cluster that validates bearer tokens against the realm's JWKS, maps a token claim to a principal, and authorizes that principal with ACLs. Each step ends with a check you can run before moving on.

Create the realm and confirm its issuer URL
An administrator creates a dedicated realm in Keycloak for Kafka workloads, separate from human sign-in realms, so token issuers and key sets stay scoped to this cluster. Record the realm's issuer URL, which takes the form of the realm base path under your Keycloak hostname. Confirm the realm is reachable from every broker network segment and that TLS on the Keycloak endpoint is trusted by the brokers. The check is a successful GET of the realm's OpenID discovery document (the .well-known/openid-configuration path under the issuer), which must return an issuer value that matches the URL you recorded character for character, including the trailing path and scheme.
Create a confidential client for each workload
Still in the Kafka realm, an administrator creates one confidential client per producer, consumer, Connect worker or service account, with client authentication and the client credentials grant enabled. Keep the client IDs descriptive and stable, because the ACL principal you build later will depend on them. Store each client secret in your secrets manager rather than in configuration files. Limit each client's scope to the Kafka audience or scope your broker validation expects, so tokens carry only what the cluster needs. The check is a token request from a workstation using curl against the realm's token endpoint with grant_type=client_credentials, which should return an access token whose decoded payload shows the expected client ID and audience.
Configure the broker listener for SASL_SSL and OAUTHBEARER
A platform engineer sets a dedicated broker listener to use the SASL_SSL protocol with the OAUTHBEARER mechanism, and keeps TLS enabled on that listener because Kafka sends the bearer token over the connection. Set the listener's SASL mechanism to OAUTHBEARER, add the JAAS login module configuration for the listener, and point the token validator at your realm's JWKS endpoint. Also configure the expected issuer and audience so the broker rejects tokens from other realms. Keep the inter-broker listener on a mechanism you have tested separately, so a token problem cannot take down replication. The check is that the broker starts without listener errors and its logs show the OAUTHBEARER listener bound, and that a deliberately malformed token is refused.
Set the principal claim and test the mapping
The broker derives a Kafka principal from a claim in the validated token, and this mapping determines which ACLs apply. Choose a claim that is stable for each workload, such as the client ID or a dedicated service-account claim, and set the broker's principal builder or claim configuration to use it. Document the resulting principal string for each client, because ACLs match on it exactly. Confirm the mapping against the principal format your Kafka version documents, and check the exact configuration keys in the Kafka security overview for the version you run. The check is an authenticated test connection from one client, with broker logs or the admin tool showing that the session is authorized as the expected principal string.
Apply ACLs to the mapped principal
Authorization stays on Kafka's side. In KRaft mode, the StandardAuthorizer stores ACLs in the cluster metadata log, and resources without a matching ACL are denied to non-super users unless allow.everyone.if.no.acl.found is set to true. Create ACLs for each mapped principal covering only the topics, consumer groups and cluster operations that workload needs, and keep super user membership to a small set of administrative principals. Leave the default-allow setting off unless you have a deliberate reason to change it. The check is that a describe or produce test from the client succeeds on its own topics and fails on a topic it should not reach, with the denial visible in broker authorization logs.
Roll clients over one workload at a time
Move workloads from their current authentication method to OAUTHBEARER in small batches, starting with a low-risk consumer. Each client receives its secret, switches to the SASL_SSL listener with the OAUTHBEARER mechanism and a token callback that refreshes tokens before expiry, and then runs its normal traffic. Keep the old listener available until every batch is confirmed, and track which principals still connect through it. Confirm token refresh works across at least one expiry cycle in the client's own logs before moving the next batch. The check is that each migrated client shows steady produce or consume activity under its new principal, with no authentication failures in broker logs, before the old credentials are revoked.

How Keycloak token claims map to Kafka principals and ACLs

This table traces one authenticated request from the Keycloak token claim the broker reads, to the Kafka principal it becomes, to the ACL check that decides what that principal may do. Read it top to bottom when designing the mapping, and check the default-deny row before any client goes live.

Stage	What happens	What to decide or confirm
Token claim	The broker reads a claim from the validated access token, such as the client ID, preferred username or a custom claim you add to the token	Choose one claim per client type and confirm it is present in tokens from your Keycloak realm, not only in a test realm
Principal name	The claim value is turned into the Kafka principal name the broker uses for authorization and logging	Confirm the principal name format your broker configuration produces, and keep service accounts and human users distinguishable
ACL resource	A topic, consumer group, cluster or transactional ID that the principal tries to use	List every resource each client needs, including consumer groups, which clients often forget
ACL operation	The action requested on the resource, such as read, write, describe or create	Grant the narrowest operation set per client, and review grants that combine write and describe for producers
Default with no matching ACL	In KRaft clusters, resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true	Keep this setting at its default unless you have a documented reason to change it, and review it with the broker configuration before go-live
Super users	Principals listed as super users bypass ACL checks	Keep the list short, confirm it names service identities rather than people, and review it after each Keycloak realm change
Mapping drift	A client's claim value or role changes in Keycloak but its ACLs stay the same, or the reverse	Assign ownership for the mapping and test new clients against a non-production cluster before creating ACLs

Security settings that decide whether OAuth authentication holds

OAuth authentication in Kafka is only as strong as the listeners, tokens, trust boundaries and defaults around it. The cards below cover the settings that most often decide whether a Keycloak-backed cluster is actually protected, and the checks to run before go-live.

TLS on every token-carrying listener

Kafka security is optional, so a cluster can run plaintext listeners next to secured ones. Bearer tokens sent over an unencrypted listener can be captured and replayed. Make every listener that accepts SASL/OAUTHBEARER use SSL or SASL_SSL, remove plaintext listeners where you can, and isolate any unavoidable unsecured listener on a separate network path. Confirm the listener map in listeners and advertised.listeners on every broker.

Short token lifetimes and secret rotation

Access tokens should expire quickly, and client secrets should have a rotation procedure that has been rehearsed before an incident forces it. Set token lifespans in Keycloak according to your risk tolerance, then confirm that client applications refresh tokens before expiry. Record which service accounts hold which secrets, who can rotate them, and how a leaked secret is revoked without a full cluster restart.

Restricting allowed issuer and audience

Brokers should accept tokens only from the Keycloak realm you operate and only for the audience assigned to Kafka. Without these checks, a token issued by another realm or for another application may pass signature validation. Confirm the expected issuer URL and audience values in the broker configuration, and test that a token from a different realm or audience is rejected before you rely on the setup.

Avoiding the allow-everyone default

In KRaft clusters, the StandardAuthorizer stores ACLs in the metadata log. Resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Leave that value at its default, grant access explicitly through principals derived from token claims, and review super user membership. Any change to this setting should go through change control because it widens access to every unprotected resource.

Manual approval for Keycloak operator updates

When Keycloak runs on Kubernetes through its operator, an automatic update can change the identity provider under running clusters. The Keycloak Operator documentation strongly recommends manual approval mode for operator updates. Pair that setting with a staging realm that mirrors production clients and scopes, so token format or claim changes are caught before Kafka brokers start rejecting tokens in production.

How to test token validation, ACLs and failure paths before go-live

Test the authentication path in order, and do not move on until each step passes. First, confirm that a client holding a valid token from the target realm connects over SASL/OAUTHBEARER on a TLS listener and can produce and consume on the topics it is granted. Second, confirm that an expired token is rejected, and that a token issued for a different audience is rejected even though its signature is valid. Third, confirm that a client whose principal has no matching ACL is denied, which matters because resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Record the broker log lines for each failure, since those are the evidence your operations team will use later. Run these tests against a non-production realm that mirrors the production client scopes, audience mappers and claim names, so that testing never changes production clients or users.

The most useful test is the principal change. In Keycloak, alter a role, group or claim mapping that feeds the principal name or the authorization claim your broker uses, then confirm that Kafka grants or revokes access on the next token issuance and not earlier than your documented token lifetime allows. Also confirm that a removed or disabled user cannot keep producing with a token that is still within its lifetime. If your design relies on the principal mapping in the broker configuration, test a changed claim name as well, because a silent mismatch usually shows up as denial for every client at once. Keycloak exposes metrics that can support this monitoring, and the Keycloak metrics guide is the place to confirm which metrics your version provides before you wire up alerts for token failures.

Repeat the full test set after every Keycloak upgrade, every Kafka upgrade and every change to the realm, client scopes or broker listener settings. Keycloak minor releases and the Keycloak Operator update path can change defaults, so confirm the behavior against the Keycloak upgrading guide before the change reaches production. Keep the test scripts, expected results and captured logs in version control so the results can be repeated and shown to auditors. Ask your Keycloak and Kafka vendors or support channels to confirm any behavior you cannot verify in your own environment.

Common Kafka OAuth failures and how to fix them

This table lists the failures teams most often hit when enabling SASL/OAUTHBEARER with Keycloak or another OIDC provider, with the likely cause, the fix and the log or configuration to check for each. Read it when a client cannot connect or a broker rejects a token, and work down the rows against the broker logs first. The main takeaway is that most failures come from time, trust, identity claims or listener configuration rather than from Kafka itself.

Failure	Likely cause and fix	What to check
Token rejected as expired or not yet valid	Clock skew between the identity provider, the client and the brokers. Synchronize all hosts with NTP or chrony and confirm the tolerance your broker validation settings allow before changing token lifetimes.	Compare host clocks, check the iat, nbf and exp claims in a decoded token, and search broker logs for validation errors on those claims.
Brokers cannot reach the JWKS endpoint	Network path, DNS or egress firewall rules block brokers from the Keycloak realm certificate endpoint. Open the route from every broker, or host a reachable copy of the key set inside the broker network.	Run a request to the JWKS URL from each broker host, check DNS resolution, and look for connection timeouts or refused errors in broker logs.
Certificate trust error on the issuer endpoint	The broker JVM does not trust the certificate chain of the Keycloak hostname, often because an internal CA is used. Add the CA to the broker truststore and confirm the issuer URL matches the certificate name exactly.	Check the PKIX or SSL handshake exception in logs, inspect the served chain with an TLS client, and confirm the truststore path and password in broker configuration.
Principal mismatch causes ACL denials	The principal Kafka derives from the token (by default the subject or a configured claim) differs from the principal named in ACLs, so authorization fails after authentication succeeds. Align the claim mapping with the ACL principal names, or write ACLs against the principal actually produced.	Check the authenticated principal in authorizer denial log lines, compare it with the sub or configured claim in the token, and list ACLs for that principal with the admin tooling.
Missing or wrong audience claim	The token lacks the audience value the broker expects, so validation fails even though the signature is valid. Add an audience mapper to the Keycloak client scope and set the matching audience check in broker configuration.	Decode the access token and confirm the aud claim, then compare it with the audience setting on the broker listener and the client scope in the realm.
Client cannot obtain a token with client credentials	The client is not enabled for service accounts, the secret is wrong or rotated, or the token endpoint is unreachable from the client network. Enable service accounts on the Keycloak client and update the secret in the client's secret store.	Check the token endpoint response code and error description in client logs, and confirm the client ID, secret and scopes requested.
Listener left unsecured or mixed	A listener is configured as plaintext, or an unauthenticated listener remains open alongside the OAUTHBEARER listener, so clients can connect without a token. Remove or restrict unsecured listeners and confirm each listener's protocol in the broker configuration.	Review listener and advertised listener settings on every broker, scan the open ports from the client network, and check which security protocol each connection uses in broker logs.
Token accepted but access denied on all topics	No ACLs exist for the principal, or the authorizer falls back to super-user-only behavior for resources without ACLs. Add explicit ACLs for the principal and review the allow.everyone.if.no.acl.found setting before relying on it.	Confirm the authorizer class in broker configuration, list ACLs in the metadata, and check the denial messages for the resource and operation named.

Questions buyers ask before integrating Keycloak with Kafka

What drives the effort and cost of integrating Keycloak with Kafka?

The main drivers are the number of clients and service accounts that need tokens, how many ACL rules you must design and map, the high availability layout of Keycloak, and how much testing you need before cutover. A fixed-scope engagement is priced from a written scope, so the quote should list each of these items. One-time work includes realm and client design, broker listener configuration, JWKS validation setup, principal mapping and testing. Recurring costs include token lifetime management, certificate renewal, and reviewing ACLs as teams change. DeployOpen does not publish a price list, so ask for a quote that separates the Kafka work, the Keycloak work and any retainer for ongoing support.

Can our internal team set this up without outside help?

Yes, if you have engineers who already run Kafka and Keycloak and who understand OAuth client credentials, TLS trust chains and Kafka ACLs. The hardest parts are usually not the configuration keys but the coordination: Keycloak realm ownership, certificate authority decisions and the authorization model on the Kafka side. Teams without one of these skills often spend their time on the first working login rather than on a design that holds up in review. A useful test is whether someone on your team can explain how a token's subject becomes a Kafka principal and which ACLs apply to it.

What does the timeline depend on?

The timeline depends on four things: the number of clients and service accounts, the ACL design, the high availability layout of Keycloak, and the testing scope. Keycloak documents three high availability architectures, and both supported multi-site designs require a synchronously replicated database, which can add lead time if your database team must provision it. Each additional client type adds configuration and test cases. Ask for a schedule that shows design sign-off, a non-production build, a test pass and a production cutover as separate milestones.

What do we need to provide for the integration?

You need to provide a named realm owner in Keycloak who can approve clients and scopes, network access between brokers, clients and the Keycloak endpoints, and a certificate authority or internal PKI that issues certificates for the brokers and the identity provider. You also need Kafka administrators who can review the ACL model and approve changes to broker listeners. Confirm early who holds each role, how change approvals work, and whether test environments mirror production trust chains. Missing any of these usually delays the first end-to-end token exchange.

Do other OIDC providers work with Kafka OAUTHBEARER?

Kafka's SASL/OAUTHBEARER mechanism works with any OAuth 2.0 or OIDC provider that issues signed JWT access tokens and publishes a JWKS endpoint that brokers can reach, but you should confirm this against your provider's documentation and your Kafka version. Confirm the token format, signing algorithm, how the subject or client identifier is exposed, whether the provider supports the client credentials grant, and how it handles token lifetime and key rotation. Keycloak is the provider this page covers, so the detailed configuration here applies to it. Other providers may need different claim mapping and principal rules.

How do Kafka and Keycloak upgrades affect this setup?

Kafka and Keycloak upgrade on separate schedules, so each upgrade can change the integration. Kafka 4.x supports rolling upgrades from the last three releases to the latest version, and the upgrade notes recommend moving KRaft clusters older than 3.3.x to 3.9.x before 4.x. Keycloak plans four minor releases a year, and breaking changes in minor releases are opt-in. Keycloak's Operator documentation strongly recommends manual approval mode for operator updates, and database schema migration runs automatically at startup by default, with manual migration available for large deployments. Test each upgrade against a copy of your realm and ACL configuration before production.

Does the integration require TLS and how are token principals mapped to ACLs?

Yes, you should run the OAUTHBEARER listeners over TLS, because bearer tokens are sent on every connection and must not travel in clear text. Kafka documents SSL and SASL as separate layers, so confirm the listener configuration on each broker and the trust store clients use. For authorization, Kafka's StandardAuthorizer stores ACLs in the cluster metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Map each token's client identifier or subject to a principal explicitly, and review that mapping whenever a Keycloak client is added or renamed.

Official Kafka and Keycloak documentation to read next

Start with the Kafka security overview if you need to confirm how SASL/OAUTHBEARER fits beside TLS, SCRAM and Kerberos on the same cluster, and whether unsecured or mixed listeners are allowed during migration. Read it before you write listener configuration, because it explains that security is optional in Kafka and that each listener can be set up differently. Pair it with the Kafka security overview for the mechanism list and the TLS model, then check your broker version against the Kafka release notes before you rely on any detail.

The *Authorization and ACLs* documentation answers the question your team will face after authentication works: how a token principal becomes a permission. Read it to confirm how the StandardAuthorizer stores ACLs in the KRaft metadata log and why resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set. Use the Kafka authorization and ACL guide to map each service account's sub or configured principal claim to explicit ACLs in a test cluster before production.

For Keycloak, the *Configuring the database* guide tells you which production databases are supported and that the dev-file database is for development only. The *High availability overview* sets out the three architectures and states that both supported designs need a synchronously replicated database, which is a question to settle with your platform team before sizing. Read the Keycloak database guide and the Keycloak high availability overview together. Finally, the *Keycloak upgrading guide* covers automatic versus manual schema migration. Read it before you plan a realm or client change window, and confirm the release you target against the Keycloak upgrading guide.

How DeployOpen scopes and delivers Kafka OAuth integration

DeployOpen plans, builds, tests and documents OAuth and OIDC authentication for Apache Kafka inside your own infrastructure, with Keycloak or another OIDC provider you run. Work starts from a written scope that names the clusters, clients, identity realms and ACL model covered, so the decisions behind the integration are agreed before configuration begins.

How we scope

Inventory every broker listener, client application and tool that will authenticate with OAUTHBEARER
Confirm the OIDC provider, realm layout, token lifetimes and the Keycloak version your environment runs
Agree the principal mapping from token claims to ACL principals before any configuration starts
Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work begins

How we build

Configure broker JWKS validation, issuer checks and TLS on each listener in a non-production cluster first
Create client credentials for each application and verify token acquisition and refresh behavior
Write StandardAuthorizer ACLs against mapped principals and test that denied operations fail as intended
Record each configuration decision, rollback step and test result as part of the delivery package

How we hand over

Deliver runbooks for client onboarding, secret rotation, signing key rollover and OIDC provider changes
Document the listener, authentication and authorization configuration with the reasons behind each setting
Walk your operations and security teams through failure modes such as expired tokens and JWKS fetch errors
Provide a test suite that your team can rerun after Kafka or Keycloak upgrades

How we support

Ongoing support, upgrades and continuous testing are available as retainers after handover
Retainer scope can cover reviewing Kafka and Keycloak release notes before each planned upgrade
Ask any partner for written confirmation of response expectations, since these are set in the contract, not in this page
Confirm with your auditors which evidence of access control they expect, since DeployOpen prepares evidence but does not issue audit reports

Apache Kafka OAuth and OIDC authentication with Keycloak