Exposing Apache Kafka through Apache APISIX

Exposing Apache Kafka through Apache APISIX for controlled client access

The APISIX kafka-proxy plugin lets a client reach a Kafka upstream through an APISIX route over WebSocket, while APISIX adds SASL/PLAIN authentication on the upstream connection. The route is worth considering when browser applications or external clients need to consume or produce Kafka traffic and you want the gateway to control who gets in, how the route is authenticated at the edge, and how traffic is limited before it reaches the brokers. Typical cases are a partner feed, a web dashboard that streams events, or a small set of external producers that cannot run a native Kafka client. Confirm first that the clients you need can work with a WebSocket transport, because most native Kafka clients do not connect that way.

This is a gateway-level access pattern, not a replacement for Kafka's own security. Kafka supports SSL and SASL for client authentication, TLS for traffic encryption, and ACLs stored in the cluster metadata log under the StandardAuthorizer, and those controls still decide what an authenticated principal may do on each topic. The gateway adds a layer in front of them: a route that only some consumers can reach, edge authentication, and rate limits. Plan for two credential sets, one the gateway uses to reach the brokers and one the clients present to the gateway, and confirm in writing how the kafka-proxy password is stored and rotated, since the plugin documentation says it is kept encrypted in etcd. Treat the broker-side ACLs as the authoritative control and the gateway as the outer boundary.

Before building, confirm the plugin version you will run and read its current documentation, since WebSocket and SASL behavior can change between APISIX releases. Decide which Kafka operations clients may perform, map those to topics and consumer groups, and check that broker ACLs match that list even when the gateway is in front. Test with a non-production cluster, including authentication failures, broken connections and a client that exceeds the rate limit. Ask the vendor or your own platform team to confirm how the route behaves under long-lived connections and how upgrades to the Kafka cluster will be coordinated with the gateway.

Which access pattern fits each Kafka client scenario

This table compares four ways traffic can reach Kafka through or around APISIX, with the fit, main constraint and open questions for each. Read it by matching your client type to a row, then confirming the listed items before choosing a design.

Access pattern	When it fits	Main constraint	What to confirm
kafka-proxy over WebSocket	External or browser consumers that need to read topics without a native Kafka client or direct broker reachability	The proxy authenticates with SASL/PLAIN toward Kafka, so clients depend on the WebSocket path and the plugin's protocol coverage	Which Kafka client operations the plugin supports for your workload, and how it behaves under your throughput and consumer group needs
Native Kafka clients on broker listeners	Internal services and data pipelines that need full protocol features, consumer groups and high throughput	Clients must reach broker listeners directly, so APISIX is not in the data path and edge controls must come from network design	Which SASL mechanism and TLS settings each listener uses, and whether ACLs in the KRaft authorizer cover each client identity
kafka-logger for APISIX logs	Shipping request and response logs from the gateway into Kafka for audit or analytics pipelines	This is one-way log export, not a client access path, and batching means logs arrive in groups	Batch settings for your volume, the SASL or TLS options your cluster requires, and the topic retention and access policy for log data
Internal-only pattern	Workloads inside one trust zone where brokers stay on private networks and no external consumer exists	Limited edge visibility and rate limiting for Kafka traffic, since no gateway sits in front of the brokers	Network segmentation, which teams hold broker credentials, and whether an internal gateway is needed later for auditing

Authentication and rate limits at the edge

The gateway can add a layer of control in front of Kafka that the broker does not offer on its own, including who may connect, how fast, from where and under what logging. These controls sit alongside broker-side authorization, not in place of it, so each card below names what the gateway enforces and what Kafka must still enforce.

Route-level authentication

Attach an APISIX authentication plugin to the WebSocket route so that each client is identified before the upgrade completes. Options such as key-auth, jwt-auth or openid-connect are documented for APISIX, and the choice depends on whether your clients are internal services or external partners. Check how the identity reaches the Kafka side, because the kafka-proxy plugin uses its own SASL/PLAIN credentials toward the broker. A gateway identity is not a Kafka principal unless you map one to the other deliberately.

Rate limiting at the edge

APISIX traffic-control plugins can cap request rates per consumer or per client address, which limits how aggressively a single client can open connections or send messages through the route. Kafka has its own quotas on the broker, so decide which layer owns each limit. Ask the vendor or your test team to confirm which limit types apply to long-lived WebSocket frames before you rely on them for producer fairness.

IP and access restrictions

Restrict which networks may reach the Kafka route, for example partner egress ranges or a private subnet used by internal consumers. Restrictions apply at the gateway listener, so they stop traffic before any authentication step runs. Confirm how the gateway sees client addresses when it sits behind a load balancer, because a misread address can either lock out valid clients or let untrusted ones through.

TLS on the client side

Terminate TLS for clients at APISIX so that credentials and message payloads are not sent in clear text across the client network. Kafka supports SSL/TLS for its own listeners, and the broker leg should be encrypted as well, so plan certificates for both hops. Record which CA each client trusts and who rotates the certificates.

Request and response logging

The kafka-logger plugin can send request and response logs as JSON to a Kafka topic in batches, by default every 5 seconds or at 1,000 entries. It supports TLS and SASL PLAIN, SCRAM-SHA-256 and SCRAM-SHA-512. Decide which fields reach the log, since payloads may contain personal data, and make sure the logging topic has its own ACLs so that audit records cannot be read or altered by ordinary consumers.

Interaction with Kafka ACLs

Gateway controls do not replace broker-side authorization. In KRaft clusters, the StandardAuthorizer stores ACLs in the metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Define topic, group and cluster ACLs for every principal the gateway uses, so that a request that passes the edge still meets the broker's own rules. Review both layers together during any access change.

Setting up a kafka-proxy route in APISIX

This sequence takes a Kafka cluster you already run and places an APISIX route in front of it, so clients reach Kafka through the gateway with SASL credentials, edge authentication, rate limits and logging in place. The result is a tested path where broker-side ACLs still decide what each identity can do.

Confirm Kafka version and listener setup
Confirm which Kafka release the cluster runs and whether it is in KRaft mode, since Kafka 4.0 supports only KRaft and ZooKeeper-based clusters must migrate before upgrading to 4.x. Record the listener that the gateway will use, its SASL mechanism (PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 or OAUTHBEARER are documented options), and the TLS settings between the gateway and the brokers. Check the Java runtime required by the broker version. Done means you have a written listener map and a test login against the broker from a host on the gateway's network segment.
Create the Kafka upstream with the kafka scheme
In the APISIX Admin API or your configuration-as-code files, define an upstream that uses the kafka scheme and lists every broker address reachable from the gateway. Keep the broker list in configuration so that adding or replacing brokers does not change client-facing routes. Confirm that the upstream nodes resolve and accept connections from the gateway hosts. Done means APISIX reports the upstream as present and a direct TCP connection to each broker succeeds from the gateway network.
Configure the kafka-proxy plugin and WebSocket route
Attach the kafka-proxy plugin to a route that has WebSocket enabled, and set the SASL/PLAIN username and password that APISIX uses toward the brokers. APISIX stores the password encrypted in etcd, so confirm your encryption settings before you write the secret. Clients connect to this route over WebSocket, so verify that your load balancers and any upstream proxies allow long-lived upgraded connections. Done means the route is active and a WebSocket handshake to the route completes without an error from APISIX.
Apply edge authentication and rate limits
Add the authentication plugin your identity model needs, such as key-auth, jwt-auth or openid-connect, so that only known consumers can open a session. Authenticate at WebSocket upgrade time and then set limit-count or a similar traffic control plugin to cap connection and request rates per consumer. Agree with the Kafka owners which limits apply, because the gateway cannot replace broker quotas. Done means an unauthenticated client is refused at the upgrade and an authenticated client that exceeds the limit receives a rejection.
Configure request and response logging
Set up the kafka-logger plugin, or your chosen log sink, to record request and response metadata for the route. The plugin batches entries, by default every 5 seconds or at 1,000 entries, and supports TLS and SASL PLAIN, SCRAM-SHA-256 and SCRAM-SHA-512 toward the log cluster. Do not log payloads that contain personal or regulated data without a documented reason. Done means log entries from a test session appear in the target topic or sink with the consumer identity attached.
Test with a client through the gateway
Connect a test client that supports the Kafka protocol through the WebSocket route, produce a message to a test topic, and consume it back. Confirm that the client uses only the gateway endpoint and that the broker addresses it sees are the ones your design expects. Check that the authentication and rate limit behave as configured for a second, unauthorized identity. Done means produce and consume both succeed for the authorized identity, and the rejected identity fails at the expected point.
Verify broker-side ACLs for each identity
On the Kafka side, confirm that the SASL identity used by the gateway holds only the topic and group permissions it needs. In KRaft clusters the StandardAuthorizer keeps ACLs in the cluster metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true, so test a topic with no ACL to confirm it is denied. Remember that the gateway identity may be shared by many clients, so consider whether per-client ACLs are needed. Done means denied and allowed operations match your written access matrix in a test run.

When native Kafka clients are the better route

Clients should speak the Kafka protocol directly when they run inside the network and can hold their own credentials. Server-side services, stream processors and batch jobs fall into this group. Their consumer groups, partition assignment and offset commits are part of how they process data correctly, and the gateway path adds a WebSocket hop that those clients do not need. The kafka-proxy plugin is designed for clients that connect to a route over WebSocket and sign in with SASL/PLAIN. It is the wrong tool when a service needs the full Kafka client behavior for its own processing guarantees.

High-throughput producers and consumers are the second case. Each message crossing a WebSocket route adds framing and gateway processing, and a route-level rate limit applied at the edge can throttle a bulk pipeline in ways that the pipeline's own backpressure would handle better. Before choosing a design, load test the exact route with your message sizes, partition counts and batch settings, and measure throughput, latency and gateway CPU. Confirm with your APISIX version whether the features your clients depend on, such as transactions, compression, idempotent producers and specific SASL mechanisms beyond PLAIN, pass through the path you plan to use. The kafka-proxy documentation sets the scope you can rely on, so check it against your client library versions.

Treat these trade-offs as hypotheses until a proof of concept confirms them. Ask the Kafka team to list every client feature in use, then test each one through the gateway in a non-production environment, including rebalances, offset resets and consumer failover. Keep the Kafka listeners and ACLs in place for internal clients regardless of the edge design, because the StandardAuthorizer denies resources without ACLs unless allow.everyone.if.no.acl.found is set. Where a partner or external system needs Kafka access, the gateway route is the controlled entry point, and internal services can keep native access under their own identities.

Security and operational considerations for Kafka behind a gateway

A gateway in front of Kafka adds one more place where credentials, TLS and connection state can fail. The cards below cover the controls that connect the edge to the brokers, and the checks to run before a client reaches a topic.

TLS on every hop

Client traffic to the gateway, the gateway to the brokers and the broker listeners each need an explicit TLS decision. Kafka security is optional, so a listener can be left plaintext without anyone noticing. Confirm which listeners are encrypted, which certificates the gateway trusts for the brokers, and that no plaintext listener is reachable from outside the broker network.

Choosing the SASL mechanism

SASL/PLAIN at the gateway sends a username and password that the gateway must hold, so the gateway becomes a credential store for every client. SCRAM-SHA-256 or SCRAM-SHA-512 to the brokers, or OAUTHBEARER where your identity provider issues tokens, keeps the broker-side secret out of the edge. Decide which hop uses which mechanism before writing the route, and confirm the broker listeners accept it.

Credential rotation

The kafka-proxy plugin stores the SASL password encrypted in etcd, so rotating it is a configuration change that must reach every gateway node. Plan the order: add the new credential on the broker side, update the route, then retire the old one. Ask who owns each credential, how rotation is audited, and how long clients can keep using the previous secret during the change.

KRaft ACLs and the default behavior

In KRaft clusters the StandardAuthorizer stores ACLs in the metadata log. Resources without an ACL are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Check that this setting is false in production, that each gateway identity has explicit read or write grants on its topics and consumer groups, and that a new topic does not open to everyone by default.

Logging without leaking payloads

Gateway access logs and the kafka-logger plugin record request and response data, which can include message bodies, headers or identifiers that are regulated. Decide which fields are logged, mask or drop the rest, and confirm where the log stream lands and who can read it. Treat the log destination as a data store with its own retention and access rules.

Monitoring long-lived connections

WebSocket consumers hold connections open for long periods, so a gateway can accumulate idle or stalled sessions that consume file descriptors and memory. Track open connections per consumer identity, connection duration, and reconnect rates. Set limits on connections per client and confirm how the gateway behaves when a broker restarts and clients must reconnect.

Questions buyers ask before exposing Kafka through APISIX

Do we need a consultant to expose Kafka through APISIX?

Not always, but most teams benefit from outside help on the parts they have not run before. The effort depends on how many topics and client groups need access, whether clients use the Kafka protocol directly or need a WebSocket path, how identities are issued, and how many environments must match. A consultant adds most value on the design decisions: which clients should go through the gateway, how credentials are stored and rotated, and where rate limits sit. Ask any provider for a written scope listing the routes, clients, identity integration and test cases before comparing quotes.

What does the work need from our own team?

Your team needs to provide the Kafka cluster details, the list of topics and consumer groups each client should reach, the identity provider or credential process, and a named owner for approvals. Engineers who run Kafka should confirm the listener setup and the authorization model, since APISIX does not replace it. Security staff should review how SASL/PLAIN credentials are stored and who can change routes. Plan time for a test environment that mirrors production clients, because most integration issues appear only with real client libraries.

Can our own engineers build this in-house?

Yes, if your team already runs APISIX and Kafka and has someone who understands both protocols. The APISIX kafka-proxy plugin is documented for SASL/PLAIN on Kafka upstreams with the WebSocket client connection, so the configuration itself is small. The harder work is testing: confirming that native clients, consumer groups and offsets behave correctly through the proxy, and that the gateway does not become a bottleneck. Teams new to either system often underestimate the verification and operations work, which is where an outside partner usually helps.

How long does an exposure project take?

Duration depends on scope and environment, so no fixed figure applies. The main drivers are the number of client types, whether you need WebSocket consumers, how many identity sources are involved, whether rate limits and logging must be tuned per client, and how many environments must be validated. Air-gapped networks, change-control windows and multiple data centers add time. Ask for a timeline tied to specific milestones such as design sign-off, test completion and cutover, and confirm which items the client team must deliver for each milestone.

Do Kafka version upgrades change the gateway design?

They can. Apache Kafka 4.0 supports only KRaft mode and removed ZooKeeper mode, and ZooKeeper-based clusters must migrate to KRaft before upgrading to 4.x. The Kafka upgrade notes also recommend that KRaft clusters older than 3.3.x move to 3.9.x before 4.x. Rolling upgrades are supported from each release in the past year to the latest version. Each upgrade may change client expectations, authorization behavior and Java requirements, so the gateway route, authentication settings and test suite should be reviewed against the target Kafka release before the upgrade window.

Can APISIX replace Kafka's own security controls?

No. APISIX can authenticate clients at the edge and add rate limits, but Kafka still needs its own controls. Kafka supports SSL/TLS encryption and SASL mechanisms including GSSAPI, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER. In KRaft clusters, the StandardAuthorizer stores ACLs in the metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Keep broker-side authentication and ACLs in place, and treat the gateway as one layer in a defense-in-depth design that your security team reviews.

When are native Kafka clients the better choice?

Native clients are usually the better choice for internal services that run on the same network as the brokers and need full protocol features, consumer group management and predictable offsets. Routing them through a gateway adds a hop, another component to operate and another place where protocol behavior must be verified. The gateway fits external partners, browser or WebSocket consumers, and teams that need one edge policy for many clients. Ask your architects to document which clients use each path before building anything.

Limits to confirm in your environment

Several parts of a Kafka gateway design depend on details that are specific to your APISIX release, Kafka version and network path. The cards below list the items to verify before committing to a route design, so that compatibility, connection behavior and security assumptions are tested rather than assumed.

kafka-proxy client compatibility

The kafka-proxy plugin carries Kafka traffic to clients over WebSocket with SASL/PLAIN on the upstream side. Confirm which Kafka client libraries, versions and request types your applications use actually work through the route, including producers, consumers, consumer group coordination and admin calls. Test each one in a non-production environment, since protocol coverage is not the same as a general statement that Kafka is supported.

WebSocket connection limits and timeouts

Every Kafka client session becomes a long-lived WebSocket connection through the gateway. Confirm idle timeouts, maximum connection duration, and how many concurrent sessions your APISIX nodes, load balancers and any upstream proxies will hold open. Consumers that poll slowly or sit idle can be dropped by an intermediate hop, which shows up as rebalances or reconnect storms on the Kafka side.

Throughput under load

Gateway encoding, WebSocket framing and the extra hop add latency and consume CPU and memory on the APISIX nodes. Measure sustained produce and consume rates, message sizes and batch behavior with your own traffic profile, and watch tail latency and error rates. Do not carry over throughput expectations from direct broker connections without a load test on your hardware.

APISIX release in use

The APISIX downloads page lists 3.19.0 as the latest release, dated September 28, 2026, and does not mark any release as long-term support. Pin the exact version you deploy, read its release notes for kafka-proxy and WebSocket changes, and plan how you will test and roll out future upgrades. Confirm the support window with the project before you commit to a version for a regulated service.

Kafka 4.x upgrade requirements

Kafka 4.0 and later run only in KRaft mode, so ZooKeeper-based clusters must migrate before upgrading. The upgrade notes also recommend moving KRaft clusters older than 3.3.x to 3.9.x first, and brokers require Java 17 from Kafka 4.0. Confirm your current cluster mode, broker Java runtime and rolling upgrade path against the version you plan to run, since the gateway design should not block a broker upgrade.

SASL mechanism support on each hop

The client side, the APISIX route and the Kafka broker listener each have their own authentication settings. Kafka documents GSSAPI, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER, but kafka-proxy uses SASL/PLAIN toward the upstream. Confirm which mechanism each hop supports, whether TLS protects the credentials on every segment, and how secrets stored in etcd are rotated and access-controlled.

Keeping APISIX and Kafka versions aligned

Start by confirming which Kafka line your brokers run, because the gateway should be tested against a supported release. The Apache Kafka downloads page lists 4.3.1 (June 25, 2026), 4.2.2 (September 29, 2026) and 4.1.2 (March 17, 2026) as the supported releases, and all earlier versions are archived. The project aims for three releases a year and issues bugfix releases only for supported lines, so a cluster that is current today can drop out of support within a year. Rolling upgrades are supported from each of the last three releases to the latest version. If your brokers are older than that window, plan intermediate upgrade steps before you touch the gateway configuration.

Broker prerequisites usually decide the upgrade order. Since Kafka 4.0 only KRaft mode is supported, so any ZooKeeper-based cluster must migrate to KRaft before it moves to a 4.x release. The upgrade notes also recommend that KRaft clusters older than 3.3.x first move to 3.9.x. Brokers on 4.x require Java 17. Confirm these gates with your broker team before scheduling work, and record the target version for each cluster so the APISIX team builds routes against the same line the brokers will run after the change.

Test both sides after every upgrade, whether the broker or the gateway changed. On the Kafka side, check that the SASL/PLAIN listener still accepts the credentials the kafka-proxy upstream uses, that TLS settings are unchanged, and that ACLs still allow the intended principals, since the StandardAuthorizer blocks resources without ACLs unless allow.everyone.if.no.acl.found is set. On the APISIX side, run a produce and consume test through the WebSocket route, confirm that the encrypted password in etcd still resolves, and check rate limits and kafka-logger batching. APISIX 3.19.0 was released on September 28, 2026, so upgrade the gateway on its own schedule and repeat the same tests after each change.

Primary sources for the kafka-proxy and kafka-logger plugins

This sequence sets the reading order for the official documentation behind a Kafka edge design. It starts with the APISIX plugin pages, then covers the Kafka references that govern the broker side. The output is a short list of confirmed settings and version decisions that the build will depend on.

Read the kafka-proxy plugin documentation
The engineer who owns the gateway reads the APISIX kafka-proxy plugin page first, because it defines how the route authenticates to Kafka. Confirm that the upstream uses the kafka scheme, that SASL/PLAIN is the credential mechanism at the gateway, and that clients connect to the route over WebSocket. Note where the password is stored, which the page describes as encrypted in etcd. The step is done when each open question is logged in the design notes.
Review the kafka-logger plugin settings
Review the kafka-logger page if request and response logs will be sent to a Kafka topic. The page describes JSON logs sent in batches, by default every 5 seconds or at 1,000 entries, so confirm those values suit your log volume and tolerance for delay. Check the TLS and SASL options it supports, including PLAIN, SCRAM-SHA-256 and SCRAM-SHA-512, against the mechanisms your brokers accept. The step is complete when every logger setting has a named owner and an agreed value.
Confirm the target Kafka release line
Open the Apache Kafka downloads page and identify which releases are currently supported. The page lists 4.3.1, 4.2.2 and 4.1.2 as supported, with earlier versions archived. Choose the line you intend to run, and check its support status again before the plan is approved, since the list changes as new releases ship. The step is done when the target version and its support status are written into the plan.
Check the upgrade notes for your version
Read the upgrade notes for the release you plan to run, using the Kafka 4.3 upgrade notes. Kafka 4.0 supports only KRaft mode and removed ZooKeeper mode, so any ZooKeeper-based cluster must migrate to KRaft before it moves to 4.x. The notes also recommend that KRaft clusters older than 3.3.x first move to 3.9.x. Record the Java requirements, which are Java 17 for brokers and at least Java 11 for clients, and confirm that your client applications meet them.
Select the broker listener and SASL mechanism
Use the Kafka security overview to decide how traffic reaches the brokers. Kafka documents SSL and SASL, with GSSAPI, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER as its SASL mechanisms. Security is optional in Kafka, so a cluster can mix secured and unsecured listeners, which means you must choose the internal listener deliberately. Document the listener map and the TLS and SASL settings for the leg between the gateway and the brokers. The step is complete when the listener map is approved by the platform owner.
Map ACLs for the gateway service identity
Read the Kafka authorization and ACL documentation to define what the gateway identity is allowed to do. In KRaft clusters, the StandardAuthorizer stores ACLs in the cluster metadata log, and resources without ACLs are limited to super users unless allow.everyone.if.no.acl.found is set to true. List the topics, consumer groups and operations the gateway identity needs, and confirm the value of that setting in your environment. The step is done when a per-topic ACL list has been reviewed and signed off.

How DeployOpen scopes and delivers Kafka access through APISIX

DeployOpen starts with a written scope that states which Kafka clients need access, which operations they perform and which controls the gateway must enforce at the edge. We then build, test, document and hand over the APISIX routes and the Kafka access model they depend on, inside infrastructure you control.

How we scope

Inventory each client that needs Kafka access, its language, its client library and whether it can speak WebSocket.
Agree which Kafka topics and consumer groups each client may use, and which operations stay on native Kafka listeners.
Record the written scope, deliverables and timeline before any work starts, since engagements are fixed-scope.

How we build

Configure the Kafka upstream, the kafka-proxy route and SASL/PLAIN credentials, with the password stored encrypted in etcd.
Set authentication, rate limits and TLS at the edge, then test the route with representative producer and consumer clients.
Plan the Kafka side with matching ACLs in the cluster so the gateway cannot grant more access than the broker allows.

How we hand over

Document each route, consumer identity, credential rotation procedure and the rollback steps for the gateway configuration.
Walk your platform and security teams through the test results, known limits and the decision record for the proxy approach.
Deliver configuration in a form your team can review and version, so changes follow your own change process.

How we support

Offer ongoing support, upgrades and continuous testing as a retainer after handover, scoped in writing.
Plan APISIX and Kafka version upgrades against the release notes of each project before changing production routes.
Confirm with your team which Kafka client versions and broker settings must be retested after each gateway change.