Apache APISIX access logs to Apache Kafka

How Apache APISIX access logs to Apache Kafka works

A SOC or analytics team that already runs Apache Kafka usually wants gateway request records in that stream, not in a separate pipeline. The kafka-logger plugin in APISIX solves that handoff: it takes each access log entry the gateway produces and publishes it to Kafka brokers. Kafka then acts as the buffer between the gateway and everything downstream, so a SIEM connector, an analytics job or a replay process can each read the same records at their own pace. The plugin can be enabled per route, per service or globally, and the choice determines which traffic reaches the stream. Global scope is the simplest way to capture everything, while per-route scope limits volume to the APIs that matter for investigation.

Kafka itself is licensed under Apache-2.0 and has no paid edition from the project, so the cost of this integration is the people and infrastructure that run it, not a license fee. The work covers four parts: configuring the plugin in APISIX, securing the connection to the brokers, creating the topics and retention the security team needs, and handing the stream to consumers with a documented schema. Each part has decisions. The plugin's batching and retry behavior affects what happens during a broker outage. TLS and SASL settings determine who can publish to the topic. The log format determines which fields a consumer can rely on.

Before the build starts, confirm three things with the team that owns the brokers: the Kafka version and deployment mode, the listener and authentication method each client must use, and the ACL model that will let the APISIX client write to its topic. Kafka's KRaft clusters restrict resources without ACLs to super users unless the cluster is configured otherwise, so a missing ACL shows up as a publish failure, not a warning. Ask for a test topic and a dedicated publishing identity before the production route is enabled.

What you need before enabling Kafka logging on APISIX

Kafka logging on APISIX depends on decisions made on both sides of the connection. The cards below cover the gateway version, the Kafka release line, broker security, topic and consumer ownership, and the network path from the data plane to the brokers.

APISIX version to run

Confirm the APISIX release your policy allows before writing any logger configuration. The downloads page lists 3.19.0 as the latest release, and it does not mark any release as long-term support. If your change process requires an LTS line, ask your team which release line it will track and how patch releases will be applied to the data plane nodes.

Kafka on a supported release line

Run the cluster on a release line the Kafka project still supports. The downloads page lists 4.1.2, 4.2.2 and 4.3.1 as supported, and all earlier versions are archived. Pick the line that matches your upgrade cadence, since rolling upgrades are supported from the last three releases to the latest version.

KRaft mode for 4.x upgrades

Kafka 4.0 and later run only in KRaft mode, and ZooKeeper mode has been removed. If your existing brokers still use ZooKeeper, plan the migration to KRaft before any upgrade to 4.x. Clusters older than 3.3.x should first move to 3.9.x, according to the upgrade notes. Confirm the mode and version of every broker before you agree on a logging design.

Broker listeners with TLS and SASL

Decide which listener the APISIX data plane will use and how it authenticates. Kafka supports SSL and SASL, and the documented SASL mechanisms include SCRAM-SHA-512, SCRAM-SHA-256, PLAIN, OAUTHBEARER and GSSAPI. Ask your Kafka owner for the bootstrap addresses, the CA chain for broker certificates, and the credential type for the gateway client, and store those credentials where APISIX can reference them.

Topic names and ACLs

Agree on the topic name, partition count and retention for gateway logs before the first request is logged. In KRaft clusters, the StandardAuthorizer stores ACLs in the metadata log, and resources without an ACL are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Grant the APISIX client write access to the log topic only, and confirm that no other service can write to it.

Topic and consumer group ownership

Name one team that owns the log topic, its retention and its consumer groups before go-live. The SIEM or analytics team usually reads the topic, while the platform team owns the producer configuration on APISIX. Write down who approves changes to batch size, log format or body logging, because each of these changes what downstream systems receive and how much data they store.

Network path from data plane to brokers

Verify that every APISIX data plane node can reach each Kafka broker address on the listener port, including across firewalls, security groups and any DNS names used for advertised listeners. Test from the gateway host itself, not from a workstation, because a path that works from an administrator laptop can still be blocked for the gateway. Record the path in your change request so network reviewers can approve it.

Configuring the kafka-logger plugin step by step

This sequence takes a gateway from no log shipping to a verified stream of APISIX access logs on a Kafka topic, built and tested on one non-production route first. Each step ends with a check that the broker, topic and record format behave as agreed before production traffic is changed.

Create the topic and service identity
Your Kafka administrators create a dedicated topic for gateway logs, sized and retained according to the agreed retention policy, and a service account for APISIX. They grant that account only the ACLs it needs, which normally means write access to the log topic and nothing else. Kafka's authorization model restricts resources without ACLs to super users unless allow.everyone.if.no.acl.found is set to true, so confirm the account can write before going further. Check: a test produce using the service account's credentials succeeds, and a produce with a different identity is refused.
Enable kafka-logger on a non-production route
Your platform engineers add the kafka-logger plugin to one non-production route through the Admin API or your configuration-as-code tooling, leaving all other routes untouched. Keep the initial scope narrow so that any log volume, format or connectivity issue affects only test traffic. Confirm the plugin's configuration keys against the plugin documentation for the APISIX version you run, since keys and defaults can change between releases. Check: the route returns the same responses as before the change, and the Admin API shows the plugin attached to that route only.
Set broker addresses and the topic
Enter the broker list and the target topic name in the plugin configuration, using the broker hostnames your TLS certificates are issued for. Use more than one broker address so a single broker outage does not stop log delivery. Keep the batching settings at the plugin defaults for the first run and record them, so later tuning is a deliberate change. Check: the APISIX error log shows no connection failures to the brokers after a request is sent through the test route.
Define the log format
Agree the fields your SIEM or analytics team needs, such as timestamp, client address, method, URI, status, latency and route identifier, and write them into the plugin's log format setting. Leave out request and response bodies and any header that carries credentials, tokens or personal data unless your data protection review has approved them. Body logging also has limits that you should confirm in the plugin documentation before enabling it. Check: a sample record from the test route contains exactly the agreed fields and none of the excluded ones.
Enable TLS and SASL to the brokers
Turn on TLS for the broker connection and configure SASL with the mechanism your cluster accepts, such as SCRAM-SHA-512 or OAUTHBEARER, using the service account created earlier. Kafka's security is optional, so a listener without these controls will still accept the connection, which means you must confirm the broker listener you connect to requires them. Store the credentials in the secret handling your organization uses rather than in plain configuration text. Check: the connection succeeds with the secured listener, and a deliberately wrong password is rejected in the error log.
Send test traffic and read the topic
Send a controlled set of requests through the test route, including successful calls, an authentication failure and a not-found response, so the records cover different status codes. Read the topic with a consumer group that is separate from any production consumer, using the service identity or a dedicated read-only account. Compare each record with the agreed format and confirm the number of records matches the requests sent, allowing for batching delay. Check: every test request appears as one record with the expected fields, and no unexpected body content or credentials are present.
Promote the configuration to production routes
Once the test route passes every check, apply the same plugin configuration to production routes through your normal change process, starting with the lowest-risk route. Record the broker, topic, format and security settings in version control so the configuration can be reviewed and rebuilt. Set up monitoring on the topic's lag and on APISIX error logs before the rollout is complete. Check: records arrive for production traffic, the downstream SIEM or analytics pipeline reads them, and the rollback steps have been tested on the test route.

Which access log fields reach Kafka and how they map to SIEM fields

This table compares the common APISIX access log fields against the message fields a Kafka consumer sees and the SIEM or analytics column each one typically feeds. Read the decision column as a starting point for your data classification review, not a finished schema. The main takeaway is that headers and request bodies are not sent unless you select them, and that every field you keep becomes data your Kafka retention and access controls must protect.

Field	Typical SIEM or analytics mapping	Decision and notes
Client address (remote_addr)	source.ip or client IP column	Keep. Confirm whether your load balancer or trusted proxy header is the value you want, since forwarded addresses need a header trust decision first.
HTTP method	http.request.method	Keep. Low sensitivity and useful for detection rules on unexpected verbs.
Request path (uri)	url.path or request target	Transform. Strip or mask identifiers, tokens or email addresses embedded in path segments, and confirm query strings are handled separately.
Query string	url.query	Drop by default. Query strings often carry tokens, reset codes or personal data. Keep only named parameters you have approved.
Response status	http.response.status_code	Keep. Required for error-rate and authentication failure detection.
Latency and upstream timing	event.duration or custom latency field	Keep. Confirm which timing variables your APISIX version exposes in the log format before mapping them.
Route identifier	Custom gateway route field	Keep. Map route IDs to route names in your SIEM so detections survive route renumbering.
Consumer identifier	user.name or API client identifier	Transform. Use a stable consumer name, not an API key or secret. Confirm that no credential value is written into the consumer field.
Request headers and body	Not mapped to standard fields	Drop unless explicitly selected. Authorization, cookie and custom token headers must be excluded. Body logging needs a size limit and a review of personal and payment data.

Batching, body limits and log formats decide what the SIEM receives

Batching is the setting that most directly trades delivery latency for throughput. The kafka-logger plugin groups log entries before sending them to the brokers, so a larger batch or a longer flush interval reduces produce requests but delays when events reach the SIEM. Do not copy numbers from a blog post. Read the batch size, flush timing and retry settings from the plugin documentation for the APISIX version you deploy, then test them against your measured request volume. Ask the implementation team to show, in your environment, how many events sit in the buffer at peak and what happens to that buffer when a broker is unreachable. A dropped batch is lost data unless you have designed a second path for it, and that decision belongs in the scope before the build starts.

Request bodies deserve the most caution. Logging them is a deliberate choice that needs a size limit and a written justification, not a default. For regulated data such as cardholder numbers, health records, credentials or personal identifiers, the safer position is usually to log metadata only: method, route, status, timing, consumer identifier and a request correlation ID. If a body must be captured for troubleshooting, set a strict maximum size, restrict which routes qualify, and confirm with your privacy and compliance owners that the captured content is allowed to reach Kafka and every system downstream of it. Headers such as authorization need the same review, because a single log line can carry a bearer token into a topic that many teams can read.

A custom log format controls exactly which fields and variables appear in each record, so it is the place where the SIEM contract is defined. Choose the NGINX variables and APISIX fields your detection rules need, such as timestamps, client address as seen after your trusted proxy configuration, route, upstream status and latency, and keep the field names stable across releases so parsers do not break. Remove anything your rules do not use. Before go-live, produce sample records from a staging route, have the SIEM team confirm each field maps to a column or parser, and check the output for secrets. Review the format again whenever the APISIX version or the plugin configuration changes.

Securing the broker connection and the log content

A gateway log stream carries client identifiers, paths and sometimes header or body content, so the path to the brokers and the fields written into each message both need deliberate choices. The cards below cover the transport, authentication, authorization and redaction decisions, and the checks that confirm each one is in place before production traffic flows.

Encrypt the broker connection

Kafka encrypts traffic between clients, brokers and tools with SSL/TLS, but security is optional, so a cluster can run unsecured or mix secured and unsecured listeners. Configure the APISIX side to connect over TLS and enable certificate verification against the broker's certificate authority, so the gateway does not send logs to a server it cannot identify. Confirm with your Kafka operators that every listener the gateway can reach is a secured one, and that no plaintext listener is reachable from the gateway network.

Choose a SASL mechanism

Kafka supports the SASL mechanisms GSSAPI (Kerberos), PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER. Pick the one your identity estate already operates, and confirm that the APISIX kafka-logger plugin supports that mechanism in the version you deploy before committing to it. SCRAM and OAUTHBEARER avoid sending a reusable password in the clear on each connection, while PLAIN relies entirely on the TLS layer for protection.

Store broker credentials outside plain config

The SASL username and password should not sit in a readable route or plugin definition that many administrators can export. Check whether your APISIX version lets you reference secrets from an environment variable or an external secret manager for these fields, and confirm how the Admin API returns stored plugin configuration. Use a separate Kafka identity for each gateway environment so a rotation or revocation affects one stream rather than all of them.

Scope ACLs to the logging topic

In KRaft clusters, the StandardAuthorizer stores ACLs in the cluster metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Grant the gateway identity write access only to the log topic it produces to, and avoid cluster-wide or wildcard permissions. Leave allow.everyone.if.no.acl.found unset in production so that new topics stay closed by default, and test the ACL with a produce request before go-live.

Redact sensitive fields in log format

The custom log format determines what reaches the topic. Remove Authorization headers, API keys, cookies, query parameters that carry tokens and any request body fields holding personal or payment data before they are written. Review the default fields against your data classification, and test the format with a request that carries a known dummy token to confirm the value does not appear in the produced message.

Limit body logging content

Logging request or response bodies multiplies the volume of sensitive data in the stream and the load on the brokers. Enable body capture only for routes where you have a documented need, and set the size limits so that large payloads are truncated rather than forwarded in full. Ask your security team to record which routes capture bodies, since downstream SIEM users will be able to search that content.

How the setup changes for SIEM teams and analytics teams

The SIEM team reads the gateway log stream as a feed of security events, so stable field names and delivery timing matter more to it than the exact message layout. Decide these before the gateway starts publishing, because a field rename later can break correlation rules and dashboards that depend on it.

Stable field names: Agree on the request fields that detection rules use, such as client identity, route, status code and source address. Treat any rename as a breaking change that needs advance notice and a rule review.
Topic retention for replay: Set retention long enough that the topic can refill the SIEM after an outage. The period is a decision for your team, based on your recovery objective and storage budget, so confirm it with the SIEM operators.
Alert latency expectations: Define how far the SIEM may lag behind a gateway event before an alert counts as late, then monitor consumer lag against that limit.
Redaction before the topic: Decide which headers, query parameters and body fields must never reach the topic, since every consumer and every replay copies whatever is written there.
Consumer ownership: Confirm whether the SIEM reads with its native Kafka integration or through a relay, and who owns the consumer group and its credentials.

Before relying on the replay path, check that replaying a day of gateway logs into a test SIEM produces the same alerts as the original run.

The platform team owns the Kafka cluster, the topic and the gateway's producer credentials, so it makes the decisions that both consumers depend on. Kafka security is optional and can be set per listener, and resources without an ACL are restricted to super users by default, so access has to be designed explicitly before either consumer connects.

Transport and authentication: Plan TLS on the broker listeners and a SASL mechanism for the gateway producer. The documented options include SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER, so pick one that your identity stack supports.
Separate principals: Give the gateway producer, the SIEM consumer and the analytics consumer their own identities and ACLs, so access can be revoked for one consumer without affecting the others.
Default-deny ACLs: In KRaft clusters, ACLs are stored in the metadata log, and resources without ACLs are limited to super users unless allow.everyone.if.no.acl.found is set to true. Leave it at the default unless the team has decided otherwise.
Kafka version: Kafka 4.x runs only in KRaft mode, and the supported releases are listed on the downloads page. Confirm the target version before the topic design is frozen.
Plugin settings: Confirm the batching, TLS and SASL setting names, along with body logging limits, in the kafka-logger documentation for your APISIX version.

Test each principal with a produce or consume attempt on the topic, and confirm that identities without a grant fail.

Common failures when APISIX logs stop reaching Kafka

This table lists the failures that most often stop gateway logs from reaching Kafka, with the layer where each one lives and the fix to try first. Read it in order when logs stop, because the first failing layer is usually the network path or the TLS handshake, and later layers only matter once the connection succeeds.

Failure	Likely cause	Fix and where to apply it
Broker unreachable from the data plane	Security groups, firewall rules or DNS on the APISIX nodes do not allow the broker address and port, or the broker advertises a hostname the gateway cannot resolve.	Confirm from each APISIX node that every advertised broker host and port is reachable. Fix the network rule or DNS record, then check the broker's advertised listeners so clients get a resolvable address.
TLS certificate name or trust chain mismatch	The broker certificate does not include the advertised hostname, or the CA bundle configured for the plugin does not contain the issuing CA.	Reissue the broker certificate with the correct subject alternative names, or add the issuing CA to the trust store the plugin uses. Verify the chain from an APISIX node before changing plugin settings.
SASL credentials rejected	The username, password or mechanism in the plugin configuration does not match the broker's SASL listener, or the SCRAM user was never created.	Match the mechanism (PLAIN, SCRAM-SHA-256 or SCRAM-SHA-512) on both sides, confirm the user exists on the broker, and rotate the credential through the secret handling your team uses rather than editing it in plain text.
Topic missing or auto-creation disabled	The target topic does not exist and the cluster does not create topics on first produce, so produce requests fail.	Create the topic ahead of time with the partition count and retention your analytics team agreed. Confirm the topic name in the plugin matches exactly, including case.
ACLs deny produce by default	In KRaft clusters, resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true, so the gateway's principal gets denied.	Grant the gateway principal Write (produce) permission on the topic through the StandardAuthorizer. Leave allow.everyone.if.no.acl.found at its default in production, and grant access explicitly.
Batches dropped under broker backpressure	The broker slows or rejects produce requests, the plugin's batch buffer fills, and batches exceed retry or buffer limits and are discarded.	Size the batch and buffer settings against expected peak traffic, watch broker request latency, and add a second log destination for records you cannot lose. Review retry settings with the plugin documentation for your APISIX version.
Oversized records from body logging	Request or response body logging adds large payloads, so single records exceed the broker's maximum message size and the produce request fails.	Cap logged body size in the log format, exclude binary or upload routes from body logging, and confirm the broker's maximum message size is compatible with the largest record you expect.

Questions teams ask before streaming gateway logs into Kafka

What drives the cost of streaming APISIX access logs into Kafka?

The cost follows the amount of work across five areas: APISIX logger configuration on the routes you choose, Kafka topic and ACL design, security testing of the path from gateway to brokers, mapping log fields to your SIEM or analytics schema, and the consumer work that keeps the pipeline running. Route count, the number of log fields you keep, and how many downstream consumers read the topic all move the effort up or down. Setup is largely one-time, while consumer maintenance, topic retention tuning and upgrades recur. Ask any provider for a written scope that lists each area separately, with the deliverables for each and who owns the consumer side after handover.

How long does a first Kafka logging setup take?

Duration depends on how many routes send logs, how much Kafka infrastructure already exists, and whether topics, ACLs and certificates are ready. A team with a running Kafka cluster, an agreed log schema and a consumer waiting for data will move much faster than one that must build the cluster and design the schema first. No fixed timeline applies across environments, so ask for a schedule in the written scope that ties each milestone to a named dependency, such as broker access or certificate issuance. Pilot one route before rolling out to the rest of the gateway.

What do we need to provide before the work starts?

The work needs several inputs from your side before configuration begins:

Network access from the gateway to the Kafka brokers, including the listener addresses and ports
TLS certificates or the CA chain the brokers present, and a decision on mutual TLS
SASL identities for the gateway, such as a dedicated user with the mechanism your cluster uses, plus the matching ACLs
A log schema decision covering which request fields, headers and body content may be logged
A named owner for each consumer that reads the topic, including the SIEM or analytics team

Confirm each item with your Kafka and security owners before the schedule is fixed.

Which Kafka security settings should we confirm before connecting the gateway?

Confirm whether the listeners use SSL/TLS, which SASL mechanism the cluster accepts, and how authorization is enforced. Kafka security is optional, so a cluster may expose unsecured or mixed listeners, and the gateway should only use a secured listener. In KRaft clusters, the StandardAuthorizer stores ACLs in the metadata log, and resources without ACLs are limited to super users unless allow.everyone.if.no.acl.found is set to true. That default means a new gateway identity may be blocked until you grant it explicit write access to the log topic. Check these settings in your cluster's configuration with your Kafka administrators.

Does the Kafka version affect the logging design?

Yes. Apache Kafka 4.0 removed ZooKeeper mode and supports only KRaft, so a ZooKeeper-based cluster must migrate to KRaft before it moves to 4.x. The project's upgrade notes also recommend that KRaft clusters older than 3.3.x upgrade to 3.9.x before moving to 4.x. Brokers and Kafka Connect require Java 17 from 4.0, and clients require at least Java 11. The Kafka downloads page lists 4.3.1, 4.2.2 and 4.1.2 as supported releases. Confirm which version your cluster runs and whether an upgrade is needed before adding a new producer.

Can our internal team build this without a partner?

Yes, if the team has experience with APISIX plugin configuration, Kafka administration, TLS and SASL, and the SIEM or analytics platform that receives the events. The gap usually sits in the security review of the gateway-to-broker path, the log schema decisions and ongoing consumer work. Teams often find that the hardest part is not the plugin settings but agreeing on which fields may be logged. Decide early whether the internal team, a partner or a mix will own each area, and record that split in writing.

What ongoing work follows the first deployment?

After the first deployment, the recurring work includes monitoring producer errors and consumer lag, tuning retention and partitions as log volume grows, rotating SASL credentials and certificates, updating ACLs when consumers change, and planning Kafka and APISIX upgrades. Each consumer also needs an owner who handles schema changes when the log format evolves. Ongoing support can be handled by your team or through a retainer, depending on what the written scope assigns to each party. Ask the provider to state which of these tasks transfer to your team at handover.

How DeployOpen scopes and delivers APISIX to Kafka logging

DeployOpen plans, builds, documents and hands over the Apache APISIX access log pipeline into Kafka inside infrastructure you control. Each engagement starts from a written scope that names the topics, brokers, security settings and log format your downstream team needs.

How we scope

Inventory which routes, consumers and upstreams must emit access logs to Kafka and which should stay out.
Agree the Kafka topic layout, broker list and the authentication method your cluster already uses.
Record the log fields the SIEM or analytics team needs, and which body content must be excluded from logs.

How we build

Configure the kafka-logger plugin on a test gateway, with batching and retry values set in the written design.
Enable TLS and SASL to the brokers, with credentials stored as references rather than plain text in configuration.
Test payload size limits, custom log formats and message flow against a non-production Kafka cluster before rollout.

How we hand over

Deliver the plugin configuration, log format definition and broker connection settings as versioned files.
Document the verification steps, expected message samples and the failure behavior your operations team should watch for.
Walk your platform, security and data teams through the topics, consumers and rollback steps in a handover session.

How we support

Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
Ongoing support, plugin upgrades and recurring checks of the logging path are available as a retainer.
Confirm with your Kafka administrators which Kafka version and security settings your clusters run before scoping.