How Apache APISIX logs reach Splunk and Datadog
A platform team that has APISIX in front of its APIs usually needs two separate things: gateway request logs in Splunk for security and operations, and gateway metrics in Datadog for dashboards and alerting. In APISIX these travel on different plugin paths. Request logs leave the gateway through the splunk-hec-logging logger, which sends events to a Splunk HTTP Event Collector endpoint. Metrics go through the datadog plugin, which sends gateway measurements to a Datadog agent. Because the two carry different data, one does not replace the other, and each needs its own plugin configuration, its own credentials and its own owner.
Plan the split before enabling either plugin. Decide which routes and consumers generate events that Splunk must retain, which fields are needed for investigation, and which fields must be excluded so that tokens, API keys and personal data do not reach the index. Decide also whether Datadog should receive only aggregate metrics or also log events, since Datadog log routing is a separate product decision with its own cost. Confirm the exact configuration keys, batching options and any log routing behavior against the plugin documentation for the APISIX version you run, because option names and defaults have changed across releases and this page does not restate them. The APISIX plugin documentation is the place to start that check.
Before go-live, test each path with a real request that carries a known test value in a header and a query string, then confirm in Splunk and Datadog exactly what arrived. Confirm the behavior when the Splunk endpoint or Datadog agent is unreachable, since logger batches queue inside the gateway and the queue limits and drop behavior determine whether events are lost during an outage. Ask the vendor or your own platform team to confirm the retention and ingest volume you expect from each path, because ingest cost in both products scales with what you send, and that volume is the main lever you control.
What you need before connecting Apache APISIX to Splunk and Datadog
Splunk and Datadog connections fail for mundane reasons: an unpinned gateway version, a blocked egress path or an untrusted certificate chain. Confirm the items below before you touch the logger or metrics configuration, so the first test sends real entries rather than troubleshooting the platform.
Pinned APISIX version and support policy
The APISIX downloads page lists 3.19.0 as the latest release and does not mark any release as long-term support. Pick the version your team will run, pin it in your deployment tooling, and record who approves upgrades. Logger options and backlog behavior differ between releases, so the version you test against must be the version you run in production. Confirm the exact version your plugin documentation matches.
Deployment mode and plugin config source
Confirm whether the gateway runs in traditional, decoupled or standalone mode. In traditional and decoupled modes, logger and metrics plugins are stored in etcd and applied through the Admin API. In standalone mode, the same settings live in the YAML or JSON configuration file. Your change process should match that source of truth, or a later sync will silently revert your logging settings.
Splunk HEC endpoint, token and index
Ask the Splunk team for the HTTP Event Collector URL, a token scoped to the target index, and confirmation that the token allows that index. Agree the sourcetype and index naming up front, because searches, retention and access control in Splunk all depend on them. Confirm whether HEC is served through a load balancer and which ports the platform team has opened for it.
Datadog account, agent or API route
Confirm which Datadog site your organization uses and how the gateway will reach it. The datadog plugin sends metrics to a DogStatsD endpoint, typically a local Datadog Agent, so verify that the agent is installed on the gateway host or reachable from it and that its API key is provisioned. Ask the Datadog administrator who owns the key and how it is rotated.
Network egress and firewall rules
The gateway needs outbound access to the HEC port and the Datadog agent or endpoint. In a restricted or air-gapped network, request explicit rules from the network team for each source address, destination, port and protocol. Gateway nodes often sit in a DMZ or private subnet, so confirm the route, proxy settings and DNS resolution from the actual gateway hosts, not from a workstation.
TLS trust chain for the HEC endpoint
Before you enable TLS verification, confirm that the gateway trusts the certificate chain presented by the HEC endpoint. Internal Splunk deployments often use a private certificate authority. Export the chain, install the root and intermediate certificates on the gateway, and verify with a test connection from the gateway host. Disabling verification to get a quick result leaves log credentials exposed to interception.
Configuring the splunk-hec-logging logger in sequence
This sequence takes APISIX from a Splunk HTTP Event Collector token to verified events and metrics in Splunk and Datadog, with a check at the end of each step so the team knows the path works before it reaches production routes.
Create the HEC token and target index
The Splunk administrator creates an HTTP Event Collector token and a dedicated index for gateway events, then limits the token to that index. Record the collector endpoint URL and confirm the token is enabled. The step is complete when a test event sent with curl from an APISIX host is accepted by the collector and the event appears in a search of the new index. Keep the token out of shared tickets and change files at this point.
Store the token as a secret reference
The platform engineer places the HEC token in the secret store the gateway team already uses, such as HashiCorp Vault, a cloud secret manager or environment variables, and references it from the plugin configuration instead of pasting the value into the route. Confirm which reference format your APISIX version supports for secrets in plugin fields. The check is a successful reload of the test route with the reference resolved, plus an Admin API read of the route that shows the reference and not the token value.
Enable splunk-hec-logging on a test route
The gateway engineer adds the splunk-hec-logging plugin to one non-production route that has a known upstream, leaving other routes untouched. Use the Admin API or the configuration file your deployment uses. Do not attach the plugin globally during this test. The check is that the route returns its normal response code to a request and the APISIX error log shows no plugin initialization failure after the change is applied.
Set endpoint, batching and timeout options
The engineer sets the HEC endpoint, the index and source fields, and the batching and timeout options as they are documented for the APISIX version in use. Confirm the exact parameter names, defaults and the backlog behavior for that release in the plugin reference and the changelog before copying values from older examples. Record the chosen values in version control. The check is a configuration diff that matches the reviewed values and a route that still passes a health request after reload.
Send a test request and find the event in Splunk
The tester sends a request with a recognizable header or path marker to the test route, then runs a Splunk search limited to the target index and the time range of the request. Compare the fields in the event with the request, and confirm that the sensitive header values and query parameters are absent or masked according to the log format in use. The check is one event with the expected route, status and marker, and no credential material in the fields.
Expand the logger to production routes
After the test passes, the team applies the logger to the agreed production routes in a planned change, using the same reviewed configuration. Watch the APISIX error log and the Splunk ingest volume for the first period after the change, and confirm that the event count per route matches the traffic the team expected. The check is a Splunk search that shows events from each in-scope route and no growing backlog messages in the gateway log.
Enable the datadog plugin and confirm metrics
The engineer enables the datadog metrics plugin on the same gateway, pointing it at the Datadog agent or endpoint your organization uses, and generates a few requests to the test route. Confirm the plugin options and the metric names against the plugin documentation for your APISIX release. The check is a Datadog metric query that returns the gateway request metrics for the test route within a few minutes of the requests, with the expected tags.
Mapping APISIX log fields to Splunk events and Datadog metrics
This table maps the core APISIX log fields to Splunk event fields and shows how datadog plugin metrics line up with Datadog metric names and tags. Use it as a starting point, then confirm each name against a real test event from your own gateway before building searches or dashboards.
| Field or metric | Splunk event field and index choice | Datadog metric name and tags |
|---|---|---|
| Request method and URI | Event field from the log entry, such as request method and URI. Use a dedicated index for gateway access events so retention can differ from application logs. | No direct metric. Use route or service tags to slice request volume, not the full URI, to limit cardinality. |
| HTTP status | Response status field. Set a sourcetype per log format so parsing stays stable across events. | Status code or class tag on request metrics. Confirm which status tags the plugin emits in your version. |
| Upstream latency | Upstream response time field. Useful for search and alerting in Splunk, though it is one event per request. | Latency metrics from the datadog plugin, if emitted by your APISIX version. Confirm the metric name in the plugin documentation. |
| Consumer name | Consumer identifier when an authentication plugin sets one. Treat as sensitive where it maps to a customer. | Consumer tag only if cardinality is acceptable. Many consumers can create high tag cardinality in Datadog. |
| Client address | Client IP field. Confirm whether it reflects the direct peer or a forwarded header, which depends on header trust settings. | Usually omitted from tags to avoid cardinality and personal data concerns. Confirm your policy. |
| Route and service identifiers | Route ID and service ID fields. Use them to join gateway events to route inventory. | Route or service tags on metrics. Confirm tag names with a test metric before dashboards depend on them. |
| Custom log format | Adding or removing fields changes which Splunk fields exist, so searches and field extractions must be rebuilt after any log format change. | Custom tags do not follow log format changes, but new tags may still change series counts. Review after each change. |
| Field name verification | Send a test request, inspect the raw event in Splunk, and record the exact field names before writing searches. | Generate a test request, check the metric in Datadog's metrics explorer, and record the exact names and tags. |
Batching and the 8,192 entry backlog limit from 3.18
Decide now what happens when the Splunk HEC endpoint is slow or unreachable, because APISIX 3.18.0 bounds logger backlogs at 8,192 pending entries. Once that limit is reached, the outcome depends on how the gateway's logger plugin handles new entries under pressure, and that behavior is a design choice your team has to make with the security and audit owners, not something to discover in an incident. Confirm the exact drop or retry behavior for the splunk-hec-logging plugin in your APISIX version against the plugin documentation and the APISIX changelog before you set alerting thresholds.
Batch size and flush timing trade latency against loss. Larger batches reduce request overhead on the HEC endpoint and smooth ingest cost, but each entry waits longer before it reaches Splunk and more entries sit in memory if delivery fails. Smaller batches shorten that delay and shrink the window of unsent data, at the cost of more outbound calls. A dropped batch is lost data for audit purposes: if the gateway recorded an authentication failure or an administrative call that never reaches Splunk, the evidence trail has a gap that the gateway cannot fill later. Record the batch parameters you choose, the expected maximum delay, and who accepts the residual risk in your logging design document.
Test with a stopped HEC endpoint before go-live. Stop the HEC listener or block the route from the gateway, drive steady test traffic, and measure how many entries are pending, when the backlog limit is reached, and what the gateway logs about dropped entries. Then restore the endpoint and confirm that the backlog drains and that the counts in Splunk match the counts at the gateway. Also check the body capture settings added in 3.18: max_req_body_size and max_resp_body_size (default 64 MiB) apply to about 19 plugins and determine how much request and response content each log entry can carry. Multiply your expected body sizes by request volume and batch size to estimate memory and ingest impact, and lower these limits where full bodies are not needed.
Keeping tokens, bodies and headers out of Splunk and Datadog
Gateway logs are useful because they record every request, and that is also why they can leak credentials and personal data into a third-party tool. The four controls below cover where secrets live, which fields are removed, which client addresses you can trust, and who can read the result.
Store the HEC token and Datadog key outside config
The Splunk HEC token and the Datadog API key are credentials that let anyone write to or read from your observability tenant. Keep them out of exported route and plugin configuration by referencing a secret store or environment variable, so a config export or a copy of etcd data does not reveal them. Check that your APISIX version resolves the reference at runtime and that the Admin API response does not echo the resolved value back to operators who should not see it.
Mask sensitive fields in custom log formats
Custom log formats decide what leaves the gateway. Do not add $http_authorization, $http_cookie, API key headers or request bodies to the format unless a documented need exists. Where you must log a body, apply the body size limits from APISIX 3.18.0 so large payloads are truncated, and confirm the exact masking or filtering options for your version before relying on them. Test the format with a request that carries a dummy token and search the Splunk index for it.
Trust client addresses after header sanitizing
Client IP fields in SIEM alerts are only as good as the X-Forwarded-For chain the gateway passes on. APISIX 3.18.0 sanitized X-Forwarded headers in the NGINX configuration, which makes logged client addresses more trustworthy. Still confirm that your load balancers in front of APISIX are configured to append rather than pass client-supplied values, and that trusted proxy ranges match your network. Alerts built on source IP should be tested against a known external test request.
Restrict log read access by role
Splunk and Datadog both support role-based access, and gateway logs often contain routing details, consumer names and upstream hostnames that only some staff need. Define roles for platform operators, security analysts and application owners, and limit index or log-view access to the data each role needs. Confirm how your Splunk edition scopes HEC-written indexes and how Datadog log restriction queries apply to the fields you ship, because misconfigured read roles are a common source of exposure.
Keep TLS verification on for HEC
Sending logs over a connection that does not verify the Splunk HEC certificate lets an attacker on the path read tokens or alter events before they are stored. Keep TLS certificate verification enabled on every logger that targets HEC or the Datadog intake, and install the internal certificate authority in the gateway trust store where Splunk uses a private CA. Confirm the verification setting for your plugin version in the plugin reference, and test the connection failure path so a certificate error is visible in the gateway error log rather than silently dropping events.
Controlling Splunk and Datadog ingest cost
The splunk-hec-logging plugin sends an event for every request on the routes where it is configured, so the first lever is deciding which routes it is attached to. Health probes, static assets and high-frequency internal calls usually add volume without adding investigation value. Body logging is the second lever. If request or response bodies are captured, ingest grows with payload size. The APISIX 3.18 changes added max_req_body_size and max_resp_body_size, with a default of 64 MiB, to about 19 plugins, so you can cap them.
- Scope by route: attach the logger to routes that carry authentication, customer-facing or partner traffic, and leave out health checks and static content.
- Cap body size: set request and response body limits well below the default on routes where the body is not needed for investigation.
- Log bodies selectively: capture bodies only on write methods or error responses if your investigations depend on them.
- Trim the log format: use a custom log format that drops headers and fields your Splunk searches never use.
Before removing any field or route, confirm with your security team which detections and searches depend on it, so cost reductions do not remove evidence analysts need.
Splunk applies ingest and retention settings by index, so routing decisions determine how long gateway events stay searchable and what it costs to keep them. Security detections usually need a longer and more searchable window than routine operational traffic. The plugin posts events to an HTTP Event Collector endpoint, while the token and its index assignment are configured in Splunk. Confirm the mapping with your Splunk administrator before the logger goes live.
- Separate security and operational events: send authentication failures, denials and administrative activity to a security index, and send routine successful traffic to a lower-retention index.
- Set retention per index: match retention to each index's purpose, and have your compliance owner approve the periods.
- Restrict token scope: limit each HEC token to the indexes it needs so a leaked token cannot write everywhere.
- Test routing in staging: send sample traffic and confirm events land in the intended index with the expected fields.
Ask your Splunk administrator how index-level retention and ingest are billed under your current agreement, and confirm that the index layout matches that model.
The datadog plugin sends APISIX metrics toward the Datadog Agent in your environment. Datadog pricing for custom metrics depends on how many unique combinations of metric name and tags you send. Tags such as route, consumer or upstream help locate problems, but each added tag value multiplies the series count. A consumer tag across a large customer base can grow the count far faster than teams expect. Check the plugin's current configuration options and the metric names it emits in the APISIX version you run.
- Limit high-cardinality tags: use route or service groupings for most dashboards, and reserve per-consumer tags for targeted investigations.
- Count series before enabling: estimate the series count per metric from your route and consumer inventory, and test in staging.
- Check agent-side filtering: confirm whether your Datadog Agent configuration filters or samples the APISIX metrics, and document the settings.
- Remove unused metrics: drop metrics that no dashboard or monitor uses.
Ask Datadog to confirm in writing how custom metrics are counted for your plan before you enable any per-consumer tag.
Cost has two parts. One-time work is the design, build and testing your team or a partner performs: scoping routes, designing the log format, mapping indexes, building dashboards and measuring volume in staging. Recurring charges come from Splunk and Datadog and depend on ingest volume, retention, indexes, custom metric counts or the licence model in your contract. Changing the gateway configuration does not change the vendor's pricing terms. This page does not provide any vendor figure, so get current pricing directly from each vendor.
- One-time implementation: route scoping, log format design, index and tag mapping, dashboard build and staging volume tests.
- Recurring ingest and retention: billed by the vendor on the volume you send and how long you keep it, so every reduction in logged bytes carries forward.
- Recurring metric charges: custom metric counts and tag combinations, set by Datadog for your plan.
- Ongoing review: periodic checks of volume, unused fields and new routes as the API estate grows.
Request each vendor's current pricing and licence or ingest model in writing, then compare it with a measured volume estimate from your staging environment.
Why logs or metrics stop arriving and how to fix them
This table lists the failures most often seen when APISIX sends logs to Splunk HEC or metrics to Datadog, with the likely cause, the check to run and the fix for each. Work down the rows in order, since a blocked egress or untrusted certificate hides every later symptom.
| Failure | Likely cause | Check to run | Fix |
|---|---|---|---|
| HEC token rejected or wrong index | The HEC token is disabled, mistyped, or not allowed to write to the target index | Send a test event from the gateway host with curl and read the HTTP status and Splunk response body | Reissue or re-enable the HEC token, confirm its allowed indexes, and set the matching index in the splunk-hec-logging configuration |
| TLS certificate not trusted by the gateway | Splunk uses a certificate from an internal CA that is missing from the gateway trust store | Run openssl s_client against the HEC endpoint from the gateway container or host and read the verify result | Add the internal CA to the gateway trust store and restart or reload, rather than disabling verification |
| Egress blocked from the gateway network | Firewall, security group, or proxy rules allow client traffic but not outbound connections to the HEC port or the Datadog intake | Test TCP reachability from the gateway network to the HEC port and the Datadog endpoint, then review egress logs | Open the specific outbound destinations and ports in the network policy, and confirm any required proxy settings |
| Batch backlog reaching the 8,192 entry limit | The collector is slower than the request rate, so pending entries build up until the bound from APISIX 3.18 is hit and new entries are dropped | Check the gateway error log for logger queue or drop messages and compare request volume with indexer acceptance rate | Fix the downstream bottleneck first, reduce logged fields, and size batch settings with the collector's capacity in mind |
| Log format change breaks Splunk field extraction | A custom log_format change renamed or reordered fields that Splunk search-time extraction expects | Search recent events for the expected field names and compare a sample event before and after the change | Version the log format, update the Splunk props and transforms or field extractions, and roll out the change with a test index first |
| Datadog metrics missing | The Datadog agent is not running or listening where the plugin sends data, or the API key is wrong or for another site | Check agent status, confirm the API key and site setting, and look for authentication errors in the gateway and agent logs | Correct the API key and site, make sure the agent address and port match the plugin configuration, and restart the agent |
| High metric cardinality from unbounded tags | Tags built from raw URIs, consumer IDs or request IDs create a new time series for each unique value | Count distinct tag values over a time window in Datadog and identify which tag grows without limit | Use route IDs or normalized path templates instead of raw paths, drop request-scoped tags, and review custom tags before rollout |
Questions teams ask before connecting APISIX to Splunk and Datadog
How much effort does connecting APISIX to Splunk and Datadog take?
The effort depends mostly on how many routes and plugins need logging, how much log format work the security or operations team wants, and how many secrets must be handled safely. A gateway with a few routes sending one standard log format is a small change. A gateway with many route groups, different log fields per team, and several consumers that each need different sensitive field handling takes longer, because each variation has to be designed, configured and tested. Splunk HEC tokens and Datadog API keys also need an administrator on each side to create and scope them, so admin access on both platforms is often the slowest dependency. Ask for a written scope that lists the route count, the log formats, and the secret storage approach before you agree on a timeline.
Can our team set this up alone using the APISIX documentation?
Many experienced platform teams can configure a single logger on a test route using the plugin documentation, and that is a reasonable first step. The harder part is production work: keeping credentials out of plain configuration, deciding which fields to drop or mask, tuning batch sizes and backlog limits for your traffic, verifying that events arrive intact in Splunk and Datadog, and writing the runbook your on-call team will use. If your team has someone who owns APISIX, Splunk and Datadog, the work can stay in-house. If one of those skills is missing, a fixed-scope engagement lets you buy that gap without hiring for it. Confirm the skills you have before deciding.
What do you need from us to start the integration?
The work needs a few things from your side before it can begin:
- Access to the APISIX environment where the gateway runs, including the Admin API or the configuration source you use
- A Splunk HTTP Event Collector endpoint and token, created by a Splunk administrator with the index set you approve
- A Datadog API key and the site your organization uses, created by a Datadog administrator
- A change window agreed with your operations team for enabling the loggers on production routes
- A set of test routes, ideally in a non-production environment, where log output can be checked before production use
Your security team should also confirm where secrets are stored and who may read them.
Can we lose logs if Splunk or Datadog goes down?
Yes, logs can be lost if the destination is unavailable for longer than the gateway can hold them, so the plan must cover that case explicitly. APISIX 3.18.0 bounded logger backlogs at 8,192 pending entries, which caps memory use but means entries beyond that limit are dropped when a destination stops accepting data. Plan for this by setting batch and retry values deliberately, alerting on logger errors in APISIX, and deciding whether a second log stream such as a file or message queue is needed for records you cannot lose. Confirm the retry and backlog behavior for the exact plugin version you run, and test an outage on a non-production route before relying on it.
How do we keep tokens and sensitive fields out of logs?
Start by listing which request and response fields carry credentials, personal data or session values, such as authorization headers, cookies, query parameters and body content. Then decide, per route, which fields to remove, mask or never log. Request and response body logging should be limited to the routes that need it, and the max_req_body_size and max_resp_body_size settings introduced in APISIX 3.18.0 (default 64 MiB) help bound what gets captured. Credentials for the Splunk and Datadog endpoints should be referenced from a secret store rather than typed into route configuration. Have your security team review the field list before production, since they own the decision about what counts as sensitive.
How does cost work for Splunk and Datadog ingest?
Splunk and Datadog set their own pricing, and that pricing usually depends on data volume, retention, indexing or host and feature tiers, so the largest cost driver sits outside the gateway. DeployOpen does not set those prices and does not publish a price list. The integration cost has its own parts: a fixed-scope engagement quote for design, configuration, testing and handover, and any recurring support you choose. To control ingest cost, decide which routes send full logs, filter low-value events, sample high-volume routes where your policy allows, and set retention in each platform. Ask your Splunk or Datadog account team for current pricing for your expected volume, and ask DeployOpen for a written scope so the two can be compared.
How is the cost of a DeployOpen engagement quoted?
DeployOpen works from a written scope and does not publish a price list. Most engagements are fixed-scope, which means the scope, deliverables and timeline are agreed in writing before work starts. A quote follows from that scope, so the inputs that matter are the number of routes, the log formats, the platforms in use, the secret handling approach, the testing you want, and whether you need operational support afterward. Ongoing support, upgrades and continuous testing are available as retainers and are quoted separately. Ask for the quote to name each deliverable and the assumptions about your access and change windows.
What does an illustrative first logging rollout look like?
As an illustrative example, a team with a single production gateway and one Splunk index might begin with a test route, confirm that events arrive with the fields the security team agreed on, and then enable the logger on a small set of production routes during an agreed change window. Datadog metrics could follow once the log path is stable. Each step has a rollback plan, and the team reviews logger error counts before widening scope. The actual sequence depends on your route count, platforms and approval process, so treat this as a pattern to test against your own environment rather than a fixed plan.
How DeployOpen scopes and delivers the Splunk and Datadog integration
DeployOpen plans, implements, tests and documents APISIX gateway logging and metrics inside your own infrastructure. We start from the log fields, metrics and ingest limits your Splunk and Datadog teams have agreed, then build and hand over a configuration your operators can run and change.
How we scope
- Inventory the APISIX routes and plugins whose logs or metrics must reach Splunk or Datadog
- Agree which request fields, headers and bodies are logged, masked or excluded before any build starts
- Confirm HEC endpoint, token ownership, Datadog site and index or metric volume limits with your platform owners
- Record the written, fixed-scope deliverables, timelines and retainer options for ongoing support
How we build
- Configure splunk-hec-logging and datadog plugin settings in a non-production APISIX environment first
- Set batch sizes, retry behavior and backlog limits so that a slow endpoint cannot exhaust gateway memory
- Apply sanitization rules so tokens, cookies and personal data do not reach downstream indexes
- Test failure modes, including an unreachable HEC endpoint and a Datadog agent outage, and record the results
How we hand over
- Deliver the plugin configuration as files or API payloads under your version control
- Document each logged field, its purpose, its retention setting and the owner who approves changes
- Provide a runbook for rotating HEC tokens, checking dropped batches and tuning ingest volume
- Walk your security and operations staff through the configuration and the test evidence
How we support
- Review log volume and ingest cost after go-live and propose filtering changes in writing
- Plan APISIX version upgrades with a regression check of the logging and metrics pipeline
- Update sanitization rules when new routes or plugins add sensitive fields
- Offer ongoing support, upgrades and continuous testing as a retainer, scoped in writing
