How Apache APISIX tracing with OpenTelemetry works
A team that already runs an observability stack and needs request traces from the API gateway can add APISIX as a span source without replacing anything else. APISIX exports spans over OTLP HTTP to an OpenTelemetry Collector, and the Collector receives, processes and forwards them to a backend the client chooses. OpenTelemetry is a framework and toolkit for generating and moving telemetry, not a storage or visualization product, so the trace store and the query interface are separate decisions. Both projects are licensed under Apache-2.0: Apache APISIX is an Apache Software Foundation project, and the OpenTelemetry Collector is a Cloud Native Computing Foundation project under the same license family. This page covers the exporter, the sampler, the batch span settings and the overhead tracing adds at the gateway. It does not cover backend selection.
The architecture has three parts that each need an owner. APISIX is configured with the tracing plugin and an exporter endpoint, and the Collector is configured with an OTLP HTTP receiver, processors and an exporter to the backend. The Collector can run as an agent beside the gateway or as a central gateway that accepts data from several sources. Confirm with your platform team which pattern fits your network, which team operates the Collector, and how its configuration changes are reviewed. Before enabling anything in production, confirm the OTLP HTTP path and port your Collector build exposes, and confirm the APISIX version and plugin fields against the APISIX documentation for the version you run, since field names and defaults can differ between releases.
The decisions that shape the rest of the build are the sampling rate, the batch size and export interval, and the trace headers the gateway passes to upstreams. Sampling determines how many requests produce spans, which directly controls storage cost downstream and the CPU and memory the gateway spends on tracing. Batch settings determine how long spans wait in memory before export and how much is lost if an export fails. Ask your APISIX and Collector operators to load test with your real traffic mix and record gateway latency and memory with tracing on and off. Those measurements, not general guidance, should set the production sampler and batch values.
What you need before enabling APISIX tracing
Tracing from APISIX only works when the gateway, the Collector and the network between them agree on version, protocol and trust. The cards below cover the checks to complete before you change any gateway configuration, so the first trace reaches a backend your team can read.
Confirm the APISIX version
Check the version you run against the Apache APISIX downloads page before you copy plugin settings from any guide. The page lists 3.19.0 as the latest release, and it does not mark any release as long-term support, so confirm which line your organization will standardize on and when it will upgrade. Record the exact version, because plugin fields and defaults can change between releases and your configuration should match the build in production.
A reachable OTLP HTTP receiver
APISIX exports traces to a collector endpoint over OTLP HTTP, so you need a Collector deployment with an OTLP HTTP receiver enabled and listening on an address the gateway nodes can reach. Confirm the receiver path, port and whether it is exposed through a load balancer. Decide whether the Collector runs as an agent beside the gateways or as a shared gateway tier, because that choice sets how many endpoints the gateways must reach.
Collector release and distribution
Confirm the Collector version against the OpenTelemetry Collector releases page before you deploy it. The page lists v0.163.0 as the most recent release at the time of writing, and newer builds may be available when you deploy. The contrib distribution ships signed artifacts, SHA256 checksums and SBOM files, so ask your platform team to verify these in your artifact pipeline. Confirm that the receivers you use are in the distribution you choose.
Network paths and firewall rules
List every gateway node, every Collector endpoint and the ports between them before you open a change request. Gateway nodes need outbound access to the OTLP HTTP port, and any proxy or egress policy in your network must allow that traffic. Test the path from a real gateway node with the same DNS names and routes the gateway uses, since a passing test from a jump host does not prove the gateway can export.
TLS and certificate trust
Decide whether the OTLP endpoint uses TLS and which certificate authority signs it. Gateway nodes must trust that authority, and the certificate name must match the host name in the exporter endpoint. Plan rotation before go-live, including how new certificates reach each gateway node. If you use mutual TLS or an authenticator extension on the Collector, document who issues client certificates and where the keys are stored.
A named backend owner
Name one team that owns the tracing backend, its retention settings and its access rules before you enable traces. OpenTelemetry does not store or visualize traces itself, so the Collector can accept spans that nobody is positioned to query. The owner should agree on retention, who can view spans that carry request metadata, and who responds when the export queue backs up.
Configure the OTLP exporter and sampler in order
This sequence turns on the APISIX opentelemetry plugin, sends its spans over OTLP HTTP to a Collector, sets how many requests are sampled, and tunes how spans are batched. It ends with a Collector pipeline that receives the spans and prints them for verification.
Confirm version and configuration file location
The platform engineer records the APISIX version in use and opens the static configuration file, which is conf/config.yaml in a typical traditional deployment. Plugin option names and defaults can change between releases, so the engineer checks each key in this sequence against the APISIX plugin documentation for that exact version before editing anything. Done when the version, the file path and the list of keys to change are written down and the current gateway starts cleanly after a restart in a test environment.
Enable the opentelemetry plugin globally
Add opentelemetry to the plugins list in conf/config.yaml so the gateway loads the tracing plugin at startup. Enabling it in the static file makes the plugin available to routes, and a route still has to opt in through its own plugin configuration in a later step. Restart or reload the test gateway and check the startup log for plugin load errors. Done when the gateway starts with no error about the opentelemetry plugin and the Admin API accepts a test route that references it.
Point the OTLP HTTP exporter at the Collector
Under plugin_attr, set the opentelemetry collector address to the Collector's OTLP HTTP endpoint, which is conventionally port 4318 with the /v1/traces path. Use a hostname the gateway can resolve, and confirm the exact key names and whether the path must be included with the APISIX opentelemetry documentation for the version in use. If the Collector runs behind TLS or an authenticator, record those settings now as well. Done when the gateway config holds one address and the network path from the gateway host to that port is confirmed open with a simple connection test.
Set the sampler on the test route
Enable the plugin on one test route and set its sampler. The always_on option records every request and suits a test, while a ratio-based option samples a fraction of traffic and suits production volumes. Choose the sampler based on how many requests the route receives and how much trace storage the team can run, and confirm the sampler names and fields for the chosen version. Done when a test request to the route is accepted, the Admin API returns the saved route with the opentelemetry block, and the chosen sampling behavior is written in the change record.
Tune the batch span processor settings
Adjust the batch span processor settings under plugin_attr so the gateway groups spans before sending them to the Collector. The main controls are the queue size, the batch timeout, the maximum batch size and the behavior when the queue is full, and their exact names come from the plugin documentation for the version in use. Start from the documented defaults, change one value at a time, and note why each change was made, because batch size and timeout trade export latency against memory and per-request overhead. Done when the gateway restarts with the chosen values and the team has recorded the default and new value of each setting.
Add the OTLP receiver and debug exporter pipeline
In the Collector configuration, define an otlp receiver listening on the HTTP port the gateway targets, a batch processor, and a debug exporter that writes received spans to the Collector log. Wire these into a traces pipeline under service, and confirm the component names and their stability level in the README for the Collector release you deploy. The Collector configuration documentation covers the pipeline syntax. Done when the Collector starts, reports the traces pipeline as running, and shows no unknown component errors in its log.
Verify a span in the Collector output
Send a test request through the gateway to the route with tracing enabled, then read the Collector log output from the debug exporter. A successful run shows a span with the route's request attributes, and the span appears after the batch timeout passes, so the engineer may need to send a few requests or wait briefly. If no span appears, check the collector address, the port, the sampler setting and the Collector log for receiver errors before changing anything else. Done when one span from the test request is visible in the Collector output and the team has saved that output as the baseline for later changes.
How gateway spans link to service traces
The gateway span is the first span in a trace, so its job is to start the trace and pass its identity downstream. When a request reaches APISIX, the tracing plugin checks the incoming request for trace context headers. If a valid W3C traceparent header is present, APISIX continues that trace and creates its span as a child of the caller's span. If none is present, APISIX starts a new trace, and the sampler decides whether that trace is recorded. Confirm which propagation formats your APISIX version reads and writes in the plugin documentation, because services in your estate may send B3 or other headers from older tracers. Mismatched formats are the most common reason gateway spans appear as disconnected roots in a backend.
Before a request reaches an upstream, APISIX writes the gateway span's context into the outgoing request headers so that the service can attach its own spans to the same trace. Decide early which header format every upstream will read, and make sure no intermediate proxy, load balancer or WAF strips or rewrites those headers. Also decide whether APISIX should trust an inbound trace ID from external clients at all. Accepting it from the public internet lets outside parties correlate or flood your traces, so many teams accept inbound context only from internal networks and start a fresh trace at the edge.
The Collector receives OTLP from APISIX, runs processors such as batching or attribute changes, and exports to a backend. Running it as an agent beside the gateway keeps the export path short and lets each node buffer spans locally, but it means every gateway node carries a Collector to configure and upgrade. Running it as a central gateway gives one place to manage authentication, sampling policy and backend credentials, which suits teams that want fewer moving parts, though it adds a network hop and a shared dependency that must be sized and monitored. The Collector's configuration documentation lists the receiver, processor and exporter settings to compare. Test both shapes under your expected peak load before choosing.
Choosing a sampler for production traffic
Always-on sampling records every request as a trace. It is the simplest setting to reason about, and it suits a low-traffic gateway, a staging environment, or a short debugging window in which you need to see every path through routes, plugins and upstreams. The cost is span volume. Every request produces spans at the gateway, and each span must be exported, batched, stored and queried. Span volume grows with request rate, so storage and collector capacity grow with it. For most production routes, treat always-on as a temporary setting unless your traffic and retention budget clearly allow it.
- Storage growth: backend ingest and storage scale with request rate, so confirm retention settings for the backend you chose before enabling it.
- Export load on the gateway: exporting every span adds work to APISIX worker processes. Measure CPU and memory before and after enabling it.
- Pressure on the collector: a collector that drops or refuses spans under load can lose data without alerts unless you monitor its dropped and refused counters.
- Expensive dashboards: teams that see high trace costs often turn tracing off during incidents, which removes the data you need most.
Confirm the sampler name and its options in the OpenTelemetry plugin reference for your APISIX version, and set a time limit or traffic cap before you enable always-on on production routes.
Ratio-based sampling keeps a fixed fraction of traces. It is the usual choice for steady production traffic because span volume becomes predictable: it follows the request rate multiplied by the fraction you select and the number of spans each request produces. The sampling decision is made from the trace ID, so the same trace receives the same decision in any service that applies the same method. The main risk is what the sample hides. A low-frequency error or a slow request on a quiet route may never be sampled, so the trace you need for an investigation may not exist.
- Volume control: ask the team to model expected spans per day from request rate, fraction and spans per request before choosing a number.
- Hidden rare failures: errors on low-volume routes can be sampled out, so keep error logs and gateway metrics as a second source of evidence.
- Consistency across services: trace-ID-based decisions only stay consistent if upstream services use a compatible method. Confirm this with each application team.
- Tuning later: the fraction is a configuration value. Confirm whether a change takes effect on reload or needs a restart in your APISIX version.
Confirm the ratio parameter name and its accepted range for your version, then test the chosen fraction against representative traffic in staging before you rely on it.
Parent-based sampling honors the decision made by the caller. If an incoming request carries a trace context that is marked as sampled, APISIX records its span and continues the sampled trace. If the caller marked the trace as not sampled, APISIX follows that decision. This keeps traces whole across the edge. A gateway that samples independently can split one request into fragments, with the gateway span missing from the end-to-end trace or the reverse. The trade-off is control. Your gateway setting applies mainly to requests that arrive without a decision, so the effective volume is partly set by clients and services outside APISIX.
- Whole traces: gateway spans attach to the trace that started in the browser, partner system or upstream service.
- Volume set outside the gateway: a client that samples every request sends every request into your pipeline, and the gateway cannot reduce that on its own.
- Untrusted callers: requests from external networks can set sampling flags. Decide whether to accept inbound trace context from outside your perimeter.
- Root sampler still matters: requests with no incoming context use the root sampler you configure, so that setting determines the volume for direct traffic.
Confirm how your APISIX version handles an incoming sampling flag and which root sampler applies when none arrives. Then test with requests that carry sampled and unsampled trace headers.
Span attributes and headers mapped between APISIX and the Collector
This table maps the request fields a gateway trace carries, such as route name, HTTP method, status and upstream address, to the span attributes and Collector resource attributes that identify APISIX. Read the third column first: fields marked as standard HTTP semantics are stable across tools, while route, service and upstream fields depend on how your APISIX configuration and tracing plugin are set up. Confirm exact attribute names in the APISIX tracing plugin documentation before you write dashboards or alerts against them.
| Field | Where it appears | Basis |
|---|---|---|
| HTTP request method | Span attribute for the HTTP method of the client request | Standard HTTP semantic attribute |
| HTTP response status code | Span attribute set from the status returned to the client | Standard HTTP semantic attribute |
| URL path and host | Span attributes for the request target as seen at the gateway | Standard HTTP semantic attribute, check whether query strings are redacted |
| Route name or route ID | Span attribute or span name, set by APISIX from the matched route | Depends on APISIX configuration, confirm the exact name in the plugin docs |
| Upstream address | Span attribute for the backend the gateway proxied to | Depends on APISIX tracing settings, confirm before relying on it |
| Trace context headers (W3C traceparent) | Incoming header read by the gateway and forwarded to upstreams | Depends on the propagation settings you choose, confirm supported formats |
| Gateway service name | Collector resource attribute identifying the APISIX instance | Set in your OpenTelemetry resource configuration, not a standard HTTP field |
| Deployment environment and cluster | Collector resource attributes added by a processor or the exporting service | Set by your team, so define the naming standard before rollout |
Securing the trace pipeline
Trace data carries request paths, upstream names and timing for every call through the gateway, so the path from APISIX to the collector needs the same controls as the APIs it observes. The cards below cover transport, authentication, data minimization, network reach and configuration hygiene, and each one names a check you can run before traces leave your network.
Encrypt the OTLP connection
Serve the collector's OTLP HTTP receiver over TLS and point the APISIX tracer at an https endpoint so span data is not readable on the network between the gateway and the collector. Confirm which CA the APISIX nodes trust and rotate the collector certificate on a schedule your PKI team already runs. Test with a failed certificate to confirm the export is refused rather than silently falling back to plain HTTP.
Authenticate senders at the receiver
Use an authenticator extension on the OTLP receiver, such as the OIDC option the Collector supports, so only APISIX nodes holding a valid token can write spans. Decide how gateway nodes obtain that token, whether through a service identity or a short-lived credential, and who rotates it. Without this step any workload that can reach the port can inject spans into your traces.
Authenticate exports with oauth2client
When the collector forwards to a backend that requires credentials, the oauth2client extension can add a bearer token to outgoing exporter requests. Store the client secret in an environment variable or a secret store rather than in the collector file. Check that token expiry and refresh are tested, because a silent authentication failure on the exporter drops spans without alerting anyone.
Keep sensitive data out of spans
Span attributes are copied into backends and dashboards, so query strings, authorization headers, cookies and tokens passed as parameters can end up in storage. Decide which request headers APISIX may tag on spans, drop the rest, and remove query parameters with the collector attribute processor before export. Ask the team that owns your observability backend which fields they index and who can read them.
Restrict who can reach the receiver
Bind the OTLP receiver to a private interface and allow only the gateway subnets or pod selectors through firewall rules or network policy. Keep the collector's health and diagnostic endpoints off public listeners, since they reveal version and pipeline details. Confirm the path with a connection test from outside the allowed set, which should fail.
Review the collector configuration
Follow the Collector configuration best practices for secrets, endpoints and component stability: use environment variables for credentials, disable components you do not use, and review the stability level listed for each component before you rely on it. Pin the collector release you tested and record the release notes that change receiver or processor behavior. Repeat the review on each collector upgrade.
Why APISIX spans are missing or incomplete
This table lists the failures that most often leave APISIX traces empty or partial, with the likely cause and a check the team can run for each. Work down the rows in order, since the first two checks confirm that spans leave the gateway at all before you look at sampling or propagation.
| Symptom | Likely cause | Check to run and fix |
|---|---|---|
| No spans reach the Collector | The exporter endpoint or path is wrong, so OTLP HTTP posts go to a URL the Collector does not serve | Confirm the OTLP HTTP receiver port and path in the Collector configuration, then send a test request to that exact endpoint from the gateway host and check the Collector logs for received spans |
| Export fails with certificate or handshake errors | The gateway does not trust the Collector's certificate chain, or the endpoint uses a scheme that does not match the receiver's TLS setting | Check the gateway error log for TLS errors, confirm the scheme in the endpoint matches the receiver, and add the internal CA to the trust store the gateway uses before retrying |
| Spans appear for some requests but not during peaks | The batch span processor queue fills under load and newer spans are dropped | Watch for dropped-span messages in the gateway error log during a load test, then review the batch size, queue limit and export timeout settings and confirm the Collector accepts the export rate |
| Traces look sparse or only show a few requests | The sampler ratio is set too low for the traffic being tested | Compare the sampled request count with the request count from access logs over the same window, and confirm the sampler setting and ratio in the tracing configuration used by the running gateway |
| Traces stop at the gateway span | Upstream services do not read the incoming trace headers, so their spans start new traces | Send one request with a known trace ID, then check each upstream service's logs or tracing backend for that ID and confirm the propagation headers are forwarded and read by each service |
| Spans or fields are missing or behave unexpectedly after a Collector change | Some Collector components are at development or alpha stability, and their behavior can change between releases | Check the stability level of each receiver, processor and exporter in its README on the Collector repository, pin the Collector version, and test configuration changes in a staging pipeline before rollout |
Measuring the overhead of tracing at the gateway
Tracing adds work to every request it samples, so the cost is never zero and should be measured before production rollout. The overhead comes from several places: the sampling ratio decides how many requests create spans, the batch span processor settings control how many spans are held and exported at once, and the export frequency and network path to the Collector determine how much time is spent sending them. Span attribute volume matters too, because a route that records large headers or many custom attributes produces heavier spans than one that records only the defaults. Each of these levers changes the result, so a single test at one setting does not describe the others.
Build a staging gateway that matches production in plugin set, route count, TLS settings and upstream latency. Run a fixed load profile with tracing turned off and record request latency at the percentiles your service levels use, along with APISIX worker CPU and memory. Then enable the OpenTelemetry plugin with the sampler you intend to ship, keep the batch settings and Collector address identical to the planned production values, and repeat the same load profile. Compare the two runs at the same request rate, and repeat the sampled run at a higher sampling ratio to see how the curve moves. Confirm the exact plugin and sampler configuration keys against the APISIX plugin reference for your release before you write the test plan.
Test the network path as well as the gateway. A Collector in the same cluster, reached over a short internal route, behaves differently from one across a WAN link or behind a congested proxy, and an unreachable Collector can change how much work the gateway does while it retries or drops spans. Ask your team to define pass criteria before the test starts, including the maximum acceptable latency increase, the CPU headroom to keep at peak, and the behavior you expect when the Collector is down. Keep the results with the change record, because they become the baseline for later sampler or batch changes.
Questions teams ask before enabling APISIX tracing
What does it cost to add OpenTelemetry tracing to APISIX?
The cost is the labor to configure the tracing plugin, connect it to a Collector, set sampling and batch settings, and test them against real traffic. Effort rises with the number of routes and upstreams that need trace coverage, whether a Collector already exists, which backend receives the spans, and how much sampling logic must be designed. Software licenses are not the main driver here, because APISIX and the OpenTelemetry Collector are both Apache-2.0 licensed. Ask for a quote that lists each of these items separately, and ask which recurring costs (backend storage, Collector hosting, and upgrades) fall to your team after handover.
How long does an APISIX tracing setup usually take?
The duration depends on the scope agreed in writing before work starts, so no fixed figure applies to every estate. A narrow scope covers one Collector endpoint, a sampler, and tests on a few routes. A wider scope adds several environments, existing Collector pipelines to extend, and validation of trace propagation to upstream services. Ask for a written timeline with milestones for Collector connectivity, sampling tests, overhead measurement and handover. Timelines that do not name the number of routes and environments in scope are hard to verify later.
What must our team provide before tracing work starts?
The work goes faster when the following are ready before the first session:
- Access to the OpenTelemetry Collector, including its OTLP HTTP receiver endpoint and the configuration repository or deployment pipeline that holds it
- Network rules that allow APISIX nodes to reach the Collector on the agreed port, in each environment
- A named owner for the tracing backend who accepts the trace volume, retention and access rules
- Test traffic on representative routes, including authenticated calls and error paths
- Staging capacity to measure gateway latency with tracing on and off
Confirm each item with your platform and security teams early, because firewall changes often take the longest to approve.
Can our internal team enable APISIX tracing without a partner?
Yes, a team that already runs APISIX and an OpenTelemetry Collector can enable tracing alone, because the configuration is a plugin and a handful of settings. The work usually needs help in three places: sampler choices that balance trace cost against incident value, batch span settings that hold up under peak load, and verification that spans link correctly across the gateway and upstream services. Teams new to OpenTelemetry often also need help deciding how the Collector should process and export data. Your team can test these areas itself, or a partner can plan and run them under a written scope.
How does tracing differ from OpenTelemetry as a backend?
Tracing is a behavior that APISIX performs: it creates spans for requests and exports them to a destination. OpenTelemetry is the framework and toolkit that defines how telemetry is generated and exported, and it is not a backend. OpenTelemetry does not store or visualize traces, so you still need a separate tool to search and display them. The OpenTelemetry Collector sits between APISIX and that tool, receiving, processing and forwarding the data. Choosing a backend is a separate decision from enabling tracing, and it should be made with its own owner, retention plan and cost model.
How much overhead does tracing add to the gateway?
Tracing adds some work to each sampled request, including span creation, attribute capture and batched export. The size of that overhead depends on your sampling rate, the amount of data attached to each span, batch settings, and the throughput of your gateway. No general figure applies, so measure it on your own traffic in a staging environment with tracing on and off at the load you expect in production. Keep the sampler and batch settings under change control, because they determine both overhead and the volume of data your backend must accept.
What security checks apply to trace data and the Collector connection?
Traces can carry request paths, header values and identifiers, so treat them as sensitive data. Check which headers and attributes APISIX records, remove credentials and personal data from spans, and confirm that the connection to the Collector uses the transport protection your policy requires. Collector configuration also has its own security guidance, including authentication on receivers and exporters, which you should review before production. Ask the partner or your team to document these choices as part of the handover.
How DeployOpen scopes and delivers APISIX tracing
DeployOpen plans, implements, tests, documents and supports self-hosted open-source platforms inside your own infrastructure. For APISIX tracing, we start from a written scope that names the routes to trace, the OpenTelemetry Collector you already run or will run, and the sampling and overhead limits your team has agreed to.
How we scope
- Inventory which APISIX routes and deployment mode will emit traces, and where the gateway runs.
- Confirm the OpenTelemetry Collector endpoint, OTLP HTTP path, and who owns that Collector.
- Agree the sampler choice, batch span settings, and a load test plan before any configuration changes.
How we build
- Configure the APISIX tracing plugin and exporter in a staging environment that mirrors production routes.
- Verify that trace context propagates to upstream services and that spans appear in your chosen backend.
- Measure latency and resource overhead under representative traffic, then adjust sampling and batching settings.
How we hand over
- Deliver the configuration, Collector pipeline notes, and a record of tested sampling and batch values.
- Document the runbook for changing sampling rates, rotating exporter credentials, and reading gateway spans.
- Walk your platform and security teams through the test results and the open items they must confirm.
How we support
- Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
- Ongoing support, APISIX upgrades and continuous testing are available as a retainer.
- Review tracing configuration after APISIX or OpenTelemetry Collector version changes that affect your deployment.
