Apache APISIX tracing with Zipkin and SkyWalking: what the integration does
When an APISIX gateway sits in front of services that already report to a Zipkin or SkyWalking backend, the gateway's spans have to join those traces, and the zipkin and skywalking tracers are the supported route for that. The opentelemetry plugin is a different method: it exports OTLP to an OpenTelemetry collector and is the right choice only when OpenTelemetry is the standard in your estate. A gateway span records the request as it enters APISIX, the route it matched, the upstream it was sent to, and the time spent at the edge. That makes the first hop of every request visible in the trace. It does not show what happens inside the upstream service, and it cannot link to downstream spans unless the trace headers are carried through the gateway to each upstream.
The APISIX plugin list places the zipkin and skywalking tracers alongside the prometheus metrics plugin, node-status and datadog metrics, and several loggers, so tracing is one observability plugin among others that you configure on the same routes. Before you write any configuration, confirm the option names, sampling settings and endpoint format against the plugin reference for the APISIX version you run, since these details can change between releases. Ask the team that owns the Zipkin or SkyWalking backend which collector address, protocol and service naming convention they expect, because a gateway span that arrives with an unfamiliar service name will be hard to find during an incident.
What you need before enabling APISIX tracing
Tracing through APISIX depends on five things lining up before any plugin configuration is applied: the gateway version, a tracing backend, network reachability, Admin API access and upstreams that carry trace headers. Each card names the team that usually owns the item and what to confirm before the build starts.
Matching APISIX version
Read the plugin reference for the same release line you run in production, because the zipkin and skywalking tracer options and defaults can differ between versions. The current release line noted on the downloads page is 3.19.0, but a gateway already in production may sit on an older line. Owner: platform team. Confirm the running version with the Admin API or your package inventory, and record it in the change ticket before copying any configuration example.
Reachable Zipkin or SkyWalking backend
Decide which backend receives spans before writing the plugin block. For Zipkin, the collector endpoint is an HTTP URL that the gateway posts spans to. For SkyWalking, the tracer targets the OAP collector. Owner: observability team. Confirm the backend version, the exact collector endpoint, whether it is served over TLS, and who is responsible for retention and capacity on that backend.
Network path from gateway nodes
Every APISIX data plane node sends spans directly to the collector, so each node needs an outbound route and firewall rule to that endpoint. Traces dropped at the network layer look like missing spans rather than errors, which makes this easy to miss. Owner: platform or network team. Test the path from each gateway node, not just from one jump host, and confirm DNS resolution from inside the gateway containers or hosts.
Admin API access for plugin changes
Enabling a tracer means adding plugin configuration through the Admin API in traditional mode, or through the configuration file in standalone mode. The Admin API listens on port 9180 in traditional mode, so the change path must reach that port from your automation or operator host. Owner: platform team. Confirm which accounts can change global or route-level plugin settings, how the Admin API key is stored, and whether changes go through a reviewed pipeline.
Upstreams that forward trace headers
Tracing only links gateway spans to service traces if each upstream reads the incoming trace headers and passes them to its own downstream calls. A service that drops or regenerates headers starts a new trace at that point. Owner: application teams. Before the build, confirm which header format each service expects, whether any proxy or service mesh in the path rewrites headers, and which services already emit spans to the same backend.
Choosing between Zipkin, SkyWalking and OpenTelemetry for APISIX
Choose the zipkin tracer when your services already report to a Zipkin-compatible backend and the team that runs it expects spans in that model. APISIX records the entry span for each request, so the trace graph starts at the gateway and continues into each upstream service that propagates the trace context. The choice is driven by the instrumentation and backend you already operate, not by the gateway alone.
- Trace start: The gateway span appears first, so time spent at the gateway is visible before any service span.
- Backend team provides: A reachable Zipkin endpoint, retention settings and an owner for trace queries. Confirm the endpoint and sampling fields in the plugin documentation for your APISIX version.
- Service instrumentation: Existing Zipkin libraries in your services must read the headers APISIX forwards. Verify this with a test request before rollout.
- Change cost: Keeping Zipkin avoids changing every service's instrumentation, which is often the largest effort in a tracing project.
- Sampling: Set the sampling rate at the gateway deliberately, because it decides which requests produce traces at all.
Before committing, confirm with the backend team which header format each service reads and that they accept the service name APISIX will report.
Choose the skywalking tracer when your services already run SkyWalking agents and the platform team uses the SkyWalking backend for service topology and trace analysis. The gateway appears as a node in that topology, so the value depends on the gateway's service name matching the naming convention your agents already use. The choice follows the agents and backend you operate.
- Topology view: APISIX appears as an entry point in the service map, which helps show how gateway traffic reaches each service.
- Backend team provides: A reachable SkyWalking collector, an agreed service naming convention and an owner for topology dashboards.
- Agent compatibility: Confirm that the agent versions in your services support the propagation format APISIX sends. Test with a sample request.
- Naming discipline: Inconsistent gateway and service names split one service into several entries in the topology, so agree names before rollout.
- Mixed estates: If some services use other tracers, decide whether they will be bridged or left out of the same trace graph.
Before committing, ask the backend team to show how an existing request trace looks end to end, then run the same request through APISIX in a test environment and compare the results.
Choose the opentelemetry tracer when your organization has standardized on OpenTelemetry for applications and infrastructure. The plugin exports APISIX spans so they sit beside application spans from the same collector pipeline. The main constraint is the transport: the APISIX opentelemetry plugin supports only binary-encoded OTLP over HTTP, so the collector must accept that format. The setup is covered on the separate OpenTelemetry tracing page.
- Samplers: The plugin offers
always_on,always_off,trace_id_ratioandparent_basesamplers. Choose one that matches how much tracing you need and how you want upstream sampling decisions respected. - Collector provides: An OTLP over HTTP receiver that accepts binary encoding, plus pipelines to your chosen trace store.
- Service instrumentation: Applications already using OpenTelemetry SDKs need the gateway and services to share trace context. Verify propagation end to end.
- Transport limit: If the collector accepts only other OTLP encodings or transports, add a compatible receiver in front of it or choose a different tracer.
- Driver: Pick this path when the standard already exists, because it gives one pipeline for every signal.
Before committing, confirm the collector's OTLP receiver settings and test one sampled request from the gateway through to the trace store.
How to enable the Zipkin or SkyWalking tracer in APISIX
This sequence takes a gateway from no tracing to a request that appears in your tracing backend as one trace, with the APISIX span and the upstream span linked under a shared trace ID. Each step ends with a check you can run before moving on.
Confirm the APISIX version and plugin reference
Before touching any configuration, record the APISIX version your gateway runs and open the plugin reference for that exact version. The zipkin and skywalking tracers are both listed among the observability plugins, but their option names and defaults can change between releases, so the reference for your version is the only valid source for keys and values. Confirm which tracer your backend expects, since the Zipkin and SkyWalking formats are not interchangeable. Check is complete when you have written down the version and the plugin page you will follow, and your backend team has confirmed which format they ingest.
Enable the chosen tracer in plugin configuration
Add the chosen tracer to the plugin set, either in the global plugin configuration so every route is traced, or on individual routes where you want to limit overhead or start with a pilot API. The plugin entry is the place where you set the collector endpoint, the service name that will label gateway spans, and any sampling settings the plugin version supports. Look up each configuration key in the reference for your version rather than copying one from an older example. Check is complete when the Admin API or your configuration file accepts the change and the gateway reloads without errors in its error log.
Point the tracer at the collector and name the service
Set the collector endpoint to the address of your Zipkin or SkyWalking OAP receiver as reachable from the gateway network, and confirm that the network path and any TLS or authentication the collector requires are in place. Give the gateway a service name that is clearly distinct from the upstream service names, so its spans are easy to filter in the backend. Confirm the endpoint from the gateway host with a basic connectivity test. Check is complete when the collector accepts a test connection from the gateway host and the service name you chose is visible as a registered service, or as an expected label, once traffic flows.
Set the sampling rate deliberately
Choose a sampling rate for the first environment, starting low enough that tracing does not add unplanned load to a production gateway. Use the sampling option your plugin version exposes and record the value and the reason for it. Sampling decides which requests produce traces, so a low rate means some requests will have no trace at all, which is expected. Confirm the behavior with your backend team, since their storage and retention settings determine how many traces are practical. Check is complete when a documented sampling value is set in configuration and a batch of test requests produces traces at roughly the rate you chose.
Confirm the upstream forwards the trace headers
A gateway span only links to a service span when the upstream receives the trace context. Check which propagation headers your tracer format uses, and confirm that the gateway forwards them to the upstream without stripping or rewriting them. Look at any header filtering rules, transformation plugins, or proxies that sit between APISIX and the service. Also confirm that each upstream service is instrumented with a tracer that reads the same format. Check is complete when an echo or debug endpoint on one upstream shows the trace header arriving from the gateway with the same trace ID that the gateway reports.
Send a test request through the gateway
Send a small number of test requests through a traced route, ideally with a known path and a recognizable header so you can find them later. Use a sampling setting high enough to guarantee the test request is traced, and note the time and route. Keep the test on a non-production route if one is available. Check is complete when the response returns normally from the upstream and the request is recorded by the gateway access log with the expected route and status code.
Verify one trace ID across gateway and upstream spans
Open the tracing backend and find the test request. The success condition is one trace containing a span from the gateway, labeled with the service name you configured, and a child span from the upstream service, both sharing a single trace ID. If the gateway span appears but the upstream span does not, return to the header check in the previous step. If the two spans appear as separate traces, the propagation headers are not being read or forwarded correctly. Record the trace ID, the service names and the span durations as your baseline for later comparison. Check is complete when the linked trace is visible and the expected service names appear on both spans.
Trace headers and fields mapped between the gateway and the backend
This table compares how APISIX and each backend tracing format name the trace identifiers, sampling decision, service and span names that must match across the gateway and upstream services. Read each row as a value the gateway sets or forwards and check the header name against the plugin reference for your APISIX version. Most broken traces come from a header format that the upstream service does not parse, so confirm each row with the team that owns that value.
| Trace element | Zipkin B3 style | SkyWalking style | Who must confirm |
|---|---|---|---|
| Trace ID | X-B3-TraceId, or the single b3 header when the multi-header form is not used | Trace ID carried inside the sw8 header | Gateway team sets it. Service team confirms the upstream reads and forwards it unchanged. |
| Span ID | X-B3-SpanId, generated for the gateway span | Gateway segment reference inside the sw8 header | Gateway team confirms the span it creates. Service team confirms its own span links to that value. |
| Parent span ID | X-B3-ParentSpanId, set when the request already carries a parent | Parent segment and span index inside the sw8 header | Service team confirms the parent is passed on. Mismatches here split one request into separate traces. |
| Sampled flag | X-B3-Sampled with 1 or 0 after the sampler decides | Sample flag within the sw8 header | Gateway team sets the sampler. Service team confirms it honors the decision and does not resample. |
| Service name | localEndpoint service name set in the plugin configuration | Service name set in the plugin configuration and sent in the sw8 header | Gateway team sets the gateway name. Service team confirms it matches the name in the tracing backend. |
| Span name | Derived from the route or upstream name used by the request | Operation or endpoint name for the gateway span | Gateway team agrees the naming convention with the service team so spans group correctly in the UI. |
| Instance identifier | Endpoint address of the APISIX node in the span | Service instance name sent with the segment | Platform team confirms each gateway node reports a distinct instance name. |
How sampling decides which requests get traced
Sampling decides which requests produce spans at the gateway, so it determines what you can investigate later. The tracing plugin applies a sampling rate to incoming requests, and only the sampled share generates spans for APISIX and is exported to the Zipkin or SkyWalking backend. The rate is set in the plugin's configuration for the tracer you choose, and the same plugin may also accept an upstream trace context that already carries a sampled flag. Whether the gateway respects that incoming flag, overrides it or ignores it is a behavior you must confirm for your chosen tracer and APISIX version before you rely on it. Test this with a request that carries a sampled header from a known client and check whether a span appears in the backend.
Use full sampling during integration testing so every request in a test run has a trace, then lower the rate in production to control storage cost and gateway overhead. Be aware that a low rate means a single failed request may have no trace at all, which is often the request an engineer needs. If failure investigation matters to your service owners, decide in advance whether errors should be traced regardless of the rate, and confirm whether your tracer supports that behavior. Document the chosen rate per environment so that a rate change is a reviewed decision rather than an accidental edit.
Sampling must agree across the gateway and the services behind it. If APISIX samples a request but a downstream service does not, or the reverse, the trace is split into fragments that do not join in the backend, and the gateway span appears without its service children. Agree on one policy for the path: either the gateway makes the decision and services honor the sampled flag, or the services make it and the gateway propagates the decision unchanged. Test the agreed policy with a sampled request, a non-sampled request and a request with an upstream sampled header, and record the results as acceptance evidence.
Protecting trace data and the collector path
Trace data often carries more than timing. Request paths, headers and tag values can expose tokens and personal data, and the collector path can leak that data if it is left open. The controls below are the ones you can check in your own APISIX configuration and your tracing backend before production traffic flows.
Limit what gets traced
Decide which routes and request headers the tracer may record before enabling it globally. Authorization, cookie and API key headers should never reach span attributes, and query strings may carry tokens or identifiers. Check a sample of exported spans on a non-production route and confirm that sensitive values are absent. Where a route handles personal data, consider excluding it from tracing or enabling tracing only on the routes that need it.
Keep the collector private
The zipkin and skywalking tracers send spans from the gateway to a backend endpoint over the network. Place that endpoint on a private network segment that only APISIX nodes can reach, and use TLS where the backend supports it. Confirm the backend's TLS support and certificate handling with the vendor or project documentation for your version, since not every collector accepts encrypted connections in the same way.
Restrict tracer changes
Tracer configuration is set through the Admin API, so anyone who can write plugin configuration can change where spans go, which headers are captured and how often requests are sampled. Limit Admin API access to named administrators and the deployment pipeline, and review changes to tracing plugin settings in the same change process as route changes. Hardening of the Admin API itself is covered separately.
Review span attributes and tags
Before production, export a representative set of spans and list every attribute and tag name the gateway writes. Look for values that identify users, customers or account numbers, and for custom tags that copy request bodies or variables you did not intend to expose. Record the approved list so that future plugin or route changes are compared against a known baseline.
Set trace retention policy
Trace data is operational data that may still be personal data or contain security-relevant details. Agree on a retention period with your data protection and security teams, and configure the tracing backend to enforce it rather than relying on manual cleanup. Ask the backend owner how deletion works for the storage you use and whether backups are held for a separate period.
Keep APISIX current
Tracing plugins change between releases, and security fixes are published in the same release stream. Track the APISIX version you run against the published release list, plan upgrades on a regular schedule, and re-check your trace-header and sampling settings after each upgrade. The current release and its date are listed on the project's downloads page.
Why APISIX traces break and how to fix them
This table lists the five failures teams most often see when tracing through APISIX, with the cause, a check to run and the fix for each. Read it as a diagnostic sequence: confirm spans leave the gateway first, then check header propagation, then identifiers, sampling and naming. The main takeaway is that most trace gaps come from the path between the gateway and the tracing backend, not from the gateway itself.
| Symptom | Likely cause | Check to run | Fix |
|---|---|---|---|
| No spans reach the tracing backend | Collector or Zipkin/SkyWalking endpoint unreachable from the gateway, wrong port, or the tracing plugin not enabled on the route | Confirm the plugin appears in the route or global plugin configuration, then test network reachability from the gateway host or pod to the backend endpoint and port | Correct the endpoint and port in the plugin settings, open the network path, and enable the plugin on the routes that must be traced |
| Gateway spans appear but upstream spans do not | Trace headers are not forwarded to the upstream, or a header rule removes them | Send a test request, then inspect the headers the upstream receives and review any header rewrite or removal rules on the route | Stop removing or overwriting the tracing headers, and confirm the service's tracing library reads the headers APISIX sends |
| One request shows up as two trace IDs | Header format mismatch between the tracer in APISIX and the instrumentation in the service, such as B3 headers sent while the service expects W3C traceparent | Compare the trace header names in the gateway's outgoing request with the names the service's instrumentation expects | Align the propagation format on both sides, or configure the service to accept the format APISIX sends |
| Test request is missing from the backend | Sampling rate too low to capture an occasional test request | Temporarily raise the sample ratio on a non-production route, then send several test requests and count the traces that arrive | Set the sample ratio for test and troubleshooting routes deliberately, and document the production value separately |
| Spans show missing or generic service names | Service name not set in the plugin configuration, so the backend has no label to group spans by | Open the stored plugin configuration and confirm the service name field is populated, then search the backend for that name | Set a clear, stable service name for the gateway in the plugin configuration and reapply it |
Questions buyers ask about APISIX tracing with Zipkin and SkyWalking
Should we choose Zipkin, SkyWalking or OpenTelemetry for APISIX tracing?
Choose the tracer that matches the backend your services already report to. If your estate already runs a Zipkin-compatible or SkyWalking backend, the matching APISIX tracer avoids a second collection path. If you are standardizing on OpenTelemetry across applications and infrastructure, the opentelemetry plugin is usually the better long-term fit, because its supported export format and sampler options are the ones to plan around. APISIX's opentelemetry plugin supports only binary-encoded OTLP over HTTP, so check that your collector accepts it before you commit. Confirm this with your architecture review, not from a default setting.
How long does a basic APISIX tracing setup take?
Duration depends mainly on how many routes need tracing, whether upstream services already propagate trace headers, and whether the tracing backend is in place. A single gateway with one backend and a few routes is a short exercise, while many routes across several environments takes longer because each route's sampling and header behavior must be checked. Ask any partner to break the schedule into discovery, configuration, upstream propagation tests and handover, and to state which of those depend on your teams. A fixed-scope written plan is the reliable way to get a date.
What does the work need from the platform and application teams?
The platform team provides access to the APISIX Admin API, the tracing backend endpoint, network paths from the gateway to the collector, and change windows for the gateway. Application teams need to confirm that their services read and forward the trace headers that APISIX sets, and that they keep the same trace identifiers across internal calls. Without that second part, gateway spans appear as isolated traces and do not link to service traces. Agree these responsibilities and named contacts in the scope before work starts.
Can an internal team set up APISIX tracing on its own?
Yes, a team with APISIX configuration experience and access to the tracing backend can configure the plugin, sampling and header propagation itself. The harder parts are usually the cross-team checks: verifying that every upstream propagates headers, confirming sampling does not drop the traces you need, and measuring the effect on gateway latency. Teams without recent gateway operations experience often benefit from an external review of the configuration. Decide based on who will own the setup after handover.
What drives the cost of APISIX tracing work?
The main cost drivers are the number of routes and upstream services, the number of environments, whether the tracing backend is already owned and operated by your team, and how deeply the work is tested. Each additional route adds sampling and propagation checks, and each service that does not forward trace headers adds a fix on the application side. Testing depth matters because verifying trace linkage end to end takes more effort than confirming that spans appear in one backend. Ask for the quote to list these drivers as separate line items.
What is one-time work and what recurs after tracing goes live?
The initial configuration, header propagation tests and the handover are one-time costs. Recurring work includes reviewing the configuration when routes change, adjusting sampling when traffic patterns or storage limits change, and upgrading APISIX and the tracing plugins as new versions are released. Plan for a periodic check that trace linkage still works after each upstream release, because a service change can silently stop header forwarding. Ask whether the partner offers these as a retainer or as separate fixed-scope tasks.
What should we ask a partner to include in a written quote?
Ask the quote to state the tracer chosen and the reason for it, the list of routes and upstream services in scope, and the tracing backend that receives the data. It should name which upstream services are covered by propagation testing, how sampling will be configured per route, and what evidence confirms that gateway spans link to service traces. Also ask for the responsibilities split between your teams and the partner, the handover deliverables, and the terms for changes to scope. Confirm which items are one-time and which are recurring.
How DeployOpen scopes and delivers APISIX tracing work
DeployOpen plans the APISIX tracing setup as part of your gateway and observability design, then builds and verifies it in infrastructure you control. The work covers which tracer you choose, how sampling and header propagation behave, and how gateway spans join your service traces.
How we scope
- The written scope names the tracer, the backend (Zipkin or SkyWalking) and the services whose spans must link to gateway spans.
- We record the sampling rule, expected request volume and which upstream headers must be propagated, agreed before build starts.
- Engagements are fixed-scope, with deliverables and timelines agreed in writing before work begins; ask for the scope to list each tracing environment.
How we build
- We configure the tracer in the APISIX plugin settings for each route or global rule that needs tracing, stored as configuration you own.
- We test trace header propagation end to end with real upstream services, checking that each request carries the trace identifiers expected.
- We verify span names, parent-child links and sampling behavior in the tracing backend before marking the build complete.
How we hand over
- You receive the configuration, a record of the sampler and propagation choices, and the list of routes covered by tracing.
- We document how to check that a request appears as one linked trace across the gateway and each upstream service.
- We walk your platform and on-call teams through the runbook for changing sampling rates and diagnosing missing spans.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers, including checks after APISIX version changes.
- Retainer work can cover tracing backend changes, new routes and periodic review of sampling against your traffic.
- Confirm with us during scoping which APISIX version and tracer documentation applies to your environment before the build starts.
