Apache APISIX service discovery: when static node lists stop working
Static upstream node lists work until the addresses behind a route start moving. Every deployment, autoscaling event or node replacement then means someone edits the gateway configuration by hand, and a missed edit sends traffic to an instance that no longer exists. Apache APISIX can take upstream instances from a discovery module instead, with Consul, Nacos, Eureka, DNS or Kubernetes as the sources it supports, and the right choice depends on where your services run. A VM estate registered in Consul or Eureka points you one way, while a Kubernetes cluster that already exposes Services and endpoints usually points to the Kubernetes option.
A discovery module feeds the upstream node list that APISIX uses for load balancing. The module queries its registry, builds the set of healthy instances and updates the upstream as instances register and deregister, so routes keep their definitions while their targets change. The trade-off is that the registry becomes a dependency of the data path. If it becomes unreachable or returns incomplete data, the gateway depends on how the module handles cached results, so you need to plan the failure behavior before go-live, not after the first registry outage. Decide which registry is the source of truth for each environment, and avoid running two registries for the same services without a clear rule for which one wins.
The supported modules, their configuration keys and their failure behavior change between releases, so confirm them against the APISIX documentation for the exact version you run. The latest release supplied for this page is 3.19.0, but your estate may be on an older line, and some modules may be marked as differing in maturity or behavior by version. Ask your implementation team to list every discovery module in use, its settings and how it caches results, then test a node removal and a registry outage in a non-production environment before you rely on the setup for live traffic.
What your estate needs before APISIX can read a registry
Discovery only works when several things are in place at once: a supported APISIX setup, a registry the gateway can reach, open network paths, credentials scoped to read instances, and trust for any HTTPS endpoint. Work through these cards before choosing a registry, because each one narrows the options.
APISIX version and deployment mode
Confirm the APISIX version you will run and whether it uses traditional, decoupled or standalone mode. In traditional and decoupled modes, the control plane writes routes and upstream definitions to etcd. In standalone mode, there is no etcd, so discovery settings live in the same YAML or JSON file as the rest of the configuration, and the gateway reloads that file when it changes. Check the version's discovery documentation before you plan the registry configuration.
A registry the gateway can reach
Consul, Nacos, Eureka or Kubernetes must already run and hold the instances you want to route to. Confirm that each service registers with a health status and that the registry returns the addresses the gateway should use, not container-internal or loopback addresses. Decide which registry is the source of truth for each estate, since a service registered in two places creates conflicting upstream lists.
Network paths to registry and services
The gateway needs two kinds of reachable path: one to the registry API endpoints, and one to every service port it will proxy to. Map both before configuration. Check firewall rules, security groups, Kubernetes network policies and any proxy between sites. Confirm the registry ports in your own environment, because the client traffic and Admin API ports for APISIX are 9080 and 9180 in traditional mode, and those ports do not tell you what the registry uses.
Read-only registry credentials
The gateway should hold a credential that can only read service instances, such as a Consul ACL token with read access to the service catalog, a Nacos user limited to naming reads, or a Kubernetes service account with list and watch rights on endpoints or endpoint slices. Do not reuse administrator tokens. Plan where the credential is stored, since APISIX configuration can reference secret managers or environment variables instead of holding the value directly.
TLS trust chain for registry endpoints
If the registry serves HTTPS with an internal certificate authority, the gateway must trust that chain before discovery works. Export the root and any intermediate certificates, confirm the server name in the certificate matches the endpoint the gateway calls, and plan rotation so a renewed certificate does not silently stop discovery. Test the handshake from the gateway host itself, because a browser or a workstation check does not prove the gateway can connect.
Choosing Consul, Nacos, Eureka, DNS or Kubernetes for your estate
Estates built on virtual machines usually already run a service registry, a configuration store or a DNS zone that tracks where services live. The usual fit is whichever of those your application teams already use, because a second registry would add a system nobody watches. Consul and Eureka are common choices when a registry was adopted for microservices, while Nacos often appears where teams also want dynamic configuration. Where no registry exists, a DNS-based approach with stable records may be simpler than adding one for the gateway alone.
- Reuse first: check whether an existing registry already holds accurate instance records and health checks for the services APISIX must route to.
- Ownership of registration: decide whether the deployment pipeline, the service itself or a sidecar agent registers each instance, and who fixes it when registration drifts.
- Health reporting: confirm how unhealthy instances are removed, and whether health comes from the registry, from APISIX active health checks or both.
- Additional operations: budget for running the registry cluster, its backups, its access control and its upgrades, which are separate from the gateway.
- Network reach: confirm APISIX nodes can query the registry across the network zones your estate uses.
Confirm each discovery module's supported options and configuration keys against the APISIX discovery documentation before you commit to a registry.
In a Kubernetes estate, the cluster already tracks every pod, its readiness and its endpoints, so the Kubernetes discovery option is usually the first candidate. Reading endpoints from the API server avoids running a second registry and keeps the gateway aligned with the same readiness signals that control traffic inside the cluster. An external registry such as Consul or Nacos makes sense mainly when workloads outside the cluster must also be routed through the same gateway.
- Reuse first: the cluster's service objects, endpoints and readiness probes are often enough, so confirm they reflect the health you want the gateway to act on.
- Access to the API server: plan a service account with read-only permissions limited to the namespaces APISIX serves, and review it under your RBAC model.
- Ownership of routes: settle who writes routes and upstream references, and whether the Ingress Controller or the Admin API manages them.
- Additional operations: the team must still operate the Kubernetes discovery settings, the upgrade path for both APISIX and the cluster, and the rollout order when they change.
- Scope of discovery: decide whether APISIX should watch all namespaces or a fixed list, since broader access widens what the gateway can see.
Confirm the Kubernetes discovery module's supported fields and permissions in the APISIX discovery documentation, and check them against your cluster version.
Mixed estates run some services on virtual machines and others in Kubernetes, often during a migration. The core decision is whether one registry can serve both sides, or whether APISIX must read from two sources. A single registry reachable from both environments keeps routing logic in one place, but it creates a dependency that every cluster and VM must reach. Two sources keep each platform native but require route ownership rules so that one service is never published twice with conflicting upstreams.
- Single source of truth: decide whether one registry covers both environments, and whether its health data is accurate for each side.
- Cross-network reach: confirm the network paths, firewall rules and name resolution that let APISIX reach registries and instances across both zones.
- Duplicate publication: agree which system owns each service name so that a migrating service does not appear with two upstreams.
- Failure domains: check what happens to routing if the registry is unavailable, and whether APISIX keeps the last known good upstream list.
- Operating two models: the team carries both registration paths, so document the runbooks and assign one owner for the discovery configuration.
Confirm which discovery modules can be combined in one APISIX deployment and how each handles registry outages, using the APISIX discovery documentation.
Wiring APISIX to a registry in sequence, with a check at each step
This sequence connects APISIX to a service registry in a controlled order, so each change is verified before the next one builds on it. The result is a gateway that routes to registry-discovered instances, with evidence that registrations and deregistrations are followed.
Confirm the registry is healthy and holds test instances
The platform or registry team checks that the registry cluster or agent is healthy, that its own health checks run, and that at least two test instances of a sample service are registered. Use the registry's own UI or API to list them, including their addresses, ports and health status. You are done when the registry lists the test instances as healthy and the team can name the service identifier APISIX will query. If the registry is unstable at this stage, stop here, because every later check depends on it.
Create a read-only identity for the gateway
Create a dedicated credential or ACL token for APISIX that can only read service catalog entries and health status for the namespaces or service names in scope. Do not reuse an administrator token or a token that can register or deregister services. Confirm the permissions by calling the registry's read endpoint with the new credential, which should succeed, and attempting a write, which should be denied. The check passes when both results match these expectations. Exact permission names depend on the registry product and version, so confirm them against the registry's documentation.
Configure discovery in a non-production APISIX instance
The gateway operator enables the discovery module for the chosen registry in a non-production APISIX instance, then sets the registry address, service identifiers and credential reference. Take the exact module keys, field names and defaults from the APISIX discovery documentation for the APISIX version you run, and keep the credential out of the file by referencing a secret source. Reload or restart as that version requires. The check is that APISIX starts without errors in its logs and reports the discovery module as loaded.
Create a test route that uses discovered upstreams
Create one test route through the Admin API that points to the discovery service name rather than a static node list. Use a path that only this test uses, so production traffic is not affected. Send a request through the gateway and record which instance answers, using a response header or body field that identifies the instance. The check passes when the request reaches one of the test instances and the route's upstream shows no hardcoded nodes. If the request returns a gateway error, check the discovery logs before changing the route.
Compare the node list APISIX sees with the registry
Query the Admin API for the upstream or discovered node list and compare it with the registry output from the first step. Addresses, ports and health state should match one for one. Pay attention to tags, metadata and weights, because mismatches here often come from filters in the discovery settings rather than from the registry. The check passes when the two lists match exactly. A difference means the gateway is reading a different scope, so review the service identifier and namespace before proceeding.
Test a registration change
Start an additional test instance of the same service and register it with the registry. Wait for the registry to mark it healthy, then query the Admin API again and send several requests through the test route. The check is that the new instance appears in APISIX's node list within the propagation window you observed in the discovery logs, and that some requests reach it. Record how long the update took in your own environment, since the timing depends on the registry's watch or poll behavior and the module's settings.
Test a deregistration and confirm routing follows
Deregister or stop one test instance and confirm the registry removes or marks it unhealthy. Check that APISIX drops it from the node list and that subsequent requests through the test route never reach it. Also confirm the gateway returns successful responses from the remaining instances. Only after this check passes should the team consider moving a production route to discovered upstreams. Keep the test route and the non-production instance in place as a regression check for future APISIX upgrades and registry changes.
What registry fields map to APISIX upstream nodes
This table matches the fields each registry exposes with the parts of an APISIX upstream that consume them. Read each row as a decision point: confirm the exact field names and behavior in the discovery module documentation for your registry before you write any configuration.
| Registry concept | What APISIX uses it for | What to decide |
|---|---|---|
| Service name or identifier | Groups instances into one logical upstream that a route can reference by name | Pick one naming convention across Consul, Nacos, Eureka and Kubernetes so route definitions stay readable across estates |
| Instance address and port | Becomes the node host and port APISIX proxies to | Decide whether registrations carry routable IPs or host names, and confirm which address the registry returns on VM and container networks |
| Health or status | Determines whether an instance is eligible to receive traffic, and how fast removed instances drop out | Set expectations for deregistration delay and confirm how each registry marks instances unhealthy or draining |
| Metadata or tags | Used for filtering or grouping instances, for example by version or canary label | Standardize tag keys such as version or environment and confirm which tags the discovery module can filter on |
| Weight | Can influence how traffic is spread across instances when the registry provides a weight value | Confirm whether your registry supplies weights at all and whether the discovery module passes them to APISIX nodes |
| Namespace or datacenter scoping | Limits which instances APISIX can see, which matters for multi-datacenter or multi-tenant estates | Decide whether one APISIX cluster reads across scopes or whether each environment gets its own discovery settings and credentials |
| Service name in route definitions | The upstream reference a route uses to find discovered nodes, rather than a static node list | Agree on the discovery prefix and name format in route and upstream config, and test that a renamed service fails visibly rather than silently |
What changes when instances come and go
Dynamic upstreams move the failure point from a config file to the registry. With a static node list, a bad change is a bad edit that someone reviews. With Consul, Nacos, Eureka or Kubernetes discovery, APISIX adds whatever instances the registry reports, so a wrong registration, a stale health status or a mislabeled service can send live traffic to an instance that is failing or not meant to serve that route. Before go-live, confirm which registry fields APISIX reads for each route, who can write those fields, and how a registration is reviewed or rolled back. Also confirm how the health checks configured on the upstream interact with the health the registry reports, so you know which signal removes a node from rotation first.
Registry outages and deregistration timing both shape what the gateway sends. If the registry becomes unreachable, APISIX may not learn about new nodes, and it may keep serving the last set of instances it knows about, including ones that have already gone away. Deregistration is a separate race: a stopping instance can still receive requests for a period after it begins shutdown if the registry and the gateway have not yet caught up. Test graceful shutdown and health checks together in a staging environment: drain an instance, confirm in-flight requests finish, then confirm no new requests arrive once it is marked out. Confirm the exact discovery and refresh behavior for your APISIX version in the APISIX documentation rather than assuming it.
Decide now how the gateway behaves when the registry is unavailable. Choose whether APISIX should keep the last known instance list, fail requests for affected routes, or fall back to a static set of upstreams for critical services. Each option trades stale routing against outright failure, so document it per route group and rehearse it. Keep a static fallback or a written recovery path that names who restores the registry, how the gateway is pointed at known nodes if the registry cannot be recovered quickly, and how the fallback is removed once discovery is healthy again. Revisit this plan whenever the registry or the APISIX version changes.
Securing registry access and the gateway's view of the estate
A service discovery integration widens what the gateway can see and what it can route to, so the registry becomes part of the gateway's trust boundary. The controls below cover who the gateway authenticates as, what it is allowed to read, how that traffic is protected, and how changes to the upstream set are recorded and reviewed.
Least-privilege read access for the gateway
Give the gateway its own registry identity with read-only permissions. Consul ACL tokens, Nacos credentials and Eureka access should allow catalog or instance reads and nothing else. The gateway identity should never be able to write, deregister or change health status. Confirm the exact permission set with the registry vendor's documentation and test it with a token that is denied a write.
Scoped namespaces and services
Limit discovery to the services the gateway actually routes to. Use Consul namespaces or partitions, Nacos namespaces and groups, or Eureka application names to separate production from staging and to keep unrelated services out of the gateway's view. An illustrative check is to list the services APISIX can resolve and compare that list against the route inventory. Anything extra is a finding.
Encrypted transport to the registry
Where the discovery module and the registry support TLS, use it for every call between the gateway and the registry. Ask the registry team to confirm certificate validation settings and which CA the gateway trusts. Traffic that carries instance addresses and health data should not cross a network segment that the gateway team does not control.
Protected registry credentials in configuration
Registry tokens and passwords should not sit in plain text in APISIX configuration or in version control. Reference them from a secret store instead. The step-by-step setup is covered on the separate secrets integration page, so this card only requires that no discovery credential appears in a route, a config file or an etcd dump without encryption.
Restricted registration in production
Only the deployment pipeline or the orchestrator should register instances in production. Developers and ad hoc scripts should not be able to add an address that the gateway will then send traffic to. Confirm which identities hold register and deregister rights in each registry, and review that list on a schedule.
Auditing changes to the upstream set
Record every change to the instances the gateway resolves, including who or what registered them and when. Keep registry audit logs, and compare upstream changes with the deployment record. On Kubernetes, the Secrets stored unencrypted in etcd by default make the same point: enable encryption at rest, restrict list and watch access with least-privilege RBAC, and confirm the gateway's ServiceAccount can read endpoints only in the namespaces it serves.
Common discovery failures, their causes and fixes
This table maps common symptoms of registry-based discovery in APISIX to their likely causes and the first fix to try. Read each row as a checklist: confirm the symptom in the APISIX error log and registry console, then work through the cause column before changing configuration. Module-specific log messages differ between Consul, Nacos, Eureka and Kubernetes, so confirm exact wording in your own logs.
| Symptom | Likely cause | Fix |
|---|---|---|
| No upstream nodes appear | Discovery module not enabled, wrong registry address, or the service has no healthy registered instances | Confirm the discovery block in the APISIX config points at the right registry, then check the service exists and has passing health checks in the registry UI or API |
| Nodes appear but requests fail to connect | Instances register a container or overlay address, or the wrong port, that APISIX cannot reach | Register the address the gateway can route to, verify the port with a direct request from the APISIX host, and check that no network policy blocks the path |
| Traffic goes to instances already stopped | Deregistration is delayed, health checks are too slow, or instances exit without deregistering | Shorten health check intervals, deregister on shutdown in the service's lifecycle hook, and watch for failures on nodes that were recently removed |
| Instances visible in the wrong namespace or datacenter | Discovery queries omit the namespace, group or datacenter, or the service was registered in another one | Set the namespace, group or datacenter in the discovery config to match the registration, and compare the query with the registry's own listing |
| TLS or certificate errors to the registry | Registry certificate not trusted by APISIX, wrong CA bundle, or hostname mismatch | Install the internal CA in the APISIX trust store, match the hostname in the certificate, and confirm the scheme in the discovery config |
| Credentials rejected by the registry | Expired or wrong token, ACL policy missing read access, or secret not resolved | Create a read-only registry identity scoped to the services APISIX needs, store it through your secrets approach, and test it with a direct API call |
| Registry outage leaves stale nodes | APISIX keeps the last known node list when the registry is unreachable | Decide how long stale nodes are acceptable, monitor registry reachability, and rehearse a registry failure in a test environment before go-live |
| Routes reference a name that does not match | Route service name differs in case, prefix or suffix from the registered service name | Copy the exact registered name into the route definition, and keep naming conventions in one shared document for teams that register services |
Questions teams ask before bringing in outside help
How much effort does service discovery add compared with static nodes?
Discovery adds a registry dependency and a reconciliation step on top of the static node list you already maintain, so the effort grows with the size of your estate. The main drivers are:
- The number of registries, since each has its own access model, health semantics and failure behavior.
- The number of services and instances that publish into them, and how consistently they register.
- The number of environments that must match, from test through production.
- How many registries must be merged into one APISIX configuration, including duplicate or conflicting service names.
An written inventory of these items gives both sides a shared basis for the quote, and the quote follows from that scope.
Can our team set up APISIX discovery in-house using the documentation?
Yes, a team with APISIX and registry experience can build a single-registry setup from the APISIX documentation. The documentation does not replace testing your own failure cases, though. Before you start, confirm the discovery module and its configuration keys for your APISIX version in the official docs, then check:
- How APISIX behaves when the registry is unreachable or returns stale instances.
- How health checks interact with instances that the registry still lists as healthy.
- How credentials for the registry are stored and referenced in configuration.
Teams with several registries, or with change windows that leave little room for trial and error, often find that testing and handover is the part of the in-house effort that is easiest to underestimate.
What does our side need to provide before discovery work starts?
You need to provide access, people and a safe place to test. The specific list depends on your registries, but most engagements need:
- Read access to each registry, and the access needed to register or deregister test services.
- A non-production environment with representative services and instance counts, so tests reflect real traffic patterns.
- Named owners for service registration practices, including naming rules, metadata and health check standards.
- Owners who can approve change windows for route and upstream updates.
- Your current static node lists and route inventory, so the work starts from what APISIX serves today.
Missing owners are the most common cause of delay, because registration habits decide what APISIX will see.
Is Kubernetes discovery different from running Consul or Nacos on VMs?
Yes. On Kubernetes, instance data comes from the cluster API, so the work centers on the access APISIX has to read it. Kubernetes authenticates clients with certificates, bearer tokens or an authenticating proxy, and Secrets are stored unencrypted in etcd by default. Grant the gateway only the read permissions it needs and confirm encryption at rest is configured, as described in the Kubernetes Secrets good practices guide. A registry on VMs means you also own the registry cluster itself, including its TLS, access control lists, backups, and upgrades. That operational load is usually larger than the APISIX configuration. Check both options against your team's skills before choosing.
How should we choose between Consul, Nacos, Eureka, DNS and Kubernetes?
Start from where your instances already register and who will operate the registry, because moving registration habits is the costliest change. Compare the options on these points:
- Where most instances run today: VMs, Kubernetes or both.
- Whether a registry already exists and is operated well by a team that will keep it running.
- What health and metadata each option passes to APISIX in your version, which you should confirm in the official documentation. DNS-based lookup often carries less metadata than a dedicated registry.
- Which teams must change their deployment process, and how much change they can absorb.
For a mixed estate, a common pattern is one source per environment, but the right answer depends on the inventory and should be tested in a non-production environment first.
What does a fixed-scope discovery engagement cover?
A fixed-scope engagement covers the work listed in a written scope agreed before work starts. For discovery, that normally includes an inventory of upstreams and registries, the design of the registry integration, configuration of APISIX in your environment, a test plan that includes registry outages and stale data, and a runbook for your operators. Ongoing support, upgrades and testing can be added as a retainer. DeployOpen does not publish a price list, so the quote follows from the written scope, and the scope should name the registries, services, environments and owners involved. Ask for the deliverables and the acceptance criteria to be written out in full.
What happens to traffic when a service registry becomes unavailable?
The answer depends on how your APISIX version handles registry failures, so confirm the documented behavior before you design around it. Ask whether APISIX keeps the last known instance list, how long it keeps it, and what happens to new instances or removed ones during the outage. Then test it on purpose:
- Stop the registry in a test environment and observe routing for each upstream.
- Register and remove instances during the outage and check what APISIX sends traffic to.
- Restore the registry and confirm the gateway reconciles without manual steps.
Record the results in the runbook, since operators will need them during a real incident.
How DeployOpen scopes and delivers APISIX discovery
DeployOpen plans and delivers APISIX discovery as part of a fixed-scope engagement inside your own infrastructure. We start from the registry you run today, agree the upstream model and failure behavior in writing, then build, test and hand over the configuration with the runbooks your team needs.
How we scope
- Inventory which upstreams are on VMs, Kubernetes or both, and which registry already holds their instance lists.
- Agree in writing which of Consul, Nacos, Eureka, DNS or Kubernetes APISIX will read, and why.
- Define what happens to routes when the registry is unreachable or returns an empty instance list.
How we build
- Configure the APISIX discovery module and registry connection in a non-production environment first.
- Test instance add, removal and health-check behavior with controlled upstream restarts and failures.
- Keep discovery settings and route definitions in version-controlled configuration that your team reviews.
How we hand over
- Deliver the registry connection settings, network paths, credentials handling and an admin access model.
- Document the failure runbook covering registry outages, stale instances and rollback to static upstreams.
- Walk your platform and operations engineers through the monitoring signals to watch after go-live.
How we support
- Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
- Ongoing support, upgrades and continuous testing are available as a retainer after handover.
- Ask any partner to confirm which APISIX and registry versions they will support, and how upgrades are tested.
