How APISIX health checks and load balancing keep traffic off failing backends
When a gateway routes to several upstream nodes and one of them starts failing, clients see errors even though most of the capacity is healthy. APISIX health checks and load balancing address that gap: the gateway tracks the state of each upstream node and stops sending requests to nodes it considers unhealthy, while distributing the remaining traffic across the nodes that pass. The key decision is which signal should mark a node as down, because APISIX offers two different ones that suit different failure patterns. APISIX runs *active* health checks by probing upstream nodes over http, https or tcp on a schedule. The default interval is 1 second, and an HTTP node is marked unhealthy after 5 failed probes. *Passive* checks take a different approach: they judge the responses APISIX is already proxying, so a node that returns repeated errors to real clients is removed without a separate probe. Teams with a dedicated health endpoint usually want active checks, while teams whose services fail only on certain real request paths often need passive checks. Confirm the probe path, expected status codes and failure thresholds against the health check tutorial before you rely on the defaults.
Health checking has a timing property that surprises many teams: it starts only after an upstream receives its first request. A freshly configured upstream therefore has no active probing running until traffic arrives, so a node that is already down can receive the first requests. Test this during rollout by sending a request to each upstream before you cut traffic over, and by watching the status view from the Control API at GET /v1/healthcheck to confirm each node's state.
The second behavior matters for incident planning. If no upstream node is healthy, APISIX keeps sending traffic to the upstream instead of failing closed. That protects against a health check misconfiguration taking an entire service offline, but it means clients will still reach broken nodes during a full outage. Plan alerting on the Control API status, and decide with the service owners whether a complete loss of healthy nodes should trigger a circuit breaker or a fallback response instead.
Active, passive and retry behavior compared
Active health checks have APISIX send its own requests to each upstream node on a schedule and mark the node unhealthy when the probes fail. APISIX supports http, https and tcp probes. The documented defaults are a 1-second interval and an unhealthy decision after 5 HTTP failures, so a node that stops answering its probe is removed from rotation without waiting for client traffic to fail. Probing starts only after the upstream first receives a request, which means a quiet route will not show health state until it is used.
- What it detects: nodes that are down, refuse connections or return failing status codes on a dedicated health endpoint.
- What it costs: a steady stream of probe requests per node, multiplied by the number of upstreams and gateway instances that run checks.
- Probe design: choose a lightweight path that reflects real readiness, such as a dependency-aware health endpoint, rather than a static page.
- Protocol choice: tcp probes confirm that a port accepts connections but say nothing about application health, so use http or https where the service exposes a meaningful endpoint.
- Where it fits: services with a stable health endpoint and a clear failure signal, especially when clients have no retry logic of their own.
Before relying on active checks, confirm the probe path, expected status codes and interval against the service owner's readiness definition.
Passive health checks create no extra traffic. APISIX watches the responses that proxied client requests receive from each upstream node and counts failures against configured thresholds. A node that repeatedly returns errors on real traffic is taken out of rotation, even if its dedicated probe path still responds. This makes passive checks good at catching faults that only appear under real request patterns, such as a broken code path or a database timeout on one endpoint.
- What it detects: errors on live requests, including status codes and connection failures seen during proxying.
- What it costs: no additional probe requests, but some client requests are sent to a failing node before it is removed.
- Reaction time: depends on how much traffic a node receives, so low-traffic routes may take longer to mark a node unhealthy.
- Threshold tuning: set the failure counts and status codes that count as failures so that a single legitimate 404 or 429 does not eject a healthy node.
- Where it fits: high-traffic APIs where real responses give a faster and more representative signal than probes.
Test the thresholds with the status codes your API returns during normal business errors, because those are the codes passive checks will count.
Retries do not detect failure. They respond to it by sending a failed request again, usually to a different node, so that a client receives a successful response when a single node fails mid-request. APISIX upstreams support retries alongside passive checks, and the retry count and the conditions that trigger a retry are part of the upstream configuration. Retries help with transient errors but can multiply load on a struggling service, so they need limits and clear rules about which requests are safe to repeat.
- What it fixes: a single bad node or a brief connection error that a second attempt on another node would avoid.
- What it adds: extra upstream requests during failures, which can increase pressure on a service that is already degraded.
- Idempotency: retry only methods and operations that are safe to repeat, such as GET, and treat non-idempotent writes with care.
- Interaction with checks: a retry on a node that passive checks have not yet ejected will still hit that node, so the two settings should be reviewed together.
- Where it fits: read-heavy APIs with several healthy nodes and clients that cannot tolerate a single transient failure.
Review the retry count and retry conditions with the service owner, and confirm in the vendor documentation how retries interact with the health state of each node in your version.
In practice the three mechanisms are combined. Active probes cover quiet routes and give an explicit health signal, passive checks react to real traffic without extra load, and retries hide single failed attempts from clients. None of them replaces the others, and each has a blind spot the other can cover. A combined setup needs deliberate values for each layer, otherwise probes may remove a node that passive checks would have kept, or retries may keep sending requests to a node that is clearly failing.
- Coverage: probes catch idle failures, passive checks catch live failures, and retries mask individual errors.
- Load budget: account for probe traffic per node, per upstream and per gateway instance when sizing the platform.
- Failure mode: if all nodes are unhealthy, APISIX keeps sending traffic to the upstream, so checks alone do not stop requests from reaching a fully failed pool.
- Ownership: assign one team to own thresholds, intervals and retry rules for each upstream.
- Review cadence: revisit values after service changes, since new endpoints or new error codes change what counts as failure.
Write the chosen values for each upstream into the design record and confirm them in a staging test that deliberately fails a node.
Load balancing types and health check settings and when each fits
This table compares the upstream options APISIX offers for distributing traffic and removing failing nodes, with the setting each one needs and the trade-off it brings. Read each row as a question: if your traffic must stay on one client or session, or if it must spread evenly, which option fits. Confirm every default value in the upstream and health check documentation before you copy it into a configuration.
| Option | What it does | Trade-off | When it fits |
|---|---|---|---|
| Active health checks | Probes each upstream node over http, https or tcp. The default interval is 1 second, and a node is marked unhealthy after 5 HTTP failures. | Adds probe traffic to every node. Checks start only after the upstream first receives a request, so a cold upstream is not probed until then. | Backends with a reliable health endpoint, where you want failing nodes removed even when no client is currently hitting them. |
| Passive health checks | Judges each node from the responses APISIX proxies to it, without sending extra probes. | Only failures seen in real traffic count, so a node can fail a few requests before it is removed. Confirm the failure thresholds in the docs. | Services where a dedicated health endpoint is unavailable or does not reflect real request failures. |
| Retries | Sends a failed request to another node in the upstream, within the retry settings you configure. | Retries can repeat non-idempotent writes. Limit them to requests that are safe to repeat and confirm the retry conditions in the upstream docs. | Read-heavy or idempotent APIs where a single bad node should not return an error to the client. |
| Timeouts | Sets how long APISIX waits for connection, send and read operations before it treats the attempt as failed. | Short timeouts remove slow nodes sooner but can cut off legitimate long requests. Set them from measured service latency. | APIs with a known latency profile, such as fast lookups versus long report generation on the same gateway. |
| Round robin style balancing | Distributes requests across healthy nodes in turn or by weight. Confirm the exact default balancing type in the upstream docs. | Spreads load evenly but gives no affinity, so a client's requests can land on different nodes each time. | Stateless services where any node can serve any request. |
| chash by consumer | Hashes on the authenticated consumer so the same consumer is routed to the same node. | Keeps per-consumer locality and cache hits, but load can skew if a few consumers send most of the traffic. Requires consumer authentication upstream. | Per-tenant or per-client caches and workloads where each consumer's state lives on one node. |
| chash by cookie | Hashes on a named cookie so a browser session stays on the same node. | Gives session stickiness at the cost of even distribution. Nodes that fail lose their sessions unless the application stores state elsewhere. | Web applications that keep session state in node memory and cannot yet move it to a shared store. |
| chash by header | Hashes on a named request header, such as a tenant or account identifier. | Routing depends on the value your clients send, so validate or derive the header at the gateway. Uneven header values can skew load. | Multi-tenant APIs where a tenant identifier header should pin traffic to a node or shard. |
Why health checks start only after the first request
APISIX does not probe an upstream until that upstream receives a request. Active health checks are tied to traffic, so a newly created upstream with no clients behind it has no health state to report. Until the first request arrives, the node status returned by GET /v1/healthcheck on the Control API may show nothing useful, and a node that is already down will not be marked unhealthy in advance. For a cold upstream, the first real request is the moment the gateway begins to learn whether the backend is reachable.
This changes how you test failover. A drill that stops a backend and expects APISIX to route around it will fail if nothing has sent traffic through that upstream since the gateway started, because the checker never had a reason to run. Before a failover test, send a test request through the route that uses the upstream, then confirm the health status through the Control API. Repeat the check after the backend is stopped, and record how many requests reach the failed node before traffic moves. Note that if no node is healthy, APISIX keeps sending traffic to the upstream, so a fully failed upstream will not be bypassed automatically.
The same behavior affects readiness after a deployment. A release that routes new upstreams into APISIX can look healthy in the Admin API while the gateway has not yet checked any node, so the first users may be the first probes. Add a synthetic request to each route as a post-deployment step, and treat a successful response from the route as the readiness signal rather than the absence of errors in logs. For monitoring, alerts built only on health check state will stay silent for idle upstreams. Pair them with request-level metrics and error rates for each route, and confirm with your own environment or the health check documentation how the checker behaves for your APISIX version and configuration before you rely on it.
How to configure upstream health checks in APISIX
This sequence takes you from defining upstream nodes to confirming that APISIX tracks their health correctly. Each step ends with a check you can run, so you know the gateway is routing around failing backends before you rely on it in production.
Define the upstream nodes
Create an upstream object in APISIX that lists each backend node by host and port, with a weight for each node. Your platform or integration engineer does this through the Admin API or your configuration-as-code workflow, using the addresses the gateway can reach on your network. Confirm the upstream exists by reading it back from the Admin API and checking that every node address matches your service inventory. Do not move to the next step until each backend appears exactly once with the expected port.
Choose the load balancing type
Set the upstream type to match how traffic should spread across nodes. The default round-robin behavior suits stateless services. Use the chash type when a client needs to keep reaching the same node, and hash on a consumer, a cookie or a header. Note that chash only pins requests while the node set is stable, so ask the team to document how a node added or removed mid-session will behave. Confirm the choice by sending a few requests with the same key and checking that they reach the same node.
Enable active health checks with probe settings
Add an active health check block to the upstream and set the probe type (http, https or tcp), the probe path for HTTP checks, the interval, and the thresholds for marking a node unhealthy or healthy again. The documented defaults are a 1-second interval and an unhealthy state after 5 HTTP failures, so change them only with a reason recorded in your change ticket. Ask the service owner which path returns a fast, dependency-light response. Confirm the setting by reading the upstream back and checking that the probe fields match your design.
Enable passive health checks and retries
Configure passive checks, which judge the responses APISIX sees while proxying real traffic, and set the retry count and the conditions that trigger a retry, such as specific HTTP status codes or connection errors. Retries should only apply to requests that are safe to repeat, so confirm with the service owners which methods and endpoints qualify. Passive checks depend on live traffic, so they add nothing to an idle upstream. Confirm the setting by reviewing the upstream object and listing the status codes that will count as failures.
Send test traffic to start checks
Health checking begins only after the upstream receives its first request, so send a controlled set of test requests through the route that uses this upstream. Include requests that succeed and, in a test environment, requests that hit a deliberately stopped node so you can watch the failure path. Your operations team runs this step with the service owner present. Confirm that checks have started by seeing the first requests recorded in the gateway access logs for each node.
Confirm node status at the Control API
Query GET /v1/healthcheck on the Control API to view the health state of each node in the upstream. Expect every healthy node to report as healthy, and compare the list against the node inventory from the first step. Then stop one test node and repeat the query after the probe interval has passed to confirm it is marked unhealthy. Restore the node and confirm it returns to healthy. Keep the captured output with your change record as evidence of the working state.
Verify traffic avoids failing nodes
With one test node stopped, send a batch of requests through the route and confirm from the access logs that none of them reach the failed node. Also test the case where every node is unhealthy, because APISIX keeps sending traffic to the upstream in that situation rather than returning an error. Decide with the service owner whether that behavior is acceptable or whether another control, such as the circuit breaker plugin, should cover it. Record the results before the configuration moves to production.
Document and monitor the health configuration
Store the final upstream definition, probe settings, retry rules and test results in version control so the configuration can be reviewed and rolled back. Set up monitoring that alerts when the Control API shows nodes as unhealthy, and assign a team to respond. Ask your platform team to confirm how the upstream configuration is promoted between environments and who approves changes. The sequence is complete when the documented configuration reproduces the test results in a second environment.
Security and operational risks to plan for
Health checks and load balancing change where traffic goes when something breaks, so they also change who can see backend state, how much load failures create, and what operators can see during an outage. The five risks below connect those effects and each one has a concrete check to run before go-live.
Control API health view exposure
The health status of every upstream node is available from the Control API at GET /v1/healthcheck. That view reveals internal backend addresses and which services are failing, which is useful to an attacker mapping your estate. Confirm that the Control API listens only on a management network, that access is restricted to operator identities, and that the view is not proxied through a public route. Treat the health output as internal data in your logging and access reviews.
Probe endpoints that leak or overload
An active health check sends requests on a fixed interval, and the default interval is 1 second. A probe path that returns verbose errors, stack traces or version details leaks information, and a probe path that runs database queries multiplies load across every node and every gateway instance. Use a dedicated lightweight health endpoint that checks only what decides routing, returns a minimal status code, and sits outside authentication only when it exposes nothing sensitive.
Retries that amplify load
Retries help with transient errors but can multiply traffic during an incident. If a backend is slow and every client retries, the upstream receives several times the normal volume at the moment it has the least capacity. Set retry counts conservatively, limit retries to status codes that indicate a transient fault, and check the total request budget with your clients. Test the behavior under a simulated backend slowdown before you rely on it.
Duplicated non-idempotent writes
A retry after a timeout can reach a backend that already processed the first request, so a payment, order or ticket creation may happen twice. Retries are safe for idempotent methods such as GET, but a POST or PATCH needs an idempotency key or a server-side deduplication check. Review every upstream route that accepts writes, decide which methods may be retried, and confirm with each service owner how duplicates are detected.
Hash-key stickiness concentrating load
A consistent hash on consumer, header or cookie keeps the same client on the same node. If one consumer, tenant or large customer sends a disproportionate share of traffic, that node carries the load while others sit idle, and a failure on that node affects the heaviest users first. Review the distribution of hash keys before launch, decide whether a few high-volume keys need a different balancing strategy, and confirm how a remapped key behaves when a node leaves the pool.
Monitoring gaps when all nodes fail
If no upstream node is healthy, APISIX keeps sending traffic to the upstream rather than returning an immediate failure. Dashboards that only count gateway errors may show normal traffic while every request times out at the backend. Alert on the health status of each node from the Control API, not only on gateway response codes, and make sure someone is paged when an upstream has zero healthy nodes. Test this alert during your rollout.
Questions teams ask before configuring APISIX health checks
Can our own team configure APISIX health checks without a partner?
Yes, a team with gateway and backend experience can configure health checks on an upstream directly through the Admin API or configuration files. The work itself is small: you set the check type, the probe path, the interval, and the thresholds that mark a node unhealthy or healthy again. The harder part is deciding those values for each backend and proving that failover behaves as expected under load. Teams that already run APISIX in production often handle the basic setup in-house and bring in outside help for the testing and rollout plan. The Apache APISIX health check tutorial is the natural starting point for the configuration itself.
What drives the effort of a health check engagement?
Effort follows the shape of your traffic and your backends. The main drivers are the number of upstreams, the protocols they use (HTTP, HTTPS or TCP), whether each service has a suitable health endpoint, how many environments need matching settings, and how much failure testing you require. Each upstream needs its own probe path, thresholds and load balancing type, so a gateway with dozens of services takes more design and verification than one with three. Ask any partner to break the quote down by upstream count, protocol group and test scenarios, so you can see which items move the total.
What do we need to provide before the work starts?
Most of the work depends on inputs from your side. You will need a staging environment that resembles production, a health endpoint or agreed probe path for each backend that reports readiness accurately, and access to the logs and metrics that show how nodes behave under failure. Monitoring should already capture upstream errors and latency, so changes can be compared before and after. Name the owners of each backend service, since their answers decide what counts as healthy. Confirm these items in your own environment before you agree on a scope.
Why don't APISIX health checks start until the first request arrives?
Active health checks only begin probing an upstream after that upstream receives its first request, so a freshly configured upstream with no traffic has no health status yet. This matters for testing: if you expect a dead node to be marked unhealthy right after deployment, send a request through the route first. It also matters for capacity planning, because a quiet route will not show probe activity. Your test plan should include a warm-up request and a check of the status view afterward. The upstream documentation describes how upstreams and their node settings fit together.
How does consistent hashing affect session stickiness?
Consistent hashing (chash) sends requests with the same key to the same node, so the type you choose sets how sessions stick. The chash type can hash on a consumer, a cookie or a header. Consumer hashing keeps one client on one node, cookie hashing follows a session cookie, and header hashing follows a value such as a tenant or user identifier. When a node is removed or marked unhealthy, requests that hashed to it move to another node, so in-memory session state on the old node is lost. Confirm with your application team whether sessions are stored externally before you choose a key, and test what happens when a node fails.
What does a fixed-scope engagement cover for health checks?
A fixed-scope engagement sets the upstreams, protocols, environments and deliverables in writing before work begins. For health checks, that typically means reviewing each backend's health endpoint, defining the active and passive check settings, configuring load balancing and retries, testing node failure and recovery in staging, and documenting how to read the Control API status view at GET /v1/healthcheck. Agree in writing whether production rollout and ongoing tuning are included or handled through a retainer. DeployOpen does not publish prices, so a quote follows from the scope you agree on.
How to test failover before production traffic depends on it
Test failover in staging before production traffic relies on it. The core test is simple: stop one backend node in the upstream, then confirm that APISIX marks it unhealthy and that requests stop reaching it. Use the Control API at GET /v1/healthcheck to read the node status directly, and send a steady stream of requests through a route bound to the upstream so you can see where traffic lands. Note how long the node takes to be marked unhealthy under your configured active check interval and failure threshold, and compare that with what your clients will tolerate. Remember that health checks start only after the upstream receives its first request, so generate traffic before you stop the node or the test will prove nothing.
Next, restore the node and confirm recovery. The node should return to healthy status and receive traffic again, and you should confirm this from the same status view rather than assuming it from a successful response. Then run the case that many teams never test: every node unhealthy at once. The health check documentation states that APISIX keeps sending traffic to the upstream when no node is healthy, so decide in advance whether that behavior is acceptable, and whether your clients or your upstream service can handle the load and errors it produces. Confirm the behavior you observe matches your expectation, and record it as a finding if it does not.
Record every run in a test log: the upstream configuration, the health check settings in use, the time each node was stopped and restored, the status returned by the Control API, and the error rates your clients saw. Repeat the full sequence after any change to upstream nodes, health check settings, retries, or the APISIX version itself, since upgrades can change defaults and behavior. Confirm the exact settings and version support for your environment with the APISIX health check documentation before you finalize the plan, and keep the results with your change records so the next team can rerun the same test.
How DeployOpen scopes and delivers APISIX health check configuration
DeployOpen plans, implements, tests and documents Apache APISIX upstream health checks and load balancing inside your own infrastructure. We start from your backends, failure modes and traffic patterns, then agree the configuration in writing before any change reaches production.
How we scope
- Inventory each upstream, its nodes, health endpoints and expected failure behavior before choosing check settings.
- Agree in writing which checks are active, passive or both, and which backends are in scope.
- Define hashing needs for consumer, header or cookie affinity, and where session stickiness is actually required.
How we build
- Configure upstream health checks, retries and load balancing types as code, stored in your version control.
- Test node failure, recovery and all-unhealthy behavior in a non-production environment with your backends.
- Verify that the Control API health status view reflects what the gateway actually observes during each test.
How we hand over
- Deliver a runbook covering check intervals, thresholds, the Control API health endpoint and operator steps.
- Document known limits, including that checks begin only after an upstream first receives a request.
- Walk your platform and operations teams through alerting, changes and rollback before the engagement closes.
How we support
- Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
- Ongoing support, upgrades and continuous testing for your gateway are available as a retainer.
- Retainer reviews can revisit upstream settings as your backends, traffic and APISIX releases change.
