APISIX circuit breaking with api-breaker plugin

How the APISIX circuit breaker stops cascading failures

api-breaker marks an upstream unhealthy after 3 responses with configured status codes (default 500), and while the breaker is open it returns the response set in `break_response_code` instead of calling the upstream. Teams need this when an upstream starts failing and callers keep retrying into it. Each retry consumes gateway workers, upstream connections and client time on a service that cannot recover under that load, so one degraded dependency can exhaust capacity in every service that calls it. The plugin is configured on the routes it protects, so it runs in the request path before the upstream is contacted and judges only the responses the gateway has already seen. Read the api-breaker plugin reference before you write the configuration, and confirm the exact field names and accepted values for your APISIX version.

The breaker has three behaviors you need to plan for. While it is open, clients receive the configured break response and the upstream receives no traffic, which gives the service room to recover. The open period backs off for up to 300 seconds by default, and the max_breaker_sec setting caps that backoff. The breaker closes again after 3 healthy responses. The status code list is the decision that matters most. A list containing only 500 will miss a service that fails with 503 or 504, while a list that includes 429 can open the breaker because of upstream rate limiting, which is a sign of load rather than failure. Decide per route which codes mean the service cannot serve, and test the cutoff with deliberate failures in staging so you know how quickly callers will see the break response.

The breaker works alongside, not instead of, upstream health checks. Active health checks probe nodes on a schedule (default 1-second interval, unhealthy after 5 HTTP failures per the health check documentation), passive checks judge proxied responses, and health checking starts only after an upstream receives its first request. Health checks remove failing nodes from the load balancer, while api-breaker protects the route as a whole and returns a controlled response to callers. Retries need the same coordination. If retries are enabled, confirm how many attempts the gateway makes before the breaker trips, so retry traffic does not keep feeding a service that the breaker should be protecting. For a practical check, stop one upstream node in a test environment, send steady traffic, and record when the break response begins, when the upstream stops receiving requests, and when traffic resumes after the service is restored. Those three observations show whether the thresholds match your service.

Configuration options and when each fits

This table compares the api-breaker settings a team tunes, the default for each, and the traffic patterns that justify changing it. Read each row as a decision: keep the default unless the service's error behavior, retry pattern or recovery needs differ. Confirm exact field names against the api-breaker plugin documentation before writing configuration.

Setting	Default	What it controls	When to keep the default or change it
unhealthy.http_statuses	500	Response status codes that count as failures toward the breaker threshold	Keep 500 for most APIs. Add 502, 503 or 504 only if upstream overload or gateway errors should trip the breaker. Remove 500 if the service returns it for non-transient bugs that retries will not fix.
unhealthy threshold (failure count)	3 failing responses	Number of configured-status responses before the upstream is marked unhealthy	Keep the default for low-volume APIs where three failures signal a real fault. Consider a higher count for services with occasional, harmless 5xx responses, and confirm the exact field name in the plugin docs.
break_response_code	Set per route (no default in the supplied facts)	Status code the gateway returns while the breaker is open	Choose 503 for temporary unavailability so clients and load balancers treat it as retryable. Use a 4xx only when a fixed client-side response is intended. Confirm the field is required in your APISIX version.
max_breaker_sec	300 seconds	Upper bound on how long the backoff grows while the upstream keeps failing	Keep 300 seconds for services that recover within minutes. Lower it for user-facing paths where long outages are worse than brief retries. Raise it for dependencies with slow, known recovery windows, and test the effect on client experience.
Healthy-response recovery count	3 healthy responses	Successful responses needed before the breaker closes again	Keep 3 for most services. Raise it for flaky dependencies that pass a few probes then fail again. Lower it only after load testing shows the upstream stays stable under returning traffic.
Interaction with health checks	Health checks off unless configured	Active and passive checks remove failing nodes from load balancing, separately from the breaker	Use both when a route has several nodes: health checks isolate a bad node, the breaker stops traffic to the whole upstream. Confirm how your version reports breaker state before relying on it for alerts.

How api-breaker differs from health checks and retries

The api-breaker plugin is attached to a route and watches the responses that route returns to clients. It counts responses whose status codes match the configured list, and when the threshold is reached it stops forwarding requests and returns a configured response for as long as the breaker stays open. It does not probe anything. It only learns from traffic that is already flowing through the route, so it reacts after clients have seen failures and it applies to the route, not to a specific node.

Trigger: unhealthy after 3 responses with configured status codes (default 500).
Response while open: returns break_response_code to the client instead of calling the upstream.
Backoff: stays open for up to 300 seconds by default, controlled by max_breaker_sec.
Recovery: closes again after 3 healthy responses.
Scope: a route-level decision that does not remove individual upstream nodes from rotation.

Before you rely on these defaults, confirm the status codes, thresholds and backoff values in the api-breaker documentation for the APISIX version you run, and test what clients receive while the breaker is open.

Setting up api-breaker on a route

This sequence adds the api-breaker plugin to a test route, sets the failure codes and break response, and proves the breaker opens and recovers under controlled upstream errors. The result is a route whose behavior under failure your team has observed before any production traffic depends on it.

Confirm deployment mode and Admin API access
Your platform engineer confirms which APISIX deployment mode runs in the test environment: traditional, decoupled or standalone. In traditional mode, client traffic goes to port 9080 and the Admin API listens on port 9180, so the engineer checks that the Admin API answers from the operator workstation and that an admin key is available outside the repository. Standalone mode changes the path, because routes live in conf/apisix.yaml or conf/apisix.json and are hot-reloaded, so the team edits that file instead of calling the Admin API. The step is done when the team can list existing routes and knows exactly where route configuration will be written.
Create a test route to a controllable upstream
Create a dedicated test route, with a path that no production client uses, that forwards to an upstream you control. A simple test service that can be told to return a chosen status code works well, because it lets you produce errors on demand without touching a real dependency. Keep the route in a non-production gateway instance and confirm that a normal request returns the upstream's success response through the gateway. The step is done when a 200 response is returned through the test route and the team has recorded the route ID for later changes.
Enable api-breaker with failure codes and break response
Add the api-breaker plugin to the test route. Set the status codes that count as failures (the default is 500), set break_response_code to the status the gateway should return while the breaker is open, and review the other configured thresholds in the plugin documentation before you save. Apply the change through the Admin API or the standalone file, using the same method your team will use in production. The step is done when the route configuration returned by the Admin API or file shows the api-breaker block with the failure codes and break response you chose, and the change is stored in version control if your workflow requires it.
Trigger controlled upstream errors
Switch the test upstream into a failing state, for example by making it return the configured failure status on every request. Send a steady series of requests through the test route and record each response code. The expected pattern is that the upstream responses are passed through until the failure threshold is reached, after which the gateway returns the break response without contacting the upstream. Run this only in a non-production environment with no shared traffic, so that the errors do not affect other users. The step is done when your recorded responses show the change from passed-through errors to the break response.
Verify the break response and gateway logs
Confirm that the response returned while the breaker is open carries the break_response_code you configured and that the upstream receives no requests during that window. Check the APISIX error and access logs for the matching entries, and confirm the upstream's own logs show no traffic while the breaker is open. Record the time the breaker opened so the recovery check has a clear reference point. The step is done when the team can point to log lines that show the open state, the break response and the absence of upstream calls.
Restore the upstream and confirm recovery
Return the test upstream to healthy responses and keep sending requests through the route. Confirm the breaker does not reclose on the first healthy response: the plugin documentation describes recovery after a set number of healthy responses, and the backoff period must also pass before the gateway lets traffic through again. Record when normal responses resume and compare the timing with the logs. The step is done when the route returns the upstream's normal responses and the logs show the breaker closed after the expected healthy responses.
Document the settings and confirm the production plan
Write down the failure codes, break response, backoff ceiling and recovery behavior you observed, and list which settings your team must review for each production route, since the right thresholds depend on each service. Confirm with the vendor documentation and your own environment how api-breaker interacts with the health checks and retry settings on the same upstream, because both can change which requests reach the backend. The step is done when the test results and open questions are recorded for the change review.

Security and operational risks of a misconfigured breaker

A breaker protects upstreams only when its thresholds, responses and visibility match how the service really fails. The cards below cover four ways a poorly tuned api-breaker causes harm, and the logging, alerting and review practices that catch each one early.

Break responses that leak detail

The break_response_code value controls what clients see while the breaker is open, and a custom break_response_body or header can expose upstream hostnames, stack traces or version strings if copied from an error page. Use a generic body that names no internal component, and confirm that any response headers added by other plugins on the route do not reveal the upstream. Review the break response in a test request during each change window, since a breaker that fires is the moment an attacker is most likely to probe.

Breakers that trip on client errors

The breaker counts only the status codes you configure in unhealthy, and the default is 500. If a team adds 4xx codes such as 401 or 404 to that list, a burst of bad tokens or scanning traffic can open the breaker and block valid users on the same route. Keep client errors out of the unhealthy list, and log which codes were counted so that a trip caused by callers can be told apart from one caused by a failing upstream.

Breakers too loose to trip

If the unhealthy status codes do not match what the failing service actually returns, the breaker never opens and the upstream keeps taking the full load during an outage. A timeout, a gateway error from the upstream, or a process that returns 200 with an error body will not count as a failure unless your configuration matches it. Test the breaker against a deliberately failing upstream in a non-production environment and confirm that it opens after the configured number of responses.

Losing visibility while the breaker is open

A rejected request returns quickly and can look like a healthy gateway in dashboards built only on latency and throughput. Record the response code and route for every break response, alert when a breaker opens and when it stays open beyond your expected backoff, and review the log after each event. Without these signals, teams learn about a tripped breaker from end users rather than from their monitoring.

Reviewing open-state events

A breaker that opens once and recovers may still hide a recurring fault, and a route with repeated trips needs a root cause review rather than tuning of thresholds. Keep a short record of each open event with the route, the upstream, the codes counted and the time to recovery, and review those records with the service owner. Revisit thresholds only after the cause is understood, so the breaker does not drift toward hiding failures.

Questions teams ask before building circuit breaking

Is the api-breaker plugin enough on its own to stop cascading failures?

No. The api-breaker plugin stops traffic from reaching an upstream that is already returning the status codes you configured, but it does not fix why the upstream is failing. It works best alongside health checks that remove bad nodes from rotation, timeouts that stop slow calls from piling up, and retry limits that avoid multiplying load on a struggling service. Before you rely on it, confirm with your service owners which failure modes actually cascade in your architecture, because a breaker tuned only to 500 responses will miss slow responses, connection resets and timeouts that the plugin does not count as failures in the same way.

How long does circuit breaker setup and testing usually take?

The effort depends on how many upstreams need protection, how well their failure behavior is understood, and how much testing you require before production. The main drivers are agreeing on the status codes and thresholds per service, deciding the response callers receive while the breaker is open, and proving behavior under controlled failure. Plugin configuration itself is usually the smallest part. Ask any partner for a written scope that lists the services covered, the failure scenarios tested and the environments used, and ask for the timeline to be tied to those items rather than to a single number.

What do our platform and service owners need to provide?

You need named owners for each upstream service who can explain its normal error patterns, expected latency, idempotency of calls and acceptable fallback behavior. The platform team needs access to the gateway configuration, the ability to run controlled failure tests in a non-production environment, and a way to observe breaker state and upstream health in your monitoring stack. Without those inputs, thresholds tend to be copied from defaults and never revisited. Confirm early who approves changes to breaker settings, since a misconfigured breaker can block healthy traffic as easily as it protects a failing service.

Can our in-house team build and operate circuit breaking themselves?

Often yes, if the team already runs APISIX or another gateway in production, understands the upstream services and has a test environment where failures can be induced safely. The configuration is declarative and the plugin documentation covers the core settings. The harder parts are choosing thresholds that reflect real traffic, verifying recovery behavior and keeping settings in step with service changes. Your team should decide whether it has capacity for that work during a release cycle. A fixed-scope engagement can cover planning, configuration, testing and handover, while a retainer can cover ongoing review as services change.

How do we decide between a breaker setting and a fix in the service?

Use the breaker to limit damage while a service is failing, and fix the service when the failures are recurring or caused by its own code, data or capacity. If a service returns errors because a dependency is down, the breaker protects callers while that dependency recovers. If it returns errors because a query is slow or a pool is exhausted, the service needs a change. Ask the owning team to classify the top causes of failure for each upstream, then decide per cause. Record the decision so future incidents do not hide a defect behind a gateway setting.

How does the breaker interact with retries and load balancing?

Retries and breakers can conflict. If the gateway retries failed calls while the upstream is struggling, load increases just as the service needs relief, so retry counts and the breaker's failure threshold should be set together. Health checks operate separately: active checks probe upstream nodes on an interval, and passive checks judge proxied responses, and health checking starts only once an upstream receives a request. Confirm in the APISIX health check documentation how these settings apply to your upstream type, and test a full failure scenario so you can see which mechanism acts first.

What should we test before turning circuit breaking on in production?

Test the breaker against the failure patterns your services actually produce: sustained error responses, slow responses, a node going down and a dependency recovering. Confirm which status codes trip the breaker, what break_response_code callers receive, how long the breaker stays open under the backoff rules and how many healthy responses it needs before traffic resumes. Also test that callers handle the returned response correctly, that alerts fire when the breaker opens, and that the team knows how to inspect and reset state. Keep a documented rollback so you can disable the plugin quickly if it blocks healthy traffic.

Limits to confirm in your environment

The api-breaker plugin is simple to enable, but its behavior depends on how your APISIX deployment is run, how retries and load balancing sit in front of your upstreams, and how your services signal expected failures. Each card below is a point to confirm with the vendor documentation or a test in a non-production environment before the breaker protects live traffic.

Behavior in each deployment mode

APISIX runs in traditional, decoupled or standalone mode, and the breaker's state and configuration handling may differ between them. Confirm how a route's api-breaker settings reach each data plane node in your chosen mode, how long a configuration change takes to apply, and what happens during an etcd outage in traditional or decoupled mode. Test the exact mode you plan to run, because a passing test in a lab mode does not prove the same behavior in production.

Shared state across gateway nodes

A breaker that trips on one gateway node does not necessarily trip on the others. Confirm whether the failure count and open state are kept per node or shared, because the answer decides how quickly every node stops sending traffic to a failing upstream. Ask the vendor how many failing responses each node counts when traffic is spread across several nodes, and test this with more than one gateway instance behind your load balancer.

Interaction with retries

If the gateway or a client retries failed requests, each attempt can count toward the failure threshold and add load to a service that is already struggling. Map every retry layer between the client, APISIX, and the upstream, including retries configured in APISIX health checks or load balancing and retries in client SDKs. Decide which layer owns retries so the breaker sees a count that reflects real failures, and test a burst of errors to confirm the threshold is reached as you expect.

Interaction with load balancing

Load balancing across several upstream nodes can hide a failing node if the breaker counts responses at the route level. Confirm whether the breaker opens for the whole route or only affects traffic to the failing node, and whether it works alongside active and passive health checks. Health checks only start after an upstream receives its first request, so verify that your upstream nodes are warmed or tested before you rely on that combination.

Status codes your services use

The breaker counts only the response codes you configure, with 500 as the default. Services that return 503 for overload, 429 for rate limiting, or 504 for timeouts may need a different list, and counting client errors such as 404 can trip the breaker for reasons that are not an outage. List the expected failure codes for each route with the service owners, and decide which of them should count toward the threshold and which should pass through unchanged.

Break response and recovery

While the breaker is open, APISIX returns the configured break_response_code to clients. Confirm that your clients handle that code correctly, that your monitoring alerts on it, and that the backoff period, up to 300 seconds by default, fits your service's recovery time. Recovery requires consecutive healthy responses, so test a slow recovery scenario and confirm that clients do not hammer the upstream during the half-open window.

How the api-breaker plugin stops cascading failures

The api-breaker plugin marks an upstream unhealthy after 3 responses with configured status codes, and 500 is the default. Once that threshold is reached, APISIX opens the breaker and returns the status code set in break_response_code instead of forwarding requests to the failing service. Callers get a fast, predictable error, and the upstream gets room to recover instead of queuing more work it cannot finish. Before you enable the plugin, decide which status codes count as failures for each route. A 500 from a broken dependency is a clear signal, but a 429 or 503 may mean something else in your stack, and counting the wrong codes can open the breaker during normal load.

While the breaker is open, APISIX backs off before testing the upstream again. The backoff can grow up to the max_breaker_sec ceiling, which defaults to 300 seconds. Set this value per service based on how long a dependency typically takes to recover and how long your users can tolerate the fallback response. Recovery requires 3 healthy responses before the breaker closes. Confirm with the current api-breaker documentation exactly which requests are used to test recovery while the breaker is open, because that determines whether a service with no other traffic can ever come back on its own. Test this path explicitly in a staging environment with an induced failure rather than assuming it behaves the way the threshold does.

The api-breaker plugin works alongside, not instead of, upstream health checks. Active health checks send probes over http, https or tcp, with a default interval of 1 second and a default of 5 consecutive HTTP failures before a node is marked unhealthy. Passive checks judge the responses APISIX proxies, and health checking only begins after an upstream receives its first request. The breaker reacts to response patterns on a route, while health checks decide which nodes receive traffic at all. Plan for both layers: a single bad node should be removed by health checks, and a dependency that fails across all nodes should trip the breaker. Review the APISIX health check documentation and load test the combined behavior, including how retries interact with the breaker, before relying on either one in production.

How DeployOpen scopes and delivers circuit breaking for APISIX

DeployOpen plans, implements and documents api-breaker configuration on APISIX gateways that run in infrastructure you control. Each engagement starts with a written scope that ties breaker thresholds to the upstream services they protect, so the protection is tested before it is relied on.

How we scope

Inventory which upstream services need a breaker and which status codes count as failures for each route
Agree the break response code clients receive while a breaker is open, and how front-end and API consumers should handle it
Record how api-breaker, active and passive health checks and retries will divide responsibility, in the written scope

How we build

Configure api-breaker on the selected routes in your APISIX environment, with failure thresholds and max_breaker_sec values reviewed by your service owners
Test breaker behavior against controlled upstream failures, including open, backoff and recovery transitions in a non-production environment
Store route and plugin configuration as code in your version control system so changes are reviewed and reproducible

How we hand over

Deliver a runbook describing breaker states, the Admin API and Control API checks operators use, and how to read upstream health status
Document the agreed thresholds, the reasoning behind each value and the test cases used to verify them before handover
Walk your operations and service teams through alerting, escalation and the steps to change a breaker setting safely

How we support

Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts
Ongoing support, APISIX upgrades and continuous testing of breaker behavior are available as a retainer
Confirm with your team which upgrades or dependency changes will trigger a review of breaker settings and tests

Apache APISIX circuit breaking with api-breaker