APISIX canary and blue-green release setup

How Apache APISIX canary releases work at the gateway

Teams that ship new API versions often want to expose them to a small share of real traffic before a full switch, without changing every client. The traffic-split plugin in Apache APISIX covers this job: it lets the gateway send part of the requests for a route to one upstream and the rest to another, so the release decision moves from client code into gateway configuration. Teams need it when a new version carries enough risk that a single cutover would be too blunt, or when they want to switch back quickly without redeploying anything.

Two patterns sit on top of the same mechanism. A *weighted canary* sends a fixed percentage of requests to the new version and raises that share as error rates and latency stay acceptable. A *header-matched blue-green switch* routes only requests that carry a chosen header, such as one set by internal testers, to the new version, while everyone else stays on the current one. Both patterns depend on the gateway seeing the request early and holding the routing rules, so the services themselves do not need to know a release is underway. Confirm in the APISIX documentation for your release how the plugin's match conditions, weights and upstream definitions are expressed, since field names and rule ordering have changed across versions.

Before you rely on this pattern, decide who owns the weight changes, what signals justify moving them, and how a rollback is triggered and tested. Check whether session affinity matters for your clients, because a user bouncing between versions mid-flow can see inconsistent behavior. The Apache APISIX getting started guide is a useful place to confirm the baseline setup in the release you run, and any canary plan should be rehearsed in a non-production environment first.

Choosing between weighted canary and blue-green switching

A weighted canary sends a fixed share of live requests to the new version and the remainder to the current one. It fits when the change affects shared logic, data handling or performance, when you cannot predict which users will hit a given code path, and when you want production traffic to reveal problems that tests missed. The gateway makes each routing decision from the configured weights, so client applications need no changes. Increases are usually staged by hand in configuration, which keeps the rollout controlled but places the pace in your team's hands.

Good fit: changes where real production load or varied inputs matter more than scripted tests.
Gradual exposure: an illustrative schedule might step from a small share to a larger one across several changes, but the steps and pauses should come from your own risk assessment.
Mixed responses: a single user may receive responses from both versions across consecutive requests, so session-dependent behavior needs checking.
Longer partial exposure: the rollout stays partial for longer, which lengthens the period in which defects can reach some customers.

Confirm with application owners whether any session state, cache or database schema is shared between versions before setting the first weight.

Traffic-split configuration options and when each fits

This table compares the main choices for configuring a traffic split in APISIX: weights, match conditions, where the upstreams are defined, and how the split sits alongside authentication and rate limits. Read each row as a decision: the right column names the situation where that option fits and what to confirm in the plugin documentation for your APISIX version before relying on it.

Configuration choice	What it does	When it fits and what to confirm
Weighted split per upstream	Sends a fixed percentage of matching requests to each upstream by weight	Gradual canaries where a small share of live traffic tests the new version. Confirm how weights are normalized and whether weights can be changed without a reload.
Header match conditions	Routes requests to an upstream only when a named header or cookie has a given value	Blue-green switching and internal testing, where selected users or test clients reach the new version. Confirm the supported match operators and how the condition is evaluated for each version.
Variable match conditions	Matches on gateway variables such as URI, method or client address	Splitting by path or region during a staged rollout. Confirm which NGINX variables are available in your APISIX release and test each condition on a staging route.
Upstream defined on the route	Upstream and split rules sit directly in the route definition	A small number of routes with simple rollouts. Easy to read, but each route carries its own split logic, so changes multiply across many routes.
Upstream defined in a plugin config or shared rule	Split rules are reused across several routes through a shared plugin configuration	Many routes that roll out together. Confirm how plugin configs and precedence apply to the traffic-split plugin in your version before changing shared rules.
Interaction with authentication plugins	Authentication runs before the split decides the upstream, depending on plugin order and precedence	When each version expects different credentials or claims. Confirm the plugin execution order and test that both versions receive the same identity context.
Interaction with rate limiting plugins	Limits count requests against a key such as consumer, IP or route, regardless of which upstream answers	When limits must hold across both versions during a rollout. Confirm whether counters are shared between upstreams and set limits before shifting traffic.

Setting up a canary release on your APISIX routes

This sequence moves a share of live traffic to a new service version through the gateway, so the team can confirm the release behaves correctly before every request reaches it, and can return all traffic to the stable version without a redeploy.

Confirm the stable and candidate upstreams
The platform team registers the current production version as the stable upstream and deploys the new service version as a separate upstream, with its own address or service name. Both must be reachable from the gateway nodes and pass the same health checks. The step is done when a direct request to the candidate upstream from a gateway node returns the expected response and the stable upstream still answers as before. Check the health check configuration on both upstreams, because a candidate that fails checks will receive no traffic.
Add the traffic-split plugin to the route
An engineer adds the traffic-split plugin to the existing route through the Admin API or your configuration-as-code repository, with the stable upstream as the default and the candidate upstream as a weighted alternative. Start with a small weight for the candidate. Keep the route's existing authentication and rate limiting plugins in place so the canary is subject to the same policies as stable traffic. The step is done when the route configuration is applied and the gateway accepts it without errors in its logs.
Start with a small weighted or header-matched share
Begin with a small weight for the candidate version, or add a match rule on a request header such as one set by internal testers, so that only requests from selected callers reach the new version. Choose the header-match option when you want deliberate test traffic rather than random sampling. Confirm the exact match syntax and weight fields against the traffic-split documentation for your APISIX version before applying them. The step is done when the change is in place and the team has recorded the expected split.
Verify routing with test requests
Send a set of test requests through the gateway, including requests with the match header and requests without it. For each response, confirm which upstream served it by checking a response header you add for diagnostics, the upstream address in the access log, or a version field in the response body. Repeat the test enough times to see the weighting in practice. The step is done when the observed split matches the configured split and no request reaches an unexpected upstream.
Watch error rates and latency for the candidate
Compare status codes, latency and error logs for the candidate and stable upstreams over the observation window your team has agreed. Use the gateway's logging and metrics plugins so each request is tagged with the upstream it used, which lets you separate the two populations. The step is done when the candidate's error rate and latency are within the thresholds your team wrote down before the release, and any anomalies have been investigated.
Define promotion and rollback checks in advance
Before increasing the candidate share, the release owner and the service owner agree on the promotion criteria and the rollback trigger, such as error rate, latency, or a business metric, and who can change the route. Record these in the release ticket. Rollback means setting the candidate weight to zero or removing the match rule, then applying the change. The step is done when the criteria are written down, the rollback change has been tested in a non-production environment, and the person on call knows the exact steps.
Promote or roll back and record the result
Increase the candidate share in stages if the checks pass, or revert the split if they fail. Once the candidate carries all traffic, update the stable upstream to the new version and remove the split, or keep the split for the next release. Confirm the final route with a set of test requests and store the configuration change and the evidence in version control. The step is done when the route reflects the chosen outcome, the test results are archived, and the release ticket is closed with the decision.

Why canary routing is not rate limiting

Traffic splitting, rate limiting, health checks and circuit breaking all sit in front of the same upstream, so they are easy to confuse. Each answers a different question, and using one to answer another produces release signals you cannot trust.

Traffic split chooses the upstream

The traffic-split plugin decides which upstream receives a given request, using weights or match conditions such as headers. It does not decide whether the request is allowed. A weighted canary sends a fixed share of requests to the new version, so the split itself is the release mechanism. Before rollout, confirm which upstream each route and weight resolves to, and that both versions accept the same consumer credentials and payloads.

Rate limiting caps consumer volume

Rate limiting counts requests per consumer, key or route and rejects those above a threshold. It answers how much a client may send, not which version should serve it. If a limit is set on the canary route, the new version sees fewer requests than planned and its error and latency numbers no longer describe production load. Check that limits apply the same way to the stable and canary paths before reading results.

Health checks detect failing backends

Upstream health checks mark backends as unhealthy so the gateway stops sending them traffic. They respond to a failure that already exists. A canary that returns errors may be removed from rotation by a health check, which hides the failure rate you meant to measure. Decide in advance whether the canary should be checked actively, how many failures remove it, and where that status is reviewed.

Circuit breaking stops cascades

The api-breaker plugin stops sending requests to an upstream after failures pass a threshold and returns a response while open. It protects other services from a failing dependency. During a canary, an open breaker can cut traffic to the new version, so the split weight no longer matches what users receive. Record breaker events alongside the release window so that a tripped breaker is not read as a clean canary result.

Misuse produces wrong release signals

A canary judged with a rate limit in place, or with a breaker silently removing the new version, reports a healthy release that never got a fair test. The reverse also happens: a canary that appears to fail because a limit rejected its share is rolled back for the wrong reason. Keep the four controls documented as separate settings with separate owners, and review their state together before each weight change.

Plugin pages cover the details

The rate limiting, health check, circuit breaking and traffic split behavior each have their own configuration rules, defaults and ordering effects. Read the relevant plugin documentation before combining them on one route. This page does not restate those details, so confirm current parameter names and defaults against the APISIX documentation for the version you deploy.

Rollback, sessions and data changes during a split

A gateway rollback only reverts routing. Moving the weights in the traffic-split configuration back to the stable upstream sends new requests to the old version, but it does not undo a database schema change, a message published to a queue, a payment or email the new version already sent, or a record it rewrote. Before the first percentage is set, write down which state the new version touches, who owns each change, and whether the old version can still read and write that state after the change. Expand-and-contract sequencing, where the schema change is additive and the old code keeps working, is the usual pattern, but your team should confirm it against your own data stores rather than assume it.

Both versions must accept the same requests and produce responses the other side can handle during the overlap. Check that the new version tolerates payloads, enum values and token claims issued by the old version, and that the old version ignores fields it does not know. Use the match conditions of the traffic-split plugin to route only requests you have tested for both versions, and keep the matching rules in version control with the route definitions. If the canary runs for longer than one deployment cycle, plan for clients that cache responses or hold long-lived connections such as WebSockets, which keep hitting the version they first connected to.

Session behavior is where weighted splits most often surprise teams. Before relying on consistent hashing by consumer, header or cookie in APISIX, confirm in the documentation for your configured APISIX version how the hash key is applied, how it behaves when the key is missing, and how it interacts with the health check and retry settings of the upstream. Cookie-based affinity depends on the service issuing and honoring the cookie, and session data held in one version's memory or cache will not be visible to the other. Ask the service owners whether sessions are stored in a shared store, and test a logged-in user through several requests while the weight changes. Record the result as an acceptance check before the rollout starts.

Security and operational considerations for release traffic

Traffic splits move production load between service versions, so whoever can edit them can change what users see. The cards below cover who may change weights, how the routing header is protected, how each response is traced to a version, and how the release configuration is kept under review.

Limit who can change route weights

A weight change on a traffic-split rule shifts live production traffic, so it is a privileged action. Restrict Admin API access to a small group of release engineers and send changes through a reviewed pipeline rather than ad hoc calls. Confirm that every change is recorded with the caller's identity and time, and that the Admin API is not reachable from client networks. The hardening topic covers the Admin API controls in detail.

Protect the routing header

Header-matched blue-green switching depends on a header that only your release tooling should set. If clients can send that header, they can force requests onto the new version. Strip or overwrite the header at the edge for external traffic, and match it only on routes where it is expected. Test with a request that carries the header from outside the trusted network and confirm it is ignored or rejected.

Log the upstream that served each request

When a canary misbehaves, the first question is which version handled the failing requests. Add the chosen upstream or a version label to access logs and metrics, so error rates can be split by version. Confirm that log fields survive your log pipeline and that dashboards can compare the stable and canary groups over the same time window. Without this, rollback decisions rely on guesswork.

Keep release configuration in version control

Weights, header rules and upstream references should live in Git, reviewed like application code. Each change then has an author, an approver and a revision that can be reverted in one step. Confirm that the deployment process applies the same file to every environment and that drift between the gateway and the repository is detected. Rollback then means reverting a commit rather than rebuilding a split by hand under pressure.

Separate rollout duty from approval

The person who starts a canary should not be the only person who decides to promote it. Define who may raise weight, who approves promotion, and who may trigger rollback, and enforce those roles in your Git and identity systems. Record the approval in the change ticket. This separation limits accidental or unreviewed traffic shifts, which matters most in regulated environments where change evidence is audited.

Questions teams ask before running APISIX releases

What does a canary or blue-green release setup with APISIX cost?

The cost is the engineering effort to set up and test the traffic-split configuration, plus any observability and CI work the release process needs. Costs fall into one-time items (route inventory, configuration design, pipeline changes, testing a rollback) and recurring items (keeping rules in sync with service releases and maintaining dashboards). Effort rises with the number of routes, the number of services that release independently, how much metric and log coverage already exists, and how many environments must be tested. APISIX itself is released under the Apache License 2.0, so the software carries no license fee. Ask any partner for a written scope that lists each route in scope, each service, the observability sources, the test environments and the handover items, then compare quotes against that list.

How long does a first APISIX release rollout take?

Duration depends on scope more than on the plugin itself. A first rollout usually covers one or two services, a small set of routes and one release pipeline, and it takes longer when several teams, environments or compliance evidence requirements are added. The main drivers are how well the current routes are documented, whether the services already emit the error and latency metrics needed to judge a canary, and how quickly the team can agree on rollback thresholds. No fixed duration applies to every environment. Ask for a timeline tied to named milestones: route inventory, traffic-split design, staging test, one controlled production release and handover.

What must our team provide before a canary or blue-green project starts?

The team must provide information and decisions that only it can make. The main inputs are:

A route inventory with upstream addresses, hosts, paths and the services that own them
The current release process, including who approves a production change and how a rollback is requested
Metrics and logs for error rate and latency per service version, plus access to the dashboards already in use
A named on-call rota that can respond during a shifted release window
Test environments that mirror production routing closely enough to rehearse a switch

Without these inputs, the traffic-split rules will be guesses, and rollback will depend on manual checks. Confirm each item in the written scope before work starts.

Can an internal platform team run APISIX canary releases alone?

Yes, a platform team with gateway, Kubernetes or Linux operations experience can run this pattern without outside help, provided it has the time and the skills to write and test the rules. The work requires familiarity with APISIX routes and plugin configuration, a clear release process, and reliable metrics per service version. Teams that lack one of these usually find the gap during the first real release, when a rollback decision has to be made under pressure. Many organizations split the work: internal staff own the release process and on-call, while an outside partner designs the first configuration, tests it and documents it. Decide the split before work starts and write it into the scope.

How is a fixed-scope APISIX release engagement priced?

DeployOpen does not publish a price list. A fixed-scope engagement is quoted from a written scope that states the routes, services, environments, deliverables and timeline before work begins. The quote reflects the effort to deliver that scope, and changes to the scope are handled through a written change request. Ask for the quote to separate one-time work (design, configuration, testing, handover) from any recurring support or retainer work, and ask which environments, services and release types each line item covers. Compare quotes on the same scope list rather than on headline totals.

How does a canary release differ from rate limiting in APISIX?

A canary release sends a share of real requests to a new service version so the team can compare its behavior with the current version, and it can be shifted back quickly. Rate limiting caps how many requests a client or route may send in a period, and it protects a service from overload or abuse regardless of version. The two can run together, but they answer different questions. A traffic split decides which version handles a request, while a rate limit decides whether a request is admitted at all. Mixing them up leads to canaries that are throttled and to limits that hide a bad release. Keep the rules and dashboards for each separate in the design.

What goes wrong most often with gateway-based release switching?

The most common failures come from process gaps rather than from the gateway itself. Typical issues include:

Traffic is split by weight, but the new version lacks the metrics needed to judge it
Header-based blue-green rules are not removed after the switch, so stale clients keep reaching the old version
Session or consumer state is not handled, so users bounce between versions mid-flow
Rollback is written down but never rehearsed in a test environment
Route changes are made in the Admin API by hand and drift from the version-controlled configuration

Confirm how each item is handled in the design and test plan, and ask for a rollback rehearsal as a named deliverable.

Does the gateway replace our deployment pipeline for releases?

No. APISIX changes where traffic goes, but it does not build, test or deploy the service itself. Your existing pipeline still produces the artifact and rolls out the new version, and APISIX then shifts a share of traffic to it or switches traffic by header. The integration point is the release step, which must set the traffic rule, wait for a health and metric check, and either promote or revert the rule. Ask any partner to show where in your pipeline the rule change happens and who holds permission to make it.

How DeployOpen scopes and supports APISIX release control

DeployOpen plans, implements, tests, documents and supports Apache APISIX release control inside infrastructure you control. Each engagement starts from a written scope that names the services, routes, traffic-split rules, header matches and rollback conditions involved in your releases.

How we scope

We confirm which upstream services will receive weighted canary or header-matched blue-green routes, and which ones stay on simple routing.
The written scope lists the APISIX routes, traffic-split plugin settings and deployment mode your releases will use.
Rollback triggers, owners and the metrics that decide promotion are agreed in writing before any build work starts.

How we build

We configure weighted canary routes and header-matched blue-green switching against staging services before touching production traffic.
Test cases cover traffic percentages, header matching, sticky behavior for users and the rollback path, run against your environment.
We keep rate limiting and release splits as separate route policies so each can be changed without affecting the other.

How we hand over

You receive route definitions, plugin configuration and a runbook for changing traffic weights and switching versions.
Documentation explains how canary splits differ from rate limiting, with examples marked as illustrative for your own services.
Your team practices a planned rollback and a promotion step during handover, so the procedure is tested before the first real release.

How we support

Fixed-scope engagements end at the agreed deliverables, and changes to the release model are quoted as a new written scope.
Ongoing support, APISIX upgrades and release-control reviews are available as a retainer if your release cadence needs them.
Ask any partner to confirm how they test rollback and who holds on-call responsibility for the gateway after handover.

Apache APISIX canary and blue-green releases