Planning your APISIX 3.18 to 3.19 upgrade before a single node changes
The first decision for a team running APISIX 3.18 in production is to treat 3.19 as a change window, not a routine patch. Three changes can break live traffic without any error in the gateway itself: upstream certificate verification for https and grpcs upstreams, issuer checks on token introspection in openid-connect, and rejection of empty basic-auth passwords. According to the APISIX downloads page, APISIX 3.19.0 was released on September 28, 2026, and the page does not mark it as long-term support, so the upgrade should be scheduled against your own support policy rather than assumed to be a supported long-term baseline.
Before any node changes, list every upstream that uses upstream.tls.verify: true, every route that uses openid-connect with bearer introspection and valid_issuers, and every consumer or route that relies on basic-auth with an empty password. Confirm with the owners of each upstream that their certificates chain to a CA the gateway trusts. Under the 3.12.0 default, ssl_trusted_certificate points at the system store, but private CAs added only to your own environment may still need to be configured explicitly. Check your identity provider's issuer value against the configured valid_issuers list, since a mismatch will now reject otherwise valid tokens.
The path is a direct in-line upgrade from 3.18 to 3.19, with the APISIX changelog as the reference for every behavior change in between. Read the 3.19.0 entries in full, not only the headline items, because batch-requests response limits and WebSocket metric labels also change what your dashboards and clients see. Agree on a rollback plan, a staging environment that mirrors your certificates and identity provider, and a named owner for each affected route before the production window opens.
Which 3.19 changes affect your gateway
This table lists each APISIX 3.19 change covered by the upgrade, the traffic or dashboard effect it can have, and the action to take before cutover. Read it against your current routes, plugins and dashboards, and treat any row that matches your setup as a blocking check for the staging run.
| Change in 3.19 | Impact on traffic or dashboards | Action before cutover |
|---|---|---|
| upstream.tls.verify on https and grpcs upstreams | Upstreams already set to verify: true start rejecting certificates the gateway does not trust, which can cause 502 errors on previously working routes. | List upstreams with verify set to true, confirm each certificate chains to the trusted store, and test handshakes from the gateway hosts. |
| openid-connect introspection issuer check | Introspection responses whose iss does not match configured valid_issuers are rejected, so tokens from a mismatched issuer fail authorization. | Compare the iss value in live introspection responses with valid_issuers for every route and update the configuration where they differ. |
| Bounded batch-requests responses | Each sub-response is capped at 1 MiB and the total at 10 MiB by default, so large aggregated responses may be truncated or rejected. | Measure typical batch response sizes, decide whether the defaults fit your clients, and test the largest batch your consumers send. |
| Empty basic-auth passwords rejected | Consumers configured with an empty password stop authenticating, which can break service accounts or legacy clients. | Search consumer definitions for empty password values and set real credentials before the upgrade reaches production. |
| ai-proxy-multi instances with duplicate names rejected | Configurations that contain instances sharing a name no longer load, which can block a deployment or configuration sync. | Check every ai-proxy-multi definition for unique names and rename any duplicates in the configuration source you use. |
| WebSocket metric labels as request_type=websocket | Successful WebSocket upgrades appear under a new request_type label, so dashboards and alerts filtering on the old label show a drop in those requests. | Update Prometheus or dashboard queries and alert rules that count WebSocket traffic to use request_type=websocket. |
Why upstream TLS and token issuer checks can reject working traffic
The two checks most likely to break working traffic in this upgrade are upstream certificate verification and token issuer matching. In APISIX 3.19.0, upstream.tls.verify is honored for https and grpcs upstreams, not only for Kafka. Any upstream already configured with verify: true will now reject certificates it cannot validate, even if the service has worked for years. The change lands on top of an earlier default: APISIX 3.12.0 set ssl_trusted_certificate to the system certificate store, so the trust anchors your gateway uses may already differ from what your team assumes. Before the upgrade, list every upstream with `verify: true` and confirm that its certificate chain, including any internal or private CA, is trusted by the gateway's store. Test each chain from the gateway host itself, since a browser or a laptop may trust a root that the gateway does not.
The second change concerns openid-connect bearer token introspection. APISIX 3.18.0 made introspection fail closed when the discovery document cannot be retrieved, and it enforces audience, issuer and required scopes. APISIX 3.19.0 adds a further rule: an introspection response is rejected when its iss value does not match one of the configured valid_issuers. A mismatch is common when the identity provider is reached through an internal hostname while tokens carry a public issuer URL, or when a tenant or realm path differs between environments. Compare the iss claim in a real introspection response with each entry in valid_issuers, character for character, and confirm the exact issuer string with your identity provider's documentation rather than assuming it.
Run both checks in a staging environment that mirrors production certificates and identity provider endpoints before cutover. Record which routes fail, then decide whether to correct the trust store or issuer configuration, or to narrow the affected settings with an explicit decision recorded for your security team. Treat any relaxation of verification as a documented exception, not a quiet fix, because both checks protect the path between the gateway and the services or identity provider it depends on.
What to inventory and back up before the upgrade
Five preparation items run in parallel before any node is touched. Each one gives you a rollback point or a list of configuration that the 3.19.0 behavior changes could break. Finish them in the test environment first, then repeat them against production.
Back up etcd or the standalone file
Take a snapshot of the etcd data that holds routes, upstreams, consumers and plugin metadata if you run traditional or decoupled mode. In standalone mode, copy conf/apisix.yaml or conf/apisix.json along with the rest of the conf directory. Store the copy outside the hosts being upgraded and confirm you can restore it into a scratch environment before the change window.
Inventory upstreams with verify set
List every upstream that uses https or grpcs and record whether upstream.tls.verify is true. Upstreams already set to verify will start rejecting certificates that the trust store does not accept, so check each certificate chain against the CA bundle in use. Confirm which upstreams have verification turned off so you can decide whether to fix them before the upgrade.
List basic-auth consumers
Export every consumer that uses basic-auth and identify any with an empty password. Version 3.19.0 rejects empty basic-auth passwords, so those consumers will fail to authenticate after the upgrade. Plan a password reset or rotation with the owning application team and test each login path.
Review openid-connect issuers
Identify each route using openid-connect for bearer token introspection and record its configured valid_issuers. Version 3.19.0 rejects introspection responses whose iss does not match that list, so an incomplete or outdated list will cause authorization failures. Compare each entry with the issuer value your identity provider actually returns.
Map batch-requests and ai-proxy-multi
Find routes that use the batch-requests plugin and check whether any client depends on very large combined responses. Version 3.19.0 caps responses per request and in total, so large payloads will be cut off. Also list ai-proxy-multi instances and confirm that no two share a name, since duplicate names are now rejected.
How the 3.18 to 3.19 upgrade runs step by step
This sequence moves an APISIX 3.18 gateway to 3.19 in a controlled order, so each change is tested, observed on a canary and reversible before the full fleet moves.
Freeze configuration changes during the window
The platform owner stops route, plugin and consumer changes in the Admin API or your configuration repository for the duration of the upgrade. Export the current routes, upstreams, consumers, plugin metadata and the active configuration file so you have a reference state. The step is done when the export is stored with a version label and the change freeze is recorded in your change system. Confirm with your team whether any automation writes to the Admin API on a schedule.
Back up etcd and the current deployment artifacts
Take a snapshot of the etcd data (in traditional or decoupled mode) and copy the current APISIX configuration files, plugin code, custom Lua or plugin runner code, and the image or package version in use. Store the backup outside the cluster being upgraded. The step closes when a restore has been checked in a scratch environment or your team has confirmed the snapshot is readable. This backup is the first rollback point for the whole upgrade.
Upgrade a non-production environment first
Deploy APISIX 3.19.0 to a test environment that mirrors your production plugins, upstreams and deployment mode, using the release listed on the official downloads page. Compare the loaded configuration with the frozen export and note any warnings at startup. The step is complete when the test gateway starts cleanly and serves traffic against the copied configuration. Rollback here is simply redeploying the 3.18 image or package to the test environment.
Run the route and authentication test suite
Run your regression suite covering every route, every authentication plugin in use and the protected upstreams. Include negative cases: invalid tokens, expired tokens, requests without credentials and requests with a wrong issuer. The step is done when all expected results match the pre-upgrade baseline. Any failure found here blocks promotion, and the fix goes back into the configuration before the next attempt.
Fix upstream certificates and token issuers
Review every upstream set to verify TLS for https and grpcs targets. Since 3.19.0 honors upstream.tls.verify for these protocols, upstreams that already set verify to true will reject certificates the gateway does not trust, so confirm the trust chain or update the CA bundle. Then review openid-connect introspection settings and confirm that the iss value in each identity provider response matches valid_issuers. The step closes when test calls to every upstream and every introspection endpoint succeed with verification enabled.
Upgrade one data plane node or a canary group
Move a single data plane node or a small canary group to 3.19.0 while the remaining nodes stay on 3.18. Send a controlled share of live traffic through the canary and compare error rates, latency and authentication outcomes with the rest of the fleet. The step is done when the canary runs for a period your team has agreed and shows no new 4xx or 5xx spikes. Rollback point: revert the package or image on the canary node so it returns to 3.18 before any wider rollout.
Check logs and metrics for WebSocket labels
In 3.19.0, successful WebSocket upgrades are labeled request_type=websocket in metrics. Confirm that dashboards, alert rules and recording rules still match the new label, and that WebSocket traffic is counted where your team expects. Review gateway error logs for batch-requests responses that now hit the 1 MiB per response or 10 MiB total limits, and for basic-auth consumers that now fail because of empty passwords. The step closes when dashboards show the expected labels and no unexplained errors appear on the canary.
Roll the remaining nodes and keep the rollback path
After the canary passes every check, upgrade the remaining data plane nodes in small batches, checking the same signals after each batch. Keep the 3.18 image or package available until the final batch is verified. The sequence is complete when every node reports 3.19.0, the regression suite passes against production, and the change freeze is lifted with a written record of the results. Ask your provider or internal team to confirm whether the Ingress Controller or any control plane component also needs an upgrade in your setup.
How to test the upgrade and roll it back safely
In standalone mode there is no etcd cluster, and APISIX reads its routes and plugin configuration from conf/apisix.yaml or conf/apisix.json. The deployment modes documentation says APISIX checks that file for changes every second and hot-reloads it, so a bad edit takes effect almost immediately. Test the new 3.19.0 binary against a copy of the production file in a staging instance before anything reaches live nodes, and keep the last known-good file under version control so rollback is a file restore.
- Reload check on a copy: Load the production file into a staging 3.19.0 node and confirm it starts with no schema errors and no unexpected plugin rejections.
- Upstream TLS handshakes: For every
httpsorgrpcsupstream withupstream.tls.verifyset to true, confirm the handshake succeeds against the real certificate chain. Untrusted certificates now fail. - Token introspection with real tokens: Send live access tokens through each openid-connect route, including expired and wrong-issuer tokens, and confirm the issuer and audience decisions match your identity provider.
- Batch requests and basic-auth consumers: Replay a representative batch-requests payload and check the responses stay within the documented limits. Confirm every basic-auth consumer still has a non-empty password.
- Rollback limits: Restoring the previous binary and file is quick, but in-flight requests during a hot reload are not replayed, so run the cutover in a window where a brief error spike is tolerable.
Before promoting a file, confirm each check passes on the staging node and that the rollback file has been tested with the old binary.
In traditional mode, data and control planes run together, and in decoupled mode separate data plane nodes share one etcd cluster with the control plane. The Admin API listens on port 9180 in traditional mode, and both modes store routes and plugin settings in etcd. Because configuration lives in etcd, the rollback plan has to cover the etcd data as well as the gateway binaries, and a snapshot taken before the upgrade is the most important artifact you will produce.
- Snapshot before upgrade: Take and store an etcd snapshot, then confirm you can restore it into a separate test cluster before changing production.
- Staged node upgrade: Upgrade one data plane node first, then verify upstream TLS, introspection and batch-requests behavior through it before moving the rest.
- Dashboard check for WebSocket labels: Confirm dashboards and alerts that filter on the older label values still match, since successful WebSocket upgrades now report
request_type=websocket. - Basic-auth audit: Query consumers through the Admin API and find any with empty passwords, which 3.19 now rejects.
- Rollback limits: Reverting binaries is straightforward, but restoring an etcd snapshot rolls back every write made since the snapshot, so pause Admin API changes during the window.
Confirm the snapshot restore has been rehearsed and that the team knows who owns the etcd restore step.
With the APISIX Ingress Controller, the upgrade touches the gateway pods, the controller and the resources that describe routes. The controller project is versioned separately, and the downloads page lists APISIX Ingress Controller 2.2.0 alongside APISIX 3.19.0, so confirm the pairing you deploy is supported before rolling it out. Use a non-production cluster or a separate gateway deployment for the rehearsal, and treat the previous Helm release as the primary rollback path.
- Helm revision as rollback: Record the current Helm release revision so you can return to the known-good gateway image and values with a single rollback command.
- Canary gateway deployment: Route a small share of traffic to a new 3.19 deployment and compare errors and latency against the existing one.
- Upstream TLS and introspection: Exercise every upstream that uses verified TLS and every openid-connect route with live tokens from the pod network, not only from a workstation.
- Metrics and labels: Check that Prometheus scrape rules and Grafana panels still capture WebSocket traffic under the new label.
- Rollback limits: A Helm rollback restores the previous gateway image, but custom resources applied by a newer controller may not be readable by an older one, so confirm compatibility before applying them.
Ask your platform team to confirm the rollback order for the gateway and the controller before the change window opens.
What makes an APISIX 3.19 upgrade take longer
This table lists the factors that expand or shrink the work of moving from APISIX 3.18 to 3.19, and the direction each one moves effort. Read each row as a question to answer about your own gateway before asking for a quote. The main takeaway is that the number of upstreams with TLS verification, the number of identity-based consumers, and the availability of a production-like test environment usually matter more than the version change itself.
| Driver | Effect on effort | What to ask for in a quote |
|---|---|---|
| Routes and upstreams with verify enabled | More routes with upstream.tls.verify set to true raise effort, since each one needs its CA chain confirmed before cutover | Inventory of affected https and grpcs upstreams and a plan to test each certificate chain |
| Custom Lua or plugin runner code | More custom code that depends on the changed batch, auth or metric behavior raises effort and retest scope | Review of custom plugins against the 3.19 changelog and a list of code paths to retest |
| openid-connect consumers and introspection | More bearing-token routes with introspection and valid_issuers settings raise effort, because issuer mismatches fail closed | Count of openid-connect routes, configured valid_issuers, and checks against each identity provider's issuer value |
| basic-auth consumers | More consumers with blank passwords raise effort, since those credentials will be rejected after the upgrade | Report of consumers with empty passwords and a plan to rotate or replace them with users |
| batch-requests traffic | Clients that send large batches or rely on full response bodies raise effort and need compatibility testing | Expected batch sizes and client behavior under the 1 MiB per response and 10 MiB total defaults |
| Dashboards and alerts on metric labels | More dashboards and alerts that filter WebSocket traffic by its old label raise effort in relabeling and review | Inventory of dashboards and alert rules that reference WebSocket metrics and an update plan |
| Deployment mode | Traditional, decoupled and standalone modes differ in rollout steps and config reload checks, so mixed estates take more planning | Rollout plan per mode, including how etcd-backed and file-driven configurations are validated |
| Test environment with production-like certificates and identity providers | Without a test environment that mirrors production certificates and identity providers, verification work moves into production windows and effort rises | Confirmation of test certificates, identity provider endpoints and a rollback rehearsal in that environment |
Questions teams ask before upgrading APISIX to 3.19
How much effort does the APISIX 3.18 to 3.19 upgrade take?
The effort depends mostly on how many routes, upstreams and plugins your gateway carries and how many of them touch the five changes in 3.19.0. Teams with few upstreams that already use valid certificates and a single openid-connect configuration usually face a smaller review. Effort rises when upstreams use verify: true with private certificate authorities, when introspection depends on tokens from more than one issuer, or when batch-requests, basic-auth consumers or WebSocket dashboards are in use. Each of these needs a test case and sometimes a configuration change. The cost to your team is mainly staff time for inventory, testing and change approval, and it is largely one-time for this release. Ask any partner for a written scope that lists the affected routes, plugins and environments before agreeing to a figure.
Can our own team upgrade APISIX without outside help?
Yes, a team with working knowledge of APISIX configuration, the Admin API and its own deployment can run this upgrade internally, provided it has a non-production environment that mirrors production routes and plugins. The changes are specific and documented in the APISIX changelog, so the work is mainly finding affected configuration and proving behavior with tests. Outside help is most useful when the gateway sits in a regulated environment that needs formal change evidence, when many teams publish routes through the gateway, or when the team has no spare capacity for a parallel test run. Whichever route you take, confirm who owns the openid-connect and LDAP integrations, because those owners must sign off on the behavior changes.
What do our staff and change windows need to support the upgrade?
The upgrade needs an engineer who can read APISIX configuration, someone who owns each upstream and identity integration, and a change approver who accepts the test evidence. Plan for a staging run, a rollback rehearsal and a production window in which traffic can be watched closely. Because the gateway usually sits in front of many applications, notify the owners of consuming services before the window opens. Ask your platform team whether the rollout will be a rolling replacement of nodes or a parallel cluster, since that choice decides how long the change window must stay open and who must be available during it.
Is APISIX 3.19.0 a long-term support release?
The APISIX downloads page does not mark 3.19.0 as a long-term support release, and it was released on September 28, 2026. Do not assume LTS status from the version number or from earlier releases that carried the LTS label. Confirm the current support policy on the downloads page and in the project's own release notes before you set a production upgrade cadence. If your governance requires an LTS line, decide with your architecture board whether to stay on a supported non-LTS release with a planned upgrade cadence or to wait for a line that the project labels as LTS. Record the decision, because auditors and security teams will ask for it.
What should we test first in the APISIX 3.19 upgrade?
Start with the changes most likely to break traffic silently. First, test every upstream set to upstream.tls.verify: true over https or grpcs, because 3.19.0 now rejects certificates it does not trust, and a certificate chain that worked before may fail. Second, test openid-connect bearer introspection with tokens from each configured issuer, since 3.19.0 rejects responses whose iss does not match valid_issuers. Third, check batch-requests calls against the 1 MiB per response and 10 MiB total default limits. Fourth, confirm that consumers using basic-auth do not have empty passwords, which 3.19.0 rejects. Finally, update dashboards that count WebSocket connections, because successful upgrades now carry request_type=websocket. Also verify that no two ai-proxy-multi instances share a name.
How does rollback work if the APISIX 3.19 upgrade fails?
Rollback depends on how you deploy APISIX. In a rolling replacement, the simplest path is to redeploy the previous 3.18 image or package on the affected nodes and confirm that routes and plugins load before traffic returns. In a parallel cluster, rollback means shifting traffic back to the 3.18 gateway and keeping it warm until the new one is proven. Before any change, back up your configuration and the etcd data if you run traditional or decoupled mode, and rehearse restoring both in staging. Verify that the previous version can read the saved configuration, since schema differences can block a restore. Write the rollback steps, the owner of each step and the traffic signal that triggers the decision into the change record before the window opens.
How DeployOpen scopes, delivers and supports your APISIX upgrade
DeployOpen plans the move from Apache APISIX 3.18 to 3.19 as a written, fixed-scope engagement inside your own infrastructure. We start from your actual routes, upstreams, authentication plugins and metrics dashboards, so each of the 3.19 behavior changes is checked against what your gateway really does.
How we scope
- The written scope lists every APISIX 3.19 change that touches your routes, plugins and upstreams.
- We inventory upstreams using https or grpcs and confirm which ones set upstream TLS verification.
- We review openid-connect introspection, batch-requests use, basic-auth consumers and WebSocket dashboards before any work starts.
How we build
- We stage the upgrade in a non-production environment that mirrors your etcd and configuration layout.
- We adjust certificate trust, issuer lists and consumer credentials, then test each change against recorded traffic.
- We update dashboards and alert rules for the request_type=websocket label before the production rollout.
How we hand over
- You receive a change record listing each 3.19 behavior change, the decision taken and the evidence behind it.
- Rollback steps are written for the production cutover, including the configuration snapshot to restore.
- Runbooks cover the Admin API checks and the monitoring queries your operations team will keep using.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer after the handover.
- Support can cover follow-up on later APISIX releases, including the 3.19.0 release dated September 28, 2026.
- Ask for the support terms, response expectations and the upgrade review cadence in the written quote.
