Why an APISIX 3.10 to 3.12 upgrade needs a plugin-level plan
Moving a production APISIX 3.10 gateway to 3.12 crosses two minor releases, and the changes that matter most sit in the plugins that handle identity and AI traffic. The jwt-auth, hmac-auth, opentelemetry and AI plugin changes can break clients silently, so a green deployment does not prove the gateway still behaves as it did. Treat this as a plugin-level plan, not a routine version bump. The supported path is to step through 3.11 first and then 3.12, in that order, rather than jumping straight to 3.12. Use the project's APISIX changelog and the GitHub release notes as the source of truth for each step, and confirm every change against your own route and plugin configuration before you schedule a cutover. Your team should inventory which routes use jwt-auth signing, hmac-auth, opentelemetry and any ai-* plugins, because those are the only places this page's changes apply. This page covers only the changes in 3.11 and 3.12 that affect those plugins. It does not cover earlier or later releases, the wider Admin API, or etcd and deployment changes, which are handled on separate upgrade pages. Where a change is not described here, confirm it in the release notes for the version you are installing and in your own test environment, not from memory of a previous release.
Which 3.11 and 3.12 changes affect your routes and plugins
This table lists each 3.11.0 and 3.12.0 change affecting authentication and AI plugins, with its impact and the action to take before cutover. Read each row against the plugins and routes in your current gateway configuration, and treat any row that matches as a required change. The main takeaway is that the jwt-auth and hmac-auth changes break existing clients and automation, while the opentelemetry and AI plugin changes break configuration that loads but behaves differently.
| Change | Impact | Action needed |
|---|---|---|
| jwt-auth signing endpoint and private key upload removed (3.11.0, October 17, 2024) | Any process that requests tokens from APISIX or uploads a private key through the removed interface will fail | Move token signing to your identity provider or an external service, then confirm no automation calls the old endpoint |
| hmac-auth rewritten (3.11.0) | Existing hmac-auth consumers and signing logic may no longer validate as before | Review consumer definitions and client signing code against the rewritten plugin behavior in the changelog |
| hmac-auth access_key renamed to key_id (3.11.0 and later) | Consumer configuration that sets access_key will not match the new field name | Rename the field to key_id in consumer definitions and in any Ingress Controller resources |
| opentelemetry attributes moved to plugin metadata (3.12.0, April 1, 2025) | Attributes set on the route or service level no longer apply, so traces may lose fields | Move attribute settings into plugin metadata and check trace output in a test environment |
| ai-content-moderation renamed to ai-aws-content-moderation (3.12.0) | Routes that reference the old plugin name will not load the moderation policy | Update plugin references and confirm the AWS-specific configuration fields your policy needs |
| ai-proxy passthrough removed (3.12.0) | Routes that depended on passthrough mode will change how requests reach the model provider | Identify ai-proxy routes using passthrough and redesign them before cutover |
| Empty key attributes disallowed (3.12.0) | Consumers or plugin settings with blank key values will be rejected | Search configurations for empty key fields and populate or remove them |
| ssl_trusted_certificate set to system by default (3.12.0) | Upstreams that depend on a private CA not in the system store may fail TLS verification | Confirm which CA bundle each upstream trusts and set explicit certificates where needed |
| signed_headers defaults to date (3.17.0, if you jump further) | Teams skipping to 3.17 or later see a different hmac-auth signing default | Check whether clients sign the date header and set signed_headers explicitly if your clients do not |
How jwt-auth and hmac-auth changes affect consumers and clients
Platform operators own the gateway change itself. In APISIX 3.11.0, released October 17, 2024, the jwt-auth signing endpoint and private key upload were removed, so any automation, runbook or internal tool that calls them will fail once the gateway is upgraded. The work starts with finding every caller and assigning each one an owner before the cutover window is scheduled.
- Signing endpoint callers: search scripts, CI jobs, service catalogs and partner onboarding tools for calls to the removed jwt-auth signing endpoint, and record the owner of each call.
- Private key uploads: identify every process that pushed a private key to the gateway for jwt-auth, and move token signing to the system that holds that key.
- Staging rehearsal: run the removed calls against a staging gateway on the target version to confirm they fail in the expected way and that the replacement flows work.
- Rollback path: keep the previous gateway version and its configuration available until consumers confirm that token issuance and validation work.
Check that every removed call has a named owner and a tested replacement path before the upgrade is scheduled.
Consumers and partner clients feel the change at the request level. The hmac-auth plugin was rewritten in APISIX 3.11.0, and the APISIX Ingress Controller 2.0 upgrade guide notes that the required access_key field became key_id for APISIX 3.11 and later. A client that still sends the old field or signs requests on the old assumptions will be rejected, and partners often deploy their own changes on a separate schedule from yours.
- Field name: update every credential definition and client configuration that used access_key so that it uses key_id, and confirm the exact field your gateway version expects.
- Signed headers default: the default for signed_headers changed to date in APISIX 3.17.0. Teams planning to move past that release should not assume the headers they sign today are still covered by default.
- Partner notice: send partners the new field name and an agreed test window, since they must change their own code too.
- Signature test: have each client sign a test request against a non-production route, then confirm the gateway accepts the valid request and rejects an altered one.
- Failure visibility: check that clients log authentication failures with enough detail to tell a rejected signature apart from a network fault.
Confirm that each client has a passing test against the target version before production traffic moves. The APISIX Ingress Controller upgrade guide is the source to check for the field rename.
Security reviewers need to know where signing material lives after the upgrade. The jwt-auth change removed the signing endpoint and the ability to upload private keys to the gateway, so the gateway should no longer be a place where private keys are created or stored for that flow. Reviewers should verify this against the running configuration and the APISIX changelog, and treat any runbook statement as a claim to test.
- Key inventory: record where each jwt-auth verification key and each hmac-auth secret now lives, and which team controls it.
- Leftover material: check the running configuration and older exports for private key material or secrets created before the upgrade, and remove what is no longer needed.
- Admin access: confirm who can read or change consumer credentials, since the key identifier and secret pair must be protected like any other credential.
- Backups and logs: check backups, exported configurations and logs for copies of signing material that predate the upgrade.
- Change record: keep the changelog entries and the upgrade test results as change evidence for auditors.
Ask the team to show the key inventory and a before and after configuration comparison before you approve the change.
What to inventory and back up before changing APISIX
The 3.10 to 3.12 changes touch authentication, telemetry and AI routes, so the work before the upgrade is mostly finding what depends on the affected fields. Each card below names one artifact to capture or one population to find, and what to confirm about it before any node is changed.
Configuration backup from etcd or files
Export the full route, consumer, plugin and plugin metadata configuration before touching any node. In traditional or decoupled mode, take an etcd snapshot as well as the Admin API export, since the Admin API alone may not capture everything. In file-driven standalone mode, the conf/apisix.yaml or conf/apisix.json file is the source of truth, so store it in version control. Confirm that a restore into a scratch cluster reproduces the same routes and consumers.
Consumers using jwt-auth or hmac-auth
List every consumer and route that uses jwt-auth or hmac-auth, and record which credential fields each consumer holds. The 3.11 hmac-auth rewrite and the access_key to key_id change affect how consumers are defined, so tie each consumer to the client application that owns its key. Confirm with the owning team whether any client still calls the jwt-auth signing endpoint, which 3.11 removed, and whether it holds its own private key.
Routes using opentelemetry or ai-content-moderation
Find every route and global rule that enables opentelemetry or ai-content-moderation, along with the attributes each one sets. In 3.12 opentelemetry attributes move to plugin metadata, and ai-content-moderation is renamed to ai-aws-content-moderation, so a route that still references the old name will not behave as expected. Confirm which telemetry backend each route reports to, and who owns the dashboards and alerts built on those spans.
Custom plugins that read affected fields
Review every custom plugin written in Lua or through a plugin runner for reads of the fields this upgrade changes, including authentication context, consumer key fields, telemetry attributes and the AI plugin names. Also check for empty key attributes, which 3.12 now rejects, and for any dependence on the default certificate trust store, since 3.12 sets ssl_trusted_certificate to system by default. Confirm each plugin against the 3.12 changelog entries before scheduling the upgrade.
Staging environment with mirrored traffic
Run the restored configuration in a staging environment that receives a copy of production request patterns, including requests that carry jwt-auth tokens, hmac-signed requests and AI route calls. Compare status codes, authentication results and telemetry output between the 3.10 baseline and the 3.12 candidate. Confirm that test traffic includes failure cases, such as expired tokens and invalid signatures, because those paths are where behavior often changes.
Rollback images and restore path
Keep the exact 3.10 image or package, its configuration export and the etcd snapshot together so the team can return to the previous state without rebuilding anything. Store the rollback artifacts in a registry or repository you control, and confirm that the rollback runbook names who approves the revert and how traffic is shifted back. Check the release notes for the target version before pinning it, and record the checksum or digest of each image.
How the gateway, etcd and plugin metadata fit together during the change
The opentelemetry plugin change in 3.12.0 moves settings out of plugin attributes and into plugin metadata, so this upgrade touches where configuration is stored, not only the plugin code. APISIX runs on NGINX and uses etcd to hold routes, consumers and plugin configuration, and it hot-loads changes without a restart. That design means a gateway can accept a new configuration shape before every node runs the matching code. Before you change anything, inventory which routes and global rules use the opentelemetry plugin, record the attributes they set today, and confirm with your own test environment how the metadata object should be written for your deployment. Confirm the exact keys against the release notes rather than assuming them.
The upgrade order depends on the deployment mode in the APISIX deployment modes documentation. APISIX supports traditional mode, where the data and control planes run together, decoupled mode, where separate data planes share one etcd cluster, and standalone mode, which has no etcd and reads conf/apisix.yaml or conf/apisix.json directly. In a shared etcd cluster, a new configuration written by one team can reach data planes still running 3.10, so plan the change so that stored metadata stays readable by both versions during the rollout. In file-driven standalone mode, each node reloads its own file, so you can stage the new file and the new binary together on a canary node first and then move the rest of the fleet.
Before the cutover, confirm three things in a non-production copy of your deployment: that the etcd snapshot you take is restorable, that each data plane version can read the metadata you write, and that your Admin API automation still targets the right keys. Keep the rollback path simple by not editing stored metadata on production until every node is on 3.12. The APISIX GitHub releases page is the place to check the 3.12.0 notes for any additional storage changes that apply to your version path.
Upgrade sequence from 3.10 to 3.12 with checks and rollback points
This sequence moves a running APISIX 3.10 gateway through 3.11.0 and 3.12.0 in stages, with a named check and a rollback action at each stage so that the auth and AI plugin changes are proven in staging before any production node changes.
Freeze configuration and record the current state
Your platform team stops routine route and plugin changes for the upgrade window and exports the running configuration: routes, consumers, upstreams, plugin configs, and the list of plugins in use on each route. Record the current APISIX and etcd versions and the Admin API key handling. Confirm that every route using jwt-auth, hmac-auth, opentelemetry or ai-content-moderation is on the list, because those are the plugins this release line changes. The step is done when the inventory is stored with the change record and someone other than the author has reviewed it. Rollback point: no change has been made yet, so the action is to lift the freeze.
Back up etcd and configuration files
Take a snapshot of the etcd data used by the APISIX cluster and copy the APISIX configuration files, including the config.yaml file and any plugin or certificate files your deployment references. Store the backup outside the cluster and test that it can be restored into a scratch environment. Confirm that the backup includes the consumer credentials and any stored secrets, since those are needed to roll back auth changes. The step is done when a restore has been rehearsed and the restore steps are written down. Rollback point: the backup itself is the rollback source for every later stage.
Deploy a staging cluster on 3.10
Build a staging cluster that runs the same APISIX 3.10 release as production, loaded with a copy of the production configuration and a representative set of test consumers. Use the same deployment mode as production, whether traditional, decoupled or standalone, so that behavior matches. Run a baseline of your auth flows and AI routes against 3.10 and save the responses, status codes and headers. The step is done when the baseline results are saved and any pre-existing failures are documented. Rollback point: staging is disposable, so the action is to rebuild it from the backup.
Upgrade staging to 3.11.0 and run auth tests
Upgrade the staging cluster to 3.11.0, which removed the jwt-auth signing endpoint and private key upload and rewrote hmac-auth. Test that any client relying on the signing endpoint now has a replacement token source, and that hmac-auth requests use the key_id field where the Ingress Controller or your configuration previously used access_key. Run the saved baseline of login, token validation and signed-request tests and compare results. The step is done when every auth test matches the expected result and each removed endpoint has a documented replacement. Rollback point: revert staging to the 3.10 image and restore the staging backup.
Upgrade staging to 3.12.0 and run AI and observability tests
Move staging from 3.11.0 to 3.12.0. Verify that opentelemetry attributes now come from plugin metadata, that ai-content-moderation has been replaced by ai-aws-content-moderation, that ai-proxy passthrough is gone, and that empty key attributes are rejected. Check that ssl_trusted_certificate now defaults to the system trust store, and test outbound TLS to every upstream and AI provider. Send test traffic through trace export and AI routes and confirm spans and responses arrive. The step is done when AI, tracing and outbound TLS tests pass on staging. Rollback point: revert staging to 3.11.0 or 3.10 and restore the staging backup.
Canary production nodes on 3.12
Move a small subset of production data plane nodes to 3.12.0 while the remaining nodes stay on 3.10. Route a limited share of real traffic to the canary nodes using your load balancer or DNS controls, and watch error rates, auth failures, TLS handshake errors and AI route responses. Compare dashboards with the pre-change baseline, including any LLM metric labels that your team has built. The step is done when the canary runs through an agreed observation period without new auth or TLS errors. Rollback point: remove the canary nodes from the load balancer and return traffic to the 3.10 pool.
Complete the production cutover
Upgrade the remaining production nodes to 3.12.0 in batches, checking health and the same metrics after each batch. Keep the 3.10 image and the production backup available until the final batch passes. Confirm that every route still answers its auth and AI tests from the staging checklist, and that consumer credentials resolve as before. The step is done when all nodes report 3.12.0, the post-upgrade test suite passes, and the change record shows the final versions. Rollback point: while the 3.10 image is kept, roll back each batch in reverse order, and restore the production backup only if configuration state is damaged.
Close the rollback window and record evidence
After the production checks pass, record the final configuration export, the test results from each stage, and the version of every node. Remove the temporary rollback images only after the agreed stability period has ended and the team has signed off. Keep the backups under the same retention rules as other configuration records. The step is done when the evidence bundle is filed and the rollback window is formally closed. Rollback point: after closure, reverting means a new change with its own backup and test cycle, so confirm this before closing.
What makes this APISIX upgrade longer or riskier
This table lists the factors that most change the effort and risk of moving from APISIX 3.10 to 3.12, and what reduces each one. Read each row against your own inventory of routes, consumers and plugin code before estimating the work. The main takeaway is that the jwt-auth and hmac-auth changes affect client integrations, not only gateway configuration, so those consumers usually set the pace.
| Driver | Why it matters | What reduces it |
|---|---|---|
| Routes using jwt-auth | The 3.11 release removed the jwt-auth signing endpoint and private key upload, so any route or workflow that depended on the gateway issuing tokens must be redesigned before cutover. | Move token issuance to your identity provider and confirm which routes only validate tokens. |
| Routes using hmac-auth | The hmac-auth rewrite in 3.11 changed how keys are declared, since the required access_key field became key_id, and every affected route needs a config change. | Inventory hmac-auth routes and rename fields in version-controlled configuration before the upgrade. |
| Consumers and clients that sign requests | Client code that signs requests may need new signing inputs or header handling, and those changes sit outside the gateway team's control. | List every signing client, its owner and its release cycle, and test each one against a staging gateway. |
| Use of opentelemetry attributes | The 3.12 release moved opentelemetry attributes into plugin metadata, so existing attribute settings stop applying until they are rewritten. | Find every opentelemetry block and move its settings to plugin metadata in a dry run. |
| Use of ai-content-moderation or ai-proxy passthrough | The plugin was renamed to ai-aws-content-moderation and ai-proxy passthrough was removed in 3.12, so routes and dashboards tied to the old names fail or go silent. | Search configs and alerts for the old names and plan replacements with the owning team. |
| Empty key attributes in plugin configs | Version 3.12 disallows empty key attributes, so configs that previously loaded with blank values will be rejected. | Validate all plugin configs against the target version in staging before production. |
| Custom Lua or plugin runner code | Custom plugins that read changed fields, or that depend on the old opentelemetry or hmac-auth structures, need code review and tests. | Keep a test suite for custom plugins and run it against the target APISIX version. |
| etcd data volume | Larger etcd datasets take longer to back up, verify and roll back, and they increase the time a cluster spends in mixed state. | Take and verify an etcd snapshot before each step and confirm restore time in a test environment. |
| Number of environments and version jumps | Each environment needs its own test pass, and a jump past 3.12 to 3.17 or later adds the security default changes of those releases on top of this one. | Upgrade one environment at a time and decide the target version before scoping the work. Confirm the release notes for every version between your start and target. |
Questions teams ask before upgrading APISIX from 3.10 to 3.12
Can our own team upgrade APISIX from 3.10 to 3.12 without a partner?
Yes, a team that already runs APISIX in production can usually do this in house, if it has time to read the changes and test them against its own routes. The work is mostly discovery: finding every route, consumer and client that depends on jwt-auth, hmac-auth, opentelemetry or the AI plugins that changed. Teams with a staging gateway, a current inventory of consumers and someone who has maintained custom plugins on these versions tend to manage it well. Teams without those pieces should weigh a fixed-scope review before they start. Also confirm who can approve a change to authentication behavior before the cutover date is set.
What drives the cost and time of this upgrade?
The main drivers are the number of routes that use the changed plugins, the number of consumers and client applications those routes serve, and how many environments must be upgraded and tested in sequence. Custom plugins that read opentelemetry attributes, reference the renamed AI moderation plugin, or depend on certificate trust settings add review time. Whether you run the APISIX Ingress Controller, whether tests are automated, and how many partner clients must coordinate a change window also matter. No fee or duration is given here because the answer depends on your inventory. Ask any partner to base its estimate on a written list of those counts.
What does the upgrade need from our staff?
Route owners need to confirm which routes use which authentication and AI plugins, and what each route is expected to do after the upgrade. Client teams need to confirm how their applications send JWT tokens or signed requests, and they must test against staging before production. Security staff should review the changes to authentication and approve the acceptance tests. A platform owner needs to schedule the change window, control etcd and deployment access, and keep a rollback path ready. Plan these people's time up front, because most delays come from waiting on answers from these groups rather than from the gateway itself.
How do we handle partner clients that sign requests with hmac-auth?
Start by inventorying every partner that signs requests to your gateway, and record the signing algorithm, the headers they sign and the key identifier they send. APISIX rewrote the hmac-auth plugin in 3.11, so a signing client that worked before may need adjustment. Agree on a test plan with each partner, run signed requests against a staging gateway on 3.12, and confirm the results in writing. Schedule the production change with the partner, keep the previous configuration available for rollback, and watch authentication failures closely after cutover. If you plan to move beyond 3.12 later, also review the signed_headers default change in 3.17.
Is APISIX 3.12 supported long term?
The APISIX downloads page does not mark any release as long-term support, so you should not assume 3.12 has a fixed support window. The same page lists APISIX 3.19.0 as released on September 28, 2026, so 3.12 is several releases behind the current line. Confirm the project's release and support policy directly, and ask any vendor you use for its own support commitments in writing. APISIX is released under the Apache License 2.0, so the software itself carries no license fee, but support contracts are separate. Build your plan around a regular upgrade cadence rather than a single long-lived version.
Does the hmac-auth key_id change affect Ingress Controller users?
Yes, if you use hmac-auth through the APISIX Ingress Controller. The Ingress Controller 2.0 upgrade guide notes that the required access_key field became key_id for APISIX 3.11 and later. Review every consumer manifest that defines hmac-auth credentials and update the field before you upgrade the gateway. Then deploy the change to a non-production cluster and send test requests from each client that authenticates with those credentials. Keep the old manifests in version control so you can roll back if a client fails. The upgrade guide is the reference to check against your controller version.
What should we ask a partner for in a fixed-scope quote?
Ask for a written scope that names the APISIX versions, environments, routes, consumers and plugins included. The quote should describe how the inventory is built, the test plan and acceptance criteria, and who performs each test. Look for the rollback procedure, the client coordination responsibilities and the change window assumptions. List what is excluded, such as custom plugin rewrites, Ingress Controller changes or later upgrades. Confirm the deliverables and their timing, and ask whether post-upgrade support is a separate retainer. DeployOpen agrees scope, deliverables and timelines in writing before work starts, so a quote should follow directly from that document.
Which 3.12 changes break custom plugins or dashboards?
Several 3.12 changes can break existing configuration or tooling:
- The opentelemetry plugin moved its attributes to plugin metadata, so collectors and route configs that set them directly need review.
- The ai-content-moderation plugin was renamed to ai-aws-content-moderation, so routes that reference the old name will fail.
- The ai-proxy passthrough option was removed, and empty key attributes are no longer accepted.
- ssl_trusted_certificate now defaults to system, so upstreams that relied on a custom CA bundle need an explicit setting.
Search your repositories and route exports for each name before you test. The APISIX changelog lists the exact changes for the release.
What the upgrade asks of your security and client teams
The client communication question comes first. The 3.11.0 change to hmac-auth rewrote the plugin, and the APISIX Ingress Controller 2.0 upgrade guide notes that the required access_key field became key_id for APISIX 3.11 and later. Any consumer that signs requests with HMAC needs to know which identifier and signing details it must send after the change, and that information has to reach partner and internal API teams before cutover, not after the first rejected call. Also confirm with the client teams whether they depend on the default for signed_headers, which the project changed to date in 3.17.0, since that affects a later step in the same plugin lifecycle. Ask your release manager to hold the consumer notice until the staging test has confirmed the exact fields.
The key custody review is the second item. APISIX 3.11.0 removed the jwt-auth signing endpoint and the private key upload, so tokens that were once issued through the gateway now have to come from an identity provider or a signing service you control. Your security team should confirm where the private keys live, who can read them, how they are rotated, and whether the signing service is in the same accreditation boundary as the gateway. Record the decision in the change record, because the removal moves a control from the gateway to another system, and the auditor will ask where that control now sits.
The third item is approval for AI moderation renames on routes that fall under compliance scope. In 3.12.0, ai-content-moderation became ai-aws-content-moderation, and ai-proxy lost its passthrough behavior. Route owners and the compliance lead should approve the new plugin name, the updated configuration, and any change to what content is blocked or logged. For the evidence file, keep the before and after route configuration, the test results from staging, and the approval record. DeployOpen prepares this evidence for your review, but independent auditors and certification bodies decide whether it meets the requirements that apply to you.
How DeployOpen scopes and delivers an APISIX upgrade
DeployOpen approaches the APISIX 3.10 to 3.12 move as a written, fixed-scope engagement inside your own infrastructure. We start from your actual jwt-auth, hmac-auth, opentelemetry and AI plugin usage, then plan, test and document each change before anything reaches production.
How we scope
- We inventory which routes use jwt-auth, hmac-auth, opentelemetry and ai-content-moderation across your gateways and configuration.
- We identify consumers that depend on the removed jwt-auth signing endpoint or the rewritten hmac-auth fields.
- We agree the scope, deliverables and timeline in writing before any work starts, with a quote following from that scope.
How we build
- We move opentelemetry attributes into plugin metadata and verify that exported traces still carry the expected fields.
- We rename ai-content-moderation references and remove ai-proxy passthrough settings, then test each affected route against staging.
- We check system CA trust for upstreams and confirm that empty key attributes are no longer accepted by your configuration.
How we hand over
- You receive a change record listing each affected plugin, route, consumer and configuration key, with the before and after state.
- We document the rollback steps for each change so your team can reverse a single affected route without a full restore.
- We walk your platform and security staff through the new hmac-auth and jwt-auth behavior so runbooks match the running gateway.
How we support
- Ongoing support, further upgrades and continuous testing are available as a retainer, scoped separately from the upgrade.
- We can rerun validation checks when you move to a later APISIX release, using the same change record as the baseline.
- Confirm with your team and the project's upgrade notes which later changes apply to your environment before scoping the next step.
