Why an Apache APISIX 3.9 to 3.10 upgrade needs a pre-change review
Moving a gateway from APISIX 3.9 to 3.10 is mostly a review of changed defaults and custom plugin code, not a feature migration. APISIX 3.10.0 was released on August 14, 2024, and the changes that matter for this step affect how the gateway behaves on upgrade: the Admin API key is generated automatically when none is passed, sensitive fields are encrypted by default, the default * for CORS expose_headers is removed, and core.request.header now returns strings. A direct move from 3.9 is the path to plan for, but only after you have checked each of these against your own configuration and plugins.
Start by listing where your team depends on the old behavior. Check whether your deployment passes an Admin API key explicitly or relies on a generated one, whether any clients read CORS response headers that the browser previously received through the wildcard, and whether any custom Lua plugin reads request headers and expects a value other than a string. Also check whether you hold more than 100 headers in a single get_headers() call, since that call is now capped. Your own plugin code and your configuration files are the inputs for this review, so the review cannot be done from the release notes alone.
The page does not mark APISIX releases as long-term support, so confirm which 3.10 patch release you will target before you schedule the change. Check the APISIX downloads page for the current release line, and read the APISIX changelog for every patch between 3.10.0 and your target. Record the version you test against, because fixes in later patches can change the outcome of the checks you run.
Which 3.10 changes affect your gateway
This table lists the APISIX 3.9.0 and 3.10.0 changes that can alter gateway behavior, automation or custom plugins. Read each row against your current configuration: the impact column says what breaks or shifts, and the action column says what to check before cutover.
| Change | Impact | Action needed |
|---|---|---|
| Admin API key auto-generated in 3.10.0 | If no Admin API key is passed at startup, APISIX generates one, so unattended deployments may run with a key nobody recorded. | Check automation and provisioning scripts. Confirm the Admin API key is set explicitly and stored in your secrets system. |
| Sensitive field encryption on by default in 3.10.0 | Sensitive fields stored in etcd are encrypted by default, so tools that read raw etcd values or depend on plaintext fields may fail. | Audit tooling that reads etcd directly. Confirm the encryption key setup and test backup restore with encrypted fields. |
| CORS expose_headers default removed in 3.10.0 | The default * for expose_headers no longer applies, so browser clients may stop seeing response headers they relied on. | Check every CORS-enabled route and list the headers each browser client reads. Set expose_headers explicitly where needed. |
| get_headers() limited to 100 headers in 3.10.0 | Custom plugins that read more than 100 request headers will receive a truncated set, which can change authentication or routing decisions. | Audit custom plugins for get_headers() calls. Confirm that expected header counts stay below the limit for your traffic. |
| core.request.header returns strings in 3.10.0 | Custom plugins that expected other value types from core.request.header may fail comparisons or type checks after the upgrade. | Audit custom plugin code for type-dependent logic. Add explicit string handling and test each plugin path. |
| Keyring and key_encrypt_salt unified in 3.9.0 | Older separate keyring and key_encrypt_salt settings are consolidated, so configuration files written for earlier fields need review. | Review apisix config files and secret settings from the 3.8 line. Confirm the unified keyring fields are populated before moving to 3.10. |
| HTTP/2 enabling moved in 3.9.0 | HTTP/2 is now enabled through apisix.enable_http2, so older HTTP/2 settings no longer take effect. | Search config templates for HTTP/2 flags. Confirm the new apisix.enable_http2 setting in each environment and test client connections. |
What to inventory and back up before 3.10
The 3.10 changes touch the Admin API key, sensitive field storage, CORS responses and custom plugin header handling. Each card below names one thing to capture or verify before you change the running version, so a rollback stays possible and no consumer is surprised on cutover.
Back up etcd and configuration files
Take a snapshot of the etcd data that holds routes, upstreams, consumers and plugin configuration, and copy the conf/config.yaml file and any custom plugin files from every node. Store the backup in a location the upgrade cannot overwrite. Because 3.10 turns on encryption of sensitive fields, confirm you also keep the key material needed to read those fields after a restore. Test a restore in a scratch environment before you rely on it.
Inventory the Admin API key
List every place that sets or reads the Admin API key, including conf/config.yaml, environment variables, secret stores and deployment manifests. In 3.10 the key is auto-generated when none is passed, so a node that starts without an explicit key may run with a key nobody recorded. Decide whether you will pin an explicit key before the upgrade and who will hold it.
Find automation that calls the Admin API
Search CI pipelines, infrastructure-as-code repositories, scheduled jobs and runbooks for calls to the Admin API. Any script that depends on a key being present without being set explicitly needs an update. Record the caller, the endpoint it uses and the credential it reads, and confirm that each one fails loudly, rather than silently, if the key changes.
List CORS consumers of the default expose_headers
In 3.10 the default * for CORS expose_headers is removed. Identify every route that uses the cors plugin and every browser client that reads response headers. Ask each application owner which headers it reads, and set those names explicitly in the route or plugin configuration. Without this list, browser code may fail on headers it previously received.
Inventory custom plugins reading request headers
Review every custom Lua plugin and every plugin-runner plugin in Java, Go, Python or WebAssembly. Flag any code that calls core.request.header or relies on get_headers() returning more than 100 headers, since 3.10 changes both. Record each function's expected return type and whether it can receive a large header set. These plugins are the most likely to need code changes before cutover.
Build a test environment from production config
Create a staging gateway with the same apisix.yaml or etcd dataset, the same plugin list, the same custom plugin code and the same upstream stubs as production. Run your flagged plugins and CORS requests against it on 3.10 before any production node changes. Ask your team or partner to confirm the test copy matches production in plugin versions, certificates and key handling, not only routes.
How the Admin API key and encryption defaults change your setup
In APISIX 3.10, if you do not pass an Admin API key, the gateway generates one for you. That is a change in behavior for any environment that assumed a known key already existed. Pipelines that call the Admin API on port 9180, Helm values that set the key, and configuration management tools that template the key into config.yaml may all be written around a value you chose. If your deployment relies on the key being present before the first start, a generated key can leave automation unable to authenticate. Before cutover, confirm where your process expects the key to come from, decide whether you will set it explicitly in every environment, and check how your secrets store feeds the value into the gateway so that the key is not lost or regenerated on redeploy.
The second change is that sensitive field encryption is enabled by default in 3.10.0 (released August 14, 2024). Fields such as credentials and secrets stored in etcd are encrypted at rest with a key derived from the keyring and key_encrypt_salt settings that APISIX 3.9 unified. The part you must confirm is what happens to values already stored under your 3.9 configuration. Ask whether existing records are read and rewritten under the new defaults, whether your keyring and salt values are the ones you expect, and whether a rollback to 3.9 can still read data written after the upgrade. Test this on a copy of your etcd data, not on production.
Verify the exact behavior for your target version in the APISIX changelog and the 3.10.0 release notes on GitHub before you write the change plan. The changelog entries are the reference for default values and migration steps, and this page summarizes them. If the release notes and your test results disagree, treat the release notes and your own reproduction as the source of truth and raise the question with the APISIX community before cutover.
How to sequence the 3.9 to 3.10 upgrade
This sequence moves an APISIX 3.9 gateway to 3.10 in a controlled order, so each behavior change is checked in a non-production cluster before production traffic moves. Each stage closes with a check and a rollback point.
Freeze configuration changes
The platform owner stops routine route, plugin and consumer changes for the duration of the upgrade and records the current state of the gateway. Export the route, upstream, plugin and consumer definitions through the Admin API so the team has a reference copy to compare against later. The step is done when the change freeze is announced to the teams that publish routes and the exported state is stored in version control or another controlled location.
Back up etcd and configuration files
The platform team takes a snapshot of the etcd data used by the gateway and copies the APISIX configuration files, including config.yaml and any custom plugin code, to a location outside the cluster. The team restores the snapshot into a scratch environment to confirm it is readable. The step is done when a restore test succeeds and the backup location, owner and retention are written into the change record. This snapshot is the rollback point for the data layer.
Upgrade a non-production cluster
The team deploys APISIX 3.10 to a non-production cluster that holds a copy of the production configuration, using the same deployment method as production. The team reviews the 3.9.0 changes already in the changelog so that the 3.10 settings are applied deliberately, including the unified keyring and key_encrypt_salt fields and the move of HTTP/2 enabling to apisix.enable_http2. The step is done when the gateway starts cleanly, loads all routes and reports healthy upstreams. Rollback point: redeploy the 3.9 image or package against the same etcd snapshot.
Set an explicit Admin API key
Because 3.10 generates an Admin API key automatically when none is passed, the team sets an explicit key in the configuration before the upgrade reaches production. Every script, CI job and controller that calls the Admin API is updated to send the explicit key, and the team confirms that no caller depends on a generated value. The step is done when a test call with the explicit key succeeds and a call with a missing or wrong key is refused. Rollback point: restore the previous configuration file, which still holds the 3.9 key setting.
Verify CORS and header-dependent plugins
The team checks the effects of the removed default * for CORS expose_headers, the limit of get_headers() to 100 headers, and the change that makes core.request.header return strings. Custom plugins and any Lua code that reads headers or assumes a non-string return are reviewed and tested with representative requests. Browser clients that read response headers are checked against the new expose_headers values. The step is done when each custom plugin passes its tests and each route that needs cross-origin headers returns the expected values. Rollback point: revert the plugin package and CORS settings to the 3.9 versions in the same release.
Run traffic tests against the upgraded cluster
The team replays representative production traffic against the 3.10 cluster, including authenticated requests, error paths and large header sets, and compares status codes, latency and upstream behavior with the 3.9 baseline. Logs and metrics are checked for new errors and for dashboards or alerts that depend on changed label names. The step is done when the test results show no unexpected differences and the owners of each affected route have signed off. Rollback point: keep the 3.9 cluster running and available until production promotion is approved.
Promote to production with a rollback point
The team repeats the backup and configuration steps on the production cluster immediately before the change window, then upgrades one cluster or node group at a time while watching health checks and error rates. Each stage is confirmed before the next starts, and the freeze is lifted only after the final stage passes. The step is done when all production nodes run 3.10, traffic metrics match the baseline and the change record is closed. Rollback point: redeploy the 3.9 nodes against the pre-change etcd snapshot, as tested in the non-production cluster.
How to test 3.10 and roll back safely
Before you cut production traffic over to APISIX 3.10, run the test cases that this release is most likely to break: Admin API calls made without a supplied key, browser CORS preflight requests from real clients, custom plugins that call get_headers() on requests carrying many headers, and plugins that read core.request.header and expect a string. For the Admin API, call every automation path you run, including CI jobs, configuration scripts and dashboards, and confirm that each one either passes its own key explicitly or handles the generated key correctly. For CORS, send preflight OPTIONS requests from each front-end origin you serve, using the browsers your users actually run, and compare the Access-Control-Expose-Headers response against your 3.9 baseline. Because the wildcard default was removed, any client that relied on reading arbitrary response headers will now fail, and that failure will show up only in the browser.
For custom plugins, write a test request with more than 100 headers and confirm that your code handles the truncated result from get_headers() without silently dropping authentication or routing data. For core.request.header, run each plugin path that reads a header and confirm it does not compare the value to a number or table. Use the release notes in the apache/apisix CHANGELOG to list every custom plugin that touches these functions, so the test list comes from code rather than memory.
Rollback means restoring the saved 3.9 configuration and the 3.9 binaries or images together, then confirming the gateway starts and serves the routes you tested. Before you roll back, confirm what 3.10 has written. Check whether the Admin API key was generated and stored, whether sensitive fields were rewritten under the encryption default, and whether any configuration was saved in a format 3.9 cannot read. Ask your team to document how those values will be decrypted or reissued under 3.9 before the rollback window opens. Keep the pre-upgrade etcd snapshot and configuration export, and rehearse the restore in a non-production environment first. Confirm the exact restore steps with the APISIX releases page for your target build.
Which upgrade approach fits your environment
In a Kubernetes deployment, APISIX usually runs as the gateway data plane under the APISIX Ingress Controller, which is a separate project with its own release line. The 3.10 changes reach you through the gateway image, the chart values that pass settings to it, and any custom plugins baked into the image. Routes and consumers are expressed as Kubernetes resources, so the upgrade is mostly a change to manifests and values rather than a data migration, but you still need to confirm that the gateway version and the controller version you pin are compatible.
- Deployment manifests: Helm values or Deployment specs that set the APISIX image tag move to a 3.10 image. Review any
config.yamloverrides for the removed default*on CORSexpose_headers. - Admin API key: If your values do not set an Admin key, 3.10 generates one. Set the key explicitly in a Kubernetes Secret so controllers and automation keep a predictable credential.
- Sensitive field encryption: Encryption is on by default in 3.10, so confirm the keyring and
key_encrypt_saltsettings carried over from 3.9 are correct before the rollout. - Custom plugins: Plugins that call
core.request.headerorget_headers()must be tested against string return values and the 100-header limit.
Before changing any image tag, check the APISIX changelog and confirm the controller version you run is documented as compatible with APISIX 3.10 in its own release notes.
In traditional mode, the data plane and control plane run together on each node, and routes, consumers and plugin settings are stored in an etcd cluster. APISIX listens on port 9080 for client traffic and port 9180 for the Admin API. The upgrade therefore touches three things at once: the conf/config.yaml file on each node, the Admin API key that automation uses, and the sensitive values already stored in etcd. Because all nodes share that etcd cluster, the order and timing of node restarts matters as much as the new settings.
- Config file keys: The 3.9 change unified the keyring and
key_encrypt_saltfields and moved HTTP/2 enabling toapisix.enable_http2. Update each node's file before the restart. - Admin API key: Scripts that call port 9180 without a key will depend on the generated key, so decide and store the key deliberately.
- Sensitive fields in etcd: Back up etcd before the rolling restart, then read back a sample of encrypted values to confirm the new setting works with your existing data.
- Rolling restarts: Confirm with the deployment modes documentation how mixed-version nodes behave against the shared etcd cluster during the rollout.
Before you start, confirm each step against the deployment modes documentation and the 3.10 entries in the project changelog.
In standalone mode, APISIX runs without etcd and reads its routes and upstreams from conf/apisix.yaml or conf/apisix.json. The file is checked for changes every second and hot-reloaded without a restart. This makes the file itself the main upgrade artifact: it is the source of truth for routes, and it must stay valid for the new version. The 3.10 changes to the static configuration still apply to conf/config.yaml on each node, so the upgrade has two files to review rather than one.
- Route file: Keep the existing
apisix.yamlorapisix.jsonand validate it against the 3.10 behavior for header handling and plugin fields before swapping it in. - Hot reload risk: Because changes take effect within a second, write the file atomically and validate it before replacing the live copy, so a partial edit does not reach traffic.
- Admin API key: If you use the Admin API in standalone mode, the generated key matters. If you do not, set the key explicitly in
config.yamlanyway so the setting is documented. - Encryption settings: Confirm with the documentation whether sensitive field encryption applies to values in the file, since there is no etcd store involved.
Before you change either file, confirm the file format and reload behavior for APISIX 3.10 in the deployment modes documentation and test the new configuration on a non-production node.
Questions teams ask before an APISIX 3.9 to 3.10 upgrade
How much effort does an APISIX 3.9 to 3.10 upgrade take?
The effort depends mostly on how much of your configuration and custom code touches the three areas that changed in 3.10: the Admin API key, sensitive field encryption and CORS headers. A gateway that uses only routes and built-in plugins, with the Admin API managed by a pipeline, usually needs a short review and a test pass. Effort rises when custom Lua plugins call core.request.header or get_headers(), when automation depends on the Admin API generating or accepting keys in a particular way, or when many routes depend on CORS expose_headers behavior. Each of these adds review, test cases and rollback planning. Ask any partner to list the custom plugins, automation scripts and CORS rules in scope before agreeing on effort.
How long does an APISIX 3.9 to 3.10 upgrade take?
Duration depends on scope and environment more than on the version step itself. The main drivers are the number of gateway nodes and environments (development, test, staging and production), how many custom plugins need review and retesting, how mature your test suite is for routes and authentication, and how many change windows your organization allows. A single traditional-mode cluster with no custom plugins moves faster than a decoupled deployment with several etcd-backed clusters and plugin runners in production. Ask for a timeline that lists each environment, the test evidence expected at each gate and the rollback point, rather than a single overall figure.
What does the upgrade need from our internal team?
Your team needs to supply knowledge and access that an outside party cannot reconstruct quickly. Typical inputs include the inventory of routes, consumers and plugins, the deployment mode in use (traditional, decoupled or standalone), the list of automation that calls the Admin API, the authors of custom plugins and their test cases, and a named owner who can approve changes in each environment. Someone must also confirm the change window, the monitoring that will show regressions and who signs off on production cutover. Without these inputs, time shifts into discovery, which is usually the largest unplanned cost in an upgrade.
Can our own team upgrade APISIX 3.9 to 3.10 without outside help?
A team that already runs APISIX in production, has tested configuration and can read Lua or the plugin runner code it depends on can usually complete this upgrade internally. The release notes are short in scope, but the work still needs a staging environment that mirrors production, a rollback plan and someone who reviews every custom plugin against the changed behavior of core.request.header. Teams with limited APISIX experience, no test coverage for CORS or custom authentication, or no spare staging capacity often find outside review useful. Check the project's CHANGELOG against your own inventory before deciding.
What does a fixed-scope quote for this upgrade cover?
A fixed-scope quote is built from a written list of what will be done, not from the version number alone. It should name the environments in scope, the deployment mode, the configuration and automation to review, the custom plugins and their tests, the CORS and header behavior to verify, the Admin API key handling, the rollback procedure and the handover documentation. It should also state what is excluded, such as upgrades of etcd, Kubernetes or the Ingress Controller, and any integration work that a later release would require. DeployOpen agrees scope, deliverables and timelines in writing before work starts, so ask for these items to be listed explicitly.
What should we check in a custom plugin before upgrading to 3.10?
Check every custom plugin for three behaviors changed in 3.10. First, any code that calls core.request.header must handle string return values, so review any logic that assumed a different type. Second, any code that calls get_headers() must cope with the 100 header limit, so confirm that requests with more headers are handled as intended. Third, any plugin that reads or writes sensitive fields, or that relies on CORS expose_headers defaults, must be tested with encryption enabled and with explicit CORS values. Run each plugin's tests against a 3.10 build in staging and record the results before promotion.
Which upgrade risks should we review before the change window?
The main risks are silent behavior changes that do not produce errors. Examples include a Lua plugin that now receives a string where it expected another type, a client that stops receiving expected CORS headers after the default * for expose_headers was removed, and automation that stops working once the Admin API generates a key when none is passed. Encryption of sensitive fields is on by default, so confirm that existing stored values are readable after the upgrade and that key handling in your secrets process matches the new behavior. Plan a rollback that restores the previous image and configuration, and test it before production.
What makes this upgrade take longer
This table lists the factors that lengthen a move from APISIX 3.9 to 3.10 and the direction each one pushes effort. Read it against your own gateway inventory to see which rows apply and where the most testing time will go. The main takeaway is that custom plugins and header-dependent behavior usually set the schedule more than the core configuration change does.
| Driver | Why it changes effort | How to keep it down |
|---|---|---|
| Custom plugins written in Lua | Lua plugins that call core.request.header or get_headers() may need code changes because the return type and header limit changed in 3.10. Effort rises with each plugin that reads request headers. | Inventory every custom plugin and grep for header access. Confirm the exact behavior against the 3.10 changelog before estimating. |
| Custom plugins using plugin runners | Java, Go, Python and WebAssembly plugins sit outside the Lua code path, so each one needs its own check of header handling and its own test run. Effort rises with the number of runners in use. | Confirm with each runner's maintainers whether they depend on the affected core behavior, and test each runner separately. |
| Reliance on CORS expose_headers | Clients that depended on the old default of * for expose_headers will stop receiving headers they read. Effort rises with the number of browser clients and routes that rely on the default. | Set expose_headers explicitly on each affected route and verify with the front-end teams before cutover. |
| Header count and size in requests | Requests carrying more than 100 headers may behave differently under the new get_headers() limit. Effort rises for gateways fronting clients that send large header sets, such as some enterprise proxies. | Sample real traffic for header counts and test the heaviest clients first. |
| Number of routes and consumers | Each route and consumer is a potential place for a changed default or encrypted field to matter. Effort scales with the size of the configuration that must be reviewed and tested. | Export configuration as code and diff it between a staging and a test instance so changes are reviewed in bulk. |
| Admin API automation | Scripts and pipelines that create resources without passing an Admin API key depend on the 3.10 auto-generated key behavior. Effort rises with the number of automation jobs and their owners. | List every caller of the Admin API, set the key explicitly where automation needs it, and assign an owner for each job. |
| etcd data size and encrypted fields | Sensitive fields are encrypted by default in 3.10, so existing stored values and any tooling that reads them need checks. Effort rises with the volume of stored data and the number of tools reading it. | Back up etcd before the upgrade and confirm with a test restore that tooling reads the stored values as expected. |
| Number of environments and clusters | Each environment, data plane and control plane pair repeats the upgrade and its tests. Effort rises roughly with the count of clusters and the differences between them. | Upgrade a non-production cluster first and reuse the same runbook and test set for each later cluster. |
| Test coverage | Without automated tests for routes, authentication and plugin behavior, most of the effort shifts to manual verification. Effort falls as coverage of these paths improves. | Build regression tests for the affected routes and plugins before the upgrade, and ask any partner to state what they will test. |
How DeployOpen scopes and delivers an APISIX upgrade
DeployOpen starts from your running APISIX 3.9 configuration, Admin API usage and custom plugins, then agrees a written scope before any change is made. The work covers the 3.10 changes to the Admin API key, sensitive field encryption, CORS headers and header handling, tested in a staging copy of your environment.
How we scope
- Inventory your APISIX 3.9 deployment mode, Admin API callers, automation and configuration sources before quoting.
- Identify custom plugins that call core.request.header or get_headers() and need review for string return values.
- Agree the written scope, deliverables and timeline in writing before work starts, as engagements are fixed-scope.
How we build
- Reproduce your production configuration in a staging environment running the 3.9 release, then upgrade it to 3.10.
- Set the Admin API key explicitly so automation no longer depends on a generated value, and test encryption of sensitive fields.
- Review CORS expose_headers settings and header-limit behavior with test requests from your real client applications.
How we hand over
- Deliver a documented change record covering each 3.10 behavior change, the affected routes and plugins, and the test results.
- Provide a rollback procedure for the upgrade window, including how to restore the previous configuration and keys.
- Walk your platform and security teams through the changed defaults so the next upgrade review starts from a known baseline.
How we support
- Ongoing support, upgrades and continuous testing for APISIX are available as a retainer after handover.
- Confirm with your team which later release lines you plan to follow, since each upgrade needs its own scoped review.
- Request a quote based on the written scope, which states who performs staging, testing, cutover and documentation.
