Why upstream TLS verification in APISIX matters before you upgrade
If your gateway forwards to internal services over HTTPS or gRPC over TLS, check how upstream certificates are verified before you upgrade to APISIX 3.19.0 or later. From that release, APISIX honours upstream.tls.verify for https and grpcs upstreams, not only for Kafka. Any route whose upstream is already set to verify: true will start rejecting certificates it cannot validate, and the failure appears as upstream errors at the gateway rather than as a configuration warning. Teams that set verify: false for convenience, or that never enabled verification, may see no change at all, which is why the inventory matters more than the upgrade itself.
Start by listing every upstream that uses https or grpcs, the route or service it belongs to, and whether its certificate chains to a CA the gateway trusts. Confirm with your platform team which CA issued each internal certificate and whether the chain includes intermediates. Since 3.12.0, APISIX sets ssl_trusted_certificate to the system certificate store by default, so private or internal CAs usually need to be added to that store or configured explicitly. Test a staging gateway against the same upstreams before you change production, and review the APISIX changelog for the exact release notes that apply to your version.
Upstream mTLS is needed when a service must authenticate the gateway itself, typically for internal APIs, partner integrations or zero-trust segments where network location is not enough. In that case APISIX presents a client certificate and key to the upstream, which is a separate decision from verification. This page covers verifying upstream servers, presenting client certificates to upstreams, the system CA trust behavior, and how to diagnose services that begin failing after the upgrade. Confirm with your security team who issues gateway client certificates and how they are rotated before you plan the change window.
Which upstream TLS settings fit each service type
Internal services usually sit behind an organization's own certificate authority, which the system trust store does not include. For these backends the gateway needs the internal CA added to its trust path before upstream.tls.verify is turned on. Verification is the setting that protects the connection from impersonation, but it only helps when the CA bundle is correct, so the work is mostly about getting trust right for every backend.
- Trust bundle first: Confirm the internal root and any intermediates are available to the gateway nodes and included in the trust configuration before enabling verification.
- Hostname matching: Check that each certificate's subject or SAN matches the upstream address APISIX connects to, since a service reached by IP or an internal alias often fails here.
- Staged enablement: Turn on verification for one upstream at a time, watch logs for handshake failures, and keep a rollback path for each change.
- Certificate rotation: Ask the CA owner how renewals and CA rollovers will reach gateway nodes, because a rotated internal CA can break verified upstreams without any change on the gateway.
- Lowest migration risk: Services that already present certificates from a known internal CA are usually the easiest to move to verified mode.
Before switching a private CA upstream to verified mode, confirm the trust bundle, the hostname match and the owner of the certificate renewal process.
Backends reached over the public internet, including SaaS APIs and partner endpoints, present certificates from public CAs. For these, the system CA store is the trust source, and APISIX 3.12.0 set ssl_trusted_certificate to the system store by default, so most public upstreams need no custom trust configuration. The main risk is not missing trust but unexpected rejection when a backend's chain or certificate changes, or when a previously unverified upstream starts being checked.
- Default trust source: Confirm the gateway image or host carries an up-to-date system CA store, since trust decisions follow what the store contains.
- Upgrade effect: Upstreams already set to
verify: trueon https or grpcs routes start rejecting untrusted certificates from 3.19.0 onward, so inventory them before the upgrade. - Chain completeness: Ask partners whether they serve complete certificate chains, because incomplete chains fail verification even when the leaf is valid.
- Expiry monitoring: Track certificates of external dependencies, since an expired partner certificate now causes a gateway failure rather than a silent pass-through.
- Partner coordination: Agree on change notices with external providers before enabling verification on their endpoints.
Before upgrading, list every https and grpcs upstream with verification enabled and confirm each provider's chain and renewal practice.
Some services require the gateway to prove its identity with a client certificate, which is mutual TLS from APISIX to the upstream. The relevant settings are upstream.tls.client_cert and upstream.tls.client_key, which hold the certificate and private key the gateway presents. This adds authentication in both directions, but it also adds a credential with its own lifecycle, storage and rotation needs that the gateway team must own.
- Identity per route: Decide whether one gateway identity serves many upstreams or each upstream receives its own client certificate, since a shared identity makes revocation broader.
- Key storage: Keep the private key out of plain configuration where possible, and confirm how it is protected in etcd and on disk, as covered in the secrets guidance for the gateway.
- Rotation coordination: Ask the upstream owner how they trust the gateway certificate and how they will accept a replacement, so renewal does not cause an outage.
- Verification pairing: Use
verify: truetogether with client certificates so the gateway also checks the upstream's identity, not only presenting its own. - Highest coordination cost: These upgrades usually involve two teams and a cutover window, so they carry more migration risk than the other two situations.
Before enabling client certificates, confirm who issues and rotates the gateway identity and how the upstream owner will validate it.
Fixing services that start failing after the 3.19 upgrade
This sequence finds every upstream that relies on certificate verification, identifies which ones present certificates the gateway does not trust, and fixes the trust chain before the upgrade reaches production. The output is a tested gateway where verified upstreams connect without errors and the trust configuration is documented.
Inventory upstreams with verification enabled
Export the current route and upstream configuration from the Admin API or your configuration repository and list every upstream using https or grpcs with upstream.tls.verify set to true. Include upstreams defined inline on routes and those referenced by shared upstream objects, because both are affected. Record the owning team, the service hostname and the port for each entry. The step is complete when the list accounts for every verified upstream and each row has an owner. Upstreams that already set verify to false are not affected by this change, but record them so they can be reviewed later.
Check the certificate chain each upstream presents
For each listed upstream, retrieve the presented certificate chain with a TLS client such as openssl s_client run from a host in the same network zone as the gateway. Confirm that the chain is complete, that the leaf certificate matches the hostname APISIX connects to, and that the chain ends at a root or intermediate the gateway can trust. Note upstreams that serve only the leaf certificate, use an expired certificate or depend on a private CA. The step is complete when every upstream has a documented chain status, and the team has a list of those that will fail verification.
Confirm the trusted certificate store in use
Review the gateway configuration for ssl_trusted_certificate and check the release notes for the version you are running. From 3.12.0, APISIX sets this to the system certificate store by default, so a private CA must be present in that store on every gateway node or referenced explicitly. Confirm the setting on a staging node by reading the rendered NGINX configuration, and check that the CA bundle file exists and is readable by the APISIX process. The step is complete when you can state, for each gateway node, which store is used and which CA certificates it contains.
Add the private CA or correct the chain
For upstreams that depend on a private CA, add the root or intermediate certificate to the trust store used by the gateway, either in the system store on every node or through ssl_trusted_certificate. For upstreams with incomplete chains, ask the service owner to serve the full chain, since this is the correct fix and the gateway should not be told to skip verification. Record each change in your configuration repository with the owner's approval. The step is complete when the chain check from the previous step passes for every upstream, using the same tooling and vantage point.
Test verified upstreams in staging
Deploy the 3.19 build and the trust changes to a staging gateway that routes to staging copies of the services. Send representative requests to each verified upstream and watch the upstream health checks, the gateway error log and the upstream response status codes. Look for certificate verification failures, which appear in the error log with the upstream address, and for rising 502 or 504 responses on routes that previously worked. Repeat the test with a deliberately untrusted certificate to confirm that rejection is now enforced. The step is complete when every verified upstream returns normal responses and the error log shows no verification failures for them.
Roll out and monitor production upstreams
Roll out the upgrade and trust changes to production one gateway group at a time, keeping the previous version available for rollback. After each group switches, compare upstream health, error log entries mentioning certificate verification, and 5xx rates per route against the pre-upgrade baseline. Pause the rollout if a verified upstream begins failing, and restore the previous trust configuration for that upstream only while its owner corrects the certificate. The step is complete when every production gateway group runs 3.19 with no verification errors in the logs and the upstream inventory reflects the final state.
Document trust and set ongoing checks
Record the final trust store configuration, the list of private CAs and their owners, and the upstreams that enforce verification. Add an alert on certificate verification errors in the gateway error log and a scheduled chain check for each verified upstream, so expiry or a changed certificate authority is detected before it causes an outage. Ask your certificate owners to confirm renewal dates and the CA rotation process in writing. The step is complete when the runbook names an owner for each trusted CA and the alerts route to the team that runs the gateway.
How to configure mTLS from APISIX to upstream services
Mutual TLS from APISIX to an upstream is configured on the upstream object, where tls.client_cert and tls.client_key tell the gateway which certificate and private key to present when it connects to the service. The upstream then decides whether to accept that identity. This is a different control from client mTLS on an SSL object, where client.ca and client.depth tell APISIX to verify certificates presented by callers. The two settings point in opposite directions and protect different hops, so teams should not assume that enabling one covers the other. Client mTLS on an SSL object applies only to HTTPS listeners, while upstream client certificates are the gateway's own identity toward backends. Confirm the exact field behavior for your upstream scheme and APISIX version in the mTLS documentation before you rely on it.
Before configuring anything, decide who issues the gateway's client certificate and who trusts it. The upstream must trust the CA that signed the APISIX client certificate, and that trust is a configuration item on the service side, not something APISIX can infer. Keep the APISIX client certificate on a short, defined lifecycle issued from an internal CA or a secrets engine you operate, and record which upstreams trust which issuer. Plan storage so the private key is not written into version-controlled configuration in clear text, and give each APISIX environment its own identity so a compromised staging key cannot authenticate to production services.
Rotation is where most designs fail. A certificate that is replaced on the gateway but not yet trusted by every upstream, or an upstream that pins an old CA after the issuer changed, breaks traffic without a clear error on the gateway side. Plan overlapping validity: add the new CA to upstream trust first, roll the new client certificate to the gateway, then remove the old trust. Test this sequence in a non-production environment, and confirm with your upstream owners how they reload trust stores. Also check that upstream certificate verification is set as you intend, because verification behavior changed in APISIX 3.19.0 for https and grpcs upstreams, so untrusted server certificates may now be rejected.
Configuration options and when each one fits
This table compares the settings that control how APISIX verifies and authenticates connections to upstream services. Read each row against your own upstream inventory: which services present certificates from a public CA, which use a private CA, and which require the gateway to present its own client certificate. The main takeaway is that verification and client authentication are separate decisions, and enabling verification from 3.19.0 can break upstreams that worked before.
| Option | What it controls | When it fits | Trade-off |
|---|---|---|---|
| upstream.tls.verify | Whether APISIX checks the upstream certificate chain and hostname for https and grpcs upstreams, honored from 3.19.0 | Upstreams that present certificates issued by a CA the gateway trusts and that you want verified end to end | Upstreams already set to verify true may start rejecting connections after upgrade if their certificates are untrusted or misnamed |
| System CA trust (ssl_trusted_certificate) | Which CA bundle APISIX uses to validate upstream certificates, defaulting to the system store from 3.12.0 | Upstreams using certificates from public CAs already present in the operating system trust store | Trusts every CA in the system store, so the trust scope is broader than a single internal CA; confirm the image and base OS bundle |
| Private CA bundle | A dedicated CA file added to the trusted set so internal upstream certificates validate | Internal services signed by your own certificate authority, where the root is not in the system store | You own distribution, rotation and expiry of the bundle across every gateway node; a missed update causes outages |
| tls.client_cert and tls.client_key | The certificate and private key APISIX presents when it connects to an upstream, enabling mTLS to that service | Upstreams that require the gateway to prove its identity, such as zero-trust internal APIs or partner endpoints | Adds a private key to the gateway that must be stored securely, rotated and covered by the secrets process |
| Terminate mTLS at the gateway | Client certificates are validated by APISIX on the client side via the SSL object (client.ca, client.depth), and the request reaches the upstream as plain or re-encrypted traffic | Clients you control or partners who can present certificates, and where upstream services should not handle certificate parsing | Gateway becomes the single point that enforces client identity; the forwarded identity must be passed in headers and trusted only from APISIX |
| Pass through mTLS to the upstream | The client certificate is forwarded to the service, which performs its own validation, typically over TLS passthrough or a re-encrypted hop | Services that need the original client certificate or enforce per-service certificate policy | Gateway cannot inspect or apply plugins to the encrypted payload, which limits authentication, logging and WAF inspection |
| Decision to verify before enabling | Whether each upstream's certificate chain and hostname are tested against APISIX before the setting is turned on in production | Any environment upgrading to 3.19.0 or later with existing https or grpcs upstreams | Testing takes effort across every route, but avoids an outage that appears only after the upgrade |
Security and operational risks to plan for
Upstream TLS mistakes rarely fail loudly at first. They show up as expired certificates, services that stop answering after an upgrade, or trust settings nobody remembers changing. The cards below cover the risks we see most often and the control to put in place for each.
Client private keys left exposed
The key that proves APISIX's identity to an upstream is as sensitive as any credential. If it sits in a world-readable file, a Git repository or an unencrypted etcd record, anyone who copies it can impersonate the gateway. Check who can read the key file on each node, keep the key out of version control, and store it in a secret manager where your team can rotate it without editing configuration by hand.
Client certificate expiry
When an upstream client certificate expires, every route that presents it fails at once, and the error often looks like a generic upstream outage. Build an inventory of each upstream certificate with its expiry date, alert well before that date, and rehearse the swap on a staging gateway. Confirm with each upstream owner how they will renew their side of the trust relationship.
Overly broad trust stores
Trusting a large public or shared CA bundle lets any certificate issued under those roots pass verification for your upstream. Since 3.12.0, the default trust comes from the system store, which may contain more roots than one upstream needs. Review which CAs each upstream relies on, and where practical pin a private CA bundle for internal services.
Verification disabled as a shortcut
Setting verify to false on an upstream clears a failure quickly, but it removes the only check that the server is who the route expects. This setting often survives the incident that prompted it. Record every upstream where verification is off, set an owner and a removal date, and fix the trust chain instead of turning checks off.
Handshake failures going unlogged
A TLS handshake that fails between APISIX and an upstream may leave only a brief trace in the error log, and no request reaches the upstream. Without alerting on handshake errors, teams find out from customers. Confirm which log level and fields your deployment writes for upstream handshake failures, forward them to your log platform, and alert on a sudden rise in 502 or 503 responses per route.
Questions teams ask before changing upstream TLS
What will the upgrade break, and how do we find it before users do?
The upgrade most likely breaks upstreams that already set upstream.tls.verify to true for https or grpcs routes. From APISIX 3.19.0, the gateway honors that setting for these protocols, not only for Kafka, so a service with an untrusted or misconfigured certificate starts getting rejected. Before changing anything, list every route and upstream with verification enabled, then test each one against staging with the target version. Look for handshake errors in the gateway error log, and check whether the certificate chain includes the intermediate CA and whether the hostname matches the upstream address. Services that never set verification will not change behavior, but you should still confirm that, because the default trust store also changed in 3.12.0.
Can our own team do this work without a partner?
Yes, if your team already owns the APISIX configuration, the certificate authorities behind your upstreams, and a staging environment that mirrors production routes. The work is mostly inventory, testing and certificate handling rather than new APISIX features. A partner is more useful when many upstreams are involved, when certificate ownership is split across teams, or when the same team must also handle the upgrade, mTLS rollout and evidence collection at once. DeployOpen plans and implements this work inside your own infrastructure under a written scope, so the decision is about capacity and skills, not about where the gateway runs.
What drives the effort, and what affects the cost?
Effort rises with the number of upstreams that need verification, the variety of certificate sources, and how mature your internal PKI is. Each upstream needs its own trust path confirmed, so a few dozen services from one internal CA are far simpler than services split across public certificates, private CAs and vendor-issued certificates. Cost has one-time parts (inventory, testing, remediation, documentation) and recurring parts (certificate renewal, trust store updates and retesting on each upgrade). Ask any quote to separate these, to state which upstreams and certificate authorities are in scope, and to name who owns the CA and the renewal process. DeployOpen does not publish prices, and a quote follows from that written scope.
How long does testing take in our environment?
Duration depends on how many routes are affected, how many environments you can test in, and how quickly certificate owners respond to findings. A small estate with one internal CA and a staging copy of production routes can be verified quickly, while a larger estate with several CAs, vendor endpoints and change windows needs more rounds. No duration is fixed by the product. Ask for the testing plan to list the upstreams in each batch, the environments used, the rollback step for each batch, and the sign-off owner. Confirm the exact upgrade timing and behavior against the APISIX changelog and your own staging results, not against an assumed schedule.
What do you need from us to get started?
You need a complete inventory of upstreams that use https or grpcs, the certificate authority that signed each one, and the name of the team that renews it. Also provide a staging environment that can reach the same upstreams, read access to the APISIX configuration, and a named owner for each CA who can approve trust changes. If mTLS is planned, add the client certificate sources and their renewal owners. Without these inputs, the scope cannot be written accurately, so treat them as the first deliverable of any engagement.
Do we need mTLS from APISIX to our upstreams?
Not always. Standard TLS verification proves that the gateway is talking to the expected service, while mTLS also lets the upstream verify the gateway with a client certificate. Choose mTLS when an upstream must authenticate the caller at the transport layer, when a zero-trust policy requires it, or when a compliance control in your scope expects strong service identity. APISIX supports this through tls.client_cert and tls.client_key on upstreams, but each client certificate adds issuance, storage and rotation work. Confirm the requirement with the owner of each upstream and the relevant control owner before enabling it.
Which upstreams should we verify first, and which can wait?
Start with upstreams that already have verification enabled, because those are the ones the upgrade can break immediately. Next, verify upstreams that carry sensitive data or cross trust boundaries, such as partner or payment services. Leave low-risk internal services for later, but record their status so the gap is visible. This order lets you fix the highest-impact failures first and gives the audit trail a clear sequence of decisions.
How DeployOpen scopes and delivers upstream TLS work
DeployOpen plans, implements, tests and documents upstream TLS verification and mTLS for Apache APISIX inside your own infrastructure. The work starts with an inventory of every upstream the gateway calls, so certificate trust is settled before verification is switched on.
How we scope
- Inventory every upstream route, its scheme (http, https, grpc or grpcs) and its current verify setting.
- Record which upstream CAs are trusted today and which come from the system store or a private CA.
- Agree in writing which upstreams need client certificates and who issues and rotates them.
- Engagements are fixed-scope, with deliverables and timelines agreed before work starts.
How we build
- Test upstream certificate chains from the gateway nodes, including intermediates and hostname matching.
- Set trusted CA bundles and the system store default per environment, then enable verification one upstream group at a time.
- Configure client certificate and key pairs for upstreams that require mTLS, stored outside plain configuration.
- Run failure tests for expired, self-signed and wrong-hostname certificates to confirm the gateway rejects them as intended.
How we hand over
- Provide a per-upstream register showing verify status, trust source, certificate owner and expiry date.
- Document the rollout order, rollback steps and the upgrade checks run before moving to a newer APISIX release.
- Walk your platform and security teams through the runbook and the troubleshooting steps for handshake failures.
- Deliver configuration and test evidence in a form your auditors and internal reviewers can read.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers after handover.
- Retainers can cover certificate renewal checks, review of new upstreams and re-testing after APISIX upgrades.
- Confirm with your team which certificate authorities and renewal processes apply before a retainer is scoped.
