Why securing etcd and the Admin API with mTLS in Apache APISIX comes first
The Admin API and etcd decide what every gateway node serves, so they are the first controls to lock down. In traditional and decoupled deployments, an operator changes routes, consumers, plugins and certificates through the Admin API on port 9180, and APISIX stores that state in etcd, which listens on port 2379. Every data plane node reads from that store. A client who can write to the Admin API or reach etcd can change traffic for the whole fleet, often without touching a single gateway host. This control path is therefore a higher-value target than any one route, and it should be protected before client traffic on port 9080 is tuned.
The control path has three parts beyond the Admin API. The Control API on port 9092 exposes internal state and must never be reachable from outside the cluster. Metrics on port 9091 leak operational detail and should stay on monitoring networks. Port 2379 must be reachable only by APISIX nodes and etcd peers. APISIX also supports three deployment modes: traditional (data and control plane together), decoupled (separate data and control planes sharing one etcd cluster) and standalone, which has no etcd and loads configuration from conf/apisix.yaml or conf/apisix.json. Standalone mode removes the etcd exposure entirely but moves the risk to whoever can write that file or call the API, so the file's permissions and delivery pipeline become the control path.
mTLS is needed wherever the control path crosses a network boundary the team does not fully own, which includes most multi-node, multi-site and Kubernetes setups. Before scoping the work, confirm which of the three modes you run, which ports your firewalls and security groups currently allow, and whether any host outside the platform team can reach 9180 or 2379. Confirm the exact configuration keys for your APISIX version against the mutual TLS documentation and the installation guide, since port guidance and TLS settings are version-specific. The rest of this page covers certificates for the Admin API, etcd client certificates and trusted CAs, and binding etcd to private interfaces, in that order of dependency.
Which mTLS links to configure and what each one protects
The Admin API on port 9180 is the endpoint that creates, changes and deletes routes, upstreams, consumers and plugins. Configuring it with mTLS means the server only accepts connections from operators or automation that present a client certificate signed by a CA you trust, in addition to the admin key. APISIX documents this through the https_admin setting with admin_api_mtls certificates, which should be enabled and paired with a dedicated CA for operators rather than a general enterprise CA.
- What it authenticates: the calling operator or pipeline, at the TLS layer, before any request reaches the Admin API handler.
- What a compromise exposes: a stolen operator certificate and its private key can rewrite routing and plugin policy on every gateway node that reads the same configuration, including authentication plugins.
- What it does not cover: the admin key itself still exists, so mTLS is a second factor, not a replacement for key management.
- Network placement: mTLS does not make a public Admin API safe. Port 9180 should still be reachable only from operator networks.
- Certificate lifecycle: confirm issuance, rotation and revocation for operator and CI certificates, because revoked certificates must be rejected by the gateway.
Before you rely on this link, confirm the exact https_admin and admin_api_mtls keys for your APISIX version in the documentation and test that a request without a client certificate is refused.
etcd holds the configuration that every traditional or decoupled gateway node loads. The deployment.etcd.tls settings control how APISIX authenticates to etcd and verifies the etcd server certificate. This link protects the path between gateway nodes and the store, and it matters most when etcd runs on a separate network segment or when several clusters share infrastructure.
- What it authenticates: each APISIX node to etcd, and etcd to the node through server certificate verification against a trusted CA.
- What a compromise exposes: an attacker who can read or write etcd can change any route or secret the gateways trust, which is broader than any single Admin API session.
- Client identity: each gateway node or control plane should have its own certificate so access can be revoked per node.
- Binding and firewall rules: the etcd client port (2379 in the installation guide) should accept traffic only from gateway and control plane hosts, not from client networks.
- Trust anchors: pin the etcd CA in APISIX rather than trusting the system store, so a compromised public CA cannot impersonate etcd.
Before you rely on this link, confirm the etcd server version and TLS settings your cluster supports, and test the failure behavior when a certificate expires.
Client-facing mTLS is configured per SSL object with client.ca and client.depth, and applies only to HTTPS listeners. It authenticates the API consumers or partner systems that call the gateway, and it is a data-plane control. This link is covered in depth by the sibling topic on managing TLS certificates and SNI, so this tab only explains its place in the trust model.
- What it authenticates: the calling client, based on a certificate chained to the CA named in
client.ca. - Depth control:
client.depthlimits how far up the certificate chain APISIX will walk during verification. - What a compromise exposes: a leaked client certificate lets its holder call the APIs behind that SSL object, but it does not grant control over gateway configuration.
- Interaction with consumers: client certificates can authenticate a request, yet authorization still depends on your route and consumer design.
- Scope: the setting applies to the SSL object, so each domain or partner group may need its own CA.
Before you rely on this link, check which listeners and SSL objects your clients use and confirm that the client CA list matches the partners you have agreed to onboard.
Upstream mTLS uses tls.client_cert and tls.client_key so that APISIX presents its own certificate when it calls backend services. It protects the hop from gateway to service and proves to the upstream that the request came from an authorized gateway. This link is covered by the sibling topic on upstream TLS verification and mTLS, so this tab focuses on how it differs from the control path.
- What it authenticates: the gateway, as a client, to each upstream service.
- What a compromise exposes: the gateway's upstream client key lets an attacker impersonate APISIX to backend services, which can bypass checks that trust the gateway.
- Key storage: the private key should be protected like other gateway secrets, and the sibling topic on encrypting secrets in etcd covers how sensitive fields are stored.
- Separation from the control path: a leaked upstream key does not expose etcd or the Admin API, but it does weaken the trust boundary between gateway and services.
- Verification: upstream certificate verification and system CA trust may change across versions, so confirm behavior after each upgrade.
Before you rely on this link, list every upstream that requires a client certificate and confirm that each service trusts the gateway's issuing CA.
Configuration options for the Admin API and etcd links
This table compares the main choices for protecting the Admin API and the etcd links that every APISIX node trusts. Read each row against your topology: the right answer usually depends on whether operators reach the Admin API from a fixed network and how many environments share a certificate authority. The main takeaway is that network restriction alone is rarely enough for the Admin API, and etcd should almost always use client certificates.
| Option | When it fits | Trade-off |
|---|---|---|
| Admin API mTLS | Operators or automation reach port 9180 from changing networks, or several teams manage routes | Every client needs an issued certificate and a rotation process; a lost laptop or pipeline credential must be revoked quickly |
| Network-only restriction on 9180 | A small, fixed set of administration hosts sits on a private management network | Any process on an allowed host can change configuration; this leaves no per-operator identity in the transport layer |
| etcd TLS with client certificates | Any multi-node deployment where etcd holds routes, consumers and secrets that gateways trust | Certificates must be issued and renewed for every gateway and etcd member, and each rotation needs a planned restart or reload |
| etcd server-only TLS | A short-lived lab or test environment where the etcd network is already isolated | Gateways cannot prove their identity to etcd, so a rogue client on the path could read or write configuration |
| Private CA per environment | Production, staging and development are separate trust domains with separate operators | More CAs to protect, back up and rotate; cross-environment tooling needs explicit trust configuration |
| Shared CA across environments | A small organization with one team and one certificate lifecycle process | A compromised key or misconfigured trust store can affect every environment at once |
| Bind 9180 and 9092 to private interfaces | Admin and Control APIs should only be reached from management networks or the control plane | Operators must reach private addresses through a bastion or VPN, which adds access steps to incident response |
| Bind 9091 and 2379 to private interfaces | Metrics scrapers and etcd peers run on internal networks that are separated from client traffic | Monitoring and cluster membership must be routed explicitly; a missed scraper rule can leave metrics blind during an incident |
Setting up mTLS for the Admin API and etcd in order
This sequence moves the APISIX control path from open access to mutually authenticated connections, so that only certificate holders can change configuration and only trusted gateway nodes can read the etcd store they depend on. Each step ends with a check you can run before moving on.
Create a private CA and separate certificates
Your security or platform team creates a private certificate authority under its own control, then issues distinct server certificates for the Admin API and etcd, and distinct client certificates for each gateway node and each operator or automation identity. Keep the CA key offline or in a secrets manager with restricted access, and record which certificate belongs to which host or person. Done means every certificate chains to the CA, the server certificates carry the hostnames or addresses clients will use, and a test handshake against the CA succeeds. Rollback is simple at this stage: discard the issued material, since nothing in APISIX has changed yet.
Enable mTLS on the Admin API
Using the admin_api_mtls settings in the https_admin configuration, point the Admin API at its server certificate and key, and set the trusted CA used to verify client certificates. Restart or reload the gateway according to your deployment, then confirm the Admin API listens on its HTTPS port rather than plain HTTP. Check that a request with no client certificate is rejected at the TLS layer, and that a request presenting a certificate issued by your CA reaches the API and returns the expected response. Keep a copy of the previous configuration so you can restore plain access during a maintenance window if an operator is locked out. Confirm the exact configuration keys against the mTLS documentation for your APISIX version.
Configure etcd client TLS on each gateway node
On every gateway node, set deployment.etcd.tls with the client certificate, client key and trusted CA that etcd expects, and confirm the etcd endpoints in the configuration use the TLS scheme. Roll the change one node at a time so that a misconfigured node can be isolated without taking the whole gateway fleet down. Verify each node starts, reads its configuration, and shows healthy routes in the Admin API. A node that cannot authenticate to etcd should fail to start or report errors in its logs, which is the signal you want. Record the certificate fingerprints deployed to each node so the inventory matches what is running.
Enable TLS and client authentication on the etcd cluster
Configure each etcd member to serve client traffic over TLS, require client certificates signed by your CA, and use peer certificates for member-to-member traffic. Restart members in a rolling order that keeps quorum, checking cluster health after each restart. Verify that an etcd client connecting without a certificate is refused, and that a client using a gateway node certificate reads the APISIX keyspace successfully. The etcd documentation for your version describes the relevant flags and the order of operations. Do not skip the peer certificate step, because an unauthenticated peer channel undermines the client protection you just added.
Bind control ports to private networks
Restrict the Admin API on port 9180, the Control API on port 9092, metrics on port 9091 and etcd on port 2379 to private interfaces or operator networks. The installation guide lists these ports and advises that the Control API must never be exposed publicly, and that the Admin API should be limited to operator networks. Apply the binding in the APISIX configuration and in host firewall or security group rules, then confirm with a port scan from an untrusted network that none of these ports answers. Also confirm that client traffic on port 9080 is the only port reachable from outside, if that matches your design.
Verify each link with rejected and accepted connections
Test every link in the control path twice: once with no credentials or with a certificate from an untrusted CA, where the connection must be refused, and once with a valid certificate from your CA, where it must succeed. Cover the Admin API, the etcd client connection from each gateway node, the etcd peer and client endpoints, and the port bindings from a host outside the private network. Keep the commands and results in your change record so an auditor or future engineer can repeat them. Any link that accepts an untrusted connection is not finished, and the sequence should stop there until it is fixed.
Plan certificate rotation and rollback before go-live
Before you call the sequence complete, document who owns certificate issuance, how long each certificate is valid under your CA policy, and how you will renew the Admin API, etcd and gateway node certificates without a full outage. Write a rollback procedure that restores the previous configuration file for each component in reverse order, with the etcd change last to be undone. Confirm that the renewal procedure has been run once in a non-production environment. Ask your APISIX and etcd vendors or documentation which certificate reload behaviors apply to your exact versions before relying on them in production.
Security and operational risks of the configuration store
The Admin API, etcd and the certificates between them decide what every gateway node trusts. The cards below cover the risks that most often turn a working mTLS setup into an exposure, and the control that addresses each one.
Certificate expiry and rotation
Every mTLS link in the control path uses a certificate that will expire. When the Admin API, etcd client or etcd server certificate lapses, gateway nodes can lose their configuration source or stop accepting admin changes at the same moment. Inventory each certificate, its issuer, its expiry date and the service that loads it, and rehearse rotation on a non-production cluster before the first real renewal.
CA key custody
The certificate authority that signs Admin API and etcd client certificates decides who can join the control path. If its private key sits on a shared server or in a repository, anyone holding it can mint a client certificate the gateway will accept. Keep the CA key offline or in a managed key store, restrict signing to a named process, and record every issuance.
Who holds admin credentials
Admin API access is granted by certificate and by admin key, so both need an owner. Shared admin keys or a single client certificate used by several engineers make it impossible to tell who changed a route. Issue one client certificate per operator or pipeline identity, store admin keys in a secrets manager, and revoke access when a person changes role.
etcd exposure on port 2379
etcd listens on 2379 for client connections, and anyone who reaches it can read or rewrite the configuration every gateway node trusts. Bind etcd to private interfaces, allow only gateway and control plane hosts through the firewall, and require client certificate authentication. Confirm the bind address and firewall rules from the running hosts, not from the intended configuration.
Backup encryption of etcd snapshots
etcd snapshots contain the full gateway configuration, including routes, consumers and any credentials not yet moved to a secret reference. A plaintext snapshot in a backup bucket is a copy of the control path. Encrypt snapshots at rest with keys held apart from the backup store, restrict who can restore them, and test a restore into an isolated cluster on a schedule.
Audit logging of admin changes
Without a record of admin changes, a bad route or a removed authentication plugin can go unnoticed until users report it. Log every Admin API write with the calling identity, the time and the object changed, and ship those logs to storage the gateway operators cannot edit. Ask your team to confirm which fields your APISIX version records and where they land.
How etcd client certificates change node-to-store trust
Every APISIX node that reads its configuration from etcd is a client of that store, and the etcd client certificate is the identity each node presents when it connects. In the deployment settings, the deployment.etcd.tls block points each node at its client certificate, its private key and the CA that signed the etcd server certificate. Etcd checks that the certificate chains to a CA it trusts before it serves any keys. The practical consequence is that every gateway node, and every control plane component that writes to etcd, needs a certificate that chains to the same trusted CA. A node with a certificate from a different CA cannot read its routes, and a node that trusts the wrong CA may accept a store it should not.
Because the configuration store is what every node trusts, its protection matters more than the protection of any single route. Whoever can write to etcd can change routes, upstreams, plugins and consumer credentials across the whole fleet at once, so the client certificates and the CA that issues them should be treated as high-value control material. Confirm with your security team which systems may hold a client key for etcd, where those keys are stored, and who can issue new ones. Keep the CA used for etcd clients separate from the CA used for client-facing TLS wherever your PKI allows it.
Expiry and node replacement are the failure modes to plan for. When a client certificate expires, the affected node can lose its connection to etcd, and it will not receive configuration changes until it is renewed and reloaded. Check the existing certificate expiry dates against your renewal calendar before any upgrade or planned change window, and confirm the reload behavior in the version you run using the Apache APISIX installation guide. When you add or replace a node, issue its certificate from the same CA, test that it reads configuration in a non-production environment, and only then add it to the load balancer pool.
Limits to confirm in your environment before you rely on this
The mTLS settings for the Admin API, etcd and clients are only as reliable as the version, deployment method and network path they run on. The cards below list what to confirm with the vendor documentation and your own environment before you treat this design as settled.
Certificate field requirements per version
Certificate fields, key types, extensions and CA chain handling can change between APISIX releases, so the requirements in your docs may not match the version you run. Check the exact fields the admin_api_mtls block and the deployment.etcd.tls block expect for your release, then test a certificate issued from your own CA against a non-production node. Record the version next to the certificate profile so upgrades get re-tested.
Admin API mTLS behind a load balancer
If a load balancer or reverse proxy terminates TLS in front of port 9180, the client certificate may never reach APISIX, or the proxy may present its own identity. Confirm whether your topology passes client certificates through to the gateway and whether the Admin API should be reached directly on a private network. Test a rejected client and a valid client through the exact path operators will use.
etcd TLS across deployment methods
etcd TLS settings in a package install, a container image, a Helm chart and a standalone mode file can differ in file paths, defaults and how certificates are mounted. Standalone mode does not use etcd at all, so this card only applies to traditional and decoupled modes. Confirm the etcd client certificate, key and CA paths for each method you use, and confirm that etcd itself requires client certificates from APISIX nodes and from nothing else.
Kubernetes-managed installs
Helm charts and the APISIX Ingress Controller may generate, mount or rotate the Admin API and etcd certificates for you, or expect you to supply them through Secrets. Confirm which values the chart exposes, whether the controller connects to the Admin API with its own credentials, and how certificate renewal reaches running pods. Document who owns each certificate and its expiry date.
Port exposure and network policy
The installation guide lists 9180 for the Admin API as a port to restrict to operator networks, 9092 for the Control API as one never to expose publicly, and 2379 for etcd. Confirm that your firewalls, security groups and Kubernetes network policies match this list in your environment, including any port 9091 metrics scraping. Verify the result with a scan from outside the operator network.
Questions teams ask before securing the APISIX Admin API and etcd
Can our team secure the APISIX Admin API and etcd with mTLS on its own?
Yes, many teams can, provided they already run a certificate authority, know how their gateway nodes reach etcd, and have someone who can change the configuration on every node. The work itself is well documented: APISIX supports mTLS for the Admin API through the https_admin settings, for etcd connections through deployment.etcd.tls, and for client traffic per SSL object. The harder part is coordinating the change without breaking gateway nodes that still use plain connections. A partner is most useful when the PKI is new, when several environments must change together, or when the team cannot spare people for a planned change window.
What drives the effort and cost of this work?
The main drivers are the number of gateway nodes and etcd members, whether a CA already exists, and whether the platform runs on virtual machines or Kubernetes. Each node needs its own client certificate and trust configuration, so effort grows with the node count. Designing a new CA hierarchy, separating issuing and root roles, and agreeing on certificate profiles take more time than reusing an existing internal PKI. Kubernetes deployments add secret distribution and Helm value changes, while VM deployments usually add configuration management work. One-time costs cover design, build, testing and cut-over. Recurring costs cover certificate renewal, monitoring of expiry and any retainer for ongoing support. Ask for a quote that lists each of these as a separate line.
How long does securing the Admin API and etcd take?
Duration depends on scope, and no fixed figure applies to every environment. A narrow scope covers one Admin API endpoint and one etcd cluster in a single environment. A broader scope covers traditional and decoupled deployments across several environments, plus rollout and rollback rehearsals. Time is also driven by how quickly the client's PKI team can issue certificates and by how often change windows are available. Fixed-scope engagements agree the timeline in writing before work starts, so ask for the planned phases and the client dependencies that could move them.
What do we need to provide for the engagement?
The work needs people and access more than hardware. Plan for the following:
- PKI owners who can approve the CA design, certificate profiles and key storage.
- Network access to ports 9180 for the Admin API, 2379 for etcd and the gateway nodes that connect to them.
- Operators who can approve and host change windows in each environment.
- A list of every client that calls the Admin API, including automation and dashboards, so none of them breaks at cut-over.
Confirm early whether your change process requires separate approval for production configuration and whether etcd backups are current before any change starts.
How is certificate rotation handled after the work is done?
Rotation should be designed before the first certificate is issued. The design should record each certificate's owner, issuing CA, validity period and the location of its private key, then define how renewal reaches each APISIX node and etcd member. Rotation usually requires a reload or restart of the affected component, so the procedure should be rehearsed in a non-production environment. Teams should also decide how a compromised key is revoked and how the trust bundle changes when a CA is rotated. Ask the partner to document the renewal procedure, the alerting on upcoming expiry and the steps for a planned CA change.
Does the work cover binding etcd and the Admin API to private interfaces?
It can, and this is usually part of the same change. Binding etcd to a private interface and restricting the Admin API to operator networks reduce exposure even when mTLS is in place. The installation guide lists 9180 for the Admin API, which should be restricted to operator networks, and 2379 for etcd. Control API port 9092 should never be exposed publicly, and metrics on 9091 should be limited to monitoring systems. Confirm your network design, firewall rules and cloud security group layout before the work begins, because some clients may reach these ports through paths you have not documented.
What does a fixed-scope quote cover?
A fixed-scope quote follows from a written scope, which should name the deployment mode, environments, nodes, etcd members, certificate authorities, and the components in and out of scope. It should list the deliverables, such as the certificate design, configuration changes, test evidence, rollback steps and operator documentation, along with the client responsibilities and the timeline. Ask whether the quote includes rotation runbooks, knowledge transfer and post-change support. DeployOpen does not publish a price list, so the quote is produced from the agreed scope. Changes to scope after signing should be handled through a written change request that states its effect on cost and schedule.
Does securing these endpoints support our compliance evidence?
It can contribute evidence, but it does not establish compliance on its own. Mutual TLS, restricted admin networks and documented certificate lifecycles are the kinds of controls assessors ask about for encryption in transit and access control. DeployOpen prepares systems, configuration and evidence for compliance assessments, but independent auditors and certification bodies issue any audit report or certification. Ask your assessor which artifacts they expect, such as configuration exports, access lists and rotation records, before the work is scoped.
How DeployOpen scopes and delivers APISIX control-path hardening
DeployOpen plans, implements, hardens and documents self-hosted open-source platforms inside your own infrastructure, including the cloud accounts, data centers or air-gapped networks you already control. For APISIX control-path work, that means the Admin API certificates, etcd client certificates, trusted CAs, private interface bindings and the configuration store all gateway nodes rely on are built in your environment, not in ours. Most engagements are fixed-scope, so the written scope, deliverables and timelines are agreed before any work starts. Ask for that document to list each certificate authority, certificate, listener and etcd endpoint in scope, and name who holds each private key after handover.
Ongoing support, upgrades and continuous testing are available as retainers, which suits the parts of this work that change over time, such as certificate renewal, etcd node changes and APISIX version upgrades. Confirm with any partner how renewal ownership, key rotation and access reviews for the Admin API are divided between their team and yours, and whether the retainer covers re-verification after each upgrade. Pricing follows from the written scope and is not published, so compare proposals on the same list of components and on who performs each recurring task.
DeployOpen prepares systems, configuration, controls and evidence for compliance assessments, such as records of certificate issuance, mTLS settings and network restrictions on ports 9180 and 2379. It does not issue audit reports, attestations or certifications. Those come from independent auditors and certification bodies, so ask your assessor early which artifacts they expect and agree on the format before the build starts.
