Encrypting secrets in etcd by Apache APISIX: what it protects and when to use it
If your APISIX gateway stores plugin credentials, consumer keys or certificate private keys in etcd, from 3.10 APISIX encrypts sensitive fields before writing them to etcd, and secret references can keep the values out of etcd altogether. The feature is aimed at one threat: someone who can read the configuration store without being a gateway operator, such as a reader of an etcd snapshot, a backup archive or a shared etcd cluster that other teams can query. Those copies hold the configuration, and without encryption they would also hold the credentials in readable form. The feature does not protect against everything that touches the same data. A compromised gateway node can read decrypted values in memory, because the gateway must use them to call upstream services or validate clients. Anyone with Admin API access can read or change configuration through the API, and the encryption layer does not change what that role may see. Protecting those paths depends on the Admin API and etcd hardening controls covered elsewhere on this page, plus network isolation of the gateway nodes.
Before you rely on the default, confirm the APISIX version you run and check that the sensitive fields your plugins use are covered. Ask your implementation partner or the APISIX changelog which fields are encrypted in your version, and whether existing records written before an upgrade are re-encrypted. Decide early whether values should live in etcd at all. If your security team requires that credentials never reach the configuration store, plan secret references to an external manager from the start instead of treating encryption as the end point. Record the decision with your data classification so auditors can see which values are protected by encryption and which never leave the secret manager.
Choosing between keyring encryption, environment references and external secret managers
This table compares the three ways to keep plugin credentials out of plain text in etcd: keyring encryption of stored fields, environment variable references, and Vault or cloud secret manager references. Read each row against your own operating model, since the answer depends on who runs the dependency and how often secrets must rotate.
| Approach | Where the secret value lives | What the gateway needs at runtime | Main trade-off |
|---|---|---|---|
| Keyring encryption in etcd | Encrypted in etcd using the keyring and key settings in the APISIX configuration | The encryption key in its config file, available to every gateway node that reads the stored data | Simplest to run, but the key sits beside the data, so protecting the config files and key material becomes your job |
| Environment variable reference ($env://) | Outside etcd, in the process environment of each gateway node, with only a reference stored in the route or consumer | Each node must have the variable set at startup or when the environment is loaded | Keeps values out of the store, but rotation means changing node environments and restarting or reloading consistently |
| Vault or cloud secret manager reference ($secret://) | In Vault KV version 1, AWS Secrets Manager or GCP Secret Manager, with only a reference in APISIX | Network reach to the provider and credentials or identity that allow the gateway to read the secret | Central audit and rotation, but adds a runtime dependency; a provider outage or access change can stop credential lookups |
| Who operates the dependency | Your platform team operates the gateway and its key material | Gateway team plus whoever owns the host environment or deployment pipeline | Secret manager operations, access policies and backups sit with a separate team, so confirm ownership before choosing |
| Vault edition and support | Vault Community Edition, Vault Enterprise or HCP Vault Dedicated, depending on how you run it | A supported Vault edition and a KV version 1 engine for the APISIX integration | Check licensing and hosting terms with the vendor before standardizing on Vault, since terms changed in 2023 |
| Covered fields | Sensitive plugin fields are encrypted by default from APISIX 3.10.0 | Confirm which fields your plugins store and whether each one accepts a reference | Not every field accepts a reference, so plan the secret pattern per plugin during design |
| Key rotation effort | Keyring rotation re-encrypts stored data under the new key, so plan a maintenance window | Rotation of the referenced value happens at the source, then the gateway picks up the new value | Rotating at the source is usually lighter, but verify the reload behavior on your APISIX version |
Setting up keyring encryption and secret references
This sequence takes an Apache APISIX deployment from a version check to a verified state in which sensitive plugin fields are encrypted in etcd and at least one credential is loaded from an external secret store at runtime. Each step ends with a check you can repeat after future changes.
Confirm the APISIX version and deployment mode
The platform owner confirms which APISIX release is running in each environment and whether it uses traditional, decoupled or standalone mode. Sensitive field encryption is on by default from 3.10, and the unified keyring settings arrived in 3.9, so a release earlier than 3.9 needs an upgrade plan before this work starts. Standalone deployments keep configuration in a file rather than etcd, so confirm where the stored data actually lives. The step is done when each environment has a recorded version, a deployment mode and a named owner for its etcd cluster or configuration files.
Set the keyring in the configuration file
The platform engineer adds the keyring setting to the gateway configuration file, which is typically conf/config.yaml in the APISIX installation, and generates key material from a source your security team approves. Keep the keyring value out of version control and out of shared ticket text. Confirm the exact keys, their locations and the accepted value formats in the official secret and deployment documentation for your version, because the key names have changed across releases. The step is done when the configuration validates, the gateway starts cleanly in a non-production environment and the key is stored in your secret management process.
Verify sensitive fields are encrypted in etcd
The platform engineer creates a test consumer or plugin configuration that contains a sensitive field, such as a key-auth key, and then reads the raw value directly from etcd with a read-only client. The stored value should not match the plaintext the test client sent. Repeat the check on a plugin field you know is covered by the default encryption in your version, and record which fields you tested. If a field is stored in plaintext, do not treat the encryption as complete for that plugin. The step is done when the test record shows encrypted storage for each field class in scope and the test objects are deleted afterward.
Choose one secret reference to configure
The security architect and platform owner pick a single credential to move first, usually one that is already stored in a plugin configuration. Record which secret backend holds it: an environment variable, HashiCorp Vault KV version 1, AWS Secrets Manager or GCP Secret Manager, as listed in the APISIX secret documentation. Confirm the reference syntax for the chosen backend and plugin in the official docs, and confirm the access policy the gateway identity needs. The step is done when the owner has a written decision, the source secret exists and the gateway identity has read access limited to that secret path.
Configure the secret reference end to end
The platform engineer replaces the literal value in the chosen plugin or consumer configuration with the secret reference and applies the change through the Admin API or configuration-as-code process used in your environment. The engineer then sends a test request that depends on that credential and confirms the response matches the expected result. Check the gateway logs for resolution errors, and confirm that the Admin API response shows the reference, not the resolved value. The step is done when a request that needs the credential succeeds and the stored configuration contains only the reference.
Check the gateway resolves the reference after restart
The operations engineer restarts one gateway node in a non-production environment, or runs the equivalent controlled restart, and repeats the test request. This confirms the node reloads the keyring, reaches the secret backend and resolves the reference without manual intervention. Also test a failure case, such as a temporarily revoked token, to confirm that the failure is visible in logs and alerts instead of silently falling back to an empty value. The step is done when the restarted node serves the request, the failure case produces a clear error and both results are recorded in the change record.
Document the rotation and recovery procedure
The platform owner writes down how the keyring and the secret backend credentials will be rotated, who approves each rotation and how a lost or corrupted keyring would be recovered from backups. Confirm the rotation behavior for your APISIX version in the official documentation, since re-encryption of existing data after a key change is a version-specific detail. Assign owners for the secret backend and for the gateway, and note which checks from this sequence must be rerun after each upgrade. The step is done when the runbook has been reviewed by security and operations and stored where on-call engineers can find it.
Which plugin fields are encrypted and which are not
Encryption coverage in etcd depends on your APISIX version and on the plugin that holds the credential. APISIX 3.10.0 enabled encryption of sensitive fields by default and extended it to more fields, so plugin secrets and similar credentials are encrypted before they are written to etcd. Key-auth keys are a specific case: according to the key-auth plugin documentation, they are AES-encrypted in etcd when data encryption is enabled. Before you rely on that, confirm in the APISIX changelog which field sets were added in each release you run, and check the documentation for each plugin you use, since a field covered in one plugin may not be covered in another.
Start with an inventory. List every plugin configuration, SSL object and consumer authentication configuration in your gateway, and record the credential fields in each one, such as client secrets, API keys, HMAC secrets, LDAP bind passwords and private keys. For each field, mark whether it is covered by encryption on your version, whether it can instead be replaced by a secret reference, and who owns the value. Then test the result on a non-production cluster: create one object of each credential type, read it back directly from etcd with your standard tooling, and confirm that the stored value is not the plaintext you submitted. An encrypted field that you cannot find in the inventory is the most common gap. Repeat the check after every upgrade, because a release can extend coverage or change how existing values are handled.
Rotating the keyring and the credentials it protects
Rotating the keyring is not one step. It touches the encryption keys in the configuration store, the values already written under those keys, the credentials those values hold, and the secret manager credentials the gateway uses to read them. Plan and test all four together in staging, because skipping any one leaves a gap that looks closed from the gateway's side.
Plan the keyring rotation
Confirm with your APISIX version's documentation how the keyring and its salt are configured and how a new key is introduced alongside the old one. Decide who approves the change, which gateway nodes must be updated, and in what order, since nodes that do not share the same keyring view can fail to decrypt stored fields. Write down the rollback path before the first change in staging. Confirm the exact procedure for your version with the APISIX release notes before you start.
Re-encrypt existing stored values
Changing the active key does not automatically rewrite values already in etcd. Plan a deliberate pass over plugin configurations, consumer credentials and SSL objects so that older entries are written again under the current key. Without that pass, values stay readable only while the old key is retained, which defeats the purpose of rotation. In staging, read back a sample of each object type through the Admin API after the pass and confirm the gateway still routes traffic.
Rotate the upstream credentials
Rotating the keyring protects nothing if the credential it wraps remains valid. Each API key, client secret or database password the gateway holds should be rotated in the upstream system, then updated in the gateway through the secret reference. Plan an overlap window in which old and new credentials both work, so clients with cached values are not cut off. Test the overlap and the cut-off in staging with a real client before production.
Rotate secret manager credentials
The gateway reads Vault or cloud secret manager values with its own identity, typically a token, AppRole or cloud role. Rotate that identity on its own schedule and confirm that the new credential is loaded before the old one is revoked. If the gateway loses access to the secret manager, routes that depend on secret references can fail at reload time. Rehearse a revocation of the old identity in staging and verify that a configuration reload still succeeds.
Running Vault as the secret source: edition, license and operating model
Before you design the secret path, confirm which Vault KV engine version you will use. APISIX can reference secrets stored in HashiCorp Vault through its KV version 1 interface, so a KV version 2 mount will not work as the source without a different arrangement. Confirm this against the APISIX secret documentation for your release, then check that the path format your key-auth and plugin configurations expect matches the mount you create. Agree the KV version, mount path and naming convention with the team that operates Vault before any consumer credential is migrated into a $secret:// reference.
Choose the Vault edition and operating model as a commercial and operational decision, not only a technical one. Vault Community Edition is available as binaries or source. Vault Enterprise adds features such as replication, HSM support, namespaces and FIPS compliance, and each requires a license applied to the cluster. HCP Vault Dedicated is a managed service that runs the same binary as self-managed Vault Enterprise. Ask for written pricing from the vendor, since the public pricing pages direct self-managed Enterprise buyers to sales. Ask your legal team to review two facts: HashiCorp announced on August 10, 2023 that future releases would move from MPL-2.0 to the Business Source License 1.1, and IBM completed its acquisition of HashiCorp on February 27, 2025. Confirm the license terms that apply to the exact Vault version you plan to run.
If the license terms do not fit, OpenBao is an MPL-2.0 community fork managed under the Linux Foundation's OpenSSF. Its migration guide was tested only against Vault 1.14.1 Community Edition, not Vault Enterprise, so treat any move as a separate validation project. Whichever backend you choose, confirm APISIX compatibility for that specific product and version with the APISIX documentation and your own test cluster before production use. Record the backend, edition, license position and KV version in the design document so that the later rotation and recovery steps refer to one agreed set of facts.
Security and operational risks to control around etcd secrets
Encryption in etcd only protects the stored value. The credentials still move through snapshots, admin calls, gateway memory, key material and secret manager identities, and each of those paths needs its own control. The cards below name the control to put in place for each path and the check to run before go-live.
Etcd snapshot and backup handling
Snapshots of etcd contain the stored configuration, including any values that APISIX encrypted, so a copy of the snapshot is a copy of the credential store. Restrict who can read backup buckets and volumes, encrypt backups at rest under a key separate from the APISIX keyring, and set a retention period your security policy approves. Test a restore into an isolated cluster and confirm that the restored gateways can still decrypt their fields with the keys you hold.
Admin API read and write access
Anyone who can call the Admin API can create or change routes and consumers, and that access exposes or replaces credentials that the gateway uses. Give each operator or pipeline its own admin key, restrict the Admin API to a private network, and log every write. Confirm that read responses to the Admin API do not return plaintext values for fields you expect to be encrypted, and test this with an account that has read access only.
Decrypted values in gateway memory
To forward a credential, the gateway must decrypt it and hold it in process memory, so a compromised gateway node can expose every secret it has loaded. Limit shell and debug access on gateway hosts, run APISIX under a non-root account, and keep gateway nodes in a separate trust zone from the etcd cluster. Plan rotation for any credential a compromised node could have read, and treat the node as untrusted until it is rebuilt.
Keyring storage and version control
The keyring and salt settings are what turn stored ciphertext back into plaintext, so storing them in etcd, in a repository or in a shared configuration file defeats the encryption. Load the key material from a file or environment source that only the gateway process can read, or from a secret manager, and keep it out of Git history, container images and CI logs. Record who holds a copy of the key and where it is backed up.
Audit logs for secret resolution
When a secret reference such as an environment variable, Vault path or cloud secret name is resolved, the event is usually invisible unless you log it, which makes misuse hard to detect. Capture the secret manager's own access logs for the gateway identity, and alert on reads from unexpected hosts or at unexpected times. Confirm with your secret manager whether it records each read, and check that log retention meets your audit requirements.
Least-privilege Vault and cloud identity
The gateway's identity in Vault or in a cloud secret manager should only read the paths and secrets that its routes use, and nothing else. Scope the policy to specific secret paths, separate read rights for each environment, and avoid a shared token across gateway clusters. HashiCorp Vault documents KV engine version 1 for APISIX references, so confirm the engine version and the policy syntax against the current Vault documentation before you write the policy.
Questions teams ask before encrypting APISIX secrets
What does it cost to encrypt APISIX secrets in etcd?
The cost comes from the work, not from a product license, because APISIX is open source under the Apache License 2.0. Cost is driven by how many plugins and consumer configurations hold credentials, how many secrets move to an external manager such as Vault, AWS Secrets Manager or GCP Secret Manager, how many environments need the same change, and how much testing is required before a production rollout. Any secret manager you adopt may carry its own license or service fees, which you should confirm separately. Ask a partner for a written scope that lists each plugin, each secret source, each environment and the test evidence you will receive, so the quote can be compared line by line with an internal estimate.
How long does encrypting APISIX secrets take?
The duration depends on your environment, so this page does not state one. The main inputs are the number of gateway clusters, the inventory of routes, consumers and certificates that store credentials, the state of your existing secret stores, and how many change windows your organization allows. Teams with a clean inventory and an existing secret manager usually move faster than teams that must first discover where credentials live. Ask for a schedule in the written scope that ties each phase to a named dependency, such as access to etcd or a Vault mount, rather than a single end date.
What does our team need to provide before the work starts?
Plan to provide access and decisions, not only people. You will need administrative access to the gateway configuration and to the etcd cluster used by your APISIX nodes, a staging environment that mirrors production, and an owner for each secret manager that will hold credentials. If you use HashiCorp Vault, you will need a KV mount and a policy that lets the gateway read the required paths. If you use AWS or GCP secret services, you will need accounts, roles and network paths that the gateway nodes can reach. You also need agreed change windows, a rollback plan and a named approver for production changes.
Can our internal platform team encrypt APISIX secrets without a partner?
Often yes, if the team already runs APISIX and etcd and has experience with at least one secret manager. The APISIX documentation covers secret references for environment variables, Vault KV version 1, AWS Secrets Manager and GCP Secret Manager, and the sensitive field encryption behavior changed in 3.10.0, so the first task is confirming which version you run and how your plugin configurations look today. Internal teams often underestimate the inventory and the testing step, particularly for consumer credentials and certificates. A partner is most useful when you need an independent inventory, a documented rollout with rollback, or a second pair of hands during change windows.
Does encrypting secrets in etcd replace Admin API hardening?
No. Encryption of sensitive fields protects stored credentials, but it does not stop someone who can call the Admin API from changing routes, plugins or secret references. The Admin API is reached through its own port and admin key, and its network exposure, key management and IP allow lists need separate controls. Treat the two as complementary: encryption reduces the damage from reading the configuration store, and Admin API hardening reduces who can change the configuration in the first place. Also protect etcd access with mutual TLS and private networking, since anyone with etcd access sits outside the gateway's own controls.
Who keeps the encryption key and how often should it rotate?
The keyring and key encryption settings belong to your platform, and their custody should be assigned to a named owner who can explain where the key material lives and how it is backed up. Rotation procedures need to be tested in staging before production, because a key change that is not planned can make stored fields unreadable. Confirm with your security team how often rotation is required under your own policy, and ask any partner to document the exact rotation steps for your APISIX version, including what happens to existing secrets and how the old key is retired.
What should the written scope include for a secrets encryption engagement?
A useful scope names the APISIX version and deployment mode, the plugins and consumer types in scope, the secret sources to be used, the etcd and Admin API controls to be checked, the test cases, the rollback procedure and the handover documentation. It should also state which environments are covered and what remains with your team, such as Vault licensing, cloud accounts or audit evidence. Ask for each deliverable to be listed with an acceptance condition, so you can verify what was delivered without relying on a verbal summary.
How DeployOpen scopes and delivers secret protection for APISIX
DeployOpen plans, implements and documents sensitive field encryption and secret referencing for your APISIX deployment inside your own infrastructure, whether that is your cloud accounts, data centers or restricted networks. The work usually runs as a fixed-scope engagement with scope, deliverables and timelines agreed in writing before any change is made to a production gateway. The written scope should name the exact plugins, consumers and certificates in use, the etcd cluster and its backups, the keyring source, the secret manager that will hold credentials, and the rotation procedure you will own afterward. A quote follows from that scope, so the more precisely your inventory of stored credentials is defined up front, the more predictable the engagement will be.
Ongoing needs such as upgrades to a new APISIX release, periodic key rotation rehearsals and re-verification after plugin changes can be covered by a retainer, which is a separate written agreement from the initial build. DeployOpen also runs security testing that can verify the result from the outside and inside: whether credentials appear in plaintext in etcd snapshots, whether the Admin API exposes secret values, and whether secret references resolve only for the intended gateway nodes. That testing produces findings your team can act on, and it is a separate scope you can add or leave out. Ask for the test method and the evidence format in the proposal.
DeployOpen prepares systems, configuration records and evidence that your team and your assessors can review for compliance assessments, but it does not issue audit reports, attestations or certifications. Those decisions rest with the independent auditor or certification body you engage. Before work starts, confirm with your assessor which evidence they expect, such as configuration exports, key management records and change logs, so the deliverables match what they will request. Your own security and operations staff remain responsible for running the gateway after handover, and the handover document should say who holds the keyring and who approves rotations.
