Why Kubernetes secrets management starts with what etcd stores
Kubernetes stores Secrets unencrypted in etcd by default, so a database password, API token or certificate key sitting in a Secret is only as protected as the etcd data directory, its snapshots and its backups. Encryption at rest must be configured deliberately, through the API server's encryption configuration, before the cluster holds sensitive data. If your cluster runs in your own cloud account, data center or air-gapped network, this is your responsibility, not a default you inherit from the platform.
The topic has three parts that teams often treat as one. The first is encryption at rest, where the API server encrypts Secret objects before writing them to etcd, usually with a key held in a key management service. The second is access control on the Kubernetes API, where anyone who can get or list Secrets in a namespace can read their values regardless of how they are encrypted on disk. The third is external secret stores, such as a vault accessed through the Secrets Store CSI Driver, which keeps values out of etcd altogether. Each part answers a different risk, and none replaces the others.
Teams usually need this work when an audit or assessment asks how secrets are protected at rest, when regulated data such as cardholder or patient records may pass through workloads on the cluster, or when several teams share one cluster and their Secrets sit side by side. Ask your auditor or compliance lead which controls they expect to see evidenced, and ask your platform owner which environments hold production credentials. Those answers determine how much of the design needs to be documented before the first workload goes live.
Two details make this harder than it looks. Enabling encryption does not rewrite Secrets that already exist in etcd, so older values remain in their original form until each one is read and written again, and the rewrite must be verified and repeated for every backup taken before the change. Also, encryption does not restrict who can read a Secret through the API. Access is governed by RBAC, so a service account with broad list or watch permissions can read decrypted values even on a fully encrypted cluster. Plan the two controls together and test them separately.
Before you choose a design, confirm three things with your platform team: which Kubernetes version and distribution you run and whether its documentation covers the encryption configuration you need, where the encryption keys will be held and who can administer them, and which namespaces and service accounts can currently read Secrets. The answers set the work sequence, and they tell you whether the gap is a configuration task, an access review, or a move of values to an external store.
Which components hold, encrypt and deliver Secret values
Five components decide whether a Secret value is protected at rest, who can read it through the API, and how it reaches a workload. Each has its own configuration and failure mode, so a gap in one can expose values that the others protect. Confirm how your distribution installs and configures each one before you rely on it.
kube-apiserver encryption configuration
The kube-apiserver reads an EncryptionConfiguration file, passed in through its startup flags, that names the resources to encrypt before they are written to etcd and the providers that do the encrypting. Providers are tried in the order listed, and the first one encrypts new writes. Confirm how your distribution sets this flag, where the file sits on control plane nodes, and whether it is managed centrally or edited by hand. A misconfigured file can leave Secrets unencrypted or unreadable after a restart.
etcd datastore
etcd stores every API object, including Secret objects, and it is where encrypted values land once encryption at rest is enabled. Secrets written before encryption was turned on stay in plain text until they are rewritten, so confirm how your team re-encrypts existing data. Backups are a second copy of the same store, so confirm that snapshots are protected as carefully as the live cluster, and that access to etcd client ports is restricted to the control plane.
Key management service
The key management service plugin is the component the API server calls to protect the data encryption keys that encrypt Secret values. The key encryption key stays in the external service, so the cluster holds only protected copies of the data keys. Confirm which KMS provider your distribution supports, how the API server authenticates to it, what happens to reads and writes if the service is unreachable, and who owns key rotation and access policy.
RBAC rules for Secrets
Role-based access control decides which identities can perform get, list and watch on Secrets in each namespace. Encryption at rest does not restrict API reads, so any principal allowed to list Secrets receives the decrypted values. Review roles that use wildcard verbs or resources, cluster-wide bindings, and service accounts that can read Secrets in namespaces they do not need. Confirm these bindings against your least-privilege review before go-live.
Secrets Store CSI Driver
The Secrets Store CSI Driver mounts secret material from an external store into pods as a volume, so values can stay out of etcd entirely. A provider plugin for your store performs the retrieval, and the driver is installed as a separate component. If the sync-to-Secret option is enabled, the driver also creates a Kubernetes Secret, which brings the values back into etcd. Confirm which provider your store supports, how pods authenticate to it, and whether syncing is switched on.
Kubernetes secrets management options and their trade-offs
This table compares four ways to protect Secrets in a self-hosted cluster: leaving them unencrypted in etcd, encrypting at rest with a key management service you control, encrypting with a local key in the encryption configuration, and delivering values from an external store through the Secrets Store CSI Driver. Read each row for when the option fits, what it asks your team to operate, and what to confirm with your distribution vendor before choosing. The main takeaway is that every option moves key or secret custody somewhere, so the decision depends on who must hold that custody and how it will be rotated.
| Option | When it fits | Operational cost | Confirm with the distribution vendor |
|---|---|---|---|
| No encryption in etcd | Only as a documented, accepted risk, such as a short-lived test cluster with no sensitive data | Lowest effort now, but anyone who can read etcd data or its snapshots can read Secret values | Confirm the default storage settings and whether installation can enable encryption at rest |
| KMS provider in your own KMS | Production clusters where key custody must stay with your security team inside your key management service | Key policies, rotation schedules and access reviews move to the KMS owner, and the API server depends on KMS availability for Secret operations | Confirm the supported KMS plugin versions, how key rotation is applied and how the cluster behaves if the KMS is unreachable |
| Local key in encryption configuration | Clusters without a KMS, including some air-gapped sites, where the team accepts local key custody | The key sits in a file on control plane nodes, so node access can mean key access, and rotation is a manual procedure to rehearse | Confirm where the encryption configuration file lives, how it is handled during upgrades and backups, and the key rotation steps |
| External store via Secrets Store CSI Driver | Teams that already run an external secret store and want Secret values kept out of cluster state | Adds a driver and provider to maintain through upgrades, changes how applications receive secrets, and makes the external store a critical dependency | Confirm supported Kubernetes versions for the driver, provider compatibility, and whether Kubernetes Secret objects are still created in etcd |
How to configure encryption at rest for Secrets, step by step
This sequence turns on encryption at rest for Kubernetes Secrets in the order that avoids breaking workloads. It produces a cluster where every API server encrypts Secrets before writing them to etcd, existing Secrets are rewritten under that encryption, and each closing check gives you evidence you can keep for audit.
Inventory Secrets and the workloads that read them
The cluster administrator, with input from application owners, lists every Secret in every namespace, including Helm release Secrets, registry pull Secrets, TLS certificates and service account token Secrets. For each one, record which Pods, Deployments, operators and external controllers mount or read it, and review RBAC bindings that grant get, list or watch on Secrets. This list shows which workloads need a restart after the change and which Secrets step six must rewrite. The step is done when each namespace owner has confirmed the list and the set of readers for their Secrets.
Choose the key source for encryption
Decide where the encryption key lives before you change any node. A local key in the encryption configuration file is simple to set up, but the key sits on the same control plane nodes as the data it protects, so anyone with root access to a control plane node can read both. A key management service reached through a KMS provider plugin keeps key material outside the cluster and supports central rotation, but it becomes a dependency that every API server must reach. Confirm with your distribution and KMS vendor which provider and API versions they support. The step is done when the security owner has approved the choice and a key custodian is named.
Prepare the encryption configuration on each control plane node
Write an EncryptionConfiguration file using the apiserver.config.k8s.io API group. List the resources to encrypt, starting with Secrets, and the provider and key for each. Keep the identity provider as the last entry so that existing unencrypted data stays readable during the transition. Place the identical file at the same path on every control plane node with owner-only permissions, and keep the key material out of version control. The step is done when the file parses on every node and checksums match across all control plane nodes.
Point kube-apiserver at the configuration and restart in rolling order
Add the encryption provider configuration flag to the kube-apiserver static Pod manifest on each control plane node, and mount the configuration file and any KMS plugin socket into the Pod. Change one control plane node at a time, and wait for its API server to report healthy before you move to the next, so the load balancer always has a healthy backend. Check the health endpoint and list nodes after each restart. The step is done when every kube-apiserver runs with the flag and answers on every node.
Confirm a new Secret is encrypted in etcd
Create a test Secret in a scratch namespace with a known value. Then read its raw record from etcd with etcdctl, using the etcd client certificates, and query the key under the registry path for Secrets in that namespace. The stored value should begin with the prefix that matches your configured provider, and the known plain text value should not appear in the output. Delete the test Secret afterward. The step is done when the raw record carries the provider prefix and the plain text is not readable.
Rewrite existing Secrets so they become encrypted
Existing Secrets stay unencrypted in etcd until they are written again. Take an etcd snapshot first, then rewrite Secrets namespace by namespace with a command that reads each Secret and replaces it without changing its content. Workloads that consume Secrets as environment variables need a rollout restart to pick up the new values, so confirm the restart plan with the application owners from step one. The step is done when etcd shows the encrypted prefix for a sample of older Secrets across namespaces, and a full sweep finds no unencrypted records.
Confirm the key is reachable from every API server
Verify that each kube-apiserver can decrypt, not only encrypt, by reading Secrets through each control plane node directly, bypassing the load balancer. Check API server logs for encryption or KMS errors, and for KMS providers check the plugin health status. Then simulate the loss of one control plane node and confirm that the remaining API servers still return Secret values. The step is done when every API server returns decrypted Secret values, the logs show no provider errors, and the results are recorded as evidence.
What drives secrets management effort and cost
This table lists the factors that make Kubernetes secrets work larger or smaller, how to keep each one under control, and whether its cost is paid once or recurs. Read it before scoping, because the first two drivers set the baseline and the rotation and evidence drivers set the long-term load. No prices or durations appear here, so ask for them against a written scope.
| Driver | Why it changes effort or cost | How to keep it down | One-time or recurring |
|---|---|---|---|
| Clusters and control plane nodes | Each cluster needs its own encryption configuration, key access and verification. Control plane nodes must all carry the same encryption configuration, so more nodes add rollout steps. | Standardize one encryption configuration and one rollout runbook across clusters. Apply it through the same automation to every control plane node. | One-time for setup, recurring for each cluster you add or upgrade |
| Number of Secrets and their consumers | Existing Secrets must be rewritten to be encrypted, and each consuming workload must be traced before cutover. Shared Secrets used by many Pods raise the risk of a broken rollout. | Inventory Secrets and consumers first. Retire unused Secrets. Group workloads by owning team and migrate in waves. | One-time for migration, recurring for new Secrets |
| Existing key management service | If the organization already runs a KMS, the work is integration and access policy. If not, the team must select, deploy and operate one, which adds a new system to patch and monitor. | Reuse the KMS already approved for other workloads, provided it supports the plugin model the cluster needs. Confirm support with the KMS and distribution documentation. | One-time for integration, recurring for KMS operations |
| External stores and multiple providers | Each external store needs its own authentication, access policy and failure handling. Pods that read from more than one provider add testing paths and support questions. | Pick one external store for each class of secret. Limit the number of providers and document which workloads use which. | One-time for design, recurring for provider access reviews and upgrades |
| Application changes to read mounted files or external sources | Applications that read environment variables must be changed to read mounted files, or the Secrets Store CSI Driver must be used with a provider. Each change needs code review, testing and a release. | Change few workloads at a time. Prefer mounted files over environment variables where the application allows it. Ask owning teams to scope their own changes. | One-time for application changes, recurring when new workloads need secrets |
| Compliance evidence requirements | Assessors typically ask for proof of encryption configuration, key access, RBAC scope and change history. Producing that evidence by hand each cycle takes staff time. | Define evidence outputs at design time, such as configuration exports, RBAC reports and audit log retention. Automate the collection where possible. DeployOpen prepares evidence; auditors and certification bodies issue opinions. | One-time for evidence design, recurring for collection and review |
| Key rotation procedures | Rotating the encryption key requires re-encrypting stored Secrets and checking that old keys can still decrypt backups. Rotation of application credentials also requires coordinated updates in consuming workloads. | Automate re-encryption of Secrets after a key change. Rehearse rotation in staging before production. Keep a written schedule that the owning team signs off. | Recurring |
| Break-glass procedures | A documented emergency path for reading or restoring Secrets when the KMS or external store is unavailable must be designed, tested and access-controlled. Each path adds review and drill effort. | Keep one break-glass path with named custodians, time-limited access and logged use. Rehearse it on a schedule rather than only after an incident. | One-time for design, recurring for drills and access reviews |
| Provider and distribution upgrades | KMS plugins, the Secrets Store CSI Driver and the Kubernetes version all change over time. Each upgrade must be checked against the encryption and mount behavior already in use. | Track the Kubernetes version lifecycle and plan upgrades in staging first. Pin tested versions of add-ons. Confirm compatibility in the vendor documentation before each upgrade. | Recurring |
How secrets handling changes by cluster model
A single control plane test cluster has one kube-apiserver, one etcd member and one place to set flags. Encryption at rest is configured by pointing the API server at an encryption configuration file on the control plane node and restarting it. Because there is only one API server, consistency is simple, but the same shortcut tends to be copied into later clusters, so the file and its key handling should be built as if it will be reused.
- Configuration location: a single file on one control plane node, referenced by the API server flag, usually set through the kubeadm configuration.
- Key custody: a local key held in the file is common in test work. Anyone with root on that node can read the key and the stored Secrets together.
- etcd backups: a snapshot taken from the one member contains Secrets in their encrypted form, so the snapshot is only protected if the key is kept elsewhere.
- Matching configuration: nothing to synchronize, but a later move to multiple control plane nodes will require the identical file on each one.
- Mismatch risk: a node rebuilt without the file can start with no encryption or fail to read existing Secrets.
Before reuse, confirm where the key lives, whether it can be rotated and whether the file is in the same configuration source that will feed future control plane nodes.
In a stacked topology, each control plane node runs its own kube-apiserver and an etcd member. The encryption configuration must be identical on every node because any API server can write Secrets to the shared etcd cluster and any other must read them back. A node with a different file can store values another node cannot decrypt, which surfaces later as failed reads rather than at the time of the change.
- Configuration location: the same encryption configuration file on every control plane node, managed by one configuration source and rolled out one node at a time.
- Key custody: the key still sits on each control plane node unless a key management service is used. Every node that holds the key is a place it can be copied from.
- etcd backups: a snapshot from any member contains the same encrypted values, so one backup covers the cluster, and it must be stored off the cluster.
- Matching configuration: verify the file checksum and the key name on each node after every change, and restart the API servers in sequence so the cluster stays available.
- Rotation: adding a new key requires rewriting existing Secrets so they are encrypted with it, a step that should be scheduled and tested.
Before go-live, confirm how the team detects a drifted node and how quickly a Secret that cannot be decrypted would be noticed.
With an external etcd cluster, the control plane nodes run the API server without an etcd member. The encryption configuration still belongs to the API servers, not to etcd, so it must match across control plane nodes even though the etcd cluster is managed separately. Secrets are encrypted before they reach etcd, which means the etcd team and the control plane team need to agree on who controls the key.
- Configuration location: the encryption configuration remains on each API server node, and etcd itself has no Kubernetes-level encryption setting.
- Key custody: separating etcd from the control plane lets the platform team hold the key while a different team operates the etcd cluster, which is a useful separation of duties to document.
- etcd backups: snapshots are taken from the external cluster and still contain encrypted Secret values. Restore procedures must also restore the matching key, or the backup is unreadable.
- Matching configuration: the same file on each API server, plus a documented process to confirm the etcd endpoints in every API server configuration.
- Failure domains: etcd quorum and API server health are tracked separately, so restore drills must cover losing either side.
Confirm who can read the key, who approves a rotation and whether the backup procedure names the key as a required restore input.
In a managed service, the provider runs the control plane and etcd, so the customer does not edit the API server or etcd configuration. Encryption at rest for Secrets is typically a provider setting, and some providers let the customer supply a key from a key management service. What the customer controls depends on the product, so the first task is to read what the provider actually exposes rather than assume the upstream steps apply.
- Configuration location: a provider-side setting or a cluster creation option, not a file on a node the customer manages.
- Key custody: the provider may use a provider-managed key by default, and a customer-managed key may be available as an option. Confirm which option applies and who can disable it.
- etcd backups: the customer usually cannot take an etcd snapshot, so encrypted values in backups are the provider's responsibility. Ask for the provider's documented backup scope and restore process.
- Matching configuration: handled by the provider across its control plane, so the check is a documented setting and an export of the cluster configuration.
- Responsibility split: Secrets encryption, Secret access through RBAC and external stores remain the customer's responsibility.
Before relying on a managed service, confirm the key options, the backup and restore scope, and the audit evidence the provider will supply on request.
In an air-gapped cluster, the key management service and any external secret store must be reachable from inside the network, because they cannot be reached over the internet. The encryption configuration and etcd backups are handled the same way as in self-managed clusters, but every dependency must be local. Keys cannot be fetched from a hosted service, so the key store, its own backups and its patching all become part of the platform.
- Configuration location: the same encryption configuration file on each control plane node, referencing a KMS plugin endpoint that resolves inside the network.
- Key custody: the key service is an internal system with its own administrators, backups and access controls. Losing it can make encrypted Secrets unreadable.
- etcd backups: snapshots and the key backup must be stored in the enclosed environment and restored together, following the approved media handling process.
- Matching configuration: every API server must point to the same KMS endpoint and key name, and changes must be made through the offline change process.
- Upgrades: key service and plugin versions must be planned alongside Kubernetes upgrades, since updates cannot be pulled on demand.
Confirm that the KMS plugin, its images and its dependencies are mirrored into the environment and that a restore rehearsal includes both etcd and the key.
How the Secrets Store CSI Driver changes where Secret values come from
With the Secrets Store CSI Driver, the authoritative copy of a secret value lives in an external store, and the cluster receives it only when a pod needs it. The driver mounts secret material from the external store into the pod as a volume through a CSI volume that the kubelet calls at pod start. The value is not read from etcd at all in that path, so etcd encryption at rest no longer protects it. Access control moves with it: who can read the value is decided by the external store's policies together with the identity the pod presents to that store, not by RBAC on Secret objects in the cluster. Platform owners should write this down as a decision, because it changes which team owns the audit trail and the revocation process.
The first trade-off is availability. A pod that mounts secrets through the driver depends on the external store being reachable when the pod starts, and on the store answering the provider's request for that specific secret. If the store, the network path to it, or the identity federation behind it fails, new pods can stay in a pending or failed state even though the cluster itself is healthy. Plan for this with the store's own high availability, network policy for the provider pods, and a documented procedure for what happens during an outage of the store. Ask the store owner for its availability commitments and recovery objectives in writing, since the cluster cannot compensate for them.
Rotation is the second trade-off, and it has two parts. First, confirm whether the driver is configured to refresh mounted content when the external value changes, and at what interval, using the driver's current documentation. Second, confirm whether the workload actually reloads. A file that changes on disk does not update an environment variable already loaded by a process, so applications that read secrets only at startup need a restart mechanism or a reload hook. Test this in staging with a deliberate rotation and record how long the old value remains valid in running pods. Treat the answer as an input to your rotation runbook rather than an assumption.
Provider availability is the third check, and it depends entirely on the reader's store. The driver relies on separately maintained provider components for each external store, and support for a given store, authentication method, or feature such as syncing to a Kubernetes Secret can vary by provider and by driver version. Do not assume support from the project name alone. Build a short matrix that lists the store, the authentication method your organization requires, the rotation behavior you need, and whether the provider supports each, then verify every cell against the driver and provider documentation for the versions you plan to run. Record the versions tested, because provider compatibility shifts over time.
The last question is whether a Kubernetes Secret is still created as a copy. Some configurations sync the mounted value into a native Secret so that workloads can consume it through environment variables or existing tooling. That copy is stored in etcd, is subject to the same RBAC and encryption at rest settings as any other Secret, and can outlive the external value unless sync and deletion behavior are configured correctly. If you need the copy, treat it as a second location to protect and audit. If you do not, confirm that the chosen configuration avoids creating it. Ask the implementation partner to show both states in the test environment so the security review is based on observed behavior.
Kubernetes secrets management access controls to set first
Encryption at rest protects Secrets in etcd, but anyone who can read Secrets through the API still gets the values in clear text. The cards below cover the RBAC and workload controls to set before go-live, each with a way to confirm it on your own cluster.
Restrict list and watch on Secrets
List and watch on Secrets return full Secret values, not just names, so these verbs carry the same exposure as get. Grant them only to the few controllers or administrators that truly need them. To confirm, run an access review for each subject that holds list or watch on the secrets resource and check that every grant is justified and namespaced.
Limit get to named Secrets
Workloads that read one Secret should be granted get on that specific resource name, not on the whole namespace. Named grants stop a compromised pod or service account from reading unrelated credentials in the same namespace. To confirm, inspect each Role binding that references secrets and verify it lists resourceNames for every workload-facing grant.
Avoid wildcard rules on secrets
A rule that uses the wildcard for resources or verbs covers Secrets silently, including Secrets created after the rule was written. Wildcards also tend to pass unnoticed in reviews of broad cluster roles. To confirm, search every Role and ClusterRole for wildcard verbs or resources, and check that none of them apply to secrets in any namespace.
Control who can create pods that mount Secrets
Creating a pod is an indirect way to read a Secret, because the pod can mount the Secret as a volume or environment variable and expose it to whoever can exec into or read logs from that pod. Restrict create on pods and on workload controllers such as Deployments and Jobs to the teams that deploy them. To confirm, list subjects with create rights on those resources in each namespace holding sensitive Secrets.
Review aggregated and custom roles
Aggregated ClusterRoles and custom roles from operators or add-ons can grant Secret access without anyone editing your own bindings, because aggregation merges rules from labeled roles. Check which roles carry aggregation labels and what rules they pull in. To confirm, list every ClusterRole with an aggregation rule and verify that none of the merged rules include secrets for unintended subjects.
Protect the encryption configuration and KMS credentials
The encryption configuration file tells the API server which provider and key to use, and KMS credentials let it wrap data encryption keys. Both live on control plane nodes, so anyone with root there can read keys or change the configuration. Restrict file permissions, limit node access, and check that the file is not world readable. To confirm, inspect ownership and mode on each control plane node and confirm access is logged.
Review service account token exposure
Service account tokens let workloads call the API, and a token with Secret read rights turns any pod compromise into a Secret compromise. Check which service accounts are bound to roles with Secret access, whether tokens are mounted into pods that do not need them, and whether the default service account is used by application workloads. To confirm, list pods that mount service account tokens and compare them against workloads that actually call the API.
How to test that Secrets are encrypted and that access fails safely
The purpose of this section is to prove, with recorded evidence, that Secrets are encrypted in etcd and that the API server refuses requests it should refuse. Reading the encryption configuration alone does not show this. Run each test first on a non-production cluster that uses the same encryption settings as production, then repeat it in production during a change window agreed in writing with the platform owner. Keep the test identities, commands, timestamps and outputs together so an auditor can trace each result to the configuration in force at the time.
Start with a canary. Create a Secret whose value is a unique random string, then read the stored object directly from etcd with etcdctl and the etcd client certificates, bypassing kubectl. The stored value should begin with the provider prefix defined in your EncryptionConfiguration and should not contain the plaintext string. Next, confirm that existing Secrets were rewritten after encryption was enabled, because encryption applies to new writes by default. Rewrite them using the method your distribution documents, then check a sample of older objects the same way. Record the prefix you observed and the namespaces you covered.
Test access from two directions. For a service account that lacks get permission on Secrets in its namespace, run kubectl auth can-i with --as and confirm the answer is no, then make a real request and confirm it returns a Forbidden error. For a restricted user, confirm they cannot list or watch Secrets across namespaces by running the same check with --all-namespaces. Review aggregated roles and group bindings as well, because a broad ClusterRoleBinding is the most common cause of a failure these tests are meant to catch. Save the audit log entries for denied requests along with the command output.
Simulate a KMS outage on the non-production cluster by making the KMS plugin unreachable, then try to create a new Secret. Confirm the API server rejects the write and that no plaintext copy appears in etcd. Then read existing Secrets and record the result. How reads behave during an outage depends on your Kubernetes version and KMS provider, including whether cached data keys allow decryption, so confirm the documented behavior with the upstream and vendor documentation before the test and write the expected result down first. Restore the plugin and confirm that writes resume.
Test recovery last. Take an etcd snapshot with etcdctl snapshot save, restore it into a scratch cluster with etcdutl, and confirm the restored cluster can read a canary Secret. This only works if the key material the cluster depends on is available to the recovery team, so test key access separately from the snapshot. Repeat the full set after every key rotation and every API server upgrade, and file each run as dated evidence tied to the configuration version it tested.
Kubernetes secrets management questions buyers ask before hiring help
How much effort does enabling Secrets encryption at rest take?
The configuration change itself is small, but the effort depends on what surrounds it. The main drivers are:
- Whether you already run a key management service (KMS) that the cluster can call, and how much work its integration needs.
- How many clusters share the same KMS and key policy.
- Whether your control plane is self-managed, so you change kube-apiserver settings directly, or managed by a provider that exposes the setting differently.
- How many existing Secrets must be rewritten, and how well your backups and change process cover that step.
Ask your distribution vendor or managed service provider which encryption options they support and how they apply them. Ask your KMS owner which key policies, rotation rules and access logs apply to the cluster.
Can our in-house team turn on Secrets encryption without a partner?
Yes, if your team already administers Kubernetes control planes and has working access to a KMS. The upstream documentation describes the encryption configuration, so the steps are known. The risk sits in the surrounding work: a mistake in the key provider or the encryption configuration can leave the API server unable to read Secrets, and recovery depends on having a tested etcd backup. Check three things before deciding: whether at least one engineer has changed kube-apiserver settings in production, whether someone can operate the KMS integration after go-live, and whether the change can be rehearsed in a non-production cluster that matches production. If any of these is missing, a fixed-scope engagement can cover the gap, with the scope agreed in writing before work starts.
What does our team need to provide for an encryption engagement?
The work needs decisions and access from several people, not only cluster admins.
Typical inputs to confirm:
- Cluster access: administrative access to the control plane nodes or the managed control plane settings, plus read access to the Secret inventory.
- KMS owner: someone who can create or approve the key, set its policy, and confirm how key rotation and audit logging work.
- Application owners: people who can confirm which workloads read Secrets at startup, so a restart or rewrite does not surprise them.
- Change windows: agreed windows for control plane restarts, plus a rollback plan.
- Backups: a current etcd snapshot, stored off the cluster, that has been checked for restore.
If any of these owners sit outside the platform team, agree their time before the schedule is set.
How long does Secrets encryption take with many Secrets or clusters?
The answer depends on the inventory, and no duration applies to every environment. Two counts matter most: the number of Secrets that already exist, because each one must be rewritten before it is encrypted, and the number of clusters, because each cluster needs its own configuration change, verification and change window. Namespaces and the number of workloads that read Secrets at startup also add review time. Before you ask for a schedule, ask the partner to inventory Secrets per cluster and per namespace, and to state which steps run in parallel and which must run one cluster at a time. Ask your KMS owner how long key approval and policy review take in your organization, since that often sets the start date. Any schedule should appear in the written scope, with its assumptions listed.
Does encrypting Secrets alone satisfy audit questions?
No. Encryption at rest protects Secret data stored in etcd and in etcd snapshots, but auditors also ask who can read Secrets through the API and through workloads, how keys are managed, and what evidence shows the controls work. Those questions cover least-privilege RBAC, especially list and watch access to Secrets, which can expose values even when storage is encrypted. They also cover key rotation, key access logs, backup handling and change records. The CIS Kubernetes Benchmark is a common reference for checking cluster configuration, and your auditor will decide which controls apply to your environment. DeployOpen prepares systems, controls and evidence for an assessment, but independent auditors issue the audit report.
What happens to Secrets that already exist before encryption is on?
Turning on encryption at rest affects new writes first. Secrets already stored in etcd stay in their original form until they are written again, so a separate rewrite step is needed to encrypt the existing data. The usual approach is to read every Secret and write it back through the API so the server stores it under the new configuration. During that period, keep the old key available for reads, because the cluster must still decrypt data written before the change. Also remember that etcd snapshots taken before the change may hold the older form, so confirm how long those snapshots are kept and who can access them. Ask your partner to show the rewrite procedure, the verification query and the rollback steps in the written scope.
Should we move Secrets to an external store with the Secrets Store CSI Driver?
It depends on which values must leave etcd and who owns the external store. The Secrets Store CSI Driver mounts secrets from an external provider into pods through a volume, so the values do not need to live in etcd as Kubernetes Secret objects. That is useful when a central vault already governs access and rotation. The trade-offs are more moving parts, a provider plugin that must match your store, and a need to confirm how the driver handles sync to Kubernetes Secrets, because some workloads still expect them. Confirm with the store owner and the provider plugin vendor which versions and authentication methods they support. Encryption at rest still matters for any Secret that remains in the cluster, so plan both controls together.
What to monitor, rotate, back up and document after encryption is on
Encryption at rest protects Secrets in etcd, but it does not show who reads them through the API. Start by sending kube-apiserver audit events for Secrets to your log platform, recording get, list and watch requests with the requesting identity, namespace and source address. Set alerts for unusual read volume from service accounts that have no operational reason to touch Secrets. Also watch kube-apiserver logs for errors that mention decryption or the encryption provider, because a failing KMS plugin can leave the cluster unable to read existing Secrets. Confirm with your distribution how its audit policy is configured, since default policies vary, and check that audit logs never store Secret payloads.
Treat key rotation as a planned change with a written procedure. With a KMS provider, the key lives outside the cluster, and the API server's encryption configuration can hold more than one key. A typical sequence adds the new key first, rewrites existing Secrets through the API so they are encrypted with it, verifies the result, and only then removes the old key. Confirm with your KMS owner how key versions, automatic rotation and key deletion waiting periods work for your provider. Keep the EncryptionConfiguration file and API server flags in version control, and require a second approver for every change, because a single error can make Secrets unreadable.
An etcd snapshot contains Secrets in encrypted form, so a backup is only useful if the matching encryption key is also available to the restore team. Store the key under a separate access control from the snapshot storage, and test periodically that a restore on an isolated cluster can read a known Secret. Keep the snapshot schedule, off-cluster storage and restore steps in the etcd backup and disaster recovery topic. Record which key protected each period of snapshots, because backups taken before a rotation still depend on the old key until they expire.
Review RBAC bindings that grant access to Secrets on a fixed schedule. Check every Role, ClusterRole and binding that allows get, list or watch on secrets, including wildcard rules and aggregated roles, and remove any binding that no longer matches a job. Name the key custodian in writing, with a deputy, and document the break-glass procedure: how access to the key is obtained when the KMS is unavailable, who approves it, and how the event is logged and reviewed afterwards. Finally, track Kubernetes upgrades against the minor releases listed on the Kubernetes releases page. Confirm that your KMS plugin and encryption configuration are supported by the API server version you plan to run, and rehearse the change in staging. The upgrade sequence itself is covered in the upgrades and support topic.
How DeployOpen scopes, builds and hands over secrets management work
DeployOpen plans, configures and documents secrets handling in Kubernetes clusters that run inside your own cloud accounts, data centers or air-gapped networks. Work starts from the written scope, covers encryption at rest with a key management service, access limits on Secrets and any external secret store, and ends with documentation your team can operate.
How we scope
- Engagements are fixed-scope, with the clusters, namespaces and Secrets covered agreed in writing before work starts.
- The written scope names the key management service, key ownership and where encryption configuration will live.
- Confirm early which workloads need external secret stores and which teams hold list and watch access today.
How we build
- Configure encryption at rest for Secrets in etcd, then verify that existing Secrets are re-encrypted and new ones are stored encrypted.
- Tighten least-privilege RBAC so that list and watch access to Secrets is limited to the identities that need it.
- Where required, connect the Secrets Store CSI Driver to an external store and test pod-level retrieval and rotation.
How we hand over
- Deliver the encryption configuration, RBAC bindings and external store manifests as version-controlled artifacts with a change record.
- Provide runbooks for key rotation, access reviews and recovery of the key management service dependency.
- Document test results from verification steps so your security and compliance teams can reuse them as evidence.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers after handover.
- Plan re-verification of encryption and access controls when Kubernetes minor versions are upgraded.
- Ask any provider how key rotation, etcd backups of encrypted data and key loss recovery are handled before signing.
