Argo CD with the Secrets Store CSI Driver: when mounts beat synced Secrets
Choose a CSI volume mount when the workload must read a secret from a cloud or Vault store without that value ever living in Git or in a Kubernetes Secret object. Argo CD then deploys only the SecretProviderClass and the workload manifests, and the Secrets Store CSI Driver fetches the value and mounts it into the pod at start. Choose a synced Secret instead when a controller, ingress or another tool needs the value as a Kubernetes Secret, because the CSI driver's sync option is the exception and not the default path.
The trade-off is operational. A mount ties secret delivery to pod start and to the pod's identity, so a rotated value usually reaches a running container only after a restart or a reload the application supports. Confirm with the driver's documentation whether your version supports rotation polling and secret sync, and with your provider's documentation how it authenticates the node or pod (for example, workload identity or a service account token). Plan for the mount failing at start, since a missing permission or an unreachable store typically leaves the pod stuck in container creation rather than failing the sync in Argo CD.
Keep SecretProviderClass objects in the Argo CD repository with provider-specific parameters such as secret names, versions and vault paths, but never credentials. Per environment, use separate paths or clusters so a staging application cannot resolve production secrets. Before adopting this pattern, confirm the supported integration method for your driver and provider versions in their current documentation, since no validated version pairing has been established for this guide. Test rotation and a forced pod restart in a non-production cluster first, and record which applications need a restart to pick up a new value.
What you need before connecting Argo CD to the CSI driver
The CSI driver pattern touches four systems at once: the destination cluster, the Argo CD instance that manages it, the external secret store and the network path between them. Confirm each item below before you commit SecretProviderClass resources to Git, because a gap in any one of them shows up as a pod that will not start.
Destination cluster and node layout
Confirm the cluster distribution, the Kubernetes version and whether the CSI driver and provider run as DaemonSets on every node pool that will host the workloads. The driver mounts secrets on the node, so a node pool without the provider component cannot serve these pods. Ask the driver documentation which Kubernetes versions it supports and whether your managed control plane needs any extra settings.
Argo CD version and sync behavior
Record the Argo CD minor version you run and check its release notes for the resource tracking method and RBAC model, since both affect how Argo CD decides to prune or report a SecretProviderClass. Confirm that the Argo CD service account can manage the custom resource in the destination namespace. Plan the sync wave so the CSI driver and provider exist before any workload mounts a secret.
Secret store account and access policy
Decide which identity the provider uses to read secrets: a workload identity bound to the pod service account, or a node-level identity. Scope its policy to the specific secret paths each environment needs, and confirm how the provider authenticates against your store. The Git repository should record only the secret names and paths, never the credentials that read them.
Node and network path to the store
Nodes, not pods, make the call to the store endpoint when the driver mounts a volume, so egress rules must allow the node subnet to reach the provider's API. Check private endpoints, proxy settings and any egress firewall between the cluster and the store. Ask the provider documentation which endpoints, ports and regional hostnames it calls, and test the path from a worker node before any Argo CD sync depends on it.
TLS and certificate trust
Identify whether the provider endpoint uses a private certificate authority or a custom root. If it does, the CA bundle must be trusted by the provider pods and any component that calls the store on the node. Confirm the provider's documented method for supplying that bundle, and decide who rotates it, since a CA change that is not planned can stop every mount at once.
Access to the Argo CD instance
Confirm who can create or change SecretProviderClass resources through Argo CD, because those objects point workloads at specific store paths. Restrict write access to the platform team and review the project's allowed resource kinds. Keep the local admin user disabled once SSO is configured, so that changes to these objects are traceable to a named identity.
How the SecretProviderClass keeps secret values out of Argo CD
A SecretProviderClass tells the Secrets Store CSI Driver which external store to read and which objects to fetch from it. Its spec carries a provider name, a parameters block with provider settings such as the store identifier and the list of object names and paths, and an optional secretObjects block that asks the driver to create a Kubernetes Secret from selected objects. None of these fields holds a secret value, so the manifest can be committed to Git and deployed by Argo CD like any other resource. The value is fetched only when a pod that references the class starts and mounts the volume. Confirm the exact apiVersion and field names against the driver release you run, because they have changed across versions.
The main trade-off sits in secretObjects. Without it, the mounted secret exists only as files inside the pod, and no Kubernetes Secret object is created. Nothing in the cluster then holds a copy that the Secrets API, backup tools or other workloads can read. With it, the driver creates a Secret stored in etcd, and that copy must be governed like any other Secret, including encryption at rest and RBAC. Add secretObjects only when a consumer, such as an ingress controller or a workload that reads environment variables, needs a Kubernetes Secret. Because the synced Secret is created when a pod mounts the volume, a class that no pod mounts produces no Secret. Record the choice per workload in your design.
Argo CD tracks the SecretProviderClass as an ordinary custom resource, so the application's sync and health status shows that the manifest was applied to the cluster. It does not show that the driver could authenticate to the store or read the object. A healthy application can still have pods stuck at container creation because the mount failed, for example when the workload identity lacks read access or an object name is wrong. Those failures appear in pod events and kubelet logs, not in the Argo CD resource tree, unless you add a resource health customization or monitor pods separately. Put that check in your runbook, and confirm how your Argo CD version evaluates custom resource health before you rely on it.
When a CSI mount beats a synced Kubernetes Secret
Workloads that read credentials from files are the strongest fit for a CSI mount. The Secrets Store CSI Driver presents the value as a file in a volume at pod start, so the secret does not have to exist as a Kubernetes Secret object at all. That keeps the value out of etcd in the common configuration, which matters when your auditors ask where secret material is persisted. Rotation depends on the application: the mounted file can change while the pod runs, but the process only sees the new value if it re-reads the file or the pod is restarted.
- etcd exposure: a pure file mount avoids creating a Secret object, but enabling secret sync for the same value brings it back into etcd, so check whether sync is turned on.
- Rotation behavior: confirm whether the driver's rotation setting is enabled in your installation and how often it polls the provider.
- Reload path: verify that the application re-reads files or that your deployment tooling restarts pods after a rotation.
- Mount failure: if the provider is unreachable or the identity is wrong, the pod stays in a pending or container-creating state and does not start.
- Argo CD role: Argo CD syncs the SecretProviderClass and the Deployment, so a drift in either shows as an out-of-sync application rather than a silent gap.
Before you commit to this tab, test a rotation in a non-production cluster and record which pods picked up the new value without a restart.
Many applications and sidecars accept credentials only as environment variables. A CSI mount does not inject environment variables by itself, so teams usually either sync a Kubernetes Secret and reference it with secretKeyRef, or mount the file and have an entrypoint script export it before the process starts. The second option keeps the value out of etcd but moves the export step into your image or startup logic. Environment values are read once at process start, so rotation almost always means a pod restart.
- etcd exposure: the sync-based approach stores the value in a Secret object, while the entrypoint approach keeps it in a file and in the process environment.
- Process visibility: environment variables can appear in process listings, crash dumps and some debugging output, so review what your images and logging expose.
- Rotation behavior: a rotated value reaches the container only after a restart, so plan a rollout strategy for each workload.
- Mount failure: an entrypoint that cannot read the mounted file often fails in a way that looks like an application error, so add explicit checks to the startup script.
- Verification: confirm which tool owns the wrapper script and that Argo CD does not revert it during sync.
Confirm with the application vendor whether it can read the value from a file path instead, since that choice removes the environment variable entirely.
Some controllers, ingress components, image pull configurations and third-party operators only accept a Kubernetes Secret reference. They cannot consume a CSI volume, so the synced Secret is often the only option. In this case the mount does not replace the Secret, and the design question becomes how to limit who can read it and how to keep it from being recorded in Git. The Secrets Store CSI Driver can create a Kubernetes Secret from the provider value when sync is configured, so the value is then stored in etcd like any other Secret.
- etcd exposure: a synced Secret is persisted in etcd, so enable encryption at rest for the cluster and restrict list and get permissions on the namespace.
- Rotation behavior: the Secret object updates when the sync interval runs, but the consuming tool may cache the value until it reloads, so test each tool.
- Lifecycle coupling: when the SecretProviderClass is removed, the synced Secret may persist or be deleted depending on configuration, so check the behavior in your version.
- Mount failure: a missing synced Secret causes the consumer to fail its own startup, which can be harder to trace back to the provider.
- Argo CD role: Argo CD should manage the SecretProviderClass manifest and never the Secret's contents, so confirm that no rendered Secret with data is committed.
Ask the tool's maintainers whether they accept a file path or a projected volume, because that answer decides whether the sync is needed at all.
When the same application runs across several clusters, the mount design must repeat in each one, and the provider identity model becomes the main cost driver. Each cluster needs its own way to authenticate to the external store, whether through workload identity, a cloud role or a service account bound to the provider. Argo CD can deploy the same SecretProviderClass to every cluster through ApplicationSets, but the parameters for each provider and each cluster differ, so the templating must be explicit. A failure in one cluster should not depend on the state of another.
- Identity per cluster: each cluster needs its own trust relationship to the provider, which adds setup and review work for every new cluster.
- Provider choice: the supported providers and authentication methods differ by cloud and by store, so confirm each provider's current driver support before standardizing.
- Rotation consistency: rotation timing can differ between clusters, so define how long mixed values can run during a change.
- Mount failure isolation: one cluster's identity failure should stop only that cluster's rollout, so keep sync waves and health checks per cluster.
- Audit scope: provider access logs and Kubernetes audit logs must be collected from every cluster for the same evidence.
Ask for a written list of the identity method, provider and sync setting for each cluster before approving the design.
Configure the driver and let Argo CD manage SecretProviderClass
This sequence installs the Secrets Store CSI Driver and a provider on the destination cluster, grants a least-privilege store identity, and lets Argo CD sync SecretProviderClass manifests so that pods mount secrets directly from the external store.
Install the driver and provider on the destination cluster
The platform team installs the Secrets Store CSI Driver and the provider for your secret store on the destination cluster, using the install method each project documents. Argo CD can manage these installs too, but the driver is cluster-level infrastructure, so many teams keep it in a separate Application owned by the platform team. Confirm the driver and provider versions are compatible with your Kubernetes version before installing. Done means the driver DaemonSet and the provider pods are Running in their namespace and the SecretProviderClass custom resource definition exists (check with kubectl get crd).
Create the store identity with least privilege
In the external secret store, create an identity that can read only the specific secrets this cluster and namespace need, such as one path prefix per environment. Avoid broad read access to the whole store. Record the identity's name and the secret paths it can read, because these become the parameters in your SecretProviderClass. Done means a test read of an allowed secret succeeds with that identity and a read of a secret outside the allowed paths is denied, which you confirm with the store's own access tooling.
Register the identity with the cluster
Connect the store identity to the cluster using the mechanism your provider documents, such as workload identity federation for the cloud provider or a service account bound to the identity. Keep the binding in the platform team's repository or in a reviewed manifest, not in a secret value. Confirm the mechanism with your provider's documentation before committing anything. Done means a test pod using the bound service account can obtain a token for the store, which you verify by checking the provider pod logs or a short diagnostic pod, without exposing a secret value.
Commit SecretProviderClass manifests to Git
Write one SecretProviderClass per workload namespace in your Git repository. Each manifest names the provider, lists the secret object names and their external paths, and sets the Kubernetes Secret sync option only if a workload needs a synced Secret (the mount alone does not create one). Contain no secret values, only references. Review the manifests through your normal change flow. Done means the manifests are merged to the branch that Argo CD tracks and pass schema validation in your pipeline.
Add the Argo CD Application that syncs them
Create an Argo CD Application that points at the directory holding the SecretProviderClass manifests and targets the destination cluster and namespace. Scope it to an AppProject that allows only the resource kinds and namespaces this team needs. Enable automated sync only after a manual sync has succeeded. Done means the Application shows Synced and Healthy in the Argo CD UI or CLI, and kubectl get secretproviderclass in the target namespace lists the expected objects.
Mount the secrets into a test workload
Deploy a test pod in the same namespace that references the SecretProviderClass in a CSI volume with the driver name and the volume attributes set to that class. Keep the pod spec in the same Git path so Argo CD manages it. Done means the pod reaches Running with the volume mounted, and a read of the mounted file inside the container returns the expected value. Confirm that the secret does not appear in the Argo CD application's manifest view, since only references are stored there.
Verify rotation reaches the running pod
Rotate the test value in the external store, then wait for the driver's sync interval to pass, which your provider and driver versions document. Check that the mounted file in the test pod shows the new value without a pod restart, unless your configuration requires one. If you enabled a synced Kubernetes Secret, confirm it also updated. Record the observed refresh behavior so the owning team knows how long a rotation takes in practice. Done means the new value is visible in the running pod.
Which resource and identity maps to which system
This table traces each object in a Secrets Store CSI setup to the system that owns it, what Argo CD can read, and where the reader should confirm the binding. Read it top to bottom to follow a secret from the external store, through the cluster identity, into the pod's mounted file. The main takeaway is that Argo CD manages the configuration objects, while the store and the workload identity hold the secret value.
| Object | Owning system | What Argo CD can read | Where to confirm the binding |
|---|---|---|---|
| Argo CD Application | Git repository and Argo CD | The manifest in Git and its sync status | Application spec path and destination namespace in the Argo CD UI or CLI |
| SecretProviderClass | Argo CD (synced from Git) | Full resource, including provider name and object names | Manifest in Git and the live object reported as synced or out of sync |
| Workload service account | Cluster namespace owner | Name and annotations, not tokens in the mounted volume | Pod spec serviceAccountName and the namespace's RBAC bindings |
| Cloud identity binding (workload identity or IRSA-style trust) | Cloud IAM administrator | Nothing directly; Argo CD only sees the service account annotation | Cloud IAM trust policy or federated credential and the service account annotation |
| Store secret name and path | External secret store team | Only the names in SecretProviderClass, never the secret value | Provider console or CLI listing for the path, plus the object names in the class |
| Secret value | External secret store | Nothing; Argo CD does not read the value | Provider access policy and audit logs for reads by the workload identity |
| Mounted file path | Kubernetes kubelet and the pod | Rendered manifest shows the mount path, not the content | Volume mount path in the pod spec and file presence inside a running container |
| Optional synced Kubernetes Secret | Secrets Store CSI Driver, if syncing is enabled | The Secret object if Argo CD can list it in that namespace | SecretProviderClass secretObjects block and the Secret name the workload references |
Security controls for CSI-mounted secrets
Mounting secrets through the Secrets Store CSI Driver moves the secret out of Git and out of Argo CD's view, but it also creates new access paths that need controls. The cards below cover who can reference a SecretProviderClass, what Argo CD is allowed to manage, and how store access is recorded.
Scope store policies per namespace
Give each namespace its own identity in the external store, with read access only to the secret paths that workloads in that namespace need. A shared store role across namespaces lets one team's pod read another team's credentials through the same SecretProviderClass pattern. Ask your store team to show the policy bindings for each namespace, and test that a pod in one namespace is denied a path that belongs to another.
Restrict service accounts that can mount
A SecretProviderClass only delivers secrets to pods whose service account is authorized by the provider's identity binding. Keep that list short: one service account per workload, with no default service account in production namespaces. Review who holds rights to create pods that reference the class, because a pod with access to a mount can read whatever the class exposes.
Limit Argo CD project permissions
Argo CD projects should allow sync of SecretProviderClass objects only in the namespaces and clusters that owning teams control, and deny cluster-scoped resource creation for those teams. Without this limit, an application author can point a class at a store path they should not reach. Confirm the project's resource whitelists and destination rules in a test project before you grant them to production teams.
Keep store identifiers out of broad views
The class definition holds vault names, secret paths, tenant IDs and identity client IDs. Those values appear in Git, in the Argo CD UI for anyone with read access and in the API responses of users with broad roles. Place sensitive parameters in a narrower project or namespace, and confirm the Argo CD role bindings that expose live manifests to users outside the platform team.
Watch the repo-server rendering path
Argo CD's repo-server runs Helm and Kustomize on repository content, so a chart or overlay that generates a SecretProviderClass can carry values you did not review. Restrict who can merge changes to chart values and overlays for these resources, and ensure the repo-server does not read out-of-tree files that contain store configuration. The Argo CD security documentation lists this as a risk to control.
Audit store access logs
Enable access logging in the external store and forward it to your central log platform, so each secret read is tied to a workload identity, namespace and time. Compare reads against the expected service accounts on a schedule, and alert on reads from unknown identities. Log formats, retention and available fields differ by provider, so confirm them with the vendor before you rely on them as evidence.
Common mount failures and how to diagnose them
This table lists the failures teams most often meet when Argo CD manages SecretProviderClass resources and the Secrets Store CSI Driver mounts secrets into pods. Read each row from the symptom down to the first check, then the fix, then the logs and events that confirm the cause.
| Symptom | Likely cause and first check | Fix and where to read logs or events |
|---|---|---|
| Pod stuck in ContainerCreating with a mount error | The CSI volume cannot be prepared. Run kubectl describe pod and read the Events section for the driver error text, and confirm the SecretProviderClass name in the volume spec matches an existing object in the same namespace. | Correct the name or namespace, or let Argo CD sync the missing SecretProviderClass first. Driver logs are in the secrets-store-csi-driver pods on the node, and the kubelet events appear on the pod. |
| SecretProviderClass exists but cannot authenticate | The provider pod cannot obtain credentials. Check the service account annotations or workload identity binding that the provider expects, and confirm the identity has read access to the named secret in the store. | Fix the identity binding or the store policy, not the manifest. Read the provider container logs in the driver namespace and the status of the pod that requested the mount. |
| Secret rotated in the store but not in the pod | File-based mounts refresh only when the driver rotation feature is enabled and the kubelet polls the volume. Confirm rotation is on in your driver version and that the application reloads the file, since many apps read secrets only at startup. | Enable rotation in the driver configuration, set a poll interval you can justify, and restart or signal the workload if it caches credentials. Compare the store version with the mounted file content, and check driver rotation logs. |
| Kubernetes Secret sync never appears | Secret syncing only happens when the SecretProviderClass declares secretObjects and a pod mounts the volume. Confirm a pod actually mounts it, because the Secret is removed when no pod uses the volume. | Add the secretObjects block with the target name and namespace, then start a pod that mounts the volume. Check the SecretProviderClass status and driver logs for sync errors. |
| Argo CD Application shows Healthy while the mount fails | Argo CD judges health from the resource kinds it knows. A SecretProviderClass is usually reported Healthy once it exists, so a broken mount can sit inside a Deployment that is still rolling out or that has old ready pods. | Add a health check or review the rollout status, and confirm the Deployment condition shows available replicas. Read the Argo CD application events and the pod events together, and treat a Healthy label as a sync result, not a mount result. |
Questions teams ask before connecting CSI mounts to Argo CD
What does a CSI mount integration cost, and what drives the price?
The cost depends on how many secret stores, clusters, workloads and rotation rules are in scope, and no fixed figure applies to every estate. Costs fall into one-time work (designing the path layout, configuring cloud identities, writing SecretProviderClass manifests, wiring Argo CD applications, and testing rotation) and recurring work (updating providers, reviewing new workloads, and handling rotation incidents). Each additional store or cluster multiplies identity and access setup. Ask any partner for a written scope that lists each store, cluster and workload, the identity model used for each, the rotation test you expect, and which items are fixed-scope deliverables and which are retainer work. DeployOpen agrees scope and deliverables in writing before work starts and gives a quote from that scope.
How long does a first CSI and Argo CD integration take?
Duration follows the number of steps in your environment rather than a standard length. A first integration usually moves through the same sequence: confirm the secret store and its access model, grant the cluster a workload identity for that store, install the driver and provider on a non-production cluster, commit SecretProviderClass resources through Argo CD, prove that a test pod receives the value, and then repeat for production with rotation and failure tests. Each store with a different authentication method adds a step, and each cluster that needs its own identity adds another. Ask for the steps and their order in the written scope, along with the environments where each step is verified.
What must our team provide before the work starts?
Your team supplies the access the integration depends on. Typically this includes cloud identities that the cluster can assume or federate with, read permissions on the secret store paths for each environment, cluster administrator rights to install the Secrets Store CSI Driver and its provider, and access to the Argo CD instance and the Git repository that holds the manifests. You should also provide a non-production cluster and a test workload that can prove a secret arrives and rotates. Confirm with your security team who approves identity and policy changes, because those approvals often set the schedule. Ask any partner to list these inputs as prerequisites in the written scope.
Can our internal platform team do this without a partner?
A capable platform team can complete the work alone if it has time, Kubernetes and cloud identity skills, and a clear rollback plan. The steps are well defined, but the risk sits in the details: workload identity trust policies, least-privilege store paths, mount failures that stop pods from starting, and rotation that does not reach running workloads. Teams that choose a partner usually want those decisions reviewed and the evidence written down before production. Whichever route you take, decide who owns the SecretProviderClass manifests in Git and who responds when a mount fails at 2 a.m.
Does mounting secrets with CSI remove the need for every synced Kubernetes Secret?
No. Mounts are a good fit for workloads that read secrets from files or that must receive rotated values without a Secret object in the cluster. Many applications still expect environment variables or a Kubernetes Secret reference, and some controllers, ingress resources and image pulls need a Secret object. The Secrets Store CSI Driver can optionally sync a mounted value into a Kubernetes Secret, which brings back the cached object you were trying to avoid, so that choice needs an explicit review. Plan an inventory of which workloads need files, environment variables or Secret objects before you migrate, and keep the decision in Git so reviewers can see it.
What breaks most often after the mounts go live?
The common failures are identity misconfiguration, missing store paths, and pods that cannot start because a mount fails. Argo CD may report an application as healthy while a pod waits on a volume, so check pod events and the driver logs as well as the sync status. Rotation is another frequent gap: a mounted file can update, but the application must reload it, and some workloads read the value only at startup. Ask the partner to include a failure test for an expired identity and a rotation test in the written scope, and confirm what your application does when the file changes.
Which secret store providers should we check first?
Choose the provider from the cloud or data center where your secrets already live, and confirm that the driver and the provider support your authentication method for that store and for each cluster type. The Secrets Store CSI Driver works with provider plugins for several clouds and for Vault-compatible stores, but support, authentication modes and maintenance status differ by provider. Confirm the provider version, its release status, and its support for your Kubernetes distribution with the provider maintainers and in your own test environment before you commit to a design.
Do we need an audit trail for the secret mount configuration?
Auditors usually want to see who changed which mount definition, who approved it and how the change reached production. Keeping SecretProviderClass manifests in Git and deploying them through Argo CD gives you a reviewable change history and a sync record, but it does not replace store access logs, which show who read the underlying secret. Confirm with your compliance lead which evidence they require, then ask a partner to list the artifacts the work produces. DeployOpen prepares systems, controls and evidence for compliance assessments, but independent auditors and certification bodies issue any audit report or certification.
How DeployOpen works on this
DeployOpen plans and implements self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks. For Argo CD with the Kubernetes Secrets Store CSI Driver, we start from your secret stores, cluster layout and sync model, then deliver the mounts, GitOps layout and verification in a fixed scope agreed in writing.
How we scope
- Inventory the secret stores, clusters and namespaces that will hold SecretProviderClass resources and the workloads that mount them.
- Record which secrets move to direct pod mounts and which stay as synced Kubernetes Secrets, with the reason for each.
- Agree in writing the provider per cloud, authentication method, Argo CD Application layout and sync waves before any build starts.
How we build
- Install the driver and provider on a non-production cluster, then deploy SecretProviderClass resources through Argo CD Applications.
- Test rotation, pod restarts and sync behavior, confirming Argo CD health status reflects the mounted resources correctly.
- Review RBAC, service account bindings and what the repo-server and Redis cache can see, so no secret values enter Git or the cache.
How we hand over
- Document the GitOps repository layout, provider configuration per environment and the steps for adding a new secret path.
- Provide runbooks for rotation, failed mounts and driver upgrades, written for the engineers who will operate them.
- Walk your platform team through the running system and the verification steps used to accept the delivery.
How we support
- Engagements are fixed-scope, with deliverables and timelines confirmed in writing before work starts.
- Ongoing support, upgrades and continuous testing are available as a retainer after handover.
- Ask any partner to confirm which driver, provider and Argo CD versions the written scope covers and who owns each upgrade.
