How Argo CD with Sealed Secrets keeps values out of Git
A team that wants GitOps for every manifest still has to deal with Secret objects, and committing them in plaintext is not an option. Sealed Secrets lets you commit encrypted `SealedSecret` resources that only the controller running on the destination cluster can decrypt. Argo CD syncs those SealedSecret objects like any other manifest, and the Sealed Secrets controller, installed through its custom resource definition (CRD), turns each one into an ordinary Kubernetes Secret inside the target namespace. The Git repository holds only ciphertext, so anyone with read access to the repo sees no usable credentials.
The division of labor matters for planning. The controller holds the private key, and Argo CD never needs it. Argo CD only has to apply the encrypted objects and report their health, which keeps the key out of the repo-server, out of Argo CD's Redis cache and out of the Argo CD API. Encryption protects the values at rest in Git, but it does not limit who can use the decrypted Secret once it exists in the cluster, so Kubernetes RBAC, namespace boundaries and Argo CD project scoping still decide who can read it. Confirm the controller's current version and its supported sealing scopes with the Sealed Secrets project before you choose a scope for each secret.
The approach also moves risk to the key itself. If the private key is lost, sealed values cannot be recovered, and if it is copied to the wrong place, every committed secret it can open is exposed. Plan key backup, restore testing and rotation before the first production secret is sealed, and decide whether each cluster gets its own key or whether a shared key is acceptable for your change control model.
What you need before sealing the first secret
Sealed Secrets only works when the cluster, the controller, the Git layout and the key owners are agreed before the first secret is committed. The four items below are the checks that connect Argo CD to the in-cluster controller and decide who can seal, decrypt and recover secrets.
Destination cluster and Argo CD access
Confirm that you control the destination cluster and that Argo CD can reach it and the Git repository holding the SealedSecret manifests. The Sealed Secrets controller must run in the same cluster that will decrypt them, because only that controller holds the private key. Check that the Argo CD application for the controller is healthy before adding any encrypted resources, so that sync failures are not mistaken for sealing errors.
Controller and CLI versions
Choose one Sealed Secrets controller version and a matching kubeseal CLI, then confirm the pairing against the project's documentation and your own environment rather than assuming a version. Older controller releases can lack a current CRD, which leaves SealedSecret resources reporting a status or stuck in Progressing in Argo CD. Record the controller version in your change record so that upgrades are planned, not discovered.
Git repository layout
Decide where sealed manifests live before the first commit: one directory per cluster and per environment, with the controller's Application separate from workload Applications. Keep plaintext Secret manifests out of the repository entirely, and add a pre-commit or CI check that rejects them. Agree on who approves changes to sealed files, since a merged SealedSecret is the production change.
Private key ownership
Name the team that owns the controller's sealing key and its backups before any secret is sealed. That team decides who can run recovery, where key backups are stored and how access to them is logged. Developers can usually seal with the public certificate without holding the private key, and that separation should be written into your access model.
Network path from the workstation
Sealing needs the public certificate, which kubeseal can fetch from the cluster API or read from a file you export. Confirm which workstations and CI runners can reach the cluster API, and whether they need a VPN, bastion or a stored certificate for segmented networks. Use a certificate file where direct API access is restricted, and confirm that the exported certificate is the one the running controller uses.
Why sealed secrets fit GitOps and where they stop
Sealed Secrets lets a platform team keep secret material in Git without keeping plaintext there. A SealedSecret is encrypted before it is committed, so the manifest can be reviewed, versioned, and synced by Argo CD like any other resource. The catch is that the ciphertext is bound to the private key held by the Sealed Secrets controller in one specific cluster. Anything that can read that key can decrypt every committed secret for that cluster, so key custody, not the YAML, becomes the main control. Decide before the first commit who can export the controller's key, where the backup lives, and how it is restored into a rebuilt cluster. If the key is lost without a backup, the committed SealedSecrets become unrecoverable and must be re-created from source values.
Argo CD still renders repository content, so the manifests themselves need access control. The Argo CD security documentation notes that the repo-server runs Helm and Kustomize on repository content, which makes untrusted repositories and out-of-tree file access a risk to control. Encryption protects secret values at rest in Git, but it does not stop someone with write access from changing a SealedSecret's target name, namespace, or scope, or from pointing a workload at a secret they should not read. Limit who can merge to the deployment repository, require review for changes under secret paths, and keep Argo CD projects tight so an application cannot deploy into namespaces it does not own.
Install the Sealed Secrets controller so Argo CD reports its resources correctly. The Argo CD FAQ notes that older Sealed Secrets versions lack a modern CRD, which can leave SealedSecret resources showing an unexpected status or stuck in Progressing until a fixed CRD is installed. Confirm the controller and CRD versions in your environment against the current Sealed Secrets release notes before you onboard the first application, and include the CRD in the same sync wave or bootstrap step as the controller so the health check has something correct to read. Recheck this during every controller upgrade, because the status behavior is the first thing a reviewer will notice.
Configuring Sealed Secrets with Argo CD in order
This sequence takes a platform team from an empty destination cluster to a SealedSecret committed in Git, decrypted by the in-cluster controller and synced by Argo CD. Each step ends with a check so you can confirm the previous step worked before moving on.
Install the controller on the destination cluster
Install the Sealed Secrets controller into the destination cluster, ideally through the same process Argo CD or your platform team uses for other cluster add-ons, so its version and configuration are recorded in Git. Only the controller in that cluster can decrypt what you seal for it, so this is the first step that determines where secrets can be used. Confirm the install by checking that the controller pod is Running in its namespace and that the SealedSecret custom resource definition exists. Confirm the CRD is current, because the Argo CD FAQ notes that older Sealed Secrets versions lack a modern CRD, which can leave SealedSecret resources reporting a status or stuck Progressing.
Back up the controller key material
Before you seal any real secret, export the controller's private key material and store the backup outside the cluster in a location your security team controls, such as an encrypted vault or offline store with restricted access. Losing this key means every SealedSecret encrypted for that cluster becomes unrecoverable, and a rebuilt cluster will not decrypt old files without it. Record which key backup belongs to which cluster and who can restore it. Done means you have a backup file you have verified can be read by an authorized person, and the backup location and access list are documented. Ask your Sealed Secrets maintainers or vendor documentation to confirm the current export command for your version.
Seal a test secret with the CLI
On an administrator workstation, create a throwaway Kubernetes Secret manifest with a test key and value, then run the kubeseal CLI against the destination cluster's controller to produce a SealedSecret. Write the output to a file in a working directory, not in the Git repository yet. Check that the output is kind SealedSecret, that the encrypted values are ciphertext, and that the original plaintext does not appear in the file. Delete the plaintext manifest once sealing succeeds so the value does not remain on disk. The step is done when the sealed file exists and you have confirmed the encryption step worked without exposing the value.
Commit the SealedSecret to the repository
Place the SealedSecret manifest in the Git path that Argo CD watches for this destination cluster and namespace, then commit it through your normal review process. Only encrypted output belongs in Git, so reviewers should see ciphertext and metadata, never plaintext. Confirm that repository scanning or pre-commit checks reject any plaintext Secret manifests in the same path. The step is done when the commit is merged to the branch Argo CD tracks and the file appears at the expected path in the remote repository.
Create the Argo CD Application for the path
Create or update the Argo CD Application that points at the repository path holding the SealedSecret, with the destination cluster and target namespace set explicitly. Keep the Application inside the Argo CD project that governs that team, so the project's allowed source repositories, destinations and resource kinds apply. Sync the Application once. Check that the Application reaches Synced and Healthy, and that the SealedSecret resource shows no error status or stuck Progressing condition in the Argo CD UI or CLI. If the status reads Progressing, check the controller version and CRD first.
Confirm the decrypted Secret in the namespace
In the target namespace on the destination cluster, check that the controller has created a Secret with the same name as the SealedSecret and that it contains the expected keys. Use the Kubernetes API to list the keys without printing values to a shared terminal, and confirm the Secret type matches what your workload expects. Also verify that a workload which reads the Secret can start, since a present Secret with the wrong key names is still a failure. The step is done when the Secret exists with the expected keys and the consuming workload has read it successfully.
Record the restore and rotation procedure
Document the steps your team will follow to re-seal a value when it rotates, to restore the controller key from backup into a rebuilt cluster, and to seal for another cluster with its own key. Store this runbook next to the cluster configuration and assign an owner. The step is done when the runbook has been reviewed by the people who would perform a restore, and you have confirmed that a restore from the key backup has been tested in a non-production cluster.
How sealed resources map to Secrets, Argo CD health and ownership
This table maps each object in a Sealed Secrets setup to its owner, location and role in the delivery chain. Read it row by row to see which objects Argo CD tracks and which it never sees in plaintext. The main takeaway is that the controller's private key is the one object no Git or Argo CD workflow can recreate, so its backup matters most.
| Object | Owner and location | Does Argo CD see its value? | What breaks if lost or edited by hand |
|---|---|---|---|
| Plaintext source value | Secret owner, held outside Git (for example in a password manager or a vault) | No, it is never committed to the repository | The value cannot be re-sealed, so rotation must start from a new source value |
| SealedSecret resource | Committed to the Git repository and synced by an Argo CD Application | Only the encrypted payload, never the plaintext | Editing ciphertext by hand produces a SealedSecret the controller cannot decrypt |
| Argo CD Application | Defined in Git or the Argo CD API, tracks the path holding SealedSecrets | Sees the SealedSecret manifest and its health status | Deleting the Application with pruning can remove the controller-generated Secret |
| Controller-generated Secret | Created in the target namespace by the in-cluster controller | Not from Git, but workloads reading it see the plaintext | Deleting it by hand causes the controller to regenerate it from the SealedSecret |
| Controller private key | Stored as a Secret in the controller's install namespace on the destination cluster | No, Argo CD never needs it | Without a backup, every SealedSecret encrypted to that key becomes unrecoverable |
| Controller public certificate | Used by client tooling to seal values, can be shared safely | Not required, but it is the key used to seal | Using a certificate from another cluster seals values that the target cannot decrypt |
| SealedSecret health in Argo CD | Reported through the CRD status the controller writes | Yes, as Progressing or Healthy status | Older controller versions without a current CRD can leave resources reporting stuck status |
Protecting the controller key, the repository and cluster access
Sealed Secrets keeps the decryption key inside the cluster, so the security of the whole pattern depends on who can read that key, where its backup lives, and who can change sealed content. The cards below cover the controls to put in place and check before the first sealed secret reaches Git.
Restrict access to the controller key
The controller's private keys decrypt every SealedSecret in the cluster, so anyone who can read them can read every secret. Limit get, list and watch on Secrets in the controller's namespace to the platform team and the controller itself. Check this with a test that impersonates each Argo CD service account and confirms it cannot read the key Secrets.
Back up keys outside Git and the cluster
A cluster rebuild that loses the controller keys leaves every committed SealedSecret unreadable. Export the active key Secrets on a schedule to an encrypted store that is separate from both Git and the cluster, with access logged. Confirm the restore procedure works in a non-production cluster before you need it, and record who can authorize a restore.
Limit who can create sealed resources
Argo CD projects decide which repositories, destination namespaces and resource kinds an application may deploy, including SealedSecret. Restrict each project to its own namespace and its own repository path, and remove cluster-scoped permissions where they are not needed. Review which RBAC roles can edit SealedSecret objects directly, because a direct edit bypasses the Git review step.
Review changes that alter sealed content
A sealed value is opaque in a diff, so a reviewer cannot tell what changed. Require a second reviewer for any pull request that modifies a SealedSecret file, and ask the author to name the secret and the reason for the change. Pair this with branch protection on the repositories Argo CD watches, so that a change reaches the cluster only after review.
Keep the controller and Argo CD patched
Both the Sealed Secrets controller and Argo CD need regular updates. Argo CD publishes patch releases only for its three most recent minor versions, so an older minor release stops receiving fixes. Track the controller's release notes as well, and test both upgrades in a staging cluster that uses a copy of the production key.
Scan history before migrating
Argo CD redacts cluster, Git and OAuth2 credentials from its API responses and logs. That protection does not extend to values a team committed in plaintext before adopting Sealed Secrets, and those values stay in Git history even after the file is replaced. Scan the repository history for credentials, rotate anything found, and then migrate.
Choosing a key model for one cluster or many
When Argo CD deploys to one destination cluster, one Sealed Secrets controller and its private key are usually enough. Engineers encrypt values against the controller's public certificate, commit the resulting SealedSecret resources to Git, and Argo CD applies them. The main operational cost is protecting that single private key, because losing it means every committed secret must be re-created from its source value.
- Key backup: Export the controller's active key material to protected storage outside the cluster and test that you can restore it into a scratch cluster.
- Re-sealing: Rotating a key or changing a value means running the seal step again and committing the new file, which is a routine change in Git.
- Access review: Limit who can read the controller's secret store and who can fetch the public certificate, since the certificate alone cannot decrypt.
- Recovery drill: Confirm that a rebuilt cluster with the restored key decrypts existing SealedSecret resources before you rely on the backup.
- Ownership: Name one team that holds the backup and the access review, so the work does not drift.
Before you commit to this model, confirm the controller's key rotation behavior and backup procedure in the Sealed Secrets project documentation for the version you plan to run.
When several clusters receive the same application, each cluster normally has its own controller and key pair. A SealedSecret encrypted for one cluster cannot be decrypted by another, so each secret must be sealed separately for every destination. The cost grows with the number of clusters, because every backup, rotation and re-seal happens once per cluster.
- Per-cluster backup: Each controller key needs its own backup and restore test, and the inventory of which key belongs to which cluster must be kept current.
- Re-sealing per cluster: A value change or rotation produces one new SealedSecret for each destination, which adds review work and risk of a missed cluster.
- Access review: Access to decrypted values is spread across clusters, so audit each cluster's controller and namespace permissions separately.
- Consistency checks: Automate a comparison of the sealed files in each cluster directory so a secret present in one environment is not silently missing in another.
- Key sharing: Some teams copy one key set across clusters to avoid re-sealing. That reduces work but widens the blast radius of a leaked key, so treat it as a security decision.
Before designing multi-cluster key handling, confirm the supported key distribution and sharing approach in the Sealed Secrets project documentation, since the exact steps are not covered here.
Many teams keep one application template and vary only the values per environment, such as hostnames, credentials for test and production, or service accounts. Here the key model must match the environment boundaries. If development and production share a cluster, a single key may be acceptable, but the files still need separate paths so a sealed production value is never applied to a test namespace.
- Separate sealed files: Store one SealedSecret per environment under its own path, and let Argo CD's application definitions point only at the matching path.
- Re-sealing scope: Changing the shared template rarely requires re-sealing, but changing a per-environment value requires re-sealing only that environment's file.
- Access review: Reviewers should confirm that the people who can approve production values are not the same people who can modify the template path alone.
- Namespace and scope: Sealed Secrets can bind a secret to a namespace or name, so decide how strictly to scope each sealed value and document the choice.
- Environment drift: Keep a checklist of required keys per environment so a missing value fails review instead of failing at deploy time.
Before you split values across environments, check the scope options and default behavior for your Sealed Secrets version in the project documentation, and confirm them with your own change control rules.
Failures to expect and how to fix them
This table lists the failures most likely when Argo CD manages SealedSecret resources, with the probable cause, a check to run and the fix for each. Read it top to bottom when a sync looks healthy in Git but the workload still lacks its Secret. The main takeaway is that most failures trace back to the controller version, the controller key or the name and namespace bound into each sealed value.
| Symptom | Likely cause | Check | Fix |
|---|---|---|---|
| SealedSecret shows no status or stays Progressing in Argo CD | An older Sealed Secrets version is installed without a modern CRD, so Argo CD cannot read the resource health | Compare the installed Sealed Secrets CRD and controller version against the Argo CD FAQ guidance and the current Sealed Secrets release notes | Install a Sealed Secrets release whose CRD Argo CD can read, then resync the application; confirm the health status changes before relying on it |
| Application syncs, but the SealedSecret cannot be decrypted | The controller's private key was rotated or lost, so ciphertext sealed with an older public key has no matching key on the cluster | List the sealing key Secrets in the controller namespace and compare their creation dates with the commit that added the ciphertext; check controller logs for decryption errors | Restore the original key from backup if it exists; otherwise re-seal every value from the source secret with the current public certificate and commit the new ciphertext |
| Application decrypts on one cluster but not on another | The ciphertext was sealed for a different cluster's controller key, which cannot be shared unless the keys were copied deliberately | Fetch each cluster's public certificate with kubeseal and compare fingerprints with the one used to seal the file | Seal per cluster, or copy the controller key to the target cluster under your key management process and confirm the same certificate is used |
| Secret is not created even though the SealedSecret exists | The sealed name or namespace does not match the target Secret, because the value is bound to both at sealing time | Compare metadata.name and metadata.namespace in the SealedSecret with the Secret the workload expects; check the controller events for unseal errors | Re-seal with the correct name and namespace, or use a deliberate scope setting only after the team has agreed on its security trade-off |
| Generated Secret is overwritten or reverted after a manual edit | Someone edited the Secret in the cluster by hand, and the controller reconciles it back to the decrypted value | Run kubectl describe on the Secret and look for the controller's managed annotations; compare the live value with the decrypted SealedSecret | Change the value in the source system, re-seal it and commit; never edit the generated Secret directly, and restrict who can patch Secrets in the namespace |
| Secret value looks correct in Git but the workload still fails | The workload reads a Secret that was not refreshed, or the pod started before the Secret changed | Check the Secret's resourceVersion and the pod start time; confirm whether the workload reloads Secrets or needs a restart | Restart or roll the workload after the Secret changes, or use a reload mechanism your team has tested |
Questions teams ask before bringing in a partner
What drives the effort of a Sealed Secrets rollout?
The main drivers are how many namespaces and clusters need sealed secrets, how many plaintext secrets already exist in Git or in manual processes, and how your organization holds the controller's private key. Each cluster has its own sealing key, so a fleet with many clusters multiplies the sealing, storage and backup work. A large inventory of existing secrets adds time because each one must be located, classified, re-created from its true source, sealed and checked against the running application. Key custody is the largest variable: a team that already has a documented backup and restore process for controller keys has less to build than one starting from nothing. Ask for a written scope that lists each of these counts and the cluster list, so the quote reflects your actual inventory rather than a generic estimate.
Can our team run Sealed Secrets without a partner?
Yes, a capable platform team can run Sealed Secrets alone, provided it owns two processes from the start: key backup and re-sealing on rotation. The controller's private keys are the only way to decrypt committed SealedSecret resources, so losing them without a backup means every affected secret must be recreated from its source. Re-sealing is required whenever a secret value changes or a key is rotated, and it must be repeated for each cluster that receives the secret. Teams that already run Argo CD, manage Kubernetes secrets carefully and assign a named owner for these two processes usually have the skills in-house. A partner is most useful when the inventory is large, the clusters are many, or the team needs an independent review of the design before production use.
What do we need to provide before the work starts?
The work needs four things from your side. First, access to the target clusters and their Argo CD installation, at a level your security policy approves for the work. Second, a named key owner who can make decisions about where controller keys are stored, who can read them and how backups are protected. Third, admin or delegated access to the Git repositories and to Argo CD itself, including the ability to change projects and repository credentials. Fourth, a secret inventory: a list of secrets in use, their owners, the namespaces and clusters where they are needed, and where their current values come from. Without that inventory, the scope cannot describe what must be sealed. Confirm these items with your security team early, because key custody decisions often need approval outside the platform team.
How do we find and fix the CRD status issue?
The issue shows up in the Argo CD interface: SealedSecret resources report an unexpected status or stay in Progressing, even though the controller is running. The Argo CD FAQ attributes this to older Sealed Secrets versions that lack a modern CRD. To confirm it, compare the installed Sealed Secrets CRD with the version your Sealed Secrets release expects, and check the health status Argo CD reports for the SealedSecret objects. The fix is to install a Sealed Secrets release that ships the current CRD, then let Argo CD refresh the affected applications. Check that the controller and the CRD come from the same release, because mixing them creates the same symptom. Before upgrading, back up the controller's keys, since the upgrade must not change which key decrypts existing secrets.
Where should the sealing keys live, and who can read them?
Sealing keys should be held in a way your key owner can explain to auditors: who can read the private key, where backups are stored, how access is logged and how a restore is tested. The private key lives in the cluster as a Secret managed by the controller, so cluster administrators can typically read it unless access is restricted. Limit read access to that Secret and to the namespace of the controller, and store backups in an encrypted location that is separate from the Git repository holding the SealedSecret manifests. Anyone who holds both the private key and the repository can recover the plaintext, so separating those two paths is a core design decision. Ask any partner to document this model in the written scope and to test a restore before the first production cutover.
How do multi-cluster keys change the work?
Each cluster's controller uses its own sealing key, so a SealedSecret encrypted for one cluster cannot be decrypted by another. Teams that deploy the same application to several clusters must either seal the secret separately for each cluster or design a process that re-seals per target. This multiplies the work of rotation, because one changed value means one re-sealed manifest per cluster, plus one backup per key. Scoping decisions include whether clusters share a key policy, how environment-specific values are separated, and whether a cluster rebuild requires restoring its old key or accepting new sealing. Ask for the cluster list, the number of environments per cluster and the rotation schedule in the written scope, because these determine how much re-sealing the team will carry after launch.
What should a written scope and quote include?
A quote should follow from a written scope that lists the clusters, namespaces, secret inventory, key custody model, Argo CD and repository access, and the target Sealed Secrets version. It should separate one-time work, such as designing the key model, migrating existing plaintext secrets and testing a restore, from recurring work, such as re-sealing on rotation and upgrades. Ask which deliverables are included, including documentation of the key backup procedure, the re-sealing runbook and the CRD verification steps. Confirm who performs each step, what you must provide, what is excluded and how changes to scope are priced. DeployOpen does not publish a price list, so the quote is the place where these components are stated.
How DeployOpen scopes and delivers a Sealed Secrets integration
DeployOpen plans, implements, hardens, documents and supports Argo CD in infrastructure you control, including the Sealed Secrets controller on each destination cluster. We treat the key lifecycle, backups and the Argo CD repository layout as part of the same written scope, so the encrypted secrets in Git can be recovered and rotated without guesswork.
How we scope
- Inventory every cluster that will run the Sealed Secrets controller and confirm which Sealed Secrets version each one runs.
- Define the written scope for key backup, key rotation, re-sealing workflow and multi-cluster key handling before work starts.
- Agree which Argo CD applications and repositories will carry SealedSecret resources, and how the CRD status issue will be checked.
How we build
- Install the controller on each destination cluster with a documented key location, then confirm the Argo CD health checks report correctly.
- Lay out the Git repository so encrypted manifests sit beside their applications and each environment has its own sealing keys.
- Test decryption, sync and re-sealing in a non-production cluster before any production secret is committed to Git.
How we hand over
- Document the key backup procedure, restore steps and the owner responsible for each cluster's sealing keys.
- Deliver a runbook for re-sealing on rotation, including how to recover when a SealedSecret stays Progressing.
- Walk your platform and security teams through the deliverables and record the test results as evidence for your reviewers.
How we support
- Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts.
- Ongoing support, upgrades of the controller and Argo CD, and continuous testing are available as a retainer.
- Quotes follow from the written scope, so ask for the key backup, rotation and multi-cluster items to be priced separately.
