Upgrading a high availability Argo CD installation without losing reconciliation
A safe upgrade of a high availability Argo CD installation is a planned sequence: first do the version-specific breaking-change work from every upgrade note between your current minor release and the target, then roll out the HA manifests for the target version. Platform teams usually start this work when their current minor release is about to leave patch support. The Argo CD release process states that a minor release ships four times a year, once every three months, and that only the three most recent minor versions receive patch releases. On October 12, 2026, the project published v3.5.5 (marked latest), v3.4.11 and v3.3.16, with v3.6.0 in release candidate. Because that window moves every quarter, confirm which lines are supported on the day you plan the change.
The rollout itself applies manifests/ha/install.yaml for the target version with kubectl apply -n argocd --server-side --force-conflicts. Server-side apply with force conflicts takes field ownership from the previous client-side apply, so review the manifest diff before running it and confirm that any local patches you carry are still needed. The HA manifests run Redis in HA mode with exactly three Redis servers or Sentinels, and Redis is a disposable cache. Cached state can be rebuilt, but expect some reconciliation delay while it refills. Watch each application controller shard after the rollout to confirm it has reconnected and is processing its share of clusters. Take a backup of the Argo CD resources before you start, including the Secrets that hold repository and cluster credentials, because Argo CD keeps its state in Kubernetes objects.
After the rollout, check the replica layout rather than assuming the manifests produced it. The high availability guide says HA manifests need at least three nodes because of pod anti-affinity rules, so confirm that the API server, repo server, application controller and Redis pods are scheduled on distinct nodes. If a cluster has fewer schedulable nodes than the rules require, pods remain pending and capacity silently drops. Also confirm that each component runs the replica count you expect and that the Argo CD version reported by the server matches your target. Record these checks in your change ticket so the evidence exists if a later review asks how the upgrade was verified. Confirm the exact anti-affinity behavior and replica defaults for your target release in the high availability documentation.
What changes in the HA install and what each change requires
This table maps each Argo CD change that affects an HA installation to the action it requires before or after the manifest apply. Read it as a pre-upgrade checklist: rows on RBAC, tracking and repository configuration need policy and configuration work before cutover, while the anti-affinity row sets a cluster sizing requirement. Version-specific upgrade notes are covered on the sibling upgrade pages.
| Change | Why it matters for HA | Action before or after upgrade |
|---|---|---|
| Update and delete RBAC apply only to the application | Existing policies that granted update or delete on project-level or wildcard resources may stop matching after the HA controllers restart. | Export the current RBAC policy, rewrite update and delete rules to target the application, and test with a non-admin account in staging. |
| Logs access needs an explicit logs and get permission | Operators and automation that read pod logs through the UI or API can lose access the moment the new version is live. | Add logs, get permissions to each role that needs log access, then confirm those roles can still view logs after cutover. |
| Annotation-based resource tracking is the default | Changing the tracking method alters how resources are linked to applications, which can show resources as OutOfSync across many apps at once. | Decide whether to keep label tracking through the configured setting, check for OutOfSync noise after the first sync, and confirm the choice with your team. |
| Repositories moved from argocd-cm to Secrets | Repository entries in the ConfigMap are no longer read, so repo-server pods on every replica may fail to fetch Git sources. | Recreate each repository as a Kubernetes Secret with the correct labels before applying the new manifests, and verify every application syncs. |
| Dex RBAC uses federated_claims.user_id | Group or subject mappings that rely on the old claim name can silently stop matching, locking SSO users out of their roles on all HA replicas. | Update Dex-based RBAC subjects to federated_claims.user_id, keep at least one admin path available, and test an SSO login with a scoped role. |
| HA manifests need at least three nodes | Pod anti-affinity rules spread the controller, server, repo-server and Redis pods across nodes, so a two-node cluster cannot schedule the full set. | Confirm three schedulable nodes before upgrading, and confirm the cluster is not IPv6-only, since those clusters are not supported. |
| Redis runs in HA mode with exactly three servers or Sentinels | Redis is a disposable cache, but the HA layout expects three instances, so a partial rollout can leave caching degraded during the restart. | After apply, check that all three Redis pods are Ready and running on separate nodes, and confirm applications still refresh. |
| Controller sharding across replicas | Sharded application controllers reassign applications as pods restart, which can delay reconciliation during the rollout window. | Watch controller pod status and reconciliation timing during the rollout, and confirm each shard holds applications after the upgrade. |
What to back up and inventory before touching the HA manifests
Argo CD keeps its state in Kubernetes objects, while Redis is a disposable cache. The backup and inventory work therefore centers on the objects, settings and dependencies that define your installation, so you can rebuild it or roll back if the upgrade goes wrong. Complete these items before you apply any new manifest.
Back up Argo CD's Kubernetes objects
Export the Application, AppProject and ApplicationSet resources, plus the repository and cluster Secrets in the argocd namespace, before any change. Argo CD is largely stateless, so these objects are the state you would need to restore. Redis does not need a backup because it is a disposable cache that repopulates after a restart. Store the export outside the cluster and confirm you can read it back. Use kubectl get -n argocd -o yaml and keep the output with a clear version label.
Inventory argocd-cm and its keys
Record every key set in the argocd-cm ConfigMap, including admin.enabled, url, resource.exclusions, resource.customizations and application.resourceTrackingMethod. Compare each key against the upgrade notes for the target version, because some settings change meaning or move between releases. Store the current file in version control so you can diff it after the upgrade.
Document RBAC and SSO settings
Capture the argocd-rbac-cm policy, the default policy, scopes, and the group mappings your identity provider sends. Also record whether SSO runs through the bundled Dex or a direct OIDC connection, and the redirect URIs registered with that provider. Major releases can change how RBAC subjects are matched, so a copy of the current policy is your reference for testing access after the rollout.
List repositories, credentials and plugins
Inventory every Git and Helm repository, the Secrets that hold their credentials, and any custom plugins or Config Management Plugin definitions. Note the Kustomize and Helm versions your manifests depend on and any out-of-tree files the repo-server reads. Repository definitions must live in Secrets for current releases, so confirm which entries still sit in older locations before you upgrade.
Rehearse on a three-node HA instance
Build a non-production Argo CD with the HA manifests on a cluster of at least three nodes, because pod anti-affinity rules require that many nodes. Restore your exported objects into it, apply the target version manifest, and watch reconciliation, controller sharding and the Redis HA pods during the rollout. Confirm the rehearsal cluster is not IPv6-only, since that setup is not supported.
How to roll out the HA manifests with server-side apply
This sequence moves an HA Argo CD installation to a new version while keeping application reconciliation running, with a rollback point and explicit checks at each gate. Each step ends with an observable result that tells the operator whether to continue.
Confirm the target version and its upgrade notes
The platform owner selects the target Argo CD release and reads every upgrade note between the current version and the target, including the notes for any major or minor boundary crossed. Record the target version, the release page reference and the list of breaking changes that touch your RBAC, SSO, repository definitions, tracking method or sync behavior. Confirm the patch line you are on is still receiving patch releases under the project's release cadence, since unsupported minor lines cannot be expected to receive fixes. The step is done when a written change record lists each breaking change and the owner who has verified it against your configuration.
Back up Argo CD state and the rollback point
Because Argo CD keeps its state in Kubernetes objects, back up the argocd namespace resources, including Applications, AppProjects, ApplicationSets, ConfigMaps, Secrets and any custom resource definitions, and export the current manifests you applied. Record the currently running image tags for each component and the exact manifest version applied, so you can reapply it if needed. If your cluster runs an etcd or cluster-level backup process, confirm a recent snapshot exists and note its identifier. The step is done when the backup files are stored outside the cluster and a restore procedure has been checked by someone other than the person who made the backup.
Verify node count and anti-affinity capacity
The HA manifests rely on pod anti-affinity rules and need at least three nodes, so confirm the cluster has at least three schedulable nodes that are healthy and not cordoned. Check that each application controller shard, repo-server replica, API server replica and Redis HA member can be placed on a separate node with enough free CPU and memory for the rollout surge. Also confirm that the cluster does not run IPv6-only networking, which the HA documentation says is not supported. The step is done when a capacity check shows that the existing replicas can be moved one at a time without leaving any component with no schedulable location.
Apply the target HA manifests with server-side apply
From a workstation with access to the cluster, download manifests/ha/install.yaml for the target version from the Argo CD release and apply it with kubectl apply -n argocd --server-side --force-conflicts. Server-side apply is used so that field ownership is tracked by the API server and the upgrade does not fail on resources that were previously created by client-side apply. Review the output for each resource reported as configured or created, and note any resource that reports an unexpected conflict. The step is done when the command completes without errors and the output lists the expected resources for the target version.
Watch controller shards and Redis HA pods roll
Watch the rollout of the application controller StatefulSet, the repo-server, the API server and the Redis HA components, using kubectl rollout status and kubectl get pods -n argocd -o wide. Confirm that the controller shards restart in order and that each shard reattaches to its assigned clusters before the next shard moves. Because Redis is a disposable cache, expect it to repopulate after restarts, but confirm the three Redis members or Sentinels return to a healthy state. The step is done when every workload reports its desired replica count as ready and the pods are spread across different nodes.
Verify applications keep reconciling
Check that the Argo CD server reports the target version and that the application controller is processing applications during and after the rollout. Sample a set of applications across different clusters and projects, confirm they show a current sync status and health, and trigger a refresh on at least one application per cluster to confirm reconciliation completes. Review controller logs for repeated errors about cluster connections, repository access or authentication, since these often reveal a breaking change that missed the upgrade notes review. The step is done when the sampled applications reconcile normally and no new error pattern appears in the controller or API server logs.
Check replicas and node anti-affinity afterwards
After the rollout, compare the running replica count for each component against the manifest and confirm that no two replicas of the same HA component share a node unless your cluster has fewer nodes than the anti-affinity rules require. Check that the Redis HA members are spread across nodes and that controller shard assignments cover every registered cluster. Confirm admin and SSO access still works, and record the new image versions in the change record. The step is done when the post-upgrade inventory matches the target manifest, the change record lists every version and verification result, and the rollback point from the backup step is still stored and labeled with the previous version.
How the controller shards and Redis HA behave during rollout
The HA manifests run the application controller across shards and run Redis in HA mode with exactly three Redis servers or Sentinels, so a rolling update replaces pods one at a time while the others keep serving. Reconciliation is not paused for the whole upgrade. Each shard keeps managing its share of applications, and the Redis layer keeps a quorum as long as only one of its three members is restarted at once. Before you start, confirm in the high availability guide that your cluster still has at least three schedulable nodes, because the pod anti-affinity rules that keep replicas apart depend on it. Also confirm that the cluster is not IPv6-only, since those clusters are not supported.
During rollout, expect two temporary effects. The first is a reconciliation delay: applications belonging to a shard whose controller pod is restarting may show stale status or take longer to sync until the new pod takes over. The second is cache rebuilding: Redis is a disposable cache, so the repo-server and controller repopulate it after restarts, and the first refresh of each application can be slower than normal. Both should settle without manual action. Treat the rollout as a problem if a shard stays without a running controller pod, if Redis pods keep restarting or lose quorum, if the API server or repo-server shows sustained errors, or if applications that were Synced before the upgrade stay OutOfSync or Unknown well after the pods are ready. Record the timings you see in a test upgrade so you know what normal looks like for your environment.
Because Argo CD keeps its state in Kubernetes objects, including Applications, AppProjects and Secrets, losing the Redis cache does not lose your deployment definitions. The cache can be rebuilt from Git and from the cluster, so a flushed or corrupted cache is an operational inconvenience rather than a data-recovery event. Your recovery plan still matters, though. The backup you take before the upgrade should cover the Argo CD custom resources and the Secrets that hold repository and cluster credentials, and you should confirm that a restore can recreate them in a fresh namespace. Ask your platform team to confirm how many controller shards your instance runs and whether any custom shard assignment is set, since that determines which applications pause during each controller restart.
What makes an HA Argo CD upgrade take longer
This table lists the factors that expand or shrink an HA Argo CD upgrade and how each one changes the work. Use it to see which drivers apply to your installation before you ask for a scope or quote. The main takeaway is that version distance, sharded scale and custom tooling usually matter more than the manifest apply itself.
| Driver | Why it changes the work | How to keep it down |
|---|---|---|
| Version steps | Each minor step may carry its own breaking changes, so a jump across several releases means reading several upgrade notes and testing each change. | Plan one minor step at a time where possible, and confirm the target version's notes before you start. |
| Clusters and applications per shard | More applications and clusters mean more controller shards to check, more sync behavior to verify and a longer post-upgrade comparison. | Document the shard layout and count before the upgrade so you can confirm the same distribution afterwards. |
| Custom plugins and repo-server tooling | Config management plugins and custom Helm or Kustomize builds must be retested against the new repo-server, which can surface failures only in some apps. | Inventory every plugin and sidecar, and keep a set of representative applications that exercise each one. |
| SSO and Dex changes | Changes to Dex connectors, OIDC settings or RBAC subjects can lock users out of the UI or API if they are not tested before cutover. | Test a login with each identity path on the new version, and keep the admin account available until SSO is verified. |
| Repository configuration migration | Moving repositories from the ConfigMap to Secrets is a data change that must be done and verified before the new controller reads them. | Export and validate each repository definition in a rehearsal, and confirm that repository access still works after rollout. |
| Node count and anti-affinity capacity | HA manifests need at least three nodes for pod anti-affinity, so a cluster with fewer schedulable nodes may leave replicas pending during rolling restarts. | Confirm node count and free capacity before the upgrade, and check replica placement after each rollout. |
| Redis HA behavior during rollout | Redis is a disposable cache in Argo CD, but the HA setup expects exactly three Redis servers or Sentinels, so a partial rollout can leave the cache degraded. | Watch Redis pods during the rollout and confirm all three are healthy before moving to the next step. |
| Backups and rollback preparation | A current backup of Argo CD resources adds a step before the apply, and a rollback path must be rehearsed to be useful. | Back up Argo CD resources and secrets, and write down the rollback steps and who runs them. |
| Rehearsal on a non-production instance | A rehearsal exposes breaking changes, plugin failures and SSO issues before production, and the time it needs grows with the number of changes found. | Use a copy of production configuration, including plugins, SSO and repositories, so the rehearsal reflects real behavior. |
Questions teams ask before upgrading HA Argo CD
What drives the cost of an HA Argo CD upgrade and who gets paid for what?
The cost is driven by the number of clusters and applications Argo CD manages, how many environments need a staged upgrade, and how much of the surrounding configuration must be checked. If you use a partner, the quote usually covers planning, the backup and restore check, the staged rollout in non-production, the production window and post-upgrade verification. Your staff time is a cost even when no invoice is sent, because platform and application owners must review changes and sign off. Ask for a written scope that separates one-time work (upgrade plan, runbooks, validation) from any recurring retainer for patching or support. Ask which upgrade notes the quote assumes you will read, since a version jump with several breaking changes takes longer to review than a patch update.
Can our own team upgrade a high availability Argo CD installation without outside help?
Yes, if your team already runs Kubernetes, understands Argo CD's resources and can test changes before production. The upgrade itself applies the HA manifest for the target version with server-side apply, so the mechanics are not the hard part. The harder work is reading the upgrade notes for the versions you cross, checking RBAC, SSO and repository configuration that may have changed, and proving that reconciliation resumes. Teams without a recent Argo CD upgrade, or with many clusters and custom plugins, often bring in a partner for planning and verification. Whichever path you choose, confirm who owns the rollback decision and who checks application health afterwards.
What does the upgrade need from our staff and change window?
Plan for people who can operate the cluster, people who own the applications Argo CD syncs, and a security or identity contact if SSO or RBAC changes are involved. The change window depends on how many instances you upgrade, whether a non-production copy exists, and how your change board classifies a control plane change. Application teams should be available to confirm that workloads reconcile and stay healthy. Ask the provider to state the staff roles, the approval steps and the monitoring that will run during the window, so your change record matches what happens.
How long does an HA Argo CD upgrade take?
Duration depends on the drivers, not a fixed number: the number of minor versions crossed, the number of clusters and applications, whether you have a tested non-production instance, and how many approvals your change process requires. Upgrades that stay within one minor line and reuse existing backups are shorter than jumps that need RBAC rewrites and repository changes. Ask for the schedule in the written scope, with separate phases for planning, staging, production and verification, and confirm which items depend on your own approvals or access.
Can we roll back an HA Argo CD upgrade if something goes wrong?
Rollback is possible only if you prepared for it before starting. Take a backup of the Argo CD resources in the namespace, including Applications, AppProjects, ApplicationSets, ConfigMaps and the Secrets that hold repository and cluster credentials, and record the current manifest version. Argo CD keeps most state in Kubernetes objects, so restoring those objects and reapplying the previous manifest is the usual path, but confirm this with the upgrade notes for the versions involved. Test the restore on a non-production instance first, and agree in writing who makes the rollback decision and how long the team waits before deciding.
Do Redis and the controller shards need special handling during an upgrade?
Yes, they need planning, though not heavy handling. Redis is a disposable cache in Argo CD, and the HA manifests run it in HA mode preconfigured for exactly three Redis servers or Sentinels, so a rollout restarts cache pods and clears cached state that Argo CD rebuilds. Expect temporary slowness in the UI and reconciliation while caches refill. The application controller can be sharded across replicas, so confirm the shard count and the assignment of clusters to shards before and after the rollout, and check that every shard returns to a running state. Ask the provider to show how they verify shard assignment and replica health.
What should we check after the HA manifests are applied?
Check that every component runs the target version, that each deployment has the replica count your design expects, and that pods are spread across nodes by the anti-affinity rules. The HA documentation says these manifests need at least three nodes because of pod anti-affinity, so a cluster with fewer schedulable nodes will leave pods pending. Confirm that applications reconcile, that sync status and health match the state before the upgrade, and that SSO login and RBAC still work. Keep the verification results with the change record.
Does the upgrade need security review of RBAC, SSO and admin access?
It can, depending on the versions you cross. Some major upgrades change RBAC behavior and how Dex subjects are matched, and the user management guidance recommends disabling the built-in admin user once SSO or local users are configured. Review these settings before and after the upgrade, and confirm that a named administrator can still sign in if the identity provider fails. Ask the provider whether the scope includes a review of these controls or only verification that the platform runs.
What a DeployOpen upgrade engagement includes and excludes
A DeployOpen upgrade of a high availability Argo CD installation is normally a fixed-scope engagement, with scope, deliverables and timelines agreed in writing before any work starts. The written scope names the source and target versions, the clusters and namespaces covered, and whether the instance manages itself or is installed from the ha/install.yaml manifests for the target version. The deliverables typically follow the sequence in this guide: a written inventory of the current installation (controller shards, Redis and repo-server replicas, SSO and Dex configuration, repository Secrets, RBAC policy and projects), a review of the upgrade notes for every minor version in the path, a verified backup and restore test, a rehearsal on a non-production copy, the production rollout, post-upgrade checks, and documentation of what changed and what was checked.
Ask for the inventory and the upgrade-note review to be delivered as separate written items, so your team can approve the list of breaking changes before the rehearsal starts. Confirm in the statement of work whether the backup restore test must succeed in a separate cluster or namespace, who holds the backup artifacts, and which rollback point is accepted for production. Ongoing support, including follow-up upgrades, patch releases and continuous monitoring of replicas and node anti-affinity, is available as a retainer and is priced separately from the one-time upgrade. Hosting, cloud accounts, change approval and access to your clusters remain your responsibility and cost, and the quote should list them as client-provided items. The engagement does not include certifications, audit reports or attestations. DeployOpen prepares configuration and evidence for your own assessors, and independent auditors or certification bodies make any determination about compliance.
How DeployOpen scopes, delivers and supports the upgrade
DeployOpen treats an HA Argo CD upgrade as a fixed-scope change to a running control plane. We agree the target version, the backup and rollback plan and the verification checks in writing before any manifest is applied to your clusters.
How we scope
- Confirm the current and target Argo CD versions, the HA manifest set in use, and the clusters Argo CD manages
- Inventory the upgrade notes that apply between your version and the target, including RBAC, repository and SSO changes
- Record the replica counts, node anti-affinity layout and controller sharding settings that must still hold afterwards
- Agree in writing who takes the backups, who approves the change window and who signs off the post-upgrade checks
How we build
- Rehearse the upgrade on a non-production instance built from the same HA manifest and configuration
- Take and test backups of the Argo CD namespace and configuration before the production change starts
- Apply the target install manifest with server-side apply in the agreed window, then watch controller and Redis rollout behavior
- Compare pod counts, anti-affinity placement and application sync status against the baseline captured before the change
How we hand over
- Deliver a runbook with the exact apply command, rollback steps and the checks that confirm a successful rollout
- Document the version, configuration changes and any deviations from the upstream manifests in your own change records
- Walk your platform team through the post-upgrade checks so they can repeat them on the next minor release
- Provide a list of open items, such as settings to review or upstream notes to re-read before the following upgrade
How we support
- Support after handover is available as a retainer, which can cover patch releases and the next minor upgrade
- Retainer scope, response expectations and any recurring work are set in writing with your team
- Pricing follows from the written scope for your environment, and DeployOpen does not publish a price list
- Ask your DeployOpen contact to confirm which patch releases and clusters a retainer would include before you sign
