When Argo CD application controller sharding becomes necessary
The main question for this section is whether a single Argo CD application controller can keep up with the applications and clusters you run, or whether its reconciliation work should be spread across several controller replicas. The application controller is the component that compares live cluster state with the desired state in Git and acts on the difference. In Argo CD's high availability model, the controller can be sharded across replicas, while the rest of the system stays largely stateless, with data kept in Kubernetes objects and Redis acting as a disposable cache. Sharding therefore changes how controller work is divided, not where application definitions are stored.
Teams usually reach for sharding when one controller reconciles a large number of applications across many destination clusters and reconciliation falls behind. Typical signs include applications that stay OutOfSync for longer than expected, sync status that lags behind recent commits, and controller pods that are close to their CPU or memory limits. Confirm these symptoms against your own metrics and logs before changing the topology, because the thresholds depend on your cluster sizes, change rates and hardware. The right trigger is a measured backlog on your own environment, not an application count copied from another deployment.
Sharding does not fix every slow sync. If the delay sits in manifest generation, such as Helm or Kustomize rendering taking a long time or the repo-server running out of memory, the bottleneck is the repo-server, and adding controller replicas will not change that. Similarly, a monorepo that causes every commit to trigger broad refreshes is a repository and polling problem. Before you shard, check where time is spent, then compare the controller's behavior with the Argo CD high availability documentation to confirm what the HA layout supports in your version.
Which sharding method fits which estate
When an estate has a handful of clusters that rarely change, sharding is mostly a sizing question. A fixed assignment is easy to reason about, and most of the work is confirming that no single controller carries the heaviest cluster alone while the others sit idle.
- Pin the dominant cluster: Give the cluster with the most applications its own shard so it does not slow reconciliation for smaller clusters.
- Write the mapping down: Record which cluster lands on which replica in the same Git repository that defines the controller, so a restart does not silently change ownership.
- Keep the replica count modest: A few replicas with a clear mapping are often enough, and extra replicas add memory and coordination overhead without a matching gain in throughput.
- Watch for one hot replica: Recheck load distribution after each major application wave, since growth in one team's workloads can quietly concentrate work on one shard.
After any change to the cluster count, confirm each replica owns the clusters you expect and that reconciliation times on the busiest cluster have not drifted. The high availability documentation describes the controller's role in the HA layout.
When clusters are created, retired or re-homed frequently, hand-maintained mappings become the failure point. Hash-based placement, or dynamic distribution where your version supports it, assigns clusters automatically, so the team spends its effort on rebalancing behavior rather than on editing assignments.
- Let placement follow membership: Hash-based assignment means a new cluster is placed without a manual edit to the mapping.
- Expect movement on membership change: Adding or removing a replica or a cluster can move some clusters to another controller, so plan for a short reconciliation spike and cache rebuilds.
- Check balance, not just count: Hashing spreads clusters by identity, not by workload size, so a fleet with a few large clusters can still concentrate load on one replica.
- Confirm the version behavior: Read the release notes for your minor version to confirm how distribution and rebalancing work. Argo CD patches only the three most recent minor versions, so an older release may lack the behavior you need.
After each cluster is added or removed, confirm it appears under exactly one shard, that application counts per replica stay within the range you expect, and that no cluster is left unassigned. Ask any implementation partner to show these checks in the runbook.
Many estates combine a few very large clusters with many small ones. Spreading clusters evenly by count works poorly here, because the work follows application count and resource churn, not the number of clusters. The usual answer is to isolate the heavy clusters and group the small ones.
- Isolate the heavy clusters: Give each very large cluster a dedicated replica so its reconciliation does not delay smaller clusters.
- Group the small ones: Place many small clusters together on shared replicas, where their combined load is usually light.
- Avoid count-based spreading for skewed fleets: Round-robin balances the number of clusters per replica, so two large clusters can land on the same controller without anyone noticing.
- Size shards from observed load: Decide whether a shard needs more capacity from controller metrics and logs, not from the cluster count alone.
After clusters are added or removed, check that no shard holds two of the largest clusters by accident and that the shared small-cluster replicas still reconcile within your normal window. Dedicated shards for large clusters also need upkeep, so review them whenever a cluster grows or shrinks substantially.
How pinning a cluster changes shard placement
Pin a large or sensitive cluster to a dedicated shard when you want its reconciliation load isolated from everything else. A pin gives that cluster's applications a controller replica that does not share its work with other clusters, so a noisy neighbor on another shard cannot delay syncs for the pinned cluster, and a slow or failing destination cluster cannot drag down unrelated work. The cost is that the shard no longer takes part in the normal distribution of clusters. Every cluster you pin is one less cluster the automatic placement can use to balance the rest, and the shards that remain carry whatever load was not pinned.
Placement changes when the replica count changes. When you add or remove controller replicas, the remaining unpinned clusters are redistributed across the shards that are available, and a pinned cluster stays where you put it as long as its shard exists. If you scale down to a count that no longer includes the shard a cluster is pinned to, that assignment cannot be honored, so check the pin before you change the replica count. Because pinned clusters are excluded from rebalancing, a shard holding a large pinned cluster can stay heavy while the others are kept even, which is the uneven outcome you should plan capacity around rather than assume the controller will correct it.
Test placement in a non-production environment before you pin anything in production. Confirm the shard assignment setting and the pin key for your Argo CD version in the high availability documentation, then check how the controller reports which shard owns each cluster, and whether that report is in logs, the cluster settings or the Application status in your installed release. Record the expected owner of each cluster, scale the replicas up and down once, and compare the reported owners with your records. Then measure the load on each shard with a representative set of applications before you decide the pin is worth its effect on the other shards.
Setting up controller sharding step by step
This sequence moves an Argo CD installation from a single application controller to several shards, so each controller replica owns a defined set of clusters and the load is spread across them. Each step ends with a check you can run before moving on.
Confirm the HA manifests and three-node cluster
Confirm the Argo CD installation was deployed from the high availability manifests and that the cluster has at least three nodes, because the HA manifests use pod anti-affinity rules that need that capacity to schedule. Your platform team owns this check, and DeployOpen can review the existing install as part of the planning work. Check the node count with kubectl get nodes and confirm the controller runs as a StatefulSet with the expected replica count before you change anything. If the install is not HA, plan that change first, since sharding alone does not add the anti-affinity and Redis HA layout that the HA documentation describes.
Count and register destination clusters
List every destination cluster that Argo CD manages, including the in-cluster destination where Argo CD itself runs, and note the applications and approximate sync load each one carries. Clusters are registered as Kubernetes Secrets in the Argo CD namespace, so list those Secrets with the cluster label to get the authoritative inventory. Record which clusters are large enough to dominate a controller. The output is a table of cluster names, application counts and owners that you will use in the next steps. Confirm the list with the application owners, because a cluster that looks small by application count can still hold heavy workloads.
Choose the sharding method
Select the sharding algorithm before you set replica counts. The application controller supports legacy, round-robin and consistent-hashing distribution, and the choice affects how clusters move when replicas are added or removed. Read the sharding section of the high availability documentation for the current options, the environment variable or setting that selects each method, and any version constraints that apply to your release. Document the chosen method and the reason for it in your change record. Confirm the method against the exact Argo CD version you run, because the supported options and their defaults can change between releases.
Set the controller replica count
Set the number of application controller replicas to match the number of shards you planned, and set the sharding method and any shard-related settings in the controller configuration. Make these changes in the manifests or Helm values that your GitOps repository holds, so the change is reviewed and recorded like any other platform change. Use the field names from the official HA documentation rather than copying settings from older blog posts. The check is that the StatefulSet shows the desired replica count and each replica starts without errors in its logs.
Apply the change and wait for the controllers to settle
Apply the change through your normal review path, then watch the controller pods roll out and become ready. Expect clusters to be reassigned as the controller replicas start, and avoid making other platform changes during this window so you can attribute any sync problem to the sharding change. The check is that all controller pods report ready, the controller logs show each replica starting its shard without repeated restarts, and no application has stalled in a state that persists after reconciliation should have run.
Pin large clusters to dedicated shards
If one or two clusters carry most of the application load, pin them to a specific shard so that automatic distribution does not place them alongside other heavy clusters. Pinning is set on the cluster registration Secret, using the shard field described in the official HA documentation. Pin only the clusters that your inventory identified as heavy, and keep the rest on automatic distribution so the remaining replicas stay balanced. The check is that the pinned cluster appears under the shard you assigned and that the other shards do not pick it up after a restart.
Verify each shard reconciles its clusters
Confirm that every controller replica owns the clusters you expected and that applications on each cluster are syncing. Compare the cluster ownership reported by the Argo CD tooling for each shard with your inventory from the earlier step, and confirm that every destination cluster is owned by exactly one shard. Then check a sample of applications on each cluster for Synced and Healthy status and review controller logs for repeated reconciliation errors. Keep the verification output with the change record, since it is the evidence that the sharded install works as planned.
Signs that one application controller is overloaded
An overloaded controller rarely fails loudly. It falls behind on the clusters it owns while the rest of the fleet looks healthy, so the signals below are best read together. Compare each shard with its peers before changing any setting, because the cause is often one large cluster rather than the total application count.
Applications reconcile late
Applications on one cluster take noticeably longer than expected to move from a new Git commit to a synced state, even though the Git repository and the destination cluster respond normally. This suggests the controller responsible for that cluster is saturated, or that its share of applications has grown beyond what it can process. Check the application's last reconciled time and compare it with applications on other clusters. If only one shard lags, the cluster assignment is the first place to look.
Queues build on one shard
Sync operations or reconciliation work pile up on one controller replica while the other replicas sit mostly idle. Uneven load like this points to a cluster that holds far more resources or applications than its neighbors, which round-robin distribution does not correct on its own. Check how clusters map to shards and how many applications each shard manages. Confirm that the shard assignment matches the cluster sizes you expect before adding replicas.
Memory or CPU pressure on a replica
A controller pod that runs close to its memory or CPU limits, restarts, or gets throttled is a direct sign that it is carrying too much cached state or reconciliation work for its cluster. Restarts also cause a temporary gap in reconciliation for every cluster that replica owns. Check the pod's resource usage over time and its restart history, and confirm the limits you set against the size of the applications it manages.
A cluster stops refreshing
Applications on one destination cluster stop showing fresh status, so drift and health changes go unnoticed while the cluster itself is reachable. This can mean the controller that owns the cluster has stalled, has lost its assignment after a replica change, or is failing on that cluster's credentials. Check the controller logs for the affected cluster and confirm that a single replica is responsible for it, since a cluster without an owner or with two competing owners will not refresh correctly.
Reconciliation timing across shards
Comparing reconciliation timing across shards is the most direct way to separate an overloaded controller from a slow cluster or a slow repository. If every application on one shard is slow while the same applications perform normally on another shard, the controller or its assigned cluster is the likely cause. If slowness follows the repository instead, the repo-server is more likely to be the bottleneck. Record a baseline per shard so that a change in one controller stands out clearly.
Security and change control around shard assignments
Each shard change moves credentials, so treat it as a security change. An application controller replica connects to the destination clusters assigned to it and holds the credentials Argo CD uses to reach them. When you pin a cluster to a different shard or let dynamic distribution rebalance, the cluster's credentials are used by a different replica pod. Before you change an assignment, confirm which service account and RBAC scope Argo CD uses for that cluster, and confirm that the target replica's pod spec, service account and node placement meet the same isolation standard as the replica it replaces. Limit who can edit the application controller Deployment or StatefulSet, the sharding settings and the cluster Secrets that carry the shard label, because those are the paths that change which replica holds which cluster. Argo CD's security documentation states that API requests are authenticated only with JSON Web Tokens, that TLS protects traffic between its components, and that cluster, Git and OAuth2 credentials are redacted from API responses and logs. Those controls protect the credentials in transit and in the UI, but they do not replace RBAC on the cluster and the namespace that hosts Argo CD.
Review the repo-server risk alongside any shard change. Sharding moves application reconciliation work, but the manifests for those applications are still produced by the repo-server, which runs Helm and Kustomize on repository content. Argo CD's security documentation flags untrusted repositories and out-of-tree file access as risks to control. If a shard change brings a new set of applications under one controller, confirm that the repositories they read are approved, that Helm and Kustomize settings do not allow access to files outside the repository, and that repository credentials are scoped to the paths those applications need. Ask your security team to review these inputs in the same change request, not only the replica count.
Record every assignment in Git or another audit trail. Store the shard label on each cluster Secret, the sharding setting and any replica count in version-controlled manifests, and require a reviewed pull request for each change. The pull request should name the cluster, the old and new shard, the reason and the rollback step. Keep the Kubernetes audit log for changes to controller workloads and cluster Secrets, and confirm with your platform team how long those logs are retained and where they are stored. This record lets you answer which replica held a cluster's credentials on a given date, which is the question an auditor or incident responder is most likely to ask.
Limits to confirm before sharding in your environment
Controller sharding changes how work is spread across replicas, but it does not change the platform rules around it. Confirm the install shape, the version you run and the support window first, because each one changes whether sharding is the right fix at all.
HA manifest and node count
The HA manifests use pod anti-affinity rules, so the vendor documentation says they need at least three nodes. Sharding a controller on a two-node cluster will leave replicas unschedulable or co-located, which removes the failover you wanted. Ask the cluster team how many schedulable nodes exist in each target environment, including during node maintenance, and confirm the install uses the HA manifests before you plan replica counts.
IPv6-only cluster limitation
The high availability documentation says IPv6-only clusters are not supported. If your clusters or management network are IPv6-only, sharding will not solve the problem and the deployment design must change first. Ask the network team whether dual-stack or IPv4 reachability exists for the Argo CD control plane, and confirm the current support statement with the vendor docs before you commit to a design.
Patch support window
Only the three most recent minor versions receive patch releases, and a new minor ships about every three months. A shard layout tuned on a minor version that has left the window will need a version change anyway. Ask the platform owner which minor version is in production, the date it was released and when its patch window closes, and schedule the upgrade before tuning work starts.
Behavior after an upgrade
The 2.14 to 3.0 upgrade notes changed RBAC so update and delete apply only to the application itself, logs access needs an explicit permission, and annotation-based resource tracking is the default. Any sharding test run before an upgrade may not reflect controller behavior after it. Ask the platform team to run the sharding plan in a test environment on the target version and compare reconciliation timing before and after the change.
Installed version versus latest patch
Compare the installed Argo CD version with the latest published patch releases on the project's GitHub releases page. A controller running an older patch may carry fixes that affect reconciliation, so tuning against it can mask a known defect. Ask the platform owner for the exact image tag in use, then check the release page and confirm which patch line it belongs to before you make sharding changes.
Whether sharding fixes the repo-server
Sharding distributes application controller work across replicas, but a slow manifest generation problem usually sits in the repo-server, not the controller. Adding controller shards will not speed up Helm or Kustomize rendering, and it can add load on Git and the repo-server. Ask the operations team to check whether the delay appears in repo-server logs or controller reconciliation, and test a repo-server change separately before attributing the issue to sharding.
Questions teams ask before sharding Argo CD
Can our team shard the Argo CD application controller without a partner?
Yes, many platform teams can, provided they have someone who understands how the application controller assigns clusters to replicas and who can test a change before it reaches production. The work itself is mostly configuration: setting the number of controller replicas, choosing a sharding algorithm, and pinning clusters where needed. The harder part is knowing your current load, reading controller metrics correctly, and rehearsing the move without losing reconciliation for a cluster. A partner is most useful when the team lacks that operational history or when the change must be proven in a segmented or regulated environment. Confirm the exact sharding options against the Argo CD high availability documentation for the version you run.
What drives the cost of a controller sharding project?
Cost follows the amount of analysis and testing the environment requires, not the number of YAML lines changed. The main drivers are the number of clusters and applications the controller reconciles, how often those applications change, how the existing HA layout is built, and how much testing your change process demands. Environments with many clusters of uneven size, or with strict change control, take longer to plan and verify. Costs split into one-time work (inventory, design, configuration, test runs, documentation) and recurring work (reviewing shard balance after growth and after upgrades). Ask any vendor for a fixed-scope quote that lists each deliverable, the environments covered and the testing evidence provided. DeployOpen does not publish a price list; a quote follows from the written scope.
How long does planning and rolling out controller sharding take?
The duration depends on the factors that shape the work rather than on a fixed number of days. Planning time grows with the cluster inventory, the number of teams who own applications, and how many environments must be tested before production. Rollout time depends on your change windows, how many clusters need to be pinned or moved, and whether you can run the change in non-production first. Organizations with formal approval steps should count those approvals in the schedule. Ask for a written timeline that names the dependencies, such as access approvals and change board dates, so the plan can be checked against your calendar.
What do we need to provide before sharding work starts?
The work depends on accurate inputs from your side. Expect to provide a cluster inventory listing each registered destination, its approximate application count and its owning team, plus the Argo CD version and installation method in use. You will also need change windows that allow controller restarts, read access to the Argo CD namespace and metrics, and a non-production Argo CD instance or equivalent environment for rehearsal. Someone who can approve changes to the Argo CD configuration must be available during planning and rollout. Make sure the access model follows least privilege, and confirm with your security team how temporary access will be granted and removed.
How do we know sharding is the right fix for our Argo CD?
Sharding addresses controller load, so the first step is to confirm that the controller is the bottleneck. Signs include applications that take a long time to refresh even when the repo-server is healthy, controller CPU or memory that stays near its limits, and reconciliation delays that grow as clusters are added. Before sharding, check whether the real cause is repo-server manifest generation, Git polling, Redis latency or API server load, because those need different fixes. Sharding will not speed up a slow repo-server or a single very large cluster on its own. A written diagnosis that compares metrics against these causes helps you decide before any change is made.
What ongoing work follows an Argo CD upgrade with sharding in place?
Upgrades can change how controllers behave, so sharding configuration should be rechecked after every minor or major release. Read the upgrade notes for the target version, confirm that shard assignments still match your cluster layout, and rerun the reconciliation tests you used at rollout. Argo CD publishes minor releases on a regular cadence and patch releases only for the three most recent minor versions, so plan upgrades to stay within supported versions. Ask a partner whether a retainer covers upgrade testing, shard rebalancing as clusters are added, and documentation updates, and whether those services are included in the fixed-scope quote or priced separately.
Does sharding change our security or compliance posture?
Sharding itself does not change who can access applications, but it changes which controller replica holds credentials for each cluster. Confirm that every replica runs with the same RBAC, network and TLS controls, and that cluster credentials remain redacted in logs and API responses. Record the shard assignments and the change approvals as configuration evidence, since auditors often ask how changes to deployment tooling were made and reviewed. DeployOpen prepares systems and evidence for compliance assessments, but independent auditors and certification bodies issue any audit report or certification.
How DeployOpen scopes and delivers controller sharding
DeployOpen treats application controller sharding as a written, fixed-scope change to your Argo CD installation, not a tuning exercise. We start from your cluster count, application volume and controller behavior, then agree the shard layout, deliverables and timeline in writing before any change is made to your environment.
How we scope
- Inventory clusters, applications and controller replicas to confirm the current load and where it sits.
- Agree the sharding method, the shard count and any clusters pinned to specific shards in writing.
- Confirm the Argo CD version and upgrade path first, since sharding behavior depends on the installed release.
How we build
- Configure the shard assignment and replica settings in a staging copy of your installation before production.
- Test cluster reassignment, failover between controller replicas and controller-to-Redis connectivity under realistic application counts.
- Verify RBAC, TLS and credential handling remain unchanged after the controller topology changes.
How we hand over
- Deliver the manifests or Helm values in your Git repository, with a record of every shard assignment and its rationale.
- Document the overload signals to watch, the steps to rebalance a cluster and the rollback procedure.
- Walk your platform and on-call teams through the runbook and confirm they can run a rebalance without us.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer, scoped separately from the build.
- Revisit shard layout when cluster or application counts change and when new Argo CD minor releases are adopted.
- Ask any partner for the written scope, acceptance criteria and who on their team will run the change before you sign.
