Running Argo CD with a thousand or more applications starts with knowing which component is the bottleneck
At roughly 1,000 applications, the settings that matter most are the application controller's status and operation processors, the repo-server's timeouts and capacity, and how the controller shares application tree data with Redis. Most platform teams should tune these before they consider splitting into shards or separate Argo CD instances. The Argo CD high availability guide suggests --status-processors of 50 and --operation-processors of 25 for about 1,000 applications, up from the defaults of 20 and 10. Treat those figures as starting guidance from the high availability documentation, not as a guarantee. The right values depend on how many clusters the controller watches and how expensive each application's manifests are to render.
The three components scale differently, so a slow Argo CD is rarely slow in one place. The application controller compares live cluster state with Git and carries most of the reconciliation work, so its processor counts and its sharding decide how quickly applications refresh. The repo-server renders Helm and Kustomize output, so its cost follows manifest complexity and Git fetch frequency, not application count alone. Redis holds the cached application tree, so its traffic grows with the number of resources each application contains. Before changing any value, identify which component is busy by checking controller and repo-server logs and metrics, and by noting which applications are slow to refresh. A change that helps the controller will do nothing if the repo-server is the limit.
Plan the split decision before you need it. Record your current application count, cluster count, largest applications by resource count, and the refresh interval you need, then tune one component at a time and measure the effect. Ask your vendor or internal Argo CD owners to confirm which settings are supported for your installed version, since defaults and flags change between releases. If tuning in one instance no longer meets your refresh targets, the next step is controller sharding or a second instance, and those choices carry their own operational cost that the sibling topics cover in detail.
Which setting to change based on the symptom
When applications take a long time to refresh or syncs wait in a growing queue, the application controller is usually the bottleneck. The controller runs two worker pools: status processors, which compare live state with desired state, and operation processors, which run syncs. The defaults are 20 status processors and 10 operation processors. The HA guide suggests 50 and 25 for about 1,000 applications, so that is the starting point for this range.
- Status processors:
--status-processorson the application controller sets the number of concurrent status reconciliations. Raise it when refreshes lag behind the desired refresh interval. - Operation processors:
--operation-processorssets concurrent sync operations. Raise it when many applications sit in a pending or progressing state while the cluster itself looks healthy. - Check CPU and memory first: Confirm the controller pod has headroom. More processors on an undersized pod mostly add throttling and memory pressure.
- Check cluster API latency: If the destination clusters' API servers are slow, more workers only increase the number of requests waiting on them.
- Change one value at a time: Measure the queue depth and refresh times after each change so you can tell which pool was limiting.
Before changing either value, confirm the controller is the component that is slow, not the repo-server or a single destination cluster.
A reconciliation that fails with Context deadline exceeded means a call did not finish within its time limit. Often the controller is waiting on the repo-server to generate manifests, and the repo-server is saturated by many applications asking at once. The HA guide suggests two responses: raise the timeout and scale the repo-server. Raising the timeout alone only hides the wait.
- Repo-server timeout:
--repo-server-timeout-secondson the application controller sets how long it waits for the repo-server. Increase it in measured steps if manifest generation is legitimately slow. - Repo-server replicas: Add replicas when the repo-server CPU stays high during refresh bursts. This spreads manifest generation across pods.
- Repo-server load: Look at how many applications share one repository, since each refresh can trigger a generation. Large Helm or Kustomize trees are a common cause.
- Git and network path: Slow Git hosts or egress limits can produce the same error, so check repository fetch times before scaling.
- Controller status: Confirm whether the error appears on every application or only on the ones that use a specific repository or tool.
Before raising the timeout, check repo-server CPU, memory and generation duration. A higher timeout with an unchanged repo-server usually makes the queue longer, not faster.
Redis holds the application tree that the controller reads and writes during reconciliation. As the number of applications grows, each controller pass can move a large amount of data between the controller and Redis, which shows up as high network use and slow cache writes. The HA guide describes application tree sharding as the way to reduce this traffic. It is controlled by the environment variable ARGOCD_APPLICATION_TREE_SHARD_SIZE, and the guide suggests a value of 100 as a starting point.
- Shard size variable: Set
ARGOCD_APPLICATION_TREE_SHARD_SIZEon the application controller. Smaller shards store the tree in more pieces, so each write is smaller. - Redis resources: Check Redis memory and network throughput before and after the change. Sharding reduces per-write traffic but does not add Redis capacity.
- Redis HA layout: If Redis is a single instance with high load, review the Redis layout separately, since Redis is a disposable cache and its availability affects sync speed.
- Controller restarts: Expect the cache to rebuild after a change, so schedule the change in a quiet window.
- Pod sharding: Traffic from many clusters may need controller sharding instead, which is a separate decision covered elsewhere.
Before changing the shard size, measure controller-to-Redis traffic and Redis CPU so you have a baseline to compare against.
Most large-scale problems come from changing a setting that does not match the bottleneck. A short diagnostic pass prevents this. Start by collecting controller, repo-server and Redis metrics for a normal day and for a refresh burst, then compare them with the symptom. The HA guide's starting values are suggestions for a deployment of about 1,000 applications, not limits, so the right value depends on your clusters, repositories and hardware.
- Collect baselines: Record queue depth, refresh durations, repo-server generation time and Redis traffic before any change.
- Change one component at a time: Adjust processors, timeouts, replicas or shard size separately so the effect is attributable.
- Confirm resource headroom: Make sure each pod and Redis instance has CPU and memory to spare before you add concurrency.
- Record each change: Keep the values in version control with the reason, so the next operator can reverse them.
- Plan the next step: If the controller stays saturated after tuning, the outline's sibling topics cover controller sharding and separate instances.
After each change, verify that the symptom has eased and that no other component has become the new bottleneck.
Configuration options for large Argo CD installations
This table lists the controller, repo-server and Redis settings that most often need changing as application counts grow, with the point at which each one fits and what it costs elsewhere in the installation. Read each row as a starting adjustment to test in your own environment, not a fixed target. The main takeaway is that raising one limit usually moves load to another component, so change one setting at a time and watch the effect.
| Setting | When it fits | Trade-off |
|---|---|---|
| Status processors (default 20, HA guide suggests 50 for about 1,000 applications) | Application status refreshes lag behind actual cluster state, and the controller queue grows across many applications | More concurrent reconciliation raises CPU and memory use on the application controller and more calls to the Kubernetes API servers it watches |
| Operation processors (default 10, HA guide suggests 25 for about 1,000 applications) | Syncs wait in a queue before starting, even when the cluster and repo-server are healthy | More simultaneous syncs increase repo-server manifest generation requests and load on target clusters during rollouts |
| --repo-server-timeout-seconds | Reconciliation fails with Context deadline exceeded while the repo-server is still generating manifests | A higher timeout hides slow manifest generation for longer and can tie up controller workers while a request waits |
| Repo-server replica count | Manifest generation is slow or times out after you raise the timeout, and large Helm or Kustomize renders dominate the work | Each replica adds CPU and memory cost and cache duplication, and it does not help if one repository is the bottleneck |
| ARGOCD_APPLICATION_TREE_SHARD_SIZE (suggested 100) | Redis and controller-to-Redis traffic are high because the application tree is cached in large single entries | Smaller shards mean more Redis keys and more round trips, so the gain depends on your installation and should be measured |
| Controller sharding (see sibling topics) | One application controller replica is overloaded even after tuning processors and timeouts | Adds replicas and cluster assignment work, so it is a separate design decision covered in the sharding topic |
How to tune a large Argo CD installation step by step
This sequence moves a large Argo CD installation from guesswork to measured tuning, changing one control at a time so each effect on reconciliation, repo-server load and Redis traffic can be seen and compared.
Measure applications, sync behavior and error types
Start by recording the total application count, the number of clusters and repositories, and how many applications are OutOfSync, Progressing or failing at any moment. Note the refresh and sync latency you see in the UI and CLI, then collect the exact error strings from the application controller and repo-server logs, including whether Context deadline exceeded appears and how often. Write these figures down with the Argo CD version and the current controller, repo-server and Redis replica counts. This baseline is the reference for every later comparison, so it must be captured before any setting changes. Done when a dated record exists that someone else on the team could reproduce.
Confirm the current controller processor settings
Check the status processors and operation processors currently set on the application controller, which default to 20 and 10. Your platform team should inspect the live deployment or ConfigMap rather than rely on the install manifest, since earlier changes may have been made by hand or through Helm values. Record both values and whether any sharding setting is already in place. Done when the live values are written next to the baseline from the previous step.
Raise application controller processors
For an installation of about 1,000 applications, the high availability guide suggests 50 status processors and 25 operation processors. Apply these values in one change, controlled through your normal GitOps or Helm workflow, and restart the controller if your deployment requires it. Watch the controller CPU and memory after the change, since more processors add load on the controller. Done when the new values are live, the controller is healthy, and the post-change metrics are recorded beside the baseline for comparison.
Adjust repo-server timeout and replicas for deadline errors
Only run this step if reconciliation logs show Context deadline exceeded. The high availability guide suggests raising --repo-server-timeout-seconds and scaling up the repo-server. Increase the timeout first, then add repo-server replicas if manifest generation still stalls, because a longer timeout alone hides capacity limits. Confirm the flag is applied to the controller. Done when the deadline errors have stopped or fallen, with a before and after count of each error type and the timeout and replica values recorded.
Set the application tree shard size
Set ARGOCD_APPLICATION_TREE_SHARD_SIZE on the application controller, where the guide suggests 100 as a starting value. This setting groups application resource trees into smaller shards in Redis, which reduces the volume of traffic between the controller and Redis. Apply it as a single change so its effect is isolated. Done when the controller restarts cleanly and the Redis connection traffic and controller logs have been sampled and recorded against the earlier readings.
Verify with controller logs and sync latency
Compare the current state with the baseline. Check controller logs for the error types recorded in step one, confirm whether Context deadline exceeded has stopped, and measure refresh and sync latency for a representative set of applications. Make sure the sampling method matches the baseline so the comparison is fair. If a symptom persists, change only one control in the next iteration. Done when each recorded metric has a before and after value and the team agrees the installation responds within its operational expectations.
Decide when to shard or split instances
If the verified installation still shows a saturated controller, review the sharding and instance options with the baseline data in hand. Consider controller sharding when a single controller is the limit, and consider separate Argo CD instances when ownership, network segmentation or failure isolation requires it. Record the trigger, the metric that crossed its threshold and the chosen option. Done when the decision and its evidence are documented so the next tuning cycle starts from the same record.
When to split into shards or separate Argo CD instances
Tuning one Argo CD instance has limits, and past those limits the answer is structural: spread the application controller across shards, or run separate Argo CD instances. Both are decisions with operational cost. Every additional controller replica, Redis layout or instance is another thing to upgrade, back up, monitor and give access to, so the reader should only move to them once the tuning settings in this topic have been tried and the symptoms persist. Before choosing, confirm the signs of an overloaded controller in your own environment: reconciliation queues that keep growing, applications that stay Progressing or OutOfSync for longer than your normal sync cycle, rising controller memory, and status updates that lag behind cluster changes. Measure these over a representative period, not during a single deployment window.
The boundaries you already have should drive the choice more than the raw application count. Sharding spreads the work of managing clusters across controller replicas but keeps one API server, one set of projects and one RBAC model, so it suits a single organization with one governance model. Separate instances make sense when teams need different administrators, different SSO configurations, different change-control rules or a smaller blast radius if an instance is misconfigured, compromised or needs an upgrade. Ask which team owns each cluster, which regulator or audit scope applies to it, and how long an outage of one Argo CD would affect other teams. The build details for controller sharding are in the sibling topic on application controller sharding, and the access model for many teams is covered under multi-cluster and multi-tenancy.
Security and operational risks when scaling Argo CD
Scaling Argo CD to a thousand or more applications raises the load on components that handle untrusted content, cache state and cluster credentials. The cards below cover the risks that come with each tuning choice, so that capacity changes are reviewed with the same care as the configuration they affect.
Repo-server runs untrusted manifests
The repo-server runs Helm and Kustomize against repository content, so it is the component where a hostile or careless repository does the most damage. Raising its replicas, parallelism or timeouts to clear Context deadline exceeded errors also lets a single repository consume more CPU and memory before it is stopped. Before raising those limits, confirm which repositories and Helm chart sources can reach the controller, and restrict out-of-tree file access.
Redis loss triggers refresh
Redis holds a cache, not the source of truth, so losing it causes the controller to recompute state rather than losing applications or configuration. The cost is a burst of refresh load on the controller and repo-server at the moment the cache is rebuilt. Plan for that burst when sizing replicas, and check that Redis HA is sized for the traffic that application tree sharding is meant to reduce.
Three-node minimum for HA
The HA manifests rely on pod anti-affinity rules, which need at least three nodes to schedule replicas on separate hosts. A three-node cluster that loses one node still runs the control plane, but scaling replicas beyond the node count leaves pods pending instead of spreading load. Confirm the node count, zone layout and any node taints before adding replicas, and do not count on IPv6-only clusters, which the HA guide does not support.
TLS between Argo CD components
Argo CD uses TLS for traffic between its own components, which matters once controllers, repo-servers and the API server are split across nodes or shards. Certificate expiry or a misconfigured trust chain can break communication between components, which appears as failed reconciliations rather than a clear TLS error. Add certificate renewal and component connectivity checks to the operating runbook for every scaled deployment.
Tuning changes tracked in Git
Settings such as status and operation processors, repo-server timeouts and shard size change how Argo CD behaves under load, and a bad value can slow or stall every application. Keeping these values in the Git repository that manages Argo CD gives reviewers a diff to inspect and a commit to revert. Record which value changed, why, and the load test or incident that justified it.
Questions teams ask before scaling Argo CD to 1,000 applications
What does tuning Argo CD for 1,000 applications cost?
No single price applies because cost follows the written scope, and DeployOpen does not publish a price list. The main drivers are the number of applications, the number of clusters the controller manages, how many repositories and manifest tools (Helm, Kustomize, plain YAML) the repo-server must render, and how often reconciliation currently fails. Costs split into a one-time assessment and tuning effort, plus any retainer for ongoing support or upgrades if you want it. Ask for a quote that lists each item, the environments covered and the evidence you will receive, such as before and after metrics. Argo CD itself is Apache-2.0 licensed, so your costs are infrastructure and services, not software licenses.
How long does Argo CD scaling work usually take?
Duration depends on your environment, so it is set in the written scope rather than assumed. The biggest variables are the size of your Argo CD footprint, whether you have a non-production copy that can be used for load testing, how many change windows you can use for restarts and setting changes, and how quickly your metrics and logs can be accessed. Work that needs a controller sharding redesign or a split into multiple Argo CD instances takes longer than tuning processor counts on one instance. Ask for a timeline with milestones, and ask which steps depend on your change approvals.
What must our team provide before tuning starts?
The team needs a short list of access and inputs before the work begins:
- Administrative access to the Argo CD namespace and the clusters it manages, scoped to the change plan
- Controller, repo-server and Redis metrics, plus controller and repo-server logs covering a period of errors
- A test environment that resembles production in application count and manifest complexity
- Named change windows for restarts and configuration changes, with a rollback plan
- A contact who can approve repository and cluster changes during the engagement
Missing metrics usually slow tuning more than any other gap, because settings should be changed based on observed behavior rather than guesses.
Can our internal platform team tune Argo CD without a partner?
Yes, many teams do this in-house, provided they have the skills to read controller behavior and do it safely. The work requires Kubernetes operations experience, familiarity with Argo CD's controller, repo-server and Redis components, and the ability to interpret metrics and logs. Teams also need a way to test changes before production and a habit of changing one setting at a time. A partner adds value when the internal team lacks capacity, when a deadline depends on the result, or when a cross-team design such as sharding or repository layout is needed. Confirm skill coverage with your own engineers before deciding.
Which Argo CD settings should we check first for Context deadline exceeded errors?
Start by identifying which component returns the error, because the fix differs. The Argo CD high availability guide suggests raising --repo-server-timeout-seconds and scaling up the repo-server when reconciliation fails with Context deadline exceeded. Check repo-server CPU, memory and queue behavior at the same time, and review whether manifest generation is slow because of large Helm or Kustomize workloads. Changes should be applied one at a time in a test environment first, with results recorded. The Argo CD high availability guide lists the relevant settings for your version.
What happens if status or operation processors are set too high?
Raising processors gives the controller more parallel work, but it also increases load on the repo-server, the Kubernetes API of managed clusters and Redis. The default is 20 status processors and 10 operation processors, and the high availability guide suggests 50 and 25 for about 1,000 applications as a starting point, not a fixed answer. If the values are too high, you may see repo-server saturation, slower syncs, more timeouts and higher memory use across components. The safe approach is to raise values in steps, watch the metrics after each step, and roll back if queues or errors grow.
When should we split Argo CD into shards or separate instances?
Split when tuning on one instance no longer keeps reconciliation within your targets, or when one controller is clearly overloaded while others are idle. Sharding spreads clusters across application controller replicas and reduces per-controller load, and ARGOCD_APPLICATION_TREE_SHARD_SIZE (suggested at 100 in the high availability guide) reduces controller-to-Redis traffic. Separate instances add operational overhead, including upgrades, access policies and backups for each one, so they are usually justified by organizational boundaries, security isolation or blast radius. Decide with measured data, and confirm the design against your Argo CD version before building it.
How DeployOpen would scope a large-scale Argo CD tuning engagement
A tuning engagement starts with measurement, not with a list of settings to change. DeployOpen would first record how the existing Argo CD installation behaves at its current size: how long applications take to reconcile, how often operations queue, where Context deadline exceeded errors appear in the application controller and repo-server logs, and how much traffic the controller sends to Redis. The team would also confirm which Argo CD minor version is running, because the supported patch lines and the flags available differ between releases, and the reader should check the Argo CD release process and cadence before planning any change. Baseline numbers from your own environment decide which of the settings below are worth touching.
The written scope then names the specific levers to be evaluated, such as the status and operation processor counts, the repo-server timeout and replica count, the application tree shard size, and whether application controller sharding or a second Argo CD instance is justified. Each lever needs a stated trigger, a change, a test window and a rollback step. The scope also defines the environments to be covered (usually a non-production copy first), the verification to be delivered (before and after measurements, reconciliation timing under load and a failure test for the repo-server), and the handoff documents that record every final value with its reason. Ask for the scope to state which of these items are included, which depend on the client's own change windows, and which need work by the client's platform team.
Once the engagement closes, the configuration is documented so that the team can operate it without DeployOpen. Ongoing tuning as the application count grows, upgrades across minor releases, and continuous testing are available as retainers, and these are priced separately from the fixed-scope work. DeployOpen does not publish a price list. A quote follows from the written scope, so the most useful preparation is a clear statement of application and cluster counts, the Argo CD version in use, the current symptoms and the change windows your organization allows.
How DeployOpen delivers Argo CD scale work
DeployOpen plans and implements Argo CD scale work inside your own infrastructure, starting from the applications, repositories and clusters you run today. Each engagement begins with a written scope, then moves through measured changes, tests and documented handover so your team can operate the result.
How we scope
- Inventory applications, clusters, repositories and sync frequency before choosing any tuning changes.
- Written scope names the Argo CD version, controller, repo-server and Redis settings in play.
- Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts.
How we build
- Adjust status and operation processors, then observe controller behavior under your real load.
- Tune repo-server timeouts and replicas where reconciliation shows Context deadline exceeded errors.
- Plan application tree sharding and controller shard boundaries, testing each change in a non-production environment first.
How we hand over
- Deliver a documented configuration record with each setting, its reason and its rollback path.
- Walk your platform team through dashboards and the signals that show a controller or repo-server is overloaded.
- Record the trigger conditions that would justify splitting into shards or separate Argo CD instances.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer after handover.
- Retainers can cover review of Argo CD upgrades before each supported minor release is adopted.
- Ask for a quote that separates one-time scope work from recurring support and testing effort.
