Why an Argo CD application stays OutOfSync after a successful sync
A sync can report success and the application can still return to OutOfSync within moments. The reason is that the live object and the desired manifest differ in fields Argo CD compares, and the difference usually comes from one of four sources: Kubernetes normalizing a field you wrote, a controller or mutating webhook changing the object after it is applied, a Helm template that renders different output on each run, or label ownership that another tool is overwriting. The fix starts with identifying which field differs and who writes it, not with adding a rule that hides the difference.
Act when the status flips back to OutOfSync on every reconciliation cycle, when the same application triggers repeated automated syncs, or when alerts for OutOfSync applications become routine and teams stop reading them. Those patterns also hide real drift: once a noisy field is ignored broadly, an unexpected change to a field that matters can pass unnoticed. Start with the application's diff view and the live object, then check the managed fields and the admission controllers in the destination cluster. Confirm the specifics for your Argo CD version in the Argo CD user documentation and in your own cluster before changing any comparison settings.
Once the cause is known, choose the narrowest correction. Fix a template that renders random values, remove a duplicated label owner, or move a mutated field out of the desired manifest where the cause is in your own configuration. Use an ignore setting only for fields that a controller legitimately owns, and record the reason next to it so reviewers can tell an expected difference from an unresolved one. Ask your platform team to confirm which resource types in your clusters are mutated by admission webhooks or operators before the first sync is treated as final.
Which causes produce permanent drift
Permanent drift means Argo CD keeps comparing the live object with a rendered manifest that never matches, even right after a successful sync. The causes below share one pattern: something other than Git writes or reshapes a field, so the live state and the desired state disagree again on every comparison. Check each cause against the diff before changing any ignore rules.
Fields Kubernetes drops or normalizes
The API server fills in defaults, drops empty values and reformats quantities, so a manifest written as 1000m may come back as 1 and a null field may disappear. Argo CD then sees a difference in a field nobody edited. Compare the live object with the rendered manifest and look for defaulted fields, empty lists or equivalent quantities. Fix the manifest to match the normalized form before adding any ignore rule.
Mutating admission webhooks
Mutating webhooks such as service mesh sidecar injectors, policy engines and cost tools add containers, labels, annotations or environment variables when objects are created. Those fields are absent from Git, so every comparison reports them as extra. Run kubectl get mutatingwebhookconfigurations and check which resources each one matches. Confirm with the webhook owner which fields it writes before deciding whether to ignore them.
Controllers that write back to resources
Some controllers update the objects they manage after creation, including operators that set status-like spec fields, cert managers that annotate Secrets and autoscalers that edit workload fields. Argo CD sees the controller's writes as drift, and a sync can revert them and start a loop. Compare the managedFields entries on the live object to see which manager touched each field. Decide whether Git or the controller owns the field.
Random and time-based Helm values
Helm templates that call randAlphaNum, genPrivateKey or now produce a different output on each render, so every refresh generates a new desired state. Generated passwords and timestamp annotations are the usual cases. Render the chart twice with helm template and diff the output to confirm. Move generated secrets to a secret manager or a lookup-based pattern, and remove timestamps from rendered metadata.
HPA-managed replica counts
When a HorizontalPodAutoscaler controls a Deployment, the live replica count changes with load while Git still declares a fixed number. Argo CD reports the Deployment as OutOfSync even though the workload is healthy. Check whether the HPA exists in the cluster and whether the manifest sets spec.replicas. The usual fix is to omit replicas from the Git manifest and let the HPA own that field.
Instance label clashes and status differences
Argo CD tracks managed resources with an instance label or annotation, and another tool such as Helm, Kustomize or a second Argo CD may write the same key. Ownership then flips between tools, and status fields that Argo CD compares can differ from the desired state. Inspect the argocd.argoproj.io/instance label and the tracking method configured in argocd-cm. Confirm which tool owns each key before changing the tracking method.
How to diagnose by symptom in the diff view
When the same fields show as different after every sync, the cluster is almost always rewriting or regenerating something on each reconciliation. The diff is usually stable in shape but returns to the same state within seconds of a successful sync. Start by comparing the rendered manifest in Git with the live object, not the diff summary, because the summary hides which side changed.
- Server-side defaults: Kubernetes fills in default values that the desired manifest omits, such as probe timeouts or protocol fields on a Service port.
- Random or time-based values: Helm templates that call
randAlphaNum,nowor generate a new certificate on each render produce a different desired state every time. - Mutating admission: A webhook may inject labels, sidecars or resource limits after the object is accepted.
- Controller rewrites: A controller may rewrite a spec field it owns, which Argo CD then sees as drift.
Question to answer before choosing a fix: does the live object change between syncs, or does Argo CD render a different desired state each time? Only the first points to a controller or webhook.
When the diff touches a short list of fields and everything else matches, the cause is usually a field that Kubernetes normalizes, drops or manages through another mechanism. The pattern is narrow and repeatable, so the field path in the diff is your main evidence. Compare the field in Git, the field in the live object, and the managed fields entry that shows which manager last wrote it.
- Normalized values: Quantities such as memory units, or ordering of list entries, may be stored in a different form than the manifest.
- Dropped fields: Fields that the API server accepts but does not persist, or that a CRD schema prunes, never match the desired state.
- Other field managers: Another tool or operator may own a field that the Git manifest also declares.
- Status-derived values: Some fields are written by a controller to status or metadata and then compared against Git.
Question to answer before choosing a fix: who owns this field in the live object? If a second manager owns it, the right fix may be a documented ignore rule rather than a change to the manifest.
When drift appears only after a HorizontalPodAutoscaler adjusts replica counts, or after a workload's resources change under load, the autoscaler is the first suspect. The desired state in Git still contains a fixed replica count, and the live object reflects the controller's decision. Confirm the pattern by checking whether the drifted field matches the autoscaler's current target.
- Replica count: A
spec.replicasvalue in Git conflicts with the value the HPA sets at runtime. - Reordering: Some lists in the live object may be reordered by the API server or a controller, which shows as a changed sequence.
- Timing: The OutOfSync state may appear during scale events and clear later, which points to a transient rather than a permanent cause.
- Owner references: Check whether the replicas field is claimed by the HPA through managed fields.
Question to answer before choosing a fix: should Git control this field at all? If the autoscaler owns replicas, removing the field from the manifest is usually cleaner than masking it with an ignore rule.
When applications that synced cleanly start showing OutOfSync after a new operator, policy engine, service mesh or monitoring tool is installed, a new mutating or labeling component is the likely cause. Establish the start time, then compare it with the install or upgrade date of the tool. Also check whether the tool writes labels or annotations that clash with how Argo CD tracks resources.
- Injected fields: Sidecar injectors, policy mutators and cost tools add labels, annotations or containers to objects after they are accepted.
- Tracking clashes: Argo CD tracks resources with its own instance label or annotation, and another tool using the same key can confuse ownership. Annotation-based tracking is the default in the 3.0 line, which lowers this risk compared with label-based tracking in earlier versions.
- Admission policies: Check the cluster's webhook configurations for objects that mutate Deployments or Pods.
- Version changes: Confirm the Argo CD version and the tool version, since upgrades can change defaults.
Question to answer before choosing a fix: which tool started writing to this resource, and is that write intended? If it is, the fix belongs in ignore rules or in the tool's configuration, not in the application manifest.
Confirm the cause before you change anything
This sequence separates real configuration drift from differences caused by the cluster, admission controllers, templating or other tools. You finish with a named cause and a clear decision on whether the fix belongs in the manifest or in a comparison exception.
Reproduce the drift with a manual refresh
Start by forcing Argo CD to re-read the Git revision and compare live state against it. Use a hard refresh in the UI or CLI on the affected application, then open the diff view and note which resource kinds show as OutOfSync. Confirm the result is reproducible by running the refresh a second time with no Git commit in between. If the status returns to Synced after one refresh, the problem was a stale cache and not permanent drift. Record the application name, the resource kinds and the revision you compared.
Capture the exact differing fields
For each OutOfSync resource, export the desired manifest from Git and the live object from the cluster, then compare them field by field. Focus on fields the API server fills in, such as defaults, status blocks, managed fields and generated values, as well as fields that other controllers rewrite. Write down the JSON paths that differ. The step is complete when you have a short list of paths, each tied to one resource and one direction (Git says one value, the cluster holds another).
Check the live object for webhooks and controller owners
Read the live object's metadata for owner references and managed fields to see which controller or manager last wrote each differing path. List the mutating admission webhooks in the cluster and check whether any match the resource kind and namespace. A webhook or operator that injects sidecars, labels, resource limits or defaults will show up as a different owner. Confirm the source by disabling or excluding the webhook in a test namespace, or by checking its configuration with the platform team, and watch whether the diff disappears. Ask the platform owner which controllers are allowed to change these fields.
Re-render the chart twice to test determinism
If the application uses Helm, render the chart locally or through the repo-server twice with identical inputs and compare the two outputs. Any value that changes between runs, such as a random password, a generated certificate, a timestamp or a lookup result, will cause permanent drift because Argo CD compares against a freshly rendered manifest. Confirm the rendering is deterministic when the two outputs match byte for byte. If they differ, move the generated value into a Secret managed outside the chart or pin it in values.
Check HPA-managed replica fields
Look at whether the application defines a replica count in Git on a Deployment or StatefulSet that also has a HorizontalPodAutoscaler targeting it. When the autoscaler changes replicas at runtime, the live value differs from Git on every comparison. Confirm this by checking the HPA status and comparing its current replica count with the value in the manifest. If the HPA owns the field, the usual fix is to remove the replica count from the manifest so the autoscaler is the only writer, which you can verify by seeing the diff clear after the next sync.
Check label and annotation owners
Compare the labels and annotations Argo CD writes for tracking with the metadata found on the live object. Check which tool added each differing label or annotation, including other GitOps tools, service meshes, policy engines and backup software that write to the same resource. Pay attention to the tracking method configured for the instance, because clashes between instance labels and other tools' labels can make Argo CD lose track of ownership. Confirm the owner by identifying the manager named in the metadata and by confirming the value stops changing when that tool is excluded from the resource.
Choose a manifest change or a comparison exception
With the cause named, decide where the fix belongs. If Git is wrong, such as a generated value, a stale replica count or a missing default, change the manifest and confirm the next comparison shows Synced. If another system legitimately owns the field, add a narrow comparison exception scoped to that application and path, and document the owner. Avoid broad exceptions that could hide real drift. The step closes when the change is committed, the application reads Synced after a refresh, and the reason for the decision is written in the change record.
Which fix fits which cause
This table matches each common cause of permanent drift to the evidence that points to it, the change to make in Git, and the comparison-level setting that may be needed. Read it from the diff: find the field named in the OutOfSync view, then match its owner before you choose a fix. The main takeaway is that the fix belongs where the field is written, and ignoring a field is the last step, not the first.
| Cause | Evidence in the diff | Fix in Git | Comparison-level fix |
|---|---|---|---|
| Normalized fields | Live object has defaults or reformatted values, such as empty fields the API server fills in | Remove the redundant field from the manifest or set the value Kubernetes stores | Ignore only the named field if Kubernetes always rewrites it |
| Webhook-injected fields | Sidecars, labels or annotations added by admission webhooks not present in Git | Stop expecting the webhook field in Git, or move the injection setting to the platform layer | Ignore the injected path, scoped to the affected resource kind |
| Non-deterministic Helm values | Generated passwords, timestamps or random suffixes change on every render | Pin the value in values files or read it from an existing Secret | Rarely needed, since the template should be deterministic |
| HPA-managed replicas | Deployment replica count differs from Git after autoscaling changes it | Remove spec.replicas from the manifest when an HPA owns scaling | Ignore the replicas field only for workloads with an active HPA |
| Instance label clashes | Tool-owned labels or the app.kubernetes.io/instance label conflict with another tool's tracking | Use annotation-based tracking and avoid reusing the instance label across tools | Set the tracking method for the instance, after confirming the default in your version |
| Broad ignore rules | A wide ignore pattern hides a change that should trigger a sync | Keep Git as the source of truth for fields you control | Narrow the rule to a specific path and resource kind; see the sibling topic on ignoreDifferences |
Why instance labels cause OutOfSync and how tracking mode changes it
Argo CD decides which live objects belong to an application by stamping them with a tracking identifier. When that identifier is written into a label, any other tool that writes the same label key, such as a Helm chart, an operator or a cluster policy engine, can overwrite or duplicate it. Argo CD then sees a resource whose ownership is ambiguous, and the application can report OutOfSync even though the rendered manifests in Git have not changed. The first check is therefore whether the conflicting key is the instance label Argo CD uses. Look at the live object's labels and annotations with kubectl get ... -o yaml, and compare the key against the application.instanceLabelKey setting in argocd-cm if your installation overrides it.
Since Argo CD 3.0, annotation-based resource tracking is the default, and the 2.14 to 3.0 upgrade notes describe this change. Teams that upgraded from older releases, or that set label tracking explicitly, may still run in label mode, and a mixed fleet of applications can behave differently from one cluster to the next. Before you change any label key or tracking setting, confirm the active mode in the configuration of your installed version and read the tracking section of the documentation for that release. Changing the mode on a running installation can cause Argo CD to re-evaluate ownership of many resources at once, so plan it as a controlled change with a rollback path and a test on a non-production application first.
Once you know which mode is active, decide where the conflict should be resolved. Renaming the tracking key is a platform-wide decision that affects every application, while excluding one field or label from comparison is narrower but can hide real drift if it is applied too broadly. Ask the owners of the other tool whether their labels can move to a different key, and document the choice in the application's repository so reviewers know why the setting exists. Confirm the exact keys and defaults in the documentation for your version, since these details have changed across releases.
Security and operational risks of masking drift
Suppressing a diff is a decision about what Git no longer describes. The cards below cover the places where that decision can hide a privilege change, a security context edit or an unreviewed manual change, and the controls that keep those changes visible in Git review and on the repo-server.
Ignore rules that hide privilege changes
An ignoreDifferences entry written for a noisy status field can also cover a field that grants access, such as a container's securityContext or a role's rules. Once the rule is in place, Argo CD reports the application as synced even when the live object has been widened. Before approving a rule, list the exact JSON pointers it covers and confirm that none of them hold privilege or image settings. Review these rules in Git like any other change.
Webhooks that alter security context
A mutating admission webhook can set fields such as runAsNonRoot, capabilities or image pull policy after Argo CD applies the manifest. Argo CD then sees a permanent difference, and teams often answer by ignoring it. The result is that the security context in the cluster no longer matches what Git reviewers approved. Ask the platform team to list every mutating webhook, the fields each one writes and whether the policy belongs in Git or in the admission layer.
Helm templates that rotate secrets
Templates that call randAlphaNum, genPrivateKey or a timestamp function produce different output on every render. Argo CD compares each render with the live object, so the application never settles, and a reviewer cannot tell whether a change came from Git or from a new random value. Fix the template so values come from a stored secret, a chart value or an external secret operator, and keep the repo-server render deterministic.
RBAC scope over live state
Argo CD's RBAC decides who can sync, override or delete applications, and who can read logs. If broad sync or override rights are granted to a group, people can change live state outside Git and the diff will show it only as drift. Check which roles can run manual syncs and which can use parameter overrides, and restrict them to the team that owns each project. Confirm the mapping in your SSO group configuration before go-live.
Repo-server exposure to repository content
The repo-server runs Helm and Kustomize on whatever content it reads, so a repository that is not reviewed, or a template that reads files outside its tree, can change what gets rendered. Pair Git review with restricted repository access, pinned chart versions and limits on which sources a project may use. Treat the rendered manifest as the artifact to review, and check it when a diff appears that no Git commit explains.
Audit gaps from overwritten edits
When self-heal is enabled, Argo CD reverts manual edits to the live state, so a hotfix made with kubectl may disappear without a record in Git. Teams then cannot reconstruct why a setting changed or who approved it. Decide which namespaces allow self-heal, require that emergency changes be committed to Git within a set window, and retain Argo CD's event history in your logging system.
Questions teams ask before fixing OutOfSync drift
How much effort does diagnosing OutOfSync drift usually take?
The effort depends on how many differences you are chasing, not on Argo CD itself. A single application with one or two drifting fields can be diagnosed in a focused session, while a fleet with many clusters, custom resources and mutating admission webhooks takes longer because each source of change must be traced separately. The main drivers are the number of clusters and applications, the count of mutating webhooks and controllers that edit objects after apply, the number of custom resources without well-understood defaults, and how many Helm charts use templates that render differently on each run. Ask for a quote that lists these counts for your environment and states what is included, such as a written list of root causes and recommended fixes.
Can our own platform team diagnose this without outside help?
Yes, many teams can, provided they have someone who understands Kubernetes controllers, Git history and the Argo CD application model. The work is mostly systematic: compare the live object with the desired manifest, identify which fields the API server defaults or normalizes, find which controller or webhook changes them, and decide whether the difference is expected. Teams that struggle usually lack cluster read access, cannot see admission configuration, or have no one who owns the charts. An outside review makes sense when that ownership is missing, when the drift spans many teams, or when you need a documented record of each decision for change management.
What do you need from our team to start a diagnosis?
Plan on providing read access to the affected clusters, the relevant Argo CD application definitions and their Git repositories with enough history to see when drift began, and the Argo CD version in use. Argo CD version matters because resource tracking and RBAC behavior changed in the 2.14 to 3.0 upgrade, and the upgrade notes list those changes. You should also name an administrator who can answer questions about installed webhooks, controllers and Helm chart ownership, and who can approve changes to ignore rules or manifests. Confirm exact access scopes with the engagement lead before work starts, since read-only cluster access is usually sufficient for diagnosis.
Is it safe to ignore a difference that Argo CD keeps reporting?
It can be safe, but only when you have proven that the field is changed by a known controller or by the API server, and that the field does not affect how the workload runs. Ignoring a difference hides it from future comparisons, so a field that is later changed by a person or a bad release will not raise an alert. Keep ignore rules narrow, scope them to specific applications where possible, and record the reason next to each rule. Before adding one, ask the owner of the field whether it carries configuration that should be in Git, because some drift is a sign that the manifest is incomplete.
How do we tell real drift from harmless noise?
Real drift is a change to the live object that does not match the desired state in Git and that alters behavior, such as a replica count changed by hand or a container image set outside the pipeline. Noise is a difference created by defaults, normalization, an external controller or a template that renders different values each time. A practical test is to check whether re-rendering the manifest from Git produces the same output twice, whether the field appears in the live object without being set in Git, and whether the changing value affects runtime behavior. Document each classification so the next person does not repeat the investigation.
What does a fixed-scope drift review cover, and when is it worth buying?
A fixed-scope review is worth buying when the drift is spread across several applications or clusters, when the cause is unclear after an internal attempt, or when auditors or change managers need written evidence of each decision. DeployOpen agrees the scope, deliverables and timeline in writing before work starts, so the review can be limited to named applications and clusters with a defined set of outputs, such as a root cause per application, the ignore rules or manifest changes recommended, and the operational checks to run afterward. Ask for the deliverables list and the access you will need to grant as part of the quote.
Which costs recur after the drift is fixed?
Most of the cost is one-time, covering diagnosis, manifest and chart corrections, and documentation. Recurring costs appear when drift returns, when new webhooks or chart versions introduce fresh differences, or when Argo CD upgrades change how fields are compared. Ongoing support or periodic review can be arranged as a retainer, which DeployOpen offers. Ask any provider to separate one-time diagnosis from recurring support, and to state how they will handle drift introduced by future chart upgrades.
How DeployOpen scopes an OutOfSync investigation
An OutOfSync investigation starts with a written scope, not a log dump. Before any work begins, the scope names the affected Argo CD applications, the destination clusters they deploy to, the Argo CD version in use and the namespaces or AppProjects involved. The version matters because comparison behavior, resource tracking and RBAC changed across releases, so a fix that is correct on one minor version may not hold on another. The written scope also records what is out of bounds, such as changes to production cluster admission policies, so that any fix proposed later is weighed against an agreed boundary.
Diagnosis then runs inside your own infrastructure. DeployOpen works in the cluster and Argo CD instance you already operate, using read access to the Application resources, the live objects they point to and the mutating admission webhooks in the path. The team compares the rendered manifests from the repo-server against the live state, identifies which fields are dropped, normalized or rewritten by the API server or a controller, and records each finding against the exact application and resource. The output is a documented fix list. Each item states the cause, the proposed change (a targeted ignore rule, a template correction, a label or tracking change, or a controller setting), the risk of that change and the check that proves the application reconciles to Synced with no new diff. Changes are made by your team or by DeployOpen under the agreed change process.
The engagement closes with handover notes that a platform team can use without DeployOpen in the room. They cover the root cause for each affected application, the configuration changes made and why, the verification steps and the recurring checks worth keeping, such as watching for OutOfSync counts after upgrades or new admission tooling. Ongoing support, upgrade testing and continuous review of Argo CD are available as a retainer if your team wants a partner to keep watching drift after the fix. Confirm with DeployOpen during scoping which items are fixed-scope and which would move to a retainer.
How DeployOpen delivers Argo CD operations work
DeployOpen approaches permanent OutOfSync状态 as a diagnosis problem first: we identify which fields, controllers or tools keep changing the live objects before changing any manifest. The work runs inside your own clusters and Git repositories, with the findings and fixes documented for your team to keep.
How we scope
- We agree in writing which applications, clusters and repositories the drift investigation covers before work starts.
- The written scope lists the drift sources to examine, such as mutating webhooks, Helm template rendering and controller-managed fields.
- Engagements are fixed-scope, and ongoing support on Argo CD is available as a retainer if your team wants continued help.
How we build
- We reproduce the drift on a non-production application and record the exact diff Argo CD reports against the live object.
- We apply the narrowest correction, such as a targeted ignore rule or a template change, and test it against a sync and a re-sync.
- Each change is reviewed through your Git workflow so the configuration history stays auditable for your team.
How we hand over
- You receive a written record of each drift source, its cause and the chosen fix, with the reasoning behind it.
- We document the Argo CD settings changed, the affected applications and the checks your team can run to confirm stability.
- Handover includes a walkthrough so your platform engineers can recognize the same patterns in future applications.
How we support
- Ongoing support covers new drift reports, Argo CD upgrades and reviews of ignore rules as applications change.
- Continuous testing and upgrades are available as retainers, scoped in writing with your team.
- Ask us to confirm which Argo CD versions and controller behaviors our support covers before you sign a retainer.
