Alerting on Argo CD sync failures and degraded applications starts with three metrics
Argo CD exposes the signals you need to alert on drift and failed deployments: argocd_app_info, which carries sync_status and health_status labels for each application, argocd_app_reconcile, which records how long reconciliation takes, and argocd_cluster_connection_status, which reports whether Argo CD can reach each registered cluster. If your teams learn about a failed sync or a degraded workload from users or from a manual check, a GitOps platform that deploys quietly has a visibility gap, and these three metrics are the place to close it. Alerting is most valuable once several teams or clusters depend on Argo CD, because nobody watches every application's status page all day.
Alerts on these metrics do not replace the Argo CD web UI, the argocd CLI or the audit trail. Dashboards help people investigate after an alert fires, and audit logs record who changed what, which an alert on sync status cannot tell you. Before you build rules, confirm with your own environment which metric names and label sets your Argo CD version exposes, since the metrics guide is the authority for names and labels. Also decide who owns the response: an alert that fires into a shared channel without a named owner per project tends to be ignored, so plan routing by project alongside the rules themselves.
Treat the alerting design as a short list of decisions. Choose which applications are critical enough to page someone and which only warrant a ticket. Decide how long a non-healthy state may persist before it counts as a problem, so that normal rollouts do not generate noise. Confirm that the Prometheus instance scraping Argo CD is itself monitored, because an alert pipeline that silently stops is the failure you most want to avoid. The sections that follow turn each metric into a concrete rule.
Which Argo CD signal fits which alert
This table maps each Argo CD metric to the alert it supports, the failure it catches and the noise risk to confirm before paging. Read it as a starting point: sync and health labels catch state, reconcile durations catch slowness, and cluster connection status catches lost reach to a destination.
| Signal | Alert it supports | Failure it catches | Noise or false positive risk to confirm |
|---|---|---|---|
| argocd_app_info sync_status | Applications stuck OutOfSync for longer than a team's agreed window | Drift from Git, failed automated sync, or a change made outside Git | Expected drift from mutating webhooks or controller-managed fields, and short OutOfSync windows during normal rollouts |
| argocd_app_info health_status | Applications Degraded, or Progressing for too long | Failed pods, crash loops, unhealthy custom resources | Progressing during healthy rollouts, and custom resources with no health check or flapping Lua checks |
| Combined sync and health labels | Page only when an application is OutOfSync and Degraded together | Real delivery breakage, not transient drift | Lower noise, but may miss a Degraded application that is Synced to a broken revision, so confirm which combination matters |
| argocd_app_reconcile durations | Slow reconciliation for a project or repo-server | Repo-server saturation, slow Git fetches, or manifest generation timeouts | Large applications and monorepos are naturally slower, so set thresholds per application group from a baseline |
| argocd_cluster_connection_status | Cluster unreachable from the Argo CD instance | Expired credentials, network path changes, or API server outages on the destination | Brief network blips and planned maintenance windows on the destination cluster |
| Failed sync events (confirm metric availability) | Sync operation failures routed to the owning team | Invalid manifests, admission rejections or hook failures | Retried or self-healing syncs can produce short-lived failures, so confirm the exposed series in your Argo CD version |
| Project label on alerts | Routing to the team that owns each Argo CD project | Unrouted alerts when project labels are missing or inconsistent | Shared or unowned applications may need a default receiver, so confirm the label set your metrics expose |
How to build OutOfSync and Degraded alerts without paging on normal change
A sustained OutOfSync or Degraded state matters because it means the cluster no longer matches Git or a workload is not healthy, and nobody is acting on it. A brief transition is normal during a rollout: Argo CD marks an application OutOfSync as soon as a new commit lands and Degraded while pods cycle. The alert should fire when that state outlasts a normal sync or rollout window, not on the first sample. Build the rule on the argocd_app_info metric, which carries sync_status and health_status labels, and filter for the states you care about. Then add a for duration in the Prometheus alerting rule so the condition must hold continuously before the alert fires. Choose that duration by measuring how long your own syncs and rollouts take, including slow manifest generation, rather than copying a number from another team.
Group alerts by project and application so that one failing application produces one notification to the owning team, not a page per resource. Alertmanager handles grouping, inhibition and silences, so route by project label and use inhibition so a cluster connection loss does not also page for every application on that cluster. Prometheus values reliability over precision, as its overview documentation states, which means a short scrape gap or a single missed sample can shift a threshold. Rule expressions should therefore look at sustained counts rather than single points, and each threshold should be tested against change history from a few weeks of real deployments, including freeze windows and large batch merges. Confirm with your Prometheus and Alertmanager versions how the for clause and grouping interval behave in your setup before you rely on them for paging.
Set up Argo CD alerts in Prometheus and Alertmanager
This sequence confirms that Prometheus scrapes Argo CD's metrics, checks the label values the alerts depend on, and turns sync and health states into alerts routed to the teams that own each project. Each step closes with a check so the alerting chain is proven before it is trusted.
Confirm Argo CD metrics endpoints are scraped
The platform engineer responsible for monitoring adds scrape configuration for the Argo CD components that expose metrics, including the application controller, the API server and the repo-server, using the service discovery method your Prometheus already uses. Confirm the exact metrics ports and service names against the Argo CD metrics guide for the version you run, since they vary by component and install method. Done means every target shows as UP on the Prometheus targets page and the argocd_app_info series appear when you query them by name.
Verify sync and health label values
Using the Prometheus expression browser, the engineer queries argocd_app_info and lists the distinct values of the sync_status and health_status labels, along with the project label your routing will depend on. Record the exact spellings returned, because alert rules match on them literally and a typo produces an alert that never fires. Also query argocd_cluster_connection_status and argocd_app_reconcile to confirm they return data. Done means you have a written list of label values and a query for each signal that returns series from every cluster you expect.
Write alert rules for sync and health states
The engineer writes Prometheus alerting rules that count applications in OutOfSync and Degraded states per project, with a
forduration long enough to ignore normal rollouts, and separate rules for cluster connection loss and slow reconciliation. Choose the thresholds with the application owners, since a healthy estate and a noisy one need different values. Load the rule file, then check the Rules page in Prometheus for syntax errors and confirm each rule evaluates without error. Done means every rule reports a healthy state and the expressions return the counts you expect today.Configure Alertmanager receivers and project routes
The engineer defines one receiver per owning team, maps each project label to its receiver in the Alertmanager route tree, and sets grouping so one incident does not produce a flood of messages. Configure inhibition and silences only where a cluster-wide outage should suppress per-application alerts. Point Prometheus at every Alertmanager instance, not at a load balancer, if you run Alertmanager as a cluster. Done means
amtoolor the Alertmanager status page confirms the configuration loads and that a test alert with a given project label reaches the matching receiver.Test sync failure and degraded alerts with a controlled change
In a non-production cluster, the engineer deploys a deliberately broken manifest to a test application, such as an image tag that does not exist or a resource that the cluster rejects, so the application enters a failed or degraded state. Watch the alert move through pending to firing, then confirm the notification reaches the correct team's receiver. Revert the change and confirm the alert resolves. Done means the test produces a firing and a resolved notification for the intended project only.
Test cluster connection loss and reconciliation alerts
The engineer simulates a cluster connection problem in a non-production environment, for example by temporarily blocking network access from Argo CD to a registered test cluster, and watches argocd_cluster_connection_status and the cluster connection alert. Then the engineer checks that the reconciliation duration alert responds to a slow condition in the same way. Restore connectivity and confirm both alerts resolve. Done means each alert fires once with the correct cluster label and resolves after recovery, with no duplicate notifications.
Document the alert catalog and review ownership
The team writes a short catalog that lists each alert, its expression, its threshold rationale, the receiver it routes to and the runbook owner. Store the rule files and Alertmanager configuration in Git so changes go through review, as your change process requires. Note that Prometheus local storage is not replicated, so confirm with your platform team how long metrics are retained and whether remote storage is needed for the evidence you must keep. Done means the catalog and configuration are merged, and each alert has a named owner who has acknowledged it.
Routing Argo CD alerts to teams by project
This approach gives every Argo CD project its own Alertmanager receiver, so an alert about a payments application goes straight to the payments team's channel or paging service. Alerts from Argo CD carry the application's project as a label, and Alertmanager routes on that label through a route tree in alertmanager.yml. Ownership is clear because each team controls its own receiver, silences and escalation path.
- Label dependency: Routing depends on the project label being present on every series you alert on. Confirm which labels your Argo CD metrics expose in your version before writing matchers.
- Tree growth: Each new project adds a route and a receiver, so the file grows with the number of teams and must be reviewed when projects are created or retired.
- Escalation: Each team defines its own escalation, which works well for teams with 24/7 coverage and poorly for small teams with one engineer on call.
- Drift risk: A renamed or deleted project leaves a stale matcher that silently routes nothing, so alerts can fall through to the default receiver.
- Catch-all: Keep a default receiver for unmatched alerts so no failure goes unseen.
Before adopting this model, check that every project name in Argo CD matches a team in your routing file, and test one alert per receiver.
In this approach, all Argo CD alerts go to one platform or on-call receiver, and the message carries the project, application, cluster and sync or health state so the responder can hand off to the owning team. The routing tree stays short and changes only when the on-call rota changes. The trade-off is that ownership moves from the routing configuration into the responder's process.
- Short routing tree: One route and one receiver are simple to review, version and audit.
- Team fields in the message: Put the project, owner and runbook link into annotations so the handoff does not depend on memory.
- Handoff delay: The platform responder must identify and page the owner, which adds time for application-level failures.
- Alert fatigue: A single channel that receives every OutOfSync event from every team can bury the alerts that matter, so use severity labels.
- Ownership source: Keep team-to-project mapping in one maintained file, such as a service catalog export, rather than in the message template alone.
Before relying on this model, confirm that the on-call rota covers all projects and that the owner annotation is populated for every application.
The hybrid model applies per-project routing for most projects and a stricter path for clusters that run regulated or customer-facing workloads. Critical-cluster alerts go to a dedicated receiver with shorter grouping intervals and paging, while other alerts use the shared or per-project routes. This matches escalation effort to business impact without giving every team its own paging policy.
- Two tiers of routes: Match on cluster and severity first, then on project, so critical alerts take a dedicated path regardless of owner.
- Dual routing: Decide whether critical alerts also notify the owning team, and document it, so both the platform and application owners see the event.
- Inhibition: Use inhibition rules so a cluster connection loss suppresses the application alerts it causes, which prevents duplicate pages.
- Maintenance cost: Two routing models mean two sets of receivers, silences and tests to keep current.
- Clear criteria: Write down which clusters count as critical and who can change that list.
Before going live, trigger a test alert on each tier and confirm the page reaches the expected person within your agreed response window.
Keeping the alerting stack reliable during Argo CD upgrades
Argo CD ships a minor release four times a year, once every three months, and only the three most recent minor versions receive patch releases. That means your alerts will face a moving target: a sync or health alert that worked on one minor version can change its label values or metric behavior after an upgrade, and an older minor will stop getting fixes on a fixed schedule. Plan an upgrade window for the alerting path itself, not only for the Argo CD control plane. Before each Argo CD upgrade, confirm which minor line you are on, check the release notes for changes to the argocd_app_info labels and reconciliation metrics, and schedule the next move before the current line leaves the patch window. Prometheus and Alertmanager run on their own release cycles, so treat them as separate upgrade tracks with their own review dates.
After any Argo CD or Prometheus upgrade, re-test every alert rule against live data. Run each expression in the Prometheus expression browser and confirm it still returns the series you expect, especially the rules that count OutOfSync and Degraded applications from the sync_status and health_status labels. Then trigger a controlled test alert, for example by pointing a non-production application at a branch with an invalid manifest, and confirm the notification reaches the correct team channel for its project. Keep a short record of each test so that the next upgrade has a baseline to compare against.
Run Alertmanager as a cluster so that a single instance failure does not silence your notifications. Start each instance with the --cluster flags so they gossip and deduplicate alerts, and list every Alertmanager instance in the Prometheus alerting configuration. Prometheus should point at each instance directly, not at a load balancer in front of them. Confirm that each Alertmanager node is in the cluster after restarts or upgrades, and test a notification path with one node stopped. Storage is a separate concern: Prometheus local storage is not clustered or replicated, so decide early how long you retain metrics and whether you need remote write to external storage for the history you will rely on during audits.
Security and operational risks of Argo CD alerting
An alerting pipeline for Argo CD adds its own attack surface and failure modes: a metrics endpoint, scrape paths, Alertmanager, receiver credentials and a Prometheus server that holds the history. The cards below cover each risk and the control that reduces it, so the alerts stay trustworthy and do not leak what they describe.
Metrics endpoint access
Argo CD exposes application and cluster metrics over HTTP, and those series reveal application names, namespaces, sync states and cluster names to anyone who can reach them. Before you add alert rules, confirm which Argo CD metrics services are exposed through Services, ingress or network policy, and restrict them to the Prometheus server that scrapes them. Check the metrics guide for the exact endpoints in your Argo CD version.
Scrape network paths
Prometheus reaches Argo CD pods across the cluster network, and in multi-cluster setups it may reach targets in other clusters or segments. Each scrape path is a route that can carry sensitive operational data, so map which network policies allow the scrape and make sure no other workload can read the same port. Test the path after every network policy change, because a silent scrape failure removes the alerts that depend on it.
Alertmanager and receiver credentials
Alertmanager stores the credentials for chat, paging and ticketing receivers, and those tokens can post into channels or open incidents. Keep receiver configuration in a secret store, limit which operators can read the Alertmanager configuration, and rotate tokens on a schedule your security team sets. Alertmanager can run as a cluster for high availability, so apply the same controls to every instance.
Secrets in alert annotations
Alert annotations and labels are copied into notifications, dashboards and ticket bodies, so a template that includes a repository URL with a token, a cluster name tied to credentials, or a sync error message containing a secret spreads that secret to every receiver. Write annotation templates that reference the application name, project and link to the Argo CD view, and review sync error text before it is routed. Ask your team to test one alert end to end with a deliberately noisy error.
Local Prometheus storage is not replicated
Prometheus stores samples on local disk in a database that is not clustered or replicated, and the default retention is 15 days. If that single node fails, the history used to judge recurring sync failures is lost, and alerts stop until it recovers. Plan backups or remote write to external long-term storage, and set an alert on Prometheus itself so a dead alerting server does not go unnoticed.
Alert routing by project
Routing alerts to teams by Argo CD project keeps an OutOfSync alert for a payments application away from a team that cannot act on it, but a misconfigured route can send sensitive application names to a broad channel. Define routes from the project label, restrict who can change routing rules, and review the receiver list whenever a team or project is added. Confirm that the team owning each project has an on-call path before the route goes live.
What do Argo CD alerting projects cost and require?
What does it cost to run Prometheus and Alertmanager for Argo CD alerts?
The software itself has no license fee. Prometheus and Alertmanager are Apache-2.0 projects with no paid edition, so the cost comes from the infrastructure that runs them, the storage that holds metrics, and the people who build and maintain the alerts. Prometheus stores data in local blocks with a default retention of 15 days, and local storage is not clustered or replicated, so long retention or high availability adds storage, a remote long-term store or a second instance. Ask any partner to price each of these separately in the written scope.
Which parts of an alerting project are one-time and which recur?
The one-time work covers collecting the Argo CD metrics, writing the alert rules, building the Alertmanager routing tree, and testing that alerts fire and reach the right people. The recurring work covers rule tuning after noisy periods, updating alerts when Argo CD releases change metric behavior, patching Prometheus and Alertmanager, watching storage growth, and reviewing routing when teams or projects change. Argo CD itself receives patch releases only for its three most recent minor versions, so the recurring work includes keeping pace with that cadence.
How long does it take to get the first useful Argo CD alert?
The time depends on how much of the monitoring stack already exists. If Prometheus already scrapes the Argo CD metrics endpoints and Alertmanager already delivers to your chat or paging tool, a first alert on sync failures can be tested quickly. A new stack, a new notification channel or a routing model that needs agreement with several teams takes longer. Ask for a milestone that delivers one tested critical alert first, then the wider rule set, so you can see progress before the full scope is complete.
What must our team provide before alerting work can start?
Expect to provide access and decisions, not just time. The usual inputs are:
- Network access from Prometheus to each Argo CD metrics endpoint and to every cluster that hosts Argo CD
- A named owner for each Argo CD project or application group who receives its alerts
- On-call rules, escalation paths and maintenance windows
- The chat, email or paging tools that Alertmanager should use, with credentials stored in your secrets system
- Agreement on which alerts page someone and which only create tickets
Missing owners cause most routing rework, so confirm them before the build starts.
Who owns the alert rules after the project is delivered?
Ownership should be written into the scope before work begins. Your platform or SRE team usually owns the rule files, the Alertmanager configuration and the on-call policy, and the application teams own the thresholds for their own projects. A partner can build, document and hand over the rules as version-controlled files, and can offer ongoing support as a retainer if you want someone to tune them over time. Confirm in the quote which party handles each change request.
Can our own team build Argo CD alerting without a partner?
Yes, if the team already runs Prometheus and understands PromQL, the Argo CD metrics and your on-call process. Prometheus is queried with PromQL and collects metrics by pulling them from targets, and Alertmanager handles grouping, inhibition and silences, so the tools are learnable. The hard parts are usually the routing design across teams and the tuning that keeps alerts useful. A partner makes sense when your team lacks that experience, needs the work finished on a fixed schedule, or wants an independent review of the rules before they go live.
Does alerting on Argo CD guarantee that we never miss a failure?
No. Prometheus values reliability over precision and is not a good choice where 100% accuracy is required, so alerting reduces the risk of missed failures but does not remove it. Alerts can be delayed by scrape intervals, lost when a Prometheus instance fails without redundancy, or suppressed by a silence or inhibition rule someone forgot to remove. Plan for a second check, such as a dashboard review or a periodic report, and test the alert path on a schedule.
How DeployOpen scopes and delivers Argo CD alerting
Alerting work is scoped in writing before any rule is written. DeployOpen agrees the scope, deliverables and timeline for an Argo CD alerting engagement in a written statement of work, and that document is the basis for a quote. For this topic the scope usually names the Argo CD instances and clusters in view, the Prometheus and Alertmanager deployments that will evaluate the rules, the Grafana instance (if any) used for dashboards, the projects and teams that will receive notifications, and the on-call channels involved. It also records what is out of scope, such as changes to Argo CD itself or to application manifests, so the boundary is visible to both sides. Engagements are fixed-scope, so scope changes are handled through a written change to the agreement rather than by informal additions.
The deliverables are concrete artifacts your team keeps. Typically they include the Prometheus rule set covering argocd_app_info sync and health states, reconciliation duration from argocd_app_reconcile, and argocd_cluster_connection_status; the Alertmanager routing configuration that maps Argo CD project labels to receivers; runbook links for each alert; and test evidence showing that each rule fires against a controlled condition and resolves when the condition clears. Ask for the test method in the quote, including who triggers the test conditions and in which environment. Confirm with your own team which metric labels your Argo CD version exposes, because label names should be checked against the Argo CD metrics documentation before rules are fixed.
Ongoing tuning is optional and sold separately as a retainer. Alert thresholds usually need adjustment after the first weeks of real use, when noisy applications or planned maintenance windows become visible, and a retainer covers that review, rule changes and re-testing after platform upgrades. Costs recur under a retainer and are one-time under a project. Cost drivers include the number of clusters and projects routed, the number of receivers and escalation paths, whether the monitoring stack is new or existing, and how much test evidence your auditors or change board expect. DeployOpen does not make claims about alert outcomes or certification status; ask for the deliverable list and acceptance criteria in writing.
