Monitoring Apache Airflow with Prometheus and Grafana starts with a metrics route
Apache Airflow has no native Prometheus endpoint, so a self-hosted Airflow cannot be scraped directly. Metrics have to reach Prometheus through an OpenTelemetry Collector or through statsd_exporter, and that route is the first decision in any monitoring project. Until it is in place, scheduler delays, slow DAG parsing and failed tasks stay invisible, and the first signal is usually a complaint from a data consumer whose report is late. Teams should choose the route before they build dashboards, because the metric names, histogram behavior and labels that reach Prometheus depend on it.
Prometheus is an open-source monitoring and alerting toolkit. It pulls metrics over HTTP from targets it discovers or is configured to scrape, stores them in its local time series database and evaluates alert rules written in PromQL. Grafana is the visualization layer that queries Prometheus and presents the results as dashboards, and it can also send alerts. Neither product runs Airflow for you, and neither one decides which Airflow signals matter. That judgment belongs to the team that operates the pipelines, which is why this work is usually planned with the people who own scheduling and on-call.
This work becomes necessary when Airflow carries pipelines that business owners depend on, when more than one scheduler or worker pool is running, or when your current checks only tell you that a task failed after the fact. Confirm two things before you start: which metrics your Airflow version emits and under which names, using the Airflow metrics configuration reference, and whether your Prometheus version supports the native histograms your timing metrics need. Those answers set the collector configuration and the cost of running it.
Choosing between the OpenTelemetry Collector and statsd_exporter
This table compares the two supported routes for getting Airflow metrics into Prometheus, along with the Helm chart exporters for PgBouncer and Flower. Read the first block to pick a route, then confirm the naming and label details against the Airflow metrics configuration in your own version before you write any alert rules.
| Decision point | OpenTelemetry Collector | statsd_exporter |
|---|---|---|
| Best fit | Teams that want timing histograms and a single telemetry pipeline that can also carry other signals | Teams that already run StatsD-style metrics or want the smallest new component |
| Timing histograms | Works with native histograms when Prometheus scrapes them (scrape_native_histograms on Prometheus 3.8 or later) | Timing metrics are converted to Prometheus histograms or summaries through statsd_exporter mapping, so bucket choices are yours to set |
| Extra components to run | Collector deployment, its configuration, and its own upgrade and monitoring | statsd_exporter process plus a UDP or TCP path from Airflow to it |
| Metric naming | Confirm how Airflow dotted names are mapped to Prometheus names in your Collector pipeline | Dotted names are rewritten through mapping rules, so dashboards depend on the rules you write |
| Label handling | Tags such as dag_id and task_id can be carried as labels; check cardinality before enabling them broadly | Tags depend on the StatsD tag format and mapping configuration you define |
| Settings to confirm in Airflow | Metrics configuration options for OpenTelemetry in your exact Airflow 3 release | Metrics configuration options for StatsD, including host, port and prefix, in your exact Airflow 3 release |
| Helm chart exporter: PgBouncer | Chart provides Prometheus metrics for PgBouncer, scraped separately from Airflow metrics | Same. Scrape the PgBouncer metrics endpoint in addition to whichever Airflow route you choose |
| Helm chart exporter: Flower | Chart provides Prometheus metrics for Flower to monitor Celery workers, relevant only with CeleryExecutor | Same. Flower metrics matter only when Celery is in use, so skip this row for other executors |
How to set up the Airflow metrics pipeline step by step
This sequence takes Airflow from no metrics to a verified path into Prometheus and Grafana, with a check at each stage so that a gap is found at the step that caused it.
Choose the emitter and collection route
Decide between StatsD and OpenTelemetry in the Airflow metrics settings, then choose the collection route that matches it: an OpenTelemetry Collector with native histogram support, or statsd_exporter. Record the Prometheus version you run, because native histogram scraping depends on it. Your platform engineer and the Airflow team should agree on this before any configuration changes. Check: a one-page decision note names the emitter, the collector or exporter, and the Prometheus version, and every team that will run the pipeline has read it.
Enable metrics in every Airflow component
Turn on the chosen emitter in the Airflow configuration used by the scheduler, DAG processor, API server, workers and triggerer, since each component emits its own metrics. Set the host and port the emitter should reach, and give each scheduler a distinguishable identity so its series can be told apart. In the Helm chart, this is done through the chart values for StatsD or Prometheus metrics. Check: restart one component in a test environment and confirm it sends samples to the emitter, for example by watching the collector or exporter receive data.
Deploy the OpenTelemetry Collector or statsd_exporter
Deploy the collector or exporter in the same cluster or network zone as Airflow so that it can receive samples without crossing a firewall. If you use the OpenTelemetry route, configure a pipeline that exposes Prometheus-format output, and keep timing histograms as native histograms where your Prometheus version supports them. If you use statsd_exporter, load a mapping file so that dotted Airflow names and tags become stable metric names and labels. Check: the exporter's own endpoint lists Airflow series, and a curl against it returns scheduler metrics with the expected labels.
Add Prometheus scrape configuration and discovery
Add a scrape job for the collector or exporter, using service discovery where the platform supports it, or static targets where it does not. Keep the scrape interval consistent with the alert rules you plan to write, and set a clear job label so dashboards can filter by environment. Prometheus stores data locally by default with 15 days of retention, so confirm that retention or remote write meets your audit needs before you rely on the data. Check: the Prometheus targets page shows the new job as up, and no target is in a failed state after a reload.
Enable Helm chart exporters for PgBouncer and Flower
Only where your deployment uses them, enable the Prometheus metrics that the Helm chart provides for PgBouncer, which fronts the metadata database, and for Flower, which monitors Celery. Add scrape jobs for each exporter endpoint and keep them separate from the Airflow job so that a failure in one source does not hide the others. Check: each exporter target is up in Prometheus, and a query against PgBouncer or Flower series returns values from the running environment.
Verify core Airflow series appear
Run queries in Prometheus or Grafana Explore for scheduler_heartbeat, dag_processing.import_errors, dag_processing.total_parse_time and pool.open_slots, pool.running_slots and pool.queued_slots. Note that series names may appear with underscores or dots depending on the route, so confirm the exact form your exporter produces before writing alerts against it. Trigger a test DAG run to confirm that task-level counters such as ti_failures appear with dag_id and task_id labels. Check: each series returns data for the current scheduler and a test run, and the names you verified are written into your runbook.
Build dashboards and alerts in Grafana
Connect Grafana to the Prometheus data source, ideally defined through provisioning files so the setup can be reviewed and rebuilt. Create dashboards for scheduler health, parse time, pool occupancy and exporter status, then add alert rules for a stale scheduler heartbeat, new import errors and sustained queued pool slots. Route alerts through Alertmanager with a named receiver and test each rule with a controlled condition. Check: a deliberately triggered test alert reaches the receiver, and the provisioned dashboards load after a clean Grafana restart.
Which Airflow metrics to chart and alert on first
Start with a small set of Airflow metrics that answer three questions: is the scheduler alive, is DAG code parsing cleanly, and are tasks failing or waiting. The metrics docs list scheduler_heartbeat, which shows the scheduler is still running its loop, so a flat or missing series is the first alert to build. Pair it with dag_processing.total_parse_time, which shows how long DAG files take to parse, and dag_processing.import_errors, which counts DAG files that fail to import. A rising parse time often precedes scheduling delay, and a non-zero import error count usually means a DAG has silently disappeared from the UI. Confirm the exact metric names your Airflow version emits, because the Prometheus names differ from the dotted StatsD names once the statsd_exporter or OpenTelemetry Collector has translated them.
The task-level signal is ti_failures, which the docs tag by dag_id and task_id. Those labels make failures easy to attribute, but every unique DAG and task combination becomes its own Prometheus series. In an estate with thousands of generated DAGs or dynamically mapped tasks, that multiplies storage and query cost quickly. Decide before go-live whether task-level failure counts belong in Prometheus at all, or whether they are better left in Airflow's own UI and logs, and confirm with your platform team how the metric names are mapped and which labels survive the Collector pipeline.
Pool metrics are the third group. pool.open_slots, pool.running_slots and pool.queued_slots show whether a pool is saturated and tasks are waiting for capacity. Chart queued slots against open slots for each pool that protects a database or API, and alert when queued slots stay high while open slots stay at zero. Pool names are a bounded set set by your team, so they are a safer label choice than per-task identifiers. Verify the metric set against the Airflow version you run, since the metrics docs are the source of truth for names and tags.
Dashboards and alert rules to build for Airflow
Each card below is a dashboard or alert group to design once metrics reach Prometheus. Set thresholds from your own baseline over a representative period, not from generic values, and review them after every major change to schedulers, pools or the database.
Scheduler health
Build a panel and alert on scheduler_heartbeat for every scheduler replica, since Airflow can run several schedulers against the same metadata database. A heartbeat that stops for one replica usually means that replica is down or blocked on the database, even when the others keep scheduling. Confirm the expected heartbeat interval in your own environment, then alert on a gap several times longer than that interval. Use the scheduler job metrics to compare loop behavior across replicas.
DAG parsing and import errors
Chart dag_processing.total_parse_time over time and alert on dag_processing.import_errors staying above zero for longer than a deploy window. A rising parse time after a code change points to new top-level code, while import errors point to missing dependencies or broken DAG files. Set the parse time threshold from the slowest healthy parse in your baseline, and review it whenever the number of DAG files grows.
Task failures by DAG
Build a panel of ti_failures tagged by dag_id and task_id, and alert when one task fails repeatedly within a short window. Grouping by DAG shows whether failures cluster in one pipeline, which usually indicates a data or code problem, while spread-out failures often point to shared infrastructure such as a database or API. Agree on the failure rate that pages someone with the owners of the highest-volume DAGs, since tolerance differs by pipeline.
Pool saturation and queued work
Plot pool.open_slots, pool.running_slots and pool.queued_slots for every pool that protects a database or external API. Alert when open slots stay at zero while queued slots keep growing, because that means work is waiting on a limit rather than on capacity. Confirm which pools exist in your deployment before building the panel, and compare queued time against the pool sizes you set on purpose.
Database connection pressure via PgBouncer
Use the PgBouncer Prometheus metrics that the Helm chart can expose to chart client connections, waiting clients and pool usage against the metadata database. Alert when clients wait for a server connection for a sustained period, since that shows scheduler, worker or API processes competing for too few database connections. Set thresholds against the connection limit you configure for PostgreSQL and the number of Airflow processes you run.
Celery queue depth via Flower
Where you run the CeleryExecutor, build a panel from the Flower metrics the Helm chart can expose, covering queued and active tasks per queue. Alert when a queue grows while worker activity stays flat, which suggests stalled workers, a broker problem or a scaling rule that never fires. If you do not use Celery, omit this card and build the equivalent view for your executor.
Running Prometheus and Grafana without a single point of failure
Prometheus local storage is not clustered or replicated, so a single Prometheus server is a single point of failure for your Airflow alerts. The Prometheus storage documentation says to manage it like any other single-node database. Default retention is 15 days, configurable by time or size through the retention flags, and that setting determines how far back your Airflow dashboards and alert history can look. If your audit or capacity planning needs longer history, the route is remote write to an external long-term storage system, which Prometheus supports through its remote write and remote read protocols. Decide the retention window and the remote storage target before the first dashboards are built, and confirm both with the team that owns long-term metrics storage.
Grafana keeps its configuration in SQLite 3 by default, and the installation documentation says SQLite does not scale for production. For production and high availability, configure MySQL 8.0 or later or PostgreSQL 12 or later as the Grafana database. High availability uses two or more active-active Grafana servers behind a load balancer sharing that database, with database-backed auth tokens. The high availability guidance also notes that alerting needs extra configuration to avoid duplicate notifications, so confirm how alert notifications will be routed before you run more than one node. Store dashboards and data source definitions as provisioning files in version control so a rebuilt Grafana node comes back with the same setup.
Alertmanager can run as a cluster for high availability using its cluster flags, and each node must be given the others. Prometheus should be configured to send alerts to every Alertmanager instance rather than to a load balancer in front of them, so that alerts still reach a receiver when one node fails. Confirm the cluster peer addresses and the receiver configuration on each node in a test before relying on them for paging. For upgrades, Prometheus publishes a new minor release every 6 weeks, and a minor release generally stops receiving bug fixes after those 6 weeks. Long-term support releases receive bug fixes for one year, so pick a line that matches how often your team can schedule upgrades, and read the Prometheus release cycle page before choosing it.
Limits to confirm in your environment before building
The monitoring path depends on details that differ between environments: the Airflow version and metric names you emit, the Prometheus release you run, and how Grafana is licensed and clustered. Confirm these six points first, because each one changes the collector configuration, the dashboards or the alert design.
Airflow version and metric names
Confirm which Airflow version you run and which metric names it emits in your configuration. Metric names such as scheduler_heartbeat, dag_processing.import_errors and pool.open_slots are the basis for dashboards and alerts, and naming changes after an upgrade can break panels silently. Check the emitted names against the metrics reference for your exact version before you write queries. Airflow 3 is the current major line, and Airflow 2 reached end of life on April 22, 2026.
Native histogram support in Prometheus
Timing metrics that arrive as native histograms need a Prometheus version and a scrape setting that accept them. Confirm your Prometheus release, the scrape_native_histograms option and whether your OpenTelemetry Collector exports in the format your Prometheus expects. If you cannot meet those conditions, plan a statsd_exporter route for timing data instead, and test both before committing to dashboards that depend on histogram quantiles.
Grafana alerting in high availability
Two or more active Grafana servers behind a load balancer share one database, but alerting needs extra configuration to avoid duplicate notifications. Confirm how your alerting instances coordinate, who receives each notification, and how a failed node is detected. Test a deliberate failover against a rule that fires, so you can see whether a single page or two reaches the on-call channel.
Grafana Live limits in HA
Grafana Live, which pushes live updates to open dashboards, has limitations when Grafana runs in high availability. If your operators depend on live panels during an incident, confirm which features work across nodes and which fall back to refresh intervals. Decide the refresh rate for critical operations dashboards on that basis, and avoid designing an alert workflow that depends on live updates alone.
Grafana license and Enterprise features
Grafana, Loki and Tempo moved from Apache License 2.0 to AGPLv3 on April 20, 2021, so review how AGPLv3 applies to your use, including any modifications you distribute or offer over a network. Separately, SAML, team sync, active sync and SCIM are available only in Grafana Enterprise and Grafana Cloud. Confirm with your legal and identity teams whether your SSO and provisioning needs require a commercial license before you design the access model.
Prometheus favors reliability over precision
The Prometheus project values reliability over precision and is not suited to workloads that need 100% accuracy, such as per-request billing. Treat Airflow metrics as operational signals for scheduler health, queue depth and failure trends, not as an exact audit record of task outcomes. Confirm with your finance and compliance owners which numbers must come from the Airflow metadata database instead, and document that split before alerts rely on Prometheus counts.
Questions teams ask before monitoring Airflow
What drives the cost of monitoring Airflow with Prometheus and Grafana?
The software is free to use, so the cost comes from the infrastructure that runs it, the storage that holds metrics, the implementation work and the people who operate it afterward. Prometheus is licensed under Apache-2.0 and has no paid enterprise edition, and Grafana's core is available for self-hosting, although Grafana's relicensing in 2021 means you should check which edition and license terms apply to your use. Recurring costs include compute and disk for Prometheus, retention settings that set how much history you keep, and staff time for upgrades and alert tuning. One-time costs cover the OpenTelemetry Collector or statsd_exporter setup, dashboards and alert rules. Ask any partner for a quote that separates implementation, infrastructure and ongoing support, and state the retention period and expected number of DAGs and tasks it is based on.
How long does a Prometheus and Grafana monitoring setup take?
Duration depends on how many Airflow components you run, whether you already operate Prometheus and Grafana, and how many dashboards and alert rules your operations team needs. The main variables are the metrics path you choose (OpenTelemetry Collector or statsd_exporter), whether you deploy with the Helm chart or on virtual machines, and whether PgBouncer and Flower metrics are in scope. Teams with an existing Prometheus stack often need less time for the collection layer, while new stacks also need storage, high availability planning and access controls. A written scope lists the components, dashboards and alerts, which lets you compare timelines between a partner and your own team. No duration is given here because the answer depends on your environment.
What does our team need to provide before monitoring work starts?
Expect to provide access and decisions rather than only approvals. The list usually includes:
- Airflow administrators who can confirm the version in use, the executor, and the deployment method
- Kubernetes or virtual machine access with rights to create the collector, exporter and monitoring workloads
- DNS names and certificates for Prometheus, Grafana and any Alertmanager endpoints
- Persistent storage sized for your chosen retention, plus backup location for Grafana configuration
- A decision on who receives alerts and which on-call channel they use
Airflow 2 reached end of life on April 22, 2026, so confirm your version first, since Airflow 3 changes which components expose metrics and how they run.
Can our own platform team build this monitoring without a partner?
Yes, a capable platform team can build this, because Prometheus, Grafana, the OpenTelemetry Collector and the Airflow Helm chart are all open source and documented. The work requires familiarity with PromQL, Kubernetes or VM operations, and Airflow metric names such as scheduler_heartbeat and dag_processing.import_errors. Teams usually find the hardest parts are choosing a metrics route, handling timing histograms, planning storage, and keeping alert rules accurate over time. Assess your team's capacity against those tasks and the on-call load they add. A partner can fill specific gaps, such as the first design, a fixed-scope build or a retainer for upgrades.
Which Airflow metrics should we alert on first?
Start with the metrics that show whether scheduling and DAG parsing are healthy, then add task-level signals. Useful first alerts include scheduler_heartbeat going stale, dag_processing.import_errors rising above zero, dag_processing.total_parse_time climbing over time, and ti_failures grouped by dag_id and task_id. For capacity, watch pool.open_slots, pool.running_slots and pool.queued_slots, since sustained queued slots point to pool limits or worker shortage. Tune thresholds against a baseline from your own environment, and route each alert to a named owner. Prometheus values reliability over precision, so treat alerts as signals to investigate rather than exact accounting.
Is Grafana Cloud a cheaper alternative to self-hosted Grafana?
Grafana Cloud can replace a self-hosted Grafana deployment, and the right choice depends on data residency, operating capacity and total cost. Grafana's pricing page lists a free tier, a Pro tier starting with a 19 US dollar monthly platform fee plus usage, and an Enterprise tier with a minimum commitment of 25,000 US dollars per year. Usage-based charges will vary with metric volume, so model your expected series count before comparing. Self-hosting keeps metrics inside infrastructure you control, but you carry upgrades, high availability and backups. Confirm current pricing and limits directly with Grafana before deciding.
How do Airflow and Prometheus upgrades affect the monitoring stack?
Airflow upgrades can change metric names, the components that emit them and how they are scraped, so every upgrade needs a metrics check. Airflow 3 replaces the webserver with an API server and requires a separate DAG processor, both of which affect where you collect data. On the Prometheus side, minor releases receive bug fixes for only about six weeks, so plan regular upgrades, and use the LTS line if you need a longer support window. The current LTS line is 3.13, supported until July 31, 2027. Grafana minor releases are supported for 9 months, so schedule those upgrades too. Keep dashboards and alert rules in version control and test them in staging before each change.
What does a DeployOpen engagement cover for Airflow monitoring?
DeployOpen plans, implements and documents open-source platforms inside your own infrastructure, which can include a monitoring build for Airflow. A typical engagement is fixed-scope, with the components, metrics route, dashboards, alert rules and handover agreed in writing before work starts. Support and upgrades can be added as a retainer for ongoing operation. DeployOpen does not publish a price list, so a quote follows from the written scope. Ask for the scope to name the Airflow version, the metrics path, the storage and retention design, and who receives the alerts.
How DeployOpen scopes and supports Airflow monitoring
DeployOpen plans, implements, hardens and supports self-hosted open-source platforms inside the infrastructure the client controls, and a Airflow monitoring engagement follows the same pattern. DeployOpen is a Chainscore Labs company. For this topic, the work runs in your cloud accounts, data centers or air-gapped network, against your own Airflow deployment, your Prometheus and Grafana instances, and your Alertmanager routing. Before any build starts, the scope should name the metric path you choose (an OpenTelemetry Collector with native histogram scraping, or statsd_exporter), the Helm chart components in use (StatsD or Prometheus metrics, PgBouncer metrics, Flower for Celery), the dashboards and alert rules to deliver, and who receives each alert.
Most engagements are fixed-scope. Scope, deliverables and timelines are agreed in writing before work starts, so the written scope becomes the reference for acceptance. For monitoring, a useful scope lists each dashboard, each alert rule with its threshold logic and owner, each data source and retention setting (Prometheus keeps local data for 15 days by default and can be configured by time or size), and the handover material your team needs. Ask for the list of deliverables to be specific enough that your team could check each one. Because DeployOpen does not publish a price list, a quote follows from that written scope, and the main drivers are the number of Airflow components and environments covered, the number of integrations with your existing Grafana or Alertmanager setup, and how much of the alert design your team wants to review.
Ongoing support is available as a retainer. Monitoring needs continuing attention after go-live: Prometheus, Grafana and Airflow each publish their own release and support cycles, so dashboards and alert rules need periodic review when versions change. Ask any provider to confirm, in writing, which of these recur and which are one-time: the initial build, dashboard and alert tuning after the first production incidents, version upgrades of each component, and on-call escalation paths. Confirm also what your own team must keep running, such as Prometheus storage capacity, Grafana database backups and access control. DeployOpen does not claim certifications or audit outcomes for this work. It prepares monitoring evidence and controls for your assessors, and independent auditors make their own determinations.
