Apache Airflow metrics with StatsD and OpenTelemetry: which emitter fits your platform
Airflow emits metrics through the [metrics] section of its configuration, and you can send them to either StatsD or OpenTelemetry. Until you choose one, a self-hosted Airflow can look healthy from the UI while scheduler lag, parse failures and pool saturation go unseen. The choice matters most once you run more than one scheduler, because the way each emitter identifies its source determines whether the numbers you alert on are trustworthy. This work is worth planning before the first production deployment, and again before you add a second scheduler or change how DAGs are parsed.
The difference that drives the decision is attribution. StatsD cannot tell which process produced a metric, so several schedulers can overwrite each other's values under the same name. OpenTelemetry sets service.instance.id for each process, which keeps schedulers distinct. Confirm this behavior against your Airflow version in the Airflow metrics configuration reference, and check which emitter your existing monitoring stack already accepts. Also confirm how your platform team will receive OpenTelemetry data, whether directly or through a Collector.
This section covers the emitter choice, the settings that switch it on, and the operational scope around it. It does not cover the backend that stores, queries and charts the metrics, such as Prometheus, Grafana or a commercial service. Those choices, their retention and their alert routing are separate decisions, and they should be made with the same team that owns on-call. Ask your platform team which backend they will use, how long metrics must be kept, and who receives each alert before you fix the emitter.
StatsD or OpenTelemetry: how the two emitters differ for Airflow
This is the factor that most often decides the question. Airflow can run more than one scheduler at once for performance and resiliency, but StatsD has no way to tell which process produced a given metric. When several schedulers send the same metric names, their values overwrite or blend into each other, so a heartbeat or parse-time reading may describe a scheduler you are not looking at. OpenTelemetry avoids this by setting a service.instance.id for each process, so each scheduler's series stays separate and can be filtered individually. If you run a single scheduler, StatsD attribution is less of a concern. If you run several, plan on OpenTelemetry or confirm that your StatsD pipeline can distinguish senders in some other way.
- Single scheduler: StatsD attribution is rarely a problem, so this factor alone does not force a choice.
- Multiple schedulers: StatsD series from different schedulers can overwrite one another under the same metric name.
- Per-process identity: OpenTelemetry sets
service.instance.idper process, which keeps each scheduler's readings distinct. - Heartbeat checks: A per-instance
scheduler_heartbeatis only reliable per scheduler if each one is identifiable. - Test before rollout: Start two schedulers in a non-production environment and confirm that each one's series appears separately.
Before you go live, check that your chosen emitter shows one series per scheduler for scheduler_heartbeat and dag_processing.total_parse_time.
The second factor is what your organization already runs. If your monitoring already ingests StatsD, the Airflow StatsD settings in the [metrics] section may be the shortest path, and the existing dashboards and alert rules can be reused. If your teams already use OpenTelemetry for applications, Airflow metrics can enter the same pipeline and share its resource attributes and export configuration. Neither choice requires a specific backend, and OpenTelemetry does not store or visualize data by itself, so the backend decision is separate. Confirm which ingestion paths your current tools accept before choosing.
- Existing StatsD receiver: Reuse the receiver and its naming conventions if they already serve other systems.
- Existing OpenTelemetry pipeline: Route Airflow through the same collector configuration used for applications.
- Dashboards and alerts: Count the existing panels and rules you would need to rebuild if metric names or types change.
- Backend independence: OpenTelemetry is not a backend, so storage and visualization remain a separate choice.
- Ask your monitoring owner: Confirm which protocols the current backend ingests before assuming either path works.
List the dashboards and alerts that depend on Airflow metric names, then check whether each emitter preserves those names.
OpenTelemetry usually means running an OpenTelemetry Collector, either as an agent next to services or as a gateway that receives data from many sources. The Collector is vendor-agnostic and its repository is licensed under Apache-2.0, but it is one more component to deploy, size, secure and upgrade. Collector releases are published with signed artifacts, SHA256 checksums and SBOM files, which helps with supply-chain review. The Collector also has components at mixed stability levels, so the README of each component you use should be checked. StatsD adds a simpler path when the receiver already exists, but it still needs an address to send to and a plan for its failure modes.
- Collector placement: Decide between agent and gateway deployment based on how many Airflow components send metrics.
- Authentication: Collector authenticator extensions can authenticate incoming connections and add credentials to outgoing exports.
- Component stability: Check the stability level of each Collector component in its README before relying on it.
- Upgrades: Each Collector upgrade is a change to your telemetry path and needs its own test cycle.
- Release verification: Use the published checksums and SBOM files as part of your intake process.
Confirm who will operate the Collector, how it is upgraded, and what happens to Airflow metrics if it is unavailable.
The last factor is who will keep the pipeline working. OpenTelemetry configuration is written in the Collector's YAML and involves receivers, processors and exporters, so the team needs comfort with that model. StatsD settings in Airflow are fewer and simpler, which can suit a team that wants a small surface area. The trade-off is that the simpler option may not give you per-process identity or the flexibility to add resource attributes later. Whichever you pick, name an owner for the Airflow metrics configuration, the allow and block lists, and the alert rules, because those drift over time as Airflow versions change.
- Configuration model: OpenTelemetry requires understanding Collector pipelines, which is a larger skill set than StatsD settings.
- Future flexibility: StatsD may limit later changes to attribution or attributes.
- Named owner: Assign someone to maintain the
[metrics]settings and the alert rules that depend on them. - Training: Budget time for the team to learn the Collector configuration before go-live.
- Vendor support: Ask your monitoring provider which configuration patterns they support.
Ask the team who will answer a 2 a.m. alert about missing Airflow metrics, and confirm they can trace the path from the scheduler to the backend.
Metrics configuration options and when each fits
This table compares the [metrics] settings and custom metric calls that shape how Airflow emits telemetry, and the situation each one fits. Read each row as a decision point: pick the backend first, then add naming and filtering controls. Confirm exact key names against the Airflow metrics configuration reference before you change a production file.
| Setting or call | What it controls | When it fits |
|---|---|---|
| statsd_on or otel_on (backend choice) | Turns on StatsD or OpenTelemetry emission from the [metrics] section | Choose StatsD when you already run a StatsD receiver; choose OpenTelemetry for per-process attribution with several schedulers |
| statsd_host and statsd_port | Address of the StatsD receiver that Airflow sends UDP metrics to | Single-scheduler setups or existing StatsD pipelines; confirm the receiver accepts the metric names Airflow sends |
| statsd_prefix | Prefix added to every StatsD metric name | Multiple Airflow environments sharing one receiver, so staging and production series stay separate |
| otel_host, otel_port and related OTLP keys | Endpoint, port and transport for OpenTelemetry export, typically a Collector | Multi-scheduler deployments where each process should be identifiable through service.instance.id |
| metrics_allow_list | Regular expression list of metric names to keep | Cutting volume to a known set of alert metrics such as scheduler heartbeat and pool slots |
| metrics_block_list | Regular expression list of metric names to drop | Removing noisy metrics after the allow list is too broad to narrow further |
| stats.incr (airflow.sdk.observability) | Increments a counter from task code | Counting business events such as rows processed or files skipped inside a task |
| stats.gauge | Records a point-in-time value from task code | Reporting a current size, backlog or queue depth observed during a run |
| stats.timing | Records a duration from task code | Tracking how long a call to an external API or database takes per run |
Setting up Airflow metrics in order
This sequence takes a self-hosted Airflow from no metrics to a verified pipeline that shows which scheduler emitted each signal, filters the noise, and carries task-level metrics into your alerting. Each step ends with a check so you know the layer below works before you build on it.
Choose StatsD or OpenTelemetry as the emitter
Your platform team decides which emitter Airflow uses, based on what already receives telemetry in your environment. StatsD is simpler when you already run a StatsD receiver, but it cannot attribute metrics to the process that produced them, so several schedulers will overwrite each other's values. OpenTelemetry sets a service.instance.id for each process, which keeps schedulers distinct. Confirm the choice against the Airflow metrics configuration before you write any settings. The output is a written decision that names the receiver and the backend that will store the data.
Set the metrics section in airflow.cfg
The engineer responsible for the Airflow deployment sets the [metrics] section for the chosen emitter, using the same configuration on every component that emits metrics. Keep these settings in version-controlled configuration, not in a hand-edited file on a single host. Set the host and port of the StatsD receiver or the OpenTelemetry endpoint, and use one prefix convention across all environments. Check the exact keys against the metrics documentation for your Airflow version, since names differ between StatsD and OpenTelemetry settings. Done means a configuration review has passed and the values are deployed to the scheduler, DAG processor, API server and workers.
Confirm each component's process identity
Before you rely on dashboards, confirm that each scheduler, DAG processor, API server and worker reports under its own identity. With OpenTelemetry, check that service.instance.id differs per process. With StatsD, plan a per-instance naming scheme or move to OpenTelemetry if you run more than one scheduler, since StatsD cannot separate their series. Start two schedulers in a test environment and compare their emitted series. Done means each component appears as a separate source in your backend, and you can name the one that produced any heartbeat.
Apply allow and block lists
Set metrics_allow_list and metrics_block_list as regular expressions to limit what leaves Airflow. Start from a short allow list covering scheduler, DAG processing, task failure and pool metrics, then widen it only when a dashboard or alert needs more. Block high-cardinality series that carry identifiers you do not need to query, and review the list whenever you add DAGs with many tasks. Test each expression against the metric names in a non-production environment, since a mistake here silently drops data. Done means the emitted metric set matches the list you reviewed, and the alert metrics are still present.
Deploy the Collector or StatsD receiver
The platform team deploys the receiver that sits between Airflow and your backend. For OpenTelemetry, that is usually an OpenTelemetry Collector, which can run as an agent beside Airflow or as a gateway that receives from several sources. Configure its receivers, processors and exporters in version-controlled files, and restrict the receiver endpoints to the networks where Airflow runs. Follow the collector configuration best practices for authentication and transport settings, and use authenticator extensions if your security policy requires them. Done means the receiver is running, reports healthy, and accepts a test metric from an Airflow component.
Verify scheduler heartbeat and pool slot metrics
Confirm that the core metrics reach your backend. Look for scheduler_heartbeat from every scheduler, dag_processing.import_errors and dag_processing.total_parse_time from the DAG processor, and pool.open_slots, pool.running_slots and pool.queued_slots for a pool with a known workload. Trigger a test DAG to create task activity, and confirm that ti_failures appears with dag_id and task_id tags after a deliberate failure in a sandbox. Done means every example metric is visible with the expected source label, and a missing heartbeat from one scheduler would be detectable on a dashboard.
Add task-level custom metrics
Once the platform metrics are verified, DAG authors can emit their own signals from tasks using airflow.sdk.observability with stats.incr, stats.gauge and stats.timing. Agree on a naming convention and a small set of tags with the platform team before anyone adds metrics, so that series from different teams stay comparable. Keep tag values bounded, since unbounded values create many series. Test a custom metric in staging and confirm it passes the allow list you set in the earlier step. Done means one task emits a custom metric that appears in your backend under the agreed name, and the team has documented what each metric means for alerting.
Telling schedulers apart when several run at once
Running more than one scheduler is a supported way to improve throughput and resilience in Airflow, because the schedulers coordinate through database row-level locks rather than through a central leader. That design means each scheduler does real scheduling work, so its metrics describe a different slice of the workload. The metrics problem starts with StatsD. StatsD cannot attribute a metric to the process that produced it, so when several schedulers send the same metric names, such as scheduler_heartbeat, the values overwrite each other and the dashboard shows one blended series that looks healthy even when one scheduler has stopped.
OpenTelemetry avoids this by attaching a service.instance.id to each process, so every scheduler produces its own series. Choose OpenTelemetry if you run more than one scheduler and need to see each one separately. Confirm the exact resource attributes your Airflow version emits by checking the metrics configuration documentation and a test run, since attribute names and where they appear in your backend depend on your Collector and storage setup. Once you have a test environment with two schedulers, stop one on purpose and confirm that its series goes stale while the other continues to report.
Per-process identity has a cost. Every scheduler instance adds a set of series for each metric and each label combination, including dag_id and task_id tags on metrics such as ti_failures. Many instances multiplied by many DAGs and tasks can grow your storage and query load quickly, so agree on a label policy before you enable it. Use allow and block lists to drop metrics you do not alert on, and ask your backend team what series count they can support. Also confirm that your dashboards filter by instance and aggregate deliberately, so a per-instance view does not become the default view for every panel.
Which metrics matter for alerting on Airflow
Most Airflow alerts come from a small set of signals: whether schedulers are alive, whether DAGs parse cleanly, whether tasks fail, and whether pools are saturated. Each signal points to a different part of the system, so design each alert around the question it answers rather than a single generic threshold.
Scheduler heartbeat
The scheduler_heartbeat metric shows that a scheduler process is still looping. Alert on the absence of the heartbeat for each scheduler, not on the aggregate, because one dead scheduler can hide behind a healthy peer. Set the window from the scheduler's observed loop cadence in your own environment, and confirm how your StatsD or OpenTelemetry setup labels each process before relying on per-instance alerts.
DAG import errors and parse time
dag_processing.import_errors counts DAG files that fail to load, and dag_processing.total_parse_time shows how long a full parsing pass takes. Alert on a sustained nonzero import error count, since a broken file can stop a pipeline from being scheduled at all. Trend parse time against a baseline you set for your environment, and treat a sudden jump after a deploy as a change signal.
Task failures by DAG and task
ti_failures, tagged by dag_id and task_id, shows where failures concentrate. Alert on a rate change for a specific DAG or task rather than on the total, so a noisy but expected failure does not drown out a new one. Keep the tag cardinality in mind when you allow or block metrics, because every distinct DAG and task pair becomes its own time series.
Pool slot saturation
pool.open_slots, pool.running_slots and pool.queued_slots describe how close each pool is to its limit. Sustained zero open slots combined with a growing queue usually means a pool is protecting a database or API correctly but is undersized for its demand. Alert on queue growth that persists over time, and check whether the queued tasks are waiting for slots or for executor capacity.
Missing metrics as a failure signal
A metric that stops arriving is often more serious than a metric with a bad value. Alert on the absence of expected series from the StatsD exporter or the OpenTelemetry Collector, including the heartbeat, which can go silent when the metrics pipeline breaks rather than Airflow. Confirm how your collector reports stale series, and test the alert by stopping one scheduler in a non-production environment.
Questions teams ask before building Airflow metrics
Should we choose StatsD or OpenTelemetry for Airflow metrics?
Choose OpenTelemetry when you run more than one scheduler, because StatsD cannot attribute metrics to the process that produced them, so schedulers overwrite each other. OpenTelemetry sets a service.instance.id for each process, which keeps series separate. StatsD remains a workable choice for a single scheduler with an existing StatsD pipeline. Set the choice in the [metrics] section, and confirm which exporter and backend your monitoring team already runs before committing. The Airflow metrics configuration lists the settings for each option.
How much effort does Airflow metrics work take, and what drives it?
Effort depends on how many schedulers, workers and DAGs you run, whether a metrics backend already exists, and how many alerts you need on day one. The core configuration is short. Most of the work sits in deciding which metrics feed which alerts, building dashboards, and testing that series stay separated across processes. Ask any partner for a written scope that lists the components, the environments and the alerts covered, since that scope sets the quote. No duration is given here because it depends on those factors in your environment.
What must our team provide before metrics work starts?
You need to provide four things:
- A metrics backend such as a Prometheus-compatible store, or an OpenTelemetry Collector path to your chosen backend.
- Network access from Airflow components to the StatsD host or the Collector endpoint, including firewall rules in each environment.
- Named owners for alerts, with an on-call rota that receives them.
- A list of schedulers, workers and DAG teams whose metrics you need.
OpenTelemetry Collector releases are licensed under Apache-2.0, so confirm the distribution your security team approves. Confirm each item with your own platform owners before work begins.
Can our in-house team set up Airflow metrics without a partner?
Yes, if your team already operates Airflow, a metrics backend and an alerting workflow. The configuration itself is documented in the Airflow metrics reference, and the OpenTelemetry Collector has published configuration docs. The skills gap is usually in tuning: deciding which series to keep, writing alert thresholds that match your scheduler behavior, and keeping the setup working across Airflow upgrades. A partner is useful when those skills are missing or when the work must fit into a scoped, written deliverable.
How do we govern custom metrics emitted from tasks?
Set a naming convention and an owner for each custom metric before tasks emit them. Tasks can emit custom metrics through the airflow.sdk.observability module with stats.incr, stats.gauge and stats.timing. Review those calls in code review like any other change, and keep tag values bounded so that no unique identifier becomes a label. Require each team to document the business meaning of its metrics so that dashboards and alerts do not depend on undocumented names.
How do we control metric volume and cardinality?
Use metrics_allow_list and metrics_block_list regular expressions to limit what Airflow emits. Start with the metrics that drive alerts, such as scheduler heartbeat, import errors and pool slots, then add others only when a team needs them. Watch tags like dag_id and task_id, because each unique combination creates a separate series. Review series counts after each new DAG wave, and ask your metrics backend team what retention and ingestion limits apply.
What changes after an Airflow upgrade, and what should we check?
Metric names, labels and available options can change between releases, so treat each upgrade as a change to your alerting. Airflow 3 is the current major line, and Airflow 2 reached end of life on April 22, 2026, so plan any upgrade on the vendor's published support table. After upgrading, compare emitted metric names against your alert rules, check that schedulers still report separate instances, and confirm dashboards still load data. Keep a test environment that runs the same metrics configuration before production.
Why the OpenTelemetry Collector is a neutral layer
OpenTelemetry is a framework, not a monitoring backend. It defines how telemetry is generated, exported and collected, but it does not store or visualize anything. The storage and dashboards are separate tools that you choose. For Airflow, this means the decision about where scheduler and task metrics end up can be made later, and changed later, without rewriting how Airflow emits them. The OpenTelemetry Collector is the component that sits between Airflow and those tools. It receives telemetry, processes it (for example, dropping or renaming series), and exports it to one or more destinations. The Collector is licensed under Apache-2.0, the same license as Airflow, so the layer itself carries no license fee for your team to plan around.
Run the Collector as an agent next to the schedulers, workers and API server, or as a gateway that accepts data from many sources, and decide which pattern fits your network before you pin a version. Confirm where your Collector distribution comes from and whether the components you need are contrib or core, because component stability varies and is documented in each component's README. The project publishes releases frequently (v0.161.0 on September 16, 2026, v0.162.0 on September 29, and v0.163.0 on October 16, 2026), so check the OpenTelemetry Collector releases page for the current release, its signed artifacts, checksums and SBOM files, and pin a version your change process can test and roll back. Authenticator extensions can protect the receiving endpoint and add credentials on export, which matters when metrics leave your network.
