Troubleshooting Airflow DAG import errors and missing DAGs

The metric to alert on is dag_processing.import_errors, which counts the files the DAG processor could not import. Any sustained value above zero means at least one DAG file is failing to load, and the broken DAG may be missing or stale in the UI. Alert when the value stays above zero for longer than a normal deploy window, so a brief error during a commit does not page anyone. Metric names and emission settings can differ by version and by StatsD or OpenTelemetry configuration, so confirm the exact name in your environment against the metrics configuration reference before you write the rule. Pair this alert with a check of the import error list in the UI or with airflow dags list to identify which file is failing.

Track dag_processing.total_parse_time, which reports how long a full parsing pass over your DAG files takes. A steadily rising value usually means top-level code is doing too much work, the number of DAG files has grown, or the parsing interval is too short for the file count. Chart the metric over days rather than alerting on single spikes, because a deploy that adds a few heavy files will move it briefly. Set the alert threshold from your own baseline after a few weeks of normal operation, and record the baseline in your runbook. Confirm the metric name and whether it is emitted by your DAG processor version before building the dashboard.

The scheduler_heartbeat metric shows that a scheduler process is alive and still looping. If it stops advancing, the scheduler is hung, crashed or unable to reach the metadata database, and no DAG will be scheduled even if parsing looks healthy. Alert on the absence of the heartbeat for a period longer than your normal loop interval, and run the alert against each scheduler when you operate more than one, because the scheduler documentation describes running several at once. Heartbeat tells you the scheduler is running, not that the DAGs it loaded are correct, so use it alongside the import error and parse time metrics.

Pool metrics help when the DAG loads without errors but its tasks never start. The pool metrics pool.open_slots, pool.running_slots and pool.queued_slots show whether tasks are waiting for capacity that another workload is holding. If pool.open_slots is zero while pool.queued_slots keeps growing, the pool is full and the tasks are waiting on it, not on parsing. Compare these values with the DAG's schedule and its concurrency settings before changing limits. Treat pool alerts as a separate rule from import alerts, so an on-call engineer can tell a parsing failure from a capacity problem at a glance.

Route the alert to the platform on-call first, and copy the DAG author team as a secondary recipient. Import failures often come from the environment rather than the DAG code, for example a missing Python dependency in the image, a broken DAG bundle sync, a database connection problem or a parser that has run out of time, and only the platform team can fix those quickly. The platform on-call can then identify the owning team from the failing file path and hand off with evidence. Authors who receive only email may not see a page at night, so decide the escalation path before the first incident and write it into the runbook.

Set thresholds from your own measured baseline, not from a number copied from another team. Collect the metrics for a representative period, note the normal import error count, typical parse time and heartbeat interval, then alert on sustained deviation from that baseline, such as an error count above zero for longer than a deploy window or a parse time well above its usual range. Use shorter windows for the heartbeat, because a silent scheduler stops all work, and longer windows for parse time trends. Review the thresholds after each Airflow upgrade or major change in DAG volume, since the metric behavior and volume can shift. Ask your team to confirm the chosen values in a test environment before they reach production paging.

Confirm the metric names and emission settings for the exact Airflow version you run, because Airflow 3 changed the component layout, with the DAG processor running as its own component. Airflow 3 is the current major line, and the version support table lists the lifecycle dates for each release line, so check that your version still receives fixes before you invest in a long alerting setup. Also confirm whether your deployment emits metrics through StatsD or OpenTelemetry, since that choice changes how the metric names appear in your monitoring system. Keep a short record of the version, the emitter and the metric names in your runbook so the alerts survive upgrades.

Yes, a team that already runs Prometheus, Grafana or another metrics system can set up these alerts internally, provided someone owns the metrics emitter configuration and the on-call runbook. The work is mostly confirming metric names for your version, routing the alerts, setting baselines and testing the escalation path. DeployOpen can plan and implement this monitoring as part of a fixed-scope engagement when your team prefers a partner, with the scope, deliverables and timeline agreed in writing before work starts. Ask any provider to name the Airflow version and the metrics backend they will configure, so the quote reflects your actual environment.

The time depends mostly on how many DAG files are affected and whether the cause sits in DAG code or in the platform. A single broken import in one file is usually a short investigation, while a fleet of files that time out during parsing takes longer because each cause needs its own fix and test. Other drivers include the number of teams that own DAGs, whether parsing settings need changing, and how much monitoring already exists to show which files fail. Ask for a written scope that lists the DAG files and environments in question, since that is what turns a rough sense of effort into a plan with dates.

Your team needs to provide access to the Airflow environment, the DAG repository and the deployment pipeline, along with the people who own each DAG. Someone who can approve changes to DAG code and someone who can change Airflow configuration are both needed, because some import errors are fixed in the DAG and others in settings such as dagbag_import_timeout. Logs from the DAG processor and the current import error list help narrow the work quickly. Confirm the owners, change windows and review process before work starts, because waiting on those is often the longest part of the timeline.

Often yes, if the team already runs Airflow 3 with the DAG processor as its own component and has access to its logs and metrics. Platform engineers can usually find the file and error behind an import error with airflow dags list and the DAG processor logs. The harder cases are duplicate DAG IDs, dependency conflicts across teams and parse timeouts across thousands of files, because they touch code owned by several groups. A partner is most useful when the team lacks time, when the same problem keeps returning, or when the fix needs a documented process that auditors can review.

A DAG code fix is usually narrower, since it changes one file, reviewed by the team that owns it. A platform fix changes shared settings, the DAG processor, dependencies in the Airflow image or the metadata database, so it affects every DAG and needs more testing and a wider change window. Cost follows that difference in scope, not a fixed rate. Ask a quote to separate the two, list which files and components each item changes, and state which items are one-time fixes and which need ongoing checks, such as an alert on the import error metric.

A fixed-scope engagement works well when the problem is bounded: a known set of failing DAG files, a defined number of environments and a clear list of deliverables. It is less suited to open-ended investigation, where the scope can only be set after the first findings. A common approach is a short, fixed investigation that produces a written list of causes, followed by a separately scoped fix. DeployOpen agrees scope, deliverables and timelines in writing before work starts, and ongoing checks can be covered by a retainer if your team wants continued support.

Parse time grows with the number of DAG files and with the work done at the top level of each file, such as database calls, API requests or heavy imports. Airflow's FAQ points to dagbag_import_timeout for files that exceed the limit, and for more than 1,000 DAG files it suggests setting file_parsing_sort_mode to modified_time and a higher min_file_process_interval, such as 600. Those settings change how often files are reparsed, so they need to be tested against how quickly your teams expect changes to appear. Track dag_processing.total_parse_time and dag_processing.import_errors before and after any change.

Several deployment choices change how import errors appear and how they are fixed:

• Whether DAGs are baked into the image or synced from a repository, since a broken sync can leave an old or partial file set in place.
• Whether more than one team shares an environment, which raises the chance of duplicate DAG IDs.
• Whether the DAG processor runs as its own component, which Airflow 3 requires.
• Whether monitoring exists for dag_processing.import_errors, so failures are seen before users report missing DAGs.

Write these answers down in the scope so the fix plan reflects your real setup.

Confirm the Airflow version in use, since Airflow 3 changed the DAG processor and API server model, and the Airflow version support table lists which releases still receive fixes. Gather the number of DAG files, the teams that own them, the deployment method, the metadata database type and the monitoring tools in place. Also confirm your change windows and who approves DAG and configuration changes. With those facts, ask the quote to state scope, deliverables, assumptions, owners and what happens after the fix, so you can compare options on the same basis.