Testing Apache Airflow DAGs in CI
A CI pipeline for DAG repositories: DAG loader tests, unit tests for tasks and structure, Ruff AIR checks, matching the scheduler environment, and blocking merges that would add import errors.
Apache Airflow DAG development and testing
Engineering practices for DAG code that stays reliable in production: testing and CI, staging promotion, dependency isolation, dynamic generation and data quality checks.
Promoting Airflow DAGs from staging to production
Running a staging Airflow that catches problems before production: parameterized paths and connections, environment-specific variables, versioned DAG bundles or images, and approval gates for regulated pipelines.
Handling conflicting Python dependencies in Airflow tasks
Running tasks that need libraries Airflow cannot share: PythonVirtualenvOperator, ExternalPythonOperator, DockerOperator and KubernetesPodOperator compared on startup cost, isolation and operational effort.
Generating Airflow DAGs dynamically from configuration
Building many similar DAGs from YAML or database config without slowing the parser: generation patterns, one DAG per file versus many, parse-time costs, and when dynamic task mapping is the better tool.
Apache Airflow data quality checks with SQL check operators
Stopping bad data inside the pipeline: column and table checks, value and threshold checks, tolerances and partitions, where checks sit in the DAG, and alerting when a check fails.
