Apache Airflow and Snowflake integration
Orchestrating Snowflake from self-hosted Airflow with the Snowflake provider: SQL API and stage-copy operators, Snowpark and Snowpark container jobs, connection setup and where credentials should live.
Apache Airflow integrations
Integration pages for the provider packages that connect Airflow to data warehouses, compute platforms, secrets stores, lineage tools and identity providers, based on the Airflow providers documentation.
Apache Airflow and Databricks integration
Running Databricks jobs, notebooks and SQL from Airflow with the Databricks provider: submit-run versus run-now, deferrable operators that free worker slots, and the DatabricksWorkflowTaskGroup for Databricks Workflows.
Apache Airflow and dbt Cloud integration
Triggering and monitoring dbt Cloud jobs from Airflow with DbtCloudRunJobOperator and DbtCloudJobRunSensor, deferrable waiting, artifact download, and what the provider does not cover, since it targets the hosted dbt Cloud service rather than dbt Core.
Apache Airflow and Apache Spark integration
Submitting Spark work from Airflow with SparkSubmitOperator, SparkSqlOperator, SparkJDBCOperator and PySparkOperator over Spark Connect, including the Spark connection setup each one needs and the JDBC transfer modes.
Apache Airflow KubernetesPodOperator setup
Running isolated task containers with the KubernetesPodOperator on any executor: pod specs, deferrable mode, cleanup actions, and handling secrets through Kubernetes Secrets or Airflow connections rather than environment variables. Separate from executor selection.
Apache Airflow and HashiCorp Vault secrets backend
Moving Airflow connections and variables out of the metadata database into HashiCorp Vault with the Vault secrets backend: KV mount layout, KV version 1 or 2, connection and variable paths, and multiple path lookups.
Apache Airflow data lineage with OpenLineage
Emitting data lineage from Airflow tasks with the OpenLineage provider so pipelines, datasets and runs can be traced in a lineage backend, and what a team needs to configure and operate around it.
Apache Airflow and Keycloak auth manager
Using the Keycloak auth manager in Airflow 3 to hold users, roles, groups and permissions in Keycloak, generating API tokens, and the caveats of an alpha component and multi-team clients. Narrower than the general SSO page.
Apache Airflow and Google BigQuery integration
Running BigQuery work from Airflow with the Google provider: BigQueryInsertJobOperator for queries and loads, durable job reconnects on retry, deferrable mode on the triggerer, table and partition sensors, and built-in check operators for data quality.
Apache Airflow with AWS: S3, Redshift, EMR, Glue and Athena
Orchestrating AWS data services from self-hosted Airflow with the Amazon provider: which services have operators, IAM connection design, S3 and Redshift transfers, EMR and Glue jobs, and running Airflow outside AWS while still driving AWS workloads.
Running Azure Data Factory pipelines from Apache Airflow
Triggering and monitoring Azure Data Factory pipelines from Airflow: AzureDataFactoryRunPipelineOperator, passing parameters, waiting on the triggerer instead of a worker, the run status sensor, and when to keep ADF for copy activities while Airflow owns the schedule.
Apache Airflow and Trino integration
Running Trino SQL from Airflow pipelines: the Trino provider and its common SQL dependency, SQLExecuteQueryOperator against Trino, GCS to Trino transfers, lineage through OpenLineage, and connection security for a shared Trino cluster.
Apache Airflow and Apache Kafka integration
Producing to and consuming from Kafka in Airflow with the Kafka provider, and starting DAGs from Kafka messages with asset watchers: offset handling, idempotent consumers, triggerer sizing, and when Kafka belongs in a streaming tool instead of Airflow.
Orchestrating Airbyte syncs with Apache Airflow
Triggering Airbyte connections from Airflow so ingestion and transformation run in one dependency graph: AirbyteTriggerSyncOperator in synchronous or asynchronous mode, AirbyteJobSensor, client credentials for self-hosted or cloud Airbyte, and retry safety.
Apache Airflow alerts to Slack, PagerDuty and Opsgenie
Routing Airflow failures and missed deadlines to the on-call channel: the Slack, PagerDuty and Opsgenie providers, notifiers on tasks and Deadline Alerts, avoiding alert floods from retries, and what each alert should link back to.
Apache Airflow SQL operators for PostgreSQL, MySQL and SQL Server
Running SQL against operational databases and warehouses from Airflow with SQLExecuteQueryOperator and the PostgreSQL, MySQL and Microsoft SQL Server providers: connections, transactions, splitting statements, returning results to XCom and keeping credentials out of DAG code.
Apache Airflow SFTP and SSH file transfers
Moving files between partner SFTP servers, legacy hosts and object storage with the Airflow SFTP and SSH providers: key-based connections, sensors that wait for drop files, host key checking, and running remote commands without opening broad network access.
Apache Airflow and ClickHouse integration
Loading and transforming data in ClickHouse from Airflow: the community ClickHouse provider, running SQL steps and checks, batching inserts from upstream systems, and keeping heavy transformation inside ClickHouse rather than on Airflow workers.
