How Airflow can trigger Airbyte syncs in one dependency graph
Airbyte moves data from source systems into your warehouse or lake, while Airflow decides when that movement happens and what must run after it. The supported method is the Airbyte provider's AirbyteTriggerSyncOperator, which starts a sync for an existing Airbyte connection by its UUID. When you need asynchronous monitoring, you pass the operator a job ID through asynchronous=True and follow it with AirbyteJobSensor, so the wait does not hold a worker slot. By default the operator waits for the sync to finish, which keeps a simple ingest-then-transform chain easy to read. This section explains how the pieces fit together. The rest of the page covers setup, credentials, retry safety and the points where a run can fail.
The main design decision is which side owns the schedule. If Airflow owns it, Airbyte connections should generally be triggered from DAGs and not run on their own timers, so that a failed load blocks the transformations that depend on it instead of letting them run against stale data. Confirm early whether any Airbyte connections are already scheduled inside Airbyte, because two schedulers firing the same connection is a common source of overlapping syncs. Also confirm which Airbyte deployment you run, self-hosted or Airbyte Cloud, since the connection UUIDs and the client credentials your DAGs use depend on it. Mapping each connection to one owning DAG before you write any operators prevents most of the confusion later.
What you need in place before wiring Airbyte to Airflow
Five things must exist before the Airflow DAG can start an Airbyte sync: a supported Airflow 3 deployment, a reachable Airbyte instance, a connection UUID, a client credential pair, and a clear license position for self-hosted Airbyte. Confirm each one with the team that owns it, because a gap in any of them shows up as a failed task rather than a clear error.
Airflow 3 deployment on a supported line
Run the Airbyte provider on an Airflow 3 deployment that is still receiving fixes. Airflow 3.3.2 is the current patch release in the supplied version data, and the prerequisites list Python 3.10 to 3.14 with PostgreSQL 14 to 18 or MySQL 8.0, 8.4 and Innovation releases as the metadata database. Confirm your Python and database versions against that list before installing the provider, since SQLite is not suitable for production. Check the metadata database first.
A reachable Airbyte instance
Decide whether the target is self-hosted Airbyte Core or Airbyte Cloud, because the network path and the credential source differ. Self-hosted Airbyte is normally deployed into a Kubernetes cluster with Helm, and its API endpoint must be exposed to the Airflow environment. Confirm the exact API base URL, the TLS certificate chain and any ingress rules with the team that operates Airbyte.
Network path from workers and triggerer
The operator calls the Airbyte API from the Airflow process that runs it, so the network path must exist from the Airflow workers, and from the triggerer when you use asynchronous mode with deferral. Open the route in firewall, security group and proxy rules, and test it from the same subnet and container image Airflow uses. A passing test from a laptop does not prove the worker can reach the API.
Connection UUID for each sync
The operator starts a sync for an existing Airbyte connection identified by its UUID, so the DAG needs that exact identifier rather than a connection name. Record one UUID per connection in your configuration system, and confirm that each one points to the production source and destination you expect. Keep the UUID in a variable or secrets backend, not hard-coded in DAG files, so promotion between environments does not require code edits.
Client ID and secret for the trigger account
Both Airbyte Cloud and Airbyte OSS authenticate the provider with a client_id and client_secret. Create a dedicated account or application for Airflow, grant it access only to the connections it must trigger, and store the pair in an Airflow connection backed by your secrets store. Ask who rotates the secret and how a rotation reaches every worker and triggerer at the same time.
License position for self-hosted Airbyte
Airbyte Core is licensed under the Elastic License 2.0, which permits internal operations and client work for a data agency but forbids hosting Airbyte as a managed service or exposing its UI or API to customers. Airbyte's connectors and Connector Development Kit are MIT licensed. Have legal review the license FAQ against your intended use before you run Airbyte for other business units or customers.
How to configure the Airbyte connection and first DAG
This sequence takes a working Airbyte connection and turns it into an Airflow task that starts the sync, waits for it, and reports the result inside one dependency graph.
Install the Airbyte provider package
The platform team adds the Airbyte provider package to the Airflow image or environment, using the same dependency pinning process used for other providers so the Airflow version and provider version stay compatible. Build the image or environment through your normal change process, not by installing into a running scheduler. The step is done when a DAG file can import AirbyteTriggerSyncOperator and the Airflow import errors view shows no failures for that file. Confirm the provider version against the Airbyte provider documentation before pinning it.
Create the Airflow connection with client credentials
An administrator creates an Airbyte connection in Airflow that points to your Airbyte API host and stores the client_id and client_secret for the Airbyte application that will run syncs. Use a dedicated Airbyte service identity with permission only on the connections it needs, and store the secret in your secrets backend rather than in DAG code. The step is done when a connection test in the Airflow UI succeeds. If the test fails, check the host URL, the path your deployment exposes, and whether the credentials belong to the same Airbyte instance.
Identify the connection UUID in Airbyte
In the Airbyte interface, open the connection you want Airflow to run and copy its connection ID, which is a UUID. The operator starts syncs by this UUID, so a copy error or a connection that was recreated in another workspace will cause the wrong sync to run or the task to fail. Record the UUID in your DAG configuration or Airflow Variable, and note which environment it belongs to. The step is done when a single manual sync started from Airflow appears under that connection in the Airbyte job history.
Add AirbyteTriggerSyncOperator in synchronous mode
The data engineer writes a first DAG with one AirbyteTriggerSyncOperator task that references the Airflow connection and the connection UUID. In the default synchronous mode, the operator starts the sync and keeps the task running until Airbyte reports that the job has finished. Trigger the DAG manually in a non-production environment. The step is done when the task turns green and the task log shows the Airbyte job reaching a successful final state. Keep this version until you have confirmed the job history matches what Airflow reported.
Check the job ID returned to XCom
Before changing modes, the engineer confirms what the operator passes downstream. In the task's XCom tab, look for the Airbyte job ID returned by the operator, and match it to the same job in the Airbyte job history. This value is what the sensor needs in the next step, so a missing or mismatched ID means the downstream wiring will fail. The step is done when a downstream task can read the job ID from XCom in a test run.
Switch to asynchronous mode with the operator
The engineer sets asynchronous=True on the AirbyteTriggerSyncOperator. The task now starts the sync and returns the job ID without waiting for completion, so the worker slot is released immediately. Confirm that the trigger task finishes quickly and that the job continues running in Airbyte. The provider documentation also warns that the operator does not guarantee idempotency if triggered again, so review retry settings on this task before promoting the DAG. The step is done when the job runs in Airbyte after the trigger task has already succeeded.
Add AirbyteJobSensor to monitor the job
The engineer adds an AirbyteJobSensor task after the trigger task and passes it the job ID from XCom. The sensor polls Airbyte until the job reaches a terminal state, so downstream transformation tasks only start after ingestion has finished. Set the sensor so that a failed Airbyte job fails the Airflow task, and make transformation tasks depend on the sensor, not on the trigger. The step is done when the sensor moves to success after a real sync, and a deliberately failed test sync causes the sensor to fail and blocks the downstream task. Record both results before you rely on the graph.
Confirm the graph in a test run
Run the complete DAG in a non-production environment with the trigger, sensor and at least one downstream transformation task in place. Review the run in the Airflow UI and compare each task state with the Airbyte job history for the same run. The DAG is ready for a wider rollout only when ingestion and transformation order is correct, retries do not start duplicate syncs, and failure in Airbyte stops the downstream work. Document the connection UUID, credential owner and retry settings so the operations team can support the DAG after handover.
Why synchronous and asynchronous modes behave differently
In synchronous mode, AirbyteTriggerSyncOperator holds its worker slot until the Airbyte sync reaches a final state, so the Airflow task is only as cheap as the time the sync takes. That is acceptable for a small pipeline with one or two connections, a short sync window and no other work queued behind it on the same pool. The operator then reports success or failure directly, and the downstream transformation task can start with no extra wiring. Confirm the expected sync duration for each connection and the size of your worker pool before accepting that trade-off, because a long sync that blocks a slot can starve unrelated DAGs.
With asynchronous=True, the operator starts the sync, returns the Airbyte job ID as its result and finishes. Waiting moves to a separate task, AirbyteJobSensor, which polls that job ID until Airbyte reports completion. Trigger and wait are split into two tasks, so the dependency graph shows each stage, retries can target the wait without starting a second sync, and a failed wait does not automatically mean the sync was started twice. Pass the job ID from the trigger task to the sensor through XCom, and set the sensor's poll interval and timeout deliberately, since a sensor that runs too often adds load and one that times out too early reports false failures.
Plan the sensor pattern for production pipelines that share a worker pool or run many connections in parallel. The provider documentation warns that the operator does not guarantee idempotency if it is triggered again, so the retry behavior of the trigger task needs its own review, and the sensor is the safer place to retry. Ask your Airbyte administrator which API identity owns the connection, and confirm how your Airflow version handles sensor waiting before you choose the pattern for each pipeline. The Airbyte provider documentation for the operator lists the parameters that apply to each mode.
Which Airbyte identifiers map to which Airflow objects
This table shows where each Airbyte identifier and credential lands in the Airflow DAG, connection and task configuration. Read it when wiring a new sync, so the connection UUID, job ID, client credentials and task names stay in the right place. The main takeaway is that the client secret belongs in Airflow's secret store, while the UUID and job ID are not secrets but still need consistent naming.
| Airbyte identifier | Airflow object | How it is used | Handling |
|---|---|---|---|
| Connection UUID | connection_id parameter of AirbyteTriggerSyncOperator | Names the existing Airbyte connection that the operator starts a sync for | Not secret; keep it in DAG code or a variable so changes are reviewed |
| client_id | Login field of the Airbyte Airflow connection | Identifies the API application that authenticates the operator to Airbyte | Sensitive; store in Airflow's connection record or a secrets backend |
| client_secret | Password field of the Airbyte Airflow connection | Lets the operator obtain access to Airbyte Cloud or self-hosted OSS | Secret; never in DAG code or environment variables, use a secrets backend |
| Returned job ID | job_id input of AirbyteJobSensor, read via XCom | Produced when asynchronous=True; the sensor polls this job until it finishes | Not secret; passes between tasks through XCom in the metadata database |
| Airbyte connection name | DAG ID and task_id naming | Human-readable label that helps operators match a task to the Airbyte connection | Not secret; agree a naming convention so alerts and logs are easy to trace |
Securing client credentials and Airbyte API access
The Airflow operators call the Airbyte API with a client_id and client_secret, so anyone who can read those values or reach the API can start syncs. The cards below cover where the credentials live, how far the Airbyte application reaches, who can change Airflow connections, and what the network allows between the two systems.
Keep credentials out of DAG code
Store the client_id and client_secret in an Airflow connection, and back that connection with a secrets store your security team already runs rather than the metadata database. DAG files are read by many authors and often end up in Git history, so a credential written into a DAG should be treated as exposed. Check that the connection is referenced by its conn_id only, and that no environment variable or log line carries the secret.
Scope the Airbyte application
Confirm in Airbyte how the application behind the client credentials is granted access, and limit it to the workspaces and connections the Airflow DAGs actually trigger. A credential that can start every connection in a workspace lets one compromised DAG or connection edit trigger unrelated syncs. Ask Airbyte for the current permission model for your edition, because the grant options differ between self-managed and cloud deployments.
Limit who can edit Airflow connections
Anyone who can edit an Airflow connection can usually repoint it at a different host or read the stored secret, so connection editing is a privileged action. Use Airflow's access controls to restrict connection management to a small group, and review changes to Airbyte connections as you would review code. Confirm how your chosen auth manager maps roles to connection permissions before relying on it.
Restrict network paths to the Airbyte API
Allow traffic from Airflow workers and the triggerer only to the Airbyte API endpoint, and block other paths to it. Because asynchronous runs use a sensor or the triggerer to poll job status, confirm that every component that polls can reach the API, and that the allowed route does not expose the Airbyte UI to the Airflow network. Document the source addresses so that a firewall or policy change does not silently stop polling.
Mask secrets in logs
Confirm in the Airflow documentation for your version that connection passwords and similar sensitive values are masked in task logs, and test this with a deliberately failing task before trusting it. Masking depends on the value being registered as a secret, so a secret that is copied into a custom field, a request body or an error message may still appear. Check the log storage location and its access rules as well.
Plan client secret rotation
Plan how the client secret is rotated in Airbyte and then updated in the secrets store, without editing DAG code. Confirm in the Airbyte documentation for your environment whether an existing secret can be replaced in place, whether old and new secrets can overlap during the change, and what happens to in-flight sync jobs. Record the owner, the schedule and the Airflow connection that must be updated in the same change window.
Making Airbyte retries safe for downstream transformations
The provider does not guarantee idempotency if AirbyteTriggerSyncOperator is triggered again, so a retry of the trigger task can start a second sync on the same connection. Treat every trigger retry as a possible duplicate run. Before the first deployment, decide what a second sync would do to your destination: append-only tables would receive duplicate rows, while deduplicating or full-refresh loads would absorb the repeat. Confirm the exact behavior of your Airbyte connection's sync mode and destination with your Airbyte administrator, since the answer determines whether the retry policy can stay at the Airflow default.
Guard the trigger with a check for an already running job before starting a new one. In practice, this means querying the Airbyte API for the connection's current job status from a short task placed ahead of the trigger, or branching so the trigger runs only when no job is active. Confirm the exact API calls and authentication with the Airbyte provider documentation and your Airbyte version, because the client credentials used by the provider must have permission to read job state. Where a previous job is still running, skip or wait rather than start a second sync.
Separate the retry policy of the trigger from the retry policy of the sensor. When asynchronous=True is used, the trigger returns a job ID and AirbyteJobSensor monitors it, so a sensor retry only re-polls an existing job and is generally safe. A trigger retry is the risky case. Set the trigger task to fewer retries than the sensor, and sequence every downstream transformation only after the sensor reports success, so dbt or SQL steps never run against a partially loaded sync. Test both paths in a non-production environment by forcing a failure at each task before relying on them.
Failures to expect when Airflow drives Airbyte
This table lists the failures teams most often hit when an Airflow task starts or waits on an Airbyte sync, with the likely cause and a first fix for each. Read it before the first production run, then confirm each fix against your own Airbyte URL, credentials and network paths. The main takeaway is that most failures trace back to connection identity, credentials, network reachability or retry behavior, and each has a specific check.
| Failure | Likely cause | Fix to check first |
|---|---|---|
| Operator cannot find the connection | The connection UUID is wrong, copied from another workspace, or the connection was deleted and recreated with a new ID | Copy the connection UUID from the Airbyte UI for the connection being synced, then update the DAG or variable that holds it |
| Authentication rejected | The client_id or client_secret in the Airflow connection is wrong, expired, or belongs to a different Airbyte instance than the host field | Recreate the client credentials in Airbyte, store them in the Airflow connection or secrets backend, and test with a single trigger |
| Connection timeout to the Airbyte API | Workers or the triggerer cannot route to the Airbyte API host or port, often because of firewall rules, DNS or a proxy | Test HTTP reachability from the same host and network namespace that runs the task, not from your laptop |
| Duplicate syncs after a retry | The provider docs say the operator does not guarantee idempotency, so a retried task can start a second sync while the first still runs | Set retries deliberately, check for an active job before triggering, and keep downstream steps tolerant of a repeated load |
| Sensor times out on a long sync | The sensor's timeout is shorter than the sync duration, or the sync is slower than the window you planned | Set the sensor timeout from observed sync durations, and confirm the sensor mode suits your worker capacity |
| Self-hosted Airbyte not reachable at the connection URL | The host in the Airflow connection is an internal name that resolves only inside the Airbyte cluster, or the ingress path differs from the URL configured | Confirm the externally reachable URL in your Airbyte deployment settings and use that exact host and path in the Airflow connection |
| Job status never completes | The asynchronous job ID is passed to the wrong sensor, or the sensor points at a different Airbyte instance from the one that issued the job | Pass the job ID returned by the same connection and host, and log both values so they can be matched during investigation |
Questions teams ask before connecting Airbyte to Airflow
What does the effort to connect Airbyte to Airflow depend on?
The effort depends on how many Airbyte connections you need to trigger, how Airbyte is deployed, how retries should behave, and how much testing the pipeline needs before it runs in production. A few connections triggered by one DAG are a small job. Dozens of connections with downstream transformations, alerting and separate staging and production environments add design and test work. Retry design deserves early attention because the Airbyte provider docs warn that the operator does not guarantee idempotency if triggered again, so a retried task could start a second sync. Ask any partner to walk through each of these four drivers against your actual connection list before agreeing on a fixed scope.
Can one internal team connect Airbyte to Airflow without outside help?
Yes, a team with working Airflow and Airbyte experience can usually do the core setup alone. The core steps are creating a client ID and secret in Airbyte, storing them as an Airflow connection, writing a DAG that uses AirbyteTriggerSyncOperator, and adding AirbyteJobSensor where asynchronous mode is used. Outside help tends to add the most value on retry safety, secrets handling, testing and operating the result over time. Whether to bring in a partner depends on how many people on your team have run Airflow in production and whether anyone owns the Airbyte deployment. A written scope lets you compare the work against your own capacity.
What does the work need from our side before it starts?
The work needs four things from your side. First, administrator access to the Airbyte instance so the connections to be triggered can be identified and a client credential created. Second, a named owner for those credentials who can rotate them and approve their use. Third, access to the Airflow environment, including permission to add connections, deploy DAG code and view task logs. Fourth, the list of Airbyte connection UUIDs and the downstream transformations that depend on each sync. Agree on these items in writing before work starts so that delays do not come from waiting for access.
How long does the integration take, and what changes the timeline?
Duration depends on scope rather than on a standard number of days. Timing is driven by the number of connections, whether asynchronous monitoring with sensors is needed, how many environments must be covered, the retry and failure-handling rules agreed with your team, and the testing and change approval steps your organization requires. Your Airflow and Airbyte environments also matter, especially if access requests or network changes take time. No duration is stated here, so ask for a timeline tied to a written scope with named deliverables and the access you need to provide.
Does a self-hosted Airbyte Core deployment change the setup?
The Airflow side stays the same, but the Airbyte side changes. The Airbyte deployment guide covers the open Core version for self-managed installations and recommends deploying with Helm and its documented chart values, with abctl and Docker documented for single-machine and local setups. Your Airflow DAGs reach Airbyte through its API using a client ID and client secret, the same authentication method used for Airbyte Cloud. What you must confirm is the network path from Airflow workers or the triggerer to the Airbyte API endpoint, the TLS certificates in use, and who upgrades Airbyte and when. Those choices affect reliability more than the operator code does.
What does the Elastic License 2.0 mean for internal use of Airbyte?
Under the Elastic License 2.0, your organization may use Airbyte Core for internal operations, which covers moving your own data into your warehouse and orchestrating those syncs from Airflow. The license also allows building applications powered by Airbyte data and using it for client work as a data agency. It forbids providing Airbyte to third parties as a hosted or managed service, selling it as an ELT tool, exposing its UI or API to customers, and circumventing license key functionality. Airbyte's connectors and Connector Development Kit are under the MIT license. Have your legal or licensing team confirm how your planned use fits these terms before deployment.
What should we check about retries before scheduling Airbyte syncs from Airflow?
Check retries first, because a retried Airflow task can start a second Airbyte sync for the same connection. The Airbyte provider docs say the operator does not guarantee idempotency if triggered again. Decide whether a retry should start a new sync, wait for the existing job, or skip when a sync is already running, and test each path. Also decide how failures reach your on-call channel, and confirm that task logs show the Airbyte job ID so engineers can trace a run. These decisions shape the DAG design more than the choice of operator does.
How DeployOpen scopes and delivers an Airbyte and Airflow integration
DeployOpen scopes the Airbyte and Airflow integration in a written, fixed-scope agreement before any work starts. The agreement names the Airbyte connections in scope, the Airflow DAGs that will trigger or wait on them, the connection objects and secret locations Airflow will use, the test plan that proves each DAG behaves correctly on retry, and the handover items the client's team will receive. Each item is written so the client can check it against its own environment. Ask for the connection inventory early, including which Airbyte deployment each connection lives in, because a self-hosted Airbyte instance and an Airbyte Cloud workspace need different client credentials and different network paths from Airflow workers.
DeployOpen works inside the client's own infrastructure, so the Airflow environment, the Airbyte deployment, the secrets store and the network routes between them all stay under the client's control. DeployOpen does not publish a price list. A quote follows from the written scope, and the scope items above are what drive it: the number of connections and DAGs, whether the client needs asynchronous sensors or synchronous waits, how many environments must be covered, and how much credential and network work sits between the two systems. Ask the vendor to show each cost component separately, and to mark which items are one-time build work and which recur.
Ongoing support, upgrades and continuous testing are available as a retainer after handover. That matters for this integration because Airbyte and Airflow both release on their own schedules, and a change on either side can alter connection IDs, provider package behavior or authentication. Before signing, confirm with DeployOpen who owns each upgrade decision, how a failed sync is escalated to the client's on-call team, and what the handover pack contains for operating the DAGs without DeployOpen present.
