Open source data pipelines: which stack fits your team
Most data teams start with data spread across SaaS tools, operational databases and file drops, and move it with hand-written scripts or a commercial ETL product that cannot run inside the organization's own boundary. Apache Airflow, Dagster and Airbyte cover ingestion and orchestration, and the right pick depends on whether your team writes Python-first pipelines, needs managed connector sync, or both. Airflow (Apache-2.0, 3.3.2 published September 17, 2026) and Dagster (Apache-2.0, 1.13.26 published October 9, 2026) are code-first orchestrators that schedule and monitor workflows your engineers author. Airbyte handles connector-based sync from many sources, but the platform repository is licensed under the Elastic License 2.0, so confirm what that license permits for your intended deployment before you commit.
This stack fits regulated organizations that must keep source extracts, staging data and credentials in their own cloud accounts, data centers or air-gapped networks. Deciding which orchestrator to run is only part of the work. Ask your team three questions: how many pipelines will be written by data engineers versus analysts, how many sources need prebuilt connectors versus custom code, and who will be paged when a sync fails at night. Teams with a small number of hand-built pipelines often start with Airflow or Dagster alone. Teams with many SaaS sources often pair an orchestrator with Airbyte. Confirm connector coverage for your specific sources against each project's current documentation, since connector support changes between releases.
DeployOpen implements these platforms inside your environment and does not sell them as a hosted service. The first engagement step is a written scope that names the orchestrator, connector set, the environments that will run them and the people who will own them after handover. Use that scope to test each option against your data residency rules and your team's skills. Ask any implementation partner to show how they will install, configure and back up the orchestrator and connector runtime in your accounts, and to state in writing which components they will hand over and which they will keep supporting.
Requirements to agree before choosing a pipeline platform
Pipeline platforms differ most in how they schedule work, who writes it, and where data is allowed to go. Agree these six points with data owners, security and the teams who will run the pipelines before comparing Airflow, Dagster and Airbyte, so the platform choice follows from real constraints.
Source inventory and volumes
List every source system, its owner, its change rate and the rough size of each feed, including SaaS APIs with rate limits and databases with change data capture options. Connector coverage differs widely between Airbyte and custom code, and a source with no maintained connector moves the work from configuration to engineering. Ask each owner which fields are sensitive and whether the source can expose incremental change markers.
Latency: batch or near real time
Decide which datasets must refresh on a fixed schedule and which must arrive within minutes. Scheduled batch runs suit most orchestrators, while near-real-time feeds usually need streaming tools or event-driven triggers alongside them. Write down the freshness each consumer expects, because a dashboard refreshed daily and a fraud signal have very different platform requirements.
Data residency and access
Map which datasets may leave a region, which may be seen by contractors, and which carry personal or regulated data. Self-hosting keeps data in your accounts, but logs, staging tables, connector credentials and metadata stores all hold copies that need the same controls. Agree row and column access rules before the first pipeline is built, not after.
Team language skills
Establish whether builders write Python, SQL or low-code configuration. Airflow and Dagster are authored as code, which suits data engineers who test and review pipelines in version control. Airbyte shifts much of the work to configuration, but custom connectors still require development. The skill mix decides who can change a pipeline during an incident and how much training the rollout needs.
Target storage or warehouse
Name the destination systems, such as a lakehouse, a cloud warehouse or an operational database, and confirm which write patterns each supports. Some connectors load into a specific warehouse format, and orchestration tools need credentials and network paths to every target. Choosing the destination early also determines where lineage and catalog metadata must be captured.
On-call and operating model
Decide who gets paged when a nightly load fails, who owns each pipeline after handover, and what the response window is. Open source platforms need upgrades, scheduler capacity planning, secret rotation and backup testing, and those tasks must sit with a named internal team or a support contract. Ask any implementation partner which parts they will run, which they will hand over and what documentation each handover includes.
Airflow, Dagster and Airbyte: when each one fits
This table compares Apache Airflow, Dagster and Airbyte by typical job, licensing checks, team skills and operating effort. Read each row as a question to answer for your own team, since the right choice depends more on who will maintain the pipelines than on feature lists. The main takeaway is that Airflow and Dagster orchestrate pipelines you write, while Airbyte moves data between systems through connectors, so many teams need one of each.
| Dimension | Apache Airflow | Dagster | Airbyte |
|---|---|---|---|
| Typical job | Schedule and monitor code-defined batch workflows across many systems | Orchestrate assets and data products with typed, testable pipeline code | Move data from SaaS apps and databases into warehouses or lakes using connectors |
| License and what to check | Apache-2.0. Confirm the license of any provider packages and third-party operators you add | Apache-2.0. Confirm the license of integrations and any commercial add-ons you adopt | Platform repository under Elastic License 2.0. Confirm the license of each connector separately |
| Current release noted in research | 3.3.2, published September 17, 2026 | 1.13.26, published October 9, 2026 | Confirm the current release and connector versions in your environment |
| Team skills required | Python, DAG design, container or Kubernetes operations for workers | Python, software engineering practices, asset-based modeling | Connector configuration, source credentials, warehouse loading and schema change handling |
| Operating effort | Scheduler, webserver, workers and metadata database to run and upgrade | Webserver, daemon, run launcher and storage to run and upgrade | Platform deployment plus connector upgrades and sync monitoring for each source |
| Fits best when | Many heterogeneous jobs need a mature scheduler and wide operator coverage | Teams want strong testing, lineage-aware assets and a code-first developer experience | Source systems are many and standard connectors cover most of them |
| Works poorly when | Teams need low-code data movement or do not want to write Python | Your main need is moving SaaS data with no custom transformation logic | Sources are custom or proprietary, or connectors lack the fields you need |
| Self-hosted caveat | Self-hosting is the usual pattern, so backups and upgrade windows are your responsibility | Same: you own the deployment, secrets and run storage | A self-hosted Airbyte deployment is not the same as the hosted service. Confirm support scope and upgrade path |
How the reference architecture connects its parts
A data pipeline stack is a chain of components that each own one job: pulling data from sources, moving it, scheduling and retrying work, storing it, protecting credentials, and telling operators when something breaks. The decisions below are the ones your team must make before a quote can be scoped, because each choice changes who operates the stack later.
Source connectors
Connectors read from SaaS applications, databases and files. Each connector carries its own API limits, schema drift behavior and credential type, so connector count matters less than whether each one you need is maintained. Decide whether to use prebuilt connectors, write custom ones, or accept batch extracts for sources without stable APIs. Check the license of any connector platform you adopt, since Airbyte's platform repository is licensed under the Elastic License 2.0, which affects how you may offer it to others.
Ingestion layer
The ingestion layer copies raw data from sources into the landing zone on a schedule or change feed. Decide whether ingestion is full refresh or incremental, and who owns the cursor and state that tracks progress. Incremental loads reduce load on source systems but need reliable state storage, so confirm where that state lives and how it is backed up.
Orchestrator for scheduling and retries
The orchestrator decides when each job runs, in what order, and how it retries after failure. Apache Airflow 3.3.2 (Apache-2.0) and Dagster 1.13.26 (Apache-2.0) both fit this role, with different authoring models and asset concepts. Decide which one your teams can write and debug, and define retry limits so a failing source does not flood downstream systems.
Landing zone and warehouse
The landing zone holds raw files or tables, and the warehouse holds modeled data for analysts and applications. A key decision is whether transformations run inside the warehouse with SQL tooling or in the orchestrator's compute. Confirm storage location, encryption, retention periods for raw personal data, and which team owns schema changes.
Secrets store
Connectors and orchestrators need database passwords, API tokens and cloud keys. Decide whether secrets live in a dedicated secrets manager that the orchestrator reads at runtime, or in the orchestrator's own configuration store. Keep secrets out of DAG code and connector configuration files, rotate them on a schedule, and confirm that each pipeline has its own least-privilege identity.
Monitoring, alerting and catalog
Monitoring covers job status, freshness, row counts and failed runs, with alerts routed to a team that can act. Decide which signals page an on-call engineer and which create a ticket. Pipeline metadata, lineage and ownership should feed a metadata catalog such as those covered on our open source data catalog page, so consumers can see where data came from and who to ask.
Build in-house, partner or buy SaaS for pipelines
This table compares three ways to run ingestion and orchestration: building on open-source tools with your own staff, implementing open source with an implementation partner, or buying a commercial managed SaaS service. Read each row as the question to settle before choosing, and note that the cost drivers in the cells apply to every column in different ways. The main takeaway is that the open-source options shift cost from license fees to people and infrastructure, so the right model depends on who will carry pipeline on-call and upgrades after handover.
| Dimension | In-house on open source | Open source with a partner | Commercial managed SaaS |
|---|---|---|---|
| Control over data location | Pipelines, connectors and logs run in your cloud accounts or data centers, and you decide where staging data and credentials sit. | Platforms run in your environment under your accounts, with the partner working inside agreed access paths and documented boundaries. | Data passes through the vendor's hosted control plane and often its processing regions. Confirm regions, subprocessors and where logs and staging copies are kept. |
| Who writes and maintains connectors | Your engineers write custom connectors and maintain them as source APIs change. Confirm the license of any prebuilt connector before reuse. | The partner can build, test and document custom connectors and hand over code, runbooks and ownership. Ongoing connector upkeep depends on the support scope agreed in writing. | The vendor supplies a catalog of managed connectors. Coverage for your specific sources, and the speed of fixes for source changes, should be confirmed in writing. |
| On-call coverage | Coverage is internal and must be staffed around the clock if business users depend on overnight loads. Define severity levels and escalation before go-live. | Coverage is defined by a retainer or support agreement, which can include monitoring and response windows. Confirm hours, response targets and who holds the pager after handover. | The vendor typically provides platform-level incident response. Your team still owns source-side failures, schema drift and downstream impact. Confirm the split. |
| Upgrade effort | Your team plans and tests every release of the scheduler, ingestion platform and database, including breaking changes between major versions. | The partner plans upgrades, tests them in a staging copy and schedules them, often under a retainer. Upgrade scope should be stated in the statement of work. | Upgrades are applied by the vendor on its schedule. Your team validates pipelines against changes and may have limited control over timing. |
| Cost structure | Mostly one-time build effort plus recurring infrastructure and staff time. Costs rise with number of sources, data volume, job frequency and the depth of monitoring you choose to run. | One-time implementation fees for design, build and handover, plus recurring support and infrastructure. Ask for the cost of each component separately in the quote. | Mostly recurring subscription or consumption charges, often metered by data volume, rows or connector runs. Ask how overage, backfills and failed-run retries are billed. |
| Main cost drivers | Team skills, number of sources, data volume, job frequency and monitoring depth. Skills gaps usually cost more than infrastructure in the early months. | Scope of sources and transformations, volume, schedule complexity and the depth of monitoring and documentation required by your controls. | Metered volume, connector count, sync frequency and enterprise features such as private networking or audit logs. Model cost at your expected peak, not your average. |
| When it fits | You have data engineers who will own pipelines long term, a small and stable set of sources, and the appetite to maintain upgrades. | You want open-source control and portability but need a delivery team for design, handover and regular upkeep, with clear support terms. | You have few engineers, need fast coverage for common SaaS sources, and accept vendor-hosted processing under contract terms your legal team has reviewed. |
How a pipeline rollout runs from pilot to handover
The sequence moves from an agreed source inventory to a proof-of-concept pipeline, then to hardened production onboarding by domain, and ends with runbooks and named ownership so your team can operate the pipelines after handover. Each phase closes with a deliverable you can accept or reject in writing.
Inventory sources and agree scope
DeployOpen works with your data owners to list every candidate source system, its access method, data sensitivity, expected change frequency and the downstream consumers that depend on it. The output is a source inventory and a written scope that names which sources are in the first phase, which destinations are targeted, and what is explicitly excluded. The step is complete when the data owners and your security team have signed off on the inventory and the scope document. Timing follows the agreed written scope.
Build a proof-of-concept pipeline
The team deploys the chosen orchestrator, such as Airflow or Dagster, and one ingestion connector, such as Airbyte, in your environment with one source and one destination. The pipeline runs on a schedule, writes to a staging area and records run history. The deliverable is a working pipeline, a short architecture note, and a test report showing successful runs, failed runs and how each failure was surfaced. Confirm the connector license for your intended use before this step begins.
Harden secrets, retries and alerting
Before any production data moves, the team moves credentials into a managed secret store, sets retry and backoff rules per task, and configures alerts for failed runs, late runs and schema drift. Backup and restore for the orchestrator metadata database and the staging storage is tested by restoring to a separate environment. The deliverable is a hardening record listing each control, its configuration location and the restore test result. Your security team reviews the record before the step closes.
Onboard production sources by domain
Sources are added in groups aligned to business domains, such as finance or customer operations, one domain at a time. Each group goes through the same checks: access approval, data classification, a test load, reconciliation of row counts or control totals against the source, and sign-off from the domain data owner. The deliverable for each domain is a onboarding record with the source list, reconciliation results and the named owner who accepts the data. Domains that fail reconciliation stay in staging until the gap is explained.
Hand over runbooks and ownership
DeployOpen delivers runbooks covering how to add a source, rotate a credential, rerun a failed task, backfill a date range, upgrade the orchestrator and restore from backup. Each pipeline and alert has a named team responsible for it, agreed in writing. Your engineers complete a supervised run of the runbooks, with DeployOpen observing, before handover is recorded. The deliverable is the runbook set, an ownership matrix and a handover sign-off document. Ongoing support, upgrades and continuous operation are available as separate retainers if you want them.
Systems the pipelines must connect to and what each one involves
A pipeline is only as safe as its weakest connection. Each system below needs its own access model, credential plan and test set before it carries production data, and the same orchestration layer must handle all of them.
Relational databases and change capture
Operational databases are usually the most sensitive sources. Incremental loads depend on timestamps or on change data capture from the transaction log, and each approach needs a read-only replica or a dedicated service account with the narrowest grants. Check how the database handles log retention, how long a stalled pipeline can fall behind before the log is purged, and how schema changes will be detected and surfaced to the team.
SaaS applications and API quotas
Business applications expose data through vendor APIs that impose rate limits, pagination rules and token lifetimes. Authentication is often OAuth with refresh tokens tied to a named user, so offboarding that user can silently stop every sync. Test against a sandbox tenant, confirm how the vendor reports deleted records, and document who owns each integration token and when it expires.
File and object storage
Object stores and file shares feed batch pipelines with CSV, Parquet and document drops. The risks are overly broad bucket policies, unencrypted landing zones and files that arrive partially written. Use scoped roles with short-lived credentials where the platform supports them, define a naming and manifest convention so loads are complete, and test what happens when a file is malformed or duplicated.
Identity and secrets managers
Connectors need passwords, keys and tokens, and these should never sit in DAG code or connector configuration in plain text. A secrets manager, whether self-hosted or provided by the cloud, should issue and rotate credentials, while the orchestrator reads them at runtime. Confirm the rotation schedule for each source, test that a rotated secret is picked up without a redeploy, and make sure the pipeline's service identity is reviewed like any other privileged account.
Warehouse or lakehouse destinations
The destination defines the cost model, the write semantics and the permission boundary. Confirm which tables the pipeline account can create, overwrite or drop, how merges and late-arriving records are handled, and whether the platform supports schema evolution without breaking downstream queries. Test restore from a failed partial load and verify that row counts and checksums match the source before any consumer reads the data.
Alerting, ticketing and chat channels
Failure alerts only help if they reach someone who can act. Route pipeline failures to a ticketing queue with an owner and a runbook link, and send only summarized notices to chat so that channel membership does not become a data access path. Store webhook URLs as secrets, rotate them with the other credentials, and test that a forced failure creates a ticket, pages the right group and does not leak record contents.
What pipeline evidence supports data protection and audit work
A pipeline platform can produce much of the evidence that data protection and audit work asks for, but only if the team configures it to keep that evidence. The most useful records are run histories that show what ran, when, with which parameters and what it returned, plus audit logs of job and DAG changes so you can show who altered a schedule, a connector or a transformation and when. In Apache Airflow, run and task history lives in the metadata database, so its retention and backup directly determine how far back you can answer an auditor's question. In Dagster, run records and asset materialization events serve the same purpose. Before go-live, confirm with your platform team how long these records persist by default, where they are stored, and whether change events are captured outside the orchestrator's own database.
Access evidence matters as much as run evidence. Document which roles can view, trigger, edit and pause pipelines, and which service accounts hold credentials for source systems and landing zones. Export the role assignments from the orchestrator and from each connector platform, for example Airbyte's workspace and user settings, and compare them to your identity provider groups at a set interval. Lineage and ownership records close the loop: a metadata catalog such as DataHub or OpenMetadata can show which datasets a pipeline reads and writes, and a named owner for each one. Retention settings for landing zones should be written down per dataset, with the deletion job itself logged. Backup and restore evidence needs tested restores, not just backup jobs; keep the dated restore test records as proof that the pipeline metadata and data can be recovered.
Frameworks such as ISO/IEC 27001 and SOC 2 ask whether these controls operate, not just whether they exist, so plan to sample evidence over a period rather than produce it once. DeployOpen prepares systems, configuration and evidence packs for assessments, but independent auditors issue the reports and certifications. The EU AI Act is narrower. Its record-keeping duties, including Article 12(1) on automatic event logging and Article 19(1) on keeping logs for at least six months, apply to AI systems, and they matter for pipelines only where those pipelines feed a high-risk AI system. If that applies to you, confirm the scope with your legal or compliance team and with the AI Act Service Desk article on record-keeping.
What buyers ask before hiring help with data pipelines
How much does a self-hosted data pipeline stack cost to run?
The running cost depends on the infrastructure you already own, how many pipelines and connectors you operate, and how much data moves each day. Expect recurring costs for compute and storage for the orchestrator, its metadata database, workers and the ingestion layer, plus logging, backups and monitoring. One-time costs cover design, installation, connector configuration and handover. Ask any provider for a quote that separates these items, lists which cloud or data center resources it assumes, and states who pays for each licence, if any. Ask also what changes when you add sources or raise volumes.
How long does a data pipeline rollout take?
Duration depends mainly on how many source systems you connect, how clean and documented those sources are, and how many approval steps your security and data governance teams require. A first rollout usually covers one or two priority sources, the orchestrator, scheduling, alerting and access controls, then grows source by source. Ask for a phased plan with written milestones, the acceptance criteria for each phase, and the dependencies on your team, such as network access, service accounts and test data. Treat any timeline given before the source inventory is complete as provisional.
What does the work need from our own team?
Your team supplies decisions and access that no outside party can provide. That includes named data owners for each source, network and firewall changes, service accounts with least-privilege rights, test environments, and sign-off on schemas and data quality rules. Platform engineers should be involved in the container platform, secrets management and backups, and analysts or data engineers should review transformations. Ask a provider to write down these responsibilities in the scope so that delays are visible early and are not discovered during testing.
Can an internal team run a self-hosted pipeline platform alone?
Many internal teams can run it, provided they have engineers who understand Python, containers, Kubernetes or a similar runtime, and database operations. The harder part is sustained operation: upgrades, dependency patching, connector maintenance when source APIs change, and responding to failed runs at night. Assess whether you have that capacity for the full lifecycle, not only the first deployment. A partial model, where the provider builds and documents the platform and your team operates it, is often easier to staff. Ask the provider which runbooks, dashboards and training it will hand over.
Should we choose Airflow, Dagster or Airbyte for our pipelines?
The three tools solve different parts of the problem, so the choice is often a combination rather than a single pick. Apache Airflow (3.3.2, published September 17, 2026) is licensed under Apache-2.0 and is a mature option for authoring, scheduling and monitoring workflows as code. Dagster (1.13.26, published October 9, 2026) is also Apache-2.0 and models pipelines around software-defined assets. Airbyte focuses on ingestion connectors, but its platform repository is under the Elastic License 2.0. Decide based on your team's language skills, asset-based or task-based thinking, and which connectors you need. Ask a provider to show a short evaluation against your own sources.
How do licenses affect running pipelines in production?
Licenses can differ between the orchestrator, the ingestion engine and any enterprise add-on, so review each component separately. Airflow and Dagster are under Apache-2.0. The Airbyte platform repository is under the Elastic License 2.0, which restricts providing the software to third parties as a hosted or managed service. Using it internally for your own data is a different question from offering it to customers, so have counsel confirm your use case against the license text. Also check connector licenses and any features marked as enterprise, because they may require a commercial agreement.
Who handles upgrades and on-call after handover?
After handover, responsibility depends on the support model you contract for, and it should be written down before work starts. The usual options are that your team runs everything, that a provider runs upgrades and incident response under a retainer, or that a mix applies, with your team handling data issues and the provider handling platform faults. Ask who receives alerts, what response expectations are stated, how upgrades are tested against your connectors, and how backups and restores are exercised. DeployOpen offers ongoing support and upgrades as retainers, and its scope terms are agreed in writing.
Questions to ask a data pipeline partner before signing
Ask the partner to answer these questions in writing, inside the statement of work, before you sign. Start with who writes, reviews and tests each connector. Ask whether the partner builds custom connectors or configures existing ones from Airbyte, and who owns them after go-live. Ask for the test approach for each connector: schema change handling, incremental sync checks, and failure replay in a non-production copy of your source systems. Ask which of your staff will be trained to modify a connector and what the partner will do if an upstream API changes during the engagement.
Next, ask how credentials are stored, rotated and scoped. For Airflow, Dagster and Airbyte, ask where connection secrets live (a secrets backend in your environment, not plain text in DAG files or Airbyte configuration), which service accounts each pipeline uses, and how rotation is performed without breaking scheduled runs. Ask how lineage and ownership are documented, and whether the partner will populate a catalog such as DataHub or OpenMetadata or hand over lineage exports in a format your team can keep current. Finally, ask for the handover package in writing: runbooks for each DAG or asset job, alert rules and their owners, a documented restore test with evidence that it was run, and a list of what is explicitly excluded.
Close with two commercial questions. Ask what support after go-live covers, which severity levels get response, who is on call, and whether support is a separate retainer priced from the same written scope. Ask the partner to show, for each connector and platform in the design, how the license was checked for your intended production use. Airbyte's platform repository is under the Elastic License 2.0, and Airflow and Dagster are Apache-2.0, so each connector's own license may differ. Ask for a written license register with the source for each entry and confirm it with your legal team before the design is approved.
How DeployOpen scopes, builds and hands over pipeline platforms
DeployOpen plans, implements, customizes, integrates, hardens, backs up, documents and supports self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks. For data pipelines, that means Airflow, Dagster or Airbyte running where your data already lives, with the connectors, schedules and monitoring your teams need to operate them.
How we scope
- Inventory source systems, destinations, data volumes and freshness requirements before choosing an orchestrator or ingestion tool
- Record connector licensing checks for each source, including any Elastic License 2.0 or enterprise-only components
- Agree deliverables, environments and handover tasks in a written, fixed-scope statement before work starts
How we build
- Deploy the chosen orchestrator and ingestion components into your infrastructure with infrastructure-as-code you can review
- Configure secrets handling, role-based access, scheduling, alerting and logging against your existing identity and monitoring tools
- Test pipeline runs, failure paths and retries, and document results for your data and security teams
How we hand over
- Provide runbooks covering deployment, upgrades, backups, connector onboarding and incident response for each pipeline
- Walk your platform and data engineering teams through operations, ownership boundaries and escalation paths
- Deliver configuration records and evidence that support your data governance and audit reviews
How we support
- Offer ongoing support, upgrades and continuous testing as a retainer, scoped separately from the build
- Plan version upgrades of Airflow, Dagster and Airbyte components with tested rollback steps
- Confirm with your team which pipeline operations stay in-house and which are covered under the retainer
