What a data pipeline costs when you build, partner or buy SaaS
For a team comparing self-run Airflow, Dagster, Airbyte and Kafka with managed orchestration and ELT, the software license is usually not the main cost. Apache Airflow and Apache Kafka are both released under the Apache License 2.0 with no paid edition from the projects themselves, and Dagster's open-source core is Apache-2.0 as well. The real answer depends on connector count, data volumes, the staff who will run the platform and the compliance evidence the pipelines must produce. Those four inputs move the total far more than the price of any tool.
The cost categories are compute and storage for schedulers, workers, brokers and the warehouse, connector build and maintenance, pipeline engineering, upgrades, monitoring, on-call coverage and the controls needed for audits. Some costs are one-time, such as initial architecture, connector development and migration. Others recur, including infrastructure, patching, upgrade cycles and on-call rotation. Managed tiers add a per-user or usage charge on top. Dagster's pricing page, for example, places role-based access control, SAML SSO, column lineage and audit logs in its paid Dagster+ tier rather than the open-source core, so a team that needs those features must price them into the comparison. Confirm the current pricing directly with each vendor before modeling.
DeployOpen does not publish a price list. Quotes follow a written scope that names the sources and destinations, the orchestrator and connectors in use, expected volumes and latency, the upgrade and support model, and any evidence requirements. Ask any partner to itemize one-time build work separately from recurring operations, to state which component versions and support windows the estimate assumes, and to explain how connector maintenance and schema changes are priced. Compare each option against the same scope so the numbers are actually comparable.
Cost components by build, partner and SaaS option
This table compares the main cost components of a data pipeline across in-house build, an implementation partner, and a managed SaaS service. Read each row to see who is paid, whether the cost is one-time or recurring, and what drives it up. The main takeaway is that open-source orchestrators remove license fees but not the engineering, operations and connector work that dominates total cost.
| Cost component | In-house build | Implementation partner | Managed SaaS |
|---|---|---|---|
| Software license | Apache-2.0 for Airflow, Kafka and Dagster open source, so no license fee; Dagster+ features such as RBAC, SAML SSO, lineage and audit logs are paid | Same open-source license terms; partner fees cover scoped work, not the software itself | Subscription or usage fees to the vendor, typically recurring; Prefect Cloud Team lists $100 per user per month and Starter $100 per month (Prefect pricing page) |
| Compute and storage | Recurring, paid to your cloud or data center provider; sized by volume, latency and scheduler and worker count | Same infrastructure cost, paid to your provider; partner sizes and documents it in the written scope | Often bundled into the usage meter, so it recurs and scales with the vendor's pricing units |
| Connector build and maintenance | One-time build plus recurring maintenance by your engineers when source APIs or schemas change | One-time build fee in the fixed scope, plus support retainer if maintenance is contracted | Included for supported connectors; custom connectors may be extra and still rely on your engineers |
| Pipeline engineering | Recurring salaries for data engineers writing and testing DAGs, assets and flows | Fixed-scope build fee, often one-time, with handover documentation | Low-code setup is quicker, but complex transformations and data quality logic still need engineers |
| Platform operations and on-call | Recurring cost of staff on rotation for scheduler, broker and database health, backups and failures | Optional retainer for monitoring, patching and incident response; confirm response terms in writing | Mostly absorbed by the vendor, but you still own failure handling for your own jobs |
| Upgrades and security testing | Recurring planning for Airflow, Kafka and Dagster versions; Airflow 2 reached end of life on April 22, 2026, so upgrades are not optional | Upgrade and testing work can be a scoped one-time project or a recurring retainer | Vendor-managed upgrades, but you still need to validate pipelines and evidence the change |
| Managed-tier fees | Not applicable unless you buy a commercial tier for features you need | Pass-through or included in the scope; confirm which tiers are assumed | Per-user tiers or consumption pricing; Informatica prices IDMC on IPUs with no published prices, so request a quote |
What moves the data pipeline cost up or down
Most of the cost of a self-run pipeline estate comes from people, not licenses, because the open-source components are free to use. The six drivers below set how many engineering and on-call hours you carry, how much infrastructure you run, and how often you must upgrade, so they belong in any build, partner or SaaS comparison.
Sources and connector count
Each source system adds a connector to select, configure, test and maintain, and every schema change in a source can break a sync. Cost rises with the number of distinct systems and with the share that have no maintained open-source connector, since those must be written and owned in-house. Ask a partner which connectors they expect to build, which they will adopt from the community, and who patches them when a source API changes. Check the license terms of any connector platform you adopt, since the Airbyte platform repository is licensed under the Elastic License 2.0.
Volume and latency
Batch pipelines on a scheduler are usually cheaper to run than streaming pipelines that must stay up around the clock. Streaming adds brokers, storage, monitoring and capacity planning, and each latency target you tighten raises the compute you keep idle for peaks. Estimate peak throughput, retention and the freshness each consumer needs before choosing a design. A partner should size compute from those numbers, not from a default cluster.
Number of pipelines and DAGs
The count of pipelines, asset graphs and DAGs drives scheduler load, metadata database size, test effort and review time for every change. A few hundred jobs managed through a shared template cost much less to run than the same jobs written ad hoc by different teams. Ask how pipelines are tested in continuous integration, how many environments are provisioned, and whether the platform team or the data teams own each change.
Compliance and evidence needs
Regulated pipelines need lineage, audit logs, access reviews, retention rules and change records, and each of these adds setup and recurring work. Open-source Dagster keeps role-based access control, SAML SSO, catalog search, column lineage, alerts and audit logs in Dagster+ rather than the free edition, so a self-run deployment must build or buy those controls separately. Confirm which evidence your auditors will request before costing the build, because the missing items often decide whether a pipeline can go live.
Staffing and on-call hours
Staffing is usually the largest recurring cost, covering platform engineers who run the scheduler, brokers and metadata stores, plus on-call rotations for failed jobs and lagging consumers. Managed tiers shift part of this work to the vendor but charge per user or per usage, so the break-even point depends on team size and run volume. Ask a partner for the named roles in the run model, the hours assumed for on-call, and the escalation path for a failed backfill at night.
Upgrade cadence and version support
Upgrades are a recurring cost that is easy to underestimate. Airflow 2 reached end of life on April 22, 2026, and Apache Kafka 4.0 supports only KRaft mode, so older clusters must migrate before upgrading. Each forced migration consumes test, change approval and rollback effort, so plan a fixed upgrade budget and confirm how the partner tests Airflow 3.3.2 or the current Kafka release before production.
Where self-run open-source costs recur
The license line in a self-run comparison is usually zero, because Apache Airflow, Dagster, Airbyte's platform code under the Elastic License 2.0 and Apache Kafka all carry no project-level fee. The recurring cost sits in people and infrastructure: a production Airflow deployment needs a PostgreSQL or MySQL metadata database, since the project warns that SQLite in production can lose data, plus scheduler and worker capacity sized to your DAG count and latency targets. Dagster's open-source deployment adds its own moving parts, including a daemon that runs schedules and sensors and a run storage backend you must back up and patch. Price these from your own environment: database administration hours, compute for schedulers and workers, and the on-call rota that covers failed runs at night.
Kafka carries the heaviest upkeep because it is a stateful platform with release and security windows you must plan around. The downloads page lists 4.3.1, 4.2.2 and 4.1.2 as supported releases, and older versions are archived. Kafka 4.0 removed ZooKeeper mode, so any legacy cluster needs a KRaft migration before it can move to 4.x, and brokers and Kafka Connect require Java 17. Security is also recurring work: you configure TLS on listeners, choose a SASL mechanism such as SCRAM-SHA-512 or OAUTHBEARER, and maintain ACLs in the KRaft metadata log, where resources without ACLs are limited to super users unless you deliberately change that default. Each certificate rotation, credential change and new topic adds an operational task.
Confirm with your platform team how many hours per month these tasks consume, who owns each one, and whether the skills exist in-house for Kafka operations and Airflow upgrades. Airflow 2 reached end of life on April 22, 2026, so an older estate already carries a migration cost that will recur if upgrades slip. Ask any implementation partner to itemize upgrades, patching, backup testing and on-call as separate line items in the written scope, so you can compare them with a managed tier's usage or per-user fees on the same terms.
How to build your own data pipeline cost estimate
This sequence turns a vague budget question into a costed scope that every candidate platform or partner can price against the same inputs.
Inventory sources and destinations
List every source system, such as ERP, CRM, product databases, SaaS tools, files and device feeds, and every destination, such as a warehouse, lake or downstream application. Record the owner, data classification and network zone for each. Your data engineering lead and the system owners do this together, and it is done when each source and destination has a named owner and a data classification agreed with security.
Classify batch versus streaming and volumes
For each flow, state whether it runs in batch or streams, how fresh the data must be, and the expected record and byte volumes for a typical day and a peak. Streaming flows push the cost toward brokers and always-on compute, so they are priced separately from scheduled batch jobs. Done when every flow has a frequency, a latency target and a volume estimate that the business owner has signed off.
Count connectors to build versus use
Split the source list into connectors that already exist in the open-source ecosystem and connectors that must be written and maintained, such as proprietary APIs or legacy databases. Check each open-source connector's license and maintenance status before counting it as available, because some connector repositories are licensed differently from the orchestrator. Done when you have a two-column list with a named owner for every custom connector.
Decide the orchestrator and deployment target
Choose the orchestrator that fits your team, for example Apache Airflow or Dagster, and the place it runs, such as Kubernetes, containers or virtual machines in your own cloud account. Confirm the supported versions and prerequisites of the chosen release, including the metadata database. Note that the Airflow 2 line reached end of life on April 22, 2026, so a new build should target a supported release. Done when the architecture decision names the orchestrator version, the metadata database and the hosting platform.
Price the managed-tier alternatives from pricing pages
Collect written pricing for each managed option you are considering. The Prefect pricing page lists Starter at $100 per month and Team at $100 per user per month, with Enterprise at custom pricing. Dagster+ gates features such as role-based access control, SAML SSO and audit logs outside the free open-source edition, and Informatica prices IDMC on consumption through IPUs with no published prices. Confirm which features your security and audit requirements need, then compare the same workload across each model. Done when every managed option has a cost model tied to your user count or usage volume.
Add run costs for on-call, upgrades and evidence
Add the recurring work that self-run platforms create: on-call coverage for failed runs, scheduled upgrades, security patching, backup testing, connector maintenance and the logs and change records that auditors may request. Kafka releases three times a year and supports rolling upgrades from the last three releases, so upgrade planning is a standing cost. Done when the run model lists each recurring activity, its owner and how often it occurs.
Request written quotes against the same scope
Send the same written scope, covering the inventory, flow classification, connector list, orchestrator choice and run requirements, to each partner and each managed vendor, and ask for itemized quotes. Ask each partner to separate one-time build work from recurring support, and to state what is excluded. Done when every quote is comparable line by line and the assumptions behind each figure are documented.
When each option usually fits
Building in-house means your own engineers select, deploy, and operate the orchestrator, connectors, and streaming layer. The software is free to use under its license, so the main costs are people, compute, storage, and the time spent building connectors and maintaining pipelines. This model fits organizations with a platform team already running Kubernetes or similar infrastructure, a steady flow of pipeline changes, and a need to keep every credential and intermediate file inside their own network, including air-gapped environments.
- Cost profile: Mostly recurring engineering salaries, on-call rotations, and infrastructure. Open-source licenses for Airflow, Kafka, and Dagster carry no license fee, but Dagster+ features such as role-based access control and SAML SSO sit outside the free edition.
- Good fit: Strict residency, air-gapped networks, or a large number of custom internal sources that no vendor connector covers.
- Weaker fit: A small team with few sources, or no one available to handle upgrades and failed jobs at night.
- Upgrade load: Airflow 2 reached end of life on April 22, 2026, so in-house teams must plan Airflow 3 migrations themselves.
- Hidden cost: Security controls such as SSO, audit logs, and access policies often need extra engineering if you do not buy the commercial tier.
Confirm your team's on-call coverage, your upgrade calendar, and whether you need Dagster+ features before you commit to this model.
In a partner-led model, an implementation partner designs, builds, and hands over the pipelines and platform, and your team keeps ownership of the running system. You pay for the project as a defined scope, then carry the operating cost yourself. This suits organizations that want a production-grade setup quickly but do not want to build the skills first, or that need an outside party to document controls for auditors.
- Cost profile: A one-time implementation fee tied to written scope, followed by your own infrastructure and staff costs. Ongoing support, upgrades, and testing may be sold as a separate retainer.
- Good fit: Regulated data flows that need documented lineage, change approval, and failure handling from the start, and teams that will own the platform afterward.
- Knowledge transfer: Confirm whether the handover includes runbooks, architecture records, and training for your own staff.
- Scope boundaries: Ask whether connector development, backfills, and first-year upgrades are included or priced separately.
- Ownership: Your team must still be able to operate the stack after the project closes, so skills transfer matters more here than in other models.
Confirm in writing who owns the code, credentials, and infrastructure accounts, and which support tasks remain with your team after go-live.
A managed service runs the orchestration, connectors, or ELT jobs for you in the vendor's cloud. Costs usually follow a subscription, a per-user seat, or consumption measures. Informatica, for example, prices its IDMC service on consumption through Informatica Processing Units and publishes no list prices, while Prefect Cloud lists a per-user Team tier. This model fits teams with many standard SaaS sources, limited infrastructure staff, and data that can legally leave your environment.
- Cost profile: Recurring subscription or usage charges that rise with users, runs, or data volume. Compute and vendor-side operations are bundled, so internal headcount is lower.
- Good fit: Standard connectors to common SaaS tools, small teams, and workloads with no residency restrictions.
- Residency: Intermediate files and credentials may be processed in the vendor's cloud regions, which can conflict with sovereignty or sector rules.
- Lock-in: Pipelines written against proprietary connectors or tiers can be hard to move later.
- Usage surprises: Consumption pricing makes forecasting harder during backfills or schema changes.
Request a written quote that states the metering unit, included connectors, data residency of processing, and exit terms before you sign.
Data pipeline cost questions buyers ask before hiring help
How much does a self-hosted data pipeline cost compared with managed ELT?
A self-hosted pipeline replaces per-connector or per-row subscription fees with costs you pay directly: compute, storage, engineering time and on-call coverage. The open-source orchestrators carry no license fee. Apache Airflow and Apache Kafka are both released under the Apache License 2.0 and have no paid edition from the project itself, while Dagster's role-based access control, SAML SSO, audit logs and lineage sit in Dagster+ rather than the open-source build. Managed tiers publish their own rates, such as Prefect Cloud's Team plan at $100 per user per month, and usage-priced services like Informatica price consumption through IPUs without public prices. Ask each vendor for a rate card that matches your row volumes, connector count and user count, then compare that total with your own infrastructure and staff estimate.
Can our team run Airflow or Dagster without outside help?
Often yes, if the team already operates Kubernetes or Linux services and has Python skills, but the first production deployment is where gaps show. Airflow's production guidance says SQLite is for testing only and recommends PostgreSQL or MySQL, and Airflow 3.3.2 lists Kubernetes 1.30 to 1.35 and PostgreSQL 14 to 18 among its prerequisites. Dagster's open-source deployment separates a webserver, a single daemon and code location servers, and its metadata storage defaults to SQLite. Check whether your staff can own upgrades, secrets handling, executor choice and monitoring before deciding to build internally. Confirm the supported versions with each project before you size the team.
How long does a pipeline implementation take, and what drives the timeline?
The timeline depends mainly on the number of sources, their access constraints, the transformation logic and how many environments must be built and tested. Connector work is often the largest variable, because each source needs credentials, schema mapping and change handling. Kafka adds time when clusters must be migrated: Kafka 4.0 supports only KRaft mode, and the upgrade notes recommend moving clusters older than 3.3.x to 3.9.x before moving to 4.x. Engagements at DeployOpen are fixed-scope, so the duration is written into the agreed scope, not estimated here. Ask for a milestone plan that names source count, environments, cutover and handover.
What does the work need from our staff?
Plan for named owners in four areas: source system administrators who grant read access and confirm schemas, a data owner who signs off transformations, a security or identity contact who approves credentials and network paths, and an operations contact who receives alerts. Kafka security depends on your choice of SSL or SASL mechanisms such as SCRAM-SHA-512 or OAUTHBEARER, and the authorizer denies resources without ACLs unless you change that setting, so your security team has to take part in design. Ask the implementation partner which decisions they need from you and in which week.
What is one-time cost and what recurs after go-live?
One-time costs usually cover architecture, connector build, pipeline engineering, testing, documentation and initial hardening. Recurring costs include compute and storage, backups, version upgrades, connector maintenance when source APIs change, on-call coverage and any subscriptions for managed control planes. Airflow 2 reached end of life on April 22, 2026 and gets no fixes, so each platform carries an upgrade obligation that repeats. Kafka publishes three releases a year, and rolling upgrades are supported from the last three releases, which sets a recurring testing cadence. Ask a quote to split these lines explicitly so you can compare build, partner and SaaS options on the same basis.
How do managed per-user tiers scale as the team grows?
Per-user tiers scale linearly with seat count, while usage tiers scale with rows, runs or consumed units. Prefect Cloud's Team plan is priced at $100 per user per month, and its Enterprise tier adds object-level RBAC and multiple workspaces at custom pricing. Dagster+ reserves features such as SAML SSO and audit logs for its paid service. A small set of engineers may cost less per year than a large group of read-only viewers, so model the number of people who need edit rights separately from those who only view dashboards. Confirm with each vendor how viewers, service accounts and non-production environments are counted.
What should we ask for in a quote?
Ask for a written scope that lists sources, destinations, batch and streaming volumes, latency targets, environments, connector count and the exact systems in scope. Request a split of one-time and recurring costs, the assumptions behind any compute or storage estimate, and the upgrade plan for Airflow, Dagster and Kafka over the next 12 months. Ask which licenses are required, such as Dagster+ features or Camunda's license for production use, and who pays for them. Also request the acceptance criteria, the handover documents, the support model after go-live and the exit terms if you move platforms later.
How compliance evidence adds to the cost
Compliance evidence is a recurring cost line, not a one-time setup task. For a regulated pipeline, the orchestrator must keep run history, task logs, lineage, schema change records and proof of who could change or trigger each job. Each of these needs storage, retention periods, access controls and someone who can produce the records on request. In a self-run stack, that work falls on your engineers: configuring audit logging in Airflow or Dagster, shipping logs to a store you control, mapping lineage and keeping access reviews current as DAG authors and operators change. In a managed tier, part of this may be bundled, but you pay for it through per-user or usage pricing. Ask any partner or vendor which evidence is included, how long it is retained and whether you can export it without a separate fee.
Log duties in the EU AI Act are narrower than many teams assume. Article 12(1) requires high-risk AI systems to technically allow automatic recording of events over their lifetime, and Article 19(1) requires providers to keep those logs for at least six months unless other law provides otherwise. These duties apply only where a pipeline feeds a high-risk AI system. A pipeline that only loads reporting tables does not take on them by default, so confirm with your legal and compliance team whether any downstream model is in scope before budgeting for AI log retention. The primary sources are the AI Act Article 12 Record-keeping and AI Act Article 19 Automatically generated logs pages.
DeployOpen prepares systems, controls and evidence for compliance assessments, including pipeline logging, lineage capture and access configuration in your own infrastructure. It does not issue audit reports, attestations or certifications; independent auditors and certification bodies do that work. When requesting a quote, ask for the evidence items explicitly in the written scope, the retention settings to be configured and who will own them after handover. For depth on specific obligations, read the sibling pages on data pipelines for financial services and open source data pipelines for GDPR, which cover lineage, records of processing and minimization in more detail.
How DeployOpen scopes and quotes a data pipeline implementation
DeployOpen is a Chainscore Labs company that plans, implements, hardens, documents and supports self-hosted open-source platforms inside the client's own cloud accounts, data centers or air-gapped networks. We start from a written scope for the pipelines, sources and destinations in question, so the build, partner or SaaS comparison rests on the client's actual workload rather than a generic estimate.
How we scope
- We list every source system, destination warehouse, and batch or streaming requirement the pipelines must meet.
- We record which orchestrator, connectors and Kafka topics are in scope, and which managed services are being compared.
- We document where sensitive data may flow, where credentials live, and which network zones data crosses.
- Engagements are fixed-scope, with deliverables and timelines agreed in writing before any work starts.
How we build
- We deploy the chosen open-source orchestrator, such as Airflow or Dagster, with a production-grade metadata database rather than SQLite.
- We build connectors and pipeline code under version control, with tests and reviewed deployments for each change.
- We configure Kafka authentication, TLS and authorization in line with the client's security requirements before data flows.
- We produce a cost model for compute, storage and engineering effort, so the client can compare options on the same assumptions.
How we hand over
- We deliver architecture diagrams, runbooks for failure handling and backfills, and the upgrade path for each component.
- We document the licensing of each component, including which features sit behind paid tiers such as Dagster+ or Prefect Cloud.
- We train the client's own engineers on operating, monitoring and changing pipelines in their environment.
- We leave the client with a written record of what was built, what was tested and what remains open.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers, scoped separately from the build.
- We plan component upgrades around each project's support lifecycle, such as Airflow and Kafka release and end-of-life dates.
- We can run monitoring and on-call coverage for self-hosted pipelines under a retainer, with responsibilities agreed in writing.
- We revisit the build, partner or SaaS decision when volumes, connectors or licensing terms change.
