Why data pipelines for SaaS products need a different design
For a SaaS team, self-hosting on Apache Airflow or Dagster with ClickHouse as the warehouse suits organizations that need tenant data to stay in their own cloud accounts, and the decision usually turns on two things: connector licensing and per-tenant separation. A managed ELT service can move product database, billing and support data quickly, but the team then depends on that vendor's connector catalog, its data residency and its terms for every tenant record that passes through. Before choosing, list each source (product database, billing, support desk, usage events) and check the license and deployment terms of every connector you intend to run, because some connector platforms carry licensing conditions that differ from the orchestrator's license. Confirm those terms with the vendor rather than assuming them.
SaaS pipelines carry data from many tenants in the same flows, so separation, deletion and contractual obligations shape the design more than raw volume. Decide early whether tenants are isolated by a tenant_id column with enforced row-level access, by separate schemas or databases, or by separate pipelines for high-value customers. Each choice changes how backfills, schema changes and retention rules are applied. Customer contracts and data processing agreements also set how long copies may persist and what must be deleted on request, so the team needs a deletion path that reaches staging tables, orchestrator logs and backups, not only the warehouse. ClickHouse backups are a useful test case: replication alone does not protect against human error, and a backup that has never been test-restored will probably not restore when needed.
Plan for the failure modes of multi-tenant flows before the first connector goes live. A bad transformation can expose one tenant's rows to another, and a failed backfill can re-load deleted records. Ask any implementation partner how they will prove tenant isolation in tests, where credentials for each source sit, how DAG and connector changes are approved, and how the team will confirm a deletion has propagated. These answers determine the operating cost, which recurs in on-call time, connector upgrades and restore testing, far more than the initial build.
Which requirements shape a SaaS pipeline build
This table maps the regulatory and contractual requirements that most often change how a SaaS company builds its data pipelines, with the source of each rule and what it means for design decisions. Read each row as a question to answer before choosing connectors, orchestration and tenant isolation, since several rows constrain where data may flow and what evidence the pipeline must produce.
| Requirement | Source | What it means for the build |
|---|---|---|
| Customer data processor contracts | GDPR Article 28(3) requires processor contracts to limit processing to documented instructions, including transfers, and to apply Article 32 security measures | Pipelines must process customer data only for documented purposes, so extraction jobs, staging tables and transfers to any external ELT or warehouse vendor need to be listed in the processing instructions and subprocessor records |
| SOC 2 Trust Services Criteria when customers request a report | AICPA 2017 Trust Services Criteria; security common criteria CC1 to CC9 apply to every SOC 2 examination | Pipeline access controls, change management for DAGs and connector configurations, monitoring and incident handling must be designed to produce evidence over a period, since a Type 2 report tests operating effectiveness |
| NIS2 classification as a service provider | NIS2 Annex I treats cloud computing, data center and managed service providers as high-criticality sectors | Confirm whether the company is itself a cloud or managed service provider to other essential entities; if so, pipeline operations, supplier management and incident reporting may fall under NIS2 obligations and the implementing requirements in Regulation (EU) 2024/2690 |
| Customer switching under the EU Data Act | Data Act Articles 23, 25 and 50 require data processing services to allow switching to another provider or to on-premises infrastructure; Article 29 bans switching charges from 12 January 2027 | Export formats, schemas, orchestration definitions and exports of pipeline outputs should be portable so customers can move their data out, and switching costs should be documented before the charge rules take effect |
| AI Act logging if pipelines feed high-risk AI | AI Act Article 12(1) requires automatic event logging for high-risk AI systems; Article 19(1) requires providers to keep logs for at least six months unless other law provides otherwise | If pipelines supply training, retrieval or input data for a high-risk AI feature, lineage and run logs must be retained and linked to the AI system's own event records, so the retention period and storage location need to be set at design time |
Which orchestrator and connectors fit a SaaS data team
Apache Airflow is the most common choice when a team wants to write pipelines as code and schedule them with explicit dependencies. The project is released under the Apache License 2.0 (Apache-2.0), and there is no paid edition from the project itself, so license cost is zero while operating cost sits with your team. Version 3.3.2 is the current patch release, and Airflow 2 reached end of life on April 22, 2026, so new work should start on Airflow 3.
- Runtime prerequisites: Airflow 3.3.2 supports Python 3.10 to 3.14 and Kubernetes 1.30 to 1.35. Confirm your base images and cluster versions against the prerequisites page before standardizing.
- Metadata store: Use PostgreSQL 14 to 18 or MySQL in production. SQLite is for testing only, and MariaDB is not supported as a metadata database.
- Executor choice: LocalExecutor suits small teams, CeleryExecutor and KubernetesExecutor suit larger fleets, and since Airflow 2.10.0 several executors can run in one deployment. The hybrid CeleryKubernetesExecutor and LocalKubernetesExecutor are no longer supported as of Airflow 3.0.0.
- Multiple schedulers: Airflow supports more than one scheduler at once using database row-level locks, which helps with resiliency when the scheduler is a single point of failure.
- Upgrade planning: Limited maintenance receives only security and critical fixes, and end-of-life versions receive nothing, so budget recurring upgrade work.
Ask your partner to show the executor, metadata database and upgrade plan for your tenant volume before committing to a design.
Dagster is built around software-defined assets, meaning tables, files and models are declared as the objects a pipeline produces, and the orchestrator tracks their freshness and lineage. It is a good fit for SaaS data teams that want engineers to reason about datasets rather than task graphs. Dagster is licensed under Apache-2.0, and version 1.13.26 was published on October 9, 2026.
- Asset-based design: Models per-tenant or per-product tables as assets, which makes freshness checks and partial reruns easier to explain.
- Local development: Teams can test asset definitions locally before deploying, which supports the review and change control that SaaS customers ask about.
- Tenant partitions: Partitioned assets can represent tenant or time slices, so a failed backfill for one customer does not force a full rebuild.
- Smaller ecosystem than Airflow: Hiring and third-party operator availability may differ, so check the skills in your market before choosing.
- Upgrade cadence: Pin the Dagster version and the integration libraries together, then test both on a staging deployment.
Confirm with your partner how assets map to tenants, and whether your team will write Python assets or rely on Airflow DAGs for most work.
Airbyte provides prebuilt connectors for SaaS APIs and databases, which can reduce the time spent writing extraction code for common sources. The license needs close reading before you rely on it. The Airbyte platform repository is licensed under the Elastic License 2.0, so confirm whether your intended use, particularly offering a hosted data service to customers, is permitted before you deploy it for tenants.
- License review: Ask counsel to confirm the Elastic License 2.0 terms against your product model before hosting the platform for customers or partners.
- Connector maintenance: Connector versions change with source APIs, so plan recurring updates and regression tests for each connector you run.
- Connector licensing: Check the license of each individual connector, which may differ from the platform license.
- Resource profile: Sync workers consume compute and memory during large backfills, so size them separately from the orchestrator.
- Alternative for custom sources: Where a source is internal or niche, a Python extraction task in Airflow or Dagster may be simpler to govern than a connector you do not control.
Before committing, ask your partner to list every connector in scope with its license and the owner responsible for updating it.
ClickHouse is a column-oriented analytical database that suits product analytics and per-tenant reporting at volume. The server is open source under Apache-2.0, while ClickHouse Inc. separately sells ClickHouse Cloud. Running it yourself means owning replication, backup and version support, so the operating model matters as much as the query speed.
- Supported versions: Security updates currently cover versions 26.9, 26.8, 26.7 and 26.3, and all other versions are unsupported. Choose a line from that list and track its end of support.
- Replication: ReplicatedMergeTree tables are coordinated by ClickHouse Keeper, and the replication guide recommends at least three Keeper nodes on dedicated hosts in production.
- Backup and restore: BACKUP and RESTORE support full and incremental backups to local disk, S3-compatible storage or Azure Blob Storage. Replication does not protect against human error or corruption.
- Restore testing: The documentation warns that a backup never test-restored will probably not restore properly, so schedule restores to a spare cluster.
- Authentication: External authenticators include LDAP, Kerberos, SSL X.509 certificates, an HTTP authenticator and JWT, which can connect the warehouse to your identity provider.
Ask your partner for the restore test results and the Keeper topology they will deploy.
Tenant-facing dashboards are the layer customers see, so the choice of analytics tool carries license and security consequences. Apache Superset is licensed under Apache-2.0, with version 6.1.0 published on May 13, 2026. Metabase is licensed under AGPL outside its enterprise directory, and that directory uses the Metabase Commercial License, so features in it may require a commercial agreement.
- Superset license: Apache-2.0 places fewer restrictions on embedding in a product, but row-level security and tenant isolation still need to be designed and tested.
- Metabase license: The AGPL and the commercial enterprise directory mean you must confirm which features you use and whether distribution to customers triggers obligations.
- Tenant isolation: Whichever tool you choose, verify that row-level rules prevent one tenant from querying another tenant's rows through the embedded interface.
- Identity: Confirm SSO support and how tenant users map to warehouse roles.
- Scope of use: Internal analytics and customer-facing embedding have different license exposure, so separate them in your planning.
Ask your partner to document the license position for each feature in use and to run a tenant isolation test before customers see the dashboards.
How to separate tenant data in the warehouse and pipelines
Choose the tenant isolation pattern before the first pipeline is written, because changing it later means rewriting table layouts, credentials and backup jobs. The three common patterns are shared tables with a tenant key, one database per tenant, and separate ClickHouse clusters for large or regulated tenants. A shared table is the cheapest to operate, but every query path depends on the tenant key being present and enforced, so row policies and role grants must be tested rather than assumed. Separate databases or clusters push isolation into the storage layout and add operational overhead for each tenant. Confirm in testing, with a tenant user who attempts to read another tenant's rows, which pattern your design actually enforces.
The orchestrator is the most common place where tenant boundaries leak. In Airflow or Dagster, avoid one shared service account that can read every source and write every destination. Give each tenant or tenant group its own connection and secret reference, store those secrets in your secrets backend rather than in DAG code, and scope each run to a tenant parameter that the task validates before it touches data. Backfills need the same scoping: a backfill for one tenant should target only that tenant's partitions and sources, and the run history should show which tenant it affected. On the warehouse side, ClickHouse access rights control who reads which tenant, and its external authenticators documentation covers LDAP, Kerberos, SSL X.509 certificates, an HTTP authenticator and JWT authentication. Map each pipeline identity and each analyst group to the narrowest role that works, and review those grants on a schedule.
Replication does not protect against human error or corruption, so a tenant's accidental deletion will replicate to every healthy copy. Backups must therefore be tenant aware: the ClickHouse backup documentation supports full and incremental backups of tables, partitions or databases, which lets you align backup scope with the tenant layout you chose. Test restores of a single tenant on a spare cluster, because the same documentation warns that a backup never test-restored will probably not restore properly when needed. Ask your implementation partner to show how a single tenant can be restored without touching others, how restore evidence is recorded, and how tenant offboarding deletes data and backups within your retention rules.
What drives cost against a managed ELT service
A managed ELT service bills for connectors and usage, while a self-run stack moves cost into infrastructure, engineering time and operations. The cards below cover the components that change the comparison for a SaaS product, and the questions to settle before any quote is requested.
Connector licensing and usage fees
Managed ELT services typically charge per connector, per row or per monthly active record, so cost rises with the number of product tenants and source tables. Self-run Airbyte is licensed under the Elastic License 2.0 for the platform repository, which restricts offering it as a hosted or managed service to third parties, so the license terms should be checked before it is used as a product feature. Ask each vendor how tenant count and schema changes affect the bill.
Compute and storage for self-run stacks
A self-run stack pays for the servers or cloud accounts that run the orchestrator, workers, connector containers and the ClickHouse cluster, plus object storage for backups. These costs recur monthly and scale with data volume and refresh frequency. Ask a partner to size staging, warehouse and backup storage separately, because backups usually grow faster than live data.
Connector build for unsupported sources
Managed services cover popular SaaS sources, but a product's own databases, legacy tenant stores or niche APIs often need custom connectors. Building and maintaining these is a recurring engineering task that the managed tier does not remove. Confirm which sources need custom code, who owns it, and how API changes will be tested before they reach production.
Upgrades, on-call and version support
Self-run orchestration adds an upgrade and on-call burden. Airflow 2 reached end of life on April 22, 2026, so any remaining Airflow 2 deployment gets no fixes and must be migrated. ClickHouse security updates cover only the listed supported versions, so the cluster must track that policy. Ask for the upgrade plan, the on-call coverage model and the named responsibilities for each component.
Backup and restore testing
ClickHouse replication does not protect against human error or corruption, so a separate backup regime is required. The ClickHouse documentation warns that a backup never test-restored will probably not restore properly when needed. Budget for recurring restore tests on a spare cluster, storage for full and incremental backups, and the engineer time to run and record them. Ask whether the quote includes scheduled restore drills.
One-time switching obligations
The EU Data Act requires data processing service providers to let customers switch to another provider or to on-premises infrastructure, and from 12 January 2027 switching charges may not be imposed at all. These rules can make an exit from a managed ELT service simpler and cheaper than before, although data export and pipeline rebuilding still take one-time effort. Confirm the contract's notice period, export formats and any switching fees before committing.
How a SaaS pipeline rollout runs in phases
The sequence moves a SaaS team from an inventory of sources and tenant data to a tested, handed-over pipeline platform that replaces any managed ELT service without losing data or tenant boundaries. Each phase closes with a written output that the client signs off before the next phase starts.
Inventory sources and map tenant data
DeployOpen works with the client's engineering and data owners to list every product database, SaaS tool and API feeding analytics, with its owner, volume, change pattern and the tenant identifier that marks each row. The output is a source inventory and a tenant data map that flags which tables hold personal data and which fields tie records to a customer account. The step is done when the client confirms the inventory matches what production actually sends.
Decide platform and check licensing
The team selects the orchestrator, normally Airflow or Dagster, and the extraction tooling, then checks the license of each component against how the client will distribute or host it. Connector licenses are reviewed separately, because some connector sources carry terms on commercial use that differ from the orchestrator's license. The output is a written platform decision with a license register, and the client's legal team confirms that no component blocks the intended use.
Build pipelines in non-production
Engineers build DAGs or assets, connectors and transformations in a non-production environment loaded with representative data, and stand up a pre-production ClickHouse cluster that matches the intended production topology. Queries that analysts and product teams depend on are rerun against this environment to confirm results. The output is a tested pipeline set and a pre-production warehouse, and the step closes when row counts and key metrics reconcile with the source systems.
Test tenant separation and access
The team enforces tenant boundaries in the warehouse through database roles, row policies or separate schemas, depending on the isolation model the client chooses, and tests them with accounts assigned to different tenants. Authentication for warehouse users is wired to the client's directory, such as LDAP, Kerberos or JWT, using the external authenticator options ClickHouse documents. The output is a test report showing that a tenant account cannot read another tenant's rows, and the client's security owner signs it off.
Prove backup and restore on a spare cluster
Engineers configure ClickHouse BACKUP jobs to the storage the client has approved, then restore the backups to a spare cluster that is not connected to production. The test checks that restored tenant data matches the source, that access rules survive the restore, and that the restore time is recorded. ClickHouse documents that a backup never test-restored will probably not restore properly, so this step is not optional. The output is a dated restore report that the client keeps with its recovery runbook.
Cut over with a parallel run
Before any managed ELT job is switched off, the new pipelines run in parallel with it for a period the client agrees in writing, and the two outputs are compared table by table and tenant by tenant. Discrepancies are traced to source, transformation or timing and fixed before the managed service is disabled. The output is a reconciliation log and a cutover decision record, and the step closes when the client accepts the parallel results. Any termination steps for the managed service follow the contract the client holds.
Hand over runbooks and upgrade plan
DeployOpen documents the pipeline architecture, on-call procedures, failure and backfill handling, secret rotation and the upgrade plan for Airflow or Dagster and ClickHouse, including how the client tracks ClickHouse's supported version lines and changelogs. The client's operators run a supervised failure drill before handover is accepted. The output is a runbook set, an upgrade calendar and a completed drill record, and the step is done when the client's team takes ownership in writing.
Evidence auditors and customers ask for from the pipeline
This table maps each piece of evidence that SOC 2 auditors, ISO assessors and SaaS customers commonly request from a pipeline to what it shows and the component that produces it. Read it as a collection plan: if a row has no owner or retention rule in your environment, the control is not yet evidenced. DeployOpen prepares these records and controls for your assessment, but independent auditors issue any SOC 2 or ISO report.
| Evidence item | What it shows | Component that produces it |
|---|---|---|
| Pipeline run logs and retries | Which jobs ran, when, whether they succeeded, and how failures were retried or escalated | Airflow or Dagster run history, exported to your log store |
| DAG and asset change approvals | Who changed a pipeline definition, who reviewed it, and that it was tested before release | Git pull request history and CI/CD pipeline records for the orchestrator repository |
| Connector credential access reviews | Which people and service accounts can use each source credential, reviewed on a set schedule | Secrets manager audit logs and periodic review exports from the orchestrator and connector platform |
| ClickHouse access rights and grants | Which users, roles and row policies can read each tenant's tables | ClickHouse access control definitions and role grant exports |
| ClickHouse authentication logs | Successful and failed logins, and the external authenticator (LDAP, Kerberos, SSL or JWT) used | ClickHouse server logs and the external identity provider's sign-in records |
| Backup and restore test records | That backups complete and that a restore has been tested on a spare cluster, with the outcome | ClickHouse BACKUP and RESTORE job logs plus signed test reports from the restore exercise |
| GDPR processing records | Which personal data categories flow from each source to each destination, the purpose and retention period | Pipeline inventory and data catalog entries maintained by the data team, reviewed with legal |
| Tenant data separation tests | That one customer's records cannot be queried or joined into another customer's results | Automated test queries run after each release and stored with the release record |
What do SaaS teams ask before self-hosting pipelines?
How does self-hosting pipelines cost compare with a managed ELT service?
The answer depends on which costs move to your side, so compare the full set rather than the license line alone. A managed ELT service bills per connector, row volume or usage tier, and that bill recurs. A self-hosted stack moves those costs to compute and storage in your cloud account, the engineering time to build and maintain connectors, and on-call coverage for failed runs. Connector licensing is the first driver to check: some open-source connector projects carry licenses that limit how you may run them for customers. Volume, number of sources, schema change frequency and the number of people on call push the total up. Ask any partner for a quote that separates one-time build work from recurring hosting, upgrades and support, and that shows the assumed row volumes and source count behind each line.
How long does a self-hosted pipeline rollout take for a SaaS product?
Rollout length is driven by the number of sources, the tenant model and how many schema and access questions have to be settled first. A pipeline from one product database into a warehouse is a much smaller job than a set of SaaS tools feeding per-tenant tables with separated access. Work usually splits into platform setup, connector build and testing, tenant isolation design, and cutover with monitoring. Each of these depends on the source owners answering questions quickly. Do not accept a duration before the written scope lists the sources, the destination, the tenant model and the acceptance tests. Confirm with the implementation partner which of these items sit inside the fixed scope and which would be change requests.
What does our team need to supply for a pipeline project?
The project needs named access, named owners and a defined change process. On the access side, you need read credentials or replication permissions for each source, network routes from the orchestration environment to those sources, and a place in your cloud account or data center for the platform. On the ownership side, each source system needs an owner who can answer schema questions, and the warehouse needs an owner who approves new tables and access grants. The change process covers how DAG or flow changes get reviewed, tested and deployed, and who approves them. Ask the partner to confirm this list in writing before work starts, so that delays from missing access or unanswered questions are visible in the plan.
Can our own engineers run a self-hosted pipeline after handover?
Yes, provided you staff the operation with people and on-call capacity. Running Airflow or Dagster, a warehouse such as ClickHouse and the connectors means someone must watch failed runs, apply upgrades, restore from backups and handle schema drift. Confirm that your team can own the runbooks, that alerts reach a person who can act, and that restore tests are scheduled. Many teams combine internal ownership with a support retainer for upgrades and incident help. Ask the partner to document the runbooks and handover tests as deliverables, and to state which operational tasks remain with your staff after go-live.
Does Airbyte's license affect hosting pipelines for our customers?
It can, so have counsel review it before you build on it. The Airbyte platform repository is licensed under the Elastic License 2.0, and that license restricts providing the software to third parties as a hosted or managed service. Using it internally to feed your own analytics is a different question from running it so that customers can use it, and the answer depends on how the connectors and platform are offered. Also check the licenses of individual connectors, which may differ from the platform. If the license is a blocker, compare alternatives such as Dagster-based ingestion code or other connector frameworks, and confirm each option against its own license text.
Do we need to upgrade from Airflow 2 before building new pipelines?
Yes, you should not start new pipelines on Airflow 2. According to the Airflow version support table, Airflow 2 entered limited maintenance on October 22, 2025 and reached end of life on April 22, 2026, after which it gets no fixes. Limited maintenance covered only security and critical bug fixes. Airflow 3 was first released on April 22, 2025, and the current patch release is 3.3.2. Before you migrate, review the upgrade guide, check that DAGs and custom operators work on Airflow 3, and confirm your metadata database meets the prerequisites. PostgreSQL 14 to 18 and MySQL 8.0, 8.4 are supported metadata databases, and SQLite is not suitable for production.
How does the EU Data Act affect moving pipelines off a managed ELT service?
The Data Act gives you a legal route to switch away from a data processing service, which matters when you plan a move to self-hosting. Under Regulation (EU) 2023/2854, which has applied since September 12, 2025, providers must let customers switch to another provider or to on-premises ICT infrastructure, with a maximum two-month notice period and a 30-day transitional period. From January 12, 2027, providers may not impose any switching charges. Until then, reduced charges may not exceed costs directly linked to switching. Review your current contract for export formats, data return and deletion terms, and ask counsel to confirm how these rules apply to your provider and your data.
How do we keep each tenant's data separated in a shared warehouse?
Choose the separation model before you build connectors, because it changes table design, access control and cost. Common options include a shared table with a tenant identifier enforced by row-level policies, a schema or database per tenant, or a separate cluster for high-sensitivity tenants. In ClickHouse, access can be controlled through its access control features, and external authentication options include LDAP, Kerberos, SSL X.509 certificates and JWT. Test that a tenant's credentials cannot read another tenant's rows, and include that test in acceptance. Ask the partner to show the isolation design and the test evidence as part of the written scope.
What to ask an implementation partner before signing
Start with tenant isolation, because it decides most of the rest. Ask the partner how they separate each customer's data in the warehouse: one database per tenant, one schema per tenant, or shared tables with a tenant key enforced by row policies and access grants. Ask who can read raw staging tables, where intermediate files are written, and which service account Airflow or Dagster uses to write to each tenant's destination. Then ask how they test isolation. A useful answer names a specific test, such as a scripted query run as a tenant-scoped user that must return zero rows from another tenant, and says how that test runs again after every pipeline change. Ask them to show the access-control settings they would use in ClickHouse, including how users, roles and grants are defined in the access_rights documentation terms, and who approves changes to them.
Next, ask which orchestrator and connectors they recommend and why. For orchestration, expect a reasoned choice between Airflow and Dagster that reflects your team's skills, your deployment model and your support window, not a default. Ask which Airflow version they would install, and confirm it is on a supported line before you sign, since the version life cycle page lists when each line moves to limited maintenance or end of life. For connectors, ask how each connector's license is checked before use. The Airbyte platform repository is under the Elastic License 2.0, so ask whether the partner will run it as an internal service, whether that use is permitted under the terms, and who signs off on that reading with your legal team. Ask whether they will use connectors that are not built on that license, and what they will build themselves. Also ask how backfills are scoped per tenant, so a replay for one customer never rewrites another customer's rows, and how upgrades of ClickHouse and Airflow are scheduled against published support windows, with a pre-production environment loaded with representative data for each upgrade.
