Data pipelines for EU data sovereignty

The first question for any managed ELT setup is not where the database sits but where the processor's access comes from. Under the EDPB's Guidelines 05/2021, remote access from a third country counts as a transfer even when the operator only views data on a screen for support, and storage in a cloud outside the EEA is also a transfer. Chapter V analysis therefore has to follow the people and systems that can reach the data in the clear, including the vendor's support staff, its connector control plane and its telemetry pipeline, not just the storage region shown in a console. Ask each vendor, in writing, which entities can access customer data, from which countries, and under what circumstances.

Consider an illustrative case: a customers table extracted from an on-premises CRM and loaded into a warehouse in an EU region. The bytes at rest may stay in the EU, yet the connector vendor's control plane receives the connection string and sample rows for schema detection, a managed staging bucket holds Parquet files of the full table in a region outside the EU, a failed sync pushes the offending rows into a dead-letter queue that the vendor's log service retains, and a SaaS observability tool stores row counts and column names with error messages. Each of these is a copy of personal data, and each needs a documented transfer decision. Intermediate files, dead-letter queues, orchestrator logs and schema metadata all count as data for this purpose, and they often outlast the pipeline run that created them.

When you review a managed ELT service, map each component to a row in a transfer register: what data it touches, where it is processed, who can access it, and which Chapter V mechanism applies. Confirm which vendor components are optional or can be disabled, and where credentials are stored. Self-hosted orchestration such as Apache Airflow keeps the scheduler, the metadata database, task logs and connection secrets inside your own infrastructure, which moves these copies under your control. Where a vendor must remain in the path, the EU processor analysis in the EDPB guidance and the third-country access exposure under GDPR Article 48 and the CLOUD Act still need written assessment before the contract is signed.

The Data Act gives customers a right to switch away from a data processing service, including to on-premises ICT infrastructure, and that right is the main reason to keep pipeline orchestration in your own hands. Under Regulation (EU) 2023/2854, the switching rights in Articles 25, 29 and 50 apply since 12 September 2025, with a maximum two-month notice period and a 30-day transition. Switching charges are banned from 12 January 2027. The practical question for a data team is whether the pipelines can actually move. A managed ELT service that holds the DAGs, connector configurations and run history in its own proprietary format makes a 30-day transition difficult to meet in practice, whatever the contract says.

Open-source orchestration changes that calculation because the logic lives in files you control. Confirm for each pipeline that the DAG code (Python for Apache Airflow, asset definitions for Dagster), connector configuration, environment variables and secret references are stored in your own version control and can be redeployed to a different scheduler without the vendor's tooling. Check which parts exist only in a UI or a metadata database: Airflow connections, variables and run history sit in the metadata database, so decide what to export, in what format, and how often. For NiFi, a flow exported from the canvas is a file you can keep under version control, and the flow configuration format changed in NiFi 2.0, so test the export and re-import path before you need it. Write these checks into a runbook and rehearse them at least once.

Portability also has a time dimension. Apache Airflow 2 reached end of life on April 22, 2026, so an Airflow 2 deployment gets no fixes, and the Airflow version life cycle table is the source to confirm current status for any version you run. Airflow 3 is the line to plan around, and the upgrading guide lists changes that affect existing DAGs, so budget the upgrade as a planned engineering task rather than an emergency. Ask your implementation partner for a written upgrade path that covers the Python and Kubernetes prerequisites of the target release, a test environment that runs the production DAGs, and a rollback plan. An exit you cannot execute on schedule is not an exit, so treat these upgrade paths as part of the switching plan.

DeployOpen is a Chainscore Labs company that plans, implements, hardens, documents and supports self-hosted open-source platforms inside the client's own infrastructure, including EU cloud accounts, data centers and air-gapped networks. For a sovereignty pipeline, that means the orchestrator, connectors, staging storage and warehouse are deployed where the client already controls the hosting, and the work starts with a written scope that names every source system, every destination and every place where credentials or intermediate files will sit. Most engagements are fixed-scope, so the scope, deliverables and timeline are agreed in writing before build work begins. Ask any partner to show you the scope document format before you sign, and confirm whether it lists the EU locations of each component, the named administrators and the exit path if you later move to another platform.

The main deliverables are a transfer register and an evidence pack. The transfer register lists every flow that leaves the EU or EEA, including remote support sessions, backup replicas, vendor telemetry and any managed connector that calls an external API, along with the legal basis relied on for each. The evidence pack holds the configuration baselines for Airflow or Dagster, the secrets backend settings, network policies, access reviews and the test results from the pipeline security review. DeployOpen prepares these artifacts for your assessors and your data protection officer, but it does not issue audit reports, attestations or certifications. Confirm with your counsel which legal basis applies to each entry and whether your supervisory authority expects a specific format.

Security testing of the pipeline covers the orchestrator web interface, the scheduler and worker hosts, the connector credentials and the paths where extracted data is staged, since those are the places where a third-country administrator or an unexpected sync could expose data. Testing results feed the hardening backlog, and remediation is tracked against the transfer register so that a closed finding is reflected in the documented flows. Because open-source platforms receive upgrades on their own schedule, retainers can cover patching, version upgrades, restore tests and periodic re-testing after changes. Ask whether the retainer includes testing of each upgrade before production, what the notice period is for security releases, and who is accountable for the register after the initial build.