Open source data pipelines for GDPR

Article 5(1)(e) of the GDPR requires personal data to be kept in a form that permits identification of data subjects for no longer than is necessary for the purposes of the processing. In a pipeline, that means retention must be set per dataset and enforced by scheduled jobs, not left to a quarterly clean-up request. For each dataset, the team should record the purpose, the storage period or the criteria used to set it (the same information privacy notices must state under Articles 13(2)(a) and 14(2)(a)), and then encode it as a parameter in the orchestrator. In Apache Airflow or Dagster, that usually means a dated purge task per table or partition, with its run history kept as evidence that it executed and what it removed.

The copies that persist are rarely the ones people plan for. Staging tables that were meant to hold a single load, intermediate files written to object storage between extraction and transformation, failed-run residues, and backfill outputs can all keep identifiable records long after the source record was deleted. Warehouse partitions in ClickHouse or a lakehouse table are another place where history quietly accumulates. Article 89(1) permits longer storage only for archiving in the public interest, scientific or historical research or statistical purposes, and only with appropriate technical and organisational measures, so a retention exception should be a documented decision with its own owner, not a default setting. Ask your partner to list every location where each dataset is written, including temporary paths, and to show the purge job that covers each one.

The EDPB's news item of 18 February 2026 on the challenges hindering the full implementation of the right to erasure is a useful reference for what auditors and supervisors will ask. The practical test is whether a deletion or rectification request reaches every downstream copy, including replicas, aggregates derived from identifiable rows, and exports. Pipelines should carry a stable subject identifier that lets deletion jobs locate records across staging, warehouse and extracts, and should log each propagated deletion. Confirm with your partner how deletions are applied to append-only stores and to backups, and what happens when a late-arriving batch re-inserts a record that was already erased.

Pipeline metadata can supply most of the fields a controller must record under Article 30(1), but it cannot be the record itself. The controller owns the written record of processing activities, which must name its contact details, purposes, categories of data subjects and personal data, categories of recipients, third-country transfers, erasure time limits where possible, and a general description of Article 32 security measures. In a self-hosted stack, the practical question is which of those fields already exist in machine-readable form. DAG definitions in Airflow, asset definitions in Dagster, connector configurations in Airbyte and the tags and owners in a catalog such as DataHub or OpenMetadata can carry purpose, data category, source system, destination and owner. Lineage then shows which downstream tables a personal data field reaches, which is the evidence needed to list recipients and to confirm that a transfer did or did not occur.

Map each Article 30(1) field to a named source before you build anything. For example, a team could require every DAG or asset to carry a purpose tag, a data_category tag drawn from a controlled list, and a retention_days value, then have a scheduled export pull those values into the record. This is an illustrative pattern, not a feature of any product. Test it with the data protection team, because lineage captures movement, not intent: it cannot tell you why a field was collected or whether a legal basis exists. Erasure limits are often the weakest link, since they depend on retention jobs that must actually run. Confirm that the warehouse retention logic matches the value stated in the record and in privacy notices under Articles 13(2)(a) and 14(2)(a).

Processor records under Article 30(2) are a different document. A DeployOpen-style implementation partner or an internal platform team acting for a client is usually a processor for the pipeline work, so it must record the categories of processing it carries out for each controller, any third-country transfers and, where possible, a general description of its security measures. Do not reuse the controller's metadata for that purpose. Agree with the controller who maintains which record, where it is stored, and how changes to DAGs, flows or connectors trigger an update. Pipeline metadata supports both records but does not replace the written, approved documents that a supervisory authority will ask to see.

Before you write a single pipeline requirement, read the GDPR text itself, starting with the GDPR on EUR-Lex. Article 5(1)(e) on storage limitation, Article 30 on records of processing and Article 32 on security of processing are the provisions that translate directly into pipeline design: they decide how long staging tables may hold identifiable data, what each pipeline must document about its recipients and transfers, and which technical measures your orchestrator and connectors must support. Note that the GDPR has no audit or certificate requirement, so the controls you build are the evidence you will later show your data protection officer or a supervisory authority.

Next, read the EDPB Guidelines 01/2025 on pseudonymisation, which were adopted in January 2025 as a version for public consultation, so confirm whether a final version has since been published. Their central point for pipeline architects is that pseudonymised data remain personal data while the additional information needed to re-identify them exists. In practice, that means the key table or tokenization service must sit in a separate, access-restricted zone from the warehouse, and the pipeline's pseudonymization step should be tested as a control, not assumed to work. Finally, read the EDPB Guidelines 9/2022 on personal data breach notification and the Kafka security overview. The breach guidance shapes how quickly your monitoring must detect a compromised connector or exposed staging bucket, because notification clocks start at awareness. The Kafka overview explains that security is optional and that clusters can run mixed secured and unsecured listeners, so confirm in writing that every topic carrying personal data uses encrypted, authenticated listeners before you sign off the design.