Can Apache Airflow help with GDPR records of processing and security?
Airflow helps with the technical inputs to a GDPR record of processing activities and with some Article 32 measures, but it does not produce the record, decide the lawful basis for any processing, or certify compliance. If your data or compliance lead runs scheduled pipelines that touch personal data, the DAG code is the most useful starting point, because each DAG, task and connection shows where data is read, transformed and written. From that code you can draft the processing inventory that Article 30(1) asks for, covering purposes, categories of data subjects and personal data, recipients, third-country transfers and, where possible, erasure time limits. The controller still supplies the contact details, the purposes, the legal basis analysis and the recipient list, because none of those live in a scheduler.
On the security side, the platform can support several Article 32 measures. Access to DAGs, triggering and clearing runs, and viewing logs can be restricted through the auth manager that your version uses, and Airflow 3 allows only one auth manager at a time, set by the auth_manager option. Run history and audit logs give evidence of who acted on which pipeline. Connection secrets belong in a secrets backend rather than in DAG files, and task logs need a retention decision of their own. Hosting is also a choice you control: deploying the metadata database and workers in an EU region, with PostgreSQL or MySQL rather than the SQLite default that the production guide says is for testing only, keeps the processing inside infrastructure you select. Confirm with your cloud or data center provider where backups and log shipping endpoints are located, because those copies count as processing too.
Two points set expectations. The GDPR requires no audit, attestation or certificate, and Article 42 certification is voluntary and does not reduce your responsibility. The Article 30(5) relief for organizations with fewer than 250 employees applies only when processing is occasional and unlikely to create a risk, and it falls away for special category or criminal offence data. Scheduled pipelines that run regularly on personal data will usually fall outside that relief, so plan to keep the record. Ask a partner to produce the DAG-derived inventory and the Article 32 control mapping, and have your legal or privacy team own the final record and its review.
Mapping Article 30 and Article 32 requirements to Airflow configuration
This table maps each Article 30(1) record field and the Article 32 security concerns to the Airflow capability or configuration that supports it. Read each row as a question to answer for your own DAGs, then confirm option names and behavior against the Airflow documentation for the version you run. The main takeaway is that Airflow supplies the controls, but the record content and the decisions behind it still come from your team.
| Requirement | Airflow capability or configuration | Evidence to keep |
|---|---|---|
| Controller contact (Art. 30(1)(a)) | Not an Airflow setting. Record the controller and DPO contact in the processing register and reference it in DAG ownership tags or documentation. | Register entry linked to each owning team |
| Purposes of processing (Art. 30(1)(b)) | One DAG per processing purpose where practical, with a description and owner tag set in DAG code. | DAG source in version control with purpose stated in the DAG docs |
| Data subject and data categories (Art. 30(1)(c)) | Tag DAGs and tasks that touch personal data, and map each connection ID to the data categories it carries. | Connection inventory and tag report exported from the metadata database |
| Recipients and third-country transfers (Art. 30(1)(d)(e)) | Review each connection and operator for destinations outside the EEA, including remote access and cloud storage, before adding it to a DAG. | Connection list with hosting location and transfer assessment reference |
| Erasure time limits (Art. 30(1)(f)) | Set run-history cleanup and task log retention to the period your register states, and confirm metadata database cleanup settings for the version you run. Pipeline outputs are governed by the target system, not Airflow. | Cleanup configuration in version control and cleanup job logs |
| Risk of destruction or loss (Art. 32(2)) | Run PostgreSQL or MySQL as the metadata database, not SQLite, and back it up with tested restores. Use more than one scheduler only on a database that supports the row-level locking Airflow relies on. | Backup schedule, restore test records and supported-database version |
| Risk of unauthorised access (Art. 32(2)) | Use the auth manager set by the auth_manager option with role-based access, limiting who can trigger, clear or view runs and connections. Confirm role names in the auth manager documentation. | Role matrix, access review records and auth manager configuration |
| Risk of unauthorised disclosure (Art. 32(2)) | Store connection credentials in a secrets backend rather than in DAG code or variables, and restrict who can read connections. Confirm masking behavior for your chosen backend. | Secrets backend configuration and list of accounts with read access |
| Risk of alteration (Art. 32(1)) | Keep DAG code in version control, separate development, test and production environments, and promote changes through review. | Pull request approvals and deployment records per DAG version |
Building the processing record from DAG code
This sequence turns the DAGs in your Airflow environment into a reviewed inventory of personal data flows, with each flow tied to its GDPR Article 30 record fields and to the Article 32 controls that protect it. The result is a record you can maintain as pipelines change.
Export the DAG source and deployment configuration
Collect the DAG code from the repository or bundle that your Airflow deployment loads, along with the deployment configuration for each environment, including the DAG processor, workers and connection definitions. Read the inventory from these files rather than from the metadata database, because Airflow 3 runs the DAG processor as a separate component and task code must reach Airflow through the REST API or Task SDK, not database sessions. The output is a frozen list of DAG files and the commit each one came from. Confirm that every production environment is covered, since a DAG that exists only in one environment is still a processing activity.
Identify DAGs and tasks that touch personal data
For each DAG, review its tasks, operators, connections and data sources to decide whether it reads, writes, transforms or forwards personal data. Pay attention to tasks that call external APIs, load files from object storage, or write to shared tables, because these often carry identifiers that are not obvious from the DAG name. Record the DAG ID, the task IDs and the connection IDs involved. The output is a register of personal data DAGs, and a short list of DAGs you have reviewed and ruled out, with the reason for each.
Tag each flow with its Article 30 attributes
For every personal data DAG, record the purpose of processing, the categories of data subjects and personal data, the categories of recipients, and any third-country transfers, including remote access from outside the EEA and storage in a cloud outside the EEA. Your privacy or legal owner should confirm the purpose and recipient entries, while the engineering owner confirms the technical facts from the code. Store these tags as structured metadata alongside the DAG, such as in DAG tags, documentation or a version-controlled register, so the record can be exported. The output is one tagged record per flow.
Map erasure periods to run data cleanup
Article 30 asks for erasure time limits where possible, so set a retention period for each personal data flow and compare it with how Airflow keeps run data. Identify which artifacts hold personal data: task logs, XCom values, rendered templates, data written by tasks, and records in the metadata database. Then configure the cleanup that applies to each, and check the database cleanup and log retention settings in the environment rather than assuming defaults. The output is a retention table that links each flow to the cleanup job or external store policy that enforces its limit, plus a note of any limit you cannot yet enforce.
Link each DAG to its Article 32 control set
Article 32 requires security measures appropriate to the risk, so assign each personal data DAG a control set covering access to trigger and clear runs, secrets and connection handling, masking of personal data in logs, encryption in transit for connections and log shipping, and restricted access to task logs. Note which controls are provided by Airflow, which by the platform (identity provider, network, database), and which by process. The output is a per-DAG control list with an owner for each control, and a list of gaps that need a decision before go-live.
Record the link between each DAG and its version
Connect every record to the commit or bundle version that introduced the flow it describes, so an auditor can see which code produced which processing activity on a given date. Keep the record in the same repository as the DAG code, so a change to a personal data DAG requires an update to its record in the same pull request. The output is a record that references the commit, with the review that approved both the code and its Article 30 entries.
Review the record on every DAG change
Add a check to the pull request template or CI pipeline that flags changes to personal data DAGs, new connections, new external destinations or changes to cleanup settings. A named reviewer confirms that the tags, retention entries and control set still match the code before merge. Run the same comparison during Airflow upgrades, since the upgrade guide for Airflow 3 changes how DAG code and components are structured. The output is a reviewed record and a log of the approval for each change.
Evidence Airflow produces for a GDPR file and how to keep it
A GDPR file for a self-hosted Airflow combines evidence the platform generates with evidence the controller must hold elsewhere. Each item below names what to keep, how long to keep it, and who reviews it, with retention periods set by the controller's own policy.
Versioned DAG source in Git
Every DAG is Python code, so the Git history is the record of what a pipeline did to personal data at a given time. Tie each run to the commit or bundle version it executed, so a question about a past run can be answered from source. Check that merges require review and that the repository keeps history rather than rewriting it. Retention is set by your policy, but the repository should outlive the longest data retention period of any pipeline it contains. Reviewed by the data platform owner and the privacy lead when processing changes.
Run history and task logs
Run history shows which DAG ran, when, with what outcome, and which tasks touched which systems. Task logs record the detail behind each step, and they are the most likely place for personal data to appear by accident. Decide which fields are logged, mask or exclude identifiers, and set a retention period for logs that matches the pipeline's purpose. Reviewed by the platform team for log hygiene and by the data protection officer for retention exceptions.
Audit logs of access and changes
Audit logs record who triggered, cleared, paused or edited runs and who changed connections or variables. They support the Article 32 question of who could reach personal data and whether that access was authorized. Confirm in your Airflow version and log shipping setup which events are captured and where they are stored, and make sure the store is write-protected from the people whose actions it records. Reviewed by the security team on a set schedule and after any access review.
Scheduler and metadata database backups
The metadata database holds run state, connections references and history, so it is both an operational asset and a record. Back it up on a schedule, encrypt the backups, and store them in the same jurisdiction as the primary data. Test restores, because a backup that has never been restored does not show that the record survives. Backup retention is a controller decision and must balance recovery needs against the erasure duties that apply to copies. Reviewed by platform operations, with restore tests signed off by the security owner.
Access role assignments
Role assignments show who could view, edit or run each pipeline and which connections they could use. Export the current assignments on a fixed cadence and keep dated copies, so you can show access at a past date. Confirm how roles are defined in the auth manager you run, since Airflow 3 allows only one auth manager at a time. Reviewed by the access owner for each data domain, with leavers and movers removed promptly.
The processing record kept outside Airflow
The Article 30 record itself should live in your records management system, not inside Airflow. Use DAG metadata, tags or documentation to point each pipeline to its record entry, and keep the entry with controller contact details, purposes, data subject categories, recipients, transfers and erasure periods. Where full erasure time limits are not yet decided, record that as an open item. The data protection officer or privacy lead owns the record, and it should be reviewed whenever a pipeline changes its data.
Where EU hosting and transfer rules change the Airflow design
When your organization runs Airflow on infrastructure it controls and that is located in the EEA, the design question is mainly internal access and storage location. Personal data stays in EEA systems, and no transfer occurs simply because the scheduler, metadata database, and workers run there. EDPB Guidelines 05/2021 treat storage in a cloud outside the EEA as a transfer, so the hosting location of every component matters, including backups and log storage.
- Component location: Confirm that the metadata database, worker nodes, log storage, secrets backend, and backup targets all sit in approved EEA regions.
- Support access: Decide whether vendor or partner staff outside the EEA can reach consoles, logs or task output, because remote access from a third country is a transfer even when data is only displayed.
- Container images and packages: Image registries and package mirrors are not personal data stores, but confirm where they are pulled from during deployments.
- Backups and replicas: Disaster recovery sites outside the EEA would bring transfers into the picture, so record them in the processing record.
- Third-country law: Even an EU-based provider can be subject to third-country law, so assess that exposure under Article 48 and the CLOUD Act before choosing a provider.
Confirm each location against your own cloud or data center configuration rather than relying on a general description from a provider.
If a provider headquartered or operating outside the EEA runs the Airflow platform, or hosts it in a non-EEA region, the processing involves a transfer that must meet Chapter V conditions. Article 44 requires every transfer, including onward transfers, to meet those conditions, and infringements of Articles 44 to 49 carry fines of up to EUR 20 million or 4% of worldwide annual turnover under Article 83(5)(c). The analysis must cover both where data sits and who can reach it.
- Transfer mechanism: Identify the basis for the transfer, such as an adequacy decision or standard contractual clauses under Commission Implementing Decision (EU) 2021/914, and document the transfer impact assessment.
- Government access: Under the US CLOUD Act provision at 18 U.S.C. 2713, providers must preserve or disclose data in their possession, custody or control regardless of location. Ask the provider how it handles such requests and whether it will challenge them.
- Article 48 limits: A third-country court order can be recognized in the EU only under an applicable international agreement, so ask how requests would be handled in practice.
- Supplementary measures: EDPB Recommendations 01/2020 state that where a provider needs data in the clear and foreign authority powers go beyond what is necessary and proportionate, no effective technical supplementary measure can be envisioned.
- Contract terms: Require notification commitments, audit rights and clear sub-processor lists in the processing agreement.
Ask the provider in writing which jurisdictions can compel access to your data and under what process.
A common hybrid model runs Airflow in an EU environment while a support partner outside the EEA troubleshoots through screen sharing, remote sessions, or log exports. EDPB Guidelines 05/2021 state that remote access from a third country, even only displaying data on a screen for support or administration, is a transfer. The platform being EU-hosted does not remove the Chapter V analysis for these sessions, so the support model must be designed alongside the hosting model.
- Access inventory: List every support channel, including screen sharing, VPN access, shared consoles, and ticket attachments that contain task logs or DAG output.
- Just-in-time access: Route support through approved bastions with session recording and time-limited credentials, so access to personal data is granted only for specific tickets.
- Masking before sharing: Redact task logs, connection details and XCom values before they leave the EEA, and keep sensitive payloads out of logs where possible.
- Transfer documentation: Record each remote access scenario as a transfer in the Article 30 record, along with its safeguards.
- Escalation rules: Define when a support engineer may view personal data, who approves it, and how it is logged.
Ask your support partner to state which of its personnel and locations would access production systems, then test that answer against your access logs.
Transfer decisions for US-linked providers depend partly on the EU-US Data Privacy Framework. Commission Implementing Decision (EU) 2023/1795 finds adequate protection only for US organizations on the Data Privacy Framework List, and an appeal against the General Court's September 2025 judgment upholding it is pending as Case C-703/25 P. The outcome is therefore not settled, and designs that rely on the framework alone carry a planning risk.
- List check: Confirm that the specific legal entity you contract with appears on the Data Privacy Framework List, not only the parent brand.
- Fallback plan: Keep standard contractual clauses and a transfer impact assessment ready as an alternative basis, given the pending appeal.
- Schrems II history: The Court of Justice invalidated the Privacy Shield in Case C-311/18 while upholding standard contractual clauses, which is why clause-based transfers still need case-specific assessment.
- Data minimization: Keep personal data out of pipelines that do not need it, so fewer transfers need to be justified.
- Review trigger: Schedule a review of the transfer basis when the appeal is decided or when the provider changes its sub-processors.
Confirm with your legal counsel which transfer basis applies to each provider entity and document the reasoning in your records.
What Airflow does not cover for GDPR
Airflow can show where personal data moves and how pipelines are run, but it leaves several GDPR duties with the controller. The cards below mark the gaps your records, security measures and incident process must fill before an assessor will accept the platform as evidence.
The Article 30 record itself
Airflow does not write your record of processing. Controller contact details, purposes, data subject categories, recipients, third-country transfers and erasure periods must come from your privacy function, and lawful basis is a legal decision no scheduler makes. Use DAG definitions as an input to the record, then confirm each activity with the business owner who approved the pipeline.
Personal data discovery in payloads
Airflow does not classify data or detect personal data inside the records that tasks pass through. A DAG that reads a customer table looks identical to one that reads product codes. Your data classification process must tag sources and targets, and you should sample XCom values, staged files and task output to confirm what the pipelines actually carry.
Data subject requests downstream
Airflow cannot find or erase a person's data in the warehouses, lakes or SaaS tools its tasks write to. Access and erasure requests need a mapping from each DAG to its downstream stores, plus a documented process for re-running or excluding data. Ask who owns that mapping and how a deleted record stays deleted after a backfill or a clear-and-rerun.
Breach notification under Articles 33 and 34
Airflow detects failed runs, not personal data breaches. Notifying the supervisory authority within the Article 33 window and informing individuals under Article 34 depends on your incident process, which must weigh the EDPB breach guidelines. Confirm that a misdirected DAG output or an exposed connection secret reaches your incident team, and that task and audit logs are preserved for the investigation.
No audit or certification
Airflow is open-source software and does not issue audit reports, attestations or certificates. The GDPR itself requires none, and Article 42 certification is voluntary and does not reduce your responsibility. Evidence from your run history and configuration is only a starting point, and you should ask an assessor or certification body which artifacts they will accept.
Personal data inside logs
Airflow task logs, scheduler logs and XCom views can capture personal data printed by a task or an error message. Those logs are themselves processing and need masking, access limits and a retention period set by your policy. Test a failing task with a sample record, confirm what reaches the log store, and check whether your log shipper stores it outside the EEA.
Why the fines and enforcement cases matter for pipeline security
Article 83 sets two fine tiers. Infringements of Articles 25 to 39, which include the security of processing in Article 32, carry maximum administrative fines of EUR 10 million or 2% of total worldwide annual turnover, whichever is higher. Infringements of the core principles, lawful basis, data subject rights and transfer rules carry the higher tier of EUR 20 million or 4%. For an Airflow platform, this means a weak pipeline security posture usually sits in the lower tier, but the record of processing under Article 30 and the transfer conditions under Chapter V can fall in the higher one. Treat the DAG inventory as evidence for both, and ask your counsel which tier a given gap would most likely fall under.
Two recent national cases show where regulators focus. The Swedish supervisory authority fined Miljödata i Karlskrona for insufficient technical measures, and the CNIL fined Hôpital Privé de la Loire EUR 500 000 after a health data breach. Read both through the EDPB national news summaries for the findings that were published. Neither case shows that a particular platform would have prevented the failure. The useful question for your team is which of your own pipelines would be assessed the same way: which ones move health or other sensitive data, who can read their task logs and connections, and whether you could show a regulator the controls that applied on the date of an incident.
Two procedural and legal points affect planning. Regulation (EU) 2025/2518 lays down additional procedural rules for cross-border enforcement and applies from 2 April 2027, so your evidence should be kept in a form that a lead authority in another Member State could request. The Omnibus proposals, COM(2025) 501 and COM(2025) 837, have not changed the GDPR text as of the supplied date of October 2026, so the articles cited on this page still apply. Confirm the current status of both procedures with your legal team before each review, because a text agreed in negotiations is not yet law until adopted.
What do buyers ask before setting up Airflow for GDPR?
Does a small organization have to keep an Article 30 record of processing?
Not always. Article 30(5) exempts organizations with fewer than 250 employees, unless the processing is likely to result in a risk to individuals, is not occasional, or includes special category or criminal offence data. The EDPB's small business guide says such organizations need not record purely occasional activities. A scheduled Airflow pipeline that runs every day on customer or employee data is usually not occasional, so the exemption is unlikely to cover it. Confirm the position with your DPO or privacy lead, using your headcount and the categories of data each DAG touches.
What does a GDPR records and security engagement cost, and what drives the price?
DeployOpen does not publish prices. A quote follows from the written scope, and most engagements are fixed-scope. The main cost components are the inventory of DAGs and data flows, interviews with data owners and DAG authors, the write-up of records entries, the Airflow configuration work for access, masking, logs and retention, and any testing you request. The biggest drivers are the number of DAGs and connections, how many systems each pipeline touches, how well existing documentation matches the code, and whether the deployment already exists or must be built. Work on the records is largely one-time, while reviewing new DAGs and keeping records current recurs. Ask each bidder to separate those items.
How long does the work take, and what depends on the size of the DAG inventory?
Duration depends mainly on the size of the DAG inventory and how many personal data flows it contains. A small set of pipelines that move data between two systems is faster to map than a warehouse estate with cross-border sources and many teams. Other factors are the time needed to get answers from data owners, the state of the deployment (existing, partly built or not yet installed), and the number of approval rounds on the records. Ask for a timeline tied to the written scope, with the inventory phase, configuration phase and review phase shown separately, and confirm which dates depend on your staff's availability.
What does the work need from our team?
Plan for four groups. Data owners confirm what personal data each pipeline handles, why, and for how long it is kept. DAG authors explain the code paths, sources, destinations and any tasks that copy data to other systems. Your DPO or privacy lead approves the purposes, lawful basis references, recipient categories and retention periods. Infrastructure staff provide access to the Airflow environment, the metadata database, log storage and secrets backends, and they review the configuration changes. Agree on named contacts and response times before the inventory starts, because slow answers are one of the most common reasons these projects run late.
Can an internal team do this without a partner?
Yes, if it has the skills and time. The work needs someone who can read DAG code and Airflow configuration, someone who understands the GDPR records requirements, and someone who can run the access and logging changes in your environment. Many internal teams handle the records and the privacy review while a platform team handles the configuration. A partner is most useful when the inventory is large, the deadline is fixed, or the same people cannot cover both the engineering and the privacy work. Either way, the records remain the controller's responsibility.
Can a partner certify that our Airflow deployment is GDPR compliant?
No. The GDPR requires no audit, attestation or certificate, and DeployOpen does not issue audit reports, attestations or certifications. Article 42 certification is voluntary, is issued for at most three years by accredited certification bodies or supervisory authorities, and does not reduce the controller's or processor's responsibility. What a partner can deliver is a documented inventory, records entries, configured controls for access, masking, logs and retention, and evidence of those controls. Ask any provider which of those outputs they will hand over, and keep the compliance decision with your DPO and legal team.
What a DeployOpen engagement covers and what stays with you
A DeployOpen engagement for this topic is a fixed written scope covering five workstreams: an inventory of the DAGs that touch personal data, configuration of access to those DAGs and their connections, secrets handling for the credentials the pipelines use, retention settings for task logs and run history, and a deployment hosted in the EU or another location you specify. The scope also includes packaging the resulting configuration, access lists and run evidence so your assessors can review them. Scope, deliverables and timelines are agreed before work starts, so the boundary of each workstream, including which DAGs and data sources are in and out, should be settled at that stage rather than discovered during the build.
The controller keeps ownership of the Article 30 record of processing and of every legal decision in it. You decide the purposes, the lawful basis, the categories of data subjects and recipients, whether any transfer to a third country occurs, and the erasure periods you can justify. DeployOpen can help you map which DAGs, connections and tasks move personal data, and can document what the platform does with that data, but the record itself and the conclusions drawn from it remain yours. DeployOpen prepares systems, controls and evidence for a compliance assessment. It does not issue audit reports, attestations or certifications, and certification under Article 42 is issued by accredited bodies, not by an implementation partner.
Several items stay with your team after handover. Your data protection officer or privacy lead should approve the inventory and the retention periods, your security team should own the identity provider, secret store and key rotation schedule, and your operations team should run the scheduled reviews of access and logs. Before scoping, confirm with your legal team whether any connected service, support access or backup location could expose data to a third country, because the EDPB's Guidelines 05/2021 treat remote access from outside the EEA as a transfer. Ask for the written scope to name who performs each task and what evidence each deliverable produces.
