Open source lakehouse GDPR erasure: Iceberg and Delta

Open source data lakehouse GDPR erasure: what Article 17 asks of your tables

When a data subject asks for erasure, the platform owner of an open source lakehouse has to find that person's records across Iceberg or Delta tables, remove them, and show that the removal reached every place the data could still be read. The short answer is that a row-level delete only marks rows as gone in the current table state. Older snapshots (Iceberg) or earlier table versions (Delta), the data files they reference, and any backups can keep the personal data readable until those copies are expired, rewritten and cleaned up. Erasure is therefore a sequence of table operations and retention settings, not a single DELETE statement.

Article 17(1) of the GDPR lists six grounds for erasure without undue delay: the data is no longer necessary for its purpose, consent is withdrawn, the data subject successfully objects, the processing was unlawful, erasure is required by a legal obligation, and the data was collected from a child for information society services. The controller decides whether a request falls under one of these grounds and which exceptions apply, so the platform team needs the legal and privacy owners to classify requests before anyone runs a delete. Confirm with your data protection officer how each request type is recorded and routed. Under the GDPR text on EUR-Lex, the controller must also be able to show what it did.

The GDPR requires no audit, attestation or certificate, and an erasure process cannot be certified into compliance. What a partner can prepare is the set of controls and evidence an auditor or supervisory authority would ask to see: documented deletion procedures, snapshot and version retention settings, orphan file cleanup schedules, backup handling, and logs of each erasure run. Independent auditors and certification bodies issue any assurance report, and DeployOpen prepares the systems and records for that review rather than issuing it.

Mapping Article 17 and Article 5 duties to lakehouse controls

This table maps erasure duties under Article 17, notification under Article 19 and security under Article 32 to the lakehouse functions that carry them out. Read each row as a control to build, the setting to confirm in your environment and the record that shows it worked.

Duty or function	What it requires	Configuration to confirm	Evidence an assessor expects
Erasure grounds (Art. 17(1))	Erase without undue delay when one of the six grounds applies, such as withdrawn consent or a successful objection.	Intake queue that records the ground, the requester, the due date and the owner for each request.	Request log with ground, decision date and completion date for each erasure.
Row-level delete	Remove the person's rows from every table where they appear, including derived and staging tables.	Identifier mapping across tables, delete support in the chosen engine and the delete write mode for each table.	Delete statements with row counts, plus queries showing zero matching identifiers after the job.
Snapshot expiry	Old snapshots keep deleted rows reachable for time travel, so they must expire on a set schedule.	Maximum snapshot age per table, confirmed with the DPO, and an expiry job scheduled in Airflow or Dagster.	Expiry job run history and metadata showing that snapshots older than the window are gone.
Orphan file cleanup	Files left by failed or replaced writes may still hold personal data after deletion.	Orphan cleanup interval. Iceberg's default retention interval is 3 days, so confirm whether that default is kept or changed.	Cleanup job logs and a file listing showing no unreferenced data files older than the interval.
Position delete rewrite	Position delete files must be rewritten so they stop pointing to data files that are no longer live.	Scheduled rewrite of position delete files after expiry, and the order in which expiry and rewrite run.	Maintenance job history and metadata checks confirming delete files reference only live data files.
Catalog-level retention	Retention rules must apply across the catalog so every table follows the same erasure schedule.	Retention policy per namespace or table in the catalog, with owners named and changes logged.	Retention schedule document mapped to tables, plus change records for policy updates.
Backups (Art. 32 and erasure)	Backups can hold erased data, and restoring an old backup can bring deleted rows back.	Backup retention period, restore procedure that reapplies pending erasures, and backup encryption and access controls.	Restore test records showing erased identifiers stay deleted after a restore.
Recipient notification (Art. 19)	Recipients of the data must be told about erasure unless this is impossible or involves disproportionate effort.	Lineage or catalog records of downstream consumers, exports and embedded dashboards that received the data.	Recipient register and notification records, or documented reasons where notice was not possible.

Which open-source platforms support erasure controls

Erasure in a lakehouse depends on four layers working together: the table format that records deletes, the catalog that tracks snapshots, the engine that runs the delete and cleanup, and the scheduler that runs maintenance on a fixed cadence. Each platform below covers one of these layers, and each has a gap you should confirm before you commit to a design.

Apache Iceberg table format

Iceberg is licensed under the Apache License 2.0 and supports Spark, Trino and Flink against the same tables. Its snapshots and time travel mean a row-level delete does not remove old data files until those snapshots expire, so erasure is a two-step process: delete the rows, then expire snapshots and remove orphan files. Confirm which delete file types your Iceberg format version and engine support, and how your team will verify that expired files are actually gone from storage.

Delta Lake table format

Delta Lake is released under the Apache License 2.0 and has been a Linux Foundation sub-project since 2019. It supports delete operations and time travel to earlier table versions, so the same two-step erasure applies: the delete writes a new version, and the older versions keep the personal data until they are removed by vacuum. Check the retention settings on each table, and confirm which engines in your stack can run the vacuum step.

Catalog for snapshot visibility

The catalog records which snapshot or table version is current, so it determines which files engines can read. Trino's Iceberg connector supports Hive metastore, AWS Glue, JDBC, REST and Nessie catalogs, selected with the iceberg.catalog.type key. Choose a catalog that every engine in your stack can reach, and confirm that catalog changes, such as a commit that expires snapshots, are logged in a way your auditors can trace.

Trino as the query and delete engine

Trino runs SQL deletes against Iceberg and Delta tables, which makes it a common choice for targeted erasure requests from a data protection team. Use it to find a person's rows across tables, delete them, and then run the maintenance procedures that expire snapshots. Confirm in the connector documentation which statements, procedures and file formats your Trino version supports on each table format, because support differs between Iceberg and Delta.

Airflow or Dagster for scheduled maintenance

Erasure is only provable if maintenance runs on schedule, not when someone remembers. Apache Airflow (Apache-2.0, with 3.3.2 published on September 17, 2026) and Dagster (Apache-2.0, with 1.13.26 published on October 9, 2026) can run snapshot expiry, orphan file cleanup and delete-file rewrites as tracked jobs. Ask your partner to define the job schedule, alerting on failures, and the run logs that will serve as erasure evidence.

Evidence auditors ask for when they test erasure

This table lists the records a supervisory authority or auditor may ask for when testing erasure on a lakehouse, and the component that produces each one. Read it as a checklist to design logging before go-live, since evidence that was never written cannot be reconstructed later. The main takeaway is that no single system proves erasure; the case rests on linking the request, the delete, the snapshot expiry and the backup rotation.

Evidence item	Produced by	What it shows
Erasure request log	Ticketing system with the case record	Date received, identity verification, the Article 17 ground relied on, and the decision and response date
Delete job run records	Scheduler (for example Airflow or Dagster) and the engine's job output	Which tables and data subject keys were targeted, when the job ran, and whether it succeeded or failed
Row-level delete history	Table format transaction log and engine audit history	The delete commit for each affected table, showing the rows removed from the current table state
Snapshot expiry logs	Scheduler running Iceberg expire snapshots or Delta vacuum	When old snapshots that could still reference the personal data were expired and removed from metadata
Orphan file cleanup logs	Scheduler running the Iceberg delete orphan files action	Which unreferenced data files were removed, and the retention interval used
Retention schedule and configuration exports	Catalog and table properties, plus version-controlled config	The retention periods, snapshot and file retention settings, and the change history of each
Backup rotation records	Backup platform and storage lifecycle policy	How long copies persist, when expired backup sets were overwritten, and how restores handle erased data
DPIA and record of processing	Privacy team documents, linked to the platform inventory	The processing purposes, storage locations, recipients and the erasure process the controller has documented

Questions buyers ask before hiring help with lakehouse erasure

Can our in-house team handle GDPR erasure on an Iceberg or Delta lakehouse?

Yes, if the team has someone who can run row-level deletes, snapshot expiry and file cleanup on the table formats you use, and who can explain the results to legal. The skills are mostly data engineering: writing delete jobs, scheduling maintenance and checking that old files are gone. The harder parts are usually outside the code: agreeing which systems hold personal data, which backups are in scope and who signs off the retention schedule. Ask your team to delete one test subject end to end, including snapshots and orphan files, and to show the evidence. If that exercise takes more than one attempt, a partner can help build the runbook and the proof. DeployOpen plans, implements and documents these controls inside your infrastructure.

What makes erasure effort grow on a lakehouse?

The effort scales with the surface that holds personal data. The main drivers are:

The number of tables and partitions that contain personal data, and whether identifiers are spread across many of them.
The backup estate, including how long backups are kept and whether they can be restored without re-exposing deleted rows.
The number of query engines and catalogs, since each one needs its own maintenance and access checks.
The catalog choice, such as a Hive metastore, a REST catalog or a managed catalog, because it determines where metadata lives and how table maintenance is triggered.

Ask for an inventory of tables, derived copies and backup locations before agreeing scope. That inventory is the single biggest input to any quote.

What does the erasure work need from legal and data owners?

Legal must confirm the grounds for erasure, the exceptions that apply and the retention periods for each data category, since Article 17 of the GDPR lists when erasure is required and Article 19 covers notifying recipients. Data owners must identify which tables and fields hold personal data, which derived datasets and dashboards copy them, and who approves a deletion request that falls outside normal rules. Security and platform teams need to agree how backups are handled. Without these decisions, the engineering work cannot be finished, so ask for a named owner for each area during scoping.

How long does a fixed-scope lakehouse erasure engagement take to agree?

Most DeployOpen engagements are fixed-scope, and the scope, deliverables and timeline are agreed in writing before any work starts. The agreement depends on the table inventory, the number of engines and backup locations, and how quickly your legal and data owners can confirm decisions. No duration is given here because it follows from that written scope. Ask any partner to show the scope document, the named deliverables and the acceptance criteria before you sign, and check that the document names the systems that are in and out of scope.

What recurs after handover of a lakehouse erasure setup?

Erasure is not a one-time project. Recurring work includes running deletes for each new request, scheduling snapshot expiry and orphan file cleanup, reviewing the retention schedule when rules change, and checking backup expiry. Iceberg's delete orphan files action defaults to a retention interval of 3 days, so the schedule must be set deliberately and tested. Engines, catalogs and table formats also receive upgrades that can change maintenance behavior. Ask for the maintenance calendar, the owner of each recurring job and the evidence each run produces. Ongoing support, upgrades and continuous testing are available as retainers from DeployOpen.

Can DeployOpen certify that our lakehouse is GDPR compliant?

No. DeployOpen prepares systems, controls and evidence for compliance assessments, but it does not issue audit reports, attestations or certifications. The GDPR itself requires no audit or certificate, and Article 42 certification is voluntary, issued by accredited bodies, and does not reduce your responsibility as controller. Your data protection officer, legal team and supervisory authority remain responsible for compliance decisions. What DeployOpen can provide is the configuration, runbooks, test records and deletion logs that your assessors will ask to see.

Which parts of erasure does a partner usually test before handover?

A useful test plan checks the full path, not just the delete statement. Teams should verify that the row is gone from the current table state, that no query can read it through a time travel or rollback snapshot after expiry, that orphan files are removed after the retention interval, and that position delete files no longer reference removed data. Backups and derived copies should be tested against the same request. The EDPB's 2025 coordinated enforcement action on erasure identified ineffective anonymisation used instead of deletion and deletion in backups as recurring problems, so both belong in the test scope. Ask for test records that show each step for a sample request.

Do we need to move off our cloud warehouse to get erasure on a lakehouse?

Not necessarily. Erasure requirements apply to personal data wherever it is stored, so the same deletion, snapshot and backup controls are needed in a cloud warehouse, a lakehouse or both. The lakehouse makes the mechanics visible because you control the files, snapshots and maintenance jobs directly. Many organizations start by applying the erasure process to one dataset and then extend it. Ask any partner to map which existing copies of personal data sit outside the lakehouse, since those copies remain in scope regardless of the platform decision.

What tooling alone does not satisfy

A lakehouse can delete rows and expire snapshots, but erasure under the GDPR is decided by the controller and tested across every place the data has spread. The cards below cover the gaps that no table format closes on its own, and the questions to settle before the first deletion runs.

Lawful basis and retention period

Article 17(1) requires erasure without undue delay on defined grounds, such as data no longer necessary or consent withdrawn. The controller, not the table format, must decide which ground applies, how long each data category is kept, and what triggers the delete. Check that each dataset has a documented purpose and a retention rule before anyone writes a delete statement.

Backups and replicas outside the lakehouse

Object storage replicas, disaster recovery copies and backup sets often hold data that a row-level delete on the live table never touches. The EDPB's 2025 coordinated enforcement action on erasure identified deletion of personal data in backups as one of its recurring challenges. Ask who owns each backup, how long it is retained, and how a restored backup would be re-scrubbed before it returns to production.

Copies in engines, extracts and exports

Query engine caches, BI extracts, notebook outputs, dashboard snapshots and CSV exports create copies that sit outside the table metadata. A delete in the catalog does not reach them. Inventory every consumer of the lakehouse, decide which ones materialize personal data, and set expiry or refresh rules on each so that stale copies do not outlive the source row.

Anonymisation mistaken for deletion

Replacing identifiers with hashes, pseudonyms or aggregates is often presented as erasure. The EDPB's 2025 enforcement action flagged ineffective anonymisation used in place of deletion as a recurring failure. Where the controller relies on anonymisation, document why the remaining data cannot reasonably identify the person, and test that claim against the other datasets the organization holds.

Notifying recipients under Article 19

Article 19 requires the controller to communicate erasure to each recipient to whom the personal data was disclosed, unless this proves impossible or involves disproportionate effort. Lakehouse tooling cannot see who received an extract by email or who holds a shared dataset. Maintain a recipient register linked to the data catalog, so the team can notify downstream parties and record the outcome.

Pending Omnibus proposals, GDPR unchanged

As of October 2026 the GDPR text has not been amended. The Omnibus IV proposal has an agreed text awaiting Parliament's first-reading vote, and the Digital Omnibus proposal awaits committee decision. Build the erasure design against the current regulation, record the proposals in your legal review, and confirm with counsel before changing retention or deletion rules because of them.

Building a retention schedule you can prove

This sequence turns an erasure obligation into a schedule with named owners, configured table maintenance and dated evidence, so the organization can show what it deleted, when, and how it verified the result.

Classify tables by personal data
The data owner and privacy lead inventory every Iceberg or Delta table in the lakehouse and mark which ones hold personal data, which hold pseudonymized keys, and which hold none. Tables derived from personal data, such as aggregates or feature stores, are flagged separately because they may still need erasure or re-computation. The step is done when each table has a recorded classification and a named owner, and the catalog entry reflects it.
Set retention per data category
The controller sets a retention period for each category of personal data, based on its own legal basis, contractual terms and retention decision, not on a default from a platform. The privacy lead records the period, the trigger for erasure (for example, a successful objection or withdrawn consent) and the legal exceptions that apply. The output is a signed retention schedule that maps each table to its category and its erasure triggers.
Configure snapshot expiry windows
Because Iceberg keeps data files referenced by snapshots that may be used for time travel or rollback, the data platform team sets snapshot expiry on each table so that old snapshots, and the files only they reference, are removed on a schedule. The window must come from the controller's retention decision, and the team should confirm that finance or audit time travel needs are documented separately. Delta Lake table history and vacuum settings follow the same logic and need their own review. The step closes when the expiry configuration is versioned and matches the schedule.
Schedule orphan file cleanup
The platform team schedules orphan file removal for each table to clear files left by failed writes that no metadata references. Iceberg's default retention interval for this action is 3 days, and the team should confirm whether that value fits its write patterns before accepting it. Position delete rewrites are scheduled as well, so delete records that point to files no longer live are discarded. The step is done when a scheduled job runs, its output is captured, and a test table shows no unreferenced files after a run.
Test a sample erasure end to end
The privacy lead and data engineers pick a test subject whose rows exist in several tables and run an erasure through the documented procedure: row-level delete, snapshot expiry, orphan cleanup and a check of derived tables. They then query every engine that reads the tables, including time travel queries, to confirm the rows are gone from current and historical views. Any copies in backups or exports are listed and handled under the backup policy. The step is complete when the test subject is absent from every location the test covered, and gaps are logged as findings.
Keep dated run logs as evidence
Every snapshot expiry run, orphan cleanup run, erasure request and verification test is written to a log with the date, table, operator, request reference and result. The logs are stored in the organization's own infrastructure with restricted access and kept for the period the controller sets. Auditors and supervisory authorities reviewing erasure will ask for this trail, so the team confirms that logs can be retrieved per request. DeployOpen prepares the logs and controls for review, and independent assessors decide whether they are sufficient.
Review the schedule on a fixed cadence
The privacy lead and platform owner review the retention schedule, table classifications and maintenance settings at a cadence the organization sets, and after any new source system, table or legal change. Changes to expiry windows or orphan settings go through the same change record as other platform changes. The step closes when the review is minuted, the schedule version is updated, and any re-run of the sample erasure confirms the new settings still work.

Decisions to make before the build

Each of these decisions changes how quickly a deleted person's data actually leaves the platform. Time travel, delete strategy, catalog, backups, ownership and location all interact, so settle them together before the first table is created.

Time-travel retention versus erasure speed

Snapshots keep older data files reachable for time travel and rollback, so a row deleted today can still be queried from an earlier snapshot. Decide how long time travel must work for reporting and audit, then set snapshot expiry to match that window. Ask the team to show the exact query that returns an erased row after a delete, before and after expiry runs.

Delete strategy per engine

Row-level deletes can be applied by rewriting data files (copy-on-write) or by writing delete files that are merged at read time (merge-on-read). The choice affects how quickly personal data is physically removed and how much rewrite work each erasure triggers. Confirm which mode each engine in your stack uses for Iceberg and Delta tables, and test a delete end to end.

Catalog choice and access paths

The catalog records which snapshots and files each table references, so a deletion is only as complete as the catalog's view of the table. Compare Hive metastore, AWS Glue, JDBC, REST and Nessie options against your engines and confirm that every engine reads through the same catalog. Ask who can register tables outside it, because unmanaged copies of personal data are hard to erase.

Backup purge approach

Backups and replicas hold erased rows until they age out or are rewritten, and the EDPB's 2025 enforcement action flagged deletion of personal data in backups as a recurring failure. Define backup retention, whether restores re-apply erasure requests, and how a restored table is checked against the erasure log. Ask the partner to document this as a runbook, not a verbal assumption.

Ownership of erasure requests

An erasure request needs a named owner who verifies the requester, locates every table holding the person's data, runs the deletes, triggers snapshot expiry and orphan file cleanup, and records completion. Split these duties across privacy, data engineering and platform teams in writing. Without a single accountable role, requests stall at the handoff between teams.

Cross-border transfers outside the EEA

If the lakehouse, its backups or its support access sit outside the EEA, erasure evidence and copies of personal data may fall under the GDPR transfer rules in Chapter V. Map where object storage, catalog services and remote administrators sit, and document the transfer basis for each location. Confirm this with your data protection officer before choosing a hosting region or a managed catalog.

How DeployOpen scopes and delivers lakehouse erasure controls

DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside the client's own infrastructure. For erasure on an Iceberg or Delta lakehouse, that means agreeing in writing which tables, snapshot retention and maintenance jobs are in scope, then delivering and evidencing the deletion workflow in your environment.

How we scope

Inventory every table holding personal data, including copies in staging, sandboxes and exports
Agree the snapshot expiry, orphan file cleanup and backup retention periods the erasure process must respect
Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts

How we build

Implement row-level delete jobs for identified subject keys on the chosen table format and query engine
Configure snapshot expiration and orphan file removal jobs so deleted rows leave the storage layer
Test erasure end to end, confirming deleted rows no longer appear in time travel queries after expiry

How we hand over

Document the erasure runbook, retention schedule and maintenance job configuration for your operations team
Produce evidence artifacts such as job logs, test results and configuration records for your compliance reviewers
Train the staff who will run erasure requests, so the process does not depend on one engineer

How we support

Ongoing support, upgrades and continuous testing are available as retainers after handover
Re-test erasure workflows when table format, engine or catalog versions change in your environment
DeployOpen prepares controls and evidence for assessments but does not issue audit reports or certifications