How GDPR erasure applies to self-hosted Kafka topics
If personal data flows through your Kafka topics, you are the controller, and erasure requests under Article 17 apply to those records wherever they sit. Kafka handles some of the mechanics: time-based retention through retention.ms and cleanup.policy, tombstones that remove earlier values for a key in compacted topics, encryption in transit with SSL/TLS, and ACLs stored in the cluster metadata log that restrict who can read or write each topic. Kafka does not decide whether an erasure request is valid, which subjects appear in which topics, or what happens to copies held elsewhere. Those belong to your platform and your processes, and they should be designed before the first request arrives.
The work outside Kafka starts with a lawful grounds assessment. Article 17(1) lists six grounds on which erasure must happen without undue delay, and Article 5(1)(e) limits storage to what the purpose requires. Your team then needs an index that maps each data subject to the topics, partitions, keys, connectors and sink tables where their data lands, because a tombstone only works if the key is the subject identifier and every producer uses it consistently. Decide early whether topic keys carry direct identifiers at all, and whether a pseudonymous key with a separately protected lookup table, in the sense of Article 4(5), is a better fit for your flows. Confirm the mapping with your legal and data protection teams rather than assuming it from the schema.
Downstream deletion is where most erasure gaps appear. Sink connectors, stream processors, search indexes and data lakes keep their own copies, and Article 17(2) requires reasonable steps to inform other controllers where data has been made public. Document a deletion procedure for each sink, test it on a non-production cluster, and record evidence of each run. Legal holds also need a defined path, since a hold can justify retaining specific records beyond the normal period. For a compliance readiness review, DeployOpen can help prepare these controls and their evidence, but the GDPR requires no certificate, and only independent auditors or certification bodies issue audit opinions or attestations.
Mapping GDPR articles to Kafka settings and platform controls
This table maps each GDPR provision in scope for erasure and storage limitation to the Kafka settings that support it, and to what the surrounding platform must add. Kafka provides configuration and mechanisms, not compliance: read each row as a set of inputs to your own erasure design, and confirm every setting against your Kafka version before relying on it.
| Requirement | Kafka capability | Configuration to set | What the surrounding platform must add |
|---|---|---|---|
| Article 5(1)(e) storage limitation | Time-based and size-based log retention deletes old log segments. Retention does not depend on the content of individual records. | retention.ms and retention.bytes per topic, set to the purpose and lawful basis of each topic. Confirm the default of 7 days with your version. | A documented retention schedule per topic, justified by purpose, plus a check that producers cannot override retention without approval. |
| Article 17(1) erasure without undue delay | Kafka has no per-record delete for ordinary topics. Records leave the log only when retention removes their segment or when a compacted key is replaced by a tombstone. | cleanup.policy set to compact for keyed state topics, with a tombstone (null payload) written on erasure and delete.retention.ms set to the time tombstones must remain visible to consumers. | An erasure request workflow that identifies every topic and key holding the subject's data, issues tombstones or key destruction, and records completion. |
| Article 17(2) informing other controllers of copies and replications | MirrorMaker 2 replicates topics to other clusters, so a tombstone or retention change in one cluster does not remove replicas elsewhere. | Confirm that tombstones and retention settings are replicated and applied on every target cluster, and that replication targets are listed in your processing register. | A register of every cluster, mirror and connector that copies a topic, and a process to propagate erasure to each recipient that may hold the data. |
| Article 19 notification to recipients | Kafka does not track which consumers or external systems received a record. Consumer groups and offsets are visible through admin tooling, but they do not show downstream copies. | Use dedicated consumer groups per recipient, and restrict consumption with ACLs so each recipient is known. | Recipient inventory, contracts or processing terms, and a notification procedure that reaches every recipient that received the data, including those reached through connectors. |
| Article 32(1)(a) pseudonymisation and encryption | Kafka encrypts traffic with SSL/TLS between clients, brokers and tools, and authenticates with SASL mechanisms. Encryption of record payloads at rest is not a broker feature. | listeners using SSL, inter-broker and client SASL mechanisms such as SCRAM-SHA-512 or OAUTHBEARER, and TLS on MirrorMaker 2 links. Security is optional, so disable plaintext listeners. | Encryption at rest for log directories, field-level encryption or pseudonymised identifiers applied before produce, and key management outside the cluster. |
| Article 4(5) pseudonymisation with separately kept additional information | Kafka stores whatever bytes producers write. It does not separate identifiers from payloads or hold a mapping table. | ACLs on topics that hold the mapping or key material, so only the key service and authorized roles can read them. The StandardAuthorizer denies resources without ACLs to super users unless allow.everyone.if.no.acl.found is set. | A separate store for the additional information, with its own access control and protection, and a documented decision on who may reverse pseudonymisation. |
Choosing an erasure pattern for each topic
Time-based retention fits topics whose records are events with a clear useful life, such as clickstream, notifications or transient status messages. Personal data leaves the log when segments age past the configured retention, which makes the storage limitation principle under Article 5(1)(e) the primary control. The erasure request itself is handled by ensuring the purpose does not outlast the retention period, not by deleting individual records.
- Retention is per topic:
retention.msdefaults to 7 days, so set it to the shortest period the stated purpose needs and record the reason for that value. - Deletion is segment-based: Kafka removes whole log segments, so a record may persist until its segment is eligible for deletion, not the moment the retention period ends.
- Consumers hold copies: Data already read and written by downstream systems is outside Kafka's retention and must be covered by those systems' own deletion rules.
- Individual erasure is not available: A single event cannot be removed on request without rewriting the log, so a request must be answered by policy and timing.
- Document the retention basis: Record the purpose, the retention value, the review date and who approved both.
Confirm with your legal and privacy team whether the retention period is defensible for each purpose before you rely on it.
Compacted topics fit keyed state, such as a current customer profile or account status, where consumers need the latest value per key. A tombstone, a record with a key and a null payload, tells the broker to delete earlier values for that key. This gives a way to remove personal data for a specific subject without waiting for time-based expiry, but only for the key the tombstone names.
- Compaction removes superseded values: Only earlier records for the same key are eligible for removal, so data under other keys or in unrelated topics is unaffected.
- Tombstones are not immediate: Tombstones are removed after
delete.retention.ms, 24 hours by default, so the erasure is complete only after that period and a compaction run. - Active segments are not compacted: The current active segment is not cleaned, so recent values may remain until it rolls. Confirm the segment settings for your topic.
- Copies persist: Older segments, consumer caches, replicas in other clusters and sink systems keep the value until they are cleared or they process the tombstone.
- Key design matters: If the key is a pseudonymous identifier, the mapping must be maintained separately and protected, as Article 4(5) requires.
Verify with a test topic that a tombstone removes the value after compaction and that downstream sinks handle null payloads as intended.
Crypto-shredding fits personal data that must become unrecoverable on request even though immutable or long-lived log segments keep the ciphertext. Each data subject gets a key, the producer encrypts personal fields with that key before publishing, and erasure means destroying the key. The encrypted records remain in the log but no longer reveal the data. This approach relies on the protection measures named in Article 32(1)(a) and on keeping the key store separate from the log.
- Keys become the erasure point: Destroying a per-subject key in a key management system renders every copy of its ciphertext unreadable, including copies in backups and downstream systems that hold only ciphertext.
- Key store design: Keys must be held in a system separate from Kafka, with access control, audit logging and verified deletion, not just removal from a configuration file.
- Consumers need the key path: Any consumer that reads plaintext must fetch keys through a controlled service, and a missing key must produce a defined outcome rather than an error loop.
- Backups and replicas: Destroyed keys must also be absent from key backups, or the shredding is incomplete.
- Pseudonymisation is not erasure by itself: Under Article 4(5) the link information must be kept separately, so the controller must still document whether shredded data counts as anonymous or pseudonymous.
Have a qualified privacy and security reviewer confirm the key lifecycle and the legal status of the remaining ciphertext before the design is adopted.
Erasure obligations do not stop at the broker. Article 17(2) requires controllers who made data public to take reasonable steps, including technical measures, to inform others about erasure requests for links, copies or replications. Within a self-hosted platform, the same logic applies to Kafka Connect sinks, stream processors, search indexes and replicated clusters, each of which can hold a copy the topic pattern does not control.
- Connector sinks write their own copies: Databases, search indexes and object stores receive data through connectors and need their own deletion or update path.
- Replication copies data: MirrorMaker 2 replicates topics to other clusters, so retention or tombstones must be applied in each cluster where the data lives.
- Stream state is persistent: Kafka Streams state stores and changelog topics keep derived data that outlives the input topic's retention.
- Inventory every flow: Keep a list of which topics carry personal data, which consumers read them and where each derived copy is written.
- Notification duties: Where a controller made data public, record the steps taken to inform other controllers.
Before sign-off, trace one test subject through every topic, connector and replica to confirm each copy is removed or shredded under the chosen pattern.
Crypto-shredding with per-subject keys
Crypto-shredding makes a stored record unreadable by destroying the key that encrypts it, without touching the ciphertext already sitting in Kafka segments. Apache Kafka does not provide field-level encryption natively. TLS protects data in transit between clients, brokers and tools, and it does nothing for the bytes on disk. The encryption has to happen before the record reaches the producer's send call, in the application or in a serializer layer that wraps the value with a data encryption key tied to one data subject. Each subject's key is held in a key store that is separate from the cluster, so a broker operator, a compromised topic reader or a retained backup cannot recover the plaintext once that key is gone. The design team should decide early which fields are encrypted, which key identifier travels with each record (for example, in a header or inside the envelope), and how consumers that replay old offsets will handle a missing key.
The approach connects to the GDPR through pseudonymisation. Article 4(5) describes pseudonymised data as data that cannot be attributed to a person without additional information, which must be kept separately and protected by technical and organisational measures. Article 32(1)(a) lists pseudonymisation and encryption among appropriate security measures. A key store that sits in the same cluster, the same namespace or the same backup set as the ciphertext does not meet the separation test, so the key store needs its own access model, its own audit trail and its own backup and destruction procedure. Whether the ciphertext remains personal data is a decision the controller must document, and it depends on who can realistically reverse it. Read EDPB Guidelines 01/2025 on Pseudonymisation, which is a version for public consultation, before relying on this approach, and confirm with counsel whether the guidance has changed by the time you build.
Downstream copies made by connectors and consumers
Erasing a record from a Kafka topic does nothing to the copies that consumers and connectors have already written elsewhere. Each downstream system holds its own data, with its own retention and deletion behavior, so each needs an explicit erasure path. The cards below cover the copy points most often found in a Kafka estate.
Kafka Connect sink connectors
Sink connectors write topic records into databases, search indexes and other stores. Their upserts and deletes only reach the target if the connector maps tombstones or delete events to a delete in that store. Confirm that each connector turns a null-payload record into a delete, and check whether it handles deletes at all. Test the path with a known subject, then verify the row or document is gone from the target, including any replicas and snapshots.
Data lake files
Sink connectors that write Parquet, Avro or JSON files to object storage create immutable files that contain personal data. Kafka retention does not rewrite them. Plan a rewrite or compaction job that removes records for an erased subject, and confirm that old table snapshots, versions and manifests stop exposing those records. Ask the connector and table format documentation how files are rewritten and whether deletes are supported in place.
Consumer application caches
Consumers often keep personal data in local state, in-memory caches, Kafka Streams state stores or key-value caches. Kafka does not notify these stores when a tombstone arrives. Each consuming team should handle tombstones or erasure events in its own code, and should clear state on a schedule that matches the erasure process. Confirm that restarts and changelog restores do not bring erased values back.
MirrorMaker 2 replicas
MirrorMaker 2 copies topics, configuration and offsets to other clusters. A deletion or tombstone on the source reaches the target only as a normal replicated record, so the target keeps the original data until its own retention or compaction removes it. Confirm that retention and compaction settings match on both sides, and that the erasure procedure runs in every cluster that holds a replica. The same applies to disaster recovery clusters that the platform team keeps current.
Backups and logs
Broker backups, volume snapshots and application logs can contain personal data long after the topic has expired. Kafka does not track which backups hold a given subject. Set a backup retention period that is documented against the erasure procedure, and confirm how restores handle subjects erased after the backup was taken. Review application and connector logs for payloads or keys, and reduce what they record.
Configuration sequence for erasure controls
The sequence turns an erasure requirement into concrete Kafka settings, key handling, connector mappings and a runbook that proves each step worked. It starts with data inventory because retention, compaction and key choices all depend on which topics hold personal data and why.
Inventory personal data per topic
The data protection lead and the platform owner list every topic, its producers, its consumers and the personal data fields in each message. For each topic, record the purpose, the lawful basis as documented by the controller, the data subjects involved and whether the topic is also a source for Connect sinks or stream processors. Done means every topic has a row in the register, and each row names an owner who has confirmed the purpose and retention need in writing. Topics with no personal data are marked so the later steps do not touch them.
Set retention and cleanup policy per topic
The platform team sets cleanup.policy per topic from the inventory: time-based retention with delete for event streams that only need to live for a defined operational window, and compact for topics that hold the latest state per key. Retention values are taken from the documented purpose, not from the cluster default of 7 days, and the decision is recorded with the reason. Done means the topic configuration export shows the chosen cleanup policy and retention.ms for every personal-data topic, and the register links each setting to its justification.
Define tombstone and delete.retention.ms behavior
For compacted topics, the platform team defines how erasure is expressed as a null-payload record for the affected key and confirms how long tombstones remain. Kafka removes tombstones after delete.retention.ms, which defaults to 24 hours, so the team decides whether that default fits the operational process or needs a documented change. Done means a test topic shows that a tombstone removes earlier values for its key after compaction, and the runbook states that compaction timing is not an immediate guarantee.
Design per-subject keys and a separate key store
The security architect defines a key hierarchy in which each data subject, or each subject group where that is justified, has a data encryption key used to encrypt personal fields before they reach Kafka. The keys are held in a key management service that is separate from the Kafka brokers and from the topic data, with access limited to the serializer identity. Deleting a subject's key is the erasure action for ciphertext already stored in logs. Done means the key design document is approved, key deletion is tested on a sample subject, and the team confirms that backups and key service snapshots are covered by the same rule.
Configure serializers for field-level encryption
Developers add a serializer layer that encrypts the designated personal fields with the subject's key and writes a key reference in the message header or payload, while non-personal fields stay in clear for routing and analytics. Deserializers fetch keys through the key service and fail closed when a key has been deleted. Done means unit and integration tests show that a message encrypted for a subject cannot be read after the key is removed, and that messages for other subjects remain readable with their own keys.
Map connector and downstream copies
The data flow owner lists every Kafka Connect sink, stream processor output and replica that copies personal data out of Kafka, including the target system, its retention and its access path. MirrorMaker 2 replicas are included because it replicates topic data and configuration to other clusters. Each copy is assigned an erasure method, such as a delete or overwrite in the target, a key deletion where the stored value is encrypted, or a documented exception. Done means every copy has a mapping row and the owner of each target system has confirmed how erasure reaches it.
Lock ACLs and TLS
The platform team enables TLS on every listener that carries personal data and restricts topic operations, including delete and alter configuration, to named service principals using the StandardAuthorizer in KRaft mode. Because resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true, the team confirms that setting is not enabled. Done means an access test shows that unauthorized principals cannot read, write, alter or delete the personal-data topics, and the ACL export is stored as evidence.
Write the retention and erasure runbook with tested evidence capture
The operations team writes the runbook that covers how an erasure request is received, validated against the Article 17 grounds, executed through tombstones, key deletion and connector actions, and recorded. The runbook defines the evidence each action produces, such as the request ID, configuration exports, tombstone or key deletion logs, and downstream confirmations. The team runs a rehearsal on a non-production cluster and keeps the captured evidence with the register. Done means the rehearsal produces a complete evidence pack that a reviewer can follow from request to closed action, and any gaps found are assigned to an owner.
Questions buyers ask before hiring help with GDPR erasure on Kafka
Can Kafka delete a single record on request?
Not in place. Kafka is an append-only log, and the broker does not offer a supported operation that removes one arbitrary message on demand. What Kafka does provide is retention: time-based and size-based cleanup removes whole log segments once they age out or exceed a limit. For keyed data in a compacted topic, a null-payload record (a tombstone) removes earlier values for that key, but the tombstone itself stays until delete.retention.ms passes, and older segments must be cleaned before the data is gone. Records that must disappear sooner usually need a design choice made before the data is written, such as short retention for that topic or per-subject encryption keys. The Kafka design documentation and the topic configs reference describe the cleanup behavior you will need to test on your own cluster.
What drives the cost and effort of crypto-shredding?
Crypto-shredding means encrypting each data subject's fields with a key held separately, then destroying that key to make the data unreadable. The main cost drivers are the design of the key hierarchy, the key management service you run or buy, how producers and consumers fetch keys safely, and how many topics and connectors carry the affected fields. Effort rises when personal data is spread across many topics, when schemas are not versioned, or when downstream copies must also be found and shredded. Much of the work is one-time (design, producer and consumer changes, key lifecycle policy), while key storage, rotation and audit logging recur. Ask any provider for a written breakdown of these components, which systems they will change, and who owns key destruction evidence. Pseudonymisation guidance from the EDPB explains why the separately kept key must be protected as well.
How long does retention design take for our topics and connectors?
The duration depends mainly on how many topics hold personal data, how many connectors copy them, and how many systems outside Kafka keep the same records. A small set of topics with one sink connector is far quicker to review than dozens of topics feeding warehouses, search indexes and object storage. Each topic needs a documented purpose, a retention rule, a decision on compaction, and a check that each downstream copy follows the same rule. Legal and data owners must sign off on those decisions before engineering changes start. DeployOpen plans this work in a written scope with agreed deliverables and timelines, so ask for a quote based on a topic inventory from your own environment. No general duration applies, and any figure quoted without that inventory should be questioned.
Can our own team handle GDPR erasure on Kafka without a partner?
Many platform teams can do the Kafka settings themselves, such as retention, compaction and topic configuration, because these are documented and testable. The harder parts are usually cross-team: agreeing the legal basis for each data category, mapping every connector and downstream copy, designing per-subject keys, and proving deletion works after a restart or a replay. Ask whether your team has capacity to run those tests and keep them working through upgrades. A partner is useful when you need an outside plan, a tested build, or documented evidence in a fixed scope. Deciding between the two is a question about capacity and skills in your own organization, and the answer should come from an honest review of your current topic and connector estate.
What does the work need from legal and data owners?
Legal and data owners set the rules that engineering cannot decide alone. They need to confirm which erasure grounds under Article 17(1) apply to each dataset, how long each category may be kept under Article 5(1)(e), and whether any retention exception applies. They also decide whether a topic's purpose justifies keeping data after the original processing ends, and whether notification duties under Article 17(2) reach any downstream recipient. Data owners must name the systems that consume each topic and approve changes to schemas and retention. Ask your legal team to confirm these points in writing before the engineering scope is fixed, since a provider cannot give legal advice on your behalf.
Can DeployOpen certify that our Kafka setup is GDPR compliant?
No. DeployOpen does not issue audit reports, attestations or certifications, and the GDPR itself requires no audit or certificate. Certification under Article 42 is voluntary and is issued by accredited certification bodies or supervisory authorities, and it does not reduce your responsibility for compliance. DeployOpen can plan, build and document the technical controls, prepare evidence for a compliance assessment, and run security testing of the erasure paths. Independent auditors and certification bodies make any formal judgment. Treat any provider who promises a certificate as a warning sign.
How does the Kafka version affect the erasure controls?
The version determines which settings and tooling you can rely on. The supported Kafka releases on the downloads page are 4.3.1, 4.2.2 and 4.1.2, and older releases are archived, so an erasure design should target one of these. Kafka 4.0 and later run only in KRaft mode, and clusters still on ZooKeeper must migrate before upgrading to 4.x. Rolling upgrades are supported from each release in the past year to the latest version, according to the release plan. Test your retention, compaction and tombstone behavior on the target version, because cleanup behavior and client requirements can change between releases. Also confirm the Java runtime your brokers, Connect workers and clients need.
How DeployOpen scopes and delivers erasure controls on Kafka
DeployOpen plans and implements the erasure controls around a self-hosted Kafka cluster in infrastructure you control. The work covers retention settings, compacted topics with tombstones, per-subject key handling and the connector copies that sit downstream, documented in a written scope before build starts.
How we scope
- Inventory which topics carry personal data and which subjects, purposes and retention periods each one serves
- Map every connector, sink and downstream copy that receives records from those topics
- Agree in writing which erasure paths are in scope, including the deliverables and timelines
How we build
- Set topic retention and compaction settings to match the documented purposes, then test expiry behavior on a non-production cluster
- Implement tombstone production, per-subject key management and key deletion where crypto-shredding is chosen
- Configure connector and sink deletion or expiry jobs and test them against sample erasure requests
How we hand over
- Document each erasure path, its owner, its trigger and the evidence it produces for your assessors
- Deliver runbooks for erasure requests, key deletion, backup handling and exception review
- Walk your engineering, privacy and compliance teams through the controls and the open questions that remain
How we support
- Engagements are fixed-scope, and ongoing support, upgrades and continuous testing are available as a retainer
- Revisit retention and erasure controls when Kafka releases change behavior or when new topics or connectors are added
- Confirm with your legal and privacy advisers how the obligations apply to your organization, since DeployOpen does not provide legal advice
