Encryption at rest for Apache Kafka data: what Kafka leaves to you
A regulated team storing event data on self-hosted Kafka needs a clear answer for the data sitting on disk. Kafka does not encrypt topic data or KRaft metadata itself, so protection at rest comes from disk or volume encryption, encryption of tiered storage buckets, application-level payload encryption, or a combination of these. TLS does not close this gap. It protects traffic in motion between clients, brokers and tools, and it does nothing for the log segments and metadata files that remain on a broker's disk after a message has been written.
The Kafka KRaft documentation states that sensitive data is stored in metadata records and is not encrypted by Kafka. That statement is the only basis for the claim about metadata, so treat it as the starting point for your data inventory. Ask which topics and configuration values carry personal, financial or health data, and which of those values reach the metadata log, such as SCRAM credentials and ACL definitions. Confirm the current wording against the Kafka KRaft vs ZooKeeper documentation for the release you run, since the storage model matters for any backup or snapshot copy you make.
Once the inventory is clear, decide which layer owns each risk. Full-disk or volume encryption protects the broker data directories and the copies taken in backups, but it does not stop a user with broker access from reading plaintext records while the volume is mounted. Payload encryption protects specific fields no matter where they land, at the cost of key management, schema changes and the loss of broker-side inspection of those fields. Ask your platform team to confirm how key custody, rotation and recovery work for each layer before the cluster stores regulated data.
Which encryption layer covers which data
Disk and volume encryption protects the broker's log directories and metadata files when the underlying media is removed, copied or stolen while powered off. It is usually provided by the operating system, the hypervisor, the storage array or the cloud volume service, so Kafka itself does not manage keys for it. The layer is transparent to producers and consumers, and it does not change how records are stored or read.
- What it protects: segment files, indexes and metadata logs at rest on the broker host, including data copied off a decommissioned disk.
- What it does not protect: data read by a running broker, which sees plaintext once the volume is mounted, or by anyone holding an authorized client connection.
- Who operates it: the platform or infrastructure team, through the storage or hypervisor key management service.
- Key handling: confirm with your storage vendor where keys live, how they rotate, and whether a node reboot requires an unlock step that your automation must support.
- Recovery: check that backups of the volumes or snapshots are encrypted under keys you can still reach after a disaster.
Before you rely on this layer, confirm the exact encryption and key management features of your storage platform with its vendor, and test a restore from an encrypted volume.
When tiered storage moves older log segments to object storage, the bucket becomes a second copy of your data at rest. Disk encryption on the brokers does not cover that copy, so the bucket needs its own encryption controls. Ownership usually sits with the team that runs the object store, and the access policy for the bucket must be reviewed separately from Kafka ACLs.
- What it protects: segment data uploaded to the object store, from unauthorized reads of the stored objects if the encryption and access policy are configured correctly.
- What it does not protect: objects readable by any credential with bucket access, including broker service accounts and administrators of the object store.
- Who operates it: the storage team, with the broker team owning the bucket policy and the service identities Kafka uses.
- Key handling: confirm with your object store vendor whether server-side encryption is supported, who holds the keys, and how key rotation affects existing objects.
- Backups and replicas: check that any replication or lifecycle copy of the bucket inherits the same protection.
Confirm the tiered storage plugin and object store combination you plan to use, and verify encryption on sample objects after a test upload.
In KRaft mode, Kafka keeps cluster state in a metadata log that the controllers replicate. The KRaft vs ZooKeeper documentation states that sensitive data is stored in metadata records and is not encrypted by Kafka. This includes data such as SCRAM credentials and ACL definitions, so disk encryption on the controller and broker hosts is the layer that matters for this data. Treat metadata as sensitive even though it is small.
- What it protects: the metadata log on controller and combined nodes, when the host volumes are encrypted and access to the hosts is restricted.
- What it does not protect: metadata content from anyone who can read the log through an administrative path, including operators with host access and holders of super user accounts.
- Who operates it: the platform team for host encryption, and the Kafka administrators for who can change credentials and ACLs.
- Credential handling: avoid placing credentials in plain configuration files or tickets, and plan rotation as a metadata change.
- Disposal: plan how metadata directories are wiped when controllers are retired or re-formatted.
Review the KRaft vs ZooKeeper documentation before your design review, and confirm with your team which metadata items your policy classifies as sensitive.
Application-level encryption encrypts message values or selected fields before the producer sends them. Kafka then stores ciphertext, and the broker never sees the plaintext payload. This is the only layer in this set that protects record contents from people who can read the broker's disks, the object store and the broker's logs, but it adds key management and schema work to every producing and consuming application.
- What it protects: record contents from readers of the broker, the log segments and the tiered storage bucket, when keys are held outside the cluster.
- What it does not protect: keys, topic names, record keys, headers, timestamps and message sizes, which remain visible unless you encrypt them too.
- Who operates it: application teams own encryption and decryption code, and a security or platform team owns the key service and key policy.
- Consumers: every consumer that needs plaintext must be authorized for the key, so access control moves partly from Kafka ACLs to the key service.
- Operational cost: plan for key rotation, replay of old ciphertext, and schema changes that affect encrypted fields.
Confirm the key management service, envelope encryption approach and library choices with your engineering and security teams before implementation starts.
Why sensitive data in KRaft metadata needs its own plan
In KRaft mode, Kafka keeps sensitive data in metadata records and does not encrypt that data itself, so the disks holding the metadata log need their own protection plan. The metadata log is replicated across the controllers and written to the disks of the brokers and controllers that run it. Three kinds of sensitive content live there: the ACLs enforced by the StandardAuthorizer, the SCRAM credentials for SCRAM-SHA-256 and SCRAM-SHA-512 users, and any other configuration stored as metadata. Anyone who can read those files on a single node can read the material that governs access to the whole cluster, so disk encryption on its own does not solve the problem.
Start with access. Restrict operating system read access to the metadata directories to the Kafka service account and to the administrators who must maintain the node, and make sure backups, snapshots and log shipping do not copy those directories to wider audiences. Encrypt the volumes that hold the metadata directories with the same key management standard you apply to other sensitive stores, and confirm who can unlock them at boot. Keep secrets out of server properties files, startup scripts, container environment variables that others can inspect, and topic names, since topic names and configuration values are visible to anyone with describe rights. Where a secret must be supplied at runtime, use a secrets manager outside Kafka.
Before you write the plan, confirm with your Kafka version's documentation and your own test cluster which metadata record types it writes, how credentials are stored in them, and whether any of them are hashed rather than stored in a recoverable form. Check that the answers hold after each upgrade, and document the result as part of your encryption at rest evidence. The KRaft vs ZooKeeper documentation is the starting point for this check.
Trade-offs when choosing each encryption approach
Each encryption layer protects a different part of the data path, and none of them covers the others. The cards below set out what each approach hides, what it leaves visible, and what it costs to run, so you can match the layers to the threats and data classes you have.
Volume encryption stays transparent
Disk or volume encryption works underneath the broker, so Kafka reads and writes data as usual. It protects log segments if a disk is lost, stolen or decommissioned without wiping. It does not hide records from the running broker or from anyone who can log in to it, because the broker process sees decrypted data. Check who holds the unlock keys and whether they are released at boot or by an external key service.
Bucket encryption covers tiered copies
When tiered storage moves older segments to object storage, those copies sit outside the broker disks and need their own protection. Bucket-level encryption and access policies must be set on the storage backend, and the broker's access credentials become a high-value secret. Confirm which tiered storage backend and version you run, because object storage features and key options differ between vendors.
Payload encryption protects content end to end
Application-level encryption encrypts the message value before the producer sends it, so brokers, disks and backups only hold ciphertext. The cost is that broker-side features reading payloads stop working on encrypted fields. Compaction keyed on a plain key still works, but routing or filtering on encrypted content does not. Decide which fields are encrypted and keep keys and identifiers readable where the platform needs them.
Key management and rotation burden
Every approach creates keys to store, rotate and retire. Volume and bucket keys can usually sit in a cloud KMS or a hardware-backed store, while payload keys must be distributed to producers and consumers, which adds an application dependency. Rotation of payload keys needs a plan for messages already written under the old key. Ask who owns each key, how rotation is tested, and what audit trail is kept.
Performance overhead to measure
Encryption adds CPU work on the path it touches. Volume encryption cost depends on the disk and crypto hardware, payload encryption adds work in every producer and consumer, and bucket encryption adds latency to tiered reads. Measure throughput and consumer latency on production-like hardware with your message sizes and compression settings before choosing a layer. Do not rely on generic overhead figures, because results vary with the workload.
Recovery when keys are lost
Losing a volume key, bucket key or payload key can make data permanently unreadable, and that is often a worse failure than a breach. Document key backup and escrow, test restoring a broker from encrypted disks with the keys held elsewhere, and confirm that backups include the key material needed to read them. Also check that KRaft metadata holding sensitive values is covered by the same recovery plan.
How to plan and set up encryption at rest for Kafka
This sequence moves from identifying where sensitive data lands on disk to a verified state in which every store, backup and mirrored cluster carries an agreed encryption control and its keys can be rotated and restored.
Classify topics and metadata holding sensitive data
The security owner and the data owners list every topic, consumer offset store and configuration value that carries personal, financial or regulated data, and mark the data classes that must be protected at rest. In KRaft mode, sensitive values can also sit in metadata records, such as SCRAM credentials and topic configuration, which Kafka does not encrypt, so the inventory must include the controller metadata log as well as topic logs. Done means a written register that names each store, its classification and the owner who signed off, reviewed before any broker is changed.
Choose the encryption layer for each data store
The platform team matches each store to a layer: disk or volume encryption for broker and controller data directories, encryption managed by the bucket or object store for any tiered storage, and application-level payload encryption for fields that must stay protected even from operators with disk access. The decision record should state what each layer does not cover, for example that volume encryption does not hide plaintext from a process with broker access. Done means a signed decision table that the architecture review accepts.
Set up key management and key ownership
The security team selects the key management system, assigns who can create, rotate and disable keys, and separates key administrators from Kafka administrators. The team confirms with the storage and cloud vendors which key options each layer supports, since the supported key types and rotation features differ by product. Done means a documented key hierarchy, named key custodians, and a tested rotation on a non-production key, with the recovery path for lost or disabled keys written down.
Encrypt broker and controller volumes
The infrastructure team enables encryption on the volumes or file systems that hold broker log directories and controller metadata directories, using the storage platform's own mechanism. This is done before cluster formation where possible, and for running clusters it is done through a planned rolling replacement of brokers so that each node is rebuilt on an encrypted volume and replicas resync. Done means that each volume reports encryption enabled in the storage platform, and a broker restart on the encrypted volume succeeds with the expected log data present.
Configure bucket encryption for tiered storage
If tiered storage is used, the storage team enables default encryption on the object buckets that hold tiered segments, sets the key policy so only the Kafka broker service identity can use the key, and confirms that bucket versioning and lifecycle rules do not keep unencrypted copies. Ask the storage vendor to confirm how its server-side encryption handles existing objects, because objects written before the setting changed may need re-encryption. Done means a bucket configuration review showing default encryption on, a policy test that denies access without the key grant, and a sample segment confirmed as encrypted.
Add payload encryption where the data class requires it
For fields that must remain protected inside topics, the application teams encrypt values before producing them, using keys held outside the Kafka cluster, and define how consumers obtain and cache the keys. The team documents which fields are encrypted, how schema changes affect encrypted fields, and how downstream connectors handle the ciphertext. Done means a producer and consumer test where a raw dump of the topic log shows no plaintext for the protected fields, and a key-unavailable test shows that consumers fail closed.
Test rotation, restore and replicated copies
The operations team runs a key rotation on a test cluster and confirms that brokers and tiered storage continue to read existing data, then restores a backup into an isolated environment and checks that it can be decrypted with the documented keys. The same checks run on any mirrored cluster built with MirrorMaker 2, which replicates topic configuration, consumer offsets and ACLs but not the storage layer's encryption settings, so each target cluster needs its own volume and bucket controls. Done means dated test records for rotation, restore and each replica, signed off by the security owner.
Verify controls and record the evidence
The security team runs a final verification against the register from the first step: it confirms volume and bucket encryption on every store, checks that no sensitive plaintext appears in topic logs or the metadata log for protected fields, and reviews key access logs. The results are stored as evidence with the configuration baselines, so that the next review, audit preparation or upgrade can compare the current state against what was approved. Done means a signed verification report and a scheduled review date set by the owner.
Questions teams ask before encrypting Kafka data at rest
Does Apache Kafka encrypt data on disk by itself?
No. Kafka encrypts traffic between clients, brokers and tools with SSL/TLS, but the project does not encrypt the log segments it writes to disk. In KRaft mode, sensitive values are also stored in metadata records without Kafka-level encryption. Encryption at rest therefore has to come from the layer below Kafka (disk or volume encryption, encrypted object storage for tiered data) or from encrypting payloads before they reach Kafka. Confirm the exact behavior of your Kafka version in the KRaft vs ZooKeeper documentation before you rely on it in an assessment.
What does disk encryption cost, and who pays for it?
No public figure is supplied for this work, so treat any number you see as a quote to verify. Costs fall into four groups: the licensed encryption features of your storage platform or cloud provider, hardware or key management services you run, the engineering time to design and test the change, and ongoing operations such as key rotation and restore drills. Most of the software cost is open source, since Kafka is released under the Apache License 2.0. Ask a partner for a quote that separates one-time build work from recurring operations and names every paid component.
How long does encryption at rest take, and what drives the duration?
Duration depends on the storage layer you already run and how many clusters need the change. Work usually moves through an inventory of data stores and retention periods, a design for keys and access, a test on a non-production cluster, and a staged rollout with restore tests. The biggest drivers are the number of brokers and volumes, whether you need to re-encrypt existing data, the state of your key management system, and how many compliance evidence items must be produced. Ask for a written schedule tied to a fixed scope, since no duration is supplied for this topic.
Can an internal platform team encrypt Kafka data alone?
Often yes, if the team already runs the storage platform and a key management service and has tested restores. Disk or volume encryption is a configuration change at the infrastructure layer, but it must be validated against broker recovery, expansion and backup procedures. Payload encryption is harder because it changes producers, consumers and Connect connectors. A useful check is whether your team has owned a Kafka upgrade or a TLS rollout before. If not, a fixed-scope engagement can cover design, testing and handover while your engineers keep ownership.
How does application-level payload encryption affect consumers and Kafka Connect?
Payload encryption protects message contents even where disks, backups or operators can read the log. Its cost is paid by every application that produces or consumes the topic, because each must hold the decryption path and the right key. Connect connectors that transform, sink or index those messages must either decrypt in a trusted step or pass the ciphertext through unchanged, which limits what downstream search and analytics can do. Key lookup failures then appear as consumer errors. Plan which fields are encrypted, how schemas describe them, and how consumers handle a missing or retired key before you start.
How does key rotation work for encrypted Kafka data?
Rotation depends on which layer holds the keys. For disk or volume encryption, the storage or key management platform usually rotates a key-encryption key without rewriting data, but you must confirm that behavior with your vendor and test a restore afterward. For payload encryption, new messages can use a new key version while consumers keep access to older versions until retention expires or data is re-encrypted. Sensitive values in KRaft metadata need the same planning. Ask a partner to document rotation steps, owners, rotation frequency set by your policy, and the evidence each rotation produces.
What should we ask for in a quote for this work?
Ask for the written scope first: which clusters, topics, tiered storage buckets and client applications are covered, and what is excluded. Then request line items for discovery, key and access design, test environments, rollout by cluster, restore and rotation drills, documentation, and handover. Each item should say who performs the work and who owns the result after handover. Ask which licenses, cloud services or hardware you must buy separately, which costs recur, and how changes in scope are priced. Confirm how the partner tests on your infrastructure and what evidence you receive for your own compliance assessment.
Security and operational considerations
Encryption at rest only protects data if the keys, the backups, the replicas and the metadata around the topics are handled with the same care. The cards below cover the places where protection most often breaks down after the volumes are encrypted, and what to check in your own environment before go-live.
Key custody and separation of duties
Decide who can create, rotate and revoke the keys that unlock disks, volumes and tiered storage buckets, and keep that group separate from the Kafka platform operators who run the brokers. If one team can both read the key and start brokers, disk encryption adds little against an insider. Confirm where keys are stored, how they are escrowed for recovery, and how a key rotation is tested against a restored volume before it is needed in an incident.
Backup and mirror copies
Every backup, snapshot and cross-site copy of a Kafka log directory holds the same records as the original, so each one needs its own encryption at rest with keys that are managed as carefully as the primary. MirrorMaker 2 replicates topics, consumer group offsets and ACLs between clusters, so a replica cluster is a full copy of your access model as well as your data. Check that the target site encrypts its storage to an equivalent standard before mirroring starts.
Access to running brokers versus stored bytes
Encrypted disks protect against someone who removes a drive or copies a volume, but a process that is running on the broker host sees data in plain text once the volume is mounted. Treat shell access to broker hosts, the Kafka process user and any debugging or log-dump tooling as the real access boundary. Restrict who can read log segments from a mounted directory, and confirm that dump and inspection tools are run only under audited sessions.
Logging and monitoring of key use
Keys used by disk encryption, cloud key services or tiered storage should produce an audit trail of every unwrap, rotation and access denial. Feed those events into your central log platform and alert on unexpected key use, such as a key accessed from a host that is not a broker or at a time outside maintenance windows. Confirm the retention period of these logs matches your other security evidence, because a key audit trail that is rotated away cannot answer later questions.
KRaft metadata and ACL data
Kafka does not encrypt the sensitive data stored in KRaft metadata records, and the StandardAuthorizer keeps ACLs in that same metadata log. Stored SCRAM credentials and other configuration records therefore sit in the metadata log on disk, not only in topic data. Put the controller and broker metadata directories on encrypted storage, restrict read access to them, and include them in backup and restore tests so that the access model is protected wherever the topic data is.
How DeployOpen scopes and delivers Kafka encryption at rest
Encryption at rest for Kafka is mostly a set of decisions about storage, metadata and payloads, so the work starts with a written scope rather than a tool choice. The cards below follow the path from classifying what sits on disk to handing over evidence that internal and external assessors can review. Each step runs inside the client's own infrastructure.
Written scope before work starts
Engagements are fixed-scope, so the systems, brokers, storage backends, buckets and environments in play are listed in writing before any change is made. The scope states which disks, volumes and tiered storage targets are covered, which KRaft metadata exposure is accepted or mitigated, and where payload encryption applies. Ask for the scope document to name every item the work will and will not touch.
Data classification and storage review
Before choosing controls, the team maps which topics and records carry sensitive fields, then traces where those bytes land: log segments, remote tiered storage, snapshots, backups and KRaft metadata records. Kafka does not encrypt stored data itself, and sensitive values can appear in metadata, so this review decides which layer must protect each location. Ask for the classification output to be reviewed by the data owners.
Disk and volume encryption design
Volume-level encryption protects broker data directories against loss of physical media, and the design covers key custody, unlock at boot, and how keys are rotated without a full cluster rebuild. Brokers must still start unattended during rolling restarts, so the unlock path is tested against the platform's maintenance process. Ask how keys are escrowed and who can release them.
Tiered storage bucket protection
Where tiered storage sends segments to object storage, the bucket needs its own encryption, access policy and key ownership, separate from the broker volumes. DeployOpen reviews bucket policies, lifecycle rules and replication targets so that copies made for disaster recovery follow the same rules. Confirm with your storage vendor which server-side encryption modes and key managers it supports.
Key management and rotation
Encryption keys need a home that the platform team does not fully control on its own, such as a managed key service or an internal key manager in your infrastructure. The work defines key hierarchy, rotation cadence, revocation steps and the recovery path if a key is lost. Ask which systems hold key material and whether each one is covered by the same access review.
Verification evidence for assessors
Each control is checked on the running cluster, and the results are packaged as configuration exports, test records and runbooks that internal reviewers and external auditors can follow. DeployOpen prepares this evidence but does not issue audit reports, attestations or certifications, which remain with independent auditors and certification bodies. Ask for the evidence list in the scope so the expected artifacts are agreed up front.
Optional retainers for ongoing support
Storage layers, key rotations and Kafka upgrades change what is encrypted and how, so controls drift over time. An optional retainer covers periodic re-review of the classification, key rotation support and checks after each upgrade step. Ask what the retainer covers, how changes are requested, and how it connects to the original scope.
