When does Apache Kafka tiered storage cut broker disk cost?
Tiered storage cuts broker disk cost when most of your retained data is older than the records consumers actually read. Without it, a team that keeps weeks of data on brokers buys disk for every broker and every replica, even though recent records receive nearly all the reads. With tiered storage, Kafka keeps the recent log segments on the local tier, which is the broker disk, and copies older closed segments to the remote tier, an object store reached through a RemoteStorageManager plugin. The disk you size is then the hot window, not the full retention period, and the remote tier holds the long tail.
The workloads that gain most are those with long retention driven by occasional reads: audit replay, backfills, reprocessing after a bug fix and recovery from a downstream failure. Topics with high write volume are strong candidates when disk is the binding constraint on the cluster. Workloads gain less when retention is short enough to fit on local disk, or when consumers constantly read old offsets with tight latency targets, because reads from the remote tier are slower than reads from local disk. Measure read latency for historical consumers in your own environment before you rely on the remote tier for them.
Savings depend on your retention per topic, your replication factor, how much data stays in the local window, the object storage price in your region and the request and transfer charges your provider applies to remote reads. Model each candidate topic with its local and remote windows against your current broker disk spend, and include object storage costs for both capacity and access. Confirm the feature's status and configuration in the Kafka 4.3 documentation and in the supported release list for the version you plan to run, since availability and limits can differ by release.
Which tiered storage settings to decide before enabling
This table lists the settings a team must decide before turning on tiered storage, what each one controls, and what it trades off. Read it as a checklist: settle each row with your platform and compliance owners, then confirm the exact configuration keys and defaults in the Kafka 4.3 documentation and the apache/kafka repository before you change a production cluster.
| Setting | What it controls | Trade-off | Choose it when |
|---|---|---|---|
| Enablement scope | Whether tiered storage applies to selected topics or the whole cluster | Per-topic rollout limits risk; cluster-wide enablement is simpler but moves every eligible topic at once | Only some topics hold long history, so start with those and widen later |
| Local retention | How long segments stay on broker disk before they become eligible for remote storage | Longer local retention keeps recent reads fast but uses more broker disk | Consumers regularly read back through a window you must serve from local disk |
| Total retention | How long data is kept across local and remote tiers combined | Drives object storage growth and cost; too short a value deletes data consumers or auditors still need | Retention is set by business, legal or compliance requirements, not by disk size |
| RemoteStorageManager plugin | The component that copies segments to and reads them from object storage | Plugin choice sets supported stores, operational burden and the support path for upgrades | You already operate an object store the plugin supports and your team can maintain it |
| Plugin configuration and credentials | Endpoint, bucket, access credentials and encryption settings passed to the plugin | Errors block uploads or reads; credentials add a secrets rotation duty | Secret storage, rotation and bucket policy are agreed before the first copy runs |
| Remote log manager schedule | How often the broker runs copy, expiry and cleanup work for remote segments | Frequent runs move data sooner but add broker load; infrequent runs leave local disk fuller | You have measured copy backlog on a test cluster and know the disk headroom you need |
| Historic read latency | Fetch speed for offsets that now live only in the remote tier | Object storage lowers cost but makes replays and backfills slower than local reads | Replays and backfills are infrequent, or consumers can tolerate slower catch-up |
| Compacted topics | Whether a topic with log compaction can use tiered storage | Compacted topics are not supported, so they must stay on local storage | Check each topic's cleanup policy before enabling; confirm the current limitation in the 4.3 documentation |
How to evaluate a RemoteStorageManager plugin
Tiered storage hands segment upload, fetch and deletion to a RemoteStorageManager plugin, so the plugin sits on the data path for every tiered partition. The six criteria below cover support status, storage targets, credentials, failure handling, observability and licensing, and each one should be checked per plugin because the Apache Kafka project itself does not sell a paid edition.
Support for your Kafka release
Confirm that the plugin is maintained and explicitly supports the Kafka version you run or plan to run, including the 4.x line you are upgrading to. A plugin built against an older storage interface can break during a rolling upgrade, so check its release notes against the Kafka release plan before you commit. Ask the vendor or maintainers who fixes a defect found in production and how quickly security patches are shipped. Treat an unmaintained plugin as a blocker, not a risk to accept.
Object storage targets
List the object stores the plugin supports and compare them with where your data will actually live. If you run self-hosted S3-compatible storage such as Ceph RGW, confirm the plugin's request signing, path style addressing, and multipart upload behavior against that endpoint rather than assuming compatibility. Also check whether the plugin needs bucket versioning or lifecycle features that your object store must provide.
Credentials and encryption handling
Find out where the plugin reads access keys, whether it supports short-lived credentials or instance roles, and whether secrets appear in broker configuration files or logs. Confirm which encryption options exist: server-side encryption at the object store, client-side encryption by the plugin, and key rotation without a broker restart. Tiered segments hold the same records as local logs, so the encryption and key management decisions must match your data classification policy.
Failure and retry behavior
Ask how the plugin behaves when the object store is slow, returns errors, or is unreachable. Check the retry count, backoff, and whether failed uploads leave orphaned objects that you must clean up. Confirm how fetches for old offsets behave under load and whether the plugin signals a clear error to consumers. Test these cases in a non-production cluster by injecting faults, since the answers decide your recovery runbook.
Metrics and observability
Check which metrics the plugin exposes for upload and fetch latency, error counts, bytes transferred, and remote storage lag, and whether they are published through the same JMX or metrics pipeline you already monitor. Without these numbers you cannot tell whether a slow consumer is caused by the local tier, the object store, or the plugin. Define alert thresholds with your own baseline during the pilot.
Licensing and vendor support
Apache Kafka is licensed under the Apache License 2.0, but each RemoteStorageManager plugin carries its own license and support terms. Read the license for compatibility with your redistribution and internal-use rules, and ask whether support is community-only, commercial, or bundled with another product. Plugin costs, if any, come from the plugin owner, not from the Kafka project, so request those terms in writing with your quote.
Where tiered storage does not fit: compacted topics and reads of old data
Tiered storage suits topics that keep records under a time or size based delete policy. Compacted topics are a different case: they keep the latest value for each key rather than aging records out, and the Kafka 4.3 documentation has to be checked for its current statement on compacted topics before you write any topic plan. Confirm that statement in the Apache Kafka 4.3 upgrade notes and the tiered storage section of the operations documentation, then split your topics into two groups: compacted topics that stay on local disk with their own sizing, and delete-policy topics that are candidates for the remote tier. Do not assume a compacted topic will be offloaded because it is large.
Consumers that read far back in history are the second boundary. Records served from the remote tier arrive with higher latency than records on local segments, so a consumer group that starts from the earliest offset, replays a backlog after an outage, or runs a periodic full reprocess will feel the difference. Map which consumers read old data before you move any topic, and separate replay jobs from real-time consumers if their latency targets differ. Measure fetch latency for a replay on a test topic in your own environment, because the cost of a remote read depends on your object store, network path and fetch settings.
Workloads with short retention gain little from tiered storage. If a topic keeps a few hours or days of data, most reads already hit local segments, and the local tier still holds the active data, so the storage saving is small relative to the added components to run and monitor. Ask your team to compare the retention window with the amount of data that would actually leave the broker disks before committing. For each topic, record its cleanup policy, retention settings, read pattern and owner, which gives you the list that decides where tiered storage belongs.
How to enable tiered storage on a KRaft cluster and check it works
This sequence moves older log segments from broker disks to object storage on a KRaft cluster, with a check at each stage. It ends with a topic that keeps recent data local, serves older data from the remote tier, and has a tested route for turning the feature off.
Confirm a supported KRaft release
The platform team confirms that every broker runs a supported Apache Kafka release on KRaft. The downloads page lists 4.3.1, 4.2.2 and 4.1.2 as supported, and older releases are archived. ZooKeeper-based clusters must migrate to KRaft before upgrading to 4.x, and KRaft clusters older than 3.3.x should move to 3.9.x first. Run the metadata quorum status command against the controllers and check the broker versions in the cluster. Done means every broker reports a supported version and the quorum has a healthy leader.
Choose and test a RemoteStorageManager plugin
Tiered storage needs a RemoteStorageManager plugin that writes segments to your object store. Choose the plugin by confirming that it supports your object storage service, the Kafka version in use, its license, and its support model. Then deploy it to a non-production cluster and run upload, read, delete and broker restart tests against a test bucket. Include a broker failure during an upload. Done means the plugin copies segments, serves reads from the remote tier and recovers cleanly after a restart, and the test results are written down before production.
Create the bucket and scoped credentials
The storage or cloud team creates a dedicated bucket for Kafka remote segments, with encryption at rest and access logging enabled according to your policy. Create a service identity that can only read, write, list and delete objects under that bucket or prefix, and keep it separate from administrative accounts. Confirm object lifecycle rules do not delete or transition segments that Kafka still needs, and confirm the plugin's exact permission requirements with its documentation. Done means the identity writes and reads a test object in the bucket and is denied access to other buckets.
Configure brokers with the plugin settings
The platform team enables the system-wide tiered storage setting on the brokers and adds the plugin class and its connection settings, following the Kafka 4.3 documentation for the exact property names. Secrets go into your secret store, not into plain configuration files. Apply the change with a rolling restart, one broker at a time, and check that the cluster stays healthy between restarts. Done means each broker starts, logs that the remote storage manager initialized, and the controllers show no under-replicated partitions after the last restart.
Enable tiering on a test topic
The Kafka operator enables tiered storage on one test topic by setting the topic-level remote storage property to true. Set the local retention so that recent data stays on broker disk and older segments are eligible for upload, and keep the total retention at the value your data policy requires. Describe the topic afterward to confirm the settings took effect. Done means the topic configuration shows remote storage enabled with the intended local and total retention values.
Produce and consume across both tiers
The application team produces enough data to roll several segments, then waits for the broker to copy closed segments to the remote tier. After that, a test consumer reads from the earliest offset, which forces reads from remote storage for the older records, and then reads the most recent records from local disk. Compare record counts and checksums with the source data. Done means the consumer receives every expected record in order, with no gaps or duplicates across the tier boundary.
Verify remote copies and reads
The operations team confirms that segments are present in the bucket under the expected topic and partition paths and that local disk usage drops as segments move off the broker. Review the broker's remote log metrics and error logs for copy, read and delete failures, using the metric names in the 4.3 documentation. Confirm alerting covers remote copy lag and failed operations before you widen the rollout. Done means the bucket matches the topic's remote segments, the metrics show successful copies and reads, and no errors appear during a test period you define.
Disable tiering on a topic later
Before you turn tiered storage off for a topic, read the disable and data handling behavior in the Kafka 4.3 documentation, because it determines what happens to segments already in remote storage. Rehearse the change on a non-production topic, set the topic-level remote storage property to false, and confirm the topic configuration and the state of remote segments match the documented behavior. Keep a copy of any remote data you need before the production change. Done means the rehearsal shows the expected result for local and remote data, and the production change has a written rollback plan.
Securing remote segments and operating the remote tier
Remote segments hold the same records as local logs, so they need the same protection plus controls for the object store itself. The cards below cover credentials, encryption, bucket access, transport security, error monitoring and deletion. Confirm each control against the RemoteStorageManager plugin you choose, because Kafka's core documentation does not define how a given plugin handles them.
Object storage credentials and rotation
Brokers authenticate to the object store with credentials that the chosen plugin reads from its own configuration. Use a dedicated identity for the Kafka brokers, not a shared administrator key, and confirm how the plugin picks up rotated credentials without a broker restart. Ask the plugin maintainer whether it supports short-lived tokens or instance roles, and write down who rotates the secret and how often. Store the secret in your secrets manager rather than in broker property files.
Encryption of remote segments
Remote segments are copies of topic data, so they fall under the same data classification as the local logs. Confirm whether the object store encrypts objects at rest, who holds the keys, and whether the plugin can apply a customer-managed key. Check how the plugin handles key rotation for objects written before a rotation. Record the answers in your data inventory so auditors can trace where each copy of a topic lives.
Bucket access limited to broker identities
Grant read, write and delete rights on the tiered storage bucket only to the broker identities and, where operations need it, the administrative identity that runs cleanup. Deny public access at the bucket level and block consumers and other applications from reading the bucket directly, since they should read through brokers. Review the policy whenever a broker is added or removed so that retired identities do not keep access.
TLS and authentication to the object store
Traffic between brokers and the object store should use TLS with certificate validation enabled, and the plugin should not fall back to plaintext endpoints. Broker-to-broker and client-to-broker security is governed by Kafka's own listener configuration, which the security overview describes through SSL and SASL. Confirm which authentication methods your plugin supports for the object store connection and whether it validates custom certificate authorities.
Monitoring remote copy and read errors
Failed copies to remote storage and failed reads from it can leave segments stranded or slow consumers that read older offsets. Alert on remote copy failures, remote read errors and growth in the backlog of segments waiting to be copied, and confirm the metric names exposed by your Kafka version. Test the alert by making the object store unreachable in a non-production cluster and checking that the alert fires and that the broker recovers after access returns.
Retention and deletion for compliance
Remote retention determines how long copies of records persist after they leave the local disk, so set it deliberately for each topic that carries regulated data. Confirm whether deleting a topic removes its remote segments, and whether the object store has versioning or lifecycle rules that keep deleted objects. Document the deletion process and test it, because a deletion request that clears local logs but leaves remote copies may not meet your retention policy.
Limits to confirm in your environment before you commit
Before you commit to tiered storage, confirm that the code path you plan to run is covered by the release line you intend to support. The Kafka downloads page lists 4.3.1, 4.2.2 and 4.1.2 as the supported releases, and the project only issues bugfix releases for supported lines. Check the Apache Kafka downloads page for the current status of your version, then confirm in the Kafka 4.3 upgrade notes and the project's release plan which tiered storage behaviors changed between your current and target versions. Treat any feature still marked preview or early access in the documentation as a risk you must own, and ask the project or the community which open issues affect your version. Pay attention to the upgrade path: rolling upgrades are supported from the last three releases, so a cluster that is further behind needs an intermediate step.
The remote storage plugin is a separate decision from Kafka itself. Kafka is Apache-2.0 licensed with no paid edition from the project, but a RemoteStorageManager implementation comes from a plugin vendor or community maintainers with their own release cadence, support terms and security patching. Ask each candidate for its supported Kafka versions, its release history, its published support window, and who is accountable for defects in the interaction between the plugin and the broker. Confirm how it authenticates to the object store, how it handles credentials rotation, and how it reports failed uploads and fetches. Write these answers into the decision record so the choice survives staff changes.
Verify the operational behavior that the documentation will not settle for your workload. Test what happens when a broker fails while segments are mid-upload, when partitions are reassigned and remote segments must be fetched by a new leader, and when consumers read historical offsets that now sit in object storage. Measure consumer lag and read latency for historical data against your own message sizes and object store latency, because remote reads trade local disk speed for network and storage round trips. Finally, define the disaster recovery position for the case where the object store is unavailable or lost: which topics can tolerate reads stalling, whether remote data is replicated, and how you would restore local retention. Confirm each item with the Kafka project documentation, the plugin vendor or maintainers, and a proof of concept on your own infrastructure.
What buyers ask about Apache Kafka tiered storage cost, effort and ownership
What drives the cost of implementing Kafka tiered storage?
Implementation cost depends mostly on how many clusters and topics will use tiered storage, how many environments need it, and how much of the work your team keeps. The main components are the design of which topics tier and for how long, the choice and configuration of a RemoteStorageManager plugin, the object storage setup and access policy, testing of failover and reads from the remote tier, and runbooks for your operators. Ask a partner to price each component separately in the written scope, and to state which items are one-time and which recur. No figure is given here because the supplied sources do not state one for this work.
What will the object storage bill look like after rollout?
The object storage bill grows with the volume of log segments moved off local disk, how long you keep them, request activity when consumers read older data, and any cross-region or egress charges from your provider. Local broker disk costs usually fall as older segments leave the broker, but you still need enough local capacity for the active tier. Ask your storage team to model retention by topic, read frequency for historical data, and the pricing terms of your chosen object store before approving a rollout. Your provider's price sheet, not this page, is the source for those rates.
How long do a pilot and a production rollout take?
Duration depends on the number of clusters, the plugin you select, the readiness of your object storage and identity setup, and how many change approvals your organization requires. A pilot usually covers one non-critical cluster with a small set of topics, so that retention, reads from the remote tier and disable steps can be tested. Production rollout then adds monitoring, capacity checks and operator training. The supplied sources do not give a duration for this work, so ask for a timeline with milestones and dependencies in the written scope.
What does the work need from our own team?
Expect to provide access to the Kafka cluster and its configuration, a named owner for the object storage buckets and their lifecycle rules, and a security reviewer for credentials, encryption and network paths between brokers and the object store. Your platform team should also confirm the Kafka version and Java runtime already in use, since the upgrade notes set Java 17 for brokers and tools from Kafka 4.0. Plan time for change control and for someone to approve the cost model. The exact hours needed depend on your environment and should be listed in the scope.
Can our own staff implement tiered storage without a partner?
Yes, if your team already runs Kafka in KRaft mode, operates object storage and has the skills to test failure scenarios. The work is mostly configuration, plugin selection, access design and testing, so the gap is usually in experience with recovery and cost tracking rather than in Kafka itself. A partner is more useful when you lack a tested rehearsal of disabling the feature, when several clusters need consistent policy, or when your security review requires documented evidence. Ask yourself which of those gaps you have before deciding.
How is tiered storage work scoped and quoted in writing?
Most DeployOpen engagements are fixed-scope, so the scope, deliverables and timelines are agreed in writing before work starts, and a quote follows from that written scope. For this topic, a useful scope names the clusters and topics in each phase, the chosen RemoteStorageManager plugin and its version, the object storage configuration, the test cases, the handover documents and the decision points for moving to production. Ask each bidder to state assumptions about your environment and what changes would alter the quote. DeployOpen does not publish a price list.
Does tiered storage require changes to our clients?
Consumers and producers keep using the same topic names and client APIs, and reads of older data are served by the brokers, so application code usually does not change. Your clients may see different latency when they read historical offsets, so test those paths with realistic consumer patterns. Check the Kafka version and the plugin's compatibility with your clients before rollout, and confirm the exact behavior with the Kafka 4.3 documentation and your plugin vendor. Compacted topics are not supported for tiering, so review any compacted topics separately.
What recurring work follows a tiered storage rollout?
Recurring work includes monitoring the remote tier's capacity and request costs, reviewing retention and topic lists as teams add data, keeping the plugin and Kafka version current, and rehearsing restore and disable procedures. Kafka's release cadence is three releases a year, with bugfix releases for supported versions, so plan regular upgrade reviews. Supported releases listed on the downloads page are 4.3.1, 4.2.2 and 4.1.2, and rolling upgrades are supported from each of the last three releases to the latest version. Ask any partner whether upgrades, support and testing are included or sold as a separate retainer.
How DeployOpen scopes and delivers Kafka tiered storage
DeployOpen plans, builds and documents Apache Kafka tiered storage inside your own infrastructure, starting from a written scope that names the topics, retention targets and storage plugin involved. Engagements are fixed-scope, so deliverables, acceptance tests and handover items are agreed before work begins.
How we scope
- Inventory which topics would move to remote storage and which must stay local, including any compacted topics
- Record target local and remote retention periods, read and restore patterns, and the storage plugin you have chosen
- Confirm the Kafka version, KRaft mode and upgrade path, and agree the scope in writing before any build starts
How we build
- Configure the RemoteStorageManager plugin and object storage access in a non-production cluster first
- Test segment offload, consumer reads from remote segments, broker restarts and failure behavior against your planned workloads
- Run a documented disable procedure in staging so the rollback path is proven before production use
How we hand over
- Deliver configuration, runbooks for retention changes, plugin credentials rotation and capacity checks
- Provide acceptance test results and a record of the decisions taken on topic selection and retention
- Walk your platform and operations teams through monitoring of local and remote storage usage
How we support
- Offer ongoing support, plugin and Kafka upgrades and periodic testing as a retainer after handover
- Review the tiered storage configuration when Kafka releases change plugin behavior or your retention needs change
- Agree response expectations and upgrade windows in the retainer scope, since these vary by engagement
