Choosing Kafka compression: gzip, snappy, lz4 or zstd

Choosing Kafka compression: gzip, snappy, lz4 or zstd for your payloads

Choosing a Kafka compression codec is a trade-off between CPU spent compressing and decompressing and the bytes saved on the network and on broker disks. Teams usually reach this decision when record volume starts pressing on brokers, inter-broker or client network links, or disk headroom, and someone asks whether turning on compression would relieve it. Kafka compresses record batches, not individual records. A producer groups records into a batch, compresses the whole batch, and sends it; brokers store the compressed batch and consumers decompress it. That means the gain depends on how much redundancy a batch contains, not only on the size of each message.

Apache Kafka is an Apache Software Foundation project released under the Apache License 2.0 (SPDX: Apache-2.0), and the project itself offers no paid edition. Compression is part of the core producer and topic configuration, so the codec choice is a settings decision you own and can test in your own environment. Before you change anything, decide whether the pressure is network, disk or CPU, because compression only helps where bytes are the constraint. If producers or consumers are already CPU-bound, a heavier codec can move the bottleneck rather than remove it. Check the Apache Kafka downloads page for the release line you run, then confirm codec support and defaults in the documentation for that version.

Confirm three things with your own data before you standardize: the codec each producer uses, whether the topic level setting overrides or keeps what producers send, and how the compressed batch size interacts with your batching settings. Replay a representative sample of real payloads through a test cluster, measure produced bytes, broker disk growth, producer and consumer CPU, and end-to-end latency for each candidate, and record the results alongside your SLOs. Synthetic or highly repetitive test data will overstate the savings, so treat a sample taken from production traffic as the only reliable input.

Compression settings and when each codec fits

This table compares the four Kafka codecs and the three places compression can be set: the producer, the topic, and the broker. Read each row for the trade-off between CPU use, compression ratio and speed, then check the workload column against your payload type and throughput. Confirm codec support and defaults in the Kafka 4.3 configuration reference before you standardize on one.

Setting or codec	Trade-off	Workload where it fits
gzip	Highest compression ratio among the four codecs, but the heaviest CPU cost on producers and consumers	Text-heavy payloads such as logs or JSON where network and disk savings matter more than CPU headroom
snappy	Low CPU cost and fast, with a modest compression ratio	Latency-sensitive pipelines with moderate volume that need some size reduction without much CPU spend
lz4	Very fast compression and decompression, with a ratio usually below gzip and zstd	High-throughput topics where broker and client CPU is the constraint and speed matters most
zstd	Good balance of ratio and speed, with tunable levels that trade more CPU for smaller output	Mixed or large payloads where ratio gains justify extra CPU, and a common general-purpose choice to test first
Producer compression.type	Compression happens on the producer, so CPU cost lands on producer machines and batch size controls the ratio	Default choice when you control producers and want compressed batches to travel over the network and into the log
Topic compression.type	Set per topic; a value other than producer will cause the broker to recompress incoming batches, adding broker CPU	Use when a topic needs one codec across producers, or when you want the broker to store a fixed codec regardless of producer choice
Broker compression.type	Broker-wide default that applies when the topic does not set one; recompression is a broker CPU cost	Confirm the default behavior and whether recompression is triggered before relying on it for storage savings
Batch size and linger	Larger batches usually compress better, but add delay before records are sent	Any codec; test batch settings together with the codec using your own payloads

Where CPU, network and disk savings pull against each other

Every compression codec trades CPU time for smaller bytes, and the bytes saved show up in different places: producer and consumer hosts, network links, broker disks and replication traffic. The right codec depends on which of these is your bottleneck, so the cards below are the places to measure before you choose.

CPU headroom on producers and consumers

Compression runs on the producer before the batch leaves the client, and decompression runs on each consumer that reads it. A codec that compresses harder costs more CPU per record, which matters most on busy application hosts that also run business logic. Check producer CPU and send latency at your peak rate, and confirm that consumer groups with many partitions still keep up after decompression.

Network egress between sites and clients

Smaller batches cross the network between clients, brokers and any remote site that receives replicated or mirrored data. If you pay for cross-site bandwidth or your links are already near capacity, the codec that compresses best may save more than its CPU cost. Measure bytes on the wire per topic before and after the change, and include MirrorMaker 2 traffic in the count.

Broker disk and retention cost

Brokers store batches in the compressed form they receive, so a better codec reduces the disk needed for each day of retention. The saving scales with retention length and replication factor, so long-retention topics gain the most. Ask your team to confirm the storage model for your Kafka version, and compare disk growth on a representative topic over several days.

Replication traffic between brokers

Followers copy the compressed batches from the leader, so a codec that shrinks data also shrinks replication traffic inside the cluster. This can matter more than client egress when partitions have several replicas or when reassignments run during business hours. Measure replica fetch bytes and under-replicated partition counts during normal load and during a planned reassignment.

Batch size and linger effects

A codec compresses a whole batch, so the compression ratio depends on how many similar records a batch holds. Raising batch.size or linger.ms gives the codec more context and usually a better ratio, but each record waits longer before sending. Test a few combinations with your real payloads and record both the ratio and the added latency for each one.

Mixed-codec topics

A topic can receive batches from producers that use different codecs, and the broker stores each batch as it arrives unless topic-level settings recompress it. Mixed codecs complicate capacity tests and make it harder to attribute savings to one change. Decide which level owns the codec for each topic, record it in your topic standards, and check that consumers in every team can decompress what they read.

Producer-level versus topic-level compression decisions

Application teams usually control compression through the producer setting compression.type. The producer compresses each batch before sending it, so the codec choice affects CPU use on the client, the bytes sent over the network and, if the topic keeps the producer's codec, the bytes written to disk. Producer settings belong to the team that understands its payloads, message sizes and throughput targets, because those factors decide whether a codec saves meaningful space.

Payload fit: JSON and text records often compress well, while already-compressed images or encrypted blobs gain little and still cost CPU.
Batch size: Compression works per batch, so larger batches from batch.size and linger.ms usually give the codec more data to work with.
Client CPU budget: Test the producer on production-sized hardware, since compression runs on the application host.
Consistent settings: Producers in one service should share a profile so results are comparable between releases.
Documented defaults: Confirm the default value and the accepted codec names for your client library against the 4.3 documentation rather than assuming them.

Before changing a producer, benchmark the codec with a sample of real records and check the compression ratio and producer CPU together.

Platform teams often set the topic-level compression.type and decide whether brokers should recompress data. A topic value of producer keeps whatever codec the client used. A specific codec value tells the broker to store records in that format, and the broker recompresses batches that arrived with a different codec. That recompression uses broker CPU, which is a shared resource across all topics on the cluster.

Storage uniformity: A fixed topic codec makes disk usage predictable even when several application teams write to the same topic.
Broker CPU: Recompression competes with request handling and replication, so measure broker CPU before and after a change.
Retention sizing: Stored size under the topic codec feeds directly into disk calculations for retention and replication.
Mirroring: Replication and cross-cluster mirroring carry the stored batches, so the codec chosen on the source topic affects the bytes those tools move.
Change control: Changing a topic codec affects only new data, so plan the effect on existing segments.

Document which topics enforce a codec and which accept the producer's choice, so application teams know what the platform will do to their batches.

How to test compression with real payloads

This sequence shows which codec gives the best trade-off for your own data by measuring compression ratio, CPU cost and latency on payloads that look like production traffic. It produces a comparison you can defend before any setting reaches a production topic.

Capture representative payloads
The data owner and a platform engineer export a sample of real messages from the topics you plan to compress, using production traffic or a masked copy if the data is sensitive. Include the full range of message sizes, key formats and field structures, since repetitive JSON, Avro and binary payloads compress very differently. Keep the sample in a separate test topic or file set so it never mixes with live consumers. The step is done when the sample covers each message type on the topic and the team has signed off that masking removed personal or regulated values without changing the structure.
Fix the batching settings under test
The engineer records the producer settings that will run in production, including batch.size, linger.ms and the expected message rate, because compression works on whole batches and a codec tested at small batches can look worse than it performs later. Write these values into the test plan so every codec runs under identical conditions. The step is done when the test plan lists one fixed batch and linger profile, plus any higher profile you want to compare against it.
Run the producer for each codec
Using a test cluster or an isolated topic on the target cluster, the engineer runs the same producer workload once per codec: gzip, snappy, lz4 and zstd, plus none as a baseline. Each run replays the same payload sample at the same rate and records the compression.type value used. Repeat runs so that one noisy run does not decide the result. The step is done when every codec has at least one complete run with the same input, rate and settings.
Check that the codec is applied
Before trusting any measurement, the engineer confirms that the codec was really applied to the batches. Use the producer metrics for the compression rate, and inspect a sample of stored batches with the Kafka dump log tool to read the compression codec in each record batch header. Also confirm that a topic-level compression setting did not override the producer setting, or the reverse, depending on how the cluster is configured. The step is done when each run's stored batches show the codec under test and the baseline run shows none.
Measure size, CPU and latency
For each run, the engineer records the average compressed batch size against the uncompressed size, producer CPU, broker CPU on the leader and follower brokers, and end-to-end latency at the same percentiles used for production. Broker CPU and disk or network savings should be read together, because a codec that saves disk but raises broker CPU may still be the wrong choice on a busy cluster. The step is done when the results sit in one table with one row per codec and the batch and linger profile stated at the top.
Measure consumer decompression time
The engineer runs a consumer group against each test topic and measures decompression time and the rate at which records are processed, with the same consumer configuration in every run. Decompression cost lands on consumers, so a codec that is cheap for producers can still slow a downstream service that is already near its limit. Include any consumers that run in other languages or client versions, since codec support differs by client. The step is done when every codec has a consumer throughput figure from a run that read the full sample.
Verify every consumer reads every codec
Before the decision, the team confirms that each downstream consumer can read every codec the producers will use, including mixed topics where a topic holds batches from several codecs during a migration. Run each consumer against a topic that contains all codec types and check that it returns every record without error and with matching payloads. Confirm the client library versions in production match the versions tested. The step is done when each consumer has read the mixed-codec topic cleanly, and any failure is logged with the client version that caused it.
Record the decision and rollout plan
The team writes down the chosen codec, the batch and linger settings it depends on, the producer and topic-level setting, and the results that justified the choice. Set a rollout that changes one producer group or one topic at a time, and repeat the size and CPU checks on live traffic before widening it. The step is done when the decision record names the codec, the measured trade-offs and the owner of the rollout, and the first production topic shows the expected compression rate on its own dashboard.

Security and operational considerations for compressed traffic

Compression changes how bytes move through Kafka, but it does not change who may read them or how they are protected in transit and at rest. The cards below cover the controls that stay in place after a codec is chosen, and the operational work a codec change creates.

TLS and SASL stay separate

Compression reduces bytes on the wire, but it does not encrypt them. The Kafka security overview describes SSL/TLS and SASL as optional and configured per listener, so a cluster can run an unsecured listener next to a secured one. Confirm that every listener carrying compressed producer and consumer traffic uses the transport and authentication you expect, and check the listener map before you measure throughput on a new codec.

ACLs apply to compressed records

Compression is applied to record batches, not to access decisions. Topic ACLs in KRaft clusters are stored in the metadata log and govern who can produce to or consume from a topic, regardless of codec. Resources without ACLs are limited to super users unless allow.everyone.if.no.acl.found is set to true. Review topic permissions after onboarding a new producer, since a codec choice does not narrow who can read the data.

Encryption at rest is a separate decision

Compressed segments sit on broker disks in compressed form, which is not the same as encrypted form. If the data is sensitive, encryption at rest must come from the storage layer, such as encrypted volumes, or from an application-level design agreed with your security team. Ask who holds the keys, how they rotate, and whether backups and replicas inherit the same protection.

Client codec support

Every producer and consumer must be able to decode the codec used on a topic. A client library that lacks support for a codec will fail to read the records, which can stall a consumer group during an upgrade or a rollout to an older runtime. Inventory the client libraries and versions across all teams, test each codec against them, and record the minimum supported version alongside the topic settings.

Monitoring and troubleshooting changes

A codec change alters the bytes per record, so dashboards built on throughput, broker CPU, and network rates shift after the switch. Troubleshooting also changes: when a consumer fails on a corrupted or unsupported batch, engineers need to know which codec the topic used. Update alert thresholds, record the codec in runbooks, and keep a baseline from before the change so that regressions are easy to trace.

Questions teams ask before choosing a Kafka compression codec

Is zstd always the best Kafka compression codec?

No. zstd often gives a strong compression ratio, but the best codec depends on your payloads, your CPU headroom and whether brokers or consumers are the bottleneck. Gzip compresses tightly but costs more CPU, snappy and lz4 favor speed, and zstd sits between them with tunable levels. Treat zstd as a strong candidate to test first, not a default you can skip testing. Compare compression ratio, producer CPU, broker CPU, end-to-end latency and consumer decompression cost on your own records before you decide.

What drives the effort of a Kafka compression test?

The main drivers are how many payload types you have, how representative your sample data is, and how many producer and topic configurations you compare. A single topic with uniform JSON is a small exercise. A platform with binary payloads, mixed record sizes and several producer profiles needs more replay runs and more careful measurement. Effort also grows if you must reproduce production traffic patterns, including key distribution, batch sizes produced under real linger settings and peak load. Ask the provider which of these they will cover in the written scope.

What does the internal team need to provide for a compression test?

Your team should provide representative payload samples, the producer client versions and configurations in use, the topic settings and replication setup, and access to a non-production cluster that matches broker hardware closely enough to be useful. Consumer applications and their language clients also matter, because decompression cost shows up there. Someone must also own the decision, since codec choice affects producers, brokers and downstream consumers. Agree on the success criteria before testing starts so that results can be judged against them.

Can our own team run a Kafka compression test without a partner?

Yes, if you have engineers who can build a repeatable load test, capture real payloads safely and read producer and broker metrics. Many teams start with a small benchmark on a staging cluster. A partner adds value when payload sampling is sensitive, when the platform spans many topics, or when the results must be defensible to auditors or leadership. Whichever route you choose, keep the test repeatable so you can rerun it after Kafka upgrades or client changes.

How long does a compression test cycle take with varied payloads?

The cycle length depends mainly on payload variety and the number of configurations under comparison. Each additional payload family needs its own sample set and its own runs, and each producer profile multiplies the work. Test duration also depends on how long you need to reach steady state under load and how much data you need for stable ratios. Ask for a timeline tied to a defined list of payload families and configurations, and ask what changes the schedule if sampling or access to a test environment is delayed.

What changes after switching the compression codec on a topic?

Compression is applied by producers and stored in batches, so a switch affects new batches while older data stays in its original form. Brokers may need more or less CPU and disk I/O, storage use will change for new data, and consumers will decompress with the new codec. Confirm the codec behavior with the Kafka documentation for your version and check that all clients support it. Plan a rollout that changes producers in stages, watch broker CPU, producer latency and consumer lag, and keep a rollback path ready.

What should we ask for in a fixed-scope quote for compression testing?

Ask for a written list of payload families, producer and topic configurations to test, the environment used, the metrics collected and the report format. Confirm which client languages and Kafka versions are covered, what the team needs to provide and by when, and how retesting is handled if the first results are inconclusive. Also ask what the deliverable recommends, whether it includes rollout guidance and what is excluded, such as changes to application code or broker hardware. A clear scope makes the quote comparable with an internal estimate.

How DeployOpen scopes and delivers Kafka tuning work

DeployOpen treats the choice of Kafka compression codec as a measured decision, not a default setting. We start from your payloads, producer behavior and cluster constraints, then test candidate codecs in your own environment before recommending settings for production topics.

How we scope

The written scope lists the topics, producers and consumers whose compression settings will be reviewed.
We ask for sample payload shapes, message sizes, batch behavior and current producer configuration before quoting.
Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts.

How we build

We set up a test harness on your infrastructure that replays representing payloads through each candidate codec.
Measurements cover producer CPU, compression ratio, broker disk use, network bytes and consumer decompression cost.
Test runs are repeated at the batch sizes and linger settings your producers actually use, then compared.

How we hand over

You receive a written recommendation with the chosen codec, the producer and topic settings, and the test evidence behind them.
The configuration changes are documented so your platform team can apply and roll them back without guesswork.
Runbook notes explain how to repeat the test when payloads change, including what to re-measure first.

How we support

Ongoing support and periodic re-testing are available as a retainer, useful when payload formats or volumes shift.
Retainers can cover reviews after Kafka upgrades, since the release notes should be checked against your settings.
Ask any provider to confirm the codec support in the Kafka release you run before relying on it in production.

Choosing Kafka compression: gzip, snappy, lz4 or zstd