Tuning Kafka producers for throughput and latency

Tuning Kafka producers for throughput and latency starts with the workload

If your self-hosted Kafka producers miss their throughput targets or send in bursts that overwhelm brokers, the fix is rarely a single setting. Producer tuning is a set of trade-offs between batching, compression, acknowledgements and buffering, and each one must be measured against your real payloads and broker setup. Before you change batch.size or linger.ms, record the message size distribution, the peak and average rate per topic, the replication factor, and the latency your consumers can tolerate. Those numbers decide which direction is worth pushing.

The reference for defaults and behavior is the producer configuration section of the Kafka 4.3 documentation, which should be the version your team reads against when you set values. Confirm which release your cluster runs before you copy any setting from older guides. The project's bugfix policy applies only to the supported releases listed on the downloads page (4.3.1, 4.2.2 and 4.1.2 at the time of writing), so a broker or client on an archived version may not receive fixes for the behavior you are tuning. Check the Apache Kafka downloads page to confirm current support status before you plan a change window.

Write down a baseline on a staging cluster that mirrors production partition counts, replication and TLS settings, because encryption and replication both change the cost of each acknowledgement. Use a load generator fed with sampled production payloads, not uniform synthetic messages, since compression ratio and batch fill depend heavily on content. Record throughput, produce request latency, record error rate and broker request queue behavior for each candidate profile. Ask your platform or performance team who owns this test, how long it takes to repeat, and which broker metrics they will review, so the tuning work has a clear acceptance point.

Producer settings that change throughput and latency

This table compares the producer settings that most change how fast records leave a client and how long they wait on the way. Read each row as a lever: what raising or lowering it does, and which profile it usually serves. Exact default values must be taken from the Kafka 4.3 producer configuration reference, not from memory.

Setting	Effect of raising it	Effect of lowering it	Typical fit
batch.size	Larger batches per partition give more records per request and better compression, but hold more memory per partition.	Smaller batches send sooner with less memory use, at the cost of more requests and lower throughput.	Raise for high throughput with steady traffic; keep modest for low latency.
linger.ms	The producer waits longer for a batch to fill, which raises throughput but adds delay to every record sent during that wait.	The producer sends sooner, reducing added latency but producing smaller batches.	Raise slightly for throughput profiles; keep near zero for latency-sensitive topics.
compression.type	Compressed batches cut network and disk bytes, at the cost of CPU on producers and brokers, and gain more from larger batches.	No compression saves CPU but sends more bytes across the network and onto broker disks.	Enable for throughput profiles with compressible payloads; test the codec against real records first.
acks	acks=all waits for the in-sync replicas, which raises durability and adds round-trip time per request.	acks=1 or acks=0 lowers latency but weakens the guarantee that an acknowledged record survives a broker failure.	Use all for data that must not be lost; reserve lower values for loss-tolerant telemetry.
enable.idempotence	Idempotent producers prevent duplicate writes on retry and keep per-partition ordering within the producer's in-flight limits.	Disabling it removes duplicate protection on retries, which matters for any retried send.	Enable by default for most workloads; confirm the current default in the 4.3 producer reference.
max.in.flight.requests.per.connection	More unacknowledged requests per connection raise throughput on high-latency links, but can reorder retried batches when idempotence is off.	Fewer in-flight requests keep ordering simpler and reduce pipelining gains.	Keep within the limit the idempotent producer allows; tune only after measuring with real traffic.
buffer.memory	A larger buffer absorbs bursts and waiting batches, but uses more client heap and can delay sends when full.	A smaller buffer caps memory use and makes send() block or fail sooner under backpressure.	Size to peak burst volume across all partitions a producer writes to; confirm limits in the reference.

How batch.size and linger.ms decide when a batch is sent

The Kafka producer does not send each record on its own. It collects records into batches, one batch per topic partition, and hands a batch to the network when a send condition is met. The two settings that matter most here are batch.size, which caps how many bytes a batch can hold, and linger.ms, which sets how long the producer waits for more records before sending a batch that is not yet full. A batch goes out when it reaches batch.size, when linger.ms has elapsed since the batch was opened, or earlier when other conditions apply, such as the producer needing to flush or close, or memory pressure forcing buffered batches out. Because the decision is made per partition, a busy partition can fill and send its batch quickly while a quiet partition waits out the full linger.

A larger batch gives the broker and the network more work per request and gives the codec more data to compress, so the compression ratio usually improves as batches grow. The cost is waiting time: the earlier records in a batch sit in the producer until the batch closes. A small linger.ms keeps that wait short, but it does not stop batching under load. When records arrive faster than the producer can send, batches fill before the linger expires and the setting has little effect on them. Batching in that case comes from the arrival rate, which is why a low-latency profile can still produce well-filled batches on a busy topic. The reverse also holds: on a quiet topic, even a modest linger may never fill a batch, so the setting adds delay without adding efficiency.

Treat these values as starting points and measure them with your own record sizes, key distribution and arrival pattern. Average record size decides how many records fit in a batch.size buffer, so a topic with 200 byte records and one with 20 kilobyte records behave very differently at the same setting. Confirm the batch fill ratio, the request rate and the producer-side latency in your metrics before and after each change, and check the Kafka producer configuration reference for the exact defaults in the version you run. The Apache Kafka documentation lists the release notes for each version, which is the place to confirm whether default values changed during an upgrade.

Example producer profiles for high throughput and low latency

This illustrative profile trades a small amount of per-record delay for fewer, fuller requests. It suits bulk ingestion such as log shipping or backfills, where a record waiting a few extra milliseconds before it is sent has little business cost.

Larger batches: Set batch.size well above the client default so each request carries more records and the broker handles fewer requests per record.
Longer linger: Raise linger.ms so the producer waits for batches to fill before sending. This adds send latency in exchange for fuller batches.
Stronger compression: Set compression.type to a codec with a better ratio, such as zstd, so each record uses less network and disk. The codec choice is covered on the compression page.
Larger buffer and more in-flight requests: Increase buffer.memory and max.in.flight.requests.per.connection so the producer keeps the pipeline full. Confirm the ordering and idempotence behavior for your release before changing the in-flight value.
Costs: Client memory use rises, records reach consumers later, failed batches take longer to resend, and producer CPU increases for compression.

Check the average batch size, record send rate and request latency under peak load before accepting this profile.

This illustrative profile sends each batch as soon as it is ready. It suits interactive workloads, such as request-driven events or alerting paths, where a record that waits on a timer delays a user-visible result.

Near-zero linger: Set linger.ms at or near zero so records are not held for a timer. Batches then contain whatever records arrived during the previous request.
Modest batch size: Keep batch.size moderate so the producer does not allocate or wait for a batch that rarely fills at low volume.
Low-CPU compression: Choose a codec with low compression cost, such as lz4 or snappy, so compression does not add to send time. For very small payloads, disabling compression may give the lowest latency, and the effect should be measured.
Keep delivery guarantees: Leave acks and idempotence at settings that protect against loss and duplicates. Latency tuning should not weaken delivery guarantees, and the current defaults for each release should be confirmed in the Kafka documentation.
Costs: Smaller batches mean more requests, which raises broker CPU, network overhead per record and disk use, because compression ratios fall.

Watch request latency percentiles, the request rate and the average batch size, because a profile that looks fast in a quiet test can queue under load.

Neither profile is portable across workloads. Record size, compressibility, key distribution and broker load all change how each setting behaves, so the values must be checked with captured or representative payloads from your own systems before they reach production.

Representative payloads: Replay samples that match real record sizes and field content. Compressibility in particular differs widely between JSON events, binary records and already-compressed data.
Producer metrics: Track record-send-rate, batch-size-avg, request-latency-avg, compression-rate-avg and buffer-available-bytes, along with the error and retry counts your client exposes.
One change at a time: Compare runs at the same load and vary a single setting per run, so each result can be attributed to that setting.
Peak and failure tests: Measure at peak volume and during a broker restart or leader change, since retries and buffer pressure often show up only under those conditions.
Recorded profile: Store the final settings in version control with the test results and the reasoning, so later changes can be reviewed against the same evidence.

Treat both profiles as starting points until the chosen settings pass these tests with your own data and your own broker configuration.

How to tune and verify a producer profile before production

This sequence turns producer settings from guesswork into a documented profile by measuring real traffic, changing one group of settings at a time and checking both producer and consumer outcomes at every step. The result is a tested profile for each client that can be defended before go-live.

Capture real record sizes and send rates
Before changing any setting, measure what the producers actually send: average and maximum record size, records and bytes per second per topic, key distribution, and peak versus steady-state load. Capture this from production traffic or from a replay of representative payloads, since synthetic messages of a single fixed size rarely reproduce batching behavior. The step is done when the team has a short traffic profile per topic that the test plan can reproduce, including the peak window that the profile must survive.
Set a baseline profile with defaults
Start from the client library defaults for batch.size, linger.ms, compression.type, acks, enable.idempotence, max.in.flight.requests.per.connection and buffer.memory, and record the exact values the client reports at startup. Run the captured traffic profile against a test cluster and record throughput, producer request latency, batch size average, record-send error counts and buffer exhaustion events. This baseline is the reference every later change is compared against, so it must be stored with the test configuration and client version.
Confirm idempotence before changing in-flight limits
Check that enable.idempotence is true on every producer that will keep ordering or deduplication guarantees, and that acks is set to all where the topic requires durable writes. Idempotence matters here because it is the condition that allows more than one in-flight request per connection without risking reordering on retry. Confirm the setting in the client's effective configuration logs, not only in the source file. The check passes when every producer reports idempotence enabled before any in-flight change is tested.
Tune batching with linger and batch size
Change batch.size and linger.ms together, one pair at a time, and keep acks, compression and in-flight settings fixed. Increase linger.ms in small steps and watch average batch size, records per request, producer latency at the 99th percentile and the share of batches sent full versus sent on timeout. Stop when additional linger adds latency without a meaningful gain in batch fill or throughput. The output is a batching setting chosen from measured curves, with the latency cost written next to it.
Evaluate compression on the same payloads
With the chosen batching fixed, test each candidate codec on the captured payloads rather than on sample text. Compare compression ratio, producer CPU use, broker CPU use, network bytes and end-to-end latency. Compression efficiency depends on batch size, so re-check the ratio after batching changes. The step is complete when the team has one codec per topic or client, supported by the measured ratio and CPU figures, and when the broker and consumer CPU budgets still hold.
Adjust acks, in-flight requests and buffer memory last
Only after the earlier groups are settled, test acks, max.in.flight.requests.per.connection and buffer.memory, changing one at a time. Raise in-flight requests only on clients where idempotence was confirmed, and watch for retries, out-of-order errors, buffer waits and blocked send calls. Confirm durability trade-offs with the topic owner before relaxing acks. Each change should close with the same metric set so that a regression can be traced to a single setting.
Run a load test on a matching cluster and check consumers
Run the final profile on a test cluster with the same replication factor, min.insync.replicas, broker count and disk type as production, under peak load and a broker restart or leader move. Check producer error and retry rates, broker request latency and under-replicated partitions. Then confirm that consumer-side latency targets still hold, because larger batches and linger can raise end-to-end delay even when producer metrics look better. The step closes when the test shows the profile meets the agreed producer and end-to-end targets under failure conditions.
Document the chosen profile for each client
Record the final values for each producer client, along with the client library version, the traffic profile it was tested against, the cluster configuration, the metrics at each step and the trade-offs accepted, such as added linger latency or a specific acks setting. Store the record with the client configuration in version control so changes are reviewed. The step is done when each production producer has a documented profile that another engineer can reproduce and that is re-tested after client or cluster upgrades.

How acks and idempotence limit what tuning can trade away

The acks setting changes durability, not only latency. With acks=0 the producer does not wait for any broker acknowledgment, with acks=1 it waits only for the partition leader, and with acks=all it waits until the in-sync replicas have the record. Throughput tuning often pushes teams toward the faster end of that range, but the choice decides how many broker failures a record can survive. With acks=all, the broker-side min.insync.replicas setting determines how many replicas must hold the write before it is acknowledged. A topic with a replication factor of 3 and min.insync.replicas set to 2 keeps accepting writes through one broker loss, while a lower value quietly shrinks that margin. Check both values per topic before you change any producer batching parameter.

Idempotence prevents duplicates from retries, but only inside its preconditions. When enable.idempotence is true, the producer attaches a sequence number to each record so the broker can discard a retried batch it has already written. This matters more once you raise max.in.flight.requests.per.connection and retries to chase throughput, because reordered retries are exactly what idempotence guards against. Confirm in the Kafka 4.3 producer configuration reference which defaults apply in the release you run, since default values for acks and idempotence have changed across recent releases, and confirm the client library you use exposes the same behavior. Do not assume a profile you copied from an older guide still holds.

Treat any weakening of these guarantees as a recorded decision. Setting acks=1, lowering min.insync.replicas, or disabling idempotence can lift throughput on paper, but each one changes what a broker failure can lose or duplicate. Write down which data classes may accept that risk, who approved it, and how the consumers downstream detect duplicates or gaps. Then verify the chosen profile with a failover test on a non-production cluster, stopping a leader under load and checking the record counts at the consumer. If the business cannot state an acceptable loss or duplicate rate, keep acks=all and tune batching and compression instead.

Security and operational considerations for producer changes

Producer tuning changes what reaches the brokers and how fast, so the same review that covers settings should cover who may write, how traffic is protected, and how changes reach every client. The cards below cover the controls that sit around the producer configuration itself.

Producer authentication with SASL

Every producer should authenticate to the cluster with one of the documented SASL mechanisms: GSSAPI (Kerberos), PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 or OAUTHBEARER. Choose one per client group based on your identity system, and confirm that each mechanism is enabled on the listener the producer uses. Shared credentials across services make throughput incidents hard to trace.

Authorization with ACLs

In KRaft clusters, the StandardAuthorizer stores ACLs in the cluster metadata log. Grant each producer write permission only on the topics it publishes to, and confirm that resources without an ACL are not open to all users. Leave allow.everyone.if.no.acl.found unset or false so that a new topic does not accept writes by default.

TLS for client traffic

Encrypt traffic between producers and brokers with SSL/TLS, because security is optional in Kafka and a plaintext listener will accept producer traffic if one is left open. Check that the producer's truststore, and any keystore used for client certificates, are set in the client configuration. Batching and compression do not protect payloads on the wire, so the encryption setting is a separate check.

Buffer memory and heap on client hosts

The producer buffer.memory setting caps the memory used for unsent records, and it counts against the heap of the host running the application. Raising buffer.memory together with a larger batch.size can push a service into garbage collection pressure or out-of-memory errors under load. Load test the new profile on a client host with the production heap size before rollout.

Rolling out producer changes safely

Producer settings take effect when a client restarts, so a fleet can run old and new values side by side during a rollout. Ordering is preserved per partition only within the producer's in-flight limits, so changing max.in.flight.requests.per.connection or idempotence settings mid-rollout needs a deliberate plan. Roll changes one service at a time and watch for reordering or duplicate-related alerts.

Producer metrics and alerting

Track record send rate, request latency, batch size average, record error rate and retry rate for each producer client, so that a tuning change shows its effect in numbers. Alert on sustained error or retry growth rather than on single spikes. Compare metrics before and after each change, and keep the baseline in the same dashboard as the change record.

Change control for shared client configuration

When several teams share a producer library or a common configuration file, a single edit changes every service that uses it. Put producer defaults under version control, require review from the platform owner, and record which release of the client library each team runs. Ask teams to confirm in writing when they have tested a profile before it becomes the default.

Questions teams ask before tuning Kafka producers

Do we need a consultant to tune Kafka producers, or can our team do it alone?

Many teams can tune producers in-house, especially when one application owns one topic and the team already runs Kafka. A partner adds the most value when several producer applications share a cluster, when latency and durability targets pull in different directions, or when nobody on the team has measured producer behavior under load before. The work is mostly measurement and disciplined change control: you need a repeatable load test, a baseline, one change at a time, and a written record of why each setting was chosen. DeployOpen plans and runs this work as fixed-scope engagements, with scope and deliverables agreed in writing before any tuning starts.

What drives the effort of a producer tuning engagement?

Effort follows the shape of the producer estate more than the number of settings. The main drivers are:

The number of producer applications and how many languages or client libraries they use
How varied the workloads are, from small event messages to large payloads with different key patterns
How strict the latency targets are and how they are measured at the percentile level
The durability requirements, since acknowledgment and idempotence choices change both throughput and failure behavior

Each added application multiplies the test matrix, so ask for an inventory of producers before requesting a quote.

How long does a producer tuning cycle take?

The duration depends mostly on how ready the test environment is. A tuning cycle needs a cluster that resembles production in broker count, replication settings and network path, plus a way to replay realistic traffic. When that exists, the time goes to running load tests and comparing results. When it does not, the first phase often becomes building the test environment and collecting representative payloads, which can take longer than the tuning itself. Ask any provider to state the assumptions behind its timeline, including which environment it will use and who provides access.

What does the work need from our team?

Tuning cannot be done well from settings files alone. Plan to provide:

Sample payloads or a faithful sample of message sizes, key distributions and send rates from each producer
Read access to producer and broker metrics, such as request latency, batch size, record send rate and retry counts
A test cluster that matches production topology closely enough that results carry over, or a clear statement of the differences
A named contact who can approve changes to producer configuration and schedule test windows

Missing any of these usually shifts the work from tuning to guessing.

Are Kafka's default producer settings already good enough?

Often the defaults are safe, and for low-volume producers they may be enough. The defaults favor durability and broad compatibility, and they do not know your latency target or message profile. Batching behavior, compression and the number of in-flight requests are the settings most likely to leave throughput on the table, and the acknowledgment setting decides how much latency and durability you trade. Check the defaults against your producer version's documentation, then measure them against your own load before deciding whether to change anything.

What happens to producer tuning when we upgrade Kafka?

Producer tuning is tied to the client library and the cluster behavior you measured, so an upgrade is a reason to retest rather than a reason to discard the work. Upgrading brokers and upgrading clients happen on separate timelines, and each can change defaults or behavior. The Kafka upgrade notes and release plan describe which versions can roll forward together, so confirm the client and broker versions you will run before a rollout. Keep the load test suite and the settings record under version control so that each upgrade can be checked against the same baseline.

Does producer tuning change our security or compliance posture?

Tuning itself does not change who can write to a topic, but some choices affect what you can prove. The acknowledgment setting and idempotence affect whether a failed send can be lost or duplicated, which matters for audit trails and data integrity. Authentication and authorization for producers are configured separately, through SSL or SASL and ACLs in the cluster. Record every durability-related setting and its reason, so that a reviewer can connect the configuration to the business requirement it serves.

How DeployOpen scopes and delivers producer tuning

DeployOpen approaches producer tuning as a scoped piece of work on a Kafka cluster you already run or are building. We start from your throughput, latency and durability targets, then measure, change settings one group at a time and document what each change did.

How we scope

Written scope lists the producer applications, topics and payload sizes covered by the tuning work.
Agreed targets state the throughput, latency and durability requirements that define a successful profile.
Engagements are fixed-scope, so deliverables and timelines are confirmed in writing before any tuning begins.

How we build

We benchmark producers against representative payloads in a test environment that matches your cluster settings.
Each change to batch.size, linger.ms, compression, acks and in-flight requests is tested and recorded against the baseline.
Idempotence and ordering behavior are verified under broker failover before any profile is proposed for production.

How we hand over

Deliverables include the final producer configuration, the test results behind each setting and the reasons for the defaults kept.
A runbook explains how to adjust each setting safely, which metrics to watch and which changes need a retest.
Your team receives a walkthrough so the profiles can be applied, reviewed and reproduced without DeployOpen present.

How we support

Ongoing support for profile reviews, Kafka version changes and new producer workloads is available as a retainer.
Retainer work can include retesting producer profiles after broker upgrades within the supported Kafka release line.
Ask any provider to confirm the response model, reporting cadence and who owns production changes before signing.

Tuning Kafka producers for throughput and latency