Why Kafka large messages need a deliberate design
When producers start failing on payloads above the 1 MB default, the fix is either to match size limits across brokers, topics, producers and consumers, or to move the payloads out of the record entirely. Raising one limit in isolation usually just moves the failure to the next component in the path. The decision belongs to the platform team and the application owners together, made before the first large record reaches production.
The default maximum record size is set by message.max.bytes on the broker. A topic can override it with its own max.message.bytes, so a single topic can accept larger records while the rest of the cluster keeps the default. Teams usually run into this with documents, images, batches of events combined into one record, or large JSON documents that grow over time. Before changing anything, confirm which limit rejects the record, since the producer, broker and topic each report failures differently and the error text is the fastest way to find the bottleneck.
Raising limits has real operating costs. Larger records consume more broker memory and network buffers, increase replication traffic across every replica, and lengthen the time a consumer needs to fetch and process a single record. Consumers must accept the same maximum size or they will stall on a record they cannot read, so the producer, topic, broker and consumer settings need to be reviewed together. Ask the team to test with a realistic maximum payload and to confirm current limit names and defaults against the Apache Kafka upgrade notes for the version you run.
Which size settings to change and when
This table compares the settings that cap record size at each point a large message travels, from broker and topic limits to producer and consumer fetch limits. Read each row as a place where a record can be rejected or stall, and check that every limit is at least as large as your largest record after compression.
| Setting | Where it applies | When to change it | What breaks when it is mismatched |
|---|---|---|---|
| message.max.bytes | Broker: largest record batch the broker accepts | Raise it when the largest record you must accept exceeds the current cluster limit | Producers receive errors for records the broker rejects, even if topic settings allow them |
| max.message.bytes | Topic: largest record batch accepted for one topic | Set it per topic for the few topics that carry large payloads, so other topics keep tighter limits | If lower than the broker limit, the topic rejects records that the broker would otherwise accept |
| max.request.size | Producer: largest request the producer sends | Raise it so one request can carry the largest record plus batch overhead | Producer fails locally on send, before the broker sees the record |
| replica.fetch.max.bytes | Broker: fetch size followers use when copying partitions | Raise it when it is smaller than the largest record batch, so followers can copy it | Followers fall behind or stop replicating large records, leaving partitions under-replicated |
| fetch.max.bytes | Consumer: total bytes returned in one fetch across partitions | Raise it when consumers must fetch several large batches in one request | Consumers receive smaller fetches, which slows reads but does not drop data |
| max.partition.fetch.bytes | Consumer: bytes returned per partition in one fetch | Raise it to at least the largest record batch for every subscribed topic | A record larger than this limit can stall a partition, so the consumer makes no progress on it |
| Compression | Producer or topic: shrinks records before limits are applied | Enable it when payloads are text, JSON or other repetitive data that compresses well | Already compressed data, such as images or video, gains little, so limits still apply at near full size |
| Claim-check payloads | Application: stores the payload outside Kafka and sends a reference | Use it when records stay large after compression or consumers only need part of the data | Consumers must retrieve the stored object, so storage access and availability become part of the pipeline |
How to raise the limit without breaking the cluster
This sequence raises the record size limit for one topic in a controlled order, so producers, brokers, replicas and consumers all accept the same payloads without destabilizing the cluster.
Measure the largest payload and its compressed size
The platform team and the application owner pull the largest serialized record the producer actually sends, measured before and after the compression codec the producer uses. Sample a representative window of production traffic rather than relying on the application's design estimate. Record both figures per topic and add a margin for headers and framing, since the broker checks the size of the batch that carries the record. The step is done when the team has a written maximum for the affected topic, both raw and compressed, and has confirmed which codec the producer uses. Ask the application owner to confirm whether the largest messages are rare outliers or the normal case, because that answer decides whether a limit increase is the right fix at all.
Set topic-level max.message.bytes on the affected topic
The Kafka operator changes max.message.bytes on the one topic that needs larger records, using the topic configuration tooling and not the broker-wide default. Leave message.max.bytes on the brokers at its current value unless the measurement shows that other topics are also affected. A topic-scoped change limits the blast radius, because other topics keep the original cap and their memory use does not change. The step is done when describe-configs output for the topic shows the new value and the broker-wide setting is unchanged. Confirm that the new value covers the measured compressed maximum with the margin from the previous step, and record who approved the change and why.
Raise replica fetch limits on every broker
The cluster operator raises replica.fetch.max.bytes on all brokers so follower replicas can copy records that are now accepted by the leader. This step comes before any producer change because a follower that cannot fetch a large batch will fall behind and remain out of sync. Apply the change through the configuration process your team uses, roll it broker by broker, and wait for each broker to rejoin before moving on. The step is done when every broker reports the new setting and all partitions of the topic show the full set of in-sync replicas. Watch replication lag during the rollout, and stop if a broker's lag grows instead of recovering.
Raise producer and consumer limits in the application
The application team raises the producer's max.request.size to cover the new record size, and the consumer team raises the fetch limits that govern how much data each fetch can return, so a single large record can be read without stalling. Producers and consumers use separate settings, and both must be changed or the record will fail at one end. Deploy the producer and consumer changes in a non-production environment first, then in production with the topic's traffic. The step is done when the application configuration files or deployment manifests reflect the new values and the owners have reviewed them against the measured maximum. Note that the consumer configuration must also accommodate the record, and the team should confirm the exact settings for its client library version in the client documentation.
Test with a record at the limit and one above it
The test team sends two test records to a non-production copy of the topic, one just under the new limit and one just over it, and checks the outcome of each. The record under the limit should be accepted by the producer, stored by the broker, replicated to the followers and returned to the consumer intact. The record over the limit should be rejected with a size error at the producer or broker, which proves the limit is enforced where you expect it. The step is done when both outcomes match the expectation and the test records round-trip with identical payloads. Include a record that uses the production compression codec so the test reflects real batching.
Confirm the producer error clears and consumers fetch the record
In production, the application team checks that the message-too-large error no longer appears in producer logs or metrics for the topic, and that the records which previously failed are now delivered. The consumer team confirms that consumers read the large records, that processing time per record is within the expected range, and that no consumer stalls on a single fetch. Use the same dashboards and error counters used to identify the original problem so the comparison is consistent. The step is done when the team has a dated record of error counts before and after the change, and the owners of the affected consumers have signed off on processing behavior.
Watch broker memory and replication lag under load
The cluster operator observes broker heap, direct memory, request queue times and replication lag during a period of normal peak load, then during a period that includes large records. Larger batches use more memory per request and more network capacity during replication, so the goal is to confirm these stay stable rather than climbing or hitting alert thresholds. Compare against the baseline recorded before the change. The step is done when the team has seen a full peak period with no sustained growth in memory use, no replica falling out of sync, and no rise in request latency that the baseline does not explain. If any signal drifts, roll back the topic setting first, since it is the narrowest change to reverse.
Document the limits and the rollback path
The platform team records the final values of max.message.bytes on the topic, replica.fetch.max.bytes on the brokers, and the producer and consumer limits in the application, together with the measurement that justified them. It also writes down the rollback order, which reverses the steps: restore the topic setting, then the application limits, then the broker fetch setting once no larger records remain in the topic. Add a monitoring alert for records approaching the limit so the team sees growth before the next failure. The step is done when the runbook for the topic lists the values, owners, measurement date and rollback order, and the alert is active. Ask whether your organization's change management process requires a separate approval for each of these settings.
When the claim-check pattern is the better choice
The claim-check pattern stores the large payload in an external store, such as object storage or a database, and sends only a small reference record through Kafka. Choose it when payloads are large enough that raising message.max.bytes and max.message.bytes would push every broker, producer and consumer toward heavier memory use and slower replication, or when the same payload is needed by many consumers and copying it into every partition multiplies storage cost. Whether it fits depends on three things you should measure before deciding: the payload size distribution, the message volume per topic, and which consumers read the data and how quickly they must act on it. Small payloads at high volume usually stay inline, and a single occasional large record may not justify a second system.
The pattern adds costs that belong in the design review. Every consumer gets an extra read path, so each one needs credentials, network reach and error handling for the external store, and a missing or slow blob becomes a failure mode Kafka itself never had. The blob and the event can also disagree: a producer that writes the object and then fails before publishing leaves an orphan, while a reference published before the upload completes points at nothing. Decide the write order, whether you use a transactional outbox or idempotent retries, and how consumers respond to a reference that does not resolve. Retention needs the same care, because Kafka deletes records on its own schedule while the object store follows its lifecycle rules, so a reference can outlive its payload or the reverse.
Access control moves to the external store and must be designed alongside Kafka's own controls. Kafka ACLs protect the topic, but anyone who can read the bucket or table can read the payload, so the store needs its own identities, least-privilege policies and encryption, and the reference should not carry credentials or be usable by parties outside the topic's consumers. Confirm with your security and compliance owners which data classes may leave Kafka at all. Once the team has these answers, document the decision with the payload size threshold, the write and deletion sequence and the owner of each store, so later topics follow the same rules.
Security and operational risks of larger records
Raising record size limits moves cost and exposure from the message itself to the brokers, the replicas and any system holding the payload. The cards below cover the four places that usually surprise teams, plus the access model that applies once a payload store sits beside Kafka.
Broker memory and disk pressure
Each large record occupies broker heap during request handling and fills log segments faster, so a few oversized producers can push disk usage past what the topic sizing assumed. Check the largest record size per topic against segment size, page cache headroom and disk alerts before raising max.message.bytes. Test with the largest realistic payload, not the average one.
Replication and recovery time
Every large record is copied to each follower, so replication traffic and catch-up time after a broker restart grow with record size. A replica that falls behind during a large burst can stay out of sync longer than expected, which affects availability guarantees for min.insync.replicas. Measure replica lag and reassignment duration with production-sized records during testing.
Client memory and timeouts
Producers buffer and compress records before sending, and consumers must hold a full record in memory before processing it. Raising broker limits without matching producer and consumer settings leads to send failures or consumers that stall while fetching. Confirm that client memory settings and fetch limits are sized for the largest record, on every client that reads the topic.
Access control on the external payload store
With claim-check, the payload sits in object storage or a database outside Kafka, and Kafka's authentication and ACLs do not protect it. Kafka authentication and TLS are optional and set per listener, and ACLs in KRaft clusters are stored in the metadata log, so the external store needs its own TLS, identity and access policy. Confirm who can read each payload, not only who can read the topic.
Blob retention outliving topic retention
Topic retention deletes the reference, but the stored blob stays until a separate lifecycle rule removes it, so payloads can outlive the events that point to them. Records may then be retained longer than your data policy allows, or deleted from Kafka while copies remain elsewhere. Define one retention rule for both systems and test deletion of expired payloads.
Encryption and listener configuration
Large payloads often carry the most sensitive data in a flow, yet TLS is configured per listener and a plaintext listener can remain open next to a secured one. Verify that every listener used by producers and consumers of large records enforces the intended protocol and authentication, and document which listeners are unsecured. Review the Kafka security overview when setting listener protocols.
Questions teams ask before raising Kafka message size
Should we raise the Kafka message size limit or use the claim-check pattern?
For most workloads, keep records small and store large payloads in object storage or a database, sending only a reference through Kafka. Raising limits is reasonable when payloads are occasional, bounded and tied to one or two producers and consumers. Larger records increase broker memory pressure, replication traffic, fetch sizes and consumer lag, and every client in the path must accept them. Claim check adds a storage dependency and a lookup step, but it keeps broker behavior predictable. Decide per topic, based on payload size distribution and how many teams read the topic.
What makes the effort larger or smaller?
The main drivers are the number of topics that carry large records, the number of client languages and library versions, the number of consumer groups and downstream systems, and the storage layers involved (broker disks, object storage, a database for claim-check references). Each additional producer or consumer stack needs its own limit checks, tests and rollout. Topics that are compacted, mirrored across sites, or read by Kafka Connect connectors add extra checks. Ask for the inventory of topics and clients before a scope is written, because the count of these items usually matters more than the size of any single change.
What does our internal team need to provide?
Expect to provide broker administration access or a change window with someone who can apply configuration changes, the owners of each producer and consumer application who can confirm their client settings, and a load test environment that resembles production in broker count, replication factor and client versions. You also need someone who can approve the topic-level retention and storage impact. Without client owners available for testing, the work usually stalls at the consumer side, because a consumer that cannot fetch a large record fails in ways the broker logs will not show clearly.
Can our own engineers make these changes without a partner?
Yes, if your team already operates Kafka and has access to broker configuration, client code and a representative test environment. The configuration keys involved, including message.max.bytes, the topic-level max.message.bytes, and the producer and consumer size settings, are documented by the project, so the change itself is small. The risk lies in verification: confirming that every producer, consumer, replica and connector handles the new size, measuring memory and disk impact under load, and rolling back safely. Teams without that testing capacity often choose a fixed-scope engagement for the verification part.
How long does load and compatibility testing take?
Duration depends on scope: the number of topics, the number of client stacks, how representative the test data is, and how many rounds of tuning are needed. A single topic with one producer and one consumer group takes far less time than a shared platform with many teams and connectors. Ask for the test plan in writing, including which payload sizes are tested, which failure cases are covered (for example, a producer sending above the limit, or a consumer fetching a record larger than its buffer), and what evidence is delivered. The schedule should be agreed in the written scope before work starts.
How does DeployOpen price this work?
DeployOpen does not publish a price list. Most engagements are fixed-scope: the topics, clients, environments, deliverables and timeline are agreed in writing before work starts, and the quote follows from that written scope. To compare quotes, ask each provider to list the topics and clients covered, the test environments and tooling used, the documentation delivered, and what is excluded. Ongoing support or retesting after future client upgrades is usually a separate retainer, so ask which items are one-time and which recur.
Do larger records create compliance or security concerns?
They can. Large payloads often contain personal or regulated data, and storing them in an external store with claim check moves that data into another system that needs access control, encryption and retention rules. Raising broker limits keeps the data inside Kafka, where topic-level ACLs and retention apply, but it can also increase the blast radius of a misconfigured consumer. Confirm which data classes flow through the topic, who may read the external store, and how deletions propagate. Kafka's authorization model is documented, so review the ACL design alongside the size change.
How DeployOpen scopes large message work
DeployOpen plans, implements, hardens, documents and supports self-hosted Kafka inside your own infrastructure. For large message handling, we start from your payload sizes, producer and consumer behavior and retention needs, then agree the work in writing before any change is made to your clusters.
How we scope
- Inventory the largest payloads per topic, their growth and the producers and consumers that touch them
- Decide between raising limits, compressing payloads or moving bodies to a claim-check store, with the trade-offs written down
- Agree in writing which broker, topic, client and storage changes are in scope, with engagements fixed-scope
How we build
- Align message.max.bytes, topic max.message.bytes and client size limits so each layer agrees on the same ceiling
- Build and configure the chosen claim-check path, including object storage access, key naming and cleanup of stored payloads
- Test oversized records end to end, including replication, consumer fetch behavior and failure paths on your own environment
How we hand over
- Document every changed limit, its owner, the reason for the value and the rollback steps
- Provide runbooks for oversized record rejections, storage growth and claim-check retrieval failures
- Walk your platform and application teams through the configuration and confirm acceptance tests passed
How we support
- Offer ongoing support and upgrades as a retainer, so limit changes are reviewed when Kafka versions change
- Re-test large message paths after client library or broker upgrades, against the release notes for your version
- Confirm with your team which support hours, response terms and escalation paths apply before the retainer starts
