Retry topics and dead letter topics for Kafka consumers

Why a poison message stalls Kafka consumers and when retry and dead letter topics help

A consumer group stops progressing when one record cannot be processed and the consumer keeps retrying that same record, because Kafka delivers records within a partition in offset order and does not skip ahead on its own. Retry topics and dead letter topics are the two common ways to move a poison message out of the way so the rest of the partition keeps flowing. Use a retry topic when the failure may be transient, such as a timeout calling a downstream service, and a dead letter topic when the record will fail again no matter how often it is retried, such as a payload that violates the schema your code expects.

Kafka itself has no built-in dead letter or retry-topic mechanism at the broker. Both patterns are built in consumer code, in a client library such as the error-handling components in a framework, or in Kafka Connect error handling, and the team that owns the application decides the topic names, retry counts, delays and what metadata travels with a failed record. Apache Kafka is released under the Apache License 2.0 and the project offers no paid edition, so the cost of building and running these patterns sits with your engineers or with whichever partner writes the code. Confirm which client library and version your services use before choosing an approach, because error-handler APIs differ between clients and frameworks.

The Apache Kafka documentation for the 4.3 release defines the client and consumer contract that these patterns depend on, including how offsets are committed and how a consumer group rebalances after it stops heartbeating. Read it alongside your client's own documentation, because the broker will not enforce the policy you choose. Decide early whether a stuck partition is acceptable for a given topic, since strict ordering for a customer account may rule out moving one record aside, and that trade-off should be recorded before any retry topic is created.

Blocking versus non-blocking retries compared

A blocking retry keeps the failing record at the head of its partition. The consumer retries the same record in place, and no later record from that partition is processed until the record succeeds or the handler gives up. Ordering within the partition is preserved because nothing passes the failing record. The cost is that one poison message can stall every key that shares its partition, and throughput for that partition drops to zero while retries run.

Partition progress: the committed offset does not move past the failing record, so consumer lag grows for the whole partition.
Ordering: strict order is kept, which suits ledgers, state machines and any stream where a later event depends on an earlier one.
Throughput: a slow or repeatedly failing record reduces throughput for every key on that partition.
Operational visibility: the stall shows up as rising lag and repeated handler errors, but the individual record is hard to isolate without logging the offset.
Retry budget: the number of attempts and the wait between them are set in the consumer, so the design must confirm how long a stuck partition can be tolerated before alerting.

Before choosing this pattern, confirm with your business owners how long a partition may stall and whether a skipped record can be recovered later.

Configuration choices for retries, delays and dead letters

This table compares the decisions a team sets when it builds retry and dead letter handling, with the trade-off each one carries. Most of these settings live in the consumer library you choose, not in the Kafka brokers, so confirm the names and defaults in your library's documentation. Read each row as a choice to make deliberately before the first production incident.

Decision	When it fits	Trade-off
Retry count per stage	Transient faults such as a brief downstream outage, where a few attempts usually succeed	More attempts delay the poison message's final outcome and add load on the failing dependency
Backoff and delay per stage	Downstream systems that need time to recover, with longer delays for later stages	Long delays hold messages for minutes or hours, which increases lag and makes ordering harder to reason about
Retry topic naming and partitioning	Separate retry topics per delay tier, named so operators can tell stages apart at a glance	More topics mean more partitions, metadata and monitoring to maintain; keying must preserve the order you need
Dead letter topic retention	Long enough for triage and replay after a fix, often longer than the source topic	Longer retention costs disk and keeps sensitive payloads around; confirm retention rules with your data owners
Error metadata headers	Operators need the original topic, partition, offset, exception and attempt count to replay safely	Headers can grow large with stack traces and may carry sensitive text, so limit what is copied
Commit timing	Commit the offset after the record is handled, or after it is republished to a retry or dead letter topic	Committing before publishing risks losing the record; committing late raises duplicate processing after a restart
Retryable versus fatal exceptions	Timeouts and unavailable dependencies are retryable; schema or validation errors usually go straight to the dead letter topic	Misclassifying a fatal error as retryable wastes attempts; misclassifying a transient error as fatal strands good messages
Blocking versus non-blocking retry	Blocking retry suits strict per-partition order with short, rare failures; non-blocking suits independent records	Blocking stalls the whole partition; non-blocking gives up strict order for the retried records

What ordering guarantees you give up when records leave the partition

Kafka guarantees order within a partition, and every retry path breaks that guarantee in a different way. Decide which ordering rules are business requirements for each topic before choosing a retry design, because the answer determines whether a retry topic, a blocking retry or a dead letter topic is acceptable.

Same-key records processed out of order

When a record for a key fails and moves to a retry topic, later records for the same key on the original partition keep flowing. The retried record is then processed after them, so the consumer sees a newer state before an older one. For an account balance or order status, this can overwrite a correct value with a stale one. Check whether each handler can tolerate a late older event, or whether it compares a version or timestamp before applying the change.

Delayed retries reorder later records

A retry topic with a delay holds a failed record for a fixed interval, and the records behind it on other partitions or later offsets are not held. The retried record can therefore land after events that were produced well after it. Ask which events must wait for a retried event to finish, and whether the consumer group can afford the extra lag while a backlog of retries drains.

Dead-lettered records break causal sequences

A record sent to a dead letter topic after its retries are exhausted is removed from the live flow. Any later event that depends on it, such as a payment confirmation that follows a rejected payment, may be applied without its prerequisite. Map which event types depend on each other in your domain, and decide whether dependent events should pause, be dead-lettered together, or be checked against the state the missing event would have created.

Idempotence and transactions stop at the consumer boundary

The idempotent producer and Kafka transactions protect writes to Kafka, not what your handler does with a record. A retry that calls an external payment API, sends an email or updates a database table can still run twice or out of order. Inventory each side effect the consumer performs, and confirm that each one is keyed by a stable business identifier so a repeated or reordered run is detected.

Ordering rules as business requirements

Decide ordering per topic before choosing a retry pattern. Ask the owners of each event type whether order matters across all records, only per key, only per entity version, or not at all. Topics that carry commands or ledger movements usually need strict per-key order, while independent notifications often do not. Record the decision in the topic's documentation so retry and dead letter settings follow the agreed rule.

How to build retry and dead letter topics on a self-hosted cluster

This sequence turns a consumer that stalls on poison messages into one that retries safely, parks failed records in dead letter topics with enough context to investigate them, and proves the routing works before production traffic depends on it.

Classify errors as retryable or terminal
The owning application team lists the exceptions its consumers can throw and sorts each one into retryable (a downstream timeout, a rate limit, a temporary lock) or terminal (a schema violation, a record that fails validation, a null where a value is required). Decide which failures get a bounded number of retries and which go straight to the dead letter path. Write the table down with the owners of each downstream dependency. The step is done when every exception class the consumer can raise maps to one route, and an unmapped exception has a defined default, usually the retry path with a cap.
Create main, retry and dead letter topics
The platform team creates the main topic, one or more retry topics and a dead letter topic, and records the partition count, replication factor and retention for each in the topic catalog. Retry topics usually need enough retention to cover the longest delay the consumer will apply, and the dead letter topic needs retention long enough for investigation and replay. Confirm the partition count of each retry topic matches the main topic, because keyed records must keep their partition mapping to preserve ordering within a key. Check the topics with the admin tooling and confirm the configuration matches the catalog entry.
Grant ACLs for consumers and producers
The security owner grants each consumer group read access to its main topic and write access to its retry and dead letter topics, and grants each producer only the write access it needs. In KRaft clusters, authorization is governed by the StandardAuthorizer, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true, so a new topic that lacks grants will fail for its consumers. Also confirm that the group that replays dead letters has its own identity and cannot read topics it has no business reading. The check is a test client from each identity producing or consuming against each topic and receiving the expected allow or deny.
Implement routing with diagnostic headers
The application team adds routing logic that attempts processing, republishes a failed record to the next retry topic with a delay mechanism, and sends it to the dead letter topic after the final attempt. Each republished record carries headers that record the original topic, partition, offset, attempt count and exception class and message. Ensure the consumer commits the offset of the original record only after the routing write is acknowledged, otherwise a crash can lose the record. The step is done when unit tests cover a retryable failure, a terminal failure and a successful retry, and each test asserts the headers.
Inject poison messages into a test cluster
In a non-production cluster that matches production configuration, the team publishes deliberately malformed records, records that trigger each classified exception and records that fail only after several attempts. They confirm that the partition keeps moving, that retryable records succeed after the expected number of attempts, and that terminal records reach the dead letter topic with the original coordinates in their headers. They also stop the consumer mid-retry and restart it to confirm no record is lost or processed silently twice in the routing path. Record the results against the classification table from step one.
Verify lag and dead letter dashboards
The operations team builds dashboards and alerts for consumer lag on the main and retry topics, the rate of records entering each retry level and the rate and total volume of records in the dead letter topic. Alerts should fire on a sustained rise in dead letter volume and on lag growth that retries cannot explain, so a bad deploy is caught early. Confirm the alerts reach the on-call rota and that a runbook names who replays dead letters and under what conditions. The step is complete when a deliberate injection in the test cluster triggers each alert and the alert resolves after the records are cleared.
Document the replay procedure before go-live
Before production traffic depends on the routing, the team writes a replay procedure that states how dead letters are reviewed, which fixes are required before replay, how replayed records are routed back to the main topic, and how duplicates produced by replay are handled by the downstream consumer. Test the replay against the injected records in the test cluster and confirm that headers survive the round trip. Record who approves each replay batch. The procedure is done when a team member who did not build the routing can replay a test batch from the runbook alone.

How to replay dead letters without corrupting downstream state

Replaying a dead letter is a new delivery, so the handler must be safe to run twice before anything is reprocessed. Start by fixing the cause that sent the record to the dead letter topic, whether that is a code defect, a bad schema, or a dependency that was down. Then confirm that the handler's side effects in databases, payment rails, email or downstream APIs are idempotent, typically through a business key, a unique constraint or a processed-records table. Without that, a replay can charge a customer twice or send a notice again, because the original attempt may have partly succeeded before it failed. Ask the application owner to write this down per dead letter type so the decision is not made during an incident.

Reprocess from the dead letter topic with a dedicated consumer group and a controlled rate. Keep the replay consumer separate from the live group so that its offsets do not interfere with normal progress, and so that it can be stopped without touching production consumption. Record the offset range or record keys you replay, along with the outcome of each attempt, in a store your operators can query. The Kafka consumer group and offset commands can show the replay group's position, but they do not record which business records succeeded, so that log is your responsibility. Records that fail again should go back to a dead letter path with the failure reason and attempt count attached, not into an open loop of manual retries.

Plan for the dead letter topic itself as a store of business data. It holds payloads that may contain personal, financial or health information, so set retention deliberately, long enough to cover your investigation and replay window but no longer than your data policy allows, and restrict read and write access with ACLs on the topic and the replay group. Confirm with your security and compliance owners who may read these records, and whether replay activity must be logged for audit. Retention values and access model settings should be checked against the Kafka version you run, using the Apache Kafka security overview as the starting reference.

Security and operational risks of dead letter topics

A dead letter topic copies the failing payload, and often its headers, out of the context that protected the source topic. The cards below cover the controls that have to follow that data: access, encryption, header content, retention, alerting and replay permissions.

Dead letter topics inherit regulated data

A dead letter topic holds the same payloads that failed processing, so personal or regulated fields arrive there intact. Treat it as a copy of the source topic, not as a scratch area. Confirm which data classes reach each source topic, then apply the same classification to the matching dead letter topic before any consumer is added to it.

Match ACLs to the source topic

Each dead letter topic needs its own ACL entries for producers, consumers and administrators. Kafka's StandardAuthorizer stores ACLs in the cluster metadata log, and a resource with no ACL is limited to super users unless allow.everyone.if.no.acl.found is set to true. Check that new retry and dead letter topics are created with explicit ACLs, so the default never grants broader access than the source topic.

Keep stack traces out of headers

Error handlers often write exception class names, messages and stack traces into record headers. Those strings can contain query text, account identifiers or internal hostnames, and they are readable by anyone who can consume the topic. Limit headers to an error code, a failure category and a retry count, and log full traces in a restricted system with its own access policy.

Bound retry topic growth

Retry topics accumulate records for every delay tier, and a slow downstream outage can fill them quickly. Set retention on each retry and dead letter topic deliberately, separate from the source topic's setting, and decide whether expired records are acceptable losses or must be archived first. Ask your platform team to confirm the broker storage limits that apply to those topics.

Alert on dead letter volume

A dead letter topic that grows steadily often signals a schema change or a broken dependency, not isolated bad records. Alert on the rate of records written to each dead letter topic and on the age of the oldest unprocessed record, and route those alerts to the team that owns the source consumer. Tie thresholds to the expected error rate for each service.

Restrict who can replay

Replaying a dead letter re-runs business logic, so it can duplicate payments or re-send notifications. Grant replay rights to a named group, separate from the consumers that process live traffic, and require a recorded approval for each replay batch. Log who replayed which offsets and when, and confirm that the replay tool authenticates with its own identity rather than a shared service account.

Questions teams ask before building retry and dead letter topics

How much effort does a retry and dead letter design take, and what drives it?

The effort is driven by how many distinct failure types your consumers meet, how many retry topics you create, how idempotent each handler is, and how much observability you build around the flow. Error classification comes first: transient failures such as a timeout need a different path than a payload that will never parse. Each extra retry tier adds topics, consumer groups, delay logic and alerts to maintain. Handlers that are not idempotent need extra safeguards before any retry is safe. Ask any partner to break the quote down by these four drivers, and to state which items are one-time build work and which recur as the platform changes.

Can our own team build retry and dead letter topics without a partner?

Yes, many teams do, especially when they already run Kafka and own their consumer code. The work is mostly design and discipline: agreeing an error policy, classifying exceptions, writing the retry consumer, and testing redelivery and replay. The hard part is usually not the code but the decisions about ordering, poison message handling and who answers the alerts at night. If your team lacks Kafka operations experience or has no capacity for a sustained build, a fixed-scope engagement with a written deliverable list can cover the gap. Either way, review the design against the Apache Kafka documentation for the release you run.

How long does it take to get retry topics into production?

The duration depends on the number of consumer groups affected, the error classes you must handle, the testing you require, and how quickly your change process approves releases. Teams that already have a tested consumer framework usually spend most of their time on policy and verification, not on wiring topics. Teams starting from scratch should plan extra time for load testing and replay drills. No general duration applies to every platform, so ask for a timeline in the written scope with milestones for policy sign-off, staging validation and production rollout. Treat any figure that lacks a stated basis as a question to clarify.

What do we need to provide, and what do we keep ownership of?

You keep ownership of the error policy, because only your business can decide which failures are retried, which are parked, and how long a message may wait before someone acts. You must supply the message schemas, the list of consumer groups and their handlers, and the topic naming and retention rules your platform follows. You also need to grant access: broker credentials, ACL changes for the new topics and consumer groups, and access to logs and metrics. Confirm with your partner who approves production changes and who owns the dead letter queue after handover, so no topic is left without an operator.

Does Kafka itself provide dead letter topics?

No. Apache Kafka is a log and does not decide what a failed message means, so dead letter topics are a pattern your application or client framework implements. Kafka supplies the building blocks: topics, partitions, consumer groups and offsets that let you publish failed records to another topic and continue. Some client libraries and frameworks offer helpers for this, but you should confirm exactly what they do, including how they set headers and commit offsets. Because the pattern lives in your code, the error policy, headers and replay tooling must be designed and tested by your team or a partner.

How does replaying dead letters differ from normal consumption?

Normal consumption reads new records in order and commits progress as they succeed. Replay reads records that already failed, usually after a fix, and republishes them to the original or a designated topic so the handler can process them again. Replay must preserve the original key and headers, avoid duplicating records that already succeeded, and respect the ordering you chose for the flow. Handlers must be idempotent, or replay can cause double effects in downstream systems. Plan replay as an audited operation with a dry run, rate limits and a clear owner who can stop it.

What are the main risks if retries are built badly?

The common failures are blocked partitions from blocking retries, out-of-order processing when retries run in parallel with newer records, duplicate side effects when handlers are not idempotent, and dead letter topics that fill silently because nobody watches them. Each risk has a design answer: separate transient from permanent errors, decide per flow whether ordering matters, and alert on dead letter growth and age. Ask any partner to show how their design covers each of these, and to list what remains your responsibility after go-live.

How DeployOpen scopes and supports retry and dead letter design

DeployOpen scopes retry and dead letter work as a fixed-scope engagement, so the decisions that shape poison message handling are agreed in writing before any code changes. The written scope covers the error policy (which failures are retried, how many attempts each tier gets, when a record is parked and who may replay it), the topic layout (retry topics, their delay tiers, the dead letter topic and its naming and retention), the ACLs that let each consumer group read its retry tier, write to the dead letter topic and no more, and a test plan. Each of these is a choice with ordering and operational consequences, so the reader should confirm them with their own owners of the consuming services before the scope is signed.

Implementation happens inside the client's own infrastructure, on the clusters and in the repositories the client controls. Because the work is fixed in scope, the deliverables are the consumer and retry wiring, topic and ACL definitions, runbooks for inspecting and replaying dead letters, and test evidence. Testing injects faults on purpose: malformed payloads, downstream timeouts, deserialization failures, and consumer crashes mid-batch. The reader should check that the test plan verifies offsets after a rebalance, that a parked record appears exactly once in the dead letter topic, and that a replay does not overwrite newer state in the target system. Retry and dead letter behavior also depends on the Kafka and client versions in use, so the client should confirm version compatibility with the platform team early.

After handover, DeployOpen can support the platform as a retainer, covering reviews of new consumers against the agreed error policy, updates when the client upgrades Kafka or its client libraries, and re-testing of fault handling after changes. DeployOpen is a Chainscore Labs company and does not publish a price list, so a quote follows from the written scope. To get a comparable quote, ask for the consumers and topics in scope, the number of error policies, the test cases to be run, and the support hours and response expectations for any retainer. Costs rise with the number of consumer groups and the number of distinct failure classes that need their own handling.