Kafka Connect error handling and dead letter queues: when your pipeline needs to survive bad records
A sink connector with default settings stops its task when one record fails conversion, transformation or write, and every later record in that partition waits behind it. Consumer lag then grows until someone intervenes. Kafka Connect gives you three levers to change this. errors.tolerance decides whether a failing record stops the task (the default, none) or is skipped (all). errors.retry.timeout and errors.retry.delay.max.ms control how long Connect retries failures that may clear on their own, such as a briefly unavailable target database. errors.log.enable and errors.log.include.messages write error context to the Connect log. Decide per connector whether a skipped record is acceptable. For a payment or audit feed it usually is not, so the decision should come with a plan for capturing the record.
The dead letter queue is the mechanism for capturing it. Set errors.deadletterqueue.topic.name on a sink connector and failed records are written to that topic instead of being dropped. With errors.deadletterqueue.context.headers.enable set to true, each record carries headers describing the original topic, partition, offset and exception. Connect documents the dead letter queue for sink connectors, so do not plan on it for source connectors without checking the Kafka 4.3 documentation and the connector's own docs. Confirm which failure types the connector routes to the queue, including converter errors. The DLQ topic holds production data, so it needs its own ACLs, a retention period set by policy, and monitoring for growth.
A dead letter queue is a holding area, not a replay mechanism. Reprocessing means reading the DLQ, finding the root cause (a schema change, a missing column, a bad converter setting), fixing it, and writing the corrected records back through a controlled path. Plan who owns that triage and how long it may take, because the DLQ grows while nobody reads it. Test the setup before production: send a deliberately malformed record through a staging connector and confirm the record, its headers and the log entries appear as expected. Then alert on DLQ message rate. Expect effort from your platform and data teams for triage and fixes, and ask any implementation partner to state in the written scope which connectors, failure types and reprocessing steps are covered.
How Connect decides what counts as a failure
Kafka Connect does not treat every error the same way. The failure class decides which settings apply, where the record goes and what a dead letter queue can actually capture, so classify errors before you tune any of them.
Converter and deserialization errors
These occur when a record's bytes cannot be parsed by the configured key or value converter, often because a producer wrote data in a different format or schema than the connector expects. Connect handles them in the framework, before the connector sees the record, so the connector's own error settings may not apply. Check which converter each connector uses, whether a schema registry is reachable from workers, and whether the bad bytes can be preserved in the dead letter queue for inspection.
Transform and predicate errors
Single message transforms can fail on missing fields, unexpected types or values they were not written for. Because transforms run inside the Connect pipeline before the sink write, a failing transform is a framework-level error that can be routed differently from a write failure. Test each transform chain against sample records that include nulls, missing keys and schema changes, and confirm which record context your dead letter queue will receive.
Connector-side write errors
Errors raised when the target system rejects a record, such as a constraint violation, a mapping conflict or an authorization failure, originate inside the connector's own code. Whether these are retried, tolerated or surfaced as task failures depends on how each connector maps its exceptions. Some connectors fail the whole task on a single bad record, while others report per-record errors. Confirm this behavior in each connector's documentation or by testing it against your target before you rely on errors.tolerance.
External timeouts and unavailable targets
Timeouts to a database, search cluster or object store are often transient, but a long outage can look like a stream of failing records. Retry and timeout settings decide how long a task waits before it gives up, and a retry loop that is too long can stall partition progress and grow consumer lag. Distinguish retriable errors from fatal ones, and set alerts on task restarts so that a target outage does not quietly fill the dead letter queue with records that were fine.
Per-connector behavior to confirm
The four classes above are general, but connectors differ in which exceptions they treat as retriable, which they pass to the error reporter and which they convert into task failures. Build a short matrix for each connector in your pipeline that records the failure class, the observed behavior and the settings that changed it. Keep the matrix with the connector configuration so that upgrades and connector version changes are tested against it.
Which error handling settings fit which failure
This table compares the Kafka Connect error handling settings by what each one controls, when it fits and what it costs in trade-offs. Read each row against the failure you expect, such as transient network errors, malformed records or schema mismatches. The main takeaway is that tolerance without a dead letter queue or logging hides bad data, so pair the settings deliberately and confirm dead letter queue behavior against the Connect documentation for your version.
| Setting | What it controls and when it fits | Trade-off |
|---|---|---|
| errors.tolerance | Set to none to fail the task on the first error, or all to skip bad records and keep the connector running. Use all only with a dead letter queue or logging in place. | none stops the pipeline on any bad record; all keeps data flowing but can silently drop records if nothing captures them |
| errors.retry.timeout | Total time Connect keeps retrying a failed operation before giving up. Set it for transient faults such as brief broker or database unavailability. | A long timeout delays failure visibility; a short one routes recoverable errors to the error path too early |
| errors.retry.delay.max.ms | Upper bound on the backoff between retries, which spaces attempts during an outage. Lower values suit short blips, higher values suit slower recovery. | Higher values reduce load on a struggling dependency but extend lag before failures surface |
| errors.log.enable | Writes error context to the Connect log for each failed record. Enable it on every connector so failures are visible even without a dead letter queue. | Log volume rises with error rates, and logs may hold sensitive values if message content is also logged |
| errors.log.include.messages | Adds the record key, value and headers to the log entry. Turn it on only while debugging a specific failure. | Can expose personal or regulated data in logs and increase storage use |
| errors.deadletterqueue.topic.name | Names the topic that receives failed records from a sink connector. Leave it unset to skip the dead letter queue, and use one topic per connector or failure domain. | Records accumulate in another topic that needs retention, access control and an owner; it does not fix the underlying fault |
| errors.deadletterqueue.topic.replication.factor | Sets replicas for the dead letter queue topic, which should match the durability of your main topics. Confirm the default behavior and whether the topic is auto-created in your version. | Lower replication risks losing failed records during a broker failure; higher replication uses more storage |
| errors.deadletterqueue.context.headers.enable | Adds headers with the original topic, partition, offset and exception details to each dead letter record. Enable it when you plan to reprocess records. | Extra headers add a small amount of size per record and the headers may include error text that needs review |
Designing a dead letter queue topic that you can reprocess
A dead letter queue (DLQ) topic only helps if you can replay what it holds, so design it as a recovery input from the start. Each failed record should be written to a dedicated topic named for its origin, for example dlq.orders-sink for records that failed in one connector, so that owners can find and filter failures without reading every record. Set the topic's retention long enough to cover your investigation and fix cycle, and confirm that the retention window will not expire while a schema or sink fix is still in progress. Use the same replication factor and minimum in-sync replica settings as your source topics, because losing the DLQ means losing the only copy of records that your business has not yet processed. Restrict producers and consumers with ACLs (see the Kafka authorization documentation): Connect workers write to the DLQ, a small group of replay operators read from it, and no general application group receives access.
The context headers are what make replay possible. Kafka Connect can attach the error class and message, the original topic, partition and offset, and the connector name to each failed record, so keep those headers enabled and check them in a test before go-live. Note that the original key and value are preserved as bytes only if the converter and error handling settings keep them intact, so verify this with your converters. Plan for a DLQ that is not a substitute for monitoring: a growing DLQ is a symptom, and alerts on record counts, error rates and consumer lag should fire before anyone opens the topic. Treat a DLQ as a queue of work with an owner and a deadline, not as storage for problems that nobody reviews.
Reprocessing needs three things decided before the first failure. First, a replay job that reads the DLQ, strips or interprets the error headers, and writes records back to the original topic or a corrected one, with its own logging and a way to stop midway. Second, a sink that is idempotent, meaning a repeated write of the same record produces the same result, because replay can deliver a record that already succeeded once. Third, a rule for schema problems: if the failure came from a schema or data contract mismatch, fix the schema or transform first, then replay, because replaying into the same error only refills the DLQ. Confirm the replay approach with your sink's documented behavior and your data owners before you rely on it.
Setting up error handling and a DLQ on an existing connector
This sequence adds error tolerance, retries, error logging and a dead letter queue to a running sink connector, tested with a malformed record before any production change. The result is a documented path from a bad record to a verified DLQ entry and a repeatable reprocessing runbook.
Confirm connector type and Connect version
Record the connector class, its version, the Kafka Connect runtime version and whether it runs as a sink or source. Confirm the Connect workers run Java 17, which the Kafka upgrade notes require for Connect from Kafka 4.0 onward. Check the connector documentation for whether it honors errors.tolerance and dead letter queue settings, since dead letter queue routing applies to sink connectors and the set of supported options varies by connector. Done when you have a written record of the connector class, versions and supported error options, and the connector's behavior on this point is confirmed against its own documentation rather than assumed.
Create the DLQ topic with replication and ACLs
A platform engineer creates a dedicated dead letter queue topic that matches the source topic's durability requirements, including its replication factor and retention, which you set to meet your reprocessing window. Grant the connector's principal write access to the DLQ topic and grant read access only to the team that will reprocess records. In KRaft clusters, ACLs live in the metadata log, and resources without ACLs are limited to super users unless allow.everyone.if.no.acl.found is set to true, so confirm the connector can write before testing. Done when a describe command shows the topic with the expected settings and an authorization test from the connector's principal succeeds.
Set error tolerance and retries in a test connector
Create a copy of the connector in a non-production Connect cluster or under a separate name, then set errors.tolerance to all, add the dead letter queue topic name, and configure the retry and timeout settings the connector supports, such as errors.retry.timeout and errors.retry.delay.max.ms. Keep the original production connector unchanged until the test passes. Document each value and the reason for it, because a permissive setting can hide a schema problem for longer than intended. Done when the Connect REST API returns the test connector's config with the new keys and the connector reports a running state.
Enable error logging with context
Turn on error logging so each failed record writes its topic, partition, offset and exception to the Connect log, and decide whether the record value is logged or only its metadata. Logging full payloads can expose personal or regulated data in logs, so set this deliberately and confirm with your data owner. Route Connect logs to your central log store and confirm the entries are searchable by connector name. Done when a deliberately failing record produces a log line with the expected source coordinates and exception, and the same line appears in the central store.
Inject a malformed test record
Produce a small set of controlled records to the test source topic: one valid record, one with a field type that violates the sink's schema, and one that is not valid for the configured converter. Use a dedicated test topic or a test key prefix so the records cannot reach production tables. Label these records as test data in their payload so they are easy to find later. Done when the valid record lands in the target system, and the malformed records are absent from the target and present in the dead letter queue topic.
Verify DLQ headers and logs
Consume the DLQ topic with a read-only consumer and inspect the headers Connect attaches to each failed record. These typically include the original topic, partition, offset, the error class and message, and the stage where the failure occurred. Confirm that each header matches the corresponding log entry and that the original record bytes are intact so the record can be replayed without loss. Check that the connector's consumer group offset advanced past the failed record, so the pipeline did not stall. Done when every test failure has one DLQ entry with complete headers that match its log line, and the connector remains in a running state.
Apply to production and document the reprocessing runbook
Apply the reviewed settings to the production connector through your normal change process, then monitor the DLQ topic's record count and the connector's task status for a period agreed with your operations team. Write a runbook that covers who receives DLQ alerts, how to triage each error class, how to fix the cause, how to republish corrected records to the source or a reprocessing topic, and when to delete or archive DLQ records. Have someone other than the author follow the runbook against a test record. Done when the runbook is stored with the connector configuration, a reviewer has completed a dry run, and DLQ alerts reach the named on-call owner.
Securing the DLQ and error logs
A dead letter queue collects the records that failed in your pipeline, and those records often carry the same personal or regulated data as the main topics. The cards below cover who can read failed data, how the DLQ topic and its logs are protected, and how long failed records should live.
Who can read failed records
A DLQ record holds the original key and value of a message that Connect could not write, so it can contain customer names, account numbers or health details. Treat the DLQ topic as a copy of the source data, not as a debugging artifact. Before you enable routing, list the teams that will consume the DLQ, decide which of them may see raw payloads, and confirm whether your data classification already covers this topic.
ACLs on the DLQ topic
In KRaft clusters the StandardAuthorizer stores ACLs in the metadata log, and resources without ACLs are limited to super users unless allow.everyone.if.no.acl.found is set to true. Grant read access on the DLQ topic only to the reprocessing service account and named operators, and keep write access to the Connect worker principal. Check that the Connect worker's own principal can write to the DLQ topic and that no broad consumer group can read it by default.
Encryption in transit
Kafka encrypts traffic between clients, brokers and tools with SSL/TLS, but security is optional, so a cluster can run unsecured or mix secured and unsecured listeners. Confirm that the Connect workers, the producer path into the DLQ and any reprocessing tool all connect over TLS, and that no plaintext listener carries DLQ traffic. Authentication with SSL or one of the documented SASL mechanisms should back the same listeners so that every DLQ reader is identified.
Redacting error logs
Connect's error logging can include the record contents and the exception text, depending on the logging and error context settings you choose. Log files are often shipped to a central log platform with wider access than the topic itself. Decide whether error logs should carry only topic, partition, offset and exception class, and test that a sample failure with a sensitive field does not reach the logs in plain text.
Retention of failed data
A DLQ topic keeps records for its configured retention, which is often longer than teams expect because nobody removes failed records while they wait for a fix. Set an explicit retention for the DLQ topic that matches your data-handling policy, and document who approves longer holds. Include the DLQ and its backups in your deletion process, so that a request to erase a person's data also reaches failed records.
Questions teams ask before routing failures to a DLQ
What drives the effort of setting up dead letter queues for Kafka Connect?
The main drivers are the number of connectors, the record formats they handle, the amount of reprocessing tooling you need, and the monitoring you expect. Each connector has its own error modes, so a team running three sinks with JSON records and one Avro source has a different job than a team running fifteen connectors across several formats. Routing failures to a DLQ topic is the smaller part of the work. The larger part is deciding which errors are retried, which are routed, which should stop the connector, and how an operator replays a record after the cause is fixed. Ask any partner to break the quote down by connector, format and reprocessing path.
Does a DLQ cost anything beyond the Kafka cluster itself?
Apache Kafka is licensed under the Apache License 2.0 and the project charges nothing for the software, so the costs sit in infrastructure and labor. The DLQ topic consumes broker storage and retention capacity like any other topic, and the cost grows with the failure rate and the retention you choose. Other recurring costs include the monitoring stack, the reprocessing service or scripts, and the people who triage failures each week. One-time costs are design, build and testing of error policies. Ask for these as separate line items so you can see what recurs after the engagement ends.
What does the client team need to provide before the work starts?
The team needs a clear list of connectors in scope, their versions, the source and sink systems, and the record schemas or converters in use. It also needs access to a non-production Kafka Connect cluster, a way to produce sample bad records, and the names of the people who will own failed records after go-live. Security approvals for new topics and ACLs usually take the longest, so start those early. If you use a Kafka version in the 4.x line, confirm it against the release listings on the official downloads page before you plan the build.
Can our own engineers build DLQ handling without an outside partner?
Yes, a team that already runs Kafka Connect in production can build this with its own engineers. The configuration keys are documented and the pattern is well understood. The harder part is the operating model: someone must own the DLQ topic, agree on retry limits, handle poison records, and rehearse replay. Teams without a clear owner or without time to test error paths often find the DLQ fills up and nobody reads it. An outside partner helps most when you need the design reviewed, the reprocessing tool built, or the runbooks tested under realistic failure conditions.
How long does a DLQ implementation take?
Duration depends on scope, which is set by the number of connectors, the number of error categories you want to handle differently, and whether you need a replay tool or only documented manual steps. A single connector with one format and simple replay will take far less time than a shared platform serving many teams. DeployOpen works on fixed-scope engagements, so the timeline is agreed in writing before work starts, based on the scope in that agreement. Ask for the timeline to be tied to named deliverables and test milestones.
Which errors should go to the DLQ and which should stop the connector?
Errors caused by a bad record, such as a schema mismatch or a malformed payload, are usually good candidates for the DLQ, because other records can continue. Errors caused by the target system being unavailable, bad credentials or missing permissions usually should not be sent to the DLQ, because routing every record would bury the real fault and create replay work. Setting errors.tolerance to all without a retry policy and alerting is a common mistake. Test each connector with deliberate failures and confirm that the connector halts when it should.
When should we bring in an outside partner for DLQ work?
An outside partner is most useful when the pipeline carries regulated or sensitive data, when failed records must be reprocessed with an audit trail, or when several teams share one Connect cluster and need consistent error policies. A partner can also review the design before production, build the replay tooling, and test the runbooks. Ask any partner what they will deliver, who will own it after handover, and how the work will be tested. DeployOpen plans, implements, hardens and documents self-hosted open-source platforms in infrastructure you control, and offers ongoing support as a retainer.
What should we confirm about licensing and the connector plugins before we build?
Confirm the license of every connector plugin that handles failed records, because some connectors come under licenses other than Apache 2.0. Check the version of Kafka and Kafka Connect you run, since the supported releases and Java requirements differ between versions. The Kafka downloads page lists the supported releases and the upgrade notes list runtime requirements. Verify both with your own environment and with the connector vendor before you commit to a design that depends on a specific plugin behavior.
How DeployOpen scopes and delivers Connect error handling
DeployOpen plans, builds and documents Kafka Connect error handling and dead letter queue routing inside your own infrastructure, on self-hosted Apache Kafka. Each engagement starts from a written scope that defines which connectors, error policies and failure paths are covered before any configuration changes.
How we scope
- We inventory each connector, its converters and its sink or source targets to decide where errors.tolerance applies.
- We agree in writing which record classes go to a dead letter queue topic and which must halt the connector.
- We define the headers, retention and access rules for the error topic before any connector is changed.
How we build
- We set errors.retry.timeout, errors.retry.delay.max.ms and error logging per connector, then test each setting against sample bad records.
- We configure dead letter queue topics with context headers and verify that the original payload and failure reason are recoverable.
- We build a reprocessing path that replays corrected records and confirms the target has no duplicates.
How we hand over
- We deliver runbooks for triaging error topics, reprocessing records and deciding when to stop a connector.
- We document each error policy, its owner and the alerts that signal a rising failure rate on each pipeline.
- We walk your operators through a failure drill on a non-production cluster before handover is accepted.
How we support
- Engagements are fixed-scope, and ongoing support, upgrades and continuous testing are available as a retainer.
- Retainer work can cover new connectors, changes to error policies and retesting after Kafka or Connect upgrades.
- Ask any quote to name which connectors, topics and environments are in scope and who runs the reprocessing.
