Preventing duplicate messages with the Kafka idempotent producer

This is the simplest case and the one where the producer does the most work. A single producer session writes to a single partition, and the broker tracks each batch by producer ID and sequence number. When a request times out and the client resends it, the broker recognizes the sequence it already committed and does not append the batch again. Within that session, the order in which records were sent is the order in which they are appended, provided in-flight requests stay within the limit the client documents for idempotent producers.

• Duplicates from retries: With enable.idempotence set to true, retried batches from this session are written once.
• Ordering: Records keep their send order inside the partition while the in-flight limit for idempotent producers is respected. Confirm that limit for your client library version.
• Acknowledgement settings: Idempotence requires acks set to all, so a write is confirmed by the in-sync replicas before the producer treats it as sent.
• Application responsibility: Handling sends that the producer reports as failed after its delivery timeout, and deciding whether to resend the same key and payload.

Check the producer settings in the running service, since a value overridden in a config file or environment variable changes the result.

Idempotence is tracked per partition, so the duplicate protection applies to each partition the producer writes. The broker keeps sequence state for every partition in the session, which means a retried batch is still written once no matter how many partitions the producer uses. Ordering, however, only exists within a partition. Records sent to different partitions have no defined order relative to each other, and a consumer reading several partitions can see them in any interleaving.

• Per-partition sequences: Each partition has its own sequence state, so retries do not duplicate records in any partition.
• No order across partitions: Consumers cannot rely on the relative timing of writes to different partitions.
• Key-based placement: Records with the same key go to the same partition while the partition count stays the same. Changing the partition count changes that mapping, so plan partition counts before production traffic starts.
• Atomicity across partitions: Idempotence does not make a set of writes to several partitions all-or-nothing. Transactions are needed when consumers must see all of the writes or none of them.

Ask which topics need consistency across partitions and whether the design depends on it before relying on idempotence alone.

Each producer instance has its own session and its own producer ID, unless the application sets a transactional.id. The broker cannot connect a retry from one instance with a resend from another. If two replicas of a service both process the same input and both write the resulting event, Kafka stores both events. Idempotence protects against one instance repeating its own sends, and it does not coordinate work between instances.

• Separate deduplication state: Two instances do not share sequence numbers, so the broker treats their writes as independent.
• No order between instances: Records written by different instances to the same partition have no guaranteed relative order.
• Duplicate business events: Duplicate processing of the same input needs an application-level identifier, such as an event ID carried in the record and checked by consumers or a store.
• Overlapping input assignment: When instances join, leave, or restart, the same input can be handled again by another instance, which produces a second write.

Confirm how the deployment starts, stops and scales instances, and whether the same input can reach two of them at the same time.

Idempotence state lives in the producer session. A restarted process normally gets a new producer ID, so the broker cannot match a request sent before the crash with one sent after it. If the process crashed after the broker wrote a batch but before the acknowledgement arrived, the restarted producer may write that batch again. Setting transactional.id gives the producer a stable identity across restarts and lets the broker fence older instances with the same identity. That identity is the basis for transactional writes, and it changes how the application must call the producer.

• New session, new identity: Without transactional.id, deduplication does not span a crash.
• Uncertain last send: The application cannot know whether the last send before a crash succeeded, so it must treat that send as uncertain.
• Transactional identity: A stable transactional.id enables fencing and requires the application to begin, commit and abort transactions.
• Replay markers: Store the source position, such as the input offset, with the output so a replay can be recognized.

Test a forced kill during sends in a staging environment and count the records that arrive downstream.

The idempotent producer only governs what reaches a topic. A consumer that processes a record, writes to a database or calls an external service, and then fails before it commits its offset will receive the record again after restart. Duplicate records that the producer removed never reach the topic, but reprocessing of records that already reached it still happens. Duplicate side effects are therefore a consumer design question, and the producer setting does not answer it.

• Redelivery after failure: A consumer restarts from its last committed offset and rereads anything processed after that point.
• External side effects: Emails, payments, and database writes run again unless the handler checks whether the work is already done.
• Commit timing: When offsets are committed sets the size of the reprocessing window.
• Idempotent handlers: Use unique event identifiers, upserts keyed by business identifiers, or transactions that include the consumed offsets.

Ask each consumer team what it writes outside Kafka and whether that write can be repeated safely.