Kafka consumer offset commit strategies

Kafka consumer offset commit strategies: when your team needs to choose

A consumer commits offsets to record how far it has read in each partition, and the timing of that commit decides what happens after a crash or a rebalance. If the offset is committed before the work finishes, a restarted consumer skips records it never processed. If the offset is committed after the work, a restarted consumer reprocesses records it already handled. Your team has to pick which failure it can tolerate, because Kafka does not pick for you. Apache Kafka is released under the Apache-2.0 license, so the commit strategy is part of the application you build and operate on infrastructure you control, not a vendor default you inherit.

Teams often start with the client defaults and never revisit them. Automatic commits run on a timer and are tied to poll calls, so an offset can be recorded for records that a slow handler has not finished. That is a reasonable starting point for logging or metrics where a small loss is acceptable, but it is a poor fit for payments, ledger entries or notifications. Manual commits move the decision into your code, where you control whether the commit follows a successful write to the downstream system. Before you choose, list each consumer group, the side effects its handler produces and whether those effects can be repeated safely. Ask the team that owns each downstream system whether it accepts duplicates, and write the answer down.

The commit choice also depends on how the consumer behaves under rebalances and on what happens when no committed offset exists yet. The auto.offset.reset setting decides whether a new group starts at the earliest or latest available offset, which matters after retention has removed old data or when a group is created for an existing topic. Confirm the value your client library uses by default and the value your operations runbook expects, because a mismatch can silently skip history or replay months of events. Plan a test that stops a consumer mid-batch, restarts it and counts the results. That test shows the real behavior of your chosen strategy more clearly than any table of guarantees.

Commit options and when each fits

This table compares the four commit approaches and the offset reset setting by delivery guarantee, failure window and fit. Read each row as the trade-off a consumer accepts when it commits progress before or after processing. The main takeaway is that the commit point, not the commit API, decides whether a crash causes replayed or skipped records.

Option	Typical delivery guarantee	Failure window	Where it fits
enable.auto.commit (default true)	At-least-once or at-most-once, depending on when processing finishes relative to the periodic commit	Up to the auto-commit interval of records may be reprocessed after a crash, or skipped if the commit runs before processing completes	Low-stakes consumers where duplicates or gaps are tolerable, and prototypes; confirm the interval and processing order in your client version
commitSync (manual, synchronous)	At-least-once when called after processing each batch	Records processed since the last successful commit are replayed after a restart; the call blocks until the broker responds	Workloads where each batch must be recorded as done before the next poll and throughput is secondary to clarity
commitAsync (manual, asynchronous)	At-least-once, with weaker ordering guarantees on failed commits	Commits can complete out of order or fail silently unless the callback is handled, so replay window may grow	High-throughput consumers that accept more replay and pair async commits with a synchronous commit on shutdown
Commit inside a transaction	Exactly-once within Kafka when output and offsets commit atomically with read_committed consumers	Aborted transactions roll back both output and offsets; limits apply to external systems that the transaction does not cover	Read-process-write pipelines that write back to Kafka and need no duplicate output inside the cluster
auto.offset.reset (earliest, latest, none)	Not a commit mode; it decides the starting position when no committed offset exists for the group	earliest can replay full retained history, latest can skip records produced before the group started, none raises an error	Set earliest for new groups that must process history, latest for live-only consumers; confirm the value your team needs for each group

At-least-once versus at-most-once: choosing the failure mode

Every consumer that commits offsets picks a failure mode, whether it intends to or not. The cards below cover the decisions that follow from that choice: when duplicates are acceptable, when loss is acceptable, where exactly-once stops, and how poison messages fit in.

Default to at-least-once

Committing after processing gives at-least-once delivery: if a consumer crashes after writing results but before its commit lands, the same records are read again. The handler must therefore tolerate duplicates, usually by keying writes on a message identifier or using upserts. Check that every side effect, such as an email, payment call or row insert, has a duplicate-safe form before choosing this mode for it.

Accept loss for telemetry only

At-most-once processing commits before the work runs, so a crash after the commit drops the records that were in flight. This suits loss-tolerant telemetry such as sampled metrics, where a gap is acceptable and duplicates would distort counts. Confirm with the data owner in writing that losing a window of records does not break alerting or billing before choosing it.

Exactly-once stops at the system boundary

Kafka transactions can make consume, process and produce steps atomic across Kafka topics. They do not make a write to a database, search index or HTTP API atomic with the offset commit. For external sinks, design an idempotent write keyed on topic, partition and offset, or store offsets in the same transaction as the result. See the sibling pages on idempotent producers and transactions for the mechanics.

Keep idempotent producers on the output path

When a consumer produces results back to Kafka, retries on the producer side can create duplicates that are separate from consumer redelivery. Enabling the idempotent producer removes that source of duplicates within a producer session. Check that producer configuration has not silently disabled idempotence, because the consumer-side failure mode is then harder to reason about.

Isolate poison messages

A record that always fails will block its partition if the consumer retries it indefinitely. Decide in advance how many attempts a record gets, where it goes after that (a retry topic or dead letter topic), and who reviews it. Log the topic, partition and offset of every parked record so the failure can be replayed after the bug is fixed.

Test redelivery before production

Failure modes only show up under failure. Test a crash between processing and commit, a rebalance during a batch, and a restart from a committed offset, then confirm the sink holds one result per input. Record the test results as evidence for the design review, since the handler's behavior under redelivery is the control that matters most.

How auto.offset.reset changes behavior for new groups

When a consumer group has never committed an offset for a partition, the broker has no position to resume from, so the client applies auto.offset.reset. With earliest, the consumer starts at the earliest offset still available in the partition. That is the start of retained history, which may be later than the first record ever written. With latest, the consumer starts at the end of the log and only sees records produced after its first assignment. The setting decides whether a new service backfills existing data or ignores it.

Earliest reads history: a new analytics group replays everything retention still holds, which can mean a large first load.
Latest skips history: a new alerting group processes only events produced after it starts, so anything written during deployment is never seen by that group.
Timing matters: under earliest, records produced between group creation and the first poll are included. Under latest, they are excluded.
Per partition: the reset applies to each partition that lacks a committed offset, so a group can start some partitions from history and others from the end if their states differ.

For each group, confirm whether the intended starting point is history or only new events, and record that choice in the service configuration.

Brokers keep committed offsets for a bounded period after a group becomes inactive, and an operator can delete a group explicitly. Once those committed offsets are gone, the broker treats the group as new, and the reset policy applies again even though the application has run before. A service that returns after a long outage can therefore jump to the start or the end of the log without raising an error. The application may not notice the change until downstream totals disagree.

Long outage: a consumer that stays down past the broker's offset retention period returns with no committed offsets.
Deleted group: an operator who deletes a group to clear a stuck state causes every partition to fall back to the policy.
Reused group ID: a service renamed under an old group.id may start from the policy rather than from its previous position, depending on what is still stored.
No exception: earliest and latest both proceed normally, so the change appears only as a gap or a duplicate burst in downstream data.

Compare the offset retention period on your brokers with the longest outage each consumer can tolerate, and alert when a group's state changes unexpectedly.

Retention deletes old log segments on its own schedule, independent of consumers. If a group's committed offset points to records that have since been deleted, the offset is out of range the next time the consumer polls. The reset policy then decides where the consumer goes. earliest moves it to the oldest surviving record. latest moves it to the end of the log. In both cases, the records between the old committed position and the new starting point are no longer in the topic and cannot be read from it.

Earliest after deletion: the consumer resumes at the oldest retained record, so everything deleted before that point is skipped without notice.
Latest after deletion: the consumer resumes at the end, so every retained record it had not yet read is skipped.
Lag as the trigger: a consumer that falls behind the retention window is the usual cause, which makes growing lag the early warning.
Recovery source: if the upstream system can reproduce the missing range, recovery depends on that system, because the topic cannot supply it.

For each topic, compare the retention window with the worst consumer lag you expect, and write down what the group does when its offset is out of range.

The reset policy only governs groups with no usable committed offset, so its effect is easy to underestimate in code review. Kafka also supports none for this setting, which causes the consumer to fail when no offset is found, forcing someone to decide how to start. For groups that process business-critical data, failing loudly is usually easier to recover from than a silent skip or replay. The exact default for each client library is documented by that library and should be checked for the version in use.

Set it in every consumer: declare auto.offset.reset in each consumer configuration so the value is visible in review and in version control.
Use `none` for critical groups: the exception stops the consumer, so a missing offset is handled through a runbook.
Separate first start from recovery: planned reprocessing should use explicit offset positioning, not a change to the reset policy, so the intent is recorded.
Test both paths: run each group against a test topic with deleted history and confirm where it resumes and what it logs.

Before relying on any default, confirm the value in the Kafka 4.3 consumer configuration reference and in the documentation for your client library.

Set up commit handling in your consumer

This sequence turns the commit choices made earlier into consumer configuration and code, then proves that restarts and rebalances resume from the right position. Each step ends with a check your team can run before the consumer goes into shared use.

Confirm the processing guarantee for each consumer
The consumer owner and the team that depends on the output agree, in writing, which guarantee applies: at-least-once, at-most-once, or exactly-once within Kafka. Write the decision next to the consumer group ID, along with the downstream system that receives the output and whether it can deduplicate. Exactly-once across an external system needs its own design, so record that as a separate item. The step is done when every consumer group in scope has a named guarantee and an owner who signed off on it.
Set enable.auto.commit deliberately
Set enable.auto.commit explicitly in every consumer configuration rather than relying on the client default, so the setting is visible in code review. Use false for any group that needs at-least-once or at-most-once behavior, and document the reason in the configuration file. If auto-commit stays on, record the commit interval and accept that offsets can be committed before processing finishes. The check is a grep of all consumer configurations for enable.auto.commit, with each hit matched to an entry in the guarantee record from the previous step.
Choose commit placement relative to processing
Decide where the commit call sits in the processing loop. For at-least-once, commit only after the output is written and acknowledged, so a crash replays records that were not finished. For at-most-once, commit before processing, accepting that a crash can skip records. Pick synchronous commits for batches where a failed commit must stop the loop, and asynchronous commits where throughput matters and a later successful commit covers earlier failures. The check is a code review of the loop against the chosen guarantee, confirming that the commit call follows the write it protects.
Configure auto.offset.reset for each group
Set auto.offset.reset for each consumer group based on what happens when the group has no committed offset or the offset is out of range. Use earliest for groups that must process history, such as a new analytics consumer, and latest for groups that only need new events, such as a live alerting consumer. Confirm the setting with the topic owner, because a new group on a long-retention topic can replay a large backlog. The check is a test run against a disposable group ID, where you confirm the consumer starts at the position your table predicts.
Add rebalance listeners that commit before revocation
Register a rebalance listener and, in the revocation callback, commit the offsets for the work already completed on the partitions being taken away. Make sure the commit happens before the partitions leave the consumer, and handle failures by logging them and stopping cleanly rather than continuing silently. Keep the callback short, because long work inside it delays the rebalance. The check is a unit test that triggers a revocation with pending processed records and confirms the committed offset matches the last completed record.
Verify with a forced restart
Run a test where the consumer processes a known set of records, then stop it abruptly during processing, without a clean shutdown. Restart it under the same group ID and compare the output with the expected result. For at-least-once, expect some replayed records and confirm the downstream system handles them; for at-most-once, confirm the records skipped match the design. The check passes when the observed behavior matches the guarantee recorded in step one, and the results are saved with the test.
Verify with a test rebalance
Start a second consumer instance in the same group while the first is processing, so partitions move between members. Confirm that the revoked instance commits its completed offsets and that the new owner resumes without a gap or an unexpected replay beyond the agreed guarantee. Record the offsets for each partition before and after the rebalance. The check is a comparison of those offset records, and it should be repeated whenever the consumer code, client version or group configuration changes.

Committing offsets inside a Kafka transaction

When a consumer reads from one Kafka topic, transforms the records and writes results to another topic, the safest commit point is inside the producer transaction. The application sends its consumed offsets with sendOffsetsToTransaction on the same transactional producer that writes the output records, so the output and the progress marker either both become visible or both are discarded. This removes the window where output has been written but the consumer group offset has not moved, which is the window that produces duplicates after a crash. The pattern only works when the producer is initialized with a transactional.id and the application commits with commitTransaction rather than relying on automatic commits.

Three settings decide whether the pattern holds. Each producer needs a stable, unique transactional.id per input partition set, because the broker uses it to fence zombie instances that a stalled process might still be running. Downstream consumers must read with isolation.level set to read_committed, otherwise they will see records from aborted transactions. Finally, the consumer that feeds the transaction should disable auto-commit (enable.auto.commit set to false) so that the broker does not record offsets outside the transaction. Confirm the exact fencing and client version requirements against the Kafka 4.3 security and transaction documentation for your release before you build on them, and test a forced consumer restart during a live transaction in a non-production environment.

The boundary matters as much as the mechanics. Transactional offset commits cover only Kafka-to-Kafka flows, where the output and the progress marker both live in the same cluster. If the handler also writes to a relational database, calls an HTTP API or sends email, those side effects sit outside the transaction and will repeat after a failure. For those steps, design the external write to be idempotent, for example by keying upserts on a message identifier, and treat Kafka's exactly-once guarantee as covering only the topic writes. Document this boundary in the design review so that teams do not assume end-to-end exactly-once across systems.

Security and operational considerations for consumer groups

Committed offsets are stored in the cluster, so they carry the same risk as the topics they point to. The cards below cover who may commit, how clients prove identity, and what to watch once commits start failing or lag grows.

Offsets are cluster data

Committed offsets live in the cluster's internal offsets topic, so anyone who can change a group's offsets can skip messages or replay them. Treat offset commits as a permission in their own right, separate from reading the topic. Review who can write offsets for each consumer group, including operators who reset offsets during incidents. Confirm that your access model covers group offset operations as well as topic reads and writes.

ACLs in the StandardAuthorizer

In KRaft clusters, the StandardAuthorizer stores ACLs in the cluster metadata log. Grant each consumer group its own read permission on the group resource, plus read on the topics it consumes. Keep grants narrow so one application cannot commit offsets for another team's group. Check ACLs in a non-production cluster before rollout, because a missing group grant typically surfaces as failed commits rather than a clear authorization error at startup.

Super user default for unprotected resources

Resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. A new consumer group without a grant may therefore fail to commit, and setting that key to true to fix it weakens every unprotected resource at once. Confirm the setting on each cluster, and keep the list of super users short and reviewed.

Client authentication with SASL or SSL

Kafka authenticates clients with SSL or SASL. The documented SASL mechanisms are GSSAPI (Kerberos), PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER. Pick the mechanism your identity system supports, and make sure consumers authenticate on the listener that carries offset commits. Ask your team how service credentials are issued, rotated and revoked for each consumer application.

Encrypting commit traffic

Security is optional in Kafka, so a cluster can run unsecured listeners next to secured ones. Offset commits carry group identity and progress, so they should travel over TLS on every listener that consumers use. Audit listener configuration after any change, and confirm that no consumer falls back to a plaintext port during a migration.

Monitoring commit failures

A failed offset commit does not always stop a consumer, so failures can build up quietly. Alert on commit error rates reported by client metrics and on broker-side authorization denials. Record the commit mode each application uses, because asynchronous commits can fail without an exception in the calling thread. Confirm which metrics your client library exposes before you write the alert rules.

Alerting on consumer lag

Consumer lag shows how far a group's committed offsets trail the end of each partition. Rising lag can mean a stalled consumer, a failed commit, or a slow downstream system. Set lag alerts per group and per topic, and tie them to the processing guarantee the application claims. Ask your operations team which tool will report lag and who receives the alert.

Questions teams ask before choosing a commit strategy

What drives the effort and cost of a consumer offset redesign?

The main driver is how many consumer groups, topics and downstream side effects are involved, not the commit API itself. Cost splits into a one-time review and redesign, test work, and any change to consumer code that has to ship across several services. Effort rises when consumers write to databases, call external APIs or publish to other topics, because each side effect needs a decision about what happens if the commit and the side effect succeed or fail separately. Ask for a quote that lists consumer groups in scope, the languages and client libraries in use, and which failure modes must be tested. Because DeployOpen fixes scope in writing before work starts, the quote follows from that list rather than from a standard rate.

How long does a commit strategy review take?

Duration depends on the factors that set the workload: the number of consumer groups, how many use manual commits, how many write to external systems, and how much test coverage already exists. A review of a small set of consumers with simple processing is much shorter than one covering many services with database writes and transactional outputs. No fixed duration applies to every environment, so ask for a timeline in the written scope, with the inventory and test environment assumed as inputs. Timing also depends on whether a staging cluster with realistic traffic and lag is available, since offset behavior is hard to validate without it.

What do we need to provide before the work starts?

Three inputs make the biggest difference: a topic and consumer group inventory, written failure requirements, and a test environment. The inventory should list each group, its topics, partition counts, current enable.auto.commit and auto.offset.reset settings, and what each consumer does after reading a record. Failure requirements state whether duplicates are acceptable, whether any message may be lost, and how long a reprocessing window the business can tolerate. The test environment should be able to replay traffic, restart consumers during processing and force rebalances, so the chosen commit pattern is exercised under the failures that matter.

Can our internal team choose and implement the commit strategy alone?

Yes, many teams can, provided they have engineers who understand consumer group rebalancing, the difference between committed offsets and processed records, and how their database or downstream API behaves on retry. The hard part is rarely the configuration. It is verifying that a crash between processing and committing produces the outcome the business expects, and that the pattern holds across every service that shares a group. Teams with a single consumer and idempotent processing often handle it in-house. Teams with many groups, transactional sinks or compliance-relevant data may prefer an outside review to get a second set of tests and a written record of decisions.

How does the commit strategy affect Kafka upgrades?

Client and broker versions both matter. The Kafka upgrade notes say that rolling upgrades are supported from each release in the past year to the latest version, and Kafka 4.0 requires Java 11 or newer for clients and Kafka Streams applications, so consumer code may need a runtime change alongside a broker change. Commit behavior is shaped by the client library, so upgrading the client can change defaults or semantics. Before any upgrade, test consumer groups against the target client version, confirm that offsets committed by the old version are read correctly, and plan a rollback path. Check the upgrade notes for the exact release you target.

Which auto.offset.reset value should we use after a restart?

The value applies only when a group has no committed offset, or when a committed offset is no longer valid, such as after retention has deleted the records it points to. Choosing earliest reprocesses history and can cause large duplicate side effects. Choosing latest skips records that arrived while the group had no offset, which can lose data. Decide per group based on whether the business can tolerate reprocessing or gaps, and document the answer. Confirm the exact behavior for your client library version, since handling differs across clients and releases.

Do the security controls change when we switch commit patterns?

The commit pattern itself does not change authentication or encryption, but it changes who can alter consumer progress. Committing offsets requires that a consumer's principal has the right ACL on the consumer group, and misconfigured ACLs can cause commits to fail silently in retry loops. In KRaft clusters, the StandardAuthorizer stores ACLs in the metadata log and restricts resources without ACLs to super users unless allow.everyone.if.no.acl.found is set to true. Review group-level permissions as part of the redesign and confirm them in a staging environment before rollout.

What evidence should we keep to show how our consumers handle failures?

Keep the written failure requirements, the chosen commit pattern for each group with its reasoning, the test scenarios and their results, and the configuration as it was deployed. For regulated data, auditors usually want to see that the chosen behavior was decided deliberately and verified, not assumed. DeployOpen prepares systems, controls and evidence for compliance assessments, but independent auditors and certification bodies issue any audit report or attestation. Ask your assessor early which records they expect for data processing and integrity controls.

How DeployOpen scopes and delivers consumer offset design

DeployOpen treats offset commit strategy as a design decision that belongs in a written scope before any consumer code is changed. We plan the delivery semantics each consumer group needs, build and test the commit logic, and document it so your own engineers can run it after handover.

How we scope

The written scope lists each consumer group, its processing guarantee and the side effects its handler performs.
We record the commit mode, auto.offset.reset policy and retry behavior each group must follow.
Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts.

How we build

We implement the chosen commit pattern in your client libraries and test it against deliberate failures and restarts.
We verify offset-reset behavior by testing new groups, expired offsets and deliberate rebalances in a non-production cluster.
Where transactions apply, we test offset commits inside the transaction and confirm read_committed consumers see the expected records.

How we hand over

You receive runbooks describing each consumer group's commit strategy, failure behavior and recovery steps.
We document how to reset offsets safely and which operator actions can cause duplicates or skipped records.
Knowledge transfer sessions walk your engineers through the code, tests and dashboards for offset progress.

How we support

Ongoing support, upgrades and continuous testing are available as a retainer after handover.
Retainers can cover reviews of new consumer groups so commit strategy stays consistent across teams.
Confirm with us during scoping which client library versions and Kafka releases the support covers.

Kafka consumer offset commit strategies