Unclean leader election in Apache Kafka

Yes, but only for a narrow set of topics where keeping the partition writable matters more than keeping every acknowledged record. The Kafka design documentation describes this as a trade-off: waiting for an in-sync replica preserves data but leaves the partition unavailable until one returns. When only lagging replicas survive, electing one of them lets producers and consumers resume, but records the old leader accepted and the new leader never received are lost, and consumers may see offsets move backward. Teams usually justify the setting for telemetry or cache-like streams where gaps are tolerable and the source can be replayed. It is rarely justified for payments, ledgers or events that trigger irreversible actions. Write the justification down before any change, and name the person who accepted the loss risk.

Usually no. The default is false, and a cluster-wide enablement exposes every topic to the loss trade-off, including topics that carry data your teams cannot regenerate. A safer pattern is to keep the default off and enable the setting only on named topics whose owners have accepted the risk in writing. Start by inventorying topics by criticality and marking which ones can tolerate gaps and replay from a source system. Cost and effort rise with the number of overrides, because each one needs a record, an owner, a monitoring rule and a review date. A cluster with a few well-understood topics is far easier to govern than a shared platform with many teams. Before you write the runbook, confirm in the topic configuration reference how your Kafka version scopes the setting, since broker-level defaults and per-topic values interact.

In KRaft mode, a dynamic change does not take effect immediately. It waits for the periodic unclean election thread, which runs every 5 minutes by default, so expect a delay of up to that interval. Plan the change window with that wait in mind, and verify the effective value on each affected topic after the change. Do not assume the change applied because the command returned without error. Monitor partition leadership and offline partition indicators during the window so the team can tell a normal wait from a real outage. If you need an election sooner, the next question covers the tooling option. Record the change time, the verification output and the approver in your change ticket so the audit trail matches what happened.

Yes. The Kafka tooling can trigger an election with the unclean option through kafka-leader-election.sh, which lets an operator act on specific partitions without changing the topic setting. The configuration can stay false while one partition is recovered under a deliberate, logged action. This is still an unclean election, so the same data loss risk applies to that partition at that moment. Use it only after the team confirms that no in-sync replica will return within the acceptable outage window, and capture the partition list, the replica state and the approving operator. Confirm the exact flags for your Kafka version in the 4.3 documentation, and rehearse the command on a non-production cluster first. Leaving the setting enabled afterward is not required for the recovery.

The effort splits into a one-time design phase and a recurring operating phase. One-time work includes inventorying topics, classifying them, agreeing risk acceptance with data owners, writing the forced-election runbook and rehearsing it. Recurring work includes reviewing overrides, checking that new topics inherit the intended default, and repeating the rehearsal after Kafka upgrades. The main drivers are the number of topics that need a decision, how many consumers must recover correctly after lost records (through idempotent processing, replay or reconciliation jobs), how many teams share the cluster, and how strict the audit evidence must be. No duration is given here. Ask any partner for an estimate tied to a written scope, and size your own topic inventory first, because it determines most of the rest.

Yes, if the team already runs Kafka with on-call coverage, change control and monitoring for partition availability. The skill to build is judgment about data loss as well as command syntax. Assign one owner for the policy, one owner for each cluster's runbook, and a named approver from the data owner side. Internal teams often underestimate the consumer side, where application teams must agree how to handle gaps and rewound offsets, and the evidence work that continues after the first change. Test the runbook on a staging cluster that mirrors topic settings, and confirm that someone other than the runbook author can complete an election drill. A partner can help with design and testing, but policy decisions and risk acceptance stay with your organization.

A partner needs your topic inventory with owners, replication settings and minimum in-sync replica values, plus the criticality of each data flow. It also needs current monitoring and alerting coverage, the consumer applications that read each topic and whether they handle duplicates and gaps safely, and any audit requirements that touch availability or data integrity. Provide access to a non-production cluster on the same Kafka version, the change approval chain and the on-call model. With these inputs, the written scope can name the topics in scope, the runbooks to write, the tests to run and the sign-off points. DeployOpen agrees scope, deliverables and timelines in writing before work starts, so gaps in these inputs usually appear later as scope changes. Expect the quote to rise with the number of topics and consumer applications that need review.