Replace a failed KRaft controller disk in Kafka

Replacing a failed KRaft controller disk without corrupting cluster metadata

Do not format and start a controller whose metadata directory was lost until a majority of controllers hold the committed data. That is the rule that keeps the cluster's metadata intact. In a KRaft cluster, the controllers form a Raft quorum that stores the cluster's metadata log: topics, partitions, leadership, configuration, ACLs and broker registrations. Each controller keeps its own copy of that log in its metadata directory. When a controller's disk fails, the other controllers still hold the committed records, and the quorum keeps running as long as a majority is healthy. The risk is a rushed rebuild: a freshly formatted controller starts with no history, and if it is allowed to vote or lead before it catches up, the cluster can lose committed metadata.

Use this procedure when a controller host lost its disk or metadata directory and the quorum is still running. A broker-only disk failure is a different job. Brokers hold partition log data, not the metadata quorum, so replacing a broker disk is handled through partition replica recovery rather than controller reformatting. Confirm which role the failed host played before you start. In combined mode, a host that runs both roles may also have a broker's log directories on the same server, so check which directories survived. Your first concrete step is to run kafka-metadata-quorum.sh describe --replication against a healthy controller and record each voter's log end offset and last fetch timestamp. Those numbers tell you whether the remaining controllers are a caught-up majority, and they are the evidence you will need in the change record.

Only reformat the replacement controller with the cluster ID once the quorum output confirms that the remaining voters are caught up. Use the cluster ID already recorded for the cluster, not a new one. The --ignore-formatted flag on kafka-storage.sh format applies only in one narrow case: when other log directories survived on a combined-mode host and the metadata directory was the only one lost. Do not use it to bypass an error on a directory that holds no valid metadata, and do not use it as a shortcut when you are unsure what survived. Ask your platform team or vendor to confirm the exact behavior for your Kafka version before you run it in production, and rehearse the full sequence on a non-production cluster. Plan for a change window, a second person to check the quorum output before and after the rejoin, and a rollback point if the replacement does not catch up.

Which recovery path fits your controller layout

This table compares three controller layouts a team may face after losing a metadata directory, with the action each one calls for and the condition that must hold first. Read the rows from the top down and stop at the first layout that matches your node. The main takeaway is that waiting for a caught-up majority comes before any reformatting, and --ignore-formatted is only for combined nodes whose other log directories survived.

Layout after the disk loss	Action	Condition that must hold first
Dedicated controller, metadata directory gone	Do not format and start yet. Wait, then reformat with the cluster ID and rejoin.	A majority of the other controllers has the committed metadata, confirmed with kafka-metadata-quorum.sh describe --replication.
Combined broker and controller, only the metadata log directory lost	Wait for a caught-up majority, then reformat the lost metadata directory with the cluster ID.	The controller quorum is healthy and the node's broker log directories are not being reformatted.
Combined broker and controller, other log directories survived	Use --ignore-formatted only for the lost metadata directory, not for a full reformat.	The surviving log directories still hold valid data, and the quorum shows a caught-up majority before the node restarts.
Any layout with no caught-up majority	Stop. Do not format any controller or broker directory.	Escalate and investigate quorum health first, since formatting an empty controller can lose committed metadata.
Controller whose cluster ID is unknown	Confirm the cluster ID from a healthy node before formatting.	The cluster ID is read from a surviving member, not guessed or copied from a different cluster.

Recovery sequence for a lost controller metadata directory

This sequence brings a controller with a lost metadata directory back into the quorum without risking the committed cluster metadata. Each step ends with a check that confirms the controller is safe to format and, finally, caught up.

Confirm the failed disk and stop the controller
The platform team confirms on the affected host that the metadata log directory is missing, unreadable or corrupted, and that no other log directory on that node holds surviving controller metadata. Then they stop the Kafka process on that node so it cannot rejoin the quorum with partial state. Done means the process is no longer running and the directory check output is saved in the change record. If any metadata data survives in a directory you intend to reuse, stop here and read the formatting notes in the later steps before going further.
Confirm the remaining controllers have a leader
Run kafka-metadata-quorum.sh against a bootstrap server or controller on a healthy node and confirm the quorum reports a leader. Check that the count of remaining voters still forms a majority of the configured controller set, since a majority must hold committed data before you proceed. Done means the describe output names one leader and the remaining voters are listed without errors. If no leader is reported, do not format anything yet, because the quorum needs to recover first.
Read replication lag and fetch timestamps
Run kafka-metadata-quorum.sh describe --replication from a healthy node and record the log end offset, lag and last fetch timestamp for each voter. A healthy voter shows small lag and a recent last fetch time relative to the leader. Save the output so you can compare it after recovery. Done means you have a written baseline of which voters are current, and you know which voter the recovered node will need to catch up against.
Wait until a majority holds committed data
Hold the procedure until the replication output shows that a majority of the voters, including the leader, are caught up to the committed offset. The operator watches the same describe output at intervals and does not format the new directory while any majority member is still lagging. Done means the majority has the committed data and the lag values have stopped growing. If the majority cannot reach that state, escalate before any reformat, because the lost node may hold the only copy of recent metadata.
Retrieve the cluster ID from a healthy node
Read the cluster ID from the meta.properties file on a healthy controller or broker in the same cluster, or from the cluster configuration record your team keeps. Confirm that the value matches on the healthy nodes you can reach, so the new directory joins the right cluster. Done means you hold a single cluster ID value that you have checked against more than one healthy node. Do not generate a new cluster ID, because a different ID means the node will not join the existing quorum.
Format the new metadata directory with the cluster ID
On the replacement disk, run kafka-storage.sh format with the verified cluster ID, the node's existing configuration file and the controller role settings. Do not add --ignore-formatted unless the directory survived in a combined-mode layout where other log directories still hold data, which is the narrow case the Kafka documentation describes. Done means the command exits cleanly and the new meta.properties file shows the expected cluster ID and node ID. Confirm the node ID matches the one in the controller configuration before starting the process.
Start the controller and confirm it is caught up
Start the controller process and run kafka-metadata-quorum.sh describe --replication again from a healthy node. Watch until the restarted voter appears in the output with its fetch timestamp updating and its lag returning to a value close to the leader. Done means the node is listed as a voter, its last fetch is recent, and its lag is in line with the other current voters. Keep the before and after outputs in the change record, and check the broker logs for errors about cluster ID mismatch or log truncation.

Why formatting a controller before the majority is caught up is risky

Do not format and start a replacement controller until a majority of the remaining controllers hold the committed metadata. A controller that comes up with an empty metadata directory has no log of its own to vote from. Kafka's KRaft quorum commits metadata once a majority of controllers has it, so a fresh controller that joins before the others have caught up can add a voter with nothing behind it. The risk is not that the quorum fails the moment you start the node. The risk is that you have changed the membership and data picture of the quorum while the committed state is still being copied to the survivors, and you have no reliable way to tell from the outside whether that copy finished.

The evidence to rely on is the output of kafka-metadata-quorum.sh with the describe --replication option, run against the healthy controllers. Read the log end offset and lag for each voter, and confirm that the surviving majority is at or near the leader's log end offset before you reformat anything. Also check the last fetch timestamp for each voter. A voter that has not fetched recently is not caught up, even if its last reported offset looks close. Do not treat a controller process that is running, listening on its ports or showing as registered as proof of safety. Those signals say the process is alive, not that the committed metadata is present on enough nodes.

Confirm the exact behavior for your version before you write a runbook step around it. Read the recovery and controller sections of the Kafka 4.3 documentation, and compare them with the release you actually run, since field names and output columns can differ between releases. Check whether your cluster runs controllers only or combined with brokers, because that changes which log directories count as surviving. Record the quorum output before and after the change, and keep it with the change ticket. If the output is ambiguous or the surviving majority is not clearly caught up, stop and get a second reviewer before proceeding, since the documentation does not give a safe way to force the replacement to catch up.

How the procedure differs for combined and dedicated controllers

On a dedicated controller, the node runs only the controller role and holds no partition data. When its disk fails, what is lost is the KRaft metadata log and snapshots for the cluster, which the remaining controllers still hold. The node has no broker log directories to protect, so the recovery question is almost entirely about metadata: whether a majority of controllers already has the committed data before this node is formatted and started again.

Survival check: confirm that the metadata directory on the failed node is the only directory that was lost, and that no other log directory on the node held data for this role.
Quorum state: run kafka-metadata-quorum.sh describe --replication and wait until a majority of the other controllers are caught up before reformatting.
Cluster ID: reformat only with the cluster ID taken from a healthy controller, so the node cannot join a different cluster.
--ignore-formatted: this flag does not apply here, because no other directory survived on a dedicated controller that would need to be preserved.
Broker side: brokers are unaffected, but confirm that metadata updates such as partition leadership continue normally while the controller is out.

Before you start, confirm the procedure against the controller guidance for your version and check the disk layout in the hardware and OS guidance.

What to check before and after the controller rejoins

Before the replacement controller is allowed to rejoin, confirm that the cluster can prove the committed metadata exists elsewhere. Run kafka-metadata-quorum.sh describe --replication against a live controller and record the lag of each voter against the leader, the last fetch timestamp, and which controller is the active leader. The replacement is only safe to format once a majority of controllers show a caught-up state. After it starts, repeat the same command and check that the rebuilt controller's lag falls toward zero and its last fetch timestamp keeps advancing. Then confirm the quorum has a single leader and that the active controller count is stable, not flapping. Watch the broker and client error rates on the same window, including request errors, metadata refresh failures and producer or consumer retries. A rate that rises or stays unsettled after the controller rejoins means the incident is not closed, even if the process is running.

Record the evidence while the work is happening, not afterward from memory. Capture the cluster ID before and after the change, the controller node IDs and their node.id values, the directory that was reformatted, and the exact kafka-storage.sh format invocation with its flags. Save the output of each quorum and metadata check with a timestamp and the name of the person who ran it. Keep the change ticket, the chosen recovery path, and any flag that was considered and rejected, including whether --ignore-formatted was judged applicable and why. This record is what an internal reviewer or external auditor will ask for when they trace how a controller was rebuilt. Confirm with your change process which storage location and retention period apply to these logs.

None of this should be worked out for the first time during an outage. Put the checks, the expected output shape, and the decision points into the runbook before an incident, and name the person who may approve reformatting. Rehearse the full sequence on a non-production cluster that has the same Kafka version, controller count, and metadata layout as production, and time how long a caught-up majority takes to appear in your environment. Use the rehearsal to confirm which fields your monitoring already exposes and which need a dashboard or script. Consult the Apache Kafka hardware and OS documentation when you check that the rebuilt node's disk, file system and resource settings match the rest of the controller quorum. Treat any step you cannot reproduce in rehearsal as a gap to close before the next real failure.

Security and operational risks during controller recovery

A controller recovery touches metadata, identity and access at the same moment the quorum has less room for error. The cards below pair each risk with the control that contains it and the check that proves the control worked. Work through them before the format command runs, not after the controller rejoins.

Stale or copied metadata

Copying a metadata directory from another controller can bring back an old view of the cluster, and a node that rejoins with stale state can mislead the quorum. Before formatting, confirm with kafka-metadata-quorum.sh describe --replication that a majority of controllers hold the committed data. Do not restore a directory from a backup or another node unless your runbook has been reviewed against the Kafka upgrade and recovery notes.

Cluster ID and node identity

The replacement controller must be formatted with the existing cluster ID and its own node ID, as set in the server configuration. A mismatch can create a second cluster or a node that the quorum does not recognize. Record the cluster ID and the node.id before the disk is wiped, store them with the change ticket, and compare them against the running controllers before starting the node.

TLS keys and SASL credentials

A copied data or configuration directory can carry private keys, keystores and SASL secrets to a host that was never meant to hold them. Limit what you copy to what the format step requires, keep key material in your secrets store, and rotate any credential whose file was left on a retired disk. Confirm file permissions on the new host before the node starts.

Who can format and start

The format and start commands change cluster state, so only named operators should run them. Restrict shell access to the controller host, require a second person to approve the format step, and keep the approval in the change record. Check that the account running the commands is the service account that owns the metadata directory, not a shared administrator login.

Logging every recovery action

A recovery involves a sequence of decisions: which node failed, which majority was checked, which lag values were accepted and why the format step ran. Capture the output of each quorum check, the exact commands run, the operator and the time, and store the log outside the affected host. An auditor or a post-incident review will ask for this record, and it is hard to rebuild from memory.

Reduced redundancy during recovery

While one controller is out, the quorum tolerates fewer further failures. Plan the window in advance, avoid other controller or broker maintenance during it, and confirm that the remaining controllers are healthy and caught up before you begin. Ask your team to define in writing who can declare the recovery complete and what triggers a rollback to the change owner.

Common questions before a controller disk recovery

How much effort and cost does a KRaft controller disk recovery add?

The work itself is usually small in steps, but the cost comes from the checks that must happen before any reformatting. Effort is driven by how many controllers remain healthy, whether the cluster is in production, how much metadata the surviving controllers hold, and how many people must approve and watch each step. Cost components are internal engineering time, any change window and rollback planning, and, if you hire help, the fixed-scope fee agreed in writing. Ask any partner for a quote built from a written scope that lists the controllers affected, the environments covered, the verification steps and the handover material. DeployOpen does not publish prices, so the quote follows from that scope.

How long does it take to recover a failed controller?

Duration depends less on the reformat command and more on how long the surviving controllers take to confirm they hold the committed metadata. The main inputs are the replication lag reported by the quorum tool, the size of the metadata log, the network path between controllers, and whether the team has a tested runbook and a maintenance window. A cluster with a healthy majority and small lag will move through the checks quickly, while a cluster with a lagging majority needs more waiting and more review. No fixed duration is stated here, so ask your team or a partner to estimate it from your own lag measurements on a test cluster first.

What must our team provide before the work starts?

Most delays come from missing inputs rather than from the recovery itself. Plan to provide:

Access to the cluster ID and the metadata directory layout on each controller
Current runbooks, broker and controller configuration files, and the Kafka version in use
A maintenance window with named approvers and a rollback decision owner
A test cluster that matches production's controller count and version, for rehearsing the steps
Monitoring access to controller lag and fetch timestamps during the work

Confirm the exact access model and change process with your own security and operations leads before scheduling.

Can an internal team recover a controller disk without outside help?

Yes, if the team already runs KRaft in production, has rehearsed controller replacement, and can read the quorum status output with confidence. The procedure relies on checks, not on a single command, so the risk lies in skipping a check under pressure. Internal teams usually need a rehearsed runbook, a second engineer to verify each step, and a clear rule for when to stop. Teams without recent KRaft operations experience, or those running a regulated workload where the metadata cannot be wrong, often benefit from a review of the runbook before the window.

When should we bring in an outside partner for this recovery?

Bring in a partner when the cluster is business critical, when the controller that failed was part of a small quorum, or when your team has not handled a KRaft metadata recovery before. A partner can also help write the runbook, rehearse it on a test cluster and document the evidence your change process needs. DeployOpen plans, implements, hardens, backs up, upgrades, documents and supports self-hosted open-source platforms inside your own infrastructure, and its engagements are fixed-scope with the scope agreed in writing before work starts.

When is --ignore-formatted appropriate, and when is it not?

Use --ignore-formatted only in combined mode when another log directory on the same node survived and already holds valid metadata, so the node can start without reformatting the directory that was lost. It is not a shortcut for a controller whose metadata directory is gone. Kafka's documentation says a controller that lost its metadata directory should not be formatted and started until a majority of controllers have the committed data, checked with kafka-metadata-quorum.sh describe --replication. Confirm your exact configuration and log directory layout with the Apache Kafka documentation for your version before using the flag.

What evidence should we keep after a controller disk recovery?

Keep the quorum replication output from before and after the change, the cluster ID used, the exact reformat command and its log output, the time the controller rejoined, and the approvals for each step. Auditors and internal reviewers usually ask how you knew the majority held committed data before the reformat. Recording those checks in the change ticket gives your team a repeatable record for the next incident. Ask your compliance lead which records your change management policy requires.

Limits and details to confirm for your Kafka version and layout

A controller disk replacement depends on details that differ between Kafka releases and between clusters. Check each item below against the documentation for your exact version or against a test run in a non-production environment before you format anything. The answers decide which steps apply to your cluster.

Kafka version and support status

Confirm the exact Kafka version running on every controller and broker, then check it against the supported releases on the downloads page. A controller on an unsupported line may need an upgrade before recovery is attempted, and recovery steps written for one release can differ in another. Record the version in the runbook so the person executing the replacement works from the right documentation.

Java runtime for the release

Check the Java version that your installed release requires before you start the replacement node. A controller that starts on the wrong runtime can fail during metadata load, and that failure looks like a recovery problem. Confirm the runtime on the replacement host, not only on the hosts that survived, because a rebuilt machine often has a different default JDK.

Controller layout and quorum size

Determine whether the failed node was a dedicated controller or a combined broker and controller, and how many voters the quorum has. The majority you must wait for depends on that count. Confirm the controller IDs and the voter list in the configuration, because a mismatched node ID can give the replacement a different identity in the quorum.

SSL or SASL listeners on the replacement

Identify every listener used for controller traffic and client connections, including whether it uses SSL or one of the SASL mechanisms. The replacement needs matching keystores, truststores, JAAS settings and credentials before it joins the quorum. Confirm these with the security configuration on a surviving controller, since a missing certificate or principal will block the node from rejoining.

Directory layout in the configuration

Read the log.dirs and metadata log directory settings on a surviving node and compare them with the replacement host. Mount points, ownership and file permissions must match, or the node may write metadata to an unexpected path. Confirm which directories held data before the failure, since that determines whether any formatting flag applies.

Flags supported by your tool version

Run the storage tool's help output on the replacement host and confirm each flag the procedure uses, including whether --ignore-formatted exists in that build. Flag availability and behavior can change between releases, so the documentation for your version is the authority. Test the sequence in a staging cluster when you can, and keep the exact output in your change record.

How DeployOpen works on this

DeployOpen approaches controller disk replacement as a planned recovery, not a quick reformat. We first confirm which controllers hold committed metadata, then agree the recovery steps in writing before anything is formatted on your cluster. The work runs inside your own infrastructure, and the result is a documented runbook your team can repeat.

How we scope

Confirm the cluster's controller count, combined or dedicated mode, and which metadata directories were lost
Record the cluster ID, Kafka version and current quorum state before any change is proposed
Agree the written scope covering replacement steps, verification checks, and documentation before work starts

How we build

Check that a majority of controllers has caught up and that fetch timestamps show healthy replication
Run the reformat only after the quorum output confirms the replacement controller will rejoin cleanly
Test the procedure on a non-production copy of your cluster where your environment allows it

How we hand over

Deliver a runbook with the exact commands, expected outputs, and decision points for each recovery path
Document the conditions under which --ignore-formatted is permitted, and the ones where it is not
Walk your operators through the quorum checks so they can confirm recovery without relying on us

How we support

Offer ongoing support and upgrades as a retainer if your team wants help keeping the runbook current
Review the runbook after each Kafka release you adopt, since upgrade notes may change recovery details
Confirm with your team which monitoring and alerting will catch a controller falling behind before a failure

Replacing a failed KRaft controller disk