Debugging KRaft metadata in Kafka

When does KRaft metadata debugging become your problem?

When a KRaft cluster shows controller elections that never settle, metadata changes that reach some brokers but not others, or brokers that fail to register, the work is finding out which layer is at fault: the controller quorum, the metadata log those controllers maintain, or a broker's view of that log. Those three layers fail in different ways and need different evidence, so the first job is to separate them before changing any configuration. In KRaft mode there is no external coordination service to blame. Kafka's own controllers hold the cluster metadata, including topics, partitions, leaders, configurations, ACLs and broker registrations, and they record every change as an ordered event in a log that brokers replicate and apply.

Controllers elect a leader among themselves, accept metadata changes from brokers and admin clients, and commit them to the log only after a majority of the voters has stored them. Brokers do not vote, but they fetch metadata from the active controller and apply it locally. A stalled election, a lagging voter or a broker stuck on an old offset will each look like "Kafka is slow" from the outside, which is why this work usually lands with the platform or site reliability team that runs the cluster, often with a Kafka specialist for the deeper log analysis. The application teams see the symptom first, but they rarely have access to controller logs or the metadata directories where the answer sits.

Before the team takes on this work, confirm three things. Someone must own access to controller hosts and their metadata directories, because the tools in this topic read files there and some must run on a stopped broker or controller. The team must agree on which KRaft commands and log levels are safe to use on a live production cluster, since changing log levels and reading quorum state are routine but reformatting storage is not. And the team needs a record of the cluster ID, voter set and Kafka release in use, so that a symptom can be matched against the Apache Kafka downloads page for supported releases and known fixes. Without those three, most debugging stalls at the access request, not at the cluster.

Which tool to reach for first in a KRaft incident

Start here when the controllers disagree about who leads, elections keep recurring, or clients report that metadata operations fail. The kafka-metadata-quorum.sh tool with the describe --status option reports the current leader, the leader epoch and the high watermark as the quorum sees them. Use it to confirm whether a leader exists at all before looking at individual nodes.

Leader and epoch: A missing leader or an epoch that keeps climbing points to repeated elections rather than a single failed node.
High watermark: Compare it across checks to see whether committed metadata is still advancing.
What it cannot answer: It does not explain why an election happened. Controller logs around the same timestamps hold that evidence.
Order: Run describe --status first, then describe --replication for voter detail, then read the controller logs.
Network and disks: Packet loss, DNS problems and slow disks show up in logs, not in this output.

Check the field names and their meanings against the documentation for the exact Kafka release you run, because output can change between versions.

Use this path when one broker seems to act on stale topic or partition state, such as leadership that does not match what clients see, or a broker that stays behind after a restart. The describe --replication option of kafka-metadata-quorum.sh lists the metadata replicas the quorum knows about and how far each one's log end offset trails the leader. It shows whether the broker is catching up or has stopped.

Replica offsets: A broker whose offset stays flat while the leader moves forward is not fetching.
Lag trend: Sample the output twice to tell a slow catch-up from a stalled fetch.
What it cannot answer: It does not show why the broker is slow. Heap pressure, disk latency and garbage collection pauses are found in broker logs and host metrics.
Order: Check the broker's own logs for fetch errors, then run describe --replication, then inspect network paths between the broker and the controllers.
Restart decision: Do not restart the broker until you have recorded its offset and the leader's offset.

Confirm how observer or broker entries appear in the output for your release before you act on a lag figure.

Use this path when a controller refuses to load metadata, when a snapshot looks wrong, or when the contents of a metadata file do not match what the cluster reports. The kafka-dump-log.sh tool decodes metadata log segments and snapshots into readable records, and the metadata shell lets you browse the state those files represent. Both read files without changing them, so they are safe to use during an incident once you work on copies.

Copy first: Copy the metadata directory of the affected controller to a separate location before running any tool on it.
Decoded records: Use kafka-dump-log.sh with the cluster metadata decoder option to see individual records and their types.
Metadata shell: Browse the snapshot state to compare topics, partitions and configuration with a healthy controller.
What it cannot answer: These tools cannot repair a file or prove that corruption caused the failure. They show what is present, not why it got there.
Order: Preserve the files, decode the log, inspect the snapshot in the shell, then compare with another controller's state.

Decoder options and shell commands differ between releases, so check the flags against the documentation for your version.

Use this path when the existing logs do not explain the problem and you need more detail from a controller that is still running. Kafka supports dynamic log levels, so you can raise the level of selected loggers on a controller with kafka-configs.sh against the controller endpoint, without a restart. The change applies to the running process and is lost when the process restarts, which is useful for a short, targeted capture.

Narrow scope: Raise one logger, such as the one covering the metadata or raft code, rather than the whole controller.
Record the baseline: Note the current level so you can return to it exactly.
What it cannot answer: Logs cannot reveal events that happened before the change, and debug output adds load.
Order: Check current levels, raise one logger for a bounded window, reproduce or wait for the event, then revert.
Ownership: Record who changed the level, when and why, so the change is not left behind.

Confirm the entity type and logger names for your release in the documentation before running the command on a production controller.

Which KRaft debugging tool fits which question

This table compares the four tools used to investigate KRaft controller and metadata problems: quorum status, log decoding, the metadata shell and dynamic log levels. Read each row against the question you are trying to answer, and check the access and write behavior before running a tool on a production controller.

Tool	What it reads	Changes anything?	Typical question it answers
kafka-metadata-quorum status and replication	Live quorum state from the controllers, including leader, epoch and replica progress	No, read-only query	Is there a leader, and is every voter keeping up with the log end?
kafka-metadata-quorum replication	Live replica fetch positions and lag for each voter and observer	No, read-only query	Which controller is falling behind, and by how much?
kafka-dump-log	Metadata log segments and snapshot files on a controller's local disk	No, reads files directly	What records were written to the metadata log around a given offset?
Metadata shell	Metadata image rebuilt from the log and snapshots on disk	No, opens a read-only view of metadata	What does the cluster metadata say about a topic, broker or ACL right now?
Dynamic log levels on controllers	Logger configuration of the running controller process	Yes, changes controller logging at runtime	Which controller component is logging the errors behind the symptom?
Access needed	Quorum tools need client access to a controller endpoint, file tools need shell access to the controller host	Confirm your RBAC or host policy before running	Who may run each tool on production, and from where?

How to set up a safe KRaft debugging session

This sequence lets an operator confirm what is running, capture the cluster's current metadata health and keep a reversible record before any inspection or change. Each step ends with a check that decides whether the session continues.

Confirm the cluster ID and running release
Start by recording the cluster ID and the exact Kafka release on every controller and broker, using the version information each node reports and the cluster ID stored in the metadata directory's meta.properties file. Check that all nodes agree on the cluster ID and that the release is one listed as supported on the Kafka downloads page, since your investigation steps and tool behavior depend on the version. If any node reports a different cluster ID or an unexpected release, stop here and confirm the topology with the team that deployed the cluster before continuing.
Capture quorum status and replication output
Before changing anything, run kafka-metadata-quorum against a controller to save the quorum status and the replication output, and store both in a dated incident file. Confirm that a leader is reported and that the voter and observer lists match your inventory of controllers. If the command cannot reach a quorum leader or the output shows missing voters, stop, preserve the output as it is, and escalate before running any further tooling against the cluster.
Record the current dynamic log levels
Use the broker and controller configuration tooling to read the current dynamic log levels on each controller and save them with the node identifiers, so you know the exact values to restore later. Confirm that the saved file lists every controller you plan to touch and that the levels match what the operations team expects from the last known configuration. If a controller does not return its log levels, do not change any level on it, and record that node as unverified in the session notes.
Copy metadata directories before inspection
Stop nothing and copy each controller's metadata log directory to a separate, access-controlled location before running kafka-dump-log, the metadata shell or any other reader against it. Compare file counts and checksums between the source and the copy, and confirm that the copy contains the segments and snapshot files you expect. If the checksums differ or any segment is missing from the copy, halt the session and do not point inspection tools at either location until the copy is repeated.
Run inspection tools against the copies only
Point kafka-dump-log and the metadata shell at the copied directories, never the live data path, and write their output to the session folder. Confirm that the tools open the copied segments and snapshots without error and that the record counts or snapshot offsets line up with the quorum output captured earlier. If a tool reports a corrupt segment or an offset that is ahead of the live quorum, stop and treat it as a finding to escalate rather than a reason to modify the live metadata.
Change log levels one node at a time
Raise the log level on a single controller only when a specific question requires more detail, and keep the change reversible by noting the value you set alongside the saved original. Re-check quorum status and replication output after the change to confirm the controller still participates and the leader is unchanged. If the quorum shows a new election, lag growth or a voter leaving the set, revert that controller to its saved level immediately and stop.
Restore levels and close the session record
When the investigation question is answered or the checks fail, restore every changed log level to the value saved at the start and confirm each controller reports the original value. Run quorum status once more and compare it with the captured baseline, then file the session record with the commands, outputs, copied directory checksums and any unverified nodes. The session is complete when the quorum matches the baseline and no node still runs a changed log level.

What metadata access exposes and how to control it

Metadata files, logs and controller debug output describe how a Kafka cluster is secured and run. The cards below cover what that material reveals, how raised log levels and copies of metadata directories widen exposure, and which trust boundaries debugging access should respect.

ACL definitions live in metadata

On a KRaft cluster that uses the StandardAuthorizer, ACLs are stored in the cluster metadata log. Anyone who can read that log, or a snapshot of it, can see which principals may read, write or administer which resources. Treat access to metadata files as access to the authorization model itself, and confirm who can read the controller data directories on every controller host before you start a debugging session.

Topic and principal names in output

Dumped records and metadata shell output include topic names, principal names, client identifiers and configuration values. Names alone can reveal business processes, customers or internal systems even when message payloads are never read. Before you share a dump in a ticket or chat, review it for names you would not expose outside the operations team and remove what is not needed for the investigation.

Raised log levels create volume

Dynamic log levels on controllers can produce very large volumes of output in a short time, and that output may include the same identifiers and configuration described above. Record the original level before you change it, set the narrowest logger that answers your question, and restore the original level as soon as the check is finished. Confirm that your log pipeline's retention applies to this debug output as well.

Controller access follows admin boundaries

Running metadata tools against controllers needs the same authorization as other administrative work on the cluster. Use named operator accounts rather than shared credentials, limit shell access to the people on the on-call or platform roster, and record who ran which command and when. Confirm with your security team which hosts count as controllers and whether they sit in the same trust zone as the brokers.

Copies follow the original retention rules

A copied metadata directory, snapshot or exported dump carries the same information as the original, so it needs the same classification, storage location, encryption and deletion schedule. Create copies only when a fix or analysis requires one, store them in the access-controlled location you already use for cluster data, and set a deletion date when the ticket closes. Ask your compliance owner which retention schedule applies to operational logs in your environment.

Symptoms that point to the metadata layer rather than the data path

The first decision is whether the fault sits in the metadata layer, which the KRaft controllers own, or in the data path, which the brokers and partition replicas own. Produce and consume traffic that keeps flowing while admin commands fail is the clearest early signal that the controller quorum is struggling rather than the partitions themselves. Brokers that register late or never appear in the cluster view, topic creations or partition count changes that take a long time to become visible, and controller leadership that changes again and again are the other symptoms that point upward. Before you act on any of them, confirm which one applies by running kafka-metadata-quorum against the controllers and comparing the leader, the voters and the observer lag with what the brokers report. Do not assume a single cause. A slow registration can come from a broker that cannot reach the controller listener, from a controller that is itself lagging, or from a broker whose metadata directory was formatted with the wrong cluster ID.

Repeated controller failovers deserve more care than a single election, because the same visible symptom can have very different roots. Frequent leader changes may come from network instability between controllers, from disk or garbage collection pauses that stall the Raft log, or from a controller that cannot keep up with the metadata log and falls out of the voter set. Admin failures such as topic creation timing out, while a producer on an existing topic keeps succeeding, show that clients are still reaching brokers that hold valid leadership for their partitions. That pattern narrows the search to controller reachability, controller load and the metadata log itself. Keep a timeline of when each symptom started, because a failover that lines up with a deployment, a certificate rotation or a disk alert is far easier to explain than one that appears without a trigger.

Treat each symptom as a hypothesis that needs a tool-backed check before you change any configuration. Record the output of the quorum status and the controller logs for the same window, and note which brokers and controllers are involved. If the evidence shows that the data path is healthy and only metadata operations are failing, plan the investigation around the controllers and the metadata log rather than partition reassignment or client tuning. Confirm the versions and listener settings of every controller and broker against your change records, since mixed configurations are a frequent source of these symptoms. The sections that follow show how to read the quorum output, decode the metadata log and adjust controller log levels to test each hypothesis.

What to confirm for your release and environment

Metadata debugging depends on the exact Kafka release, the tools shipped in it and how your network and change process reach the controllers. Check these items before you rely on a procedure, because the output format and available options can differ between versions.

Release and tool output format

The supported releases listed on the downloads page are 4.3.1, 4.2.2 and 4.1.2, and earlier versions are archived. Confirm which release your controllers run, then run each tool's help output on that exact build, because flags and output columns for kafka-metadata-quorum and kafka-dump-log can change between releases. Save a sample of the output from a healthy cluster so you have a baseline to compare against during an incident.

Metadata shell and log level support

Confirm that your build includes the metadata shell and the dynamic log level controls you plan to use on controllers. Do not assume a command is present because a neighboring version has it. Test each command in a non-production environment with the same distribution, Java runtime and packaging (tarball, container image or distribution package) that runs in production.

Controller access path

Controllers listen on their own endpoints and are often isolated from client networks. Map which hosts, jump servers or bastions can reach the controller listener and the admin client properties needed for quorum commands. Confirm that the TLS and SASL settings for tool access match your security configuration, and that any access requires an approved change record and an audited session.

Handling of copied metadata

Metadata logs and snapshots can contain topic names, configuration values, principal names and ACL entries. Before copying files off a controller, confirm the approved storage location, encryption requirements, access list and deletion date with your data owner and security team. Record who exported what, when, and from which node, so the copy can be traced later.

Live debugging change approval

Changing log levels or running read-only quorum checks on a production controller may still need a change or incident ticket under your process. Confirm with your change board whether emergency access is allowed, who must approve it, and how a level change is reverted. Write the rollback step, including the original level, into the ticket before you begin.

Release-specific upgrade notes

Release-specific behavior, removed options and compatibility changes are listed in the upgrade notes for your target version. Read the notes for every release between your current version and the one you plan to debug or upgrade to, because a procedure that works on one release may not apply on another. Treat the notes as the authority over any older example you find elsewhere.

Questions teams ask before debugging KRaft metadata

How much effort does a KRaft metadata investigation take?

The effort depends mostly on how many controllers are involved, how far the metadata has drifted, and whether the team already has a record of the cluster's normal state. A quorum that is healthy but noisy usually takes less time than one that has lost its majority or has diverged replicas. Effort also grows with the number of metadata snapshots and log segments that must be decoded, and with the number of rounds of log-level changes needed to isolate a cause. Ask any provider to break the estimate into evidence collection, analysis, fix design and verification, and to state which of those phases depend on access you have not yet granted.

Can our in-house team debug KRaft metadata without outside help?

Yes, a team with Kafka operations experience can work through the core tools themselves, including the quorum status and replication output, the dump of metadata log and snapshot records, and the metadata shell. The harder part is judging what the records mean for a live cluster and deciding whether a repair is safe, because a wrong step can widen an outage. Teams often bring in outside help when the controllers are in an inconsistent state, when no one has run a recovery before, or when the investigation has to finish inside a change window. Your own team can still own the decisions while an outside engineer supports the analysis.

What access and files does a metadata investigation need from us?

Expect to provide read access to the controller and broker hosts, the Kafka version in use, the relevant server configuration with secrets removed, and output from the quorum status and replication commands. Copies of metadata directories or snapshot files are often needed, and they should be taken from a stopped or copied state, never by working on the live directory. Also provide logs covering the period before the symptom, any recent change records, and the name of the person who can approve a change window. Agree on where copies are stored and who can read them, because metadata can contain topic names, principals and configuration values that fall under your data handling rules.

How long does an investigation usually run, and what makes it longer?

Duration is driven by evidence quality, cluster size and the number of unknowns, not by a fixed rule. Investigations run longer when logs have rotated, when snapshots are missing, when the cause spans a network partition and a disk fault, or when every fix needs a rolling restart that can only happen in a window. Access delays and waiting for approvals also add time that does not show up in the engineering effort. Ask for the timeline in phases with the dependency for each phase, and ask how the plan changes if a key artifact is unavailable.

What changes if our controllers run in Kubernetes instead of on servers?

The metadata tools and the record formats stay the same, but access and evidence collection change. Controllers in Kubernetes keep their metadata on persistent volumes, so you need to know which pod, volume and node held each copy, and whether the volume was reattached or recreated during the incident. Pod restarts can rotate container logs quickly, so capture logs from the node or your logging stack before they are lost. Running the tools from inside a pod may need extra permissions, and any change to a StatefulSet or volume claim needs its own review. Confirm with your platform team how the operator or manifests manage controller storage before the work starts.

What should a written quote for this work include?

A written quote should state the scope in terms of the clusters, controllers, Kafka version and symptoms covered, and what is excluded, such as fixing application code or changing network design. It should list the deliverables, for example a findings report, a recommended change plan, a verification checklist and a record of commands run. Ask which costs are one-time and which recur, such as follow-up support or a retainer, and whether any work is billed separately when a change window is missed or a new finding extends the scope. Confirm who performs the changes, who approves them, what access is needed, and how the work is documented for your audit trail.

Do we need the upgrade or release details before we start?

Yes. The Kafka version on each controller, the version history of the cluster, and whether it was originally ZooKeeper-based or KRaft-based all affect which records appear and how they should be read. Check the Kafka upgrade notes for your target and current releases before the investigation, because the supported upgrade paths for KRaft clusters are documented by the project and changes between releases can affect how metadata is interpreted. Share any recent upgrade or configuration change with the team, since a change made weeks earlier is often the root cause.

How DeployOpen works on this

DeployOpen treats KRaft metadata debugging as a bounded piece of work on your own Kafka cluster, starting from the symptoms you already see and the evidence you can collect. Each engagement follows a written scope, with the tools, log access and change controls agreed before anyone touches a controller.

How we scope

Written scope names the clusters, controller nodes, Kafka versions and metadata symptoms covered by the work.
Scope records which evidence you will provide, such as quorum output, controller logs and metadata directory copies.
Scope states who may change log levels on controllers and how changes are approved and reverted.

How we build

Investigation runs in a fixed order: quorum status, replication lag, controller logs, then decoded metadata records.
Each finding is checked against a copy of the metadata directory so production controllers stay untouched.
Recovery steps are tested in a non-production environment that mirrors your controller layout before any change.

How we hand over

Deliverables include a written record of commands run, outputs captured and the conclusion reached for each symptom.
Runbook steps cover quorum checks, log level changes, and the conditions under which your team should escalate.
Knowledge transfer sessions walk your operators through the tools so they can repeat the analysis without us.

How we support

Ongoing support, upgrades and continuous testing are available as retainers after the fixed-scope work ends.
Ask any quote to separate the one-time investigation from recurring retainer hours and any version upgrade work.
Confirm with your team which Kafka release you run, since supported releases and upgrade paths change over time.