When Kafka KRaft combined mode fits and when dedicated controllers do
Apache Kafka 4 runs only in KRaft mode, so the remaining decision for most teams is layout. Combined mode, where each server acts as both broker and controller, suits development and small setups. The Kafka KRaft operations docs do not recommend combined mode in critical deployment environments because the controller is less isolated from the rest of the system. The layout is set by process.roles: broker makes a server a broker, controller makes it a controller, and broker,controller makes it both. Changing that role later is possible but should be treated as a planned change to the cluster, not a config tweak, so decide before the first production rollout.
Combined mode is acceptable when the workload is modest, the environment is non-critical, and you can tolerate a broker problem also affecting controller availability. Confirm the controller quorum size in the docs before you build it: the docs say an admin typically selects 3 or 5 servers for the controller role, and a majority must stay alive, so a 3-controller quorum survives one failure and a 5-controller quorum survives two. Combined mode also needs enough resources on every node to run both roles under load, which you should test against your own throughput rather than assume.
Dedicated controllers are the default for production systems with uptime commitments, regulated data, or many brokers, because controller nodes stay small and isolated from broker load and maintenance. Also plan the quorum type. A dynamic quorum uses controller.quorum.bootstrap.servers and must not set controller.quorum.voters, while a static quorum lists every controller in controller.quorum.voters on all brokers and controllers. Check which one you run with kraft.version from kafka-features.sh, and review the Kafka KRaft operations docs before you commit to a layout.
How the two layouts differ in practice
In combined mode, each server sets process.roles to broker,controller, so the same process handles client traffic and also takes part in the KRaft controller quorum. This layout has the smallest node count, which suits development clusters and small test environments where one set of machines is enough. The trade-off, as the Kafka KRaft operations docs state, is that the controller is less isolated from the rest of the system. Heavy broker load, memory pressure, disk saturation or maintenance on a broker can also affect controller duties on the same host.
- Node count: every combined server counts toward the controller quorum, so the controller role and the broker role share the same machines.
- Isolation: broker work and metadata work compete for memory, disk and threads on one host.
- Documented fit: the docs describe combined servers as simpler for small setups such as development.
- Upgrades: restarting a server for broker maintenance also restarts a quorum member, so the rolling plan must protect quorum majority.
- Capacity planning: you size one tier for both duties, which makes growth decisions harder to separate.
Before you accept this layout, confirm with your team whether the cluster is a small or development setup or a critical production environment.
With dedicated controllers, controller servers set process.roles to controller and run no broker role, while broker servers set it to broker. The controller nodes hold cluster metadata and vote in the quorum, and the broker nodes handle partitions and client traffic. This separation is the layout the Kafka docs recommend for critical environments, because the controller is isolated from the rest of the system. The cost is more machines, since the controller tier is added on top of the broker tier.
- Node count: an admin typically selects 3 or 5 servers for the controller role, and those servers are added to the broker fleet.
- Isolation: metadata duties run on hosts that do not carry client traffic or partition storage.
- Capacity planning: controllers and brokers can be sized and scheduled separately, which keeps each tier's resource needs visible.
- Configuration: each role has its own configuration files, so changes to one tier need separate review.
- Hardware and hosting: controller nodes can use smaller footprints than brokers, but the choice must be confirmed against the Kafka docs for your version.
Ask your team to list every extra host, its owner and its patching window before you commit to a dedicated design, because those hosts recur in cost and effort.
A controller failure matters most for what it removes at the same time. In combined mode, one failed server removes a broker and a controller vote together, so the cluster loses data-path capacity and quorum margin in one event. In dedicated mode, a failed controller removes quorum margin, and a failed broker removes broker capacity, so each event can be reasoned about and handled separately. The KRaft operations docs describe majority rules: a 3-controller quorum survives one failure and a 5-controller quorum survives two. Once a majority is lost, the quorum cannot make metadata decisions until enough voters return.
- Single failure in combined mode: one host can remove a broker and a controller vote at once.
- Single failure in dedicated mode: a controller loss reduces quorum margin, and a broker loss is handled as a broker event.
- Two failures on a 3-controller quorum: the majority is lost, so metadata changes stop until recovery.
- Two failures on a 5-controller quorum: a majority remains, which is the margin the docs describe for the larger quorum.
- Broker behavior: confirm in the KRaft operations documentation for your version exactly how brokers behave during controller loss, and test that in staging.
Ask your team to write down which failure scenarios each recovery runbook covers, including controller replacement and broker replacement as separate procedures.
Both layouts need the same core KRaft skills, but the daily work differs. The quorum mode also shapes that work. A static quorum lists the IDs, hosts and ports of all controllers in controller.quorum.voters on every broker and controller. A dynamic quorum leaves controller.quorum.voters unset and uses controller.quorum.bootstrap.servers instead. The kraft.version value reported by kafka-features.sh shows which mode a cluster uses, so check it before planning membership changes.
- Combined layout: the team manages one tier, but every broker change touches a quorum member.
- Dedicated layout: the team manages two tiers with separate patch, monitor and capacity routines.
- Static quorum: voter lists must be updated on every node when membership changes, which adds coordination work.
- Dynamic quorum: bootstrap settings are simpler to reason about, but the team must still understand the quorum state during changes.
- Monitoring: alerts for quorum health and controller availability should exist in both layouts before go-live.
Before choosing, ask your team which quorum mode the design uses, who performs membership changes and how they will be tested.
Combined vs dedicated controllers: the factors that decide the layout
This table compares combined mode (broker and controller in one process) with dedicated controller nodes across the factors that drive the layout decision. Read each row against your own failure tolerance, team skills and upgrade process, and treat the Kafka docs' recommendation for critical environments as the starting point for that judgment.
| Factor | Combined mode (broker,controller) | Dedicated controllers |
|---|---|---|
| Controller isolation | The controller shares a process and host with broker workloads, so it is less isolated from the rest of the system. | Controllers run on their own nodes with no broker role, which the Kafka KRaft docs treat as the more isolated design. |
| Recommendation for critical environments | The Kafka KRaft operations docs state combined mode is not recommended in critical deployment environments. | Matches the docs' guidance for critical environments. Confirm the choice against your own risk register. |
| Typical node count | Quorum members are the same machines as brokers, so the controller count is tied to broker count and placement. | The docs say an admin typically selects 3 or 5 servers for the controller role, separate from broker capacity. |
| Quorum sizing | A 3-controller quorum survives one failure and a 5-controller quorum survives two, since a majority must stay alive. | Same majority rule applies. Size the quorum to the failures you must tolerate, usually 3 or 5 per the docs. |
| Failure blast radius | A host or process failure removes a broker and a controller voter at once, so one incident affects both data and metadata availability. | A broker failure does not remove a voter, and a controller failure does not remove partition leadership capacity. Confirm the blast radius in your own failure tests. |
| Upgrade sequencing | Each rolling restart touches broker and controller roles together, so the restart order and quorum health must be planned as one step. | Controllers and brokers can be restarted as separate groups. Confirm the exact order in the upgrade notes for your release. |
| Monitoring scope | Dashboards and alerts must cover controller quorum health alongside broker metrics on the same hosts. | Separate alerting for controller quorum state and broker health. Confirm which metrics your monitoring stack exposes. |
| Skills needed | Lower footprint and fewer moving parts, suited to small setups such as development, as the docs note. | Requires operating a second node group and tracking the quorum separately. Confirm your team can run both roles and the voter list. |
How many controllers a KRaft quorum needs
The Kafka KRaft operations docs typically have an admin select 3 or 5 servers for the controller role, and the reason comes down to majority voting. A controller quorum keeps working only while a majority of its members stay alive. A 3-controller quorum therefore survives one failure, and a 5-controller quorum survives two. The count you pick is a statement about how many controller losses you expect to absorb before the metadata layer stops accepting changes, so it is the first number to settle before you decide anything about combined mode.
An even-sized quorum adds no fault tolerance over the next odd size down. Four controllers still need three alive to form a majority, so they survive only one failure, the same as three. The fourth node adds cost and a further member to patch, restart and monitor without buying resilience. For that reason, treat odd counts as the normal choice and ask yourself directly: how many controller failures must this cluster survive while still accepting metadata changes? If the answer is one, three controllers fit; if the answer is two, five do. Make that decision explicitly and write it into your design record, because the same reasoning applies whether the controllers run on dedicated nodes or alongside brokers.
Once the count is set, confirm the membership mechanics before you provision anything. With a dynamic quorum, controller.quorum.voters must not be set and controller.quorum.bootstrap.servers is used instead. With a static quorum, every broker and controller lists the IDs, hosts and ports of all controllers in controller.quorum.voters, so adding or replacing a controller means a configuration change everywhere. Run kafka-features.sh to check which kraft.version your cluster uses, and confirm the current procedure for your release in the Apache Kafka 4.3 KRaft operations documentation before you plan a replacement window.
Static vs dynamic quorums: what changes in configuration
The quorum type decides which controller keys every node carries and how membership changes later. The cards below cover the keys that differ between a static and a dynamic quorum, how to confirm which mode a cluster is running, and what to check before you commit a layout.
Static quorum voters on every node
In a static quorum, controller.quorum.voters lists the IDs, hosts and ports of all controllers, and that list must appear on every broker and controller. Each node in a combined or dedicated layout therefore needs the same voter list. Before you choose static, confirm who edits this list during maintenance, since a missed node keeps a stale view of the quorum.
Bootstrap servers for dynamic quorums
In a dynamic quorum, controller.quorum.bootstrap.servers replaces the voter list. Nodes use it to find the controllers when they join or start, and the quorum membership itself is then managed by Kafka rather than by a list in each file. Check that every node can reach the bootstrap endpoints from every network segment where it will run.
Voters must not be set for dynamic quorums
When a cluster uses a dynamic quorum, controller.quorum.voters must not be set. A configuration file that carries both the voter list and the bootstrap servers mixes the two modes, and that is the most common cause of a failed quorum setup. Audit the configuration templates your automation renders, because a stale voters line copied from an older static template can break a dynamic cluster.
Confirm the mode with kraft.version
Run kafka-features.sh and read the kraft.version feature to see which quorum mode the cluster is using. Do this on the running cluster rather than trusting the file on disk, because a config file can differ from what the nodes loaded. Record the output in your change record so auditors and operators see the same answer.
Operational impact of membership changes
Static membership changes mean editing controller.quorum.voters on every node and restarting in a coordinated order, which adds work for each controller you add or remove. Dynamic membership moves the change into the quorum itself, so fewer files change, but your runbooks must still account for the majority rule that keeps the quorum available. Agree the procedure with your operators before the first change, not during an incident.
What to verify before choosing
Confirm your Kafka version, the controller count (typically 3 or 5 under the documented guidance), and whether your automation can render per-node voter lists or manage bootstrap endpoints. Check whether your change process can handle a membership change that touches every node. Use the upgrade notes to confirm the target release supports the quorum mode you pick.
When combined mode is acceptable for a team
Combined mode is acceptable in development and test environments, and in small non-critical clusters where an outage of the controller is an inconvenience rather than a business event. The Kafka KRaft operations docs describe combined servers as simpler for small setups such as development, with the trade-off that the controller is less isolated from the rest of the system. In practice, that means a team can run a single node with process.roles set to broker,controller to learn the platform, test client code or rehearse upgrades, without standing up a second set of machines. The same reasoning applies to a small production cluster whose owners have agreed that a slow or failed broker may also slow metadata changes for a short period.
Before choosing combined mode for anything beyond a sandbox, write down the isolation trade-off and have the accountable owner sign it through your own risk acceptance process. The docs say the controller is less isolated, so a memory spike, a long garbage collection pause or a disk problem on a broker can affect controller availability on the same host. Ask the team to state the consequence in plain terms, such as how long the cluster could tolerate losing metadata operations, and confirm with your governance function whether that risk fits your policy. Keep the decision record next to the cluster's configuration so auditors and future operators can see why the layout was chosen.
Within combined mode, the controls that reduce risk are operational rather than architectural. Monitor controller health and quorum state alongside broker metrics, and alert on lag or failed elections rather than waiting for client errors. Isolate resources where the platform allows it, for example by sizing heap and disk for the combined role, keeping other workloads off the host and avoiding co-location with heavy batch jobs. Schedule upgrades and configuration changes in defined change windows and rehearse rolling restarts in a non-production copy. These measures lower the likelihood of trouble, but they do not guarantee that a broker problem will leave the controller untouched, so confirm the expected behavior and test it against your own environment.
Upgrades, monitoring and skills for each layout
This table compares how combined and dedicated-controller KRaft layouts differ in upgrade ordering, configuration drift, monitoring, metadata recovery and staffing. Read each row as a question to answer in your own environment, and treat the documented upgrade paths as the fixed points around which your plan is built.
| Area | Combined mode (broker,controller) | Dedicated controllers |
|---|---|---|
| Upgrade path to 4.x | Clusters older than 3.3.x should move to 3.9.x before 4.x, and rolling upgrades are supported from the last three releases. Plan the same path for both layouts. | Same documented path applies. Controller nodes are upgraded as a separate group, so the order of brokers and controllers must be written into the runbook. |
| Upgrade ordering | Each node restarts both roles, so a single node restart removes a broker and a controller vote together. Confirm the quorum stays healthy before each restart. | Controllers can be rolled first or last in a planned sequence. Confirm the upgrade order in the Kafka upgrade notes for your exact source and target versions. |
| Configuration drift risk | One process.roles and voter list is applied to every node. Drift is easier to spot but a single bad change touches both roles. | Broker and controller configs diverge over time. Keep controller.quorum.voters or controller.quorum.bootstrap.servers identical everywhere and review them in version control. |
| Monitoring scope | Monitor broker and controller health on the same hosts. Watch for controller contention when broker load spikes. | Monitor controller quorum state separately from broker throughput. Alert on controller elections and lag on the controller nodes. |
| Metadata backup and recovery | Metadata lives in the controller quorum on the same machines as broker data. Confirm your recovery runbook covers restoring a quorum when disks on several combined nodes fail. | Metadata is held on fewer, dedicated nodes. Confirm the backup and restore procedure for the metadata log and test it against a rebuilt controller. |
| Skills needed to run it | Operators need broker tuning, storage planning and KRaft basics. The team must understand the layout of both roles on each host. | Operators need the same broker skills plus controller quorum administration, including kraft.version checks and voter management. |
| Fit for critical environments | The Kafka KRaft operations docs state combined mode is not recommended in critical deployment environments because the controller is less isolated. Confirm this against your risk assessment. | The layout the docs point to for critical environments, with the controller isolated from broker load. Confirm the node count with your own failure-tolerance target. |
Deciding the layout before you build the cluster
This sequence turns the combined-or-dedicated question into a written, reviewable decision covering environment criticality, controller failure tolerance, quorum type, node placement and records for audit. Each step ends with an output that the next step depends on.
Classify the environment as critical or not
The platform owner and the security lead agree in writing whether this cluster supports production workloads that the business cannot afford to lose or degrade. The Kafka KRaft operations documentation says combined mode is not recommended in critical deployment environments, and that combined servers suit small setups such as development because the controller is less isolated from the rest of the system. The output is a one-line classification, signed by both roles, that sets the default layout for the rest of the sequence. If the answer is unclear, treat the environment as critical until the owners confirm otherwise.
Count the controller failures you must survive
The platform team decides how many controller losses the cluster must tolerate without losing metadata availability. The Kafka KRaft operations documentation says an admin typically selects 3 or 5 servers for the controller role, because a majority must stay alive: a 3-controller quorum survives one failure and a 5-controller quorum survives two. The output is a stated controller count and the failure tolerance it gives, recorded next to the classification from the previous step. Confirm the choice against the documentation for the Kafka version you plan to run.
Choose a static or dynamic quorum
The platform engineers select the quorum type that matches their tooling and operating model. A static quorum lists the IDs, hosts and ports of all controllers in controller.quorum.voters on every broker and controller. A dynamic quorum leaves controller.quorum.voters unset and uses controller.quorum.bootstrap.servers instead. Confirm the chosen mode after deployment by running kafka-features.sh and checking the kraft.version output on the running cluster. The output is a written quorum decision with the configuration keys it requires and the kraft.version value observed in the environment.
Set process roles per server
The platform engineers assign process.roles on each node: broker for broker-only servers, controller for controller-only servers, and broker,controller only where the earlier decisions allow combined mode. The configuration review confirms that every node in the plan has exactly one intended role set, and that the controller nodes match the count from step two. The output is a role inventory that maps each host to its role. Check the mapping against the official KRaft operations page for the version in use before rollout.
Place controllers outside single failure domains
The infrastructure team spreads controller nodes across hosts, racks, zones or hypervisors so that one shared fault does not remove a majority. In a dedicated layout this means controllers do not sit on the same hardware or network segment as all the brokers. In a combined layout, the same logic applies to the servers that carry both roles. Ask the infrastructure team to show the placement map and where each controller runs, then record any shared dependency, such as a common storage array or hypervisor host, as an accepted risk or a fix. The output is a placement diagram reviewed by someone outside the platform team.
Confirm the access model before going live
The security team checks how clients, brokers and tools authenticate and how ACLs apply to the cluster. Kafka supports SSL and SASL, and the documented SASL mechanisms include GSSAPI, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER. In KRaft clusters, the StandardAuthorizer stores ACLs in the metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. The output is a signed access configuration that covers controller listeners as well as broker listeners. Confirm each setting against the Kafka security and authorization documentation for the version you run.
Record the decision for audit
The platform owner writes a decision record that holds the criticality classification, controller count and failure tolerance, quorum type with the kraft.version output, role inventory, placement map and access configuration. Each item links to the evidence that supports it, such as configuration files and command output. An independent reviewer, such as an internal auditor or a compliance lead, checks that the record matches the running cluster. The output is a versioned document that can be reviewed when the layout changes. DeployOpen can help prepare this evidence, but audit opinions and certifications come from independent auditors and certification bodies.
Questions teams ask before choosing a Kafka controller layout
Is combined mode supported for production Kafka clusters?
Combined mode, where a server runs process.roles set to broker,controller, is a supported configuration, but the Kafka KRaft operations docs say it is not recommended in critical deployment environments. The docs describe combined servers as simpler for small setups such as development, while the controller is less isolated from the rest of the system. For a production cluster that other teams depend on, the practical question is how much isolation you need and whether a broker workload could starve the controller. Check the current wording in the Kafka KRaft operations docs before you settle the design.
How many controllers should we run, and why 3 or 5?
The Kafka KRaft operations docs say an admin typically selects 3 or 5 servers for the controller role. A KRaft quorum needs a majority of its controllers alive to keep operating. A 3-controller quorum survives one failure, and a 5-controller quorum survives two. Five controllers add tolerance for a second failure during maintenance or an incident, at the cost of more nodes to patch, monitor and back up. Choose the count from your failure domains: three racks, zones or hosts that can fail independently usually map cleanly to three controllers.
What does converting a combined cluster to dedicated controllers involve?
Converting a combined cluster means adding controller-only nodes, moving the controller role off the existing brokers, and changing the quorum configuration. Whether you can do this online, and in what order, depends on your Kafka version and quorum type, so confirm the exact procedure in the Kafka upgrade notes and KRaft operations docs before planning a change window. Test the procedure on a non-production copy first. Ask whoever scopes the work to name every step that touches the live metadata log and the rollback path if a step fails.
Static or dynamic quorum: which should we use?
A static quorum lists the IDs, hosts and ports of all controllers in controller.quorum.voters on every broker and controller. A dynamic quorum leaves controller.quorum.voters unset and uses controller.quorum.bootstrap.servers instead. The kraft.version value reported by kafka-features.sh shows which mode a cluster is running. Dynamic quorums are the option that lets you change controller membership without a full restart, but confirm the membership change steps for your version in the KRaft operations docs before depending on them. Static setups are simpler to reason about, but every membership change touches every node's configuration.
Can our own platform team run a KRaft quorum without a partner?
Yes, many teams run KRaft clusters themselves, and the core work is manageable for teams that already operate Kafka, Linux and their monitoring stack. The skills to check are controller failover behavior, metadata log monitoring, certificate and listener configuration, and upgrade sequencing across the supported releases. Also check whether the team has a tested restore procedure and a runbook for losing a controller. If the answer to several of these is no, a fixed-scope engagement can cover the design, the build and the handover, with the scope and deliverables agreed in writing before work begins.
What drives the effort and timeline for a controller redesign?
The main drivers are the current topology, the Kafka version in use, whether the environment is virtual machines or Kubernetes, how much existing tooling (configuration management, monitoring, backup and certificate automation) you can reuse, and how many client applications depend on the cluster. Compliance evidence needs can add work if auditors expect documented change control and tested recovery. No figure here is a fixed estimate. Ask any provider for a written scope listing the environments, the number of nodes, the migration steps and the test evidence, and compare quotes against that same scope.
When should we bring in an implementation partner with a fixed written scope?
Bring in a partner when the decision is blocked by skills, deadlines or audit evidence rather than by the technical choice itself. Typical triggers are a production cluster still running combined mode with no isolation plan, an upgrade path that needs to pass through several supported releases, or a compliance review that requires documented controller design and tested failover. DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms in your own infrastructure, and most engagements are fixed-scope. DeployOpen does not issue audit reports or certifications, so the evidence it prepares would go to your independent assessor.
How DeployOpen scopes and delivers Kafka deployment work
DeployOpen plans and implements self-hosted Kafka inside your own cloud accounts, data centers or air-gapped networks, including the KRaft role layout decision. We start from your broker count, failure tolerance and isolation requirements, then write the topology down before any node is built.
How we scope
- Written scope fixes the process.roles layout, controller count and quorum type for each environment
- Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts
- Scope lists which environments stay combined and which get dedicated controllers, with the reasons recorded
How we build
- Controller and broker nodes are deployed to your infrastructure with configuration reviewed against the KRaft operations guidance
- Quorum membership is verified, including controller.quorum.voters or bootstrap settings matching the chosen dynamic or static model
- Failover tests simulate loss of controllers and brokers so the quorum behavior is checked before go-live
How we hand over
- Documentation covers the topology, quorum sizing, upgrade path and the kraft.version check used to confirm the quorum type
- Runbooks cover controller replacement, broker addition and rolling upgrades within your change process
- Handover includes evidence of tests run, so your team and any assessor can see what was verified
How we support
- Ongoing support, upgrades and continuous testing are available as retainers after handover
- Retainer scope is agreed in writing and can cover upgrades when a new Kafka release is adopted
- Your team keeps ownership of the cluster, and we advise on topology changes as your broker count grows
