Why separate KRaft controllers and brokers with Strimzi node pools
Separating controllers from brokers is worth planning when a platform team finds that a single Strimzi-managed Kafka cluster forces unrelated decisions together. In a common starting layout, the same pods run both the KRaft controller role and the broker role, and they share the same storage definition. That means a change to broker capacity, such as adding disk or moving to larger hosts, also touches the metadata quorum, and scaling brokers can change the controller footprint without anyone intending it. KafkaNodePool resources let you define each group of nodes separately, with its own roles, replica count, storage and resource settings, under one Kafka cluster.
Kafka 4.x runs only in KRaft mode, so the controller role is always present in the cluster. The question is not whether controllers exist but whether they share pods with brokers. Dedicated controller pools tend to fit when the metadata quorum should stay small and stable while broker capacity changes often, when broker storage is large or replaced on a different schedule, or when the team wants controller pods isolated from broker load and disruption. They add operational objects to manage, so a small cluster with modest traffic may not gain enough to justify them. Confirm the node pool fields, role assignment rules and any restrictions on moving roles against the Strimzi release you run, because these details change between Strimzi versions.
Before you draft a layout, ask your Strimzi vendor or community contacts, and review the Strimzi documentation for your version, how pools are assigned roles, whether a pool can change roles after creation, and how storage changes are applied to existing pods. Then compare those answers with your Kafka version using the Apache Kafka downloads page so the controller and broker plan matches a supported release. Write down who owns the controller pool, what happens to quorum during a node pool rollout, and which change you would roll back first if a rollout stalls.
Which pool layout fits your cluster
A combined layout assigns both the controller and broker roles to one KafkaNodePool. Each node votes in the KRaft metadata quorum and also serves partitions. This is the simplest shape to create and the smallest footprint to maintain, which suits development, test and small non-production clusters where availability expectations are modest.
- Fewer objects to manage: One pool means one storage template, one set of resource settings and fewer moving parts during bootstrapping.
- Shared failure domain: A node that fails removes a controller vote and broker capacity at the same time, so the quorum and the data path degrade together.
- Coupled scaling: Adding broker capacity also adds a controller, so you cannot grow throughput without changing the size of the metadata quorum.
- Resource contention: Metadata work and partition I/O compete for the same CPU, memory and disk on each node, which makes tuning harder to isolate.
- Migration later: Moving to dedicated pools requires a planned change to role assignments, so decide early whether the cluster will become production.
Before choosing this layout, confirm that a combined pool is supported for your target Kafka and Strimzi versions and that your availability requirements accept a shared failure domain.
A dedicated layout splits the roles into a controller-only pool and a broker-only pool. The controller pool carries only cluster metadata, while the broker pool carries client traffic, replication and partition storage. This is the usual shape for production because each role can be sized, placed and scaled according to its own needs.
- Isolated metadata quorum: Controllers handle only metadata, so broker load, partition reassignments and disk saturation do not directly compete with the quorum.
- Independent scaling: Broker pools can grow or shrink without changing the controller count, which is a separate decision based on the fault tolerance you need.
- Storage per role: Controllers need durable metadata storage, while brokers need data volumes sized for throughput and retention, so each pool gets its own storage template.
- More to operate: Two pools mean two sets of resource requests, two rolling update sequences and matching labels for monitoring and policy.
- Placement rules: Controllers must stay spread across failure zones, and that scheduling constraint needs its own check separate from the broker pool.
Confirm the controller count against the fault tolerance your organization requires, and check the hardware and sizing guidance for your environment before fixing pool sizes.
A specialized layout adds broker-only pools built for particular workload profiles, such as latency-sensitive topics, high-throughput ingestion or long retention. These pools sit alongside a general broker pool, and they can use different node types or storage classes. This layout pays off when profiles differ in ways that are measured and persistent, not in ways that were only expected at design time.
- Profile-matched hardware: A pool can use faster volumes or different node types for a workload that needs them, without raising the cost of every broker.
- Contained impact: A heavy workload placed on its own brokers is less likely to affect other teams, although quotas and topic placement still determine how much isolation you actually get.
- Placement discipline: Each topic must be assigned to the correct pool, which requires a naming or labeling convention and a review step in topic provisioning.
- Moves between pools: Relocating partitions to another broker pool is a reassignment that generates replication traffic and takes time, so test the procedure on a non-production cluster first.
- Ownership and cost: Each pool adds a cost line and an owner, so the number of pools should match the number of profiles the platform team can support.
Measure the profile differences on your own workloads before creating a dedicated pool for them, and confirm who approves new pools and topic placements.
KafkaNodePool settings and when each one fits
This table compares the KafkaNodePool settings a designer must decide for each pool: roles, replicas, storage, resources, pool configuration and placement. Read each row as a decision with a fit condition and a constraint it places on the cluster. The main takeaway is that roles and replica counts are the settings that are hardest to change later, so decide them before the first deployment.
| Setting | When it fits | What it constrains |
|---|---|---|
| Roles (controller or broker) | Separate controller-only pool when you want metadata isolated from data traffic; combined pool when the cluster is small and operational simplicity matters | Controller roles hold cluster metadata; moving a role between pools is a planned change, not a routine scaling action |
| Replicas on the controller pool | Odd counts such as three or five give a clear majority for the KRaft quorum; choose the count from your availability target and the hardware sizing guidance | Quorum needs a majority to elect a leader, so losing more than a minority of controllers stops metadata changes; confirm the count with your upgrade plan |
| Replicas on the broker pool | Scale brokers independently of controllers when throughput, partition count or consumer load grows | Partition replica placement and rebalancing depend on broker count; changing it affects replication traffic and disk use |
| Storage type and class per pool | Use persistent volumes for brokers and controllers that hold data or metadata; choose a storage class that matches the disk throughput the pool needs | Storage is tied to each pool and node, so the class cannot be swapped without moving data; confirm volume expansion support with your storage vendor |
| Resource requests and limits | Set explicit requests for each pool so controllers keep CPU and memory under broker load; size broker memory with page cache in mind | Requests affect scheduling and eviction; memory limits that are too tight can destabilize a JVM-based process |
| Pool-level configuration | Use per-pool settings for controller-specific or broker-specific Kafka properties that should not apply to the other role | Settings that are not shared across pools must be kept consistent with cluster-wide configuration to avoid drift |
| Node affinity and taints | Pin controllers and brokers to dedicated nodes or zones when failure domains matter for quorum or replicas | Tight placement rules can leave pods unschedulable during node maintenance or rolling upgrades; plan spare capacity |
| Move workloads between pools | Plan a migration path when a combined pool needs a dedicated controller pool, and confirm the supported procedure for your Strimzi version | Role changes touch the metadata quorum; test the procedure on a non-production cluster before running it on production data |
How to set up controller and broker pools in Strimzi
This sequence takes a Strimzi-managed Kafka cluster from a version check to two separate KafkaNodePool resources, one for KRaft controllers and one for brokers, and confirms that the quorum, pod placement and client connections work before the cluster carries production traffic.
Confirm Strimzi and Kafka version compatibility
The platform engineer checks which Kafka release the chosen Strimzi version supports before writing any manifests. Start from the Kafka downloads page, which lists 4.3.1, 4.2.2 and 4.1.2 as the supported releases, and confirm in the Strimzi documentation that the operator release you plan to install supports the same Kafka version. Record both versions in the change request. Done means you have a written pair of Strimzi and Kafka versions that both your team and the operator documentation agree on, and that the Kafka version is on the supported list. If you are moving from an older KRaft cluster, confirm the upgrade path separately before proceeding.
Define the controller KafkaNodePool resource
The platform engineer writes a KafkaNodePool resource with the controller role and a replica count that matches the quorum design agreed for the cluster. Set the pool name, the strimzi.io/cluster label that points to the Kafka resource, and the role field to controller. Keep controller pods free of the broker role so that metadata handling is isolated from client traffic. Done means the manifest passes a dry-run apply against the cluster namespace and the role field reads controller only. The number of controllers should come from your sizing review, not from this step.
Define the broker KafkaNodePool resource
The platform engineer writes a second KafkaNodePool resource with the broker role and its own replica count, listener-facing sizing and resource requests. Give it a distinct pool name and the same cluster label. Broker pools are the part you will scale and move later, so keep their settings in a separate file that can change without touching the controller pool. Done means the broker manifest also dry-runs cleanly and the two pools share one cluster label but have different names and roles.
Set storage per pool
The platform engineer defines the persistent volume configuration in each KafkaNodePool, using a storage class that your infrastructure provides. Controllers need durable metadata storage with low write latency, while brokers need capacity and throughput for log segments. Confirm with your storage team which storage classes exist, whether volumes can be expanded, and how they are reclaimed when a pool is removed. Done means each pool declares its own storage block, the volume sizes and classes are recorded in the design document, and the storage class exists in the target cluster.
Reference both pools from the Kafka resource
The platform engineer updates the Kafka resource so that it is managed through node pools rather than a fixed replica count, and confirms that the cluster name, listeners and authorization settings match the pool labels. Apply the Kafka resource alongside the two pool resources in one change so the operator reconciles them together. Done means the Kafka resource is accepted by the operator without validation errors and the operator reports both pools as associated with the cluster.
Deploy the pools and watch the rollout
The platform engineer applies the manifests through your normal delivery path and watches the operator create controller and broker pods. Controllers should come up and elect a leader before brokers register. Capture the operator log output and the events for the namespace so the rollout can be reviewed later. Done means all controller and broker pods report ready, and the operator status shows the cluster as ready without pending reconciliation.
Verify quorum health and pod placement
The platform engineer checks that the KRaft quorum has a leader and that every controller is a voting member, then confirms that controller and broker pods run on the nodes and zones your placement design requires. Use the Kafka admin tooling inside a broker pod to describe the quorum, and list pods with their node names. Done means the quorum description shows a leader and the expected voters, and no controller shares a node or failure domain with another controller unless your design explicitly accepts that risk.
Test client connectivity and authorization
The platform engineer creates a test topic and runs a producer and consumer from a host inside the approved network using the listener and authentication settings the cluster exposes. Confirm that authorization denies an unapproved principal and allows the approved one. Confirm with your security team which principals and ACLs the test should cover. Done means a message produced by the test client is consumed, the denied principal receives an authorization error, and the test results are stored with the change record.
How to scale pools and move workloads between them
Broker pools and controller pools scale for different reasons, and only broker pools are routine to resize. A KafkaNodePool that holds broker-only nodes can be scaled up or down by changing its replicas value, and new brokers take on partitions only after you move or create partition assignments for them. Adding brokers adds capacity, not throughput, until partitions are reassigned onto the new nodes. Removing brokers is the riskier direction: before a broker leaves, confirm that every partition it leads or replicates has enough in-sync replicas elsewhere, and drain its data with partition reassignment. The day-two mechanics of reassignment, broker removal and admin tooling are covered in the Apache Kafka operations and monitoring pages, which are the next read for this work.
Moving a broker role or a workload between pools is a reassignment plus a reconfiguration, and the order of steps matters. Changing which pool a node belongs to, or changing the storage or resources of a pool, can trigger rolling restarts of the affected nodes. Plan each change as a sequence: confirm the target pool's storage class and volume size, add capacity first, reassign partitions in batches, verify under-replicated partition counts return to zero, and only then retire the old nodes. Check the Strimzi documentation for the node pool fields that are allowed to change in place and those that require a new pool. Record the current Kafka version and the Strimzi operator version before each change, because rolling upgrades from Kafka are supported from each of the last three releases to the latest version, so a pool design that is only tested against the current release can leave you without a tested upgrade path.
Controller membership changes deserve the most caution because the controller quorum decides whether the cluster can accept metadata changes at all. Controllers form a Raft quorum, so adding or removing a controller changes the voter set and needs a deliberate plan, not a replica count edit made during a busy window. Keep the controller count odd, avoid running controllers and brokers in a way that makes a single node failure break quorum, and confirm the quorum state and voter set before and after each change. Treat controller moves as a change with a rollback decision point, and rehearse them in a non-production cluster first. The exact procedure for adding or removing controllers should be confirmed against the Apache Kafka documentation for your release, since it differs from broker movement and is not a step to improvise.
Security and operational risks of split node pools
Splitting controllers and brokers into separate KafkaNodePool resources moves trust boundaries, storage and operator permissions into new places. The cards below cover the controls that follow from that split, and each one names a check to run before the pools go into production.
Controller listener authorization
The controller listener carries metadata traffic, so access to it has to be scoped to the controller and broker pools only. In KRaft clusters the StandardAuthorizer stores ACLs in the metadata log, and resources without ACLs are limited to super users unless allow.everyone.if.no.acl.found is set to true. Confirm that this setting is false in your cluster and that every inter-pool principal has an explicit ACL before you scale either pool.
Client listener TLS and authentication
Separating pools does not secure client traffic by itself. Kafka security is optional, so a broker listener can run unsecured or mix secured and unsecured listeners unless you define each one. Check that every client-facing listener on the broker pool uses TLS and one of the documented SASL mechanisms, and that no internal listener is exposed on an external address.
Operator RBAC and pod security
The Strimzi operator creates and manages the pods, storage claims and secrets for every pool, so its service account holds broad rights in the namespace. Review its cluster and namespace roles, limit which teams can create or edit KafkaNodePool resources, and apply the Pod Security Standards to the namespace that holds the controllers. Confirm who can read the secrets that hold cluster certificates.
Storage encryption and backups per pool
Controller pools hold cluster metadata and broker pools hold topic data, and each pool may use a different storage class with different encryption and snapshot behavior. Confirm encryption at rest for both volume types, and confirm that backups cover the metadata logs of the controllers as well as broker data. Restores should be tested per pool, because a broker volume and a controller volume are not interchangeable.
Secrets and certificates in etcd
Kubernetes stores Secrets unencrypted in etcd by default, and Strimzi keeps cluster and client certificates in Secrets. Configure encryption at rest for the API server, restrict list and watch access on Secrets in the Kafka namespace with least-privilege RBAC, and decide whether an external secrets store is required for your environment.
Pool-level health monitoring
A single cluster-wide health signal can hide a failing controller quorum behind a healthy broker count, or the reverse. Build alerts for each pool separately: controller quorum membership, broker availability, under-replicated partitions and pod restarts per pool. Test the alerts during a planned node drain so you know which signal fires first.
Questions teams ask before separating controllers and brokers
Can our own team separate KRaft controllers and brokers with Strimzi node pools?
Yes, a capable platform team can do it, provided it already runs Kubernetes, understands Kafka's KRaft metadata quorum and can operate the Strimzi operator. The work splits into design (pool layout, storage, replica counts, network policy), deployment, testing of failure and rolling restart scenarios, and day-two operation. Most teams find the first pass is manageable and the harder part is keeping runbooks current through upgrades. If any of those skills are thin, a partner can cover the gap for a fixed scope, and your team keeps the knowledge through handover documentation and pairing.
What drives the cost of a node pool design engagement?
Cost follows the amount of design and testing work, which depends on how many clusters, environments and storage classes are involved, how strict your change control is, and how much existing automation must change. Ask any provider to price these components separately: discovery and design, the Strimzi and Kubernetes configuration, a test plan that covers controller loss, broker rolling restarts and pool scaling, handover, and any retainer for ongoing upgrades. Design and deployment are usually one-time. Operating the clusters, applying Strimzi and Kafka patch releases and running periodic recovery tests recur. Use the written scope as the basis for comparing quotes.
How long does separating controllers and brokers take?
Duration depends on your environment more than on the Strimzi configuration itself. The main variables are whether a Kubernetes cluster and storage classes already exist, how many Kafka clusters move to node pools, whether migration touches live topics, and how long your change approvals and test windows take. Ask for a timeline that lists each dependency and the approval gates it depends on. Treat any duration that does not follow from those inputs as a question to challenge, and confirm the Strimzi and Kafka versions you will use against the project's release information before fixing dates.
What must we provide before the work starts?
Plan to provide the following before design begins:
- A Kubernetes cluster with enough capacity for the planned controller and broker pods, and an agreed namespace and access model
- Storage classes with the performance and volume-binding behavior your brokers and controllers need
- Identity and access: who may change Kafka resources, and how client authentication will be integrated
- TLS certificates, including the internal certificate authority that signs cluster and client certificates
- Network policies, DNS and any external listener requirements
- Monitoring, backup targets and a change window for testing
Confirm each item in writing so the quote reflects what is actually in place.
Do we need a dedicated controller pool at our size?
Not always. A dedicated controller pool separates metadata quorum work from broker data traffic and lets you scale brokers without touching the controllers. Teams with a small cluster and modest traffic may run combined nodes and still meet their goals, while larger or busier clusters, and those with strict isolation or change-control rules, usually benefit from separate pools. Controller count and host sizing are covered in the hardware sizing guidance, so review that before deciding. Ask whether the design makes the separation a choice you can revisit later, since moving between layouts is itself planned work.
What goes wrong when controllers and brokers are moved between pools?
The most common problems are an incorrect storage mapping that leaves data on the wrong volume, controller quorum disruption during a rolling change, and client disruption from broker restarts that were not planned around replication. Test each move in a non-production cluster first, check replication health and controller quorum before and after every step, and keep a rollback plan. Confirm with your Strimzi documentation which pool changes require a restart, and budget testing time for that verification rather than assuming a change is safe because the manifest applies.
Who runs the cluster after the design is delivered?
Your team usually runs it day to day, with the option to add an ongoing support arrangement. DeployOpen plans, implements, documents and supports self-hosted open-source platforms inside your own infrastructure, and offers ongoing support, upgrades and testing as retainers. Ask for the handover package, the upgrade procedure for Strimzi and Kafka, and the recovery test schedule as part of the written scope, so you know which tasks stay in-house and which recur under a retainer.
Where to verify the Kafka and Kubernetes behavior behind this design
Before you commit to a controller and broker split, read the Kafka upgrade notes for the release line you plan to run. They state which KRaft versions can move to 4.x and which intermediate release is recommended first, which determines whether the controller pool you create today can follow a supported upgrade path. Pair that with the Apache Kafka 4.3 upgrade notes and confirm the Java runtime each role needs, because the broker and controller images you build into node pools must match it.
The Apache Kafka 4.3 authorization documentation explains how the StandardAuthorizer stores ACLs in the cluster metadata log and why resources without ACLs are limited to super users unless allow.everyone.if.no.acl.found is set. Check this before you separate pools, since controller and broker identities, listener names and the principals granted access to each pool need to be planned together. Confirm the default behavior in your target release rather than relying on older examples.
The Kubernetes version skew policy governs how the Strimzi-managed pods can run alongside the control plane during an upgrade. It says kube-apiserver must be upgraded one minor version at a time, ahead of kube-controller-manager, kube-scheduler and kubelets, and that kubelets may be up to three minor versions older. Read it with the Strimzi compatibility table for your operator version, and ask your platform team to confirm which Kubernetes minor versions are in support at the time of your rollout, since node pool rollouts are only as safe as the cluster beneath them: Kubernetes version skew policy.
How DeployOpen scopes and delivers a Strimzi node pool design
This sequence turns a node pool layout into a written scope, a reviewed design, a working split of controller and broker pools in your own clusters, and a handover your team can run without us.
Confirm the current cluster and goals
DeployOpen reviews the existing Strimzi version, Kafka version, Kubernetes platform, storage classes, topic count and partition layout, and the reasons for splitting roles, such as independent scaling or isolating controller load from broker disk pressure. Your platform team supplies cluster access, change windows and the list of client applications. The step closes with a short written findings note that states what is confirmed and what still needs your input.
Agree the fixed-scope statement of work
DeployOpen drafts a written scope that names the pools to create, which roles each pool carries, the storage and node placement per pool, the migration method for existing workloads, the verification tests and the handover materials. Both parties sign off before any change is made to your clusters. The quote follows from this scope, so ask for each item to be listed separately so you can see what is one-time work and what would recur under a retainer.
Review the node pool design
DeployOpen produces a design document with KafkaNodePool resources for controller and broker roles, including replica counts, storage definitions per pool, pod scheduling and the Kafka custom resource changes needed to reference the pools. Your architects and Kafka owners review the trade-offs, such as whether controllers run as dedicated nodes and how broker storage is sized. The step is complete when the design is approved in writing and open questions are closed.
Prepare a rollback and backup point
Before any pool change, DeployOpen confirms that you have current backups of topic data and a documented rollback path for the Kafka and Strimzi configuration. This includes saving the existing Kafka and node resources and confirming that the controller quorum is healthy. The step ends when your team confirms the backup is restorable and the change window is booked.
Implement the pools in your clusters
DeployOpen applies the approved KafkaNodePool manifests in a non-production environment first, then in your target cluster, following the order set in the design. Work runs inside your Kubernetes and Strimzi environment using your repositories and deployment tooling, and every change is applied through reviewable manifests. The step closes when all pools report ready and the cluster reaches a stable state with no under-replicated partitions.
Move workloads and verify behavior
Where brokers or partitions move between pools, DeployOpen runs the agreed reassignment steps and checks leader placement, replication, client connectivity and consumer lag. Verification tests cover controller failover, a broker restart in each pool and a scale-out of the broker pool. The step is complete when each test in the scope has a recorded result that your team has reviewed.
Hand over runbooks and operating notes
DeployOpen delivers the final manifests, the operating runbook for scaling each pool, storage expansion steps, pool-specific alerts to add, and the list of decisions that were made and why. Your team walks through the runbook with DeployOpen and runs one supervised scaling action. The handover is accepted when your team completes that action without assistance.
Continue with an optional retainer
If you want ongoing help, DeployOpen can offer a retainer for Strimzi and Kafka upgrades, periodic testing of pool failover and recovery drills. Retainer scope is agreed separately in writing and does not change the fixed-scope delivery above. Ask for the retainer terms to state which upgrade and test activities are included so you can compare them against your own team's capacity.
