Apache Kafka high availability starts with the failures you must survive
A team that needs Kafka to keep accepting and serving data through a broker or controller loss has to decide what it must survive before it touches a configuration file. High availability in Apache Kafka comes from configuration and placement decisions made before go-live. It is not a setting you switch on. Start by writing down the failures the cluster must tolerate, such as one broker, one controller, one rack or one facility. Every later choice on quorum, replication, acknowledgments and placement should trace back to that list, so the design can be tested against it.
In KRaft mode, the controller quorum is separate from the data brokers. Controllers hold cluster metadata and coordinate leader elections, while brokers store and serve partition data. A broker failure and a controller failure are therefore different events with different recovery paths, and each needs its own sizing decision. If a majority of controllers is lost, metadata changes stop until quorum returns, while brokers can keep serving partitions they still host. Confirm which roles run on dedicated nodes and which are combined, because combined roles tie failure domains together and reduce the tolerance you thought you had.
Topic-level settings decide whether acknowledged data survives a failure. The replication factor sets how many copies of each partition exist, and min.insync.replicas sets how many of those copies must be in sync before a write using acks=all is confirmed to the producer. Producers using acks=0 or acks=1 can receive confirmations for data that a single failure removes. Ask each owning team what durability it needs per topic, because one cluster-wide default rarely suits payment events, telemetry and application logs equally. Record the chosen values per topic and check them against the failure list, not only against the broker defaults.
Placement decides whether one facility outage is survivable. Rack awareness tells Kafka to spread replicas across failure domains such as racks, rooms or availability zones, but the labels only help if they match real power, network and cooling boundaries. After rollout, verify the broker rack assignments and the replica distribution for sample partitions. Also confirm that controller nodes are placed so that losing one failure domain does not remove the quorum majority. Cross-site copies of data are a separate design, covered elsewhere, so this section stays with resilience inside a single site or between failure domains you already operate.
Two assumptions make Kafka high availability harder than it looks: broker defaults that do not match the intended tolerance, and failover that has never been exercised. Plan a recovery test for each item on the failure list. Stop a broker, stop a controller, take down a rack, and measure whether producers and consumers recover within the limits the business has set. Confirm the actual values with your own environment rather than assuming them. Record what you observed, then repeat the tests after each upgrade or significant configuration change, because a design that passed once can drift.
Which high availability decisions to make before you size anything
This table compares the main Kafka availability decisions, what each one protects against, and what it costs to run. Read each row as a question to settle with your failure scenarios and recovery goal before choosing broker counts or storage. The main takeaway is that replication, acknowledgment and placement settings trade durability against availability and cost, so they should be chosen together.
| Decision | When it fits | Operational cost | Confirm before sizing |
|---|---|---|---|
| KRaft controller quorum size | Odd-sized quorum so a majority survives a controller loss; separate controllers when brokers scale independently | Each controller is an extra node to patch, monitor and back up; losing a majority halts metadata changes | Quorum size for your failure scenarios and whether controllers run co-located with brokers in the 4.3 documentation |
| Replication factor per topic | Higher factor for topics where losing data or failover time is unacceptable; lower for replayable data | More storage and inter-broker network traffic for each added replica | Default replication settings in the 4.3 configuration reference and per-topic overrides in your plan |
| min.insync.replicas with producer acks | Pair acks=all with a minimum in-sync count that matches how many broker losses you must survive without data loss | Too few in-sync replicas makes producers fail with write unavailability until replicas catch up | Interaction between min.insync.replicas, acks and replication factor in the 4.3 producer and broker documentation |
| Unclean leader election | Usually disabled where data loss is unacceptable; enabled only where availability outweighs possible loss of acknowledged records | Enabling it can elect an out-of-sync replica and lose records that were already acknowledged | Default value for unclean.leader.election.enable in your release and how your producers handle acknowledged writes |
| Rack or availability zone placement with broker.rack | Multiple racks or zones in one site where replicas must land in different failure domains | Placement constraints need enough brokers per zone; uneven zones reduce the replica spread you planned | Rack assignment rules and replica placement behavior in the 4.3 operations documentation |
| Single site versus multi-site | Single site can meet the recovery goal when the failure domains you need are inside one facility or region | Brokers spread across sites add latency to replication and acknowledgments, and add network cost between sites | Your recovery time and recovery point objectives, and whether cross-site replication needs MirrorMaker 2 as a separate layer |
How to build a highly available Kafka cluster in order
This sequence takes a self-hosted Apache Kafka cluster from failure planning to a verified first broker stop. Each step ends with a check, so resilience is confirmed before the next layer depends on it. Exact commands change between releases, so follow the 4.3 upgrade and operations documentation for syntax.
Define failure scenarios and recovery targets
Before any configuration, list the failures the cluster must survive: loss of one broker, loss of one controller, loss of a rack or availability zone, and loss of a whole site if you run one. For each scenario, record how long producers and consumers may stall and how much data loss, if any, the business accepts. The platform team owns this work with the application owners who define acceptable behavior. The step closes when the scenarios and targets are signed off in writing, because every later setting is judged against them.
Place controllers and brokers across failure domains
Map each controller and broker to a failure domain such as a rack, data hall or availability zone, using your infrastructure provider's documented zone layout. Spread controllers so that losing any single domain leaves a majority of the controller quorum running. Confirm with your provider that the domains are independent for power and networking, because shared dependencies undermine the layout. The check is a placement table reviewed against the scenarios from the previous step, with every single-domain loss mapped to a surviving quorum and surviving brokers.
Configure the KRaft controller quorum
Deploy the controllers in KRaft mode as a dedicated or combined quorum, using an odd number of voters so a majority can always form. Give each voter a stable identity and address, set the voter list consistently on every node, and format storage with the same cluster ID before the first start. Quorum size is a trade-off: more voters tolerate more failures but add replication work for metadata writes, so confirm the choice against your scenarios. The closing check is that the quorum reports a leader and every voter appears as a member, verified with the quorum describe tooling in the 4.3 operations documentation.
Set broker.rack and cluster replication defaults
Set broker.rack on every broker to the failure domain identifier from the placement table, so the cluster can spread replicas across domains. Set default.replication.factor and min.insync.replicas at the cluster level so that topics created without explicit settings still tolerate a broker loss. The replication factor must be greater than min.insync.replicas, or a single broker stop can block acknowledged writes. The check is that each broker reports its rack and the effective defaults match the values signed off in the design.
Create topics with explicit replication settings
Create every production topic with an explicit replication factor and min.insync.replicas, instead of relying on cluster defaults that a later configuration review might change. Keep partition count and retention in the separate sizing work, but record the reliability values beside each topic request. Ask the platform team to reject topic requests that omit these values if your tooling supports that policy. The check is a describe output for each topic showing that replicas for each partition sit on different racks and that the in-sync replica set matches the replica list while the cluster is healthy.
Configure producer acknowledgments and retries
Set producers to acks=all, enable idempotence so retries do not duplicate records, and configure retries and delivery timeouts so that brief leader changes are absorbed by the client. Agree with application owners how a producer should behave when in-sync replicas fall below the minimum, because the producer will return errors and the application must handle them. In a test cluster, stop brokers until the in-sync count drops below the minimum. The check is that the test producer receives errors and does not report success for those writes.
Run a first controlled broker stop
Before production traffic, stop one broker cleanly in a test or pre-production cluster while producers and consumers run at a representative load. Watch partition leadership move, the under-replicated partition count, consumer lag and producer error rates, and compare them with the recovery targets from the first step. Restart the broker and confirm it rejoins and catches up to the in-sync replica set. The step closes with a written result listing which scenarios passed, which targets were met and what must change before the next test. Repeat the test after each upgrade, because behavior can change between releases.
How Kafka handles broker and controller failures in practice
When a broker that leads a partition stops, Kafka keeps the partition available by electing a new leader from the in-sync replicas (ISR), the followers that have caught up with the leader. The controller makes this choice and publishes it to the cluster, so clients resume once a new leader is in place. The unavailable window runs from failure detection to election, and its length depends on cluster size, partition count and broker and controller settings. Measure that window in your own environment with a planned broker stop before you commit to any recovery target.
Replication factor and min.insync.replicas decide how much failure a partition can absorb. With acks=all, a write is acknowledged only after the required replicas have it, and if the ISR falls below min.insync.replicas, the partition rejects new writes with NotEnoughReplicas errors while reads from the current leader can continue. Set these two values together and check placement, because a replication factor of three with a minimum of two tolerates one broker loss only if the three replicas sit on different brokers and, where you use rack awareness, in different racks. Confirm that a full rack loss still leaves enough replicas to meet the minimum.
In KRaft mode, the controller quorum runs Raft among the controller nodes and elects one active controller. If the active controller fails, the remaining controllers elect a successor from the quorum, and metadata changes such as topic creation, partition reassignment and leader changes wait until the election completes. Losing a majority of controllers halts metadata changes even if existing partitions still serve traffic, so the number of controllers and their placement across failure domains are sizing decisions. Confirm the controller count and placement against your quorum design and rehearse a controller loss as well as a broker loss.
Unclean leader election decides what happens when no replica in the ISR is available. If unclean.leader.election.enable is set to true for a topic, an out-of-sync replica can become leader, which restores availability but can drop messages that were acknowledged to producers but never copied to that replica. Keeping it disabled favors durability and makes the partition wait for an ISR member to return. Confirm the default for your release in the configuration reference, set the value per topic where the business accepts loss, and record that decision in the runbook.
During failover, producers see retriable errors such as NOT_LEADER_OR_FOLLOWER and NotEnoughReplicas, and consumers pause on affected partitions until leadership moves and the group rebalances. Applications survive this only if they configure it explicitly: retries and delivery.timeout.ms bound how long a producer keeps trying, request.timeout.ms sets how long it waits per request, and enable.idempotence prevents duplicates when a retry follows an ambiguous timeout. On the consumer side, session.timeout.ms and max.poll.interval.ms control when a stalled member is removed from its group. Test these settings with broker kills in a staging cluster and count errors and duplicates before production.
How high availability changes across deployment models
In one data hall, the failure domains are usually racks, top-of-rack switches and power feeds. Brokers and controllers share one network fabric, so latency between them is low and acknowledgments are fast. The main question is whether losing one rack leaves enough replicas and enough controllers to keep the cluster writable.
- Failure domains: set broker.rack on every broker so that replicas of each partition spread across racks. Confirm the rack labels match real power and switch boundaries with your facilities team before the first production broker starts.
- Quorum: place KRaft controllers in different racks where the floor layout allows, so that no single rack holds a majority of the quorum.
- Latency and acknowledgments: with acks set to all, each write waits for in-sync replicas over the same fabric. The cost is small in a healthy facility but rises with congestion, so measure it under load.
- Testable: cutting a rack's power or network, stopping brokers, and observing leader election and client errors in a staging copy of the cluster.
- Not testable: losing the whole building or site, because there is no second site to fail over to.
Check that your rack labels follow actual power and switch boundaries, and record which failures the single site cannot survive so that the gap is visible to the business owners.
Availability zones or data centers within one metro have low but measurable latency between them, which is higher than within a rack. Each zone is a failure domain, and a cluster that claims zone tolerance must keep serving when one zone disappears entirely. Two zones are a common trap, because the zone holding the controller majority becomes a single point of failure for metadata changes.
- Failure domains: map each zone to a broker.rack value and spread replicas across all zones, not only across two of them.
- Quorum: place controllers across three failure domains where the metro allows. With only two zones, losing the zone that holds the majority stops controller writes, even though data may still be served.
- Latency and acknowledgments: acks=all waits for replicas in other zones, so produce latency rises. Measure round-trip times between zones under peak load, then set client timeouts and batch settings from those measurements.
- Testable: isolating one zone at the network level, killing brokers in each zone, and checking producer retries, consumer rebalances and partition leadership.
- Not testable: a full metro outage. Recovery from that scenario depends on cross-cluster replication, which is covered in a separate topic.
Ask your network team for measured inter-zone latency and inter-zone transfer paths, and confirm the controller placement before choosing min.insync.replicas.
On Kubernetes, the scheduler and the storage layer decide where broker and controller pods run and where their volumes can attach. Failure domains therefore depend on node pools, zone labels and volume binding behavior, not only on the Kafka configuration. Automatic rescheduling helps only when the placement rules hold during each restart.
- Failure domains: derive broker.rack from the zone label on each node, then use topology spread constraints and anti-affinity so that brokers do not share a node or zone. Confirm which constraint types your chosen operator supports.
- Quorum: run controllers as separate pods with stable identities, spread across zones. Confirm how your operator handles controller replacement and quorum membership changes.
- Storage: many cloud block volumes are zonal, so a pod can usually reattach only within the zone that holds its volume. Confirm volume type, reclaim policy and reattachment behavior for your provider.
- Latency and cost: cross-zone traffic adds latency to acknowledgments and may carry transfer charges. Test the acknowledgment path under load before settling on replica settings.
- Testable: draining nodes, deleting pods, and simulating loss of a zonal node pool in a non-production cluster. Regional provider incidents are not testable without a second region.
Confirm with your operator vendor which upgrade, scaling and quorum operations it supports, and verify the placement rules after every node pool change.
Air-gapped and on-premises networks have no cloud APIs, no managed block storage and often no automatic zone labels. Failure domains are whatever your facility defines from its physical power, cooling and network design. Releases, container images and patches enter the enclave through an internal process, which sets how quickly you can recover after a fault.
- Failure domains: define them from physical evidence such as separate power distribution units, cooling loops and network paths, then set broker.rack manually on each broker.
- Quorum: place controllers on separate hardware with independent power feeds. Confirm hardware independence with the facility team, because shared hardware can silently collapse two controllers into one failure domain.
- Latency and acknowledgments: latency is usually that of a single site, but internal congestion from backups or analytics jobs can slow acknowledgments. Measure acks under realistic load.
- Testable: pulling hardware, cutting power in a lab copy, restarting brokers, and failing a disk. Upgrade paths can be rehearsed fully offline.
- Dependencies: certificates, Kerberos or other authentication services, and monitoring must work without internet access, so test them in the same isolated network.
Ask who approves and imports Kafka releases and dependencies into the enclave, and time a full recovery drill against that process.
What drives the effort and cost of Kafka high availability work
This table lists the factors that most change the scope of a Kafka high availability project, with a note on how each one can be kept under control. Read each row as a question to answer during scoping, because the same design choice that protects one workload can add recurring replication and testing work across all of them.
| Driver | Why it changes effort or cost | How to keep it down |
|---|---|---|
| Failure domains and sites | Each additional rack, availability zone or data center adds broker placement rules, controller placement, network paths to test and failure scenarios to rehearse. | Agree the number of failure domains early, use rack awareness on one site before adding a second, and confirm what cross-site replication must protect. |
| Topics and partitions with different durability needs | Every topic needs its own replication factor, min.insync.replicas and producer acknowledgement settings, and each combination must be reviewed and documented. | Group topics into a small number of durability tiers and apply tier defaults, so exceptions are the only items reviewed individually. |
| Throughput and retention volume | Higher write rates and longer retention raise replication traffic, disk capacity, recovery time after a broker loss and the time needed for recovery tests. | Confirm peak throughput and retention per tier with the owning teams, and size recovery tests against the largest tier rather than every topic. |
| Client count and retry behavior | Each application needs review of producer retries, idempotence, consumer group settings and timeouts, and some will need code or configuration changes during failover tests. | Standardize client configuration templates, publish a shared set of producer and consumer defaults, and track the applications that deviate from them. |
| Environment count (development, test, production) | Failover behavior must be reproduced outside production, so each environment adds build, configuration management and test cycles. | Keep non-production environments on the same configuration templates, and scale them down where failure tests do not need full capacity. |
| Compliance evidence requirements | Assessors may ask for proof of replication settings, failover test results, access records and change history, and each item must be produced and kept current. | Agree the evidence list with the compliance owner before build, generate configuration exports and test reports as routine outputs, and confirm with the auditor which evidence they accept. |
| Recovery test scope | Full broker, controller and site failure tests across every topic take longer to run, need more people to observe, and produce more results to review. | Scope recovery tests to a representative subset of topics and clients for each tier, and repeat the full set only after major changes or upgrades. |
Controls that protect high availability settings from misuse
A Kafka cluster stays available only as long as its replication and leadership settings stay correct and the people who can change them are limited. The controls below focus on the access, secrets and audit decisions that keep those settings from being loosened by mistake or by someone without the authority to change them.
Restricting changes to replication settings
Replication factor, min.insync.replicas and unclean leader election determine whether acknowledged writes survive a broker loss. Grant ALTER and ALTER_CONFIGS on topics and brokers only to the team that owns recovery, and review which principals hold those permissions. Confirm that topic creation defaults cannot silently create topics with a replication factor below your standard.
Limiting controller access to recovery operators
In KRaft mode, controllers hold the metadata that defines the quorum and leader election. Limit who can reach controller listeners and who holds super user status, since super users bypass ACL checks. Keep recovery actions such as reassigning a quorum voter in a short list of named operators, and confirm that routine application principals have no path to controller endpoints.
Protecting quorum and inter-broker listeners
Controller quorum traffic and broker replication traffic carry the data and votes that high availability depends on. Place these listeners on private network paths, enable TLS on them, and avoid exposing them through shared load balancers or client-facing networks. Verify the listener configuration after every upgrade, because a changed listener map can expose a path that was previously internal.
Storing failover tooling credentials outside the cluster
Scripts and operators that promote consumers, reassign partitions or fail over to a second site need credentials with broad rights. Keep those secrets in a dedicated secrets manager rather than in broker configuration files, topic configs or shared repositories. Rotate them on a schedule, scope each one to the operations it needs, and confirm that a compromised failover credential cannot read or change unrelated topics.
Auditing configuration changes
Changes to replication factor, min.insync.replicas, unclean leader election or broker configs should leave a record of who made them and when. Confirm where authorization denials and administrative requests are logged, forward those logs to a store the cluster operators cannot edit, and alert on changes to high availability settings outside a planned change window.
Understanding KRaft authorization defaults
The StandardAuthorizer stores ACLs in the cluster metadata log, so ACL changes replicate with the rest of cluster metadata. Resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Confirm that this setting is false in production, and that every topic, group and cluster resource has explicit ACLs before clients depend on it. The full hardening approach is covered in the security hardening topic.
How to prove the cluster survives a broker loss without losing acknowledged data
The proof that a Kafka cluster survives a broker loss is a set of recorded tests run against the failure scenarios the design claims to tolerate, with pass criteria written down before the first broker is stopped. Start from the design document: list each scenario it covers (one broker down, one rack down, one controller down), the replication factor and min.insync.replicas values that should carry it, and the producer and consumer behavior expected in each case. A test that cannot fail against a written expectation proves little, so the expectations come first and the test run comes second. Run the positive scenarios in order of increasing severity. Stop one broker that leads partitions and confirm that producers using acks set to all keep succeeding while in-sync replicas remain at or above min.insync.replicas, and that consumers resume after leadership moves. Then stop an entire rack, which is where rack awareness either paid off or did not. Finally, stop an active controller and confirm the KRaft quorum elects a new leader and metadata operations continue. For each step, check whether partition leadership returns to preferred replicas after the broker restarts and rebalances, because a cluster can look healthy while every partition still sits on a fallback leader. The central check is data loss. Have the producer record every offset it received an acknowledgment for, then after each scenario read the topics from the earliest offset and compare the result with that record. Any acknowledged message that cannot be read is a failed test, regardless of how the cluster looked during the outage. Keep the producer load generator and the verification script under version control so the same comparison runs every time. Negative tests matter as much as positive ones. Stop enough replicas to drop the in-sync set below min.insync.replicas, then confirm that writes with acks set to all fail with an error instead of being accepted at reduced durability. Also confirm that unclean leader election is disabled on every topic where data loss is unacceptable, by checking the topic and broker configuration directly and by attempting a scenario that would require it. Ask your engineering team which topics are exempt from that rule and why, and record the answer. Treat the results as evidence. Store the test plan, the configuration snapshot, the timestamps of each injected failure, the producer and consumer logs, and the verification output together, and tie them to the release or change ticket. Repeat the full suite after each Kafka upgrade and after any change to broker counts, rack layout or replication settings, and schedule it as a standard change control item rather than an ad hoc exercise. Ask the operator how long a full run takes in your environment before you commit to a window.
What to monitor, back up and document after go-live
A Kafka cluster stays highly available only if someone watches the signals that show failing redundancy, protects the metadata that defines the cluster, and keeps the proof of recovery current. The four areas below are the controls that connect day-two operations to the design decisions made during the build.
Watch under-replicated and offline partitions
Under-replicated partitions show that followers are not keeping up with their leaders, so the replication factor you designed for is not currently in force. Offline partitions mean producers and consumers cannot use those partitions at all. Alert on both signals per cluster and per topic, then check whether the count falls back to zero after a broker restart or a rack maintenance window. Set the thresholds from your own service objectives, not from a default value copied from another team.
Track active controller changes
In KRaft mode the controller quorum decides leadership and metadata changes, so an unexpected change of active controller is an early sign of network instability, overloaded controllers or failed nodes. Record each election with its timestamp and the node involved, and correlate it with broker restarts and network events. Frequent elections over a short period deserve investigation before they become outages. Confirm the controller metrics exposed by your Kafka release and the exact alerting behavior you want with the vendor documentation for that version.
Review ISR shrink and expansion events
The in-sync replica set changes when a follower falls behind or catches up. A shrink lowers the number of replicas that can satisfy acknowledgments, which matters directly when min.insync.replicas is set to protect writes. Expansion events show recovery after a broker rejoins. Review shrink events by topic and broker, and confirm that each shrink was followed by an expansion within the time you expect. Repeated shrinks on one broker usually point to disk, network or garbage collection problems on that machine.
Monitor producer error rates
Producers are where availability loss becomes visible to applications. Errors such as timeouts, not-enough-replicas responses and leader changes tell you whether failover is hiding problems from clients or passing them through. Track error rates per producer application, per topic and per client version, since a single misconfigured client can look like a cluster fault. Confirm that retries and idempotence settings in each client match the delivery guarantees the business expects, and compare error spikes against the broker events above.
Back up configuration, topics and ACLs
Replication copies partition data between brokers, but it does not protect you from a bad configuration change, an accidental topic deletion or a deleted ACL, because those changes are replicated too. Back up broker and topic configuration, topic definitions with partition and replication settings, and ACL state from the cluster metadata. Store the exports outside the cluster and test restoring them into a separate environment. Confirm with your Kafka version how each export is produced and which secrets must be handled separately.
Plan upgrades against supported paths
Each upgrade is a planned failure test, so plan it against the release list and the rolling upgrade path before scheduling. The project publishes supported releases and states that rolling upgrades are supported from each release in the past year to the latest version. Check the upgrade notes for Java requirements on brokers and clients, and confirm that clusters older than the stated 3.x baseline move to 3.9.x first. After each upgrade, rerun your broker failure, controller failover and rack loss tests before accepting the new version into production.
Document the failure-domain map
The failure-domain map records which brokers, controllers and racks share power, network, storage or a hypervisor, and which failures the design is meant to survive. Without it, an operator cannot tell whether losing one rack leaves enough replicas and controllers for the quorum. Keep the map next to the quorum membership list, which names each controller node, its location and its role. Review both after every topology change, such as adding a rack or moving a controller, because stale documents are a common cause of wrong assumptions during an incident.
Keep runbooks and last test results
Recovery runbooks should cover broker replacement, controller loss, partition recovery and restoring configuration from backup, each with named owners and the commands your team will actually run. Pair every runbook with the date and result of its most recent test, and with the Kafka version it was written for. Test results show whether the design still works after changes, so record the failure scenarios run, the observed behavior and any gaps. Auditors and new operators will ask for this evidence, so store it where it can be found during an incident.
Questions buyers ask about Kafka high availability work
Can our own team build Kafka high availability without a partner?
Yes, if the team already runs Kafka in production and has the skills the design needs. The skills required are:
- Kafka operations, including KRaft controller quorum behavior, partition leadership, and topic configuration
- Network design across failure domains, including firewall rules and latency between sites
- Linux, TLS, and access control, since security is optional in Kafka and must be configured deliberately
- Failure testing, such as stopping brokers, isolating a rack, and checking producer and consumer behavior under load
Apache Kafka is licensed under Apache-2.0, and the project offers no paid edition, so the decision is about staff time and skills rather than a license fee. Confirm the team's experience with a failure drill in a non-production environment before committing to a schedule.
What drives the cost and effort of Kafka high availability work?
Cost follows the number of failure domains, controllers and brokers, the storage and network layout, the number of environments, and how many topics and client applications must be checked against the new settings. Replication factor and min.insync.replicas must be chosen per topic, and each choice affects how many clients need testing. The effort table on this page breaks the work into components such as design, build, testing, runbooks, and handover. Ask the quote to show:
- Which components are one-time and which recur, such as patching and failure testing
- Who performs each task: your staff, the partner, or both
- Which assumptions about environments and client applications the estimate depends on
How long does a Kafka high availability project take?
Duration follows the scope and the number of environments, such as development, test, staging, and production. It also depends on how many failure domains the design covers and how long your network and change approvals take. This page does not give a duration because it depends on your environment. The written quote states the timeline, the milestones, and the assumptions behind it. Ask for milestones tied to deliverables, so you can see when the design is approved, when failure tests run, and when production cutover is planned.
What does a fixed-scope Kafka quote cover?
A fixed-scope quote covers the work agreed in writing before the project starts. Most DeployOpen engagements are fixed-scope, so scope, deliverables, and timelines are fixed in the agreement. Check that the quote lists:
- The failure scenarios that will be tested and how results are recorded
- The documents handed over, such as the design record, runbooks, and configuration
- Items explicitly out of scope, such as application changes or upgrades after go-live
- Client responsibilities, such as approvals, test windows, and access
- Acceptance criteria, so both sides agree when the work is complete
What does our side need to provide for Kafka high availability work?
The work moves faster when the client provides accurate inputs and decisions early. The most important inputs are a failure-domain map showing racks, zones, or data centers and which brokers and controllers sit in each. Network and change approvals must be scheduled, because firewall and routing changes often have the longest lead time. Test windows must be approved by the business, since failure injection affects live traffic if it runs in production. Name owners for:
- Kafka platform operations
- Network and firewall changes
- Security and access control
- The application teams that produce and consume topics
Can a partner keep the Kafka high availability design after handover?
Yes, the design can be documented so your team owns it after handover. The handover should include the controller quorum layout, the replication and min.insync.replicas settings for each topic, the rack mapping, the failure test scripts, and the runbooks, with the reasons for each choice written down. Kafka's rolling upgrades are supported from each of the last three releases to the latest version, so an upgrade plan belongs in the handover. If you want continued help with support, upgrades, or repeated failure testing, those are available as retainers. Ask for the handover list to be part of the written scope.
Is high availability the same as disaster recovery for Kafka?
No. High availability keeps a cluster serving clients through the failure of a broker, a rack, or a zone within one deployment. Disaster recovery covers losing an entire site or region, and it usually depends on a second cluster. MirrorMaker 2 is built on Kafka Connect and can replicate topics with their configuration, consumer groups with offsets, and ACLs between clusters while preserving partitioning. Replication lag, failover steps for consumers, and the recovery targets your business sets all need design work. Confirm those targets with the business owners before the disaster recovery design is built.
What should engineers test before calling a Kafka cluster highly available?
Engineers should test each failure the design claims to survive, under realistic load, and record the results. Useful tests include stopping a broker while producers write, losing a rack, losing a controller, isolating a network segment, and filling a disk. For each test, check whether producers see acknowledged writes, whether the affected partitions elect new leaders, how long consumers take to resume, and whether min.insync.replicas behaves as designed. Keep the results with the design record, including the test date, the version tested, the observed behavior, and any follow-up actions.
How DeployOpen scopes, builds and hands over Kafka high availability
DeployOpen plans, builds, tests and documents Apache Kafka high availability inside your own infrastructure, starting from a written scope. The work covers KRaft controller quorum layout, replication settings, rack placement and recovery testing, and ends with runbooks your own team can operate.
How we scope
- Confirm the Kafka release line, with 4.1.2, 4.2.2 or 4.3.1 as the supported options to evaluate
- Agree which topics need which durability and availability targets, and which failures must be survived
- Record the broker, controller, rack and availability zone layout in the written scope before build starts
How we build
- Configure a KRaft controller quorum sized for the failure domains you need to tolerate
- Set replication factor and min.insync.replicas per topic class, and test producer acknowledgement behavior
- Apply rack awareness and run scripted broker failure and controller failover tests with recorded results
How we hand over
- Deliver configuration as code, with the quorum, replication and rack settings documented per cluster
- Provide runbooks for broker replacement, controller recovery and partition reassignment, tested with your operators
- Walk your platform team through the recovery test results and the open risks agreed in the written scope
How we support
- Offer ongoing support, upgrades and recurring recovery testing as a retainer after handover
- Review the high availability settings again when you move to a new supported Kafka release
- Confirm with us in writing what the retainer covers, since terms are set per engagement
