Kubernetes high availability

The main difference between the two kubeadm high availability topologies is what a failed machine takes with it. In a stacked design, each control plane node runs kube-apiserver, kube-controller-manager, kube-scheduler and an etcd member together. Losing one node removes a control plane instance and an etcd member at the same moment. In an external design, etcd runs on its own three or more nodes, separate from the control plane. Losing a control plane node removes an API server, controller manager and scheduler instance, but etcd keeps its full membership and the data path stays intact. Before choosing, write down which failures you need to survive and whether the two should be coupled.

Start with the odd member count, because it determines how many failures the cluster can absorb. etcd uses majority voting, so a write needs agreement from more than half of the members. Three members tolerate one loss, five tolerate two. An even count adds cost without adding tolerance: four members still tolerate only one failure, the same as three. Kubernetes documentation calls for an odd number of etcd members and recommends five in production. Confirm the exact quorum and member-change behavior for the etcd version you plan to run, since the documentation for each release describes it in detail.

When you lose one member of a three-member etcd cluster, the cluster keeps quorum. The API server keeps serving reads and writes, and controllers keep reconciling. In a stacked design, that same loss also removes one API server endpoint behind the load balancer, so the load balancer must fail health checks on port 6443 quickly enough that clients are redirected. Losing a second member in a three-member cluster breaks quorum. etcd then stops accepting writes, the API server cannot persist changes, and controllers cannot update objects. In a stacked design, two control plane nodes are gone, so you have also lost most of your API capacity. Plan a runbook for restoring members and confirm how your version reports quorum loss through etcdctl endpoint health.

A whole zone failure tests the placement decision. If a three-member stacked cluster has one control plane node per zone, losing a zone is the same as losing one member, and quorum survives. If two nodes sit in one zone, a single zone outage breaks quorum. External etcd lets you place etcd members across zones independently of control plane nodes, but it does not fix a bad layout. Three etcd members in two zones still fail when the zone holding two members goes down. Map every member to a failure domain on paper before provisioning, and ask your infrastructure team to confirm that zone labels and anti-affinity match that map.

Restoring a cluster after quorum is lost is a different job from routine failover. Recovery usually means restoring from an etcd snapshot taken with etcdctl snapshot save and restoring it with etcdutl, which is why backup schedules and restore drills belong in your planning, as covered in the etcd backup and disaster recovery section. Stacked clusters also require you to rebuild the control plane components on replacement nodes, while external clusters let you replace etcd members and control plane nodes separately. Ask your team to rehearse each scenario in a staging cluster and record how long their own runbook takes, rather than relying on general estimates.

Placing three control plane nodes in three different zones or racks is the decision that gives a Kubernetes cluster zone-level resilience. Three members in one zone satisfy the quorum arithmetic on paper, but a single zone outage, a top-of-rack switch failure or a shared hypervisor fault takes all of them down together. The etcd members that back the cluster need an odd count so a majority can survive, and that majority only helps if the members are not all behind the same point of failure. For a stacked topology, each kube-apiserver, kube-controller-manager, kube-scheduler and etcd member should sit in a different failure domain. For an external etcd cluster, apply the same rule to the etcd nodes and keep the API server nodes spread the same way.

The first thing to confirm is what your platform calls a failure domain and whether it is independent in practice. In a cloud, a zone may share a network fabric, a control plane or a storage backend with other zones in the same region, and the vendor documentation should state which services are zonal and which are regional. In a data center, check whether two racks share power feeds, aggregation switches or a storage array. Ask the infrastructure team to show you the physical or logical dependencies, then map each etcd member to a domain that does not share them. Also confirm the latency between domains, because etcd is sensitive to disk and network round-trip times, and a domain that is far away may degrade write performance even when it survives.

Worker nodes need the same thinking, with more room for variation. Create node pools per zone or rack rather than one large pool that the scheduler can place anywhere, so you can control the size of each domain and drain one without affecting the others. Label every node with its topology values, using the well-known topology.kubernetes.io/zone label where your cloud provider or kubelet sets it, and add a custom label such as a rack identifier for on-premises racks. Confirm which labels your distribution applies automatically and which you must set during provisioning. Without accurate labels, the scheduler cannot make placement decisions based on the domain a node belongs to.

Topology spread constraints and pod anti-affinity are the two mechanisms that keep replicas apart. A topologySpreadConstraints entry with a topologyKey of the zone label and a maxSkew value limits how unevenly pods of one workload can be distributed across domains, and whenUnsatisfiable decides whether the scheduler blocks placement or accepts a skew. Pod anti-affinity can require or prefer that replicas avoid nodes that already run a matching pod. Hard requirements prevent co-location but can leave pods pending during a partial outage, so choose the mode per workload after testing. Set the constraints in your deployment templates, and confirm that the same rules are enforced for stateful sets, which need a storage class that can provision volumes in the zone where the pod is scheduled.

Size each domain so the cluster keeps running when one fails. If a zone is lost, the remaining replicas must fit on the surviving nodes at the resource requests they carry, and the control plane must still hold quorum with its remaining members. Work through the arithmetic with your own workload figures: total requested CPU and memory per replica set, the number of replicas you require to serve traffic, and the headroom your autoscaler and cluster autoscaler need to add nodes in time. Exact zone counts, node sizes and capacity margins depend on your infrastructure, workload and availability targets, so test them. Run a controlled drain of one domain in a staging environment, observe whether pods reschedule and whether the API server stays reachable through its load balancer, and record the results as the basis for your design.

Run a failure test plan in staging before production, and treat every test as a written hypothesis with an expected result recorded before you run it. The goal is to prove that the control plane and your workloads behave as the high availability design says they will, not to discover the design during a real outage. Use the same topology as production where you can: the same number of control plane nodes, the same load balancer in front of kube-apiserver on TCP port 6443, the same node labels and the same failure domain layout. A test on a smaller or differently spread cluster proves very little about production.

Start with the control plane. Stop one control plane node and confirm that the API remains reachable through the load balancer, using a health check from outside the cluster network as well as kubectl from an admin workstation. Check that the load balancer removes the failed backend and that kubectl commands continue to succeed without manual kubeconfig changes. Then stop one etcd member and confirm that writes still succeed, for example by creating and deleting a test ConfigMap and reading back its resourceVersion. In a stacked topology the etcd member lives on the same node as kube-apiserver, so the first test also removes an etcd member. Confirm the member list with etcdctl and the health of the remaining members before you bring the node back.

Next, drain or isolate a whole failure domain. Cordon and drain every node in one zone or rack, or cut its network access at the hypervisor or switch level if you want a closer simulation of a zone outage. Confirm that Deployments and StatefulSets reschedule their pods onto surviving nodes, that topologySpreadConstraints, pod anti-affinity and PodDisruptionBudgets are honored, and that no workload is pushed onto a node that breaks its own constraints. Pods that cannot be placed should remain Pending with a clear scheduler event rather than silently overloading a survivor. Record how long each workload took to return to its desired replica count, and compare that with the recovery expectations your application owners have agreed.

Test the failure you hope never happens: loss of quorum. Stop enough etcd members to break majority, then confirm that the API server and etcd refuse writes instead of accepting them on a minority. There should be no split brain, two leaders making conflicting changes, or corrupted data after the members return. Confirm the behavior by reading cluster state after recovery and comparing it with the last known good state. Then rehearse a restore from an etcd snapshot taken with etcdctl snapshot save, and restore it with etcdutl into a fresh data directory on a clean node. Verify the restored object counts and a sample of Secrets, then document each step, including who authorizes the restore and where the snapshot is stored.

Add two credential checks to the same drill. Confirm that the kubeadm join token used for adding nodes has not expired or, if it has, that a new one can be created and used, and list the expiry of every control plane certificate with kubeadm certs check-expiration. Also record the certificate authority location and who can renew it. Repeat the full plan after each Kubernetes minor version upgrade, because control plane components, etcd and add-ons change between releases. Keep a dated results log with expected and observed outcomes, any deviations, and the fix for each one. That log is your evidence that the cluster was tested, and it becomes the baseline you compare against the next run.