Kubernetes stacked etcd vs external etcd: which topology fits your control plane
Stacked etcd fits most teams that build a highly available kubeadm cluster and can accept coupled node loss. External etcd fits teams that need a control plane failure to leave the key-value store untouched and have a separate group to operate it. Decide before hosts are ordered, because the choice sets the host count. Kubeadm's default is stacked: kubeadm init and kubeadm join --control-plane create a local etcd member on each control plane node automatically, so the cluster needs no extra inventory. The Kubernetes HA topology docs describe this layout as simpler to set up and manage for replication, and it is the path most teams should start from unless a requirement below rules it out.
The trade-off is failed coupling. In a stacked cluster, losing one node removes both an etcd member and a control plane instance at once. The Kubernetes docs recommend at least three stacked control plane nodes for an HA cluster, which keeps quorum after one loss, but a second failure during maintenance still costs you the store and the API server together. Stacked etcd suits clusters where the same team patches nodes, owns the control plane and accepts that rebuilding a member means touching a control plane node. External etcd fits when a security or compliance owner requires that etcd data and its access paths sit apart from kube-apiserver hosts, or when a separate database group already runs consensus clusters. The cost is real: the kubeadm external etcd topology needs at least three control plane hosts and three etcd hosts, so you pay for six machines, plus a load balancer and backup tooling for both tiers. Confirm the etcd host count and recovery steps with your hardware vendor and your own change process before you commit.
How stacked and external etcd compare on hosts, failure impact and effort
This table sets stacked etcd and external etcd side by side on host counts, failure coupling, kubeadm defaults, commands, and the backup and operating work each one adds. Read the rows on hosts and failure impact first, then the rows on operations to judge the effort your team would carry.
| Factor | Stacked etcd | External etcd |
|---|---|---|
| Where etcd runs | An etcd member on every control plane node, sharing the node with kube-apiserver, kube-controller-manager and kube-scheduler | On a separate etcd cluster of its own hosts, with control plane nodes connecting to it |
| Minimum hosts for HA | At least three control plane nodes | At least three control plane hosts plus at least three etcd hosts, six in total |
| Failure coupling | Losing one node removes both an etcd member and a control plane instance at once | Losing one control plane node does not remove an etcd member, so redundancy loss is reduced |
| kubeadm default status | The default topology: kubeadm init and kubeadm join --control-plane create a local etcd member automatically | Not the default; requires preparing the etcd cluster and passing its endpoints and certificates to kubeadm |
| Setup and join commands | kubeadm init on the first node, then kubeadm join with --control-plane on the others, with no separate etcd step | Build the etcd cluster first, then configure the external etcd endpoints in the kubeadm configuration before init and join |
| Backup and restore responsibility | The team backs up etcd on each control plane node as part of the control plane, using etcdctl snapshot save and etcdutl for restores | The team backs up the separate etcd cluster on its own hosts, with etcdctl snapshot save and etcdutl, and plans restores for that cluster separately |
| Added hardware | No extra hosts beyond the control plane nodes, since etcd shares them | Three additional etcd hosts on top of the control plane hosts, and the associated network and storage for them |
| Added operations | Fewer components to patch and monitor, but etcd upgrades and maintenance are tied to control plane node maintenance | Two clusters to upgrade, monitor, secure and back up, with a stricter ordering for maintenance windows |
| What to confirm in a quote | Node sizing and disk performance for etcd on the control plane nodes, plus the backup schedule and restore test | Host count, etcd node sizing, certificate handling, backup location and who performs restore drills |
What runs on each node in a stacked control plane and an external etcd layout
Both layouts run the same Kubernetes control plane, but they place etcd members differently, and that placement decides what fails together. The cards below cover the components, host counts, load balancer, member count, kubeadm bootstrap behavior and upgrade order that a team must plan before building either topology.
Control plane components on each stacked node
In the stacked layout, every control plane node runs kube-apiserver, kube-controller-manager and kube-scheduler alongside a local etcd member. The Kubernetes HA docs recommend at least three such nodes. Before you commit, check that each node has enough CPU, memory and disk for etcd write latency as well as the API server load, because etcd shares those resources with the control plane on the same host.
Dedicated etcd hosts in the external layout
The external layout moves etcd onto its own hosts, separate from the control plane nodes. The kubeadm docs call for at least three control plane hosts and three etcd hosts, so the topology roughly doubles the host count. Ask who patches, monitors and backs up these etcd hosts, since they become a separate system with their own upgrade and access path.
TCP load balancer in front of kube-apiserver
Both topologies place a TCP load balancer in front of the kube-apiserver instances on port 6443, so kubectl, kubelets and controllers reach a single endpoint. The balancer is an extra component to build, health-check and make highly available itself. Confirm which load balancer your team will run, how it fails over, and that its health checks target the API server correctly.
Odd etcd member count
etcd needs an odd number of members so that a majority can still form after a failure. Kubernetes documentation recommends a five-member cluster in production, while three members is the minimum for HA. An even count adds hosts without adding failure tolerance. Count members in both layouts before you size hardware, because the stacked layout ties the member count to the control plane node count.
Local etcd member from kubeadm bootstrap
With stacked etcd, kubeadm init and kubeadm join --control-plane create a local etcd member on each control plane node automatically. That default is the simplest path to a working HA cluster. In the external layout, you must configure the etcd endpoints and certificates yourself before joining control plane nodes. Confirm which bootstrap path your runbook uses and test that each joined node reports the expected member.
Version skew rules for control plane upgrades
The skew policy requires kube-apiserver to be upgraded one minor version at a time, before kube-controller-manager, kube-scheduler and kubelets. Kubelets may run up to three minor versions behind kube-apiserver. In either layout, plan control plane upgrades node by node and keep a backup of etcd before each step. Verify the skew window against your Kubernetes version and your etcd upgrade procedure before scheduling a change.
How the coupled failure risk of stacked etcd plays out in practice
The failed coupling warning in the Kubernetes HA topology docs means that in a stacked cluster, losing one node removes two things at once: an etcd member and a kube-apiserver, kube-controller-manager and kube-scheduler instance. A three-node stacked cluster tolerates one failure, but after that failure it has two etcd members, so the cluster has no spare quorum margin until the node returns or is replaced. The same loss also leaves only two control plane endpoints behind the load balancer on port 6443. Both margins shrink in the same event, which is why the docs recommend at least three stacked control plane nodes for an HA cluster.
This risk matters most when the cluster is small, when replacing a failed node is slow, and when etcd carries heavy write load. A stacked cluster with three nodes and a manual rebuild process may spend a long window with no spare margin. Heavy write traffic, such as large numbers of object updates or frequent leader elections, makes slow disks on the surviving members more dangerous because they compound the quorum problem. Ask your team to measure how long it takes to remove a failed member, join a replacement with kubeadm join --control-plane, and confirm the etcd cluster is healthy again. That measured window is the number to compare against your availability target.
The risk matters less when node rebuilds are fast and automated, when hosts are provisioned from images, and when the cluster has enough members that the remaining quorum stays comfortable. Kubernetes documentation says etcd should run with an odd number of members and recommends five members in production, so a five-node stacked design tolerates two failures with a better margin than three nodes do. Under either topology, you need a backup plan for etcd data, using etcdctl snapshot save and restoring with etcdutl, tested on a schedule. Backups do not remove the coupling risk, but they are the recovery path when quorum is lost.
Which etcd layout fits a small team, a regulated estate or a shared platform
A small team with a limited host budget usually starts with the stacked etcd topology. kubeadm places an etcd member on every control plane node, so the cluster needs fewer machines and fewer components to bring up and maintain. The Kubernetes docs describe this layout as simpler to set up and to manage for replication, which matters when the same two or three engineers also run networking, storage and upgrades.
- Host count: A stacked HA cluster needs at least three control plane nodes, and etcd runs on those same nodes.
- Bootstrap:
kubeadm initandkubeadm join --control-planecreate the local etcd member automatically. - Operations the team must own: etcd snapshots with
etcdctl snapshot save, restores withetcdutl, and a load balancer on port 6443 in front of kube-apiserver. - Coupling risk: If one node fails, an etcd member and a control plane instance go down together, so capacity planning must assume that loss.
- Upgrades: Plan kube-apiserver upgrades one minor version at a time, ahead of the other control plane components and kubelets.
Before choosing, confirm with the team whether they can run etcd backups and restore drills on their own, and confirm with the vendor or distribution which etcd version and backup tooling their installer uses.
In a regulated environment, the question is usually how a single host failure is isolated and who can change each tier. The Kubernetes HA topology docs state that external etcd decouples control plane nodes from etcd members so losing one has less impact on redundancy. That separation is a design property you can describe to an assessor, but it does not by itself satisfy any specific control, so the architecture must be documented and tested.
- Failure isolation: Stacked etcd couples a control plane instance and an etcd member on each node, while external etcd keeps them on separate hosts.
- Host count: External etcd needs at least three control plane hosts and three etcd hosts, which doubles the machine count.
- Separation of duties: External etcd lets a separate team own the datastore hosts, access controls and backups.
- Encryption and access: Kubernetes stores Secrets unencrypted in etcd by default, so encryption at rest and least-privilege RBAC apply to either layout.
- Evidence: Keep etcd backup records, restore test results and change logs for the assessor.
Confirm with the assessor which evidence they expect for control plane availability and datastore protection, and confirm with the team which hardware, network and access controls the external etcd hosts will need.
A shared platform serves several application teams, so the control plane becomes a dependency for everyone. Teams often weigh external etcd here because a datastore on its own hosts can be sized, patched and monitored separately from the API servers. The cost is more hosts and a separate operating surface, which the platform team must staff.
- Blast radius: A shared datastore affects every tenant, so its failure domain and maintenance windows need explicit ownership.
- Scale of operations: External etcd adds host lifecycle, certificate rotation and backup scheduling for a second group of machines.
- Odd member counts: Kubernetes documentation says etcd should run with an odd number of members and recommends five members in production, which affects how many etcd hosts a shared platform budgets for.
- Tenant isolation: Kubernetes RBAC and Pod Security Standards govern tenant workloads on either topology, independent of etcd placement.
- Upgrade order: Kubelets may be up to three minor versions older than kube-apiserver, so shared clusters with long-lived node pools need a planned upgrade cadence.
Confirm with the application teams what availability and maintenance windows they expect, and confirm with the team whether they can run a separate etcd lifecycle alongside the control plane.
What upgrades, etcd backups and monitoring demand from each layout
This table compares the day-two work each kubeadm topology creates: upgrades, etcd backups and restores, health monitoring, node replacement and team skills. Read each row as recurring work versus one-time setup, and confirm the details against the Kubernetes operating documentation before you commit.
| Day-two task | Stacked etcd | External etcd |
|---|---|---|
| Control plane upgrade order | kube-apiserver goes first, one minor version at a time, then kube-controller-manager, kube-scheduler and kubelets within the skew policy. etcd members upgrade on the same nodes as the control plane, so each node change touches both. | Same required order for kube-apiserver, controllers, scheduler and kubelets. The etcd cluster is upgraded on its own hosts, so the etcd and control plane change windows can be planned separately. Recurring work increases with the number of hosts. |
| etcd backup with etcdctl snapshot save | Recurring. Snapshots are taken from a member on a control plane node, so the backup job depends on that node being healthy. Confirm the backup schedule and storage location in your own runbook. | Recurring. Snapshots are taken from an etcd host that holds no control plane role. Confirm which member is used and where snapshots are stored, since this is a separate job from the control plane. |
| Restore with etcdutl | Recovery can require restoring etcd data while control plane components on the same nodes are stopped, so the restore procedure touches the control plane. Rehearse it before you need it. | Restore is limited to the etcd hosts, which makes the procedure narrower. Rehearse it on a non-production cluster and confirm the exact steps in the restore documentation. |
| Monitoring etcd health and quorum | Track member health and leader changes on the control plane nodes. Quorum loss takes the control plane down with it, so alerts must cover both layers at once. | Track member health and leader changes on the etcd hosts, separately from the control plane. Alerting covers two node groups, so the monitoring scope and on-call routing grow. |
| Certificate and node replacement | Replacing a failed control plane node also replaces its etcd member. Confirm the member removal and rejoin steps before a live replacement, because the node role and the data role change together. | Replacing a control plane node leaves the etcd cluster alone, but replacing an etcd host requires removing and adding a member. Confirm the certificate rotation and member procedure for each node group. |
| Team skills required | Administration of kubeadm, etcd and control plane components on the same hosts. Fewer node groups to run, but each node needs both sets of skills. | Administration of etcd as a separate service, host provisioning for two node groups and cluster-wide certificate handling. More surface to cover, with clearer ownership boundaries. |
| One-time setup versus recurring work | One-time: initial kubeadm bootstrap with a local etcd member on each control plane node and load balancer on port 6443. Recurring: upgrades, backups, monitoring and replacements. | One-time: provisioning three etcd hosts alongside three control plane hosts, and their TCP load balancer setup. Recurring: the same tasks as stacked, repeated across both node groups. |
How a kubeadm HA control plane is built in either topology
This sequence takes a team from shared host preparation to a verified highly available control plane, showing where the stacked and external etcd paths diverge and what each step must prove before the next begins.
Confirm hosts, load balancer and node prerequisites
The platform or infrastructure team confirms the host plan before any install runs. Stacked etcd needs at least three control plane nodes, while the external topology needs at least three control plane hosts and three separate etcd hosts, according to the kubeadm HA topology docs. Place a TCP load balancer in front of kube-apiserver on port 6443, as the kubeadm high availability guide describes. Confirm time synchronization across all nodes, a ready container runtime and the network paths between nodes and the load balancer in your own environment, because the guide sets the requirement but your hypervisor, firewall and NTP setup determine whether it holds. Done means every node reaches the load balancer address on 6443 and every node reports consistent time.
Initialize the load-balanced control plane endpoint
Before the first control plane node starts, set the load balancer address as the shared control plane endpoint so every later join points at the same VIP or DNS name. Verify that the load balancer health check reflects the state of kube-apiserver on each backend. Do this in your own environment, since the health check method depends on your load balancer product. Done means the endpoint answers on port 6443 and the backend pool is ready to accept the first API server.
Run kubeadm init on the first node
On the first control plane node, the platform engineer runs kubeadm init against the shared endpoint. In the stacked path, this command creates a local etcd member on that node automatically, which the Kubernetes HA topology docs identify as the default behavior. In the external path, the same command is used but the configuration must reference the external etcd endpoints and certificates prepared in the previous step, so the first node never starts a local etcd member. Confirm the kubeconfig works and the node registers. Done means the first control plane node reports ready to the API server and the join command with the certificate key is captured for the next step.
Join the remaining control plane nodes
On each additional control plane node, run kubeadm join with the --control-plane flag using the captured token and certificate key. In the stacked topology, each join creates a local etcd member on that node, so the cluster gains etcd quorum capacity at the same time as the API servers. In the external topology, the join reads the etcd endpoints from the configuration and creates no local member. Join nodes one at a time and confirm each becomes ready before starting the next. Done means all control plane nodes show ready and the API server is reachable through the load balancer from each node.
Build and secure the external etcd cluster first
The external path requires this step before any control plane node is initialized. Build a three-member etcd cluster on the dedicated etcd hosts, following the Kubernetes guidance to use an odd member count, with five members recommended in production according to the Kubernetes etcd operations documentation. Generate and distribute the peer and client certificates, restrict access to the client port to control plane hosts, and confirm that each member reports healthy. This is a separate platform task owned by the team that operates the datastore, and it adds hardware and operating work that the stacked path avoids. Done means etcdctl endpoint health reports every member healthy before any control plane node points at the cluster.
Verify API health and etcd membership
After all control plane nodes join, confirm the cluster state from the outside. Check that every control plane node is ready, that kube-apiserver answers through the load balancer, and that the etcd member list matches the expected topology. In the stacked topology, the member list should show one member per control plane node. In the external topology, it should show only the dedicated etcd hosts. Record the output as the baseline for later audits. Done means the member list and node status match the topology you designed.
Test an etcd snapshot and restore procedure
Take an etcd snapshot with etcdctl snapshot save, which the Kubernetes etcd operations documentation names as the backup method, and store it outside the cluster hosts. Then confirm that the snapshot status reads correctly and rehearse the restore path with etcdutl in a non-production environment, because a backup that has never been restored is an unverified control. Note that in the stacked topology the snapshot covers the etcd data on the control plane nodes themselves, so a restore rehearsal also tests the coupled failure case described in the HA docs. Ask your team to confirm the snapshot schedule, storage location and retention in their own runbook. Done means a snapshot file is stored off-cluster, its status check passes and a restore has been rehearsed with a recorded result.
Document topology and etcd recovery ownership
Record which topology was built, which hosts run etcd, which team owns each layer and how an operator would respond if one control plane node or one etcd member failed. For stacked etcd, document that losing one node removes both an etcd member and a control plane instance. For external etcd, document the separate patching and certificate rotation schedule for the etcd hosts. This record is the input a compliance assessor or an operations lead will ask for, and DeployOpen prepares it as part of documentation work within an agreed scope. Done means the runbook names an owner for every host, backup and restore step.
What external etcd costs in hardware, operations and quoting
The hardware cost of external etcd is extra hosts. The kubeadm external etcd topology needs at least three control plane hosts and three etcd hosts, so you pay for six machines where stacked etcd needs three. The three etcd hosts do no control plane work, which means they add capacity, power, network ports, patching surface and rack or cloud footprint without adding scheduling capacity. Confirm with your platform team whether these hosts sit on the same hypervisor or failure domain as the control plane nodes, because a shared failure domain removes much of the redundancy you paid for.
The recurring cost is operations. A separate etcd cluster must be backed up with etcdctl snapshot save, restored with etcdutl, monitored on its own, and upgraded in step with the cluster it serves. The Kubernetes documentation also expects a five-member etcd cluster in production, so ask whether the quote assumes three or five etcd members, since that changes host count and backup scope. Patching, certificate rotation and snapshot storage recur for every host, and someone must be on call for quorum loss, which requires skills in etcd member management that stacked etcd operators may not already hold. One-time work covers design, build, the first backup and a restore test; recurring work covers patching, certificate rotation, snapshot retention and on-call coverage.
When you request a written quote, ask for the number of etcd hosts and their placement, the backup scope (which snapshots, where they are stored, how long they are retained, and whether Secrets encryption configuration is included in the backup), the restore test scope (a full rebuild from snapshot on separate hardware, or a documented procedure only), the monitoring and alerting covered for etcd, the upgrade and patching responsibilities for each tier, and which party provides on-call coverage and quorum-loss runbooks. Fixed-scope quotes should list these as deliverables with written acceptance criteria, so you can compare an external etcd proposal against a stacked etcd proposal on the same terms.
Questions buyers ask before choosing stacked or external etcd
Is stacked etcd the default kubeadm topology?
Yes. Stacked etcd is the default kubeadm topology, because kubeadm init and kubeadm join --control-plane create a local etcd member on each control plane node automatically. The Kubernetes docs describe stacked etcd as simpler to set up and manage for replication, which is why most small and mid-sized clusters start there. External etcd is a deliberate choice you make before bootstrapping, and it requires you to provision and wire up the etcd cluster yourself. Confirm the exact flags and configuration keys for your kubeadm version in the kubeadm high availability guide before you write your automation.
How many hosts does each etcd layout need?
The minimum differs by layout. A stacked cluster needs at least three control plane nodes, each running an etcd member alongside the control plane components. An external etcd topology needs twice the hosts: at least three control plane hosts and three separate etcd hosts, so six machines in total. Kubernetes also expects an odd number of etcd members and recommends five members in production, which means a production external cluster may grow beyond three etcd hosts. Both layouts need a TCP load balancer in front of kube-apiserver on port 6443. Ask your hardware or cloud team to confirm host sizing against your own workload, since the docs do not give a per-host capacity figure.
What is the coupled failure risk with stacked etcd?
In a stacked cluster, each node carries both a control plane instance and an etcd member. If one node fails, you lose both at once, so the cluster has less redundancy than the node count suggests. The Kubernetes HA topology docs warn about this failed coupling and recommend at least three stacked control plane nodes so the cluster can keep a quorum after a single node loss. External etcd reduces this coupling because a control plane node failure does not remove an etcd member. Stacked etcd still works well for many teams, but you should model the outage you can tolerate, test a node failure in staging, and document how you restore etcd from snapshots.
Can a stacked cluster be moved to external etcd later?
Moving between topologies is possible in principle, but it is a migration project rather than a setting you flip. You would stand up a separate etcd cluster, migrate the cluster data, and then reconfigure each control plane node to point at the external endpoints, with snapshots taken beforehand. The Kubernetes docs cover etcd backup with etcdctl snapshot save and restore with etcdutl, which give you the rollback path you need before any change. Plan a maintenance window, rehearse the move in a non-production copy, and confirm the exact procedure for your Kubernetes minor version with the upstream docs, since the operational steps and version support change over time.
What does external etcd add in hardware and operations?
External etcd adds cost in three places: the extra hosts, the operating effort, and the monitoring and backup work that goes with them. The topology doubles the host count from the stacked minimum, so you pay for three more machines or VMs plus their storage, network and patching. Each etcd host needs its own backup plan, certificate rotation, upgrades and health checks, and you must keep the etcd version compatible with the Kubernetes version you run. Those costs recur every month, while the initial build is one-time work. Ask any implementation partner to price these as separate line items in the written scope so you can compare them against the availability you actually need.
Can our own team run either layout?
A team with working Kubernetes administration skills can run either layout, but the external option asks more of it. Stacked etcd keeps the surface area smaller because the control plane and etcd share hosts and lifecycle. External etcd adds a second cluster to upgrade, back up, secure and monitor, and the team must be comfortable restoring etcd from snapshots under pressure. Also consider secrets: Kubernetes stores Secrets unencrypted in etcd by default, so whichever layout you choose, you must configure encryption at rest and restrict list and watch access. Assess your team's current skills against those tasks, and where gaps exist, DeployOpen can plan, implement, document and support the work under a fixed-scope agreement.
How long does the work take and what must we provide?
No duration is fixed for this work, because it depends on the number of hosts, your network and load balancer design, your backup targets and how much hardening you require. Before a timeline is agreed, you should provide the target infrastructure (VMs, bare metal or cloud accounts), host specifications, network ranges, DNS and load balancer details, the Kubernetes minor version you plan to run, and your backup storage location. Access to a staging environment for rehearsing upgrades and restores is essential. DeployOpen typically agrees scope, deliverables and timelines in writing before work starts, so ask for a written scope that lists each of these prerequisites and the acceptance tests for both the stacked and external designs.
How DeployOpen scopes, builds and supports your Kubernetes control plane
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure, including the Kubernetes control plane layout for stacked etcd vs external etcd. The work follows a written scope agreed before any build starts, so the topology decision is made against your hosts, failure tolerance and operating model.
How we scope
- Inventory the hosts, hypervisors or bare metal, and failure domains where control plane and etcd nodes would run.
- Record the topology choice, minimum host counts and load balancer placement for kube-apiserver on port 6443 in the written scope.
- Agree which etcd backup, restore and encryption-at-rest responsibilities sit with your team and which sit with DeployOpen.
How we build
- Bootstrap the control plane with kubeadm using the chosen stacked or external etcd topology and document each node role.
- Configure etcd with an odd member count, test snapshot save with etcdctl and a restore with etcdutl before handover.
- Test node-failure scenarios in a non-production environment to confirm quorum and API availability behave as the design states.
How we hand over
- Deliver topology diagrams, host inventory, runbooks for member replacement and the upgrade order defined by the version skew policy.
- Walk your operators through backup schedules, restore drills and alerting on etcd health so ownership is clear.
- Provide evidence of configuration and test results that your auditors or internal reviewers can examine.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers, scoped in writing separately from the build.
- Retainers can cover Kubernetes minor version upgrades, etcd maintenance and periodic restore testing on a schedule you agree.
- Ask any provider to confirm in writing who responds to control plane incidents, and how that differs for each topology.
