Adding and removing ClickHouse Keeper nodes

Adding and removing ClickHouse Keeper nodes without losing quorum

Keeper dynamic reconfiguration lets you add, remove or replace ClickHouse Keeper nodes through reconfig without restarting the whole ensemble, but only when enable_reconfiguration is set, and that setting is off by default. Plan to turn it on deliberately, on every Keeper node, before you need it, because a cluster that discovers the gap during an outage has no safe path to change membership. ClickHouse Keeper is the coordination service behind ReplicatedMergeTree: it holds the replication log, part metadata and the distributed DDL queue that tell each replica what to fetch and merge. When Keeper loses quorum, replicated tables cannot accept inserts that require coordination, so membership is a durability question as much as a server-maintenance task.

Teams usually need a membership change for one of three reasons: they are growing from a minimal ensemble, they are retiring a host that is failing or being rebuilt on new hardware, or they are moving Keeper nodes to new addresses or subnets. The replication guide recommends at least three Keeper nodes, on dedicated hosts in production, with about 4 GB of RAM each, so a three-node ensemble tolerates the loss of one node only. Replacing a failed node in a three-node ensemble therefore leaves no spare margin until the replacement has joined and caught up. Confirm with your own monitoring which Keeper node is leader and which followers are in sync before you start any change, and record the current membership so you can reason about the quorum arithmetic at each step.

Before running any reconfig command, confirm three things in your environment: that every Keeper node runs a version that supports dynamic reconfiguration as described in the ClickHouse Keeper documentation, that the server identifiers and hostnames for the new node are unique and resolvable from every peer, and that your automation will not reintroduce the old host from a stale configuration file. Use hostnames in the server entries rather than raw IP addresses, because changing a hostname is treated as removing and re-adding a server. A hostname that resolves to a new address after a host move is a supported path, while editing the IP in the config is not. Practice the full sequence on a test ensemble with representative configuration first, and keep a documented rollback that restores the previous membership if the new node fails to sync.

Which membership change fits your situation

Adding a node grows the ensemble, for example to move from a three-member minimum setup to a larger membership, or to place a member in a separate failure domain. The goal is more fault tolerance for Keeper, not faster replicated writes, so confirm that reason before you change membership. The main risk is that a new member changes the majority size during the change, and an addition made while an existing member is already down can leave the ensemble with less headroom than before.

Start from a healthy ensemble: Confirm every current member is running, reachable and has a leader before you add anything.
Enable reconfiguration first: Dynamic reconfig works only when enable_reconfiguration is set, and it defaults to false, so plan a configuration change and restart on the existing members.
Use a hostname for the new member: Resolve the name identically from every node so membership does not depend on an address that may change later.
Confirm the new member joins as a voter: Check that it appears in the membership and catches up before you rely on it for quorum.
Update ClickHouse server settings: Make sure servers that point to Keeper know the new member, so the ensemble list does not drift between hosts.

Before you start, check the current membership, the reconfiguration setting on every member and name resolution from each node.

Configuration options that control reconfiguration

This table compares the settings that decide whether Keeper membership can change at runtime and how each server is identified in the ensemble. Read each row as a precondition: a setting that is wrong usually fails at the moment you try to replace a host, not when the configuration is first written.

Option	Use it when	What breaks if set wrongly
enable_reconfiguration	You plan to add or remove Keeper nodes without a full ensemble restart. The default is false, so it must be set deliberately on each node.	With the default false, reconfig commands are rejected, so membership changes require stopping and reconfiguring the whole ensemble. Set it inconsistently across nodes and behavior during a change becomes hard to predict.
Server identity (server ID) in the ensemble definition	Each Keeper node needs a unique numeric ID that stays with that server for its life in the ensemble. Confirm the exact layout in the Keeper docs.	Reusing an ID for a different host, or renumbering existing nodes, makes the ensemble treat the server as a different member. Confirm how the ID is recorded before you replace a host.
Hostname for each server	Preferred for every server entry. Changing a hostname is treated as removing one server and adding another, so keep names stable across the host's life.	Renaming a host forces a remove and re-add cycle. Plan that as a membership change, not a cosmetic edit, and confirm the sequence in the Keeper docs.
Raw IP address for each server	Avoid in server entries where the host may be replaced or re-addressed. Confirm any case where IPs are unavoidable.	A new IP on a replacement host looks like a new member to the ensemble. Changing addresses in place can also require the same remove and re-add work as a rename.
reconfig add and remove commands	Use them to change membership on a running ensemble once enable_reconfiguration is active. Confirm the exact command syntax and the required arguments in the Keeper docs.	Removing a server before the new one is healthy, or removing enough servers to lose quorum, leaves the ensemble unable to commit writes. Verify quorum after each step.
Ensemble size during change	Keep an odd number of voting members and confirm the quorum arithmetic for your target size in the Keeper docs. The replication guide recommends at least three nodes in production.	An even count or a shrink below the quorum majority stalls replicated writes. Plan one membership step at a time and check the ensemble's status between steps.
Keeper configuration on every node	Each node's config must match the membership you intend to reach. Confirm which entries must be updated locally and which are applied through reconfig.	Stale entries on nodes that were not updated can cause a restarted node to rejoin with an old view of the cluster. Compare configs on all nodes after each change.

How to change Keeper membership step by step

This sequence changes Keeper membership one server at a time while the ensemble keeps a voting majority. It produces an ensemble with the replacement hosts in place, checked against replica health at each stage, with a rollback point ready before every change.

Confirm ensemble health and quorum
Before any change, the operator checks that every Keeper node is running, that one node reports as leader, and that the ensemble has a voting majority with no member lagging. Run the same checks against your ClickHouse replicas, confirming that ReplicatedMergeTree tables show no readonly replicas and no growing replication queue. Record the server IDs, hostnames and ports of the current ensemble. Done means you have a dated record of a healthy baseline, and the change stops if the ensemble is already short of quorum.
Take a rollback point before changing
Copy the configuration files and the Keeper log and snapshot directories from every node, and store the copies outside the hosts being changed. Record the exact configuration each node uses so it can be restored if a reconfiguration stalls. Confirm that a recent ClickHouse table backup exists and has been restored in a test environment, because Keeper holds replication coordination and not the table data itself. Done means the copies are readable, the restore steps are written down, and a named person can approve a rollback.
Enable reconfiguration on every node
Set enable_reconfiguration to true in the Keeper configuration on each node, because dynamic reconfiguration stays off until this setting is enabled. Plan this as a rolling restart, one node at a time, and confirm each node rejoins before restarting the next so the ensemble keeps its majority. Use hostnames in the server list, since the documentation treats a hostname change as removing and re-adding a server. The ClickHouse Keeper documentation covers the exact setting and command syntax. Done means every node reports the same membership and accepts reconfig commands.
Add the new server before removal
Start the new Keeper node with its own server ID, hostname and storage paths, then add it to the membership with reconfig while every existing node remains in the ensemble. Adding first keeps the current members in place, so the ensemble still has its full voting set if the new node runs into trouble. Confirm the new node appears in the membership list and that the ensemble still reports a leader before continuing. Done means the membership lists the new server and the ensemble stays healthy with the addition in place.
Verify the new member joins and replicates
Check that the new node has caught up with the leader and is listed as a participant in the ensemble status output. Confirm that ReplicatedMergeTree tables have replicas on the new host, then write a marker row to a test table on one replica and confirm it appears on the others. If a replica on the new host is behind, the replication guide describes automatic resync from a healthy replica, so watch the replication queue until it drains. Done means the new member is caught up, the test write replicates, and no table shows a stuck queue.
Remove the retired server from membership
Only after the previous step passes, update the Keeper connection settings on each ClickHouse server so they list only the surviving members, restarting servers as needed. Then remove the retired server from the membership with reconfig. Stop its Keeper process last, after the membership no longer lists it. Done means no ClickHouse server still points at the retired host, the remaining nodes agree on the leader, and the membership shows only the planned servers.
Confirm health after the final change
Repeat the baseline checks from the first step against the final membership: a leader is present, the voting majority is healthy, no member lags, and replicated tables show no readonly replicas or growing queues. Confirm that scheduled backups still run against the new membership and that monitoring and alerts refer to the new hostnames. Keep the rollback record until these checks pass and the team signs off the change. Done means the live state matches the planned membership and the change record is closed.

Why hostnames beat IP addresses for Keeper identity

Keeper tracks each ensemble member by the identity it was configured with, so the address or name you choose on day one determines how painful later changes will be. The cards below cover the consequences of that choice and the planning it forces across DNS, certificates and Kubernetes.

A hostname change is a removal

The ClickHouse Keeper docs state that changing a server's hostname is equivalent to removing that server and adding it back. Keeper does not treat the renamed host as the same member, so the ensemble loses a voter and gains a new one. Before any rename, confirm the replacement plan includes a reconfig removal and a re-add, and that quorum survives the intermediate state.

IP changes cause the same effect

A new IP address on an existing hostname is not automatically safe if the configuration records the address itself. Rescheduling a pod, moving a VM or re-addressing a subnet can make a running member look like a different server to the rest of the ensemble. Check where each node's address is written, including server lists on every peer, and test a re-addressing change in a non-production ensemble first.

Plan DNS and certificates together

The hostname you choose becomes the name used in DNS records, in the Keeper configuration on every peer and in any TLS certificate used for secure Raft traffic. If these three drift apart, a node can join with a valid name that a peer cannot verify. Agree on one naming scheme, issue certificates for the exact names, and confirm forward and reverse DNS resolve consistently from every host.

Stable pod identity on Kubernetes

Pods receive new IP addresses when they are rescheduled, so an address-based Keeper configuration breaks whenever the scheduler moves a member. A StatefulSet with a stable headless service gives each pod a predictable DNS name that survives restarts. Confirm that the Keeper pods keep their names, persistent volumes and service records during node drains, upgrades and autoscaling events.

Keep the membership list in sync

Each Keeper member needs to know the others, and a reconfiguration is only as good as the list it starts from. An outdated server entry on one host can cause confusion during a replacement, even when the new host is healthy. Keep a single source of truth for the membership list, render it from that source for every node, and record which list is live after each change.

Rehearse the rename before production

Hostname and address changes are rare, so teams often attempt them under pressure with no practice. A rehearsal on a copy of the ensemble shows the exact reconfig commands, the time each step takes and where quorum is at risk. Ask your partner or internal team to document the rollback path, including how to re-add the original identity if the new name fails to resolve.

How much effort does a Keeper membership change take and what do we need?

Can our own team change Keeper membership without outside help?

Yes, if your engineers already run ClickHouse replication and can work through a staged change in a test environment first. The reconfiguration itself is a small set of Keeper commands, but the risk sits in the surrounding steps: confirming quorum before and after each change, keeping the ensemble's server IDs and hostnames consistent, and proving that replicated tables recover. Teams with no recent Keeper experience, or with a production ensemble that already shows instability, usually benefit from a second pair of hands for the first change. Your team should be able to explain what happens to quorum if a second node fails during the window, because that answer decides whether the change is safe to start.

What drives the effort of a Keeper membership change?

Effort rises with four factors. The first is ensemble size: each added or removed node is another step that must keep a majority of voting members healthy. The second is the platform: Kubernetes adds pod scheduling, stateful set and DNS work, while bare metal or virtual machines add host provisioning and network changes. The third is the change window, since a change that must finish without affecting replicated tables needs more rehearsal and a clear rollback point. The fourth is testing on a pre-production cluster with representative replicated tables and realistic load. Ask for a breakdown of the effort against each of these factors, not a single total.

What does a Keeper change need from our environment?

You need to provide four things before work starts. Administrative access to the Keeper hosts or pods, the ClickHouse servers that use the ensemble, and the configuration management that controls them. A written maintenance window with an agreed rollback decision point. A pre-production cluster with a Keeper ensemble and ReplicatedMergeTree tables that resemble production in table count, data volume and insert pattern. Current inventory of server IDs, hostnames, IP addresses and DNS records. Finally, named contacts who can approve a pause in replication if the change goes wrong. Confirm with your platform team how DNS and certificates are managed, because both affect the steps.

What should a partner quote include for a Keeper change?

A useful quote states the written scope before any figure appears. It should list the inventory and current-state review, the reconfiguration plan for each node, the pre-production rehearsal, the production change steps, the verification queries against replicas, the rollback procedure, and the handover notes. Separate one-time work from anything recurring, such as a retainer for later membership changes or upgrades. Ask which items the client must supply, which assumptions the estimate depends on, and what happens if the rehearsal shows a different risk. DeployOpen plans most engagements as fixed scope, with scope, deliverables and timelines agreed in writing before work starts, so compare quotes on the same written scope.

How long does a Keeper membership change take?

No duration is fixed by the supplied facts, so ask for one in writing rather than relying on a general figure. The schedule depends on how long the pre-production rehearsal takes, how many nodes change, how many replicated tables must be verified, and how much of the window your change board requires for approval. Ask the partner to show which phases are sequential and which can run in parallel, and what the schedule assumes about access approvals. Keep a buffer for a failed step and a rollback, because a membership change should not be rushed to meet a window.

When should we bring in outside help for Keeper changes?

Bring in outside help when the change touches production replication and your team has not performed a Keeper reconfiguration before, when the ensemble runs on Kubernetes with dynamic addresses, or when you need to retire a node that holds data your replicas depend on. Help is also useful when a failed replica or stalled replication already exists, because mixing that recovery with a membership change multiplies risk. Procurement teams should ask whether the partner will also test the recovery path, since a membership change that cannot be rolled back is a planning failure, not a technical surprise.

Why does the choice of hostnames affect the effort?

Hostnames reduce the number of steps when membership changes, because the documentation for ClickHouse Keeper recommends them over raw IP addresses. Changing a hostname is treated as removing a server and adding it again, so an IP-based design forces that remove and re-add sequence whenever an address changes, for example after a host is rebuilt. Plan DNS records and naming before the first change, and confirm with your network team how records are updated and cached. This decision is usually cheap to make early and expensive to correct later.

Does a Keeper change need a security review?

Yes, because Keeper controls coordination for replicated tables, and membership changes alter who can vote. Confirm how the ensemble's internal traffic is encrypted and authenticated, how new nodes are admitted, and who can run the reconfiguration commands. Record these controls in the change ticket so auditors can trace them later. Ask the partner to show how access is granted for the change and removed afterwards. For the exact control requirements of your framework, confirm them with your compliance lead and the assessor, since DeployOpen prepares evidence but does not issue audit reports or certifications.

What a scoped Keeper change covers

A Keeper membership change is scoped in writing before any host is touched, because the scope decides which steps are safe and in what order. DeployOpen agrees the scope, deliverables and timeline as a fixed-scope engagement, so the client knows exactly which Keeper nodes are added or removed, which enable_reconfiguration setting changes on which hosts, and what evidence is handed over at the end. The scope also states which work is not included, such as changes to ReplicatedMergeTree tables or the Distributed tables that sit on top of the ensemble.

The scope needs four inputs from the client. The first is the current topology: how many Keeper nodes run today, their server_id values, and which ClickHouse replicas depend on them. The second is the hostnames each node will use, confirmed in DNS or the inventory, because the documentation recommends hostnames over raw IP addresses and a changed hostname counts as removing and re-adding a server. The third is the target size, with the ClickHouse replication guide's guidance of at least three Keeper nodes on dedicated hosts as the baseline to confirm against the client's own risk model. The fourth is a test environment with representative data, where the sequence can be rehearsed before production. Ask the client to confirm which of these exist and who approves the production window.

Some costs are one-time and some recur. The one-time work is the planning, the rehearsal, the membership change itself and the evidence pack that records the before and after state. Recurring work is whatever the client keeps after handover, such as monitoring Keeper health, repeating reconfiguration after future host replacements, and keeping the test environment aligned with production. Ask for a quote that separates these lines, names who performs each task, and states what the client must provide, so that the effort can be compared directly with an in-house team's time.

How DeployOpen delivers ClickHouse Keeper changes

DeployOpen plans and carries out Keeper membership changes inside infrastructure you control, starting from a written scope that names the current ensemble, the target membership and the quorum plan. Each change is rehearsed against a non-production copy of your cluster before it touches production, and the result is documented so your team can repeat it.

How we scope

Inventory the current Keeper ensemble, server IDs, hostnames and the ClickHouse clusters that depend on it.
Agree in writing which nodes are added, removed or replaced, and the order each change follows.
Confirm whether enable_reconfiguration is already set, or whether a restart window is needed to enable it.

How we build

Prepare hostname-based configuration for every Keeper node so later host swaps do not look like removals.
Rehearse each membership step on a staging ensemble, checking that quorum holds at every stage.
Verify replicated tables are healthy before, during and after each change using system tables and Keeper status.

How we hand over

Provide a runbook with the exact reconfig commands, rollback steps and quorum checks used in the work.
Document the final ensemble layout, DNS records, and the configuration files changed on each host.
Walk your operators through a supervised change so they can repeat the procedure without us.

How we support

Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts.
Ongoing support, upgrades and continuous testing are available as a retainer after handover.
Ask any partner for the written scope, the rehearsal plan and the rollback path before you agree a quote.

Adding and removing ClickHouse Keeper nodes