Why a three-node ClickHouse Keeper cluster is the starting point for replication
If you run or plan self-managed ClickHouse with replicated tables, you need a coordination service that agrees on which replica holds which data and what changes have been committed. ClickHouse Keeper fills that role. It stores replication metadata and uses a Raft consensus protocol, so the ensemble keeps working only while a majority of its members are up. That majority rule is why an odd number of nodes is the usual minimum: three nodes tolerate one failure, and five nodes tolerate two. The ClickHouse replication guide recommends at least three Keeper nodes for production, placed on dedicated hosts, with about 4 GB of RAM on each. Treat those figures as the documented baseline and confirm them against your own data volumes and load before you size hardware.
ClickHouse Keeper replaces ZooKeeper as the coordination layer for ReplicatedMergeTree tables, so new deployments no longer need a ZooKeeper ensemble. The ClickHouse server itself is open source under the Apache License 2.0, but that license covers the server software only. Whether Keeper runs on the same hosts as your data nodes, on separate machines, or in a separate failure domain is a planning decision you make on its own, and it determines how a single host failure, a rack failure or a maintenance window affects replication. Before you build anything, write down your replica count, your availability target and where each Keeper member will sit. Those answers drive the server IDs, hostnames and disk layout covered in the rest of this page.
Layout choices for the ensemble: node count, hosting and async replication
This table compares the main layout decisions for a Keeper ensemble: node count, where the nodes run, and whether async_replication is enabled. Read each row as a question to settle before building, and use the confirm column to check the item against your versions and environment. The main takeaway is that three dedicated nodes on Keeper 23.9 or later with async_replication enabled is the documented starting point, and the other options change failure behavior or operational risk.
| Decision | Option | When it fits | Trade-off and what to confirm |
|---|---|---|---|
| Node count | Three nodes | Most single-site ClickHouse clusters that need replication and can tolerate one Keeper node down at a time. The replication guide recommends at least three. | Keeps quorum through one node loss. Confirm that all three nodes sit in separate failure domains such as racks or availability zones. |
| Node count | Five nodes | Larger clusters, multi-zone designs or higher tolerance for planned maintenance overlapping with a failure. | Tolerates two node losses but adds hosts, monitoring and reconfiguration work. The Kubernetes etcd guidance recommends five members in production, cited here only as a comparison for odd-numbered quorum sizing. |
| Node count | Two or four nodes | Not a recommended production layout for Keeper quorum. | An even count gives no extra failure tolerance over the next odd size down. Confirm this with your own failure scenarios before accepting any even-number design. |
| Hosting | Dedicated hosts | Production clusters where Keeper must stay responsive when ClickHouse servers are under heavy merge, query or insert load. The replication guide recommends dedicated hosts in production. | Adds machines and their patching, backup and monitoring cost. Confirm capacity for about 4 GB of RAM per node, as stated in the replication guide, and who owns the hosts. |
| Hosting | Co-located with ClickHouse servers | Small test or pre-production environments where host count is constrained and load is light. | Keeper competes with ClickHouse for memory, CPU and disk I/O, which can slow quorum decisions during heavy activity. Not advised for production without load testing. |
| Hosting | Shared disks or shared storage for log and snapshot paths | Rarely appropriate. Only consider when storage controls guarantee the Keeper log and snapshot paths are not shared with other workloads. | Keeper writes its transaction log and snapshots synchronously, so contention on shared disks raises latency. Confirm separate, low-latency volumes for the log and snapshot paths on each node. |
| Async replication | async_replication enabled | All Keeper instances run version 23.9 or later. ClickHouse recommends enabling it in this case because it improves performance without downsides. | Requires every Keeper instance to be upgraded to a supporting version first. Confirm the exact versions on every node before you set the flag. |
| Async replication | async_replication disabled (default) | Mixed-version ensembles, or environments where backward compatibility with older Keeper instances is still required during a transition. | This is the default for backward compatibility, so it is easy to leave in place by accident. Confirm the setting in your Keeper configuration and plan to enable it once all nodes are upgraded. |
Building the ensemble step by step
This sequence turns three prepared hosts into a running Keeper ensemble with a confirmed leader, secured Raft and client traffic, and async replication enabled, so ReplicatedMergeTree tables have a coordination layer to rely on.
Confirm version compatibility across all nodes
Before any configuration, confirm that the ClickHouse server and Keeper binaries on all three hosts run the same release line and that the release supports
async_replication, which the Keeper documentation says requires version 23.9 or later on every Keeper instance. Choose the release from the ClickHouse production guidance and the security policy, then pin it so no host upgrades on its own. Check withclickhouse-keeper --versionon each host. Done means all three report the same version string, and that string is one you have already reviewed in the changelog.Provision dedicated hosts and separate disks
Provision three hosts that run only Keeper, not ClickHouse data nodes, because the replication guide recommends dedicated hosts for production. Give each host roughly 4 GB of RAM as the guide suggests, and place the Keeper log directory and the snapshot directory on separate disks so that snapshot writes do not compete with log fsync. Confirm the mounts with
df -handlsblk. Done means each host has two mounted, writable Keeper paths on different devices and matching time synchronization.Assign server IDs and stable hostnames
Give each node a unique numeric
server_idin its Keeper configuration, starting at 1, and register a stable DNS name for each host in your internal zone. Use hostnames rather than IP addresses in the Raft membership so that replacing a host later does not change the ensemble definition. Verify resolution from every node withgetent hostsagainst all three names. Done means each name resolves to the expected address from every other host, and eachserver_idappears exactly once across the three files.Write raft_configuration and storage paths
Write the
keeper_serversection on each host with its ownserver_id, the client port, thelog_storage_pathandsnapshot_storage_pathpointing to the separate disks, and araft_configurationblock listing all three servers with the same hostnames and Raft ports. Keep the server list identical on every node. Enableasync_replicationin the same block when the version check passed. Done means the three files differ only inserver_id, and a diff across them shows no other mismatch.Enable secure Raft and client TLS
Create a certificate authority or use your internal one, issue a certificate per node whose subject or SAN matches its hostname, and configure TLS for Raft peer traffic and for client connections from ClickHouse servers. Restrict file permissions on private keys so only the ClickHouse service user can read them. Confirm the chain with
openssl verifyagainst the CA and check the TLS listener withopenssl s_clientto the secure port. Done means each node presents a certificate that verifies and the peer port rejects connections without a valid chain.Start the nodes and confirm a leader
Start Keeper on all three hosts within the same short window so the ensemble can elect a leader. Query each node with the
mntrfour-letter command or the equivalent client status command, and check for a single leader and two followers with matchingzk_synced_followerscounts. Review the logs for election messages and unexpected connection resets. Done means exactly one node reports leader, the other two report follower, and the ensemble reports a healthy state on all three.Verify quorum and failover behavior
Confirm the ensemble can tolerate one failure by stopping the current leader in a non-production window, then checking that a new leader is elected and that a write through a client still succeeds. Restart the stopped node and confirm it rejoins as a follower and catches up. Keep the test on the Keeper hosts only, and record the commands and outputs. Done means a new leader appears after the stop, writes succeed during the outage, and the restarted node reports as synced.
Point ClickHouse servers at the ensemble
Add the three Keeper endpoints to the
zookeepersection of each ClickHouse server configuration, using the same hostnames and TLS settings, and restart one server at a time. Create a test ReplicatedMergeTree table on two replicas and confirm the Keeper path is registered and both replicas show as active in the replication status table. Done means the test table replicates an insert from one replica to the other, and the replication health columns show no errors.
Configuration differences between lab, staging and production
A lab exists to learn how Keeper behaves: how the server IDs, configuration file and log and snapshot paths fit together, how a leader is elected, and how a replicated table reacts when one node stops. Three Keeper processes can run on one or three small virtual machines, and the hardware does not need to match production. Nothing in a lab should carry data you would need to keep, and the ensemble should be built so it can be torn down and rebuilt quickly.
- Co-location is acceptable: Keeper processes and ClickHouse servers can share hosts, because the goal is understanding the protocol, not measuring it.
- Hardware can be reduced: Memory and disk well below the production guidance are fine, but keep three members so quorum behavior is real.
- Use separate ports per process: Running three Keeper nodes on one host requires distinct client and Raft ports, which makes the lab a useful rehearsal of the same settings.
- Keep names production-shaped: Use hostnames rather than raw IP addresses in the server list so the habit carries over.
- Expect data loss on rebuild: Treat lab state as disposable.
Before relying on any lab result, confirm the exact Keeper settings for your ClickHouse version in the official Keeper documentation.
Pre-production is where configuration changes, upgrades and failure tests are proven before production. ClickHouse's production guidance recommends a pre-production environment with representative data and queries, and that advice applies to Keeper as much as to the data nodes. Keeper load comes from replicated inserts, merges and DDL, so a copy with a small dataset and a few test queries will not show the coordination pressure that production creates.
- Match the topology: Three Keeper nodes on separate hosts, with the same server ID scheme, paths and TLS setup used in production.
- Match the version: Run the same Keeper and ClickHouse versions planned for production, and read the changelog for any backward-incompatible changes first.
- Replay representative load: Use sampled or synthetic data at production volume and production-like insert and query patterns.
- Rehearse failures: Stop a Keeper node, restart a ClickHouse replica and check that quorum holds and replicas resync.
- Reduce capacity, not behavior: Smaller hosts can be acceptable if the timing and failure paths are the same, but record the difference.
Ask your team to document every intentional difference from production so the results can be interpreted correctly.
The production ensemble is the one that must be correct on the first day, and it should not share hosts with the ClickHouse data nodes. ClickHouse's replication guide recommends at least three Keeper nodes on dedicated hosts in production, with about 4 GB of RAM each. Separation means a busy ClickHouse server cannot starve the coordination service, and a host reboot or disk problem affects one Keeper member rather than a data replica and a quorum member together.
- Dedicated hosts or disks: Keep Keeper off the ClickHouse data disks so log and snapshot I/O is not competing with merges.
- Three or five members: Three is the documented minimum for production; five adds tolerance for more simultaneous failures at the cost of more coordination traffic.
- Stable hostnames: Use names that resolve consistently, because membership changes are simpler when hostnames are stable.
- Encrypted coordination traffic: Secure Raft and client connections before the first production write.
- Monitoring from day one: Alert on Keeper health and leader changes, not only on ClickHouse query errors.
Before go-live, confirm the memory, disk and network sizing with your infrastructure team against the current Keeper documentation.
The differences between environments should be deliberate and written down. Some settings change Keeper's behavior, so they must match production in every environment that is used to make a decision. Others only change capacity, and those can be reduced when the test question does not depend on them. Confusing the two is the most common reason a staging result fails to predict production.
- Must match, topology: Number of Keeper members, quorum size, and how replicas are placed relative to Keeper nodes.
- Must match, configuration: Server IDs, log and snapshot path layout, TLS and Raft security settings, and the async_replication choice on the same Keeper version.
- Must match, versions: Keeper and ClickHouse versions, since behavior changes between releases.
- Can reduce, capacity: Memory, CPU, disk size and dataset size, as long as the load shape and failure tests are preserved.
- Can reduce, scope: Number of tables and users, if representative queries and DDL still run.
For each environment, ask your partner or internal owner to list every intentional difference and the test it could affect.
The point of a pre-production environment is evidence, and the production FAQ's advice to use representative data and queries is a test of whether that evidence is real. A Keeper ensemble that has only been started and seen to elect a leader has not been proven. Operators should be able to show the results of load, restart and member replacement tests on the same configuration that will run in production.
- Representative load run: Replicated inserts and merges at a volume and concurrency close to production.
- Leader loss test: Stop the current leader and confirm a new one is elected and replicas keep working.
- Replica restart test: Restart a ClickHouse replica and confirm it resyncs from a healthy peer.
- Configuration diff record: A written list of every difference between staging and production, with the reason for each.
- Upgrade rehearsal: The planned upgrade path run in pre-production first, with rollback steps recorded.
Before approving the production build, check that these results exist and that the owner of each difference has signed off.
Securing Raft and client traffic on the Keeper ensemble
Keeper holds the coordination state for every replicated table, so a compromised or misconfigured node can affect the whole cluster. The controls below cover who can reach the ensemble, how its traffic is protected, and what happens to its data on disk. Work through them before the ensemble carries production tables, and check each one on every host.
TLS for Raft and client connections
Keeper peers exchange Raft traffic with each other, and ClickHouse servers connect to Keeper as clients. Plan TLS for both paths, not only the one that is easiest to configure. Decide how certificates are issued, which certificate authority signs them, and how renewal happens before expiry. Confirm in the Keeper configuration reference which TLS settings your version supports for each port, then test that a peer with a wrong certificate is refused.
Restricting Keeper ports to known hosts
Limit the Keeper client and Raft ports to the three Keeper hosts and the ClickHouse servers that use them. Apply the rule at the host firewall and at any network security group or load balancer in front of the ensemble. Do not place a load balancer in front of Keeper peers, since each node must reach specific peers directly. Verify the rules with a connection test from a host outside the allowed list.
File ownership on log and snapshot directories
Keeper writes its transaction logs and snapshots to the paths set in the configuration. Make the service account the only writer, and keep the directories off shared mounts that other services can write to. Anyone who can alter a snapshot or log can change replicated metadata. Check ownership and permissions after each package upgrade, because package scripts sometimes reset them.
Digest ACLs on Keeper paths
Keeper znodes that hold replication metadata can be read or changed by any client that can connect unless access control is set. Configure digest ACLs for the ClickHouse paths so that only servers with the right credentials can modify them. Store the credential in the server configuration with restricted file permissions, and rotate it through a planned change. Test by connecting with a client that lacks the credential and confirming the write is refused.
Backing up snapshots and configuration
Snapshots and the configuration files for all three Keeper nodes form the recovery set for the ensemble. Copy them to storage separate from the Keeper hosts on a schedule, and keep the server IDs and hostnames recorded with each copy. A restore test should rebuild a node from the copy on a spare host. Note that a Keeper backup does not replace a ClickHouse backup of table data, which the backup and disaster recovery topic covers.
Masking secrets in Keeper logs
Keeper logs record connection activity and errors, and a misconfigured log level can echo values from the configuration. Review which fields reach the log, keep debug output off in production, and restrict who can read the log directory. Ship logs to your central store only through channels that already meet your confidentiality rules. For the wider logging and log masking controls across ClickHouse, see the ClickHouse security hardening topic.
Running the ensemble after go-live: monitoring, snapshots and upgrades
Once the ensemble is serving ReplicatedMergeTree tables, the operating question is whether Keeper is still healthy enough to coordinate replicas. Watch five signals: whether all three nodes are up and in quorum, disk latency on the log directory, growth of the snapshot directory, the frequency of leader changes, and the replication queue depth on each ClickHouse replica. Leader churn and rising latency on the log path usually show up before replicas start to lag, so alert on the trend and not only on a node going down. Keeper exposes the four-letter commands and metrics you need for these checks, and the exact commands and endpoints for your version should be confirmed in the Keeper documentation before you wire up alerts.
Upgrade Keeper one instance at a time, keeping quorum at all times, and check the version of every Keeper instance before you change any setting that depends on version. ClickHouse recommends enabling the async_replication Keeper setting only when all Keeper instances run version 23.9 or later, because it is disabled by default for backward compatibility. A mixed-version ensemble can therefore block the change, so record the version on each host, upgrade the remaining nodes, and only then flip the setting. Confirm the exact sequence and any restart requirements against the Keeper documentation for your target release, and rehearse the procedure on a non-production ensemble first.
Detailed procedures belong to other topics in this group. The ClickHouse upgrades planning guide covers rolling server upgrades across replicas and rollback, and the ClickHouse monitoring and operations topic covers metrics, system tables and dashboards. Plan a restore test for the snapshot and log directories as part of the same routine, since a Keeper node that cannot read its state after a restart is a replication outage.
Limits to confirm before you build
A Keeper ensemble is only as sound as the versions, hosts and network paths under it. The cards below list the facts to verify in your own environment before you order hardware or write configuration, because each one changes the build.
Keeper version on your release line
Confirm that every Keeper instance runs a version that supports async_replication, which the Keeper docs place at 23.9 or later. Keeper and the ClickHouse server should move together on the same release line, so check the Keeper build your chosen server version ships with. Mixed versions across the ensemble make the upgrade path harder to plan, so write down the exact build on each host before you start.
Support window for your server version
The security policy lists 26.9, 26.8, 26.7 and 26.3 as supported with security updates, and all other versions as unsupported. Confirm that the server version you plan to run appears on that list at the time of your build, and pick a target that still has a supported lifetime after go-live. A Keeper ensemble built on an unsupported line will need a rebuild sooner than expected.
Stable and LTS release cadence
Stable releases ship roughly monthly, and LTS releases ship twice a year with support for a year. Decide whether your team will follow stable releases or the LTS line, and plan how often you will schedule Keeper and server upgrades. Read the changelog before each upgrade, because minor backward-incompatible changes can occur even within a supported line.
RAM on the Keeper hosts
The replication guide recommends about 4 GB of RAM for each Keeper node and dedicated hosts in production. Check whether your candidate hosts meet that figure without sharing memory with ClickHouse data nodes, and confirm the disk used for the Keeper log and snapshot paths is separate from heavy merge I/O. If you run Keeper on shared hosts, record that decision and the contention you accepted.
Dynamic reconfiguration needs
Decide now whether you expect to replace a host or add a member later. Dynamic reconfiguration lets you change ensemble membership without a full restart, but it has to be enabled in the initial configuration and needs quorum to remain intact during the change. Use hostnames rather than IP addresses in the server list, so a replacement host does not force a rewrite of every node's config.
Ports open between Keeper nodes
Each Keeper node must reach the other two on the Raft and client ports you configure, and ClickHouse servers must reach the Keeper ports as well. Confirm that firewalls, security groups and any Kubernetes network policies allow that traffic in both directions, and that the Raft traffic is encrypted if your policy requires it. Test connectivity from each host before you start the first ensemble, since a blocked port looks like a quorum failure later.
Questions teams ask before setting up Keeper
How much effort and cost should we expect for a three-node Keeper cluster?
The cost is mostly labor, plus the hosts that run the ensemble. Three Keeper nodes are small processes, but the work around them (host provisioning, TLS certificates, configuration, replication testing and a documented restart procedure) takes skilled time. Drivers include how many environments you run, whether hosts already exist, how your change process works and how much verification you require. The one-time cost is the build and test; the recurring cost is patching, certificate renewal and version upgrades. Ask any quote to itemize build, verification, handover and post-go-live support separately, and to state which hosts and disks are assumed to be provided by you.
Can an in-house team set up Keeper without outside help?
Yes, if the team already runs Linux servers, ClickHouse and a TLS certificate process. The Keeper configuration itself is compact, but the risk sits in details: unique server IDs, hostnames that resolve consistently, separate log and snapshot paths, secure Raft traffic and a tested restart order. An in-house team usually needs someone who has seen a quorum loss before and can rehearse it. Outside help is most useful when the team is small, when the first cluster will hold regulated data, or when there is no spare environment to rehearse failures. Whichever route you take, a written runbook and a failure test should be the acceptance criteria.
How long does the Keeper setup take, and what sets the duration?
Duration depends on how quickly hosts, network rules and certificates are available, and on how many test cycles your change process requires. The configuration work is short; the schedule is usually driven by access approvals, host readiness and the number of environments that must be built in sequence (test, pre-production, production). Failure testing, which stops one node, restarts it and confirms recovery, adds time but is the step that proves the design. Ask for a timeline that lists dependencies on your side and separates the build from the verification window. Any figure should come from the written scope, not from general estimates.
What does the work need from our team before it starts?
Plan for three or more hosts with the storage and network the Keeper nodes will use, administrative access to them, DNS or hosts entries for every node name, and a certificate authority or process that can issue the certificates for Raft traffic. You will also need a change window for the first ClickHouse restart that points replicas at Keeper, and a non-production ClickHouse environment with representative tables and realistic data volumes so that replication can be tested without touching live data. Confirm who approves production changes, who holds on-call responsibility during the window, and where the configuration will be stored under version control.
Are three Keeper nodes enough, or do we need five?
Three nodes are the usual starting point because they tolerate the loss of one node while keeping quorum. Five nodes tolerate two failures and suit larger fleets or environments where a longer maintenance window on one host would be risky. The ClickHouse replication guide recommends at least three nodes on dedicated hosts for production. The decision depends on how many hosts you can dedicate, how independent their failure domains are (power, rack, availability zone) and how much extra coordination traffic the extra members add. Adding members later is possible, but it is cleaner to choose the count before the first replicated table is created.
Can Keeper run on the same hosts as ClickHouse servers?
The ClickHouse replication guide recommends dedicated hosts for Keeper in production. Co-locating the two services means a ClickHouse memory spike, heavy merge activity or a disk fill can slow or kill a Keeper node, and that can break quorum for every replicated table at once. Co-location can be acceptable in a lab or a small test environment if you accept that risk and monitor it. If you do share hosts for cost reasons, separate disks for Keeper logs and snapshots, memory limits and clear alerting are the minimum controls. Record the decision and the failure consequence in the design document.
How do we verify replication works after the Keeper setup?
Verify with tests, not with a clean startup log. Create a replicated table, insert data on one replica, and confirm it appears on the others. Stop one Keeper node and confirm inserts and reads continue, then restart it and confirm it rejoins. Stop one ClickHouse replica and confirm it resyncs from a healthy peer when it returns. Check replication health in the system tables and confirm the Keeper settings you intend to use are active on every node. Keep the results as dated evidence, because auditors and your own change board will ask for it. Confirm the async_replication setting against your Keeper version before you finalize the configuration.
Should we migrate from ZooKeeper to Keeper instead of building new?
Build new when you have no ZooKeeper dependency yet, since a fresh ensemble avoids a maintenance window and snapshot conversion. Migrate when ZooKeeper already holds replication metadata for live tables that you cannot pause or rebuild. The migration requires planning a window, pausing background tasks, converting snapshots and verifying metadata before switching, with a rollback path. The migration is a separate piece of work with its own risks, so ask for it to be scoped and priced on its own rather than folded into a new build. The decision mainly depends on the state of your existing data and the change limits you operate under.
How DeployOpen scopes and delivers the Keeper build
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure, including your cloud accounts, data centers and air-gapped networks. For a Keeper build, that means the ensemble is designed and deployed on hosts you control, with the server IDs, hostnames, log and snapshot paths, Raft settings and TLS configuration written down as deliverables that your team can inspect and rerun. Most engagements are fixed-scope, so the number of Keeper ensembles, the target ClickHouse versions, the environments covered (for example, staging before production), and the acceptance checks are agreed in writing before work starts.
Pricing follows from that written scope. DeployOpen does not publish a price list, and a quote is produced once the scope, deliverables and timelines are defined. To compare quotes fairly, ask each provider to list the hosts and disks they expect you to supply, the Keeper versions they will target, whether they will test failover and restart behavior on your environment, how they will hand over configuration and runbooks, and which items are one-time work versus recurring. Ask whether the quote includes a Keeper failure drill, since that is the step most often cut when budgets tighten.
Ongoing support, version upgrades and continuous testing are available as retainers. A retainer is useful after go-live because Keeper membership changes, ClickHouse upgrades and certificate rotation recur over the life of the cluster. Confirm with any provider what response coverage the retainer includes, who performs changes on your hosts, and how change windows are approved. DeployOpen prepares systems, controls and evidence for compliance assessments, but it does not issue audit reports, attestations or certifications; those come from independent auditors or certification bodies.
