Which coordination service fits a replicated ClickHouse cluster
A team planning ReplicatedMergeTree replicas should start on ClickHouse Keeper, and an existing ZooKeeper ensemble should stay in place until a planned stop and conversion is scheduled. ClickHouse Keeper replaces ZooKeeper for replication coordination and distributed DDL, and its client protocol is compatible with ZooKeeper, so standard ZooKeeper clients can talk to it. The two services cannot form a mixed cluster, because their snapshot, log and interserver formats are incompatible. That makes the choice a whole-cluster decision, not a node-by-node one. Keeper can run as a standalone service or as an internal part of clickhouse-server, using nearly the same configuration file. Standalone suits teams that want coordination isolated from query load, while the embedded layout suits small clusters with fewer hosts to manage. The Keeper docs use a three-node example that keeps working if one node crashes, and the ClickHouse replication guide recommends at least three Keeper nodes on dedicated hosts in production. Confirm with your own capacity plan whether embedded Keeper leaves enough memory and disk headroom for the server process, since the guide's figure of about 4 GB of RAM per Keeper node is a starting point, not a sizing result.
If ZooKeeper already coordinates other systems, such as Kafka or HBase in the same environment, keep that ensemble for them and run a separate Keeper cluster for ClickHouse. Moving ClickHouse off a shared ensemble is a planned change, because the ClickHouse Keeper docs say seamless migration is not possible. The ZooKeeper cluster must be stopped, its logs and snapshots converted with clickhouse-keeper-converter (ZooKeeper 3.4 or later), and Keeper started. Plan a maintenance window, rehearse the conversion on a copy of the data directories, and confirm the exact procedure for your ZooKeeper version in the ClickHouse Keeper guide before scheduling. Ask your vendor or platform owner whether any other service depends on the same ensemble, because stopping it affects every client.
ClickHouse Keeper and ZooKeeper side by side
This table compares the two coordination services a replicated ClickHouse cluster can use, from language and consensus design to migration. Rows the supplied facts do not settle are marked for confirmation with the vendor, so read those as open questions for your own evaluation.
| Factor | ClickHouse Keeper | Apache ZooKeeper |
|---|---|---|
| Implementation language and consensus | Written in C++ and uses the Raft algorithm | Java service; confirm the consensus details against the ZooKeeper project documentation |
| Client protocol | Compatible with the ZooKeeper client protocol, so standard ZooKeeper clients work with it | The reference protocol that Keeper clients are built to match |
| Deployment modes | Runs standalone as a ZooKeeper replacement or inside clickhouse-server with a nearly identical config file | Runs as a separate ensemble; confirm any embedded options with the ZooKeeper project |
| Node count guidance | Docs' three-node example keeps working if one node crashes; replication guide recommends at least three dedicated nodes in production | Confirm the ensemble size guidance for your ZooKeeper version and load |
| Default consistency | Linearizable writes; reads are not linearizable by default, as with ZooKeeper | Linearizable writes; non-linearizable reads by default |
| Format compatibility | Snapshot, log and interserver formats are incompatible with ZooKeeper | Native formats; a mixed ZooKeeper and Keeper cluster is not possible |
| Migration path | No seamless migration: stop ZooKeeper, convert logs and snapshots with clickhouse-keeper-converter, then start Keeper | Source side of the conversion; ZooKeeper 3.4 or later is the supported input |
| Who maintains the software | ClickHouse project, under the Apache-2.0 licensed ClickHouse server repository | Apache Software Foundation project; confirm current maintainers and release support with the project |
How Keeper standalone and embedded layouts differ
A replicated ClickHouse cluster needs a coordination quorum whichever way Keeper is deployed. The layout you choose decides what shares a failure domain, what competes for memory and CPU, and when Keeper can be upgraded without touching the database servers.
Standalone Keeper on dedicated hosts
Keeper runs as its own service on separate machines, and ClickHouse servers point to it as their coordination layer. This isolates coordination from query load, so a heavy merge or large aggregation on a ClickHouse server cannot starve the quorum. It also lets you upgrade Keeper on its own schedule. The guidance is dedicated hosts in production, so confirm with your platform team that these machines are not shared with other workloads before you commit to this layout.
Keeper embedded in clickhouse-server
Keeper can run as an internal part of the ClickHouse server process, with nearly the same configuration file as the standalone service. This removes a set of hosts and a separate service to patch, which suits a small estate. The trade-off is shared failure and resource domains: a server that is restarted, overloaded or upgraded also affects its coordination role. Use this layout only when you have confirmed that the Keeper nodes do not depend on a server you plan to take down for maintenance.
Three-node quorum survives one failure
The ClickHouse documentation's three-node example keeps working when one node crashes, because the remaining two still form a majority. Two simultaneous failures stop the quorum, and replicated tables then stop accepting coordinated writes until it recovers. Before you rely on this, check that the three nodes sit in separate failure domains, such as different racks or availability zones, because a quorum spread across one failure domain offers little protection. Confirm the placement with your infrastructure team.
Sizing dedicated nodes for memory
The replication guide recommends dedicated hosts for production Keeper nodes, with about 4 GB of RAM each. Keeping the coordination service off the query path protects its memory from analytical workloads that can spike during large merges or queries. Ask your team to size disks for the Keeper log and snapshot directories and to monitor them, since coordination data is small but must be durable. Treat 4 GB as a starting point from the documentation, and verify it against your replica and table counts.
ZooKeeper ensemble stays separate
In a layout that already runs Apache ZooKeeper, the existing ensemble keeps serving ClickHouse replication until a planned change. ZooKeeper's snapshot, log and interserver formats differ from Keeper's, so the two cannot form a mixed cluster. The ensemble is therefore a separate failure domain, and its upgrade timing belongs to whoever runs it. Plan the switch to Keeper as a full cutover with a stopped ZooKeeper cluster, a conversion with clickhouse-keeper-converter, and a restart, not as a rolling change.
Which coordination path fits each team situation
If you are building a replicated ClickHouse cluster and do not already run ZooKeeper for it, ClickHouse Keeper is the coordination service to plan around. It replaces ZooKeeper for ReplicatedMergeTree replication and distributed DDL, so you avoid adopting a Java ensemble you would otherwise have to patch, monitor and tune alongside ClickHouse. Choosing Keeper at the start means no stop-and-convert step ever happens.
- Node layout: The replication guide recommends at least three Keeper nodes on dedicated hosts in production, with about 4 GB of RAM each. Confirm the layout with your own capacity planning before you commit to hardware.
- Standalone or embedded: Keeper can run as a separate service or inside clickhouse-server with nearly the same configuration file. Decide this during design, because it affects failure domains and patch windows.
- Restore testing: Replication does not protect against human error or corruption, so schedule backup restores to a spare cluster before go-live.
- Version discipline: Pick a stable line or LTS line that is on the current support list and read the changelog before each upgrade.
What would make this choice wrong: a requirement to share one coordination ensemble with services that only run on ZooKeeper, which Keeper cannot join because the formats are incompatible. Confirm that with your own dependency inventory before you start.
If a ZooKeeper ensemble already runs for other systems your team depends on, such as Kafka, the question is whether ClickHouse should keep sharing it or move to its own Keeper cluster. Keeper speaks the ZooKeeper client protocol, so clients connect without rewrites, but a mixed ZooKeeper and Keeper cluster is not possible. Moving ClickHouse off a shared ensemble is therefore a planned cutover, not a live change.
- Shared ensemble stays: If other services need the existing ensemble, keep it and point ClickHouse at it. Nothing changes for them, and you avoid a second coordination service to run.
- Dedicated Keeper: If ClickHouse coordination should be isolated from other services, plan a separate Keeper cluster and a conversion window. The ClickHouse Keeper documentation says the ZooKeeper cluster must be stopped, its logs and snapshots converted with clickhouse-keeper-converter (ZooKeeper 3.4 or later), and Keeper started.
- Change window: The conversion needs a window you have agreed with every consumer of that ensemble, not only the ClickHouse team.
- Backup and rollback: Keep a verified backup of the ZooKeeper data directories and a tested rollback path before the stop.
What would make this choice wrong: attempting a rolling move while other services still depend on the same ensemble, or converting without a rehearsal on a copy of the data.
For a small test or pre-production environment, the coordination choice matters less than matching the production path. The goal is to exercise replication, failover and restore on representative data and queries, so that production surprises are rare. The ClickHouse production guidance recommends a pre-production environment with representative data and queries before any upgrade or topology change.
- Match the production topology: If production will use Keeper, run Keeper in test. If production will keep a shared ZooKeeper, rehearse the conversion against a copy of that ensemble.
- Lower sizing is acceptable: Test hosts do not need the dedicated-host layout of production, but treat any shortcut as a difference you must document.
- Failure drills: Stop one Keeper node or one replica and confirm that a recovered replica syncs missing data from a healthy replica without manual steps.
- Restore drills: Run a full restore to a spare cluster and check the results, since the backup documentation warns that an untested backup will probably not restore when needed.
What would make this choice wrong: using a single-node test setup to make a production decision about replication or coordination. Ask your ClickHouse support or consultant to confirm which behaviors a single node cannot show.
Why protocol compatibility does not mean a mixed cluster
Standard ZooKeeper clients can talk to ClickHouse Keeper, but a single cluster cannot run both services at once. The client protocol is compatible, so applications and drivers that open a ZooKeeper session generally work against Keeper without code changes. The snapshot, log and interserver formats are different, however, so a ZooKeeper node and a Keeper node cannot form one quorum. Compatibility covers the conversation with clients, not the internal state a cluster keeps. For a rolling change, that means there is no half-migrated state to run in production: the move is a planned stop, conversion of the ZooKeeper logs and snapshots with clickhouse-keeper-converter, and a Keeper start. Plan the window, the rollback path and the conversion test before anyone schedules it. The ClickHouse Keeper documentation sets out the procedure.
The protocol match also says nothing about tooling that reads coordination state directly. Monitoring agents that scrape ZooKeeper four-letter commands, admin scripts that inspect znodes, and backup jobs that copy the data directory may assume ZooKeeper's on-disk layout or command set. Confirm with each owner which client libraries, exporters and scripts in your estate depend on ZooKeeper specifics, and test each one against a Keeper node in a non-production cluster. Also confirm how your team's metrics and alerts map to Keeper's output, since dashboards built for ZooKeeper may show gaps rather than errors.
Keeper's default behavior matches ZooKeeper's: linearizable writes with non-linearizable reads. Application teams that already tolerate ZooKeeper's read semantics will see the same guarantees, so the migration risk sits in the operational steps rather than in query behavior. Test the move on a copy of production metadata, time the stop and conversion, and run a restore check on the converted cluster before cutover, because a conversion that has never been exercised is an untested recovery path.
How a ZooKeeper to Keeper move runs as a planned stop
This sequence turns a ZooKeeper-coordinated ClickHouse cluster into a Keeper-coordinated one through a scheduled outage, with a verified backup and evidence recorded at each gate. Seamless migration is not possible according to the Keeper docs, so the planned stop is the mechanism, not a fallback.
Confirm a current backup and a tested restore
Before any change, take a fresh BACKUP of the tables that matter and store it outside the cluster. The ClickHouse backup documentation notes that replication alone does not protect against human error or corruption, so replicas are not a substitute. Restore that backup to a spare cluster and run row counts and key queries against it. The ClickHouse backup documentation warns that a backup never test-restored will probably not restore properly when needed. Record the backup name, timestamp, storage location and the restore test results as evidence. Do not proceed until a restore has succeeded.
Schedule the window and notify dependent teams
Agree a maintenance window with the owners of every application that writes to ReplicatedMergeTree tables, because writes will stop while coordination is unavailable. Confirm the rollback owner and the decision point for abandoning the move. Ask the ClickHouse support or implementation partner to confirm the exact ZooKeeper version in use, since the converter supports ZooKeeper 3.4 or later. Record the approved window, the named approvers and the rollback criteria in the change ticket.
Stop ClickHouse writes and the ZooKeeper cluster
Stop ingestion and DDL against the replicated tables, then stop the clickhouse-server processes that depend on the ZooKeeper ensemble. Once the servers are quiet, stop the ZooKeeper nodes. Keep the ZooKeeper data directories intact, because they are the input to conversion and the rollback path. Check that no ZooKeeper process remains running and that no replica is still accepting inserts. Record the stop time and the process check output in the change log.
Convert ZooKeeper logs and snapshots
Run clickhouse-keeper-converter against the stopped ZooKeeper data directories to produce Keeper-format snapshots and logs. The Keeper docs state that Keeper's snapshot, log and interserver formats are incompatible with ZooKeeper, so this conversion is required and mixed clusters are impossible. Keep the original ZooKeeper data copied to a separate location before conversion. Verify that the converter completed without errors and that the output files exist on each intended Keeper host. Record the converter version, command used, input paths and output checksums.
Start Keeper with a quorum of nodes
Start Keeper either as a standalone service or inside clickhouse-server, using a configuration that is nearly the same as the previous setup. The replication guide recommends at least three Keeper nodes on dedicated hosts in production, with about 4 GB of RAM each. Confirm the nodes form a quorum and that a leader is elected before going further. Check the Keeper logs for errors during snapshot loading. Record the node list, quorum status and any warnings from startup.
Point replicas at the new Keeper nodes
Update the ClickHouse server configuration so every replica points to the Keeper ensemble rather than the ZooKeeper hosts, then restart the servers in a controlled order. Because Keeper speaks a ZooKeeper-compatible client protocol, the change is in the connection settings and the target hosts. Check that each replica reconnects and reports its replication queue state. Record the configuration diff and the reconnection status for each replica.
Verify replication and distributed DDL
Insert test rows on one replica and confirm they appear on the others. Run a distributed DDL statement across the cluster and confirm every host applies it. Check that a recovered replica syncs missing data from a healthy replica without manual steps, which the replication guide describes as automatic. Compare row counts and checksums for key tables against the pre-move baseline. Record the test queries, results and any replication lag or queue errors as the acceptance evidence.
Retain rollback material and close the change
Keep the stopped ZooKeeper data, the converted Keeper files and the pre-move backup until the team has observed normal operation for the agreed period. Confirm the monitoring and alerts for Keeper are active, and that the restore runbook references the new coordination endpoints. Hand the evidence pack to the change owner, including backup and restore results, converter output, quorum status and verification queries. Close the change only after the owner signs off against the rollback criteria set in the scheduling step.
What running each option asks of your team
This table compares what each coordination option asks of your operators across upgrades, backups, monitoring, recovery and skills. Read it as a checklist of work you will own after the cluster is built, and confirm each row against your own environment and vendor documentation.
| Factor | ClickHouse Keeper | Apache ZooKeeper |
|---|---|---|
| Upgrades and release cadence | ClickHouse ships stable releases roughly monthly and LTS releases twice a year, with LTS supported for a year. Keeper versions with the server, so plan upgrades around the ClickHouse line you run. Read the changelog first, since minor backward-incompatible changes can occur. | Your ZooKeeper upgrade follows the ZooKeeper project's own release line, separate from ClickHouse. Confirm the version your ClickHouse cluster requires and the support window of that line with your vendor or distribution. |
| Supported-version policy | ClickHouse's security policy lists 26.9, 26.8, 26.7 and 26.3 as supported with security updates. Confirm that your running version appears on that list before you schedule any work. | Confirm the supported-version policy of the ZooKeeper distribution you deploy. Your ClickHouse version and your ZooKeeper version each need a separate support check. |
| Backups and restore testing | ClickHouse BACKUP and RESTORE cover table data, but replication does not protect against human error or corruption. Coordination state is not your data backup, so test restores of table data on spare clusters on a schedule. | Keep ZooKeeper snapshots and logs under your own backup routine. Converting to Keeper requires a stopped cluster and a converted copy of those files, so rehearse that conversion before any cutover. |
| Monitoring and alerting on quorum health | Watch Keeper node availability, leader state and replica lag at the ClickHouse layer. Confirm which Keeper metrics your monitoring stack can scrape before go-live. | Watch ZooKeeper ensemble health and client session behavior with your existing tooling. Teams already running ZooKeeper for other systems usually have these alerts in place. |
| Failure recovery | The replication guide says a recovered replica syncs missing data from a healthy replica automatically. The docs' three-node example keeps working if one node crashes. Test node loss and replica recovery in a pre-production environment. | Recovery procedures are your responsibility. Document quorum loss handling and rehearse it, since a ZooKeeper cluster and its ClickHouse clients must both recover cleanly. |
| Node count and hosting | The replication guide recommends at least three Keeper nodes on dedicated hosts in production, with about 4 GB of RAM each. Confirm sizing against your query and ingest volume. | Plan an odd-numbered ensemble on dedicated hosts sized to your workload. Confirm node count and resource needs with your ZooKeeper distribution's guidance. |
| Skills needed | Linux operations, Keeper configuration with a file close to the server configuration, and ClickHouse replication and DDL behavior. Teams often need someone who understands ReplicatedMergeTree and Keeper together. | Linux operations and ZooKeeper configuration and administration, plus ClickHouse replication knowledge. Keeping an existing ZooKeeper team is an advantage only if they also understand the ClickHouse side. |
Questions teams ask before choosing Keeper or ZooKeeper
Can ClickHouse Keeper run inside clickhouse-server?
Yes. ClickHouse Keeper can run either as a standalone replacement for ZooKeeper or as an internal part of the ClickHouse server, with a configuration file that is nearly the same in both layouts. The choice is about failure domains and operations, not features. Running Keeper embedded keeps the number of processes you manage lower, but it ties coordination to the same hosts that serve queries. Standalone Keeper on dedicated hosts isolates coordination from query load, which is the layout the replication guide recommends for production. Confirm the embedded layout against your own host sizing before you commit to it.
How many Keeper nodes should a replicated cluster run?
The ClickHouse replication guide recommends at least three Keeper nodes, on dedicated hosts in production, with about 4 GB of RAM each. Three nodes form a quorum that keeps working if one node crashes, which matches the three-node example in the Keeper docs. Adding nodes does not add throughput for ClickHouse data, because coordination traffic is small compared with data movement. The count is driven by how many failures you need to survive and how many hosts your operations team can keep patched and monitored. Ask for the node count and host sizing to be written into the scope.
Can a ZooKeeper cluster and a Keeper cluster run together?
No. According to the Keeper docs, Keeper's snapshot, log and interserver formats are incompatible with ZooKeeper, so a mixed cluster is not possible. Keeper does speak the ZooKeeper client protocol, so standard ZooKeeper clients can connect to it, but that compatibility covers clients, not peers. Each cluster must be entirely one or the other. This is why the migration needs a planned stop rather than a rolling swap of nodes.
Does moving from ZooKeeper to Keeper require downtime?
Yes, the replication coordination must stop. The Keeper docs say seamless migration from ZooKeeper is not possible: the ZooKeeper cluster has to be stopped, its logs and snapshots converted with clickhouse-keeper-converter, and Keeper started. Replication and distributed DDL depend on coordination, so plan a maintenance window and rehearse the steps on a copy of the cluster first. The length of that window depends on your data layout, change volume and checks, so ask the implementer to state the window and rollback steps in writing.
What does the conversion tool need as input?
The clickhouse-keeper-converter tool takes the logs and snapshots from a stopped ZooKeeper cluster, and the Keeper docs state it works with ZooKeeper 3.4 or later. You need a consistent set of those files from a cleanly stopped ensemble, and the output then feeds a new Keeper cluster. Confirm your ZooKeeper version and data directory layout before scheduling, and verify that the converted state matches the source before pointing ClickHouse servers at it. The inventory of versions and paths belongs in the written scope.
What does the move cost, and who pays for it?
The cost has three parts: the implementer's planning, rehearsal and cutover work, the hardware or virtual machines for dedicated Keeper hosts, and the internal time your engineers spend on reviews and sign-off. The first is one-time, while hosts and monitoring recur. DeployOpen does not publish prices, and a quote follows from a written scope that lists the cluster size, number of Keeper nodes, rehearsal environment and rollback plan. Ask who owns each host, who runs the cutover and who is on call during the window. Compare quotes on those lines, not on a total alone.
Can an in-house team do the Keeper migration alone?
A team that already runs ClickHouse replication, has tested backups, and can rehearse a stop and conversion on a non-production copy can plausibly do it alone. A team without those skills should weigh the risk of a failed cutover against the cost of outside help. The migration touches replication, distributed DDL and recovery, and the backup documentation warns that an untested backup will probably not restore when needed. Check whether your team can restore to a spare cluster before deciding. DeployOpen plans, implements and documents this kind of work under a fixed-scope agreement if you prefer to hand it over.
How long does the migration take, and what drives it?
The supplied facts do not give a duration, so treat any number you hear as unverified until it is tied to a scope. The main drivers are the number of ClickHouse tables and replicas, the size of the ZooKeeper data to convert, how many environments you must rehearse in, the change freeze you can accept, and how many people must approve the cutover. Ask for a written timeline with separate phases for planning, rehearsal, cutover and post-cutover checks, plus the exact conditions that would extend it.
What drives the cost of coordination choices
The cost of choosing between ClickHouse Keeper and ZooKeeper is mostly labor and risk, not license fees. Both are open source, but the supplied facts do not state the license terms of ZooKeeper or ClickHouse Keeper in detail, so confirm the terms for your exact versions before budgeting. The larger cost lines are hardware or virtual machine capacity for dedicated coordination hosts, engineering time for the migration, and the outage exposure created by a planned stop. The ClickHouse replication guide recommends at least three Keeper nodes on dedicated hosts in production, with about 4 GB of RAM each, so ask your infrastructure team what three dedicated hosts or VMs cost in your environment, including monitoring and backup of those nodes.
Migration is one-time work and it is the expensive part. The ClickHouse Keeper documentation says seamless migration from ZooKeeper is not possible: the ZooKeeper cluster must be stopped, its logs and snapshots converted with clickhouse-keeper-converter, and Keeper started. That means engineering time to plan the window, rehearse the conversion on a copy of production data, script the cutover, and prove that replicas resynchronize afterward. Replicas that recover automatically sync missing data from a healthy peer, which reduces manual repair, but you still need to measure how long your cluster is unavailable for writes during the stop. Price that outage against your business calendar, not an assumed duration.
Recurring work is what decides the total over several years. Each ClickHouse upgrade, whether to a stable line or an LTS line, needs changelog review, staging tests and a rollback plan. The backup documentation warns that a backup never test-restored will probably not restore when needed, so restore tests on a spare cluster repeat on a schedule that you set. In a quote from DeployOpen, ask for the following: a written scope covering the conversion, rehearsal and cutover; the number of environments in scope; who performs restore tests and how often; upgrade support across the versions you plan to run; and the handover documentation your own team will maintain. DeployOpen does not publish a price list, and a quote follows from that written scope.
How DeployOpen scopes and delivers a coordination move
DeployOpen treats a move from ZooKeeper to ClickHouse Keeper as a planned change to a running cluster, not a configuration swap. We scope the cutover window, the conversion and the rollback path in writing before any work starts, inside your own infrastructure. Engagements are fixed-scope, with ongoing support available as a retainer.
How we scope
- Inventory every ReplicatedMergeTree table, the current ZooKeeper ensemble size and each replica's role before agreeing a plan
- Agree in writing which Keeper layout you need: standalone three-node ensemble or Keeper inside clickhouse-server
- Define the stop window, who approves it and what rollback looks like if conversion fails
- Confirm which backup and restore tests must pass on a spare cluster before cutover is approved
How we build
- Stand up Keeper on dedicated hosts sized to the documented minimum of about 4 GB RAM per node
- Rehearse the ZooKeeper stop, clickhouse-keeper-converter run and Keeper start on a non-production copy
- Verify client compatibility for any tooling that talks to ZooKeeper directly before the live move
- Test a node crash in the three-node layout to confirm the cluster keeps serving replicas
How we hand over
- Document the final topology, configuration keys, conversion steps and the exact cutover runbook used
- Deliver restore test results from a spare cluster, since an untested backup is unlikely to restore when needed
- Walk your operators through replica recovery, Keeper health checks and the upgrade path for your chosen ClickHouse line
- Record which ClickHouse release line you run and the changelog review required before each upgrade
How we support
- Offer ongoing support, upgrades and continuous testing as a retainer after handover
- Review each ClickHouse stable or LTS upgrade against your Keeper configuration before it is applied
- Repeat restore tests on a schedule agreed in the retainer scope, not left to chance
- Confirm with your team which support window your ClickHouse release line falls under at each review
